Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new automatic metric called SMART for evaluating model-generated text. Here, sentences are used as basic units of matching instead of tokens, in order to support long and multi-sentence texts. For this, the authors use a sentence-level soft matching function, and experiment with both string-based and model-based matching functions to select the best variants of SMART. Similar to ROUGE, two types of SMART are proposed, SMART-N which is based on sentence-level n-gram overlap, and SMART-L which is based on longest common soft-subsequence. In order to support grounded evaluation, the authors also include the source documents during metric calculation. Results presented in Table 1 demonstrate that SMART, with BLUERT as the soft-matching function, has better correlation scores, with human judgements, than existing model-based metrics across all four quality dimensions of coherence, fluency, informativeness, and factuality. SMART (SMART-CHRF) also achieves comparable evaluation results with previous LM-based metrics even without using any pretrained language models. Ablation results demonstrate the advantage of using source documents in metric calculation, not only for factuality dimension, but for other dimensions as well. Results presented in the appendix further demonstrate that SMART performs better as summary length increases and is less biased than other competing metrics.

strength_and_weaknesses: Strengths
The paper is well-written and well-structured.
Using sentences as basic units of matching, although not new, is well-motivated.
The proposed metric is defined well, although it would be better if the authors could explain SMART-L more clearly, maybe using an example (in the appendix).
Experiments are well-defined and results are clearly presented.
SMART not only outperforms metrics for evaluating content quality of model-generated summaries, but also for evaluating their factual consistency. This gives SMART an added advantage over existing metrics.

Weaknesses
Some results, especially in Table 2, are difficult to follow. Text summarization is more intuitive when it is abstractive as it is closer to how humans summarize a piece of text. Reference summaries on CNN/DM are also abstractive. I believe the majority of the model-generated summaries used for analysis in the SummEval benchmark (Fabbri et al. (2021)) are also abstractive. In such a case, how is source-only SMART performing better than ref-only SMART? It seems counterintuitive to me that abstractive sentences in model-generated summaries would match better with source document sentences. Please explain.
How does using the source documents for SMART calculation improve the correlation scores for non-factuality quality dimensions?
In Table 2, how can the minimum of src-only and ref-only correlation scores be more than the lower of the two? Am I missing something?
Although not a weakness, but can SMART be used in more dynamic settings? For example, consider task-agnostic dialogue generation, where there can be multiple possible responses based on the given context, and also where the response to be generated might depend not only on the immediate context but also on the history of previous utterances.

clarity,_quality,_novelty_and_reproducibility: Clarity
The paper is very clearly presented, with adequate details included in the main draft to propose the main ideas. Extra details and results are also neatly presented in the appendix.
Quality
The authors perform adequate experiments to first justify their choices for the soft-matching functions. Thereafter, exhaustive experiments are performed to demonstrate the advantage of the proposed metric in different settings. While I would like the authors to address some questions, overall, I am satisfied with the quality of the presented results.
Novelty
The work is novel. However, the idea of using source-documents for metric calculation has been investigated in the past. Also, given that the model-generated summaries are majorly abstractive, it is not quite clear why using source documents, rather than reference summaries (which are again abstractive in case of CNN/DM), gives better scores especially for quality dimensions other than factuality.
Reproducibility
Reproducing the results could be challenging as codes are not released yet.

summary_of_the_review: Overall, the paper is a nice read. The proposed metric SMART is well-motivated, well-defined, and shows better correlation scores with human judgements than existing summary-evaluation metrics on four different quality dimensions, namely coherence, fluency, informativeness, and factuality. SMART is flexible, as different soft-matching functions can be selected depending on the task. It also performs well in reference-free settings. The authors demonstrate that SMART performs better when the length of summaries increase. However, in the majority of summarization tasks, summaries are meant to be concise. Hence, this finding may not be practically important. Also, some observations, highlighted under “Weaknesses”, are difficult to follow. I would encourage the authors to address the questions asked in order to make the submission stronger. Overall, the proposed metric is a welcome value addition to the existing literature on model-generated text evaluation metrics.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors proposed SMART, a new automatic evaluation metric text generation models.
- Proposal 1: To deal with multiple sentences, they proposed SMART-N and SMART-L, in which the calculation unit of ROUGE-N and ROUGE-L is changed from words to sentences. In SMART, several metrics such as BLEURT and ANLI are used to softly compute matching scores between sentences.
- Proposal 2: To exploit the source document, they proposed to use the maximum value of "similarity between candidate text and reference text" and "similarity between candidate text and source text".

Experiments on SummEval (document summarization) and TRUE benchmark (factual consistency) showed that the proposed method performs competitively when using a specific matching function that was selected for each task.
strength_and_weaknesses: Strengths:
- Both proposals are intuitive and easy to reproduce.
- The existence of an ablation study (§5.1) makes it easy to understand the nature of the proposed method.

Weaknesses:
- The performance of the proposed method is not robust to the choice of matching function. In fact, on SumEval (§5.1), the proposed method shows high performance with BLEURT, while it shows low performance with ANLI (Table 4). On the other hand, on the TRUE benchmark (§5.2), the proposed method shows high performance with ANLI, while it shows low performance with BLEURT (Table 7).
  - Existing methods should not be described as "very slow" (Introduction); the proposed method also uses LM-based metrics similar to existing methods, e.g., BLEURT. Moreover, the proposed method in this paper has the additional cost that the reader needs to try the number of matching functions for each data set.
- In the paper, the first proposal (sentence-by-sentence computation) is given a large amount of space, but in practice, the second proposal (comparison of generated text and source document) appears to be more effective. In fact, the performance of the first proposal alone is very low (`ref-only` in Table 2). The readers would benefit from an ablation study in which only the second proposal is applied to existing methods such as ROUGE, BLEURT, and ANLI. It would also be very attractive to present concrete examples where the treatment of the sentence-by-sentence works and add qualitative evaluation.
  - Although the authors consider the proposed method as a general-purpose evaluation metric for text generation models, it cannot be applied to machine translation. This is because the proposed method requires calculating the similarity between the source and the candidate. The authors should weaken their argument and argue that they proposed an automatic evaluation metric for automatic summarization. In any case, additional experiments on the TRUE benchmark is very attractive.
clarity,_quality,_novelty_and_reproducibility: This paper has many points that could be improved in terms of paper quality. Since there are likely many points that can be pointed out and resolved within the authors, it would be desirable to submit a paper in a more complete state. For example:
- If the equations for ROUGE-N and ROUGE-L were shown before SMART-N and SMART-L, the difference (sentence-by-sentence soft match) would become clearer and self-contained would increase.
- In Table 3, rows and columns are inverted. The reader would be easier to read if the methods are always placed on the row-side.
- Without a shorter acronym, the name of the proposed method is much easier to read.

SMART-1 is similar to Sentence Mover's Similarity (Clark et al., ACL 2019); both of which use a soft similarity per sentence. However, there is no experimental comparison or qualitative evaluation, and PROs/CONs are unknown.
summary_of_the_review: The authors propose a new evaluation metric named SMART for text generation models. Both proposals are intuitive and simple, and cross-sectional experiments suggest empirical effects.

On the other hand, the effectiveness of the main proposal is somewhat questionable, and the paper is not complete; it is considered difficult to be accepted by ICLR.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper is about an automatic reference-based evaluation metric which evaluates the quality of generations with respect to the source text and multiple references by considering sentences as the base units of evaluations and leveraging different matching methods to compute the similarity between sentences. Authors have shown that the proposed metric has better performance even for the evaluation of long summaries and also it is less biased.
strength_and_weaknesses: The contribution, idea, motivation and results from experiments are written very neatly.

There are quite a comprehensive set of experiments (comparisons to different baselines and functions) to show the effectiveness of the proposed metric.

According to the experiments the metric can be easily adopted to different tasks and domains which can be one of the major positive impact of this metric.

Overall, I do think the paper does not have any major flaws that prevent it from being accepted. Some minor concerns for me is: what was the motivation that authors tried to consider sentences as the units of the evaluation, explaining it can be beneficial. Also, is it possible to show some results of the metric on other reference-based evaluation tasks such as translation?


clarity,_quality,_novelty_and_reproducibility: The clarity of the work and its contributions can be clearly received from the paper. According to the provided details in the paper most of the results seem to be reproducible. The novelty of the work come from the substitution of metric's base unit with sentences and analyzing different matching metrics' effects on the metric's performance. Also low risk of bias introduced by the metric and its good performance for evaluating long summaries are very beneficial attributes that should not be neglected.
summary_of_the_review: This paper is in good shape and the idea behind the metric is interesting and very easily applicable in different domains. The effectiveness of the metric to evaluate long summaries without adding bias is the major benefit of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes SMART, a new metric for conditional text generation (especially summarization) evaluation. The proposed method mainly consists of two components: sentence-level soft-matching and source inclusion. For sentence-level soft-matching, it splits the prediction and reference into separate sentences. The single sentence pair similarities are computed using existing metrics (e.g. BLEURT, CHRF, etc.) The sentence-level similarities are aggregated in ways inspired by ROUGE, leading to different method variations, SMART-N (based on sentence N-gram) and SMART-L (based on longest common sequence, LCS). For SMART-L, a dynamic programming algorithm is proposed to find the best soft-alignment between prediction sentences and reference sentences. For source inclusion, it essentially treats the source (input) text as another reference, and aggregate the source score with other reference scores by taking their maximum.

The authors conducted evaluations on two meta-evaluation datasets: SummEval, for general summary evaluation; and TRUE, for summary factual verification. Experiments show that using SMART on existing good-performing methods can further improve the performance and achieves overall the best performance compared to other methods. Ablation studies are also conducted to verify different components in the method, and provide more insights into the method behaviors.
strength_and_weaknesses: #### Strength

- Evaluation metrics for summarization is a highly important topic, especially because the currently most widely-used metrics like ROUGE has been found to correlate poorly with human evaluation.

- The proposed method has non-trivial novelty (mostly in the sentence-level soft-matching component), and has exhibited good performances in the experiments.


#### Weakness
- As stated above, the novelty mostly resides in the sentence-level soft-matching techniques. However, the ablation studies did not effectively verify the usefulness of it. The ablation studies showed the usefulness precision & recall, source & reference, and max aggregation, yet it is possible that the performance gain mostly comes from source inclusion but not sentence-level soft-matching. This is likely, given the poor performance of the "ref-only" setting and relatively better performance of "src-only", implying that source is more helpful than references. A valuable (and hopefully easy) study would be to use existing methods, like BLEURT, ANLI, etc., *with* source inclusion but *without* sentence-level soft-matching. If it consistently underperforms the full version of SMART, that verifies the usefulness of sentence-level soft-matching.

- The improvement of SMART is not always consistent; it does not seem to improve all methods. For example, based on Table 1 and Table 4, it seems to me that it only improves ROUGE-L, BLEU, BLEURT and BertScore, but hurts ROUGE-1/2 and ANLI. Although it's good to be able to improve on a current SOTA method and establish a new SOTA, it becomes questionable whether the method generalizes to future SOTA methods. (By the way, I think it would be better to put information in Table 1 also in Table 4 so that it's easier to compare.)


Below are my questions, not necessarily weaknesses:
- Some results in Table 2 are quite hard for me to understand. Mainly, SX-BLEURT ref-only gets extremely bad performance (negative correlation with human on all aspects), but BLEURT without SMART is not as bad. Also, even though ref-only has negative correlation, with source added, including the references can still largely improve the performance. How would you interpret these results?

- A more high-level question: how do you justify the idea of using sentences as units in summary evaluation? As I imagine, there can be cases where information is grouped differently among sentences in prediction and reference (e.g. predicted sentences = ["A", "BC"]; reference sentences = ["AB", "C"]. The prediction is perfect, but all sentence-pair matching scores are imperfect, so the given score will be imperfect). In such cases, using sentence as units will not be better than using tokens as units.


Below are some minor points:
- Many argued strengths of the method, such as good for longer summaries, less biases, good reference-free performances, etc., are only elaborated in the appendix.

- (typo) page 9, above "Results": "SZ_{ZS}" should be "SC_{ZS}"
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is overall clearly written.

Quality: there are some concerns about experiment design, as stated above.

Novelty: the idea of sentence-level soft-matching for summarization evaluation carries non-trivial novelty. For source inclusion, the idea is mostly imported from previous work.

Reproducibility: the method does not involve process with significant randomness (e.g. neural model training), and details of methods in the experiments (proposed and compared) are provided, therefore should be highly reproducible.
summary_of_the_review: This paper proposes SMART, a new metric for summarization evaluation based on a novel technique, sentence-level soft-matching, and source inclusion. Experiments show the good performance of the method. However, the study was not rigorously done to verify the usefulness of the novel part. Also, the improvement is not fully consistent among different based methods. The work is overall interesting, but more work should be done to make it more concrete and strong.

(Scores updated after rebuttal)
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new automatic metric, SMART, for evaluating the quality of generated text w.r.t. multiple references and the source text. The metric uses sentences as the base units of evaluations (instead of tokens as commonly used in previous metrics). The metric plugs in existing metrics (e.g., BLEURT) to compute the similarity between sentences. The proposed metric shows better performance for evaluating long-form generations such as summaries. Ablation studies are provided. Overall the idea is simple and intuitive, and the results are positive. Reviewers have suggested more comparisons/analyses, such as the approach of adding source text as one of the references. 


justification_for_why_not_higher_score: Intuitive (non-surprising) idea. More comparison is desired.
justification_for_why_not_lower_score: Simple method with good performance.
Decision: Accept: poster