SMART: Sentences as Basic Units for Text Evaluation
Authors: Reinald Kim Amplayo
Peter J Liu
Yao Zhao
Shashi Narayan
Venue: ICLR 2023 poster (2023)
Keywords: summarization
evaluation

Abstract: Widely used evaluation metrics for text generation either do not work well with longer texts or fail to evaluate all aspects of text quality. In this paper, we introduce a new metric called SMART to mitigate such limitations. Specifically, we treat sentences as basic units of matching instead of tokens, and use a sentence matching function to soft-match candidate and reference sentences. Candidate sentences are also compared to sentences in the source documents to allow grounding (e.g., factuality) evaluation. Our results show that system-level correlations of our proposed metric with a model-based matching function outperforms all competing metrics on the SummEval summarization meta-evaluation dataset, while the same metric with a string-based matching function is competitive with current model-based metrics. The latter does not use any neural model, which is useful during model development phases where resources can be limited and fast evaluation is required. SMART also outperforms all factuality evaluation metrics on the TRUE benchmark. Finally, we also conducted extensive analyses showing that our proposed metrics work well with longer  summaries and are less biased towards specific models.

Published as a conference paper at ICLR 2023
SMART: SENTENCES AS BASIC UNITS FOR TEXT EVALUATION
Reinald Kim Amplayo, Peter J. Liu, Yao Zhao, Shashi Narayan Google Research
(reinald, peterjliu, yaozhacyz, shashinarayan| @googl com
ABSTRACT
Widely used evaluation metrics for text generation either do not work well with longer texts or fail to evaluate all aspects of text quality. In this paper, we introduce a new metric called SMART to mitigate such limitations. Specifically, we treat sentences as basic units of matching instead of tokens, and usc a sentence matching function to soft-match candidate and reference sentences. Candidate sentences are also compared to sentences in the source documents allow grounding (e.g-, factuality) evaluation. Our results show that system-level correlations of our proposed metric with model-based matching function outperforms all competing metrics on the SummEval summarization meta-evaluation dataset, while the same metric with string-based matching function is competitive with current model- based metrics. The latter does not use any neural model, which is useful during model development phases where resources can be limited and fast evaluation is required. SMART also outperforms all factuality evaluation metrics on the TRUE benchmark. Finally, we also conducted extensive analyses showing that our proposed metrics work well with longer summaries and are less biased towards specific models.
INTRODUCTION
One major obstacle in the progress of text gener- ation tasks (e.g., document summarization, long- form question answering, data-to-text genera- tion, etc.) is automatic evaluation. Traditionally, automatic metrics that rely on discrete token- level matching such as ROUGE (Lin) 2004) and BLEU (Papineni al. 2002) have been utilized to check whether system outputs are of high
System-L Correlation
0.6
0.4
0.2
SMART
quality across four dimensions (Kryscinski et al.) 2019 Fabbri et al. 202I): coherence. factuality, fluency, and informativeness. These metrics correlate well with human judgments all four dimensions of text quality (Fabbri et al.) 2021). Because of this, the evaluation is usu- ally coupled with human elicitation studies that ask humans to rate texts. These studies can be expensive and nearly impossible to reproduce.
ROUGE
BartScore
0.0
<40
<47
<65
Number of Tokens Reference
Figure 1: Kendall tau system-level correlations of ROUGE and SMART averaged over four dimen- sions of summary quality as the number of tokens increases. Summaries are from CNN/DM (Her) mann et al.) 20T5) and human annotations are from SummEval (Fabbri et al. [222)). Each bucket in the x-axis contains equal number of data points. More details in Section A.2
More recently, pretrained language models are leveraged for automatically evaluating system- generated texts (Zhang al. 2020; [Sellam et al.] 2020; Yuan et al.) 2021), which have shown improvements on correlation with human judgments. Nevertheless, both ROUGE and LM- based metrics have three major drawbacks. Firstly, these metrics are not good at evaluating long and multi-sentence texts. Figure T iillustrates system-level rank correlations of ROUGE in different text lengths, which shows that after a certain length, ROUGE drastically decreases its evaluative power. By design, ROUGE is also not robust evaluating possibly shuffled information in long

---

Published as a conference paper at ICLR 2023
outputs, hurting its performance evaluating coherence. On the other hand, LM-based metrics such as the ttate-tf- BARTScore (Yuan et al. [202]. are constrained to the length limitation of the pretrained LM used, thus they are not able to evaluate outputs longer than this limit. And secondly, most of these metrics only use reference texts during evaluation. This restricts the capability of the metrics from evaluating dimensions of text quality that requires grounding to the source. Yuaan et al. (202I) suggested to use the source document during evaluation, however their evaluation is still limited short documents because of length limitations in LMs.
In this paper, we propose an automatic metric called SMART (Sentence MAtching for Rating Text SMART is motivated by the pyramid method of human evaluation for summarization (Nenkova et al. 2007), where they transform text into semantic content units (SCUs), or sentences that contain a single fact. Since this kind of transformation cannot be done automatically, we use sentences as a proxy to SCUs, and treat them as basic units of matching instead of tokens. This additionally allows the metric to effectively support long and multi-sentence texts. Since sentences most likely do not have exact matches, we use soft-matching function that returns a matching score between 0 and 1, given a pair of sentences. Moreover, to allow grounded evaluation, we also include the source in the calculation of the metrie. Similar to ROUGE, introduce multiple SMART versions using sentence n- g-gamm overlap and longest common subsequence.
Our experiments show that SMART with BLEURT (Sellam et al. 2020) as soft-matching function outperforms all the competing approaches on all four dimensions of quality in the SummEval dataset (Fabbri al. 2019). We also show that SMART with T5-ANLI (Honovich et al.) 2022) outperforms all competing factuality-
evaluation metrics on the TRUE benchmark (Honovich et al. (002)). Moreover, a faster variant of SMART, which does not use any neural model for text matching, shows competitive correlations with human judgments. Finally, our extensive analyses show that SMART works better with longer summaries and is less biased towards specific models.
2 RELATED WORK
Evaluation in conditional generation tasks such machine translation and document summarization is a long-standing problem. Traditionally, evaluation involves human elicitation studies that score texts based on different metrics of quality, such as adequacy, fidelity, and fluency in machine translation (Hovy, 1999). and coherence, conciseness, fluency, readability, and content relevance in summarization Mani 2001; Nenkova et al. 2007). Automatic metrics based on token n n-gram matching have been developed to replace these expensive and timecconumiigg studies, in which ROUGE (Lin) 2004) and BLEU (Papineni et al., 2002) are most widely used in summarization and translation, respectively. Several extensions to token n-gram matching have been proposed, such as using paraphrases, synonyms (Lavic & Agarwal 2007), and word embeddings (Ng & Abrecht 2015) to handle cases that are semantically equivalent, and downweighting common n-grams to focus more on salient ones (Vedantam et al., 2015). Popović (013) instead use character-level n- h-gram matching to also match words that are conjugated differently and support morphologically rich languages.
With the introduction and success of pretrained language models such as BERT (Devlin et al. 2019) and BART (Lewis et al. 2020), evaluation metrics that leverage them have been proposed. BERTScore (Zhang* et al. 2020) leverages contextualized token emheddings from BERT and obtains pairwise matching of tokens from reference and system summaries. MoverScore (Zhao al. 2019) extends BERTScore by instead having many-to-one soft alignments using Word Mover's Distance
(WMD; Kusner al. 2015). BLEURT (Sellam et al., 2020 fine-tunes BERT to predict human scores with large-scale synthetic training data. Score Yuaan et al. 2021) uses BART and treats evaluation as text generation problem, using likelihood of predicting the system summary given the source document or the reference summary. Clark et al. (2019) and Zhao et al. (2019) also explored sentence-level matching with WMD using (contextualized) sentence embeddings, however they show no concrete improvements over other model-based metrics (Fabbri et al.. 2021). In contrast, we show that our metric correlates better with human judgments than all competing models.
Factuality in summarization (Falke et al. 2019 Maynez al. 2020 is usually evaluated separately since most automatic metrics are focused on informativeness and do not include the source document in the metric calculation. Factuality metries can be divided into three approaches: natural
g t hub. htt.:///w.pooog..eaaarr////////a
2

---

Published as a conference paper at ICLR 2023
language inference (NLI) based approaches (Falke et al. 2019 Maynez et al. 2020 Laban et al. 2022). where a summary is considered factual if all its facts are entailed by the source document, model-based approaches (Kryscinski et al.) 2020 Deng et al. (02I). where model is trained to detect factual errors in the summary, and question answering (QA) based approaches (Durmus et al. 22200: Wang al. 2020 Honovich al. 202T), where questions generated in factual summary should be answerable using the source. While we also compare correlations of automatic metrics with human judgments factuality, the goal of our work to find holistic metries for evaluation that can also effectively evaluate other dimensions of text quality. Our results show that other dimensions such as coherence, fluency, and informativeness also benefit in the use of the source documents in the metric. Finally, we also show that our metric improves the state-of-the-arr of factual consistency evaluation based on the TRUE benchmark Honovich et al. (2022).
PROBLEM DEFINITION
We primarily use document summarization - the task of generating concise and accurate summaries of input documents (Mani 2001) to explain our metric, but the proposed metric can be easily adapted to other tasks, shown in Section 55.2
Let be a list of source documents, be a list of summaries generated for S by a candidate system, and R. be list of reference summaries produced by human annotators for S. Note that R can be a nested list, i.e., for each example, there can be multiple references. Moreover, let Q be a list of dimensions of summary quality, and let Hg be list of human- aanotttd scores for C in terms of certain summary quality For each summary quality q E Q, the problem is to devise an evaluation metric f R,C) that outputs list of scores that correlates well with Hy. Note that, unlike most of previous work on summarization evaluation (Lin 2004]] Clark et al. 2019:; Bhandaar Bhandari et al. 2020
Fabbri et al., 2021), we also take into account source documents S when calculating metric (-(.). This ensures that the metric can evaluate quality dimensions that require looking at the source.
We define the list of summary quality as follows (based on definitions in Fabbri et al. 2021 and Yuan et al., 2021):
the following four dimensions of summary quality, defined
The summary should be well-structured and well-organized. The summary should not just be a heap of related information, but should build from sentence to sentence to a coherent body of information about a topic.
Coherence:
Factuality:
The summary should only contain statements that are entailed by the source document. The summary should not contain hallucinated facts that either do not exist the source contradict facts from the source.
The summary should have no formatting problems, capitalization errors or obvi- ously ungrammatical sentences (e.g., fragments, missing components) that make the text difficult to read.
Fluency:
The summary should include only important information from the source document. The summary should not include redundant information and information that are considered excessive and non-salient.
Informativeness:
4 SENTENCE MATCHING FOR RATING TEXT
We now describe our proposed metric, SMART (Sentence MAtching for Rating Text), which has two key ideas. Firstly, we treat sentences as basic units of matching between system and reference summaries, instead of tokens. At sentence-level, exactly matching sentences are most likely non- existent (in datasets with abstractive reference summaries), thus we instead utilize soft-matching functions to compare sentences. Similar to ROUGE, present types of SMART: n-gram overlap (SMART-N) and longest common subsequence (SMART-L). Secondly, SMART allows to compare the candidate system summary with both the reference summary and the source document. This is particularly important when evaluating dimensions of summary quality that rely on the source document such as factuality.
SMART-N In order for SMART to work, summaries should be split into sentences. Let R = In:] and C [cj] be the sequence of sentences of the reference and the candidate system summary.
3

---

Published as a conference paper at ICLR 2023
def soft_ice (X, Y):
lcs = [10] * (X) * (X) +1) for i in range (x)
for in range (len +Y)+1);
if i m (X]i), Y(ji)
and j 1=
[cs[i][ max (1cs [is 1cs [cs[i] (i-1])
return [-s[-1][-1]
Figure 2: Python pseudocode of the soft version of Longest Common Subsequence (Soft-LCSS) given two sets of summary sentences X and Y.
SMART-N finds pairs of sentence n n-grams in R and C' that have the highest matching scores given by sentence matching function that returns score between 0 and (detailed description in Section A.
Formally, given N the number of sentences in the sentence n-gram, SMART-N can be calculated as follows:
pred' Σ C+-
Σ En
N-1
mat hatch((stniTin///
(1)
prec pred N
(2)
red' -EE1-M |R|-N+1
Σ N-1 n=o match((rn::
(3)
red' /( N N
(4)
f = * prec* * rec/(prec + rec)
(5)
where ma matcc((;.) is the sentence matching function, prec, rec, and f are precision, recall, and f-measure, respectively. Note that unlike ROUGE, the numerators precision and recall are different due to the use of a soft-matching function, thus they are calculated separately. In our experiments, we used SMART-1 and SMART-2 but SMART-N can be easily extended work with larger Ns.
SMART-L SMART-L is essentially the Longest Common Subsequence (LCS) of sentences in the reference and the candidate system summary. However, the original LCS algorithm requires an exact match to work. We instead use a soft version of LCS, where the task defined as: Given two sequences X [x]]] and Y [y,] and matching function match(z; Mj), find two soft- subsequences Vi Vi, and ,......, Vi length with ig-1 in ik+1 and ja-1 S ju S jk+1, maximizing the sum Ek_ match(z,,, V..)
Unlike normal subsequences, soft-subseuuences allow repetition of sentences as long as they do go back to previous sentences (hence the use of S operator). This relaxation helps in cases where the meaning sentence on one side spans over multiple sentences on the other side. Furthermore, Soft-LCS is similar but different from simple sequence alignment problem since the weight of the match depends on both the positions the items in the sequence and the items themselves. It is a less-restricted version of the Heaviest Common Subsequence (HCS: Jacobson & Vol 1992) since the matching function is relaxed to allow the use of a soft match (which is essentially an exact mismatch) in the subsequence.
turns out that Soft-LCS can be solved using dynamic programming algorithm similar to that of LCS, which is illustrated a pseudocode in Figure The main difference is that since do not require an exact match, we always take the maximum among three cases: (1) choosing to soft-match I; and Vy. (2) choosing soft-match r; and y and (3) choosing skip z;.
In special cases where either the candidate the reference is single-sentent sentence summary. normal implementation SMART-2 would return zero since one of the summaries would have zero sentence bigrams. To mitigate this issue, we pad summaries with a blank sentence on both sides when calculating SMART-2, This ensures that get non-zero score for single-sentence summaries. In fact, SMART-2 reduces to SMART-1 in this
4

---

Published as a conference paper at ICLR 2023
Given the Soft-LCS function, can then calculate SMART-L follows:
prec' soft [C,((C,
(6)
prec = pred /|C1
(7)
red's soft-lcs(R,C)
(8)
rEc = recl/(A)
(9)
prec rec/ +
(10)
Comparing with Source Some dimensions of summary quality require access to source to be effectively evaluated. To cover those dimensions, SMART also compares the candidate system summary with the source, addition to comparison with the reference summary. Let S = [sx] be the sequence of sentences the source document. SMART that uses both source and reference is calculated as follows. We first calculate two SMART scores that (I) compares candidate system summary C' with reference summary R, and (2) compares with source document S. Then, we aggregate the scores by taking their maximum. For example, SMART-N(S, R, C') is calculated as:
SMART N(S, R, C) = max(
SMART-NG NSRR C), SMART-N C))
(11)
Multiple References Finally, when there are multiple reference summaries, we calculate SMART for each reference, and aggregate them by taking their maximum, as also commonly done in previous work (Fabbri et al. 2021). This is intuitive since the candidate system summary only needs to match with at least one the reference.
Shorter Acronym We use the following template to describe SMART variants space-efficient -m,
manner: $11 2 -m, where m is the sentence matching function of choice. For example, SMART- 1 with a BLEURT (Sellam et al. 2020) matching function can be shortened into SI-BLEURT.
5 EXPERIMENTS AND RESULTS
One advantage of SMART is that it is easily extensible by changing the matching functions depending on the task. We investigated and compared six different matching functions, three model-based and three string-based, in Sections A and and of the Appendix and use the ones that perform the best for each task. Specifically, for document summarization, we use BLEURT (Sellam et al. 2020) as the best model-based function and CHRF (Popović (015) as the best string-based function. For factual consistency, matching function. Using these matching functions for the corresponding tasks is intuitive; BLEURT has been specifically trained rank texts using objectives that correspond to multiple dimensions of text quality, while ANLI is an NI. classifier that outputs a binary prediction (entailed or not).
Sentence Matching Functions
ANLI (Honovich et al., 2022) as the
Implementation Details The sentence matching functions implemented as follows. For BLEURT (Sellam et al., 2020), we used the BLEURT 20 checkpoin suggested by the authors which also supports non-English languages. For ANLI, we used the same implementation Honovich et al. (002)). where T5-1 1BB is fine-tuned with 25K training steps on ANLI (Nie et al. 2020), treating both contradiction and neutral pairs as not entailed. Sentences are split using nitk For the document summarization experiments, we report f-measure scores whenever available, such as in ROUGE, BERTScore, and SMART. For the factual consistency experiments, we report precision scores. We also report the version of SMART that considers both source and reference as in Eq [1]
in
We tested our metric two different tasks: document summarization and factual consistency. We discuss them in detail in the next sections.
5.1 DOCUMENT SUMMARIZATION
Dataset and Evaluation We conducted experiments on the SummEval dataset (Fabbri et al., 2021). document summarization meta-valluatiioonnnnnnn suite consisting of summaries from the CNN/DM dataset
eurt
http:://pppi.nnn///rojjeetttt
5

---

Published as a conference paper at ICLR 2023
(Hermann et al. 2015). Annotation is done in two stages and using experts to ensure high quality and high agreement actoss annotators. There are 1600 data points (16 systems x 100 examples) in total, each of which includes score between to 5 for each of the four dimensions of summary quality, which represents the average score given by three experts. Each data point also includes 11 reference summaries: the original summary from the CNN/DM dataset and 10 numan-writted abstractive summaries from Kryscinski et al. (2020). For evaluation, we use system-level correlation using Kendall tau, where we first take the average score for each system and take the correlation
Comparison Metries We compared SMART with four types of metrics: source-free and source- dependent string-based metrics, and source-free and source-dependent model- metrics.
Source-free string-based metrics include: (1-3) ROUGE-I/2L (Lin) 2004 measures token-level overlap between reference and output summaries; (4) BLEU (Papineni et al. 2002) measures token-
level overlap between reference and output summaries with a focus on precision; and(5) CHRF (Popović 2015) measures character- based n- g-gram overlap between reference and output summaries.
Source-dependent string-based metrics include: (6-9) Bre-ROUGE-1/2L and sre-CHRF, versions of ROUGE and CHRF that compare the output summary with the source instead of with the reference; and (10-12) S1/2/L-CHRF, our best SMART metric using a string-based matching function, as shown in AA. of the Appendix.
Source-free model-based metrics include: (13) BERTScore (Zhang* et al., 2020): A metric that relies on BERT(-like) models (Devlin et al. 2019) and computes an aggregation of the token-level similarity scores; (14) MoverScore (Zhao et al. 2019): measures the semantic distance between BERT n-gram embeddings of reference and candidate summaries using Word Mover's ' Distance (WMD; Kusner cct al.) 2015); (15) BLEURT (Sellam et al.) 2020): finetunes BERT using a combination of real and synthetic training data with gold-/silver-standar human judgment scores; and (16) Sentence Mover's s Similarity (SMS: Clark et al. 2019): : uses an extension of WMD that works with sentences instead of tokens.
Source-dependent model-based metrics include: (17) sre- B-BUREEURT, a version of BLEURT that compare the output summary with the source instead of with the reference; (18) PRISM (Thompson & Post 2020): leverages zero-shot paraphrasing model and uses probabilities from force-decoding the candidate summary given the source as input; (19) Q2 (Honovich et al. 2021): employs question generation and question answering models and checks whether answers from the summary are entailed by answers from source; (20) ANLI (Honovich et al., 2022): fine-tunes T5 (Raffel et al., 2020) using the ANLI dataset (Nie et al. 2020) to produce an entailment score given the source as premise and the summary as hypothesis; (21-22) BARTScore(+CNN) (Yuan et al., 202I]: evaluates text using probabilities from force-decoding the candidate summary given the source as input using BART (Lewis et al. 2020) without (with) finetuning with CNN/DM summarization dataset (Hermann) al. 2015); and (23-25) S1/2/L- Our best SMART metric using model-based matching function, as shown in of of the Appendix.
Results Table reports the system-level correlations of different metries for each quality dimension. For all quality dimensions, SMART with BLEURT matching function has the highest correlation, where SL-BLEURT evaluates coherence and informativeness better, and S SI-BLEURT and S2- BLEURT evaluate factuality and fluency better, respectively. On average, SL- -LERURT performs best, followed by S1-BLEURT and S2-BI BLETT all three of which outperforming BARTSCore+CCNN, which is finetuned on the same summarization dataset SummEval. 52-BLEURT also performs comparably with Q² in factuality evaluation. Given that each of the SMART metrics are better at evaluating different quality dimensions, it is therefore recommended to use them as set, similar to how ROUGE metrics are used. We can also see in the table that souree-deendeen metrics are better than source-free ones, signifying the importance of the use of source documents during summary evaluation. However, just simply using the source does not improve the metric; src-ROUEE- 1/2/1., SHRR and Sre-BLEURT obtain worse correlation scores compared to their reference-only counterparts. Among foreee-free metrics, SMS performs the best, showing the superiority of sentence-
'While they claimed to report system-leve level correlation, the BARTScore paper (Yuan et al. 2021 actually calculated summary-leve correlation (Louis & Nenkova 2013). where they first get correlation for each system and then take the average. Since use evaluation metries to rank systems, report system-level comelation following Fabbri al. (202T)

---

Published as a conference paper at ICLR 2023
Metric
Coherence Factuality
Flueney
Informativeness
Source-free String-baxed baxed Metrics
ROUGE-1
.350
.550
.527
.583
.503
ROUGE-2
.233
.600
.494
.433
.440
ROUGE-L
.117
.117
.259
.350
.211
BLEU
.217
.050
.326
.383
.244
CHRF
.350
.617
.561
.550
.599
Source-dependent baring-hased Metrics
ROUGE-1
.000
.467
.226
.200
.223
src ne-ROUGE-2
.067
.500
.293
.267
.282
src- me-ROUGEE-
.117
.550
.310
.317
.323
sre-CHRF
.050
.517
.276
.50
.273
S1-CHRF
.300
.733
.494
.500
S2-CHRF
.300
.700
.460
.433
.473
SL-CHRF
.367
.733
.494
.500
523
Source-free Model-based Metricy
BERTScore
.333
-.030
.142
.200
MoverScore
.217
-005
.259
.350
.194
BLEURT
.533
.200
.410
.467
.403
SMS
.267
.600
.360
.400
.407
Source-dependent Model-hased Metrics
PRISM
.233
.600
.360
.367
.390
ANLI
.250
.583
.544
.517
.473
Q²
.250
.750
.577
.450
.507
sre- -E-BBLEURT
.317
.717
.510
.483
.507
BARTScore
.350
.617
.494
.450
.478
BARTScone+CNN
.550
.317
.594
.583
S1-BLEURT
.433
.667
.644
.667
.603
S2- BEBTEURRT
.477
.750
.628
.583
.594
SL-BLEURT
.567
.567
.611
.733
.619
Table Kendall tau system-level correlation of different metrics on the SummEval dataset. Correla- tions of metrics not significantly outperformed by any other metric (using William's Test; Graham & Baldwin 2014) for that specific dimension are boldfaced. We also show the average scores in the column, where the best values for each block underlined. Note that BART BART that fine-tuned on CNN/DM, the same data source as SummEval, thus direct comparison with the other metries is possible.
uses
level matching metrics against the token-level ones. Among string-based metrics, SL-CHRF performs the best, with a competitive system-level correlation when compared with BARTScore+CNN on average. This shows that SMART achieves comparable evaluation power with previous LM-based metrics even without using any pretrained language models.
Ablation Studies We present various ablation studies on the different components of SMART in Table 2 For simplicity, we report on SMART-X, an average of SMART-I, SMART-2 and SMART-L The first block contains SMART-X that uses only either precision recall, both of which have substantially lower system-level correlation than f-measure. The recall variant performs better, which
follows results from traditionally recall-oriented evaluation metrics in summarization (Lin 2004)
The second block contains different ways to aggregate SMART scores that compare candidate summaries (1) to the source documents and (2) to the reference summaries. When using only one of the two scores (i.e., reference- or source-only), SMART scores significantly perform worse, which implies that using both source and reference is necessary for SMART to work. We also tried aggregation through taking the average or the minimum, both of which perform worse than taking the maximum in Eq II Taking the minimum notably does increase the correlation, and this
From here on, we correspond the average of the three [1]2]]]] variants of ROUGE SMART.
7

---

Published as a conference paper at ICLR 2023
Component
Coherence
Factuality
Fluency
Informativeness
/
SX-BLEURT
.450
.750
.611
.650
.628
using precision or recall
precision
.083
.077
.259
.350
.177
recall
.283
.683
.410
.383
.440
using different source/reference aggregation
ref-only
-100
-.300
-092
-003
-.131
src-only
.267
.700
.460
.433
.465
average
.400
.750
.577
.600
.582
minimum
.267
.700
.427
.467
.465
maximum w/o sentence-level
.550
.550
.494
.517
.528
using different reference summary
ref-only (CNN/DM)
1.100
-.300
-0059
-0033
-.123
ref-only (best system) max (CNN/DM) max (best system)
.367
.100
.410
.500
.344
.500
.767
.711
.667
.661
.667
.467
.745
.867
.686
Table Ablation study. Kendall tau system-level correlation of SX-BLEURT when one component is set to different configuration.
is because it restricts giving high scores to summaries that match both the source and the reference, which is not necessary. Finally, removing the sentence-level mechanism SMART also reduces the overall correlation score.
One interesting finding that using only the reference summaries calculate SMART gives negative correlation scores on all dimensions. This can be explained in two-folds. Firstly, human annotations Sul done by only looking at source document, which means they are not biased towards summaries that look closer to the given reference summaries. And secondly, Fabbri et al. (202T) showed that CNN/DM summaries are worse than system summaries in terms of human ratings. Given these, further investigated using different reference summaries for SMART. Specifically, we tried (1) replacing the set of reference summaries from SummEval into the original summary from CNN/DM, and (2) treating the best system summary according to the average human the reference. As can be seen in the third block of Table CNN/DM reference summaries correlate negatively on all dimensions. When using the best system summary the reference, SMART now obtains positive correlations across all dimensions. The correlation further improves when SMART both the source and the new reference.
Further Analyses We provide several analyses the Appendix to further understand our metrics. In $A.2] we show that SMART works better with longer summaries, and in $A.3 we also show that it is less biased towards specific models compared to other competing metrics. Finally, in $A.4 we show that source-only SMART works well as reference-free metric and correctly ranks systems that popular metrics such as ROUGE (Lin 2004), BERTScore (Zhang* et al. 2020), and QAEval (Deutsch et al. 2021) cannot, according to Goyal al. (022).
5.2 FACTUAL CONSISTENCY
Dataset and Evaluation We conducted experiments on the TRUE benchmark (Honovich al. 2022), meta-evaluation suite for factuality, consisting of source and candidate texts from different kinds of text generation tasks: abstractive summarization, dialogue generation, and paraphrase detection. We note that there is no reference text given and the candidate text is compared only the source. In this benchmark, factuality scores are binary (positive/negative,, scoring in SummEval), and the generated text needs to be fully factual to get positive score. Given this. the metrics are evaluated using the Received Operating Characteristic Area Under the Curve (ROC AUC) remove the necessity of setting specific decision threshold.
contrast to the five-scale
8

---

Published as a conference paper at ICLR 2023
3-Ensemble ANLI + Q² SC2s
2-Ensemble
Single Metries
Task
ANLI + SX-ANLI
ANLI + Q?
SX-ANLI
ANLI
SCzs
Q²
FRANK
91.2
91.1
89.6
89.4
89.1
87.8
SummEval
82.9
82.2
80.7
80.6
80.5
81.7
78.8
MNBM
76.6
77.0
75.6
73.0
77.9
71.3
68.7
QAGS-C
87.7
85.3
86.0
81.7
82.1
80.9
83.5
QAGS-X
84.8
84.0
81.8
77.2
78.1
70.9
BEGIN
86.2
84.9
85.7
85.0
82.6
82.0
79.7
Q
82.8
81.5
83.0
82.4
72.7
77.4
80.9
DialFact
90.4
90.3
89.4
90.4
77.7
84.1
86.1
PAWS
91.2
95.7
90.5
98.3
86.4
88.2
89.7
Average
86.0
85.8
84.7
84.2
81.5
81.4
80.7
Table 3: ROC AUC results for the different metrics on the development set of the TRUE benchmark. Results can be from ensembles of multiple metrics, and are grouped by the number of metrics used in the ensemble. The highest average score for each group is boldfaced.
Comparison Metries The best matching function for factual consistency is ANLI, as shown in of the Appendix. Since there no reference text, we used the version of SMART that compares only with the source. Moreover, we aggregated S1/2/L-ANLI into SX-ANLI by taking their mean for brevity (results of individual metrics are reported in $A.5).
compared (1) Sx-ANLI with three factuality- -specfcc- evaluation metrics: (2) Q2 (Honovich et al. 2021); (3) ANLI (Honovich et al. 2022); and (4) SCzs (Summary Consistency; Laban et al. 2022) that splits the source and candidate texts into sentences, computes entailment probabilitics into a single score. The last metric is similar to S1-ANLI, but with two key differences: (a) The aggregation is different; and (b) Training is done using both NLI and fact verification datasets.
all sentence pairs, and aggregates them
Finally, following Honovich et al. averaging their scores: (5) ANLI Q (6) ANLI SX-ANLI, and (7) ANLI Q² SZ7s-
2022) we also compared three ensembles of multiple metrics by
Results Table 3 shows the ROC AUC scores of different metrics. Among single metries, SX-ANLI performs the best, substantially improving over the second best metric. Moreover, ensembling SX-ANLI with ANLI provides another boost, performing superiorly compared to the ANLI Q ensemble metric and comparably with the ANLI + Q² S77s ensemble metric. Our ensemble metric is also more cost-effective since it only requires single model, while the other ensemble metric requires at least two more (e.g., question answering and generation models for Q?).
6 CONCLUSIONS
In this paper, proposed SMART, a new metric for evaluating generated text given reference text. SMART makes use of a sentence-level soft-matching function to match sentences, which can either be string-based model-based depending the available resources. This function can easily be replaced with new and better ones, which allows us to create better SMART metric. We provided two types of SMART based on n-gram overlap and longest common subsequence, and our extensive experiments showed that SMART evaluates document summaries better in terms of all four dimensions of summary quality: coherence, factuality, fluency, and informativeness. Our analyses also showed that SMART is better as summary length increases and is less biased than other competing metrics.
and
Limitations and Future Work We acknowledge several limitations our work. Firstly, SMART currently assumes single-source inputs, which makes it non-trivial to use for evaluating texts in tasks with multi-source inputs. Secondly, we plan to develop a system to automatically transform text into list of semantic content units (SCUs), or sentences that contain single fact. This would improve SMART by supporting texts with sentences that have more than one atomic fact. And finally, also plan to explore cross- lonnull capabilities of SMART-BLEURT, and apply to tasks where the source and candidate summaries are different languages (e.g., machine translation and cross-lingual summarization).

---

Published as a conference paper at ICLR 2023
REFERENCES
Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. Re- evaluating evaluation in text summarization. Proceedings of the 2020 Conference on Empirical Methods in Natural Language P Processing (EMNLP), pp. 9347-9359, Online. November 2020. Association for Computational Linguisties. doi: 10. 18653vv 1/2020 12202..emmlpmmain..7.. URL https
//aclanthology www.ogy.ogg///. 2020. sanip-mmain 751
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (cds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901, Curran Associates, Inc., 2020, URL 96 www.eeee606111110000eaee2e2 Paper, pdf
neurips
cc pttppppp///2220////115//555
Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. Sentence mover's similarity: Automatic evaluation for multi-sentence texts. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2748 27600, Florence, Italy, July 2019. Association for Com- putational Linguistics. doi: 10. 10.18653/v1/PP 1ttp//////P19//// 1264. URL http::::s//.aaanooooo....ooo
P19-1264
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chupter of the Associa- fion for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. New Orleans, Louisiana. June 2018. Association for Computational Linguisties. doi: 10.18653/v1/N18- 2097. URL httppp///aaaaaatoooogooooo//////
2097
Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. Compression, transduction, and creation: A unified framework for evaluating natural language generation. In Proceedings of the 202/ Conference
Empirical Methods in Natural Language Processing, pp. 7580-7605, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 16t5:////02..emmlpppmo httpa////......ttta////aaaaaaaaa http:pppss//ccccaaoooooo..... 2021 emmmmmmpppppmmmm inmai.... 599
URL
Daniel Deutsch, Tania Bedrax- Redrax-Weiss, and Dan Roth. Towards question-answering automatic metric for evaluating the content quality of a summary. Transactions of the Association for Computational Linguistics, 9:774-789, 2021.
doi: 10.1162/tacl a a_003/7 URL https://
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pp- 4171-4186 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/////11114222 1423. URL https! V/aclantho ogy. 3rg/N1991423
Esin Durmus, He He, and Mona Diab. FEQA: question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pp. 5055-5070, Online, July 2020. Association for Computational Linguistics. doi: 10. 454
URL
https://aclanthhr atttg:://acLanthoooyy.oo
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: A large-scale multi-documenn summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 1074-1084, 1084, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v 1t////991111 1102. URL https: / acl antho loy..or//P19-110
10

---

Published as a conference paper at ICLR 2023
Alexander R. Fabbri, Wojeiech Kryšciński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391 -39, 2021. doi: 10.1162/tacl a_00373. URL ht.ps: accl antho 1 wgw..ogg////....oo wtc.ll1.aacll-. -1.4
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. Ranking generated summaries by correctness: An interesting but challenging application for natural lan- guage inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2214-2220, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10.18653/v 1/t1665//11/1199/// 1/P 1213. UtL/ttps:: httlltttpss//accaaahooooog/ org/p19 1213
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. News summarization and evaluation in the era of gpt-3, arXiv preprint arXiv:2209. 12356, 2022.
Yvette Graham and Timothy Baldwin. Testing for significance of increased correlation with human judgmeet. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP),, pp. 172-176, Doha, Qatar, October 2014. Association for Computational Linguisties. doi: 1031115/1/D1441100
URL pttps::/aclanttoolllt
org/ 1020
Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS' 15, pp. (693-1701. Cambridge, MA, USA, 2015. MIT Press.
Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. q²: Evaluating factual consistency
wwwwwllgggroondeee dialogues via question generation and question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7856-7870, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10.18653/w120221emmml main.619. URL https://aclantho..oo www.:tttwwwoooooooro.o.o org/2021. emnlp-manle ma--mmainn 619
Or Honovich, Roee Aharoni, Jonathan Herzig. Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. TRUE: Re-evaluating factual consistency evaluation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, pp. 9905-3920, Seattle, United States, July 2022. Association for Computational Linguistics. URL bttps://aaLaanhhoooooooooo///// 2022 www.:/2..aaaaaac.aaaaaa.aaa naacl ma 287
Eduard H Hovy. Toward finely differentiated evaluation metrics for machine translation. In Proceed- ings of the EAGLES Workshop on Standards and Evaluation Pisa, Italy, 1999, 1999.
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1419-1436, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v12021.. .naacl-main. 112. URL https: ac ant www.ooooyy.....mooooooomowwmomot /ttp:///2222 nwacl-aacaaaaaaa mm...112
Guy Jacobson and Kiem-Phong Vo. Heaviest increasing/coommon subsequence problems. In Anmual Symposium on Combinatorial Pattern Matching, pp. 62-66. Springer, 1992.
Tom Koemi, Christian Federmann, Roman Grundkiewicz, Marcin Junezys- Dowwmutt, Hitokazu Matsushita, and Arul Menezes. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Translation, pp. 478 494, Online, November 2021. Association for Computational Linguistics. URL. httpss //aclanthology. org/ 2021 wnt 1. 1.57
Huan Yee Koh, Jiaxin Ju, He Zhang, Ming Liu, and Shirui Pan. How far are we from robust long abstractive summarization? arXiv preprint arXiv:2210.166 16732, 2022.
Wojeiech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on
11

---

Published as a conference paper at ICLR 2023
Natural Language Processing (EMNLP-IJCNLLL) pp. -40-551, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10.18653/1/D119-00511 URL https? haw.aaannhhoologyor//11----0
Wojeiech Kryseinski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference Empirical
Methods in Natural Language Processing (EMNLP), pp. 9332-9346, Online, November 2020.
Association for Computational Linguisties. doi: 10.18653/v 1/20v0. http200.mmllppmaii main.700. URL https
//aclanthology.o org. org/2020 2020. emnlp- main. 750
Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kilian Q. Weinberger. From word embeddings to document distances. In Proceedings of the 32nd International Conference on International Conference on Machine Learning- Volume 37, ICML'15, pp. 957-966. JMLR.org. 2015.
Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. SummaC: Re-visiting NLJ-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguisticss, 10:163-177, 2022. doi: 10.1 10.1162/tac1 00453. URL https
7 //aclanthology.ooee wtw.hhhoooooo.ggg///ee wwg/202..022222220 www.:/2....c111... 1.10
Alon Lavie and Abhaya Agarwal. METEOR: An automatic metric for MT evaluation with high levels correlation with human judgments. Proceedings of the Second Workshop on Statis- fical Machine Translation, pp. 228 231, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URLthttps httsss///caaatthooocooo wwwpaa/hhhhoooo..oo//00000-00000
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequenie pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 787 1-810, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/ 1ttp:////2222.../ wttp:////00.....mmia....000.00 703. URL
703
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
URL wttpthtt///// ac antho log.. W04- 1013
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. stractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 2890-2903, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.
BRIO: Bringing order to ab-
URL
Annie Louis and Ani Nenkova. Automatically assessing machine summary content without a gold standard. Computational Linguistics, 39(2):267-300, 2013.
Inderjeet Mani. Automatic summarization, volume 3. John Benjamins Publishing, 2001.
Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondfej Bojar. Results of the
WMT20 metries shared task. In Proceedings of the Fifth Conference on Machine Translation, pp. 688-725, Online, November 2020. Association for Computational Linguistics. URL https //aclanthology
org/2020 wtt.aa//ohoooooo.oo/220000000 wmt-1.77
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1906-1919, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/23 1/2020 acl-main...mmmnm... main. 173. URL htwph:ttp//// https:://aclaathhooogg../ /rg/
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topie-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 1797-1807, Brussels, Belgium, October-November 2018, Association for Computational Linguistics. doi: 10.18653/v1/ D18-1206. URL htt.thttss://eee anto howethoooooyy...... ogy.lol....o.................... .rgg 10t///1881111 1206
12

---

Published as a conference paper at ICLR 2023
Ani Nenkova, Rebecca Passonneau, and Kathleen McKeown. The pyramid method: Incorporating human content selection variation in summarization evaluation. ACM Trans. Speech Lang. Process.. 4(2):4 es, may 2007. ISSN 1550-4875, doi: 10.1145//2339112123311, hothhtt:///// /doi
org/ 111111115/131122 2339133
Jun-Ping Ng and Viktoria Abrecht. Better summarization evaluation with word embeddings for ROUGE. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1925-1930, Lisbon, Portugal, September 2015. Association for Computational Linguistics, doi: 10.18653/11/D15--
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885-4901, Online, July 2020. a41-main,441, URL https
Association for Computational Linguistics. doi: 10. //aclanthology wtw///222220
441
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02/55, 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for auto- matic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/107330830077333 https://aclaante low.aanthhoooo.oooooooo.....
URL
ogy. 2rg.yoorg/022-1440
Maja Popović. chrF:
character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pp. 392-395, Lishon, Portugal, September
2015, Association for Computational Linguistics. doi: 10. L/ URL https
//allannhology.orgg///ooo -3049
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21 21((40):1-65) 2020.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Amaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations. 2022.
net. forum? htt::/oormmmmm//ooooo d=9Vrb9p0w
Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT:
Learning robust metrics for text
generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881 -7922, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020..o acl- main. 704. URL https: / http:////aaaantoolg..oooo org/200 ac 1-main in
704
Brian Thompson and Matt Post. Automatic machine translation evaluation in many languages
via zero-shot paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods
in Natural Language Processing (EMNLP), pp. 90-121, Online, November 2020. Associa-
tion for Computational Linguistics. doi: 001118655//1/2/2 ear.:0mmmmlllmmmmmemm
URL https
wwwwb. wwtp///tooogg////eeooooooooo www.::m....m.mmmamm...mmm
emn
R. Vedantam, C. Zitnick, and D. Parikh. Cider: Consensus based image description evaluation. In
2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4566 4575, Los Alamitos, CA, USA, jun 2015. IEEE Computer Society. doi: 10. 10.110/CVPR.20055.229000 URL
fttps://dos eeecomput rsoc ety. wttp:://.etttt.oo..............
201. 7299087
13

---

Published as a conference paper at ICLR 2023
Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions evaluate the factual consistency summaries. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 5008-5020, Online, July 2020. Association for Computational Linguistics. doi: 10. 1/21666/1/20222.-mmmm
URL. https: //ac anthology....... http:///...aanoooooooooooooo org/
7020.acl-ma. 450.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. BARTScore: Evaluating generated text as text generation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL
ew net/ forum?
Id=5Ya8Pbvp29
Tianyi Zhang*, Varsha Kishore*, Felix Wu", Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. 2020. URL httpttttp///eerreett eww net/
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. Mover- Score: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIP),, pp. 563-578, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19. 1053 URL https: htw.o/aonnoooooooooooooooo.
14

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 SENTENCE MATCHING FUNCTIONS
The scores we get from SMART depend on the sentence matching function match. We investigate six different sentence matching functions widely used in both machine translation and document summarization. Specifically, we compare three string- based and three model- based matching func- tions. The former do not rely accelerators (GPUs/TPUs) while the latter leverage pretrained neural models. Thus, we expect string-based matching functions to be inferior, but they are good alternatives for faster evaluation or when accelerators are not available.
ROUGE Lin 2004) A popular document summarization evaluation metric, it measures the number of overlapping textual units. As with most summarization work, we explored three types of textual units: unigrams (ROUGE-1), bigrams (ROUGE-2), and longest common subsequence (ROUGE-L..
BLEU Paapineni et al. 2002) A popular machine translation evaluation metrie, it is precision- focused metric that calculates n- n-gram overlap between two texts and also includes a brevity penalty.
Another machine translation evaluation metric, it calculates character- based n- n-gram overlap between system and reference sentences. Unlike ROUGE and BLEU which operate at the token level, CHRF is more effective especially in morphologically does not require any tokenization, lemmatization, and stemming (Mathur et al. 2020 Koemi et al.) 2021
CHRF (Popović, 2015)
languages as
model-based metric that leverages contextualized token embeddings from BERT(-like models). It computes similarity scores by aligning tokens from reference and candidate summaries, and token alignments are computed greedily to maximize cosine similarity.
BERTScore (Zhang et al. 2020)
Another model-based metric mainly used to evaluate factuality, which uses T5 (Raffel et al. (220)) fine-tuned on the ANLI dataset (Nie et al.) (2020) to produce between 0 (not entailed) and 1 (entailed) given premise and hypothesis. We use the source/reference as premise and the candidate summary as hypothesis.
ANLI (Honovich et al. 2022)
BLEURT (Sellam et al., 2020) supervised model-based metric that uses BERT that trained to predict human judgment scores using small-scale dataset. To make it more robust, the model is first pretrained with a large-scale synthetic dataset. Moreover, it is optimized using several objectives including ROUGE, BLEU, BERTSSoree and entailment. BLEURT has been shown be effective in evaluating sentence match machine translation, thus we expect it to be the better matching function among all matching functions.
One advantage of SMART'S is that it is easily extensible by changing the matching functions to better ones. This means that more domain-specifee matching function can be used for evaluation towards specific domains, or better-performing sentence matching metric can be used to improve overall evaluation.
SMART with Different Matching Functions We compare different variants of SMART using the six matching functions described in Section A. Table their system-level correlations. where Coh, Fac. Flu, and Inf stand for coherence, factuality, fluency, and informativeness. respectively. Among string- based matching functions, CHRF performs the best in terms of average correlation, followed by BLEU. This shows that machine translation metrics are better sentence matchers. Among model-based matching functions, BLEURT performs the best by large margin; SMART with BLEURT significantly outperforms all the other model-based variants on all dimensions of summary quality. We believe that this is because BLEURT is optimized to match sentences, as well as to predict ROUGE, BLEU, BERTScore. and entailment scores. Interestingly, ANLI as a matching function underperforms even in the factuality dimension. We posit that this is because sentences are passed to the matching function without their neighboring context. Finally, when compared against the no-SMART variant (e.g., BLEURT vs. S1/2/L-BI SMART improves the correlation overall,
15

---

Published as a conference paper at ICLR 2023
Matching Function
SMART Type
Coherence
Factuality
Fluency
Informativeness
SMART with burnng Matching Functions
ROUGE-1
SI
.233
.733'
.494
.500'
.490
S2
.217
.683
.477
.417
.448
.267
.460
.467
.473
none
.350
.550
.527
.583
.503
ROUGE-2
.650
.477
.432
S2
.183
.650
.477
.417
.432
SL
.217
.650
.44
.383
.423
.233
.600
.494
.433
.440
ROUGE-I.
S1
.233
.733"
.527"
.500
.498
S2
.217
.683
.477
.417
.448
.267
.460
.467
.473
none
.117
.117
.259
.350
.211
BLEU
SI
.300
.733'
.527"
.467
.507
S2
.283
.717
.510
.450
.490
SL
.377
.717
.460
.483
.494
.217
.050
.326
.383
.244
CHRF
S1
.300
.733"
.494
.500"
.507
S2
.300
.700
.460
.433
.473
.367*
.733*
.494
.500*
.523
none
.350
.617
.561
.550
.519
SMART with Model-bused Matching Functions
BERTScore
S1
.377
.683
.561
.517
.519
S2
-017
.383
.266
.183
.207
SL
.383
.683
.527
.583
.544
none
.333
-000
.142
.200
.161
ANLI
S1
.117
.550
.444
.350
.365
S2
.133
.533
.360
.333
.340
SL
.117
.483
.343
.350
323
none
.250
.583
544
.517
.473
BLEURT
S1
.433
.667
.644"
.667
.603
$2
.750"
.628"
.594
SL
.567"
.567
.611
.733'
.619
none .533
.200
.410
.467
.403
Table 4: Kendall tau system-level correlation of variants of SMART using different matching functions. For each block, correlations of metrics not significantly outperformed by any other metric (using William's Test; Graham & Baldwin, 2014) for that specific dimension are marked with an asterisk (*). Those that are not significantly outperformed by all metrics are boldfaced. We also show correlation scores the matching function without SMART, as well as average scores the μr column, where the top three values for each block underlined.
except for three matching functions: ROUGE-1/2 and ANLI. We believe this is because they effective as sentence matching function. ROUGE-I4 1/2 only matches unigrams/bigrams, in contrast BLEU (where all n-grams up to
are considered collectively) and BertScore (where the n-grams are contextualized). ANLI is an NLI classifier that produces predictions that are close to 0 or which is not ideal for ranking two systems with different degrees of errors.
A.2 SMART WORKS WELL WITH LONGER SUMMARIES
We divided the datasets into four buckets based the average number of tokens in the reference summary, where the first bucket contains the shortest reference summaries. For each bucket, we then calculated system-level correlation for all competing metrics. For each quality dimension, the relative increase correlation with respect to ROUGE-X, which illustrated in Figure As can be seen in the figure, in general, all metrics perform better relative to ROUGE as the number of tokens
report
16

---

Published as a conference paper at ICLR 2023
Coherancce
Factuality
Fluency
Informativresss
2.5
2.9
BMATTcom
2M5
-12
-40
-00
-17
-65
<10
-033
-17
-00
310
-00
-00 Ratsorcc
405
-10
-07
-05
Tuons Nehernnse
Tdkos feforse
Idooxx
Figure 3: Length Analysis. Relative increase in Kendall tau system-level correlation with respect to ROUGE-X of four evaluation metries for each length bucket (leftmost bucket has the shortest lengths).
GovReport factuality
arXiv factuality
Metric
relevance
geomean
relevance
geomean
ROUGE-1
0.52
-0.12
0.00
0.25
-0.13
0.00
ROUGE-2
0.44
-0.11
0.00
0.16
-0.13
0.00
ROUGE-L
0.39
-0.11
0.00
0.17
-0.15
0.00
BERTScore
0.38
-0.04
0.00
0.18
-0.10
0.00
BLEURT
0.06
-0.08
0.00
0.37
0.00
0.01
BARTScore
0.25
0.06
0.12
0.12
0.24
0.17
SX-ROUGE-1
0.72
0.06
0.21
0.27
0.52
0.37
SX-CHRF
0.24
0.39
0,27
0.30
0.48
0.38
SX-BLEURS
0.31
0.26
0.28
0.45
0.38
0.41
SX-BLEURS Bre-pppuR--roiiiio
-0.21
0.40
0.00
0.26
0.56
0.38
SX-B SS-BLEURT-ree
0.36
-0.04
0.00
0.53
0.17
0.30
Table 5: Spearman rank correlation of different metrics on the GovReport and arXiv datasets with annotations provided in Koh et al.] (2022). The values in the 'geomean' column the geometric averages (we put zero if one value is non-positive) of the correlation scores. The best number for each column boldfaced.
increases from 43 tokens, which shows that ROUGE is not suitable for long summary evaluation. Interestingly, ROUGE also underperforms in the first bucket, which means that it is also not good at evaluating short summaries. Among the competing metrics, SX-BLEURT (and SX-CHRF) correlate the best (and second best) when the there are more tokens in the source/reference.
We also compared SMART with different metries on two long-form summarization datasets, arXiv (Cohan et al. 2018) and GovReport (Huang et al. 202I). We used the meta-evaluation suite curated Koh et al. 2022), where they annotate scores terms of relevance (measures whether a summary contains the salient information found in the reference) and factuality (assesses whether a summary factually consistent with the source). Table 5 reports the Spearman rank correlation scores of competing metrics, including ROUGE-1/2/L. BERTScore, BLEURT, BARTScore, and SX-ROUGE- /CHRE/BLEURT In terms of the geometric average both relevance and factuality, SX-BLEURT performs the best among all metries. All previous metrics except BARTScore only positively correlates with relevance. BARTScore is positively correlated with both relevance and factuality, however it is substantially worse than SMART, When dissecting SMART into its precision and recall variants, we see that the precision variant is correlated well with factuality while the recall variant correlated well with relevance. This shows that SMART can be used as a single metrie evaluate both relevance and factuality, which can be useful for hill-climbing during model development.
7S--ROGGE-1 has the best correlation among the Sx-ROUGE-1/21. variants.
17

---

Published as a conference paper at ICLR 2023
Rank Diff = Model Rank -Human Rank
ROUGE-X
SMS
BARTScare
SX-CHRF
SX-BLEURT
0-
M11
Figure 4: Bias Analysis. Difference in ranking between human scores and metric scores for each competing metric, averaged over all quality dimensions. A negative value means that the metric ranks the system higher. MO M23 in the x-axis correspond to the summarization models, where MO to M5 are extractive and the rest are abstractive. See Fabbri et al. 2021) for detailed model descriptions.
Metric
RRank Diff) 4
Pairwise Acc. f
ROUGE-X
5,012
63.33
SMS
5.500
63.33
Coh
BARTScore
4.717
67.50
SX-CHRF
5,232
65,00
SX-BLEURT
4.228
72.50
ROUGE-X
4.228
70.00
SMS
2.915
80.83
Fac
BARTScore
2.915
80.83
SX-CHRF
2.062
86.67
SX-BLEURT
2.151
87.50
ROUGE-X
3.969
72.50
SMS
4.650
67.50
BARTScore
4.031
75,00
SX-CHRF
3.824
74.17
SX-BLEURT
2.646
83.33
ROUGE-X
3.841
75.00
SMS
4.783
70.00
Inf
BARTScore
4.500
72.50
SX-CHRE
4.198
75.00
SX-BLEURT 2.739
82.50
Table 6: The standard deviation of the difference in ranking between human and metric scores (Rank Diff) and the pairwise ranking accuracy of different metrics for different summary quality. Best scores are boldfaced.
A.3 SMART IS LESS BIASED TOWARDS SPECIFIC MODELS
While we acknowledge that all automatic metrics are not perfect as shown Table their rankings should not be hugely different from human rankings. Moreover, they should not be biased towards a single summarization model. To this end, we get the difference in rankings given by humans and by the automatic metrics for each summarization model for each quality dimension. Figure 4 illustrates the resulting differences averaged over all dimensions summary quality, in which we have two interesting observations. Firstly, all automatic metrics are in general biased towards ranking extractive systems higher. This suggests that a separate extractive and abstractive model evaluation is necessary using current automatic metrics. We leave exploration of metrics that equally unbiased towards both extractive and abstractive for future work. Secondly, we found that BARTScore scores BART (M22 in Figure
Lewis et al. 2020) significantly higher than all the
18

---

Published as a conference paper at ICLR 2023
Model
Human
ROUGE-1
CHRF
BLEURT
SX-ROUGE-1
SX-CHRF SX-BLEURT
BBC Dataset
GPT3-D2
1st
15,812
9.512
41.871
39.621
43.461
49.275
TO
2nd
10.617
5.641
39,952
29.440
30.210
42.747
BRIO
3rd
10.964
5.700
39.200
31,070
31,030
41.852
CNN Dataset
GPT3-D2 BRIO
1st 24.357
16.020
42.946
40.115
40.382
48,911
2nd
22.128
13.564
42,950
35,582
31.800
43,436
TO
3rd
15.182
9.163
40.641
31.940
29.096
41,486
Table 7: Scores of three systems on two different datasets based on five different reference- metries. Human rankings are also shown for comparison. Values that are colored red are those that do not correspond to the human rankings.
other models (BARTScore of BART is -1 11.99 vs. 22300 + 0.313 average without BART). This problem is amplified when BARTScore is finetuned using the CNN/DM dataset (-0.488 vs. -1.702 + 0.248). This shows that using pretrained encoder-decoder models for summary evaluation induces bias towards summarization models finetuned on the same model.
In Figure 4 we can see that SX-BLEURT the least biased since its rank differences with human scores are closer to zero. To quantitatively measure if the above statement is true, we use two measures. The first one is the standard deviation of the rank difference, where the score closest to zero can be considered the least biased. The second measure is the pairwise ranking accuracy, where for pairs system, we check whether human and metric rankings are equivalent. Table these numbers, which show that SX-BLEURT has the lowest standard deviation of the rank difference and the highest pairwise rank accuracy across all quality dimensions. This entails that metric the least biased among the competing metrics.
A.4 SMART IS EFFECTIVE EVEN WITHOUT REFERENCE SUMMARIES
In this section, we show that SMART does not require reference summaries to effectively evaluate system summaries. We make use of two sets of newswire data from BBC and CNN from Goyal et al. (2022). Each dataset consists of a source news article and three candidate summaries generated by (I) GPT3-D2: GPT-3 model (Brown et al. (020) which is trained to follow user instructions (uyang cet al., 2022); (2) BRIO (Liu et al.) 2022): a summarization model fine-tuned on both CNN/DM
(Hermann al. 2015) and XSum (Narayan et al. 2018 which currently achieves state-of-the-ar results; and (3) ))>r/)))) TO (Sanh et al. )022): a prompt-based model fine-tuned on multiple tasks which
include summarization datasets.
Goyal et al. 2022) showed that reference-dependeen metrics such as ROUGE (Lin 2004), BERTScore (Zhang et 2020). and QAEval (Deutsch et al. 202T) do correlate well with human judgments. We therefore compared reference-free versions of five metrics: ROUGE- 1 CHRF, BLEURT, and their SMART variants, where we calculate the scores by instead comparing the candidate summary with the source. Table reports the scores of these metrics on the BBC and CNN datasets. As can seen in the table, only SX-BLEURT is able to rank the systems correctly on both datasets, according to the given human rankings. All the other scores produce incorrect rankings on one dataset, however GPT3-D2 is ranked the highest in most cases for all metrics. This entails that using reference summaries may not be necessary (and can be detrimental) evaluate system summaries, and our results demonstrate that SMART-BLEURT is better equipped than other metrics.
A.5 SMART FOR ASSESSING FACTUAL CONSISTENCY
Table g reports the ROC AUC scores of different individual SMART metrics using three different kinds of matching functions: ANLI, BLEURT, and CHRF. Overall, SMART with ANLI performs significantly better than the other matching functions. This is expected since ANLI produces scores
ROUGG-11 ROUGF-2, and ROUGE-L. all have similar trends. We only report ROUGE-1 I for brevity.
19

---

Published as a conference paper at ICLR 2023
Task
SI-ANLI
S2-ANLI
SL-ANLI
SI-BLEURT
S2-BLEURT
SL-BLEURT
S1-CHRF
S2-CHRE
SL-CHRP
SummEval
77.
81.0
75.8
75.5
MNBM
73.0
73.0
73.0
67.9
68.0
67.9
68.0
68.0
68.0
QAGS-C
83.0
79.7
79.1
80.9
79.2
79.4
75.5
75.2
74.9
QAGS-X
77.2
77.2
77.2
61.6
61.6
61.6
59.0
59.0
59.0
DEGIN
85.0
84.8
85.0
86.7
86.8
$6.7
83.6
83.5
83.6
Quans
DialFact
90.2
90.4
65.3
69.9
66.7
68.6
65.7
PAWS
98.3
98.3
98.3
66.4
66.3
65.4
66.3
66.2
66.3
Average
Table 8: ROC AUC results for the different metrics the development set of the TRUE benchmark. The highest average score boldfaced.
that are extremely polarized (i.e., close or 1) even for slightly nonfactual pairs, which more appropriate for this task. BLEURT and CHRF, on the other hand, produce partial scores for such pairs, which make them more appropriate in ranking systems, such as in SummEval.
20


Reviews:
[{"review_id": "mQEpw0TXUZ7", "reviewer": "Reviewer_7UHv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes SMART, a new metric for conditional text generation (especially summarization) evaluation. The proposed method mainly consists of two components: sentence-level soft-matching and source inclusion. For sentence-level soft-matching, it splits the prediction and reference into separate sentences. The single sentence pair similarities are computed using existing metrics (e.g. BLEURT, CHRF, etc.) The sentence-level similarities are aggregated in ways inspired by ROUGE, leading to different method variations, SMART-N (based on sentence N-gram) and SMART-L (based on longest common sequence, LCS). For SMART-L, a dynamic programming algorithm is proposed to find the best soft-alignment between prediction sentences and reference sentences. For source inclusion, it essentially treats the source (input) text as another reference, and aggregate the source score with other reference scores by taking their maximum.\n\nThe authors conducted evaluations on two meta-evaluation datasets: SummEval, for general summary evaluation; and TRUE, for summary factual verification. Experiments show that using SMART on existing good-performing methods can further improve the performance and achieves overall the best performance compared to other methods. Ablation studies are also conducted to verify different components in the method, and provide more insights into the method behaviors.", "summary_of_the_review": "This paper proposes SMART, a new metric for summarization evaluation based on a novel technique, sentence-level soft-matching, and source inclusion. Experiments show the good performance of the method. However, the study was not rigorously done to verify the usefulness of the novel part. Also, the improvement is not fully consistent among different based methods. The work is overall interesting, but more work should be done to make it more concrete and strong.\n\n(Scores updated after rebuttal)", "strength_and_weaknesses": "#### Strength\n\n- Evaluation metrics for summarization is a highly important topic, especially because the currently most widely-used metrics like ROUGE has been found to correlate poorly with human evaluation.\n\n- The proposed method has non-trivial novelty (mostly in the sentence-level soft-matching component), and has exhibited good performances in the experiments.\n\n\n#### Weakness\n- As stated above, the novelty mostly resides in the sentence-level soft-matching techniques. However, the ablation studies did not effectively verify the usefulness of it. The ablation studies showed the usefulness precision & recall, source & reference, and max aggregation, yet it is possible that the performance gain mostly comes from source inclusion but not sentence-level soft-matching. This is likely, given the poor performance of the \"ref-only\" setting and relatively better performance of \"src-only\", implying that source is more helpful than references. A valuable (and hopefully easy) study would be to use existing methods, like BLEURT, ANLI, etc., *with* source inclusion but *without* sentence-level soft-matching. If it consistently underperforms the full version of SMART, that verifies the usefulness of sentence-level soft-matching.\n\n- The improvement of SMART is not always consistent; it does not seem to improve all methods. For example, based on Table 1 and Table 4, it seems to me that it only improves ROUGE-L, BLEU, BLEURT and BertScore, but hurts ROUGE-1/2 and ANLI. Although it's good to be able to improve on a current SOTA method and establish a new SOTA, it becomes questionable whether the method generalizes to future SOTA methods. (By the way, I think it would be better to put information in Table 1 also in Table 4 so that it's easier to compare.)\n\n\nBelow are my questions, not necessarily weaknesses:\n- Some results in Table 2 are quite hard for me to understand. Mainly, SX-BLEURT ref-only gets extremely bad performance (negative correlation with human on all aspects), but BLEURT without SMART is not as bad. Also, even though ref-only has negative correlation, with source added, including the references can still largely improve the performance. How would you interpret these results?\n\n- A more high-level question: how do you justify the idea of using sentences as units in summary evaluation? As I imagine, there can be cases where information is grouped differently among sentences in prediction and reference (e.g. predicted sentences = [\"A\", \"BC\"]; reference sentences = [\"AB\", \"C\"]. The prediction is perfect, but all sentence-pair matching scores are imperfect, so the given score will be imperfect). In such cases, using sentence as units will not be better than using tokens as units.\n\n\nBelow are some minor points:\n- Many argued strengths of the method, such as good for longer summaries, less biases, good reference-free performances, etc., are only elaborated in the appendix.\n\n- (typo) page 9, above \"Results\": \"SZ_{ZS}\" should be \"SC_{ZS}\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Q5Iia0cxW-", "reviewer": "Reviewer_k533", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new automatic metric called SMART for evaluating model-generated text. Here, sentences are used as basic units of matching instead of tokens, in order to support long and multi-sentence texts. For this, the authors use a sentence-level soft matching function, and experiment with both string-based and model-based matching functions to select the best variants of SMART. Similar to ROUGE, two types of SMART are proposed, SMART-N which is based on sentence-level n-gram overlap, and SMART-L which is based on longest common soft-subsequence. In order to support grounded evaluation, the authors also include the source documents during metric calculation. Results presented in Table 1 demonstrate that SMART, with BLUERT as the soft-matching function, has better correlation scores, with human judgements, than existing model-based metrics across all four quality dimensions of coherence, fluency, informativeness, and factuality. SMART (SMART-CHRF) also achieves comparable evaluation results with previous LM-based metrics even without using any pretrained language models. Ablation results demonstrate the advantage of using source documents in metric calculation, not only for factuality dimension, but for other dimensions as well. Results presented in the appendix further demonstrate that SMART performs better as summary length increases and is less biased than other competing metrics.\n", "summary_of_the_review": "Overall, the paper is a nice read. The proposed metric SMART is well-motivated, well-defined, and shows better correlation scores with human judgements than existing summary-evaluation metrics on four different quality dimensions, namely coherence, fluency, informativeness, and factuality. SMART is flexible, as different soft-matching functions can be selected depending on the task. It also performs well in reference-free settings. The authors demonstrate that SMART performs better when the length of summaries increase. However, in the majority of summarization tasks, summaries are meant to be concise. Hence, this finding may not be practically important. Also, some observations, highlighted under “Weaknesses”, are difficult to follow. I would encourage the authors to address the questions asked in order to make the submission stronger. Overall, the proposed metric is a welcome value addition to the existing literature on model-generated text evaluation metrics.\n", "strength_and_weaknesses": "Strengths\nThe paper is well-written and well-structured.\nUsing sentences as basic units of matching, although not new, is well-motivated.\nThe proposed metric is defined well, although it would be better if the authors could explain SMART-L more clearly, maybe using an example (in the appendix).\nExperiments are well-defined and results are clearly presented.\nSMART not only outperforms metrics for evaluating content quality of model-generated summaries, but also for evaluating their factual consistency. This gives SMART an added advantage over existing metrics.\n\nWeaknesses\nSome results, especially in Table 2, are difficult to follow. Text summarization is more intuitive when it is abstractive as it is closer to how humans summarize a piece of text. Reference summaries on CNN/DM are also abstractive. I believe the majority of the model-generated summaries used for analysis in the SummEval benchmark (Fabbri et al. (2021)) are also abstractive. In such a case, how is source-only SMART performing better than ref-only SMART? It seems counterintuitive to me that abstractive sentences in model-generated summaries would match better with source document sentences. Please explain.\nHow does using the source documents for SMART calculation improve the correlation scores for non-factuality quality dimensions?\nIn Table 2, how can the minimum of src-only and ref-only correlation scores be more than the lower of the two? Am I missing something?\nAlthough not a weakness, but can SMART be used in more dynamic settings? For example, consider task-agnostic dialogue generation, where there can be multiple possible responses based on the given context, and also where the response to be generated might depend not only on the immediate context but also on the history of previous utterances.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GoFXAp__so", "reviewer": "Reviewer_K7DY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about an automatic reference-based evaluation metric which evaluates the quality of generations with respect to the source text and multiple references by considering sentences as the base units of evaluations and leveraging different matching methods to compute the similarity between sentences. Authors have shown that the proposed metric has better performance even for the evaluation of long summaries and also it is less biased.", "summary_of_the_review": "This paper is in good shape and the idea behind the metric is interesting and very easily applicable in different domains. The effectiveness of the metric to evaluate long summaries without adding bias is the major benefit of the paper.", "strength_and_weaknesses": "The contribution, idea, motivation and results from experiments are written very neatly.\n\nThere are quite a comprehensive set of experiments (comparisons to different baselines and functions) to show the effectiveness of the proposed metric.\n\nAccording to the experiments the metric can be easily adopted to different tasks and domains which can be one of the major positive impact of this metric.\n\nOverall, I do think the paper does not have any major flaws that prevent it from being accepted. Some minor concerns for me is: what was the motivation that authors tried to consider sentences as the units of the evaluation, explaining it can be beneficial. Also, is it possible to show some results of the metric on other reference-based evaluation tasks such as translation?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "8baHQOLtVk", "reviewer": "Reviewer_TC5h", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed SMART, a new automatic evaluation metric text generation models.\n- Proposal 1: To deal with multiple sentences, they proposed SMART-N and SMART-L, in which the calculation unit of ROUGE-N and ROUGE-L is changed from words to sentences. In SMART, several metrics such as BLEURT and ANLI are used to softly compute matching scores between sentences.\n- Proposal 2: To exploit the source document, they proposed to use the maximum value of \"similarity between candidate text and reference text\" and \"similarity between candidate text and source text\".\n\nExperiments on SummEval (document summarization) and TRUE benchmark (factual consistency) showed that the proposed method performs competitively when using a specific matching function that was selected for each task.", "summary_of_the_review": "The authors propose a new evaluation metric named SMART for text generation models. Both proposals are intuitive and simple, and cross-sectional experiments suggest empirical effects.\n\nOn the other hand, the effectiveness of the main proposal is somewhat questionable, and the paper is not complete; it is considered difficult to be accepted by ICLR.", "strength_and_weaknesses": "Strengths:\n- Both proposals are intuitive and easy to reproduce.\n- The existence of an ablation study (§5.1) makes it easy to understand the nature of the proposed method.\n\nWeaknesses:\n- The performance of the proposed method is not robust to the choice of matching function. In fact, on SumEval (§5.1), the proposed method shows high performance with BLEURT, while it shows low performance with ANLI (Table 4). On the other hand, on the TRUE benchmark (§5.2), the proposed method shows high performance with ANLI, while it shows low performance with BLEURT (Table 7).\n  - Existing methods should not be described as \"very slow\" (Introduction); the proposed method also uses LM-based metrics similar to existing methods, e.g., BLEURT. Moreover, the proposed method in this paper has the additional cost that the reader needs to try the number of matching functions for each data set.\n- In the paper, the first proposal (sentence-by-sentence computation) is given a large amount of space, but in practice, the second proposal (comparison of generated text and source document) appears to be more effective. In fact, the performance of the first proposal alone is very low (`ref-only` in Table 2). The readers would benefit from an ablation study in which only the second proposal is applied to existing methods such as ROUGE, BLEURT, and ANLI. It would also be very attractive to present concrete examples where the treatment of the sentence-by-sentence works and add qualitative evaluation.\n  - Although the authors consider the proposed method as a general-purpose evaluation metric for text generation models, it cannot be applied to machine translation. This is because the proposed method requires calculating the similarity between the source and the candidate. The authors should weaken their argument and argue that they proposed an automatic evaluation metric for automatic summarization. In any case, additional experiments on the TRUE benchmark is very attractive.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster