DocAsRef: An Empirical Study on Repurposing Reference-based Summary Quality Metrics as Reference-free Metrics
Authors: Forrest Sheng Bao
Ruixuan Tu
Ge Luo
Yinfei Yang
Hebi Li
Minghui Qiu
Youbiao He
Cen Chen
Venue: EMNLP 2023 Findings (2023)
Keywords: summarization
evaluation
zero-shot

Abstract: Automated summary quality assessment falls into two categories: reference-based and reference-free. Reference-based metrics, historically deemed more accurate due to the additional information provided by human-written references, are limited by their reliance on human input. In this paper, we hypothesize that the comparison methodologies used by some reference-based metrics to evaluate a system summary against its corresponding reference can be effectively adapted to assess it against its source document, thereby transforming these metrics into reference-free ones. Experimental results support this hypothesis. After being repurposed reference-freely, the zero-shot BERTScore using the pretrained DeBERTa-large-MNLI model of $<$0.5B parameters consistently outperforms its original reference-based version across various aspects on the SummEval and Newsroom datasets. It also excels in comparison to most existing reference-free metrics and closely competes with zero-shot summary evaluators based on GPT-3.5.

DocAsRef: An Empirical Study on Repurposing Reference-based Summary Quality Metrics as Reference-free Metrics
Forrest Sheng Bao x6) Ruixuan Tu Ge Luo Yinfei Yang*, Hebi Li Minghui Qiu Youbiao He and Cen Chen
@ Department of Computer Science, Iowa State University, Ames, IA, USA Dept. of Computer Sciences, University of Wisconsin-Madison, Madison, WI, USA Sunnyvale, CA, USA
ByteDDnce, China
School of Data Science and Engineering, East China Normal University, Shanghai, China Equal contribution, forrest bao gmail com, ruixuan@cs. wisc.edu
Abstract
Requiring highly educated human labor, refer- ence summaries are very costly to obtain. There- fore, many reference-free metrics have emerged recently (Scialom et al., 2019; Vasilyev et al.. 2020; Bao et al., 2022), which directly compute score between system summary and its source document. However, the performance of reference- free metrics has historically lagged behind that of reference-basee metrics because human- reference summary serves fluent and compre- hensive representation of the key facts in the input document and thus gives reference- based metrics an advantage.
Automated summary quality assessment falls into two categories: reference-based and reference-free. Reference- metrics, his- torically deemed more accurate due the addi- tional information provided by human-written references, are limited by their reliance on hu- man input. In this paper, we hypothesize that the comparison methodologies used by some efference-based metries to evaluate system summary against its corresponding reference can effectively adapted assess it against its source document, thereby transforming these metrics into reference-free ones. Experimen- tal results support this hypothesis. After be- ing repurposed reference-freely. the zero-shot BERTScore using the pretrained DeBERTa- large-MNLI model of <0.5B parameters con- sistently outperforms its original reference- based version across various aspects the SummFvval and Newsroom datasets. also ex- cels in comparison to most existing reference- free metrics and closely competes with zero- shot summary evaluators based on GPT-3.5.
Recently, large language models (LLMs) have shown promise in building reference- frreencce-feeeeeeeeee summary quality metrics. Metrics based on LLMs like GPT- 3.5/4 (Liu et al., 2023; Wang al., 2023; Gao al., 2023) have outperformed both reference- free and reference-based baselines. However, LLMs computationally expensive, and the closed nature of GPT-3+ restricts their usage with legal and repro- ducibility limitations. A more viable solution that uses much more cost-effective language models is highly expected.
1 Introduction
To build accurate but efficient metric, we revisit the reference-based metrics and hypothe- size that they can be repurposed into reference- free metries by directly comparing summary with its source document. After being repur- posed, BERTScore outperforms not only its orig- inal reference-basee version, but also most exist- ing reference-free free metrics across the SummEval, Newsroom, and TAC2010 datasets on both seman- tic and linguistic aspects. Notably, the repurposed BERT achieves superior or comparable per-
Summarization is an important natural language generation (NLG) task. A problem that goes hand in hand with it is summary evaluation, which quantifies the quality of summarizer or sys- tem summary it generates. The traditional ap- proach to automated' summary quality assessment is reference-based, such ROUGE (Lin, 2004), BERTScore (Zhang et al., 2020) and Mover- Score (Zhao al., 2019), which assesses sys- tem summary against one or plurality of human- written reference summaries.
pattp::/hhace ingsemant I css cs..yy/20733 clased- basel ines/
"The ground truth is still human evaluation.

---

formance to GPI-3.5-based summarization eval- uators. is worth noting that these results are achieved using foundation models with signifi- cantly fewer parameters (<0.5B) compared to GPT- 3.5's extensive 175 billion parameters.
2.2 Repurposing ref-based ref-free
The idea of repurposing ref-based metrics for ref- free evaluation involves leveraging the mechanism employed by these metrics to compare two texts. Although ref-based metrics were originally de- signed to compare system summary against a reference summary, we hypothesize that they can still be effective in directly comparing the system summary with the document.
We hope this paper can inspire more work into zero-shot summarization or NLG evaluation us- ing cost-effective (c.g., <IB parameters) LMs. Our source code is at https: /gtthb.. com/ S gmaww/ DDoee/De.......... In summary, the key find- ings of this paper include:
To repurpose ref-based metric f into ref-free one, we simply feed the document in lieu of the ref- erence when using f. While the idea of using the document as the reference not new, the specific approach proposed here, which is straightforward and direct, has not been previously explored. Em- bracing the principle that simplicity is beautiful in science, we decide to give it try.
1. The proposed reference-free free repurposing does improve performances for Transformer based metrics including BERTScore and BLEURT.
2. The repurposed BERTScore can significantly outperform all non-GPT-3.5 baselines using underlying LMs of the similar capacity.
Remarkably, our simple strategy has yielded good results. Three representative ref-based met- rics gain their performances after being repurposed (Table 1). One of them, BERTScore employing generically trained LMs such as RoBERTa-large has performance very close the performances of metrics based on GPT-3.5, which utilizes hun- dreds of times more parameters (Tables 2 & 3). This outcome highlights the effectiveness of repur- posing ref-based metrics for ref-free evaluation.
3. With LMs hundreds of times smaller, the re- purposed BERTScore can further match the performance of those based on GPT-3.5 in most of the cases.
2 Approach
2.1 Background: Ref-based and ref-free summary evaluation metrics
A system summary is generated from a source doc- ument by summarizer, which is usually embodied by a neural network model today. A correspond- ing reference is generated from the same document by a human. Metrics for summary evaluation fall into two categories: the reference-based (short as ref-based) ones which are functions comparing candidate summary and human-written reference summary:
2.3 Variants of BERTScore
The promising initial results encouraged us to ex- plore modifications to the ref-based metrics for en- hanced performances. ROUGE and BLEURT have limited room for tweaking because ROUGE-1 and ROUGE-2 have been the best among its variants in the past two decades and BLEURT is already fine- tuned explicitly for summary evaluation. Hence, we focus on refining BERTScore.
(system summary, reference). and reference-free (short as ref-free) ones which are functions that evaluate a candidate summary based solely on the input document:
The first tweak we applied onto BERTScore to try different small-scale, pretrained language mod- els (LMs). We conducted experiments with three LMs: RoBERTa, DeBERTa, and BART, both their base versions (around 1 110M parameters) and large versions (around 400M parameters). Additionally, explored the variants of these LMs that have been officially fine-tuned on the MNLI dataset. Our hypothesis is that an LM fine-tuned for the MNLI task may be better suited for computing text simi- larity than generic LMs.
f'system summary, document).
Ref-based metrics, such as ROUGE (Lin, 2004). BERTScore (Zhang* et al., 2020), BLEURT (Sel- lam et al., 2020), and MoverScore (Zhao et al.. 2019), historically have advantage over ref-free ones, such Blane (Vasilyev et al., 2020), Sum- mQA (Scialom et al., 2019), SDC* (Liu et al., 2022), and SueNes (Bao al., 2022), because the human-written reference summary serves as flu- ent and comprehensive representation of the key facts in the input document. Recent GPT-based summary metrics (Gao et al., 2023; Wang et al., 2023; Liu et al., 2023) are all ref-freee in nature.
The second tweak we explored is expanding BERTScore to the sentence level by calculating the similarity between sentences instead of tokens. Various similarity measures and sentence weight-
2

---

ing schemes were proposed (Appendix B). Unfortu- nately, they rarely perform better than the original token-level BERTScore.
BLANC, SummaQA, and SueNes, share the com- mon initial checkpoint, BERT-base. MoverScore and BLUERT use RoBERTa-larg and BLUERT- 20 as the LMs.
3 Experiments
BERTScore is pairwise comparison metric. De- pending on the axis along which max pooling is done, each BERTScore variant yields three scores: P (Precision), R (recall), and F (FI). The exper- iments are carried out on individual RTX 3090 24GB GPUs. For more details, see Appendix A.
3.1 Settings
Because of their exceptional performances and im- pacts, four ref-based metrics are picked as candi- date metrics to be repurposed: ROUGE (Lin, 2004), BERTScore (Zhang* et al., 2020), BLEURT (Sellam et al., 2020), and Mover- Score (Zhao et al., 2019). ROUGE is the classic metric used in summarization. The rest three are widely used as baselines in the field in recent years.
3.2 Results
Following the trend in recent summary evaluation studies (Peyrard et al., 2017), we report the results at the summary level. Spearman's correlation coef- ficients between metrics' predictions and human- rated ground truth are the performance measure. For space sake, we present selected results here with extended results available in the uppendices.
Seven ref-free baselines are included in our study. Four of them use underlying foundation LMs of fewer than 1B parameters: SummaQA (Scialom al., 2019), BLANC (Vasilyev et al., 2020), SU- PERT (Gao et al., 2020), and SueNes (Bao al., 2022). The rest three (Liu al., 2023; Gao et al.,
3.2.1 Is repurposing useful? Before vs. after The answer is yes! Despite that ref-based met- rics historically perform better than ref-free met- rics, Table shows that the three modern metrics, MoverScore, BERTScore, and BLEURT, gain their performances after being repurposed, on nearly all aspects of all datasets. The lexicon-basee ROUGE- 1/2/L also improves its performance on some as- pects or datasets after being repurposed.
2023; Wang et al., 2023) of them are based on
GPT-3.5, which has 175B parameters.
Three multi-facet summarization evaluation datasets with human ratings are used as the test datasets: SummEval (Fabbri et al., 2021), News- room (Grusky et al., 2018) and TAC2010 (NIST, 2010). SummEval and Newsroom are for single- document summarization while TAC2010 is for multi-document summarization. SummEval covers four aspects: CONsistency, RELevance, COHer- ence, and FLUUnccy Newsroom covers four as- pects: INFormativeness, RELevance, COHerence, and FLUency. TAC2010 reports three scores: Pyra- mid (Nenkova et al., 2007), linguistic, and overall scores. For TAC2010, only Set A of TAC2010 is used in this paper because Set B "update summa- rization" does not fit the problem formulation in $ 1. Mcasuring how well summary covers key pieces of information in the source document, REL- evance or Pyramid score is generally considered the most important aspect of a summary. CONsistency raising concern recently due to the hallucination issue. Details for the datasets and their aspects can be found from their respective papers.
After being repurposed (top of half of Table 1), BERTScore outperforms all other metrics across datasets, with only couple of exceptions. It outperforms MoverScore and BLEURT signifi- cantly. While BERTScore underperforms ROUGE on SummEval before repurposing, it turns the tide after.
The ref-free metrics used in their original des- ignated way perform extremely bad on the News- room dataset (bottom half of Table I and additional evidence in Appendix D). This is due to that in Newsroom, a reference summary can be as short one sentence. Here, the reliance to reference sum- maries becomes weakness of ref-based summary quality metrics. In this case, the original document may be better than the reference summary com- pare with for judging the summary quality.
Underlying language models (LMs). The LMs used in repurposed BERTScore variants are dis- cussed in $ 2.3. The default LM is RoBERTa- large. All ref-free baselines involving finetuning:
3.2.2 Repurposed BERTScore ys. ref-free baselines
BERTScore is the most tweakable ($ 2.3) and best- performing ($ 3.2.1) metric. So we further study how compares with the ref-free baselines. As mentioned in $ 2.3, to study its robustness, different
*Wo did not run the experiments on baselines but simply copied the numbers from their original papers to here. For the three GPT3.5 hasee baselines, we pick their best results from their papers.
3

---

Table Performance before vs. after repurposing for four metrics. Summary-level. Spearman's s. On the SummEval and Newsroom datasets, Best in each column in bold while 2nd best underlined.
SummEval
Newsroom
CON
RRI.
COH
F1.U1
INF
REL.
COH
FI.U
BERTSeure
0,318
0.375
0.471
0.265
0.611
0.591
0.633
0.591
Seore R
0.235
0.343
0.258
0.162
0.750
0.658
0.639
0.500
BERTScore F
0.208
0.401
0.416
0.241
0.689
0.617
0.66.3
0.618
After repurposing,
MoverScore
0.245
0.695
0.589
0.537
used ref-freely
ROLGE-1 R
0.145
0.128
0.002
0.067
0.744
0.6639
0.564
0.476
ROUGE 2
0.262
0.155
0.049
0.163
0.746
0.648
0.591
0.511
ROUGEEL R
0.289
0.187
0.106
0.183
0.746
0.641
0.591
0.515
BLBURRT 0,221
0.252
0.336
0.172
0.549
0.507
0.596
0,562
Seore P
0.008
0.208
0.275
0.083
-0.034
0.012
0.044
0,045
0.365
0.315
0.320
Seor F
0.088
0
0.321
0.139
0.149
0.171
0.185
0.187
Before repurposing, used original
MoverScare
0.129
0.238
0.088
0.096
0.136
0.153
0.112
0.077
ref-bused way
ROUGE- R
0.148
0,250
0.117
0.109
0.105
0.128
0.071
0.073
ROUGE-2
0.166
0.194
0.109
0.102
0.089
0.087
0.016
0.057
ROL:GELI.
0.205
0.099
0.035
0.063
0.016
0.025
BLEURT
0.048
0,215
0.174
0.087
0.154
0,140
0.071
0.075
underlying LMs are used with BERTScore. Due to space limit, here we only report the results using RoBERTa-large and DeBERTa-large which give the best performance.
Table 3: Summary-level Spearman's 's correlation coeffi-
cients on dataset Newsroom. Aspect names abbreviated.
INF
REL
COH
FLU
BERTScom reperposed using respective I.Ms helow
RoBERTs Jurge R
0,750
0.638
0.659
0.590
Table 2: Summary-level Spearman's 's correlation coeffi- cients on dataset SvmmEval. Aspect names abbreviated.
RoBERT large F
0.689
0.617
0.663
0.618
RoBERTs large MNLIR
0.737
0.621
0.632
0,550
BoBBRTT--agg-MMMN
0.582
0.563
DeBFRTTs-arg large R
0.247
0.646
0.669
0.604
DeBERTs large
0,720
0.625
0.676
0,613
CON
REL
COH
FLU
DeBERTh- Jarri-----NLLLLL
0.748
0.629
0.668
0.583
DeBERTs large MNLI
0,739
0.635
0.674
0.595
NeRTS reperposed, using respectivv LMs below
RoBERTh-larg
0.318
Best of Reporposed BERTScore
0.750
0.688
0.669
0.618
Ta-rge
0.308
0.401
0.416
0.241
RaBERTa large MNLIP
0.387
0.355
0.438
0.287
Baselines, reference
KoBBERBB-arge-MNNL
0.357
0.382
0.373
0.241
Blanc
0.688
0.608
0.586
0.531
DeBERTa-lrge
0,338
0.341
0.418
0.280
SummaQA-FI
0.569
0.516
0.490
0.466
DeBERTa-large
0.289
SUPERT
0.605
0.539
0.399
0.293
0.551
0,305
SueNes
0.753
0.647
0.669
0.674
DeBERTa large MNLIF
0.344
0.333
0.291
0.239
ChaiGPT (Gao al., 2023)
0.521
0.524
0.484
0.480
CharGPT (Wang et al., 2023)
0.461
0.409
0.507
Best of Repurposed BERTSeare
0.399
0.401
0.471
0.303
Best Basetines
0.753
0.647
0.669
0.674
Buselines, reference free
Blane
0.244
0.197
0.089
0.132
Su
0.197
0.165
0.123
0.140
SUPERT
0,330
0.216
0.120
0.230
SucNes
0,190
0.177
0.167
0.228
tinct performances. Despite the challenges, repur-
ChalPP (Wang al., 2023)
0.432
0.353
G-Eval (GPT-3.5) (Lin al., 2023)
0.386
0.385
0.440
0.424
posed BERTSccre outperforms all baselines except
Best of Baselines
0.432
0.428
0.470
0.424
SueNes, which is finetued using data explicitly aug- mented for the summary evaluation task, on all aspects.
The results on SummEval are given in Table 2. Repurposed BERTScore outperforms all non-GPT baselines by a significant margin. Additionally, performs comparably to GPT3.5-based baselines on the RELevance and COHerence aspects. It is superior than one of the two GPT-3.5-based ap- proaches on the CONsistency aspect. It should be noted that SummEval is challenging due to its cov- erage of 23 modern summarizers, many of which cxhibit highly similar behavior.
Because the non-GPT baselines, BLANC, SummaQA, and SueNes, use BERT-base as the underlying LM, for a fair compari- son, we include BERTScore's results using RoBERTA/DEBENNTBBRRTTTTE Even when they use LMS of the same size, BERTScore still outperforms them.
Appendix D.
Table 4 shows the results on the TAC2010 dataset where BERTScore outperforms baselines on all aspects except linguistics. As a multi- document summarization dataset, TAC2010 pro- vides 10 source documents di. . for gener- ating a system summary s. We use the formula Eie(1.10) ff (d,, 8) to approximate the score of a
Table 3 reports the results on the Newsroom dataset. The Newsroom dataset poses signifi- cant challenge for new metrics since the basclines already perform very well on this dataset, likely because it evaluates only seven systems with dis-
4

---

Table 4: Suvmary-level Spearman's 's correlation coef- ficients on dataset TAC2010 (multi-document summa- rization). Aspect names in table header.
effectively in measuring factual inconsistency in summaries (Laban et al., 2022).
Pyramid
Linguistic
Overall
Table 5: The performance of BERTScore-P with and without IDF. Summary-level Spearman's 's correlation coefficients comparison. Model size: base. Yellow cells are when using IDF is worse than without IDF and
BERTScooe, repurposed, uzing respective LMs below
0.496
0.455
DeBERTa large MNLUR R
0.526
0.405
0.492
DeBERTa-large Baaeela-aae--MNLl
0,500
BaRT-larg--MNT MNLI
0.471
0.2772
0.415
green cells are for the opposite.
BART large MNLI
0.422
0.202
0.350
BAKI-large-MNIL
0.481
0.426
SummEval
Newsroom
GoBBeKil-lagg-MMNL
0.469
0.306
0.418
CON
REL
COH
FLU
INF
REL
COH
FLU
RoBERTa large M
RoRFKTh-large large-MMIJJ R
0.481
0.509
0.340
0.356
0,450
0.464
IDF
BART
RoBERTa
0:295
0.284
0.381
0.228
0.627
0.579
0.589
0.620
0.536
Baselines, reference-free
DeBERTa
0.262
0.252
0.316
0.206
0.614
0.556
0.613
0.544
Blanc
0:427
0.294
0.397
RoBERTa
0.307
0.315
0.408
0.240
0.597
0.551
0,579
0.531
SummmQA-FF
0
0.243
0.286
IDF
BART
0.322
0.300
0.233
0.675
0.650
0.061
0610
SUPERT
0.479
0.324
0.427
DeBERTs
0.281
0.276
0.345
0.221
0.628
0.587
0.631
0.586
SueNes
The IDF part of BERTScore may not play important role because the attention mechanism already factors in what IDF does. stopword boilerplate word has weak attention other tokens. In BERTScore's original paper (Zhang* et al., 2020), IDF makes very marginal impact on all except one datasets/tasks. Table 5 shows our ablation study on the impact of IDF. IDF makes a very small impact and in many cases, even decreases the performance.
summary given single-document summariza- tion metric f.
3.3 What makes BERTScore powerful
While the result of this paper may sound surprising because the method is very simple, it is totally ex- plainable. Comparing summary with a document is theoretically more challenging than comparing it with a reference, because information is more sparse in document than in reference. This might be the reason that strong NL. evaluation metrics are historically reference-basee However, BERTScore exhibits exceptional performance after being repurprosed from ref- based to ref-free. We attribute this to both the contextual embedding of the underlying LMs and the maxpooling step of BERTScore.
The repurposed BERTScore shows relatively ro- bust performance with respect to the choice of the underlying LMs. For example, on Newroom. BERTScore's worst performing variant in every aspect still outperforms the ChatGPT-based. The only aspect which BERTScore is not stable is the COHerence aspect of SummEval..
Conclusion
The Transformers have the ability identify important information in a context: by showing strong attentions to the important tokens as learned in pretraining. In other words, encoder-only Trans- formers used in BERTScore can identify important tokens and function as implicit summarizers. Ex- traneous information in a summary causes the sum- mary's context to diverge from that of the original document, resulting in reduction of semantic sim- ilarity, even when comparing the same token in the summary to its 'counterpart in the document. The maxpooling step of BERTScore further focuses the alignment of the most semantically proximate token pairs between the document and the sum- mary. Because the document and the summary are independently embedded in BERTScore, only when important information in the document and the summary align, the BERTScore can high. On a related note, BERTScore alone is found very
In this paper, we explore repurposing summary evaluation metrics that were originally designed or trained for reference-based use reference- frec metrics. The motivation was reuse their power in comparing texts. Comprehensive experiments on multiple datasets show that four representative metrics generally perform better after the repurpos- ing. The best among them, BERTScore, is further studied with different configurations. The repur- posed BERTScore using 0.5B- B-B--aaameter LMs can outperform all non-GPT baselines significantly and even most of the times those based on GPT3.5.
A Acknowledgments
This research partially supported by National Sci- ence Founation (NSF) grant CNS-1817089, The authors would also like to thank reviewers who have given precious feedback on improving this
5

---

work. Forrest Bao also wants to dedicate this paper the people of Ukraine who have been coura- geously fighting for freedom since February 24, 2022.
metrics for multi-document summarization. In Pro- ceedings of the 58th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 1347- 1354, Online. Association for Computational Linguis- tics.
Limitations
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers).
The test sets are all from the news domain which is the only domain that human evaluation to system summaries has been done. This is a limit beyond our control.
Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. SummaC: Re- Se-Visiting NL.-- based Models for Inconsistency Detection in Summa- rization. Transactions of the Association for Compu- tational Linguistics, 10: 10:163-177.
Unfortunately, our attempt (Appendix B) to ex- pand BERTScore from token-level to sentence- level fails. Moreover, unlike token- BERTScore, which remains stable across different LM choices, sentence-level BERTScore is highly sensitive to the selection of LMs. Extended results can be found in the appendices.
Chin-Yer Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries, In Text Summariza- fion Branches Out, pages 74, Barcelona, Spain. Association for Computational Linguisties.
BERTScore can have a variant, which is at chuck-level. This idea was proposed in REUSE for machine translation (Mukherjee and Shrivas- tava, 2022). Since we have tried token-level and sentence-level BERTScore, trying chuck-level BERTScore in summarization be part of the future work.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG evaluation using GPT-4 with better human alignment.
Yizhu Liu, Qi Jia, and Kenny Zhu. 2022. Reference-free summarization evaluation via semantic correlation and compression ratio. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2109-2115, Seattle, United States. Association for Computational Lin- guistics.
References
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL workshop on intrinsic and ex- trinsic evaluation measures for machine translation and/or summarization, pages 65-72.
Ananya Mukherjee and Manish Shrivastava. 2022. REUSE: REference-free UnSupervised quality esti- mation metric. Proceedings of the Seventh Confer- ence on Machine Translation (WMT), pages 564-568, Abu Dhabi, United Arab Emirates (Hybrid). Associa- tion for Computational Linguistics.
Forrest Sheng Bao, Ge Luo, Hebi Li, Minghui Qiu, Yin- fei Yang, Youbiao He, and Cen Chen. 2022. SueNes: weakly supervised approach to evaluating single- document summarization via negative sampling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 2450-2458, Seattle, United States. Association for Computational Linguistics.
Ani Nenkova, Rebecca Passonneau, and Kathleen McK- cown. 2007. The pyramid method: Incorporating human content selection variation in summarization evaluation. ACM Transactions on Speech and Lan- guage Processing (TSLP), 4(2)
NIST. 2010. TAC2010 guided summarization com- petition. https: http///a..iii gov/ /210// Summari commarizzaionn Gwwiodd//nndddeeeeeeeeeeeneeeeee Summ. 2010. gui wed.aiieea...eeee html. Accessed: 2021-08-18 16.
Alexander R. Fabbri, Wojeiech Kryšeiński, Bryan Mc- Cann, Caiming Xiong. Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating Summariza- tion Evaluation. Transactions of the Association for Computational Linguistics, 9:391-400 409.
Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. 2017. Learning to score system summaries for bet- content selection evaluation. In Proceedings of the Workshop on New Frontiers Summarization, pages 74-84, Copenhagen, Denmark. Association for Computational Linguistics.
Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Ship- ing Yang, and Xiaojun Wan. 2023. Human-like sum- marization evaluation with ChatGPT.
Thomas Scialom, Sylvain Lamprier, Benjamin Pi- wowarski, and Jacopo Staiano. 2019. Answers unite! unsupervised metrics for reinforced summarization
Yang Gao, Wei Zhao, and Steffen Eger. 2020. SUPERT: Towards new frontiers in unsupervised evaluation
6

---

models. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natu- ral Language Processing (EMNLP-IJCNLP,, pages 3246-3256, Hong Kong, China. Association for Com- putational Linguistics.
each token-level metric, 30 mins for each sentence- level BERTScoor-variant For Newsroom, the num- bers are minutes and 45 minutes, respectively. Software packages: We used Hugging- Face's evaluate library for the metrics and sentence-transfomee library for cosine similarity computation. The evaluate li- brary automatically downloads and plugs mod- els into the metrics. We also used numpy and scipy for general computation. For MoverScore, we used its official implementa- tion from its Github Repo https: /tte:://gthhts com A AIPHES// http/////mllppp em 9-movers The vI code has deprecated dependencies. So we used the v2 version.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text genera- tion. pages 7881-7892, Online. Association for Com- putational Linguistics.
Oleg Vasilyev, Vedant Dharnidharka, and John Bohan- non. 2020. Fill in the BLANC: fluman-free quality estimation of document summaries. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 11-20, Online. Association for Computational Linguistics.
Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is ChatGPT good NLG evaluator? preliminary study.
B Expanding BERTScore to the sentence level
We experimented with two approaches to mea- sure sentence similarity: cosine/dot-produte sim- ilarity and using text reasoning probabilities. Let us elaborate on the latter. Models trained for nat- ural language inference (NLI) tasks typically out- put three probabilitics representing the relation- ships between two input sentences: "entailing" (E), "neutral" (N), and "contradictory" (C). In other words, given a pair of sentences x and y, we ob- tain: [E, N, CJ = NLI(I, y). We experimented with three options: 1 - N, E - C, and E, and selected F. C due to its intuitive appeal and em- pirical evidence of its effectiveness.
Tianyi Zhang". Varsha Kishore", Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. Interna- tional Conference on Learning Representations.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Chris- tian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized em- beddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP),, pages 563-578, Hong Kong, China. Association for Computational Lin- guistics.
The original BERTScore uses IDF to weight to- kens. To weight sentences, we employ PageRank- style approach below. First, we decide the impor- tance of document sentences [æ1, .22 tence is considered important if can relate to many other sentences. Hence, the importance of document sentence x; can be estimated as w; (sim (z,, æ1). 22m where sim(.) is a sentence-level similarity measure and can be sum or entropy. In the simplest case, we have Wy ΣifteeN+ sim(x,, Second, we let the document sentences "vote" on the importance of summary sentences. The importance of summary sentence is determined by the sum of its similar- ities all document sentences, weighted by the importance (voting power) of document sentences. Thus, the importance of the j-th sentence y, in the summary is v, = Σi,jen w.,sim(x; Vÿ)- Unfortunately, as you can see in $ D, the sentence-level tweaks do not yield better results ex- cept on the consistency aspect Since there are too
A More Experimental Information
A sen-
More details on underline language models, BLANC and SueNes, the two training-based reference-free baselines use BERT-base while the training-based reference-based BLEURT uses BLEURT-20 32-layer Transformer model. SU- PERT uses BERT-Large-basee SentenceTrans- former while SummaQA uses a SERT--argeebasee QA model. Please understand the tremendous amount of effort and time needed to re- trair or re- benchmark all metries using the same underlying pre-trained language model.
Experimental time: The experiment can be done really quickly. For SummEval, 5 mins for
Per the BLEURT authors, BLEURT-20 is the strongest, released BLEURRT model attps://giie cot/googlee b bourt/blle htt/m/stee//hhccppooot htt.ee..cckints........ mdd the recommended-chhcccpoin-----ee-e-
7

---

many sentence-level BERTScore variants, they are referred in this A-B-C nomenclature, where A is the similarity measure which Cosine if cosine similarity and MNLI using entailment confidence from an MNLIIfinctuned model, is the underly- ing LM, and C, optional, is the sentence weighting method g.
performance after using the Leadword heuristic. The shorter the lead is, the more performance drop.
Investigating the reason is part of our future work.
C Our idea in code
We hope this code can help explain what we mean by "repurposing" and also how to directly use the conclusion of this paper.
Amport evalucte BuggingFaeo"a import functools
2ython's standard
bertscore evaluste Locd "bartssoor"""""
bertscore_ het.score_debera_aee. debeera_lasge_nneee
functools. partiell
bertscore-corgate,
lang="en",
model_type""nieeee riicrosoft/deeeeee Langttt---rgeg-ll----
scores latprr___ddbbra__aaaaeaae
predictions = ["this is summery),
12
way references ["this is the 200*! & DocAsReÉ l'recall";
i"tada referencs")
At line 13, conventional approaches plug in hurteewritten references. But in our proposed idea (line 14), just plug in the source documents, and you will get the best reference- summary quality assessor.
It's casy-to-implement, zero-shot, and reference- free.
D More comprehensive results
Please refer to Table 6 and Table
E Leadword heuristic
It common that important information is clus- tered at the beginning a document. Hence, Leadword a simple but effective method to extract important information. SUPERT build pseudo-references by extracting salient sentences and found that Leadword is better than any other simple extractive approach. So we also experiment with limiting the BERTScore-style pairwise com- parison to top-k sentences in the input document. We use top-k slightly different from its common in text generation. Here means ratio rather than an absolute number because the length of the input document varies a lot.
Is Leadword heuristic useful? In this study, no repurposed metries benefit from the Lead- word heuristic, unlike the result reported in SU- PERT (Gao et al., 2020). Nearly every metric loses
8

---

Table 6: Extended Spearman results.
SummEval
Newsroom
CONiittncy
RELevance
COfference
Fl Uvney
INFormativemen
RELanance
COllennce
Fl.Uency
BERTScone teken-level, repurposed, usiigg respective LMs below
RoblER hoorr
0.307
0.315
0.408
0.340
0.590
0.551
0.574
0.531
RodlEK
0.379
0.616
0,540
RoBl EoTobaee
0.278
0.636
0.556
Dell Ha base
0.204
0637
0.75
0.363
0.730
0.88
BART buse P
0.291
0.322
0.380
0.233
0.075
0.600
0.611
0,60
BART-buse R
0.147
0.268
0.176
0.057
0.752
0.600
0.61
0.561
BART-boax F
0.216
0.321
0.260
1.28
0.765
0.64
0.79
0.17
RodlKKIa large P
0,318
0.375
0.471
0.265
0.611
0.591
0.63
0
RoBERTo-large R
0,235
0.345
0.258
0.162
0.750
0.658
0.639
0,590
Rol HOD
0.308
0.618
MILLII
0.554
Ro large MNLIF
0.357
0.41
0,563
0.338
0.488
0.050
0.51
0.616
0.222
0.310
0.747
0.646
0.804
DeBERTo- F
0.289
0.357
0.315
0.211
0.720
0.625
0.676
0.613
0.389
0.295
0.351
0.303
0.647
0.594
0.69
0.805
DeMliKTa large MINLIR
0
0.345
0.220
0.183
0,748
0.629
0.668
0,585
0.344
0.333
0.291
0.239
0.739
0.635
0.674
0,595
BART-Ange
0.399
0,350
0.397
0.326
0.701
0.621
0.809
0.856
BAKT
0.186
0.544
BARTTAMm
BAKT large MNLIF
0.336
0.63
0.64
BART-buge-MNL
0.305
0.764
0.654
0.69
0.560
BART-Amyy-MMN MNI.
0.782
0.360
0.389
0. 1.196
0.773
0.655
0.670
0.611
Reporposed BEReSeoore
0.303
0.661
0.699
BEKTSecore, sentence level, repurposed, using respective LMs below
Coo. MPNet-huse
0.378
0.160
0.210
0.315
0.565
0.578
0.613
0.612
Coo. MPNet-hewe
0.182
0:207
0.093
01997
0.653
0.557
0.554
0.509
Cos, MIPNet base
0,322
0.218
0.156
0.220
0.657
0.599
0.629
0.594
Cos. MPNet-base Sun-wt
0,386
0.170
0.216
0.315
0.502
0.557
0.636
0.613
Coo. MPNat-bose Sum-ott R
0.287
0.232
0.130
0.204
0.670
0.598
0613
0.9%
MPNes-hase Sun-oc
DeRFRD--arg-
MNLI DeBlKIla
0.341
0.828
0,996
MNLI DeBEKID-lug--MNL
0.379
-0.026
0312
0222
MNL TaRFEC--lrrg-MN ut
0.469
0.160
0.21
0.764
0387
MNLI DeBERTa-agge ge-L Eshopy- R
0.002
0.048
-0.022
-8.026
0.174
0.127
0.71
0248
MNIJ De Barggr---ggg--- MNI Friropy-ut F
00.10
0.001
-0.015
-0.115
40051
-0.076
0.019
-0446
Exparpesed other metries
ROUGGE-1 R
0.145
0.128
0/002
0/067
0.44
0.639
0.564
0.476
ROUGEE-2 2
0,262
0.618
0.91
0.511
ROUGFF-I.
0.089
0.591
0.515
BLEURT
0307
0.96
0,562
MoverSoore
0.380
0.138
0.695
0.615
0.599
0.537
Baselines, other reference-free free metries
Blane
0.244
0.197
0,089
0.132
0.683
0.608
0.556
0,531
SummaQAA F1
0.197
0.365
0.125
0.100
0.599
0,516
0.890
0.406
SUPERT
0,330
0.216
0.120
11230
0,693
0.605
0.17
0,539
StaNee
0.190
0.177
0.166
0.328
0.751
0.647
0.669
0.674
ChoraP Kino al., 2023)
0,435
0.554
0.130
ChaGGP//Wag 2023)
0.432
3.31 SEC (La ot al., 2023)
2023)
0.386
NN
NA
-0.068
0,062
0.002
-0.627
-0536
.0.45
Reference-Bunee appreaches used in their original designated way
ROUGE-1 R
0.114
0,309
0.164
0.117
0,323
0.278
0.231
0.215
ROUGGE-2 R
0.17%
0.272
0.182
0.142
0.151
0.134
0.886
0.102
ROUGE-L
0.111
0.29%
0.119
0.109
0,301
0.263
0.206
0.201
MoserSuone
0.195
0.269
0.175
0.185
0211
0216
0174
0.143
SLoe RoULUXIs base
0,520
-000
0.006
BEXTSesse RoBERT hase
0.258
RERTS RoRFRTT-hass P
0,059
0.125
BEXTScore RoBEXDD-luge
0.008
0.208
40034
Q012
0844
0.445
0.158
0.384
0.315
0311
0.300
Score KobitKlla large P
0,888
0
0.321
0.139
0.148
0171
0130
0.177
BLEURT
0.163
0.272
0.163
0.191
0.316
0.832
0271
0.239
METFOR (Ranetjoc and Lavin 2005)
0.170
0.253
0.120
0.124
0.243
0.23
0.163
0.173
9

---

Table 7: Extended Pearson results.
SunmmEval
Newsroom
CONústency
RELevance
COHerence
FLliony
INFormativenes
RELevance
COfference
FlAeeny
HE BERTSc token-lavel, repurposed, using nespccive LM below
RobiKila
0,321
0.420
0.299
0.664
0,661
0,615
0.554
RoBERTs-huse
0.264
0.194
0.995
0.905
0.753
0.688
0.630
0.365
0.357
0.217
0.750
0.735
0.569
0.505
DebiKiia
0.294
0.319
9.271
0.703
0.557
0.586
DeBERTo-hasc
0.2115
0.317
0.183
0.123
0.809
0.766
0.703
0.641
DeBiRRio base F
0.285
0.347
0.289
0.208
0.786
0.736
0.701
0.61
BART-lose
0.306
0.351
0.418
0.243
0.720
0.720
0.678
0.64
BART-Noou R
0134
0.267
0.154
0.033
0.818
0.775
0.899
0.37
BART buse
0.219
0.322
0.262
9.122
0.855
0.783
0.719
0.61
BOBERTi-large
0.343
0.377
0.483
0.316
0.684
0.682
0.646
0.590
0,345
0.268
0.161
0.746
0.714
0.660
RoBERTs-luggs
D411
0.425
0.277
0.749
0.725
0.688
0.630
0.467
0,670
0,629
RoBiKia large MNLI
0.347
0.237
9.166
0.795
0.743
0,350
0.625
RoBERTo-lugg-MNNLL
0,414
0.390
0.298
0.744
0.711
0.564
0.594
0.356
0.430
0.347
0.79
0.735
0.677
0.85
DeBERT--lagg-
0.217
0.320
0.203
0.135
0.812
0.771
0.710
0.655
0.306
0.768
0. 309
0.252
0.794
0,767
0.712
0.500
DeBiitie large M
0.470
0.335
0.309
0.397
0.721
0.710
0.675
0.008
DeBERTo-logee MNLLIR
0.273
D.320
0.204
0.177
0.814
0,767
0,711
0.645
0.169
0.354
0.262
0.278
0.746
0,760
0,714
0.643
large
0,363
0.412
0.270
0.764
0,708
0.645
BART--ange
0.092
0.788
0.700
0.629
BAKI large
0.257
0.208
9.164
0.824
0.796
0.718
0.663
BART-bang-MNLII
0.375
0.450
0.329
0.763
0.763
0.004
0:632
RARTE-ing-MNTI1
0:313
0317
0.160
0111
0.833
0.785
0.604
0.633
BART large MILIF
0.303
0.374
0.278
0.294
0,823
0.796
0.713
0.518
Best Repurposed BERTSc0ke
0.411
0.483
0.397
0.825
0.796
0.719
0.660
BERTScore, sentesee-evdd, repurposed, using respective LMs below
Cheme MPNet-Nowr
0.436
0.218
0.226
0.365
0.721
0.758
0.665
0.642
Cocite MIPN base
0.224
0.257
0L086
0.121
0.745
0.740
0,008
0.355
Colite MPNet-bose
0,375
0.279
0.164
0.272
0.764
0.764
0.647
0.598
Coeia MPNci-heas Sum-utt P
0.435
01214
0.253
0.370
0.716
0.730
0.671
0.639
MPNet-ose
0.154
0.248
0.720
0.608
0.191
0.327
0.587
0.676
MNLI DeULXI's large
0.182
0.099
0.906
0457
0.657
0.685
MNLI D-BEEXD--arooMBR
E-CR
0.163
-0.078
0.161
0447
0.381
0.393
MNI. Der-arT--rg--MNI
0.184
-0.46
0.100
0280
0:211
0:000
MNL.I D-BBERT-lagg-MMI E-C Europyou P
0.561
02202
0.147
0.426
0404
0.445
0.454
MNI.I DooRFat.---brg--
MNIJ Mri..
E-C Famppy-wi R
0004
0011
-0.000
-0003
0.200
0.235
0.364
0.264
MNLI DeUL large MNLI B Lauropy-st otF
0.999
00011
-0025
-0102
00037
0,000
0,000
0.049
Reporpused other mitries
ROUGB-L
0.162
0.157
-0.011
0.054
0.779
0.704
0.621
0.358
ROUGE-2
0.045
0.190
0.719
0.643
0211
0.100
91185
0.789
0:714
0.000
0.588
BLBURT
0.770
0.366
0217
0.606
0.586
0.612
0:588
Menerfoone
0.184
0.202
0.137
0.144
0.675
0.611
0.596
0.549
Baselines, other reforence- free metrics
Blanc
0.259
0.234
0.089
0.172
0.731
0.680
0.619
0.587
SumiiOA
0.245
0.186
0.115
0.157
0.588
0,553
0.507
0.474
SUPERT
0.393
0.257
0.133
0.296
0.766
0.774
0.651
0.579
SucNes
0.254
02283
0.168
0.251
0.785
0.885
0.667
ChaiGPT (Waag 2023)
0.413
0.456
0.443
0.645
0.587
0.487
0.524
SUC" [Lib 2022)
00083
0.070
0.025
0.712
0.671
0.353
0.505
Refownre-luse apprruchess mad their uriginally designated way
B K
0.214
(1270
0.129
0.148
0.000
0,013
00081
00084
ROUGB-2
0.212
[1:22]
0.131
0.123
.0.991
-0,099
.0.122
-0.105
RO0XFF-11 R
0.175
[.221
0.15
0.133
20.86
-0.054
-0.107
-0. 1.104
MoverSewe
0.190
0.268
0.099
0.132
.0031
0,008
.0.056
.0.06
RERTSSaee BeRFRTs-haae P
0/003
0.171
0.269
0.068
40.099
-0.45
-0.87
-0.800
BLRISeooe KoULiKIss Huse
0.269
0.125
0.252
0.216
BERTSeore RoBERTi-Bose
(12889
0.258
0.103
0.888
0.119
0.063
0.045
RERTS ReArFT---rpp
0.201
0.925
1.00
40,000
40000
-0.044
-0.012
BERTScore RoBEETi-lage
0.375
0.292
0.147
0.282
0.230
0.342
0:314
RFRTSSowe ReReEWTs-harp
0:12
0310
0.347
0.34
0.077
0.107
0.075
0.068
BLBUKT
0021
01208
0.176
(1066
0,015
0.101
0.006
0.018
METEOR [Roterjos und Lavie, 2005)
0215
0.283
0.106
0.135
0.056
0.069
0.003
.0.010
10


Reviews:
[{"review_id": "qL9f6uqYM6", "reviewer": "Reviewer_j2Me", "summary": "", "questions": "When you say that the repurpose a ref-basic metric into a ref-free one you say you feed the document instead of the reference, but this seems to violated the need of an independent train and test, potentially leading to an over-fitting situation. How can you be sure this is not happening?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for finding our approach to be competitive over the SOTA alternatives.\n\nFor the question for the authors, we would like to **clarify that our method is zero-shot** as mentioned in Pages 3 & 6. There is no training and thus no overfitting risk. The relationship between document and reference is not one for training and the other for test.\n\nWe appreciate the feedback on **ablation study**. Kindly let us know any additional experiments you feel necessary and we will add them. The paper already included extensive results using different foundation models and settings to study the robustness of our approach and its independence to foundation models. The conclusion is that we can get consistent results. \n\nAn additional ablation study about IDF was done in response to Reviewer 2AuP's concern. Please feel free to read the results there.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The work presents a relevant contribution, and it is fairly well written. The proposed method is sound and the results indicate a promising approach.\n", "reasons_to_reject": "The work is clearly still in progress and some further analysis as an ablation study needs to be conducted, but those are natural limitations of an exploratory study in its preliminary stages.\n", "questions_to_authors": "When you say that the repurpose a ref-basic metric into a ref-free one you say you feed the document instead of the reference, but this seems to violated the need of an independent train and test, potentially leading to an over-fitting situation. How can you be sure this is not happening?", "paper_topic_and_main_contributions": "This paper aims to offer a reference free metric derived from a reference based one generating a small scale language model that is competitive over state of the art transformer-based alternatives. The proposed repurposing of metrics are compared against reference-free baselines indicating competitive results for the proposed technique.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "As far as I could tell, the presented references are enough.\n", "Typos_Grammar_Style_and_Presentation_Improvements": "As far as I could tell, there were no typos or grammar issues.\n"}, "extra_text": {}}, {"review_id": "8mUjLq0qz9", "reviewer": "Reviewer_2AuP", "summary": "", "questions": "Can you provide more details about the creation of idf in the usage of BERTScore? From my experience, using traditional BERTScore without applying the idf strategy, only replacing ref with source could not acquire such a high Pearson score. Is the improvement all come from the selection of idf_dict?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback. \n\nOur **contribution and novelty** is not using new inputs. Such a choice of inputs has been used in all baseline ref-free summary quality metrics including those based on ChatGPT. Our contribution is on using a zero-shot and extremely simple approach to achieve a ref-free metric. On top of the idea, we performed extensive experiments using different foundation models and various settings to study the robustness of this approach. Notably, we found that this approach can achieve ChatGPT-level performance with only 0.5B parameters, as compared to 175B of ChatGPT. Finally, we suggest explanations as to why such a simple approach can be effective. \n\nRegarding **the  lack of crucial experiment details**, please kindly let us know any additional info you expect and we will be happy to add. Details on the foundation models, versions of BERTScore, etc. are already given in Section 3 and Appendix A. We further provided the source code in Appendix C and included a link to the anonymized code repo in Section 1. \n\nRegarding **IDF**, first, in BERTScore's original paper (Zhang et al. ICLR 2020), IDF makes very marginal impact on all except one datasets/tasks. IDF often slightly decreases the performance. In our case, **IDF=False for all results in the manuscript**. We further reran all experiments on BERTScore-P with IDF=True. Due to space limits, we present the Spearman's CC using three base-size foundation models below, with and without IDF, respectively. The outcome is the same as in BERTScore's paper that **IDF has a very limited impact**. \n\n**IDF=True, newly got for rebuttal:**\n|  | summeval |  |  |  | newsroom |  |  |  |\n|---|---|---|---|---|---|---|---|---|\n|  | con | rel | coh | flu | inf | rel | coh | flu |\n| RoBERTa-base P |  0.295 |  0.284 | 0.381 | 0.228 | 0.627 | 0.579 | 0.589 | 0.536 |\n| BART-base P | 0.279 | 0.283 | 0.359 | 0.208 | 0.673 | 0.631 | 0.664 | 0.620 |\n| DeBERTa-base P | 0.262 | 0.252 | 0.316 | 0.206 | 0.614 | 0.556 | 0.613 | 0.544 |\n\n**IDF=False, reported in our manuscript:**\n\n|  | summeval |  |  |  | newsroom |  |  |  |\n|---|---|---|---|---|---|---|---|---|\n|  | con | rel | coh | flu | inf | rel | coh | flu |\n| RoBERTa-base P | 0.307 | 0.315 | 0.408 | 0.240 | 0.597 | 0.551 | 0.579 | 0.531 |\n| BART-base P | 0.291 | 0.322 | 0.390 | 0.233 | 0.675 | 0.650 | 0.661 | 0.610 |\n| DeBERTa-base P | 0.281 | 0.276 | 0.345 | 0.221 | 0.628 | 0.587 | 0.631 | 0.586 |\n\nSome additional note: We cannot perform IDF weighting on BERTScore-Recall and thus BERTScore-F1 because in the recall version, a token's logarithmic IDF is 0 because it appears in all documents as all documents are the same, i.e., the source document. This is a theoretical limitation per BERTScore authors in their GitHub repo `Tiiiger/bert_score` issue 8.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper makes a modification upon the input of reference-based metrics like BERTScore, transforming them into reference-free form, which is a simple way to improve their performance.", "reasons_to_reject": "1. As mentioned in the paper, the practice of using sources as references has been employed in various other metrics and is not limited to the summarization task, making it a less novel idea.\n\n2. The lack of crucial experiment details reduces the persuasiveness of the results presented in the paper.", "questions_to_authors": "Can you provide more details about the creation of idf in the usage of BERTScore? From my experience, using traditional BERTScore without applying the idf strategy, only replacing ref with source could not acquire such a high Pearson score. Is the improvement all come from the selection of idf_dict?", "paper_topic_and_main_contributions": "This paper revisits general reference-based metrics for summary evaluation, proposing that using source documents as references can improve the performance of reference-based metrics.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "TRUE: Re-evaluating Factual Consistency Evaluation (Honovich et al., NAACL 2022)  \n\nRefer to line 125, \"While the idea of using 124 the document as the reference is not new\". This work focuses on evaluating consistency and all metrics used take src and sys as input, which should be included in the bibliography.", "Typos_Grammar_Style_and_Presentation_Improvements": "1. In Section 1, the excessive use of brackets to explain the alias of sys and ref is detailed but not entirely necessary. It would be clearer to use abbreviations such as src, sys, and ref throughout the section and subsequent sections (e.g., Section 2.1).\n\n2. Adding bold formatting to highlight the best result in Table 1, Table 2, and Table 3 would enhance clarity and make it easier to identify the top-performing values."}}, {"review_id": "vuMzlSZu8w", "reviewer": "Reviewer_Je5A", "summary": "", "questions": "QA: Can you explain the generality of the approach (e.g. in multi document summarization)?\nQB: Please explain your clues about low numbers for reference-based metrics (Table 4 & 5)?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are encouraged that the reviewer found our method’s strong performance compared with non-GPT baselines and GPT-3.5.\n\n**Reason 1 to reject on generality**: A) **Domain**: Unfortunately, all human evaluation datasets for summarization are from the news domain. We'd be happy to learn and evaluate on any non-news summarization  datasets with human evaluation. B) **multi-document case**: We added summary-level, Spearman's CC on the TAC 2010 dataset below. **BERTScore-based metrics still outperform baselines on the Pyramid score and the overall score by 9% and 7% respectively.** \n\n|  | **Pyramid** | **Linguistic** | **Overall** |\n|---|---|---|---|\n| **Our Max** | **_0.539_** | **0.422** | **_0.500_** |\n| **Baseline Max** | **0.492** | **_0.460_** | **0.470** |\n| bertscore-deberta-large-mnli_precision | 0.496 | 0.401 | 0.455 |\n| bertscore-deberta-large-mnli_recall | **0.526** | _0.405_ | **0.492** |\n| bertscore-deberta-large-mnli_f1 | **_0.539_** | **0.422** | **_0.500_** |\n| bertscore-bart-large-mnli_precision | 0.471 | 0.272 | 0.415 |\n| bertscore-bart-large-mnli_recall | 0.422 | 0.202 | 0.380 |\n| bertscore-bart-large-mnli_f1 | 0.481 | 0.245 | 0.426 |\n| bertscore-roberta-large-mnli_precision | 0.469 | 0.306 | 0.418 |\n| bertscore-roberta-large-mnli_recall | 0.481 | 0.340 | 0.450 |\n| bertscore-roberta-large-mnli_f1 | _0.509_ | 0.356 | 0.464 |\n| **Baselines below**  |  |  |  |\n| Blanc | 0.427 | 0.294 | 0.397 |\n| summaQA-F1 | 0.301 | 0.243 | 0.286 |\n| SummaQA-confidence | 0.290 | 0.152 | 0.262 |\n| Supert | 0.479 | 0.324 | 0.427 |\n| SueNes | 0.492 | **_0.460_** | _0.470_ |\n\n**Reason 2 to reject and Question, the poor performance of ref-based metrics**: First, the ref-based metrics are not our work. The numbers of SummEval were obtained using the SummEval toolkit itself while the numbers on Newsroom were copied from SueNes (Bao et al., NAACL 2022). \n\nAs to why, there are two possible reasons. First, refs in the SummEval and Newsroom datasets are very short, e.g, as short as one sentence in Newsroom. A very short reference provides limited information for a ref-based metric to function. On TAC, the refs are much longer and thus ref-based metrics have higher scores than ref-free ones.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The repurposing approach improves the performance of reference-based metrics (and also making them reference-free) on multiple datasets.\n2. It outperforms non-GPT baselines including reference-free and reference-based metrics.\n3. It has comparable results with GPT3.5-based baseline while having 0.5B parameters.\n", "reasons_to_reject": "1. The metric is just tested on one domain and one type (single document), so its generalization is not clear.\n2. Results for reference-based metrics in result tables (e.g. Table 4 & 5) are very low, any clue?\n2. The writing is not well-organized", "questions_to_authors": "QA: Can you explain the generality of the approach (e.g. in multi document summarization)?\nQB: Please explain your clues about low numbers for reference-based metrics (Table 4 & 5)?", "paper_topic_and_main_contributions": "This paper proposed the idea of re-purposing the reference-based metrics to be used as reference-free metrics for summarization task. The input document is used as the second input instead of the reference summary. By experimenting on different signle-document summarization datasets, they showed that re-purposing approach imporves the performance, and outperforms the non-GPT baselines, and has comparable results with GPT3.5 while having 0.5B parameters.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on summarization evaluation and investigates whether reference-based metrics can be repurposed as reference-free metrics by substituting the reference with the input document. Overall, reviewers agree that this paper has merits (simple/sound approach, promising results) but also point out that 1/ it has a number of issues regarding clarity and that 2/ the experimental results are quite preliminary.  For 1/ I feel that the authors's responses provide some feedback to be included in a revised version of the paper. For 2/ I believe that this is not such an issue for a short paper as this format is adequate for describing a work in progress.
Decision: Accept-Findings