Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model
Authors: Qi Jia
Siyu Ren
Yizhu Liu
Kenny Q. Zhu
Venue: EMNLP 2023 Main (2023)
Keywords: Faithfulness Evaluation
Summarization Evaluation
Hallucination Detection

Abstract: Despite tremendous improvements in natural language generation, summarization models still suffer from the unfaithfulness issue. Previous work evaluates faithfulness either using models trained on the other tasks or in-domain synthetic data, or prompting a large model such as ChatGPT. This paper proposes to do zero-shot faithfulness evaluation simply with a moderately-sized foundation language model. We introduce a new metric FFLM, which is a combination of probability changes based on the intuition that prefixing a piece of text that is consistent with the output will increase the probability of predicting the output. Experiments show that FFLM performs competitively with or even outperforms ChatGPT on both inconsistency detection and faithfulness rating with 24x fewer parameters. FFLM also achieves improvements over other strong baselines.

Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model
Qi Jia
Siyu Ren
Yizhu Liu
Shanghai Jiao Tong University, China
Meituan, China
(Jia_qi, roy07e2jesjtu.co edu.cn
liuyizhuêmei tuan com
Kenny Q. Zhu"
University of Texas at Arlington, USA
kenny. zhu@uta. edu
Abstract
2020) or doing post-hoe corrections (Chaudhury et al., 2022).
Despite tremendous improvements natural language generation, summarization models still suffer from the unfaithfulness issue. Pre- vious work evaluates faithfulness either using models trained on the other tasks or in-domain synthetic data, or prompting a large model such as ChatGPT. This paper proposes to do zero-shot faithfulness evaluation simply with moderately-sized foundation language model. We introduce new metric FFLM, which is a combination of probability changes based on the intuition that prefixing a piece text that is consistent with the output will increase the probability of predicting the output. Ex- periments show that FFLM performs compet- itively with both inconsistency detection and faithfulness rating with 24x fewer parameters. FFLM also achieves improvements over other strong base- lines.
Most previous work for faithfulness evaluation either takes advantage of models trained lated tasks for zero-shot evaluation (Goodrich al., 2019; Falke et al., 2019; Wang et al., 2020), or does weakly-supervised evaluation with synthetic in-domain data (Kryšciński et al., 2020). The for- mer requires transferring out-of-box models to the summarization domain (Mishra et al., 2021), which lacks guarantees on the models' performance and suffers from error propagation (Ji et al., 2023). The latter one shows poor generalization ability (La- ban al., 2022) as a result of the limited synthetic rules that couldn't cover various kinds of halluci- nations. Recently, as ChatGPT (OpenAI, 2022) has shown amazing generation abilities on various tasks, researchers attempt to do human-like evalu- ation by designing prompts to query the model in the zero-shot manner (Luo et al., 2023). However, such strong language models still sensitive nuances, showing unstable performance with dif- ferent wording of prompts (Gao et al., 2023; Chen et al., 2023).
outperforms ChatGPT on
1 Introduction
Faithfulness evaluation for text summarization aims at measuring if the information in a sum- mary is fully covered by and consistent with the source document !. Although automatic text sum- marization has achieved remarkable improvements with pre-trained language models (Zhang et al., 2020; Lewis et al., 2020; Liu et al., 2021, 2022a; Zhang et al., 2023) in recent years, especially in the aspect of fluency and informativeness. However, these neural models tend to generate unfaithful summaries. An effective faithfulness evaluation metric not only helps for implementing summariza- tion systems in real applications but also plays key role developing more faithful summarization models, such as by data filtering (Matsumaru et al.,
Considering the above weaknesses, we think that an ideal faithfulness evaluation metric for sum- marization should be independent of other tasks and -atasct-specffic expertise, be able to general- ize among different benchmarks and robust for the same document-summn concludes that instruction tuning is just to teach the model to produce high-quality output while al- most all of the knowledge has been learned during pre-training for large language models. Based on their findings, we wonder: can we get rid of the popular prompting approaches and calculate the faithfulness score simply with a foundation lan- guage model, which meets the above expectations?
pair. Zhou et al. (2023)
The corresponding author.
We use the words "faithfulness". "consistency" and "(without) hallucination" interchangeably. Extrinsic hallucina- tions that are correct to the world knowledge are regarded as unfaithfulness in this work.
In this work, we propose a metric named FFLM for zero-shot faithfulness evaluation with a founda- tion language model. The intuition behind FFLM

---

is that the generation probability of piece of text will increase when prefixing another piece of con- sistent text. Following this intuition, we classify different kinds of probability changes into changes with prior probability and changes with conditional probability. The former contains a comparison be- tween the vanilla sequence-to-sequeee seequeeeee probabili- ties of the summary given document and uncondi- tional probabilities of the summary, and similar comparison by changing the position of the docu- ment and the summary. The latter calculates the vanilla other conditional probability by adding a piece of prefix text. Similar intuition has been considered in previous works (She et al., 2023; Son et al., 2022). The major differences are that their metrics were carried out models fine-tuned by summariza- tion data and they only consider a single kind of probability changes. Our FFLM is based on the foundation language model, and we hypothesize that these different probability changes capture dif- ferent hallucinations (see Sec. 4.4) which should be considered as whole.
2 Approach
Given source document X { Fn and the corresponding summary Y = {y), J. the goal of this work is to design a metric FFLM mea- suring the faithfulness of Y based on the founda- tion model LM-()) We adopt L.M(.) under the teacher-forcing strategy, which can provide se- quence of generation probabilities p of given text with without other conditional inputs. We first introduce three probability changes for faithfulness measurements and then propose a feasible design of our comprehensive metric FFLM. Scores pro- posed by She et al. (2023) and Son et al. (2022) are in Appendix A.
probability with an-
2.1 Faithfulness Measurements via Probability Changes
The intuition is that the generation probability of piece of text will increase when providing more related and consistent information. On the contrary. the generation probability will drop when condi- tioned inconsistent information. Accordingly, we considered three different probability changes in two categories as follows.
On top of these three components of probability changes, we introduce a feasible design of FFLM by re-weighting each token and each component to get the final faithfulness score. We did experi- ments in both the inconsistency detection setting and the faithfulness rating setting for summariza- tion evaluation. The results show the favorable performance of our FFLM across different settings and datasets 2, Our contributions are as follows:
Changes with Prior Probability: The prior probability of Y can be estimated by the foundation model (-M()):
p = LM(Y) = (r
Im P
and the sequence-to-sequee probability of y given X is:
We propose to do zero-shot faithfulness evaluation based on foundation language model(Sec. 4.6).
22 LM(Y (r M
(2)
If Y is faithful summary, the sequence-to sequence probability pÿ?² should be larger than the prior probability p\u' as more information consis- tent Y is given by conditioning on X. Therefore, faithfulness measurement can be defined as:
We introduce a comprehensive evaluation metric FFLM by calculating the probability changes of the desired output in different ways(Sec. 2) and verify the rationality of our
metric des.gn(Sec...3).
1 m
A prior Py
Jm 91
Experiments on different evaluation settings show that FFLM based on LLaMa with only billion parameters can achieve competitive performances or even outperforms ChatGPT among different datasets (Sec 4.1 and 4.2).
From another point of view, we expect that the generation of Y highly relies on X, instead of para- metric knowledge stored in LM which is main resource of hallucinations (Ji et al., 2023).
Similarly, faithful Y can support the contents in X. Thus, the differences between the sequence- to-sequence probability of X given and the prior
"The code and dataset for this paper are available at https: //gi thub com/ Ji wwt.iaaiSSSuaaaaaaa Fahtvalllaatlllllaaeaa

---

probability of X is another reasonable measure-
Finally, FFLM is combination of these metries:
ment:
PX (X) = {p=z Hi=1
FFLM = aA. + + + &dA cond
(7)
22 (X = [Pr H
Px
(4)
where a, 3, and are weighting parameters in the range of O to 1 and a + 3 + á = 1. These three weights can be tuned on validation set, or set manually as hyper-paramettrss.
Aprior
22 Px:
Pas
Changes with Conditional Probability: In- stead of comparing sequueeee-t-sequence gener- ation probabilities with prior probabilities, another way is to add more information P besides the in- put document X, leading to an influecce on the generation probability of Y. Following She et al. (2023), we simply set P = Y. In this way, if Y is inconsistent with X, prefixing P will cause in- formation contradictions in the input and decrease the probability of Y compared to a consistent one. Mathematically, the third measurement is:
Experiment Setup
We present two evaluation settings considered by previous work for faithfulness evaluation first, with the implementation details of FFLM for them later.
3.1 Inconsistency Detection
Inconsistency detection regards the faithfulness evaluation as a binary classification problem. In other words, human annotators or automatic met- rics only need to recognize whether the summary is faithful to the document not.
Derel LM (ppror)le
(5)
cond
T
22
Py
Datasets: The SUMMAC Benchmark (Laban et al., 2022) is benchmark consisting of six summarization evaluation datasets, including Co- GenSumm Falke et al. (2019), SummEval (Fabbri et al.. 2021), FRANK (Pagnoni et al.. 2021), Poly- tope (Huang et al., 2020), FactCC (Kryšciński et al., 2020) and XSumfaith (Maynez al., 2020). It stan- dardized these datasets by changing their original labels into binary label and split each dataset into validation set and a test set. Most of the original datasets are labeled by three or more annotators, except Polytope and FactCC.
1=
We didn't consider X and Y reversely here. The main reason is that inputting the sequence [P- = X, Y, X] to (-) is much more costly and may exceed the max sequence length of most models since X is much longer than vie... >>
2.2 A Feasible Design of FFLM
Goyal et al. (2022) found that high-loss tokens generally correspond to unfaithful contents during training summarization model. Inspired by this finding and the success of the loss truncation train- ing algorithms (Kang and Hashimoto, 2020), we think that more attention should be paid to such high-loss (or low-prod (ww-robabbilit) tokens when calcu- lating the faithfulness scores. So, instead of simply averaging the probability changes to get the final score for an (X, Y) pair, we adopt two operations. First, we take the logarithm of the probabilities be- fore subtraction, which will magnify changes on the low-probability tokens. Second, we re-weight each token based on p8?s and yx² correspondingly. We get:
Evaluation Metric: Balanced accuracy (Broder- sen et al., 2010) is adopted as the primary evalua- tion metric, which requires binary labels for com- putation. For approaches with continuous scores, threshold can be selected via the validation set. Baselines: We borrowed the baselines from Laban et al. (2022)'s work, including linguistic feature-based metrics NER-Overlap (Laban et al., 2021) and DAE (Goyal and Durrett, 2020), NLI- based metric MNLI-doc (Kryšcinski et al., 2020) and SUMMACZs (Laban et al., 2022), QA-based metrics FEQA (Durmus et al., 2020) and QuestE- val (Scialom et al., 2021), prompting with Chat- GPT (OpenAI, 2022) and two weakly-supervised baselines FacICC-CLS (Kryšciński et al., 2020)
A Aprior
m
Po
2x (log 222
log p
m
Aprior
1
220
(log Pz:
bm) log P
71
*As Chen et al. (2023) shows that faithfulness evaluation is less reasoning-intensive and chain-of-though (Wei et al., 2023) prompting even hurts performances, we only compared with ChatGPT using a vanilla prompt.
A
cond
m
24 e
(log 2
- log pref
i=1
(6)

---

Evaluation Metrics: Pearson(7), Spearman(p), and Kendall(T) correlation coefficients are used to measure the alignments between faithfulness rat- ings annotated by annotators and automatic metrics. The correlations are the higher the better. We con- sider the summary-level correlations for all datasets. Besides, sestem-level correlations are calculated on SummEval which contains annotations for 16 extractive or abstractive summarization models.
Setting
Dataset
Val
Test
Source
CotionSum
1281
400
C
Inconsistency
SummEEal
850
850
Detection
FRANK
671
1575
C+X
Baselines: Rouge-2 F1 (Lin, 2004), Me- teor (Banerjee and Lavie, 2005), BLEU (Papineni et al., 2002) and BERTScore FI (Zhang et al., 2019a) are widely-accepted summarization eval- uation metries. We report their best results in Son et al. (2022) by calculating between the summary and the source document. QAGS (Wang al., 2020) another QA-based metric. Others are the same as the ones for inconsistency detection.
(SUMMAC
Polytope
634
634
C
Benchmark)
FacCCC
XSumFaith
1250
1250
RANKKNN
1250
Faithrulness
QAGSCNN
235
SummFral
1600
Rating
FRANKXSUM
996
QAGSXSUM
239
Statistics of the datasets. "C" and "X" are short for CNN/DM (Nallapati et al., 2016) and XSum (Narayan al., 2018) respectively.
Table 1:
3.3 Implementation Details
and SUMMACcONV (Laban et al., 2022). Be- sides, we implemented the language modeling- based metric BARTScore (Yuan et al., 2021) and metrics based on probability changes include CoP (She et al., 2023) and HaRiM (Son et al., 2022). These three metrics were suggested to use the CNN/DM (Nallapati et al., 2016) fine-tume tne-tuned BART model for calculation. We also improved the latter two metrics with our proposal by calcu- lating with a foundation language model, LLaMa, for comparisons.
We implemented FFLM with the foundation lan- guage model LLaMa (Touvron et al., 2023). It con- tains models with different sizes, where LLaMa7b is selected for our main experiments. We add "TL:DR" between the conditional sequence and the target sequence. The weights in Eq. 7 are de- termined in {0.0, 0.1, . 1.0} according the per- formance on the corresponding validation set for inconsistency detection. For faithfulness rating, we set a, 3, as 0.25, 0.25, 0.5 respectively, with the intuition that the former two are from the same cat- egory introduced in Sec. 2.1. Our experiments are done on single RTX 3090.
3.2 Faithfulness Rating
Faithfulness rating defines the evaluation as a Lik- ert scale coring problem. Annotators or metrics score each summary according to its faithfulness. Generally, the higher, the more faithful.
Results and Analysis
This section includes the main results for incon- sistency detection and faithfulness rating, together with an ablation study, an analysis error types, and comparisons of different model sizes of our FFLM. We also discussed our metric and the prompting approach with or without instruction tuning under the same model size.
Datasets: Following Son et al. (2022), we exper- imented on five different datasets: FRANKCNN and FRANKXSUM from Pagnoni et al. (2021), QAGSCNN and QAGSXSum from Wang et al. (2020), and SummEval (Fabbri et al., 2021). For the first four datasets, human judgments were orig- inally done on the sentence level. The faithfulness rating of the whole summary is collected by doing majority voting each summary sentence among annotators and averaging among sentences. Sum- mEval contains human scores in the range of to 5 in the aspect of consistency. More details are in Table
4.1 Performance on Inconsistency Detection
The results inconsistency detection are in Ta- ble 2. Our proposed metric FFLM achieves state- of-the-art performance on 3 datasets including Co- GenSum, Summlival, and FRANK, and outper- forms ChatGPT on 5 out of datasets from the SUMMAC benchmark except XSumFaith
Both Polytope and FactCC are only labeled by a single annotator. As result, their labels may not be as convincing the other datasets. Although QuestEval, the best QA-based metric, achieves the top-1 accuracy on Polytope, performs mediocrely on the rest. The weakly FactCCcLs and SummaCConv are trained with synthetic data constructed with human expertise
baselines
aac.ugggiggaa.e..oom wtt.aaccccccaaaaaeeeeeeeeeee co/facebook/ dart-large-co

---

Metric
CoGenSur
SummEEvll
FRANK
Polytope
FaaCC
XSumFuith
NER Overlap
5300
56.8
619
52.0
55.0
63.3
MNLI doc
57.6
66.6
63.6
61.0
61.3
57.5
Fac1CCo18
63.1
60.1
39.4
61.0
75.9
57.6
DAF
63.4
70.3
61.7
62.8
75.9
50.8
TEQA
QuestEval
62.6
72.5
82.1
70.3
66.6
62.1
SummaCzs
SummaCcionn
64.7
81.7
81.6
62.7
89.5
66.4
BARTScure
62.5
66.7
80.2
57.3
68.4
56.9
COPBART
65.3
63.9
77.7
60.0
HARiMBART
58.9
76.6
81.8
55.8
67.3
56.2
ChatGT
COPLLaMs
65.4
83.6
83.1
55.4
78.6
54,1
HaRiMLLaMM
FFLM
71.8
84.4
83.9
61.5
77.3
58.9
Table 2: Balanced accuracy($) (6) on the SUMMAC benchmark. The best result for each dataset is in bold. Scores FFLM better than other metries based on the foundation model are underlined.
PRANKCNN
QAGSCNN
SummEval
HRANKXSUM
QAGSXSUM
Metric
r
T
Rouge-2
33.1
32.7
24.9
47.5
42.7
31.5
24.7
25.2
19.5
33
2.7
10.7
9.1
69
Meteor
23.0
22.9
17.4
27.7
32.4
23.4
14.3
12.2
11.2
0.5
0.4
-1.5
-7.1
-5.2
BLEU
9.3
20.2
15.3
18.0
33.7
24.5
11.7
7.3
9.1
-4.2
-4.6
-3.8
-18.6
-13.9
SeRTS
31.4
46.4
35.8
49.3
36.5
29.2
29.5
23.0
13.7
11.1
FactOC
49.2
43.8
37.6
32.0
34.0
7.2
7.2
7.1
QAGS
31.4
26.7
20.6
46.6
38.2
27.4
17.7
12.7
-2.2
-0.7
-0.6
21.7
20.3
15.3
QuestEval
49.2
44.5
37.0
33.9
9.6
DAE
44.0
44.7
34.2
20.0
27.0
11.3
9.2
BARTScore
56.1
53.0
41.3
67.3
61.3
47.0
24.9
26.2
19.7
17.4
16.8
13.7
9.7
7.2
BART
18.0
20.8
HARiMPART
61.0
53.9
42.1
67.4
58.2
47.1
42.7
37.6
29.8
14.8
13.9
11.4
15.8
16.0
13.1
ChaiGPT
CoPLLaMA
59.7
54.5
42.6
74.3
67.6
54.8
55.1
46.4
37.0
24.6
23.1
18.8
19.0
18.1
14.7
HaRiMLL.MMa
FFLM
62.2
56.0
43.7
72.3
65.3
53.0
56.3
46.9
37.4
22.0
25.3
20.6
28.3
27.1
22.2
Table 3: Summary-level correlations (6) on the faithfulness rating datasets.
that may have certain similarities with the FactCC dataset. Therefore, FactCCcus shows strong per- formance the FactCC dataset while relatively weak on the others including datasets in Table the same as the findings Laban et al. (2022). Also, that's 's why SummaCCunv shows around 12% significant improvements over our FFLM.
4.2 Performance on Faithfulness Rating
Summary-level results are in Table 3. The results of ChatGPT borrowed from Luo et al. (2023) show its inconsistency improvements among datasets: It doesn't It exceed previous baselines FRANKCNN, performs similarly on SummEval, and achieves conspicuous gains on FRANKXSUM. Besides, similar to the above analysis for comparisons among probability change-based metrics, our FFLM induces performance gains on 4 out of 5 datasets over CoPLLaMa and HaRiM espe- cially on datasets sourced from XSum. Unfor- tunately, FFLM still lags behind ChatGPT with 175 billion parameters on FRANKXSUM, showing ChatGPT's strong ability on dealing with highly abstractive summaries. This is also in line with ChatGPT's favorable performance on XSumFaith in Table 2. After all, FFLM achieves the best scores on FR FRANKCNN, SummEval, and QAGSXSUM, and performs competitively the other datasets.
Concentrating the metrics based on proba- bility changes, zero-shot metries COPBARE and HaRiMBART perform badly compared with previous SOTA SummaCzs. showing the poten- tial of using probability changes for faithfulness evaluation. After introducing the foundation lan- guage model, their performances don't drop in most cases, indicating that fine-tuning with in- domain data is not necessary. However, the leading performance between these two metrics is unsta- ble among datasets. HaRiMr,LaMa outperforms CoPLLaME FRANK and Polytope, while on the rest datasets, the opposite true. FFLM, as com- prehensive metric, successfully achieves improve- ments over both of them on out of datasets.
We also report the system-level results on Sum- mEval in Table 5. FFLM performs similarly to ChatGPT according to the Spearman correlation.

---

FRANKCNN
QAGSCNN
SummEval P
FRANKXSUM
QAGSXSUM
Metric
P
P
7
P 27.1
7
FILM
62.2
56.0
43.7
72.3
65.3
53.0
36.3
46.9
37.4
27.0
25.3
20.6
28.3
22.2
Ablations on the metric components
Agnor
26.4
20.1
11.9
7.5
20.5
20.0
15.6
20.9
21.3
17.4
orier
36.0
27.4
23.7
26.3
10.2
4.8
3.2
2.6
A
59.9
54.6
42.7
74.3
67.9
55.2
54.8
46.3
36.9
24.7
23.3
19.0
19.4
18.0
14.7
34.7
29.2
22.3
17.7
13.0
9.9
28.3
23.6
18.5
20.4
19.7
16.0
20.7
21.3
17.4
Brior Adoud
54.2
68.1
60.2
45.1
28.3
29.4
28.6
23.4
A Aamer AA Arona
60.3
54.7
42.6
73.9
66.5
54,0
54.7
46.6
37.1
24.7
23.2
18.9
19.8
18.3
14.9
Atlations on the metric derigns
wño
61.2
54.8
42.7
68.4
60.1
48.6
54.3
45.4
36.3
26.9
25.0
20.4
26.4
26.00
21.3
wro log
57.5
$2.3
40.5
69.2
60.3
48.5
56.9
45.9
36.7
19.7
17.5
14.3
11.5
12.2
10.0
who and log
36.3
51.3
39.6
66.5
57.6
46.4
54.4
45.0
36.0
18.8
16.6
13.5
11.0
12.4
10.2
Ablations on the combinationn weightt [a. 3,
same
60.9
$4.0
42.1
67.5
58.6
47.1
54.1
44.9
35.8
28.3
26.4
21.5
29.2
28.7
23,5
Table 4: Ablations of FFLM on faithfulness rating. The highest scores are in bold.
Metrie
7
P
is much more efficient for faithfulness evaluation.
Rongo-2
50.0
59.9
68.8
BLEI
45.0
28.7
62.1
4.3 Ablation Study on Metric Designs
BERTScore
68.3
68.0
86.8
RARTS
30.1
25.9
18.3
We carried out ablation studies of FFLM on faith- fulness rating in Table 4. The ablation results inconsistency detection are in Appendix B.
COPRART
19.9
35.9
25.0
HaRiMBARTT
CharGPT
83.3
CoP LoPLL
63.3
HaRiMrataMa
89.9
84.4
66.7
Ablations on the metric components: We test different combinations of the three probability changes. The results show that A is the most powerful component of FFLM. Its combination with Aprior ranks first among ablations on both FRANKXSUM and QAGSXSUM. Together with A prior our metrie FFLM shows over 5% increases in Spearman correlation on QAGSCNN, 1.8% on FRANKCNN and SummEval, without much loss on the other two datasets, records more robust re- sults. Moreover, combining different probability changes induces performance gains in most cases, reflecting the necessity of designing a comprehen- sive metric(More in Sec 4.4).
FFLM
90.4
83.2
65.0
Table 5: System-level correlations between metrics and human ratings on the SummEval dataset.
HaRiMILLaMa achieves a bit higher Spearman and Kendall correlation than FFLM, while FFLM per- forms better Pearson correlation showing bet- ter linear correlation with human scores. More- over, FFLM is more robust than HaRiM on different evaluation settings considering the poor summary-level performance of HaRiMLLaMA es- pecially for FRANKXSUM and QAGSXSUM in Table 3. Another observation is that although CoP and HaRiM backed on BART perform closely with them backed on LLaMa the summary-level eval- uation, they perform poorly the system-levee evaluation. This can be attributed to the fact that metries based on CNN/DM fine-tuned BART have inductive bias (Son et al., 2022). They tend to prefer summaries generated by abstractive models, while extractive models are generally more faithful. Mcanwhile, metries based on the foundation lan- guage model don't it show this bias, leading to the best results for ranking summarization systems.
Ablations the metric designs: We use and log annotate the token-level weights and the logarithm operation introduced in Sec 2.2. Both operations contribute to the final FFLM, where log is more effective for datasets sourced from XSum and w for the others.
Ablations the combination weights: For the faithfulness rating task where we empirically the weights a, 3 and d as 0.25, 0.25 and 0.5, we compared with the equaling weights, i.e.,
a = 3 = 8 J. FFLM performs relatively better.
4.4 Analysis on Error Types
To recap, our FFLM generalizes well among different task settings and different datasets, show- ing favorable performance over the baselines. is backed on L.LMMa with only billion parameters and performs competitively with or even outper- forms ChatGPT with 175 billion parameters, which
By taking look at the correlations between pairs of the metric components in Figure 6, we can see that the correlations vary among different datasets. None of the pairs show a high degree of correla- tion, indicating that these components may capture

---

FRANKCNN
QAGSCNN
SummEEal
FRANKXSUM
QAGSXSUM
Metric
7
7
7
P
P
7
P
7
Ay Ax
Agrion
44.7
32.0
32.5
23.8
29.5
23.7
24.9
A
Aprion Agand
26.5
12.9
14.7
1.2
20.1
15.6
10.1
23.4
20.1
13.4
-6.2
-2.4
-1.5
A Arrin
Agand
47.2
49.9
34.6
35.2
41.2
28.0
36.5
40.6
27.9
38.7
38,3
25.9
-0.4
3.8
2,5
Table 6: Correlations between pairs of the metric components on faithfulness rating.
FRI.M
28.9
30
FRANKONN
QAGSCNN
66
4/
27
30
56
66
47
33
64
46
4
10
10.4
15.9
20
18R
10
4
28.4
14.2
26.5
aрms
26.6
41.7
32.0
FRANKXSUM
QAGSXSUM
25
(a) FRANKCNN
(b) FRANKXSUM
24
24
Figure Spearman correlation (%) of different error types on FRANKCNN and FRANKXSUM Highest correlations for each A is highlighted with red boxes.
Figure 2: Spearman correlation(%) of FFLM with dif- ferent model sizes on faithfulness rating datasets.
unfaithfulness from different aspects.
to 41.7% on FRANKXSUM for Disc.
To figure out if different probability changes cor- relate well with different error types in the gen-
FFLM made further improvements over Agand on both Sem and CVer, showing that combining different probability changes is reasonable and ef- fective in most cases except Dises.
take advantage of labels in the FRANKCNN and FRANKXSUM datasets. Pagnoni et al. (2021) divides the factual errors in the generated summaries into three groups. Seman- tic frame (rrors(Sem) include errors on the predi- cate, entities, and additional information about the circumstance. Discourse errors consist of coreference errors and discourse link errors. Con- tent verifiability errors(CVer) are closely related extrinsic hallucinations (Ji et al., 2023), contain- ing the out-of-article error and grammatical error. We randomly picked 50 error cases and 10 error cases for each error type from FRANKCNN and FRANKXSUM respectively,, and mixed them with the rest faithful summaries. Spearman correlations averaged over 10 times are in Fig. 1.
erated summaries,
4.5 Performance on Different Model Sizes
To test FFLM's performance on different models sizes, we select LLaMa with billion(3B), 7 bil- Iion(7B) and 13 billion(13B) parameters that are trained on the same data volume with trillion tokens and draw the diagram in Fig. 2 for faith- fulness rating datasets. The scores consistently increase from LLaMa-3B to LLaMa-7B across the five datasets, while the improvements not sistent for 1LaMM--13B. Given certain amount of data, increasing the number of parameters can enhance the model's language modeling ability and be helpful to faithfulness evaluation. On the other hand, when the model size keeps scaling up. more unexpected biases in the pre-training corpus may be memorized and will hurt the performance. This has also been pointed out by Ranaldi et al. (2023) and Nadeem et al. (2021).
We observed that Agond captures different errors best, which accord with the ablation results in Ta- ble 4. Comparing among the scores for each hor- izontally, we can see that the probability changes with prior probability is good at CVerr errors on both datasets, and Apoud at Sem eтrors or Disc errors. The differences among datasets reflect their dif- ferent characteristics (Pagnoni et al., 2021). Sum- made up of multiple
In this way, we think that using larger foundation models may not be the best choice for faithfulness evaluation on summarization, which is also closely related to the research on investigating the optimal
maries in FRANKCNN sentences, resulting more diverse and challeng- ing situations for Disc errors than FRANKXSUM with single-sentent summaries. Thus, Agonn in- creases dramatically from 14.2% on FRANKCNN
"The corresponding checkpoints from hugging face are openIm-rescarch/open Ilama_3h, dccapoda-rescarchl 7b-hf, and decapoda-rescarcc/llaa 13b-hf.

---

FRANKONN
QAGSCNN
SummEval
FRANKXSUM
QAGSXSUM
Metric
7
7
T
P
P
Prompiing Appmaach
LB
-1.3
-0.2
-0.2
2.4
1.9
9.5
11.5
10.9
3.0
3.0
-6.9
-7.0
-6.9
Vicuna-18
17.6
17.4
15.5
18.9
19.8
17.7
13.1
11.9
11.1
7.0
5.5
5.1
10.2
8.4
8.0
Alpaca
3.4
1.4
Our Approoek
LLaMa-7B
62.2
72.3
53.0
17.4
25.3
27.1
Vicuna-7B
62.7
$6.7
44.3
73.1
67.2
54.6
55.3
47.2
37.7
25.8
23.9
19.4
23.5
22.5
12.8
Alpaca-78
61.4
55.3
43.2
71.4
52.6
55.8
47.1
33.6
20.0
24.2
25.4
20.7
Table 7: Comparisons with prompting and instruction-tuning techniques under the model size. The highest correlations are in bold in each column and are underlined among each kind of approach.
model size and dataset size for training foundation language models (Hoffmann et al., 2022).
guage tasks. Goodrich al. (2019) adopted in- formation extraction tools to extract the fact ples from both the source document and the sum- mary. Tuple mismatches reflect the hallucina- tions. The intuition behind question-answering- based metries (Wang et al., 2020; Durmus et al., 2020; Scialom et al., 2021) is that identical an- swers should be generated when asking the same question to summary and the corresponding doc- ument respectively. Natural language inference also shares commonalities with faithfulness eval- uation in the way that information in a consistent summary should be entirely entailed by the source document (Falke et al., 2019; Mishra et al., 2021; Laban et al., 2022). However, all of these metrics highly rely on the domain-transfer ability of out-of- box models and suffer from error propagation.
4.6 Comparisons with Prompting and Instruction-tuning
We compare our metric with prompting and instruction-tuning techniques under the same model size in Table for faithfulness rating. Here, LLaMa-7B is the vanilla foundation language model. Vicuna-7B (Chiang et al., 2023) and Alpaca-7B (Taori et al., 2023) are initialized from LLaMa-7B and instruction-tuner with data col- lected in different ways. We present the maximum scores for each dataset among different prompts de- signed by previous works (Chen al., 2023; Gao et al., 2023; Luo et al., 2023). The detailed prompts for each evaluation task are listed Appendix C.
First, we observe that using models containing 7 billion parameters, FFLM outperforms the prompt- ing approach across different models and datasets. The prompting results here lag behind the perfor- mance of ChatGPT dramatically. This leads to the conelusion that the effectiveness of prompting approaches relies highly on much larger models, while our metric FFLM can be cheaper alternative with smaller models. Second, instruction tuning is important for improving the prompting approach, while is not necessary for our FFLM. It enhances the models' understanding ability on instruction templates in the prompts by further tuning with relatively small datasets. However, such manually collected datasets may contain unconscious bias and hurt FFLM's performance.
Instead, weakly-supervised approaches choose to train classifiers by constructing synthetic in- domain data with heuristics by experts. Differ- ent kinds of inconsistency errors are simulated by perturbing the reference document-summary pairs (Kryšciáskii et al., 2020; Utama et al., 2022: Yin et al., 2021). The limited heuristic makes it hard to cover all kinds of errors and shows poor generalization ability among datasets (Laban et al., 2022). As language modeling based metrics (Egan et al., 2022; Liu et al., 2022b) receive more atten- tion, another small group of work for faithfulness evaluation computes probability changes with mod- els fine-tuned on summarization datasets (She et al., 2023; Son et al., 2022; Xie et al., 2021), showing biased preference for abstractive summaries. Based on this line of work, we propose FFLM based on the foundation language model. Our zero-shot met- ric doesn't 't require further training with in-domain or synthetic data and shows a strong generalization ability.
5 Related Work
5.1 Faithfulness Evaluation for Summarization
Faithfulness evaluation metrics can be classified into zero-shot ones and weakly-supervised ones. Zero-shot evaluation metrics mainly take advan- tage of the models trained with related natural lan-

---

5.2 Evaluation with Large Language Models
based on specific analysis of different error types will be considered in the future. Besides, in this work, our FFLM only calculates a single score for the whole summary without pin- pointing the exact erroneous words or the specific error type. Considering the success of CoP (She et al., 2023) on token-level inconsistency detec- tion and detailed inconsistency category evaluation, hypothesize that our metric FFLM can be also used for these evaluation scenarios by adjusting the aggregation weights or combining it with the prompting approach.
With orders of magnitude more parameters and ex- tensive training large-scale data, large language models (LLMs) (Brown et al., 2020; Touvron et al.. 2023) have exhibited surprising abilities that may not be observed in previous small language models. The strong capability in language comprehension naturally spurs research in exploring LLMs as bet- ter automatic evaluators for various text generation systems (Wang et al., 2023).
There are also some attempts of faithfulness evaluation by prompting large models (Luo et al., 2023) with different templates and strategics, such as adding detailed definitions (Gao et al., 2023) and Mfain-of-thouglle (Chen et al., 2023). None of these strategies achieve consistent improvements over the original prompt. Besides, neural models are sensitive to the choices of words (Chen et al., 2023), resulting in unstable performances(See Ap- pendix D).
Moreover, we limit our scope to faithfulness eval- uation for text summarization in this paper because the definition of faithfulness evaluation for other generation tasks has some non-trivial differences. For example, the chit-chat utterances in dialogue generation (Dziri et al.. 2022) are supposed be acceptable under the evaluation for faithfulness, in- stead of being regarded as extrinsic hallucinations.
The evaluation for sentence paraphrasing (Zhang et al., 20) should be bi-directional, i.e., the first sentence has to be consistent with the second one, and vice versa. We consider transferring FFLM with adjustment on the other tasks future work.
Our FFLM takes advantage of the strong capa- bility of LLMs for faithfulness evaluation in dif- ferent way and shows competitive performance re- quiring a much smaller number of parameters than the well-known ChatGPT (OpenAI, 2022).
6 Conclusion
References
This paper focuses on zero-shot faithfulness eval- uation for summarization and introduces a novel evaluation metric FFLM which is simply based on the foundation language model. Experiments on both the inconsistency detection benchmark and faithfulness rating datasets show the strong general- ization ability of FFLM across various task settings and different datasets. It also shows favorable per- formance over strong baselines including ChatGPT, Using our proposed metric for more fine- me-igrieeed consistency detection and designing more faithful summarization systems are future directions.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evalu- ation measures for machine translation and/or sum- marization, pages 65-72.
Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, and Joachim M Buhmann. 2010. The bal- anced accuracy and its posterior distribution. In 2010 20th international conference on pattern recog- nition, pages 3121-3124. IEEE.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1901. Curran Associates. Inc.
Limitations
The main idea of this work is to do faithfulness evaluation based on a foundation language model by a combination of different probability changes. FFLM is just feasible but not perfect metric de- sign. Although it makes improvements over each A on almost all of the datasets Table 4, it failed on the errors related discourse errors on the FRANKCNN and FRANKXSUM dataset accord- ing to Fig. 1. Designing better aggregation metrics
Subhajit Chaudhury, Sarathkrishna Swaminathan, Chu- laka Gunasekara, Maxwell Crouse, Srinivas Ray-

---

ishankar, Daiki Kimura, Keerthiram Murugesan, Ramón Fernandez Astudillo, Tahira Naseem, Pavan Kapanipathi, et al. 2022. X-factor: A cross-metric evaluation of factual correctness in abstractive sum- marization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Procexs- ing, pages 7100-7110.
Tanya Goyal and Greg Durrett. 2020. Evaluating fac- tuality in generation with dependency- entail- ment. In Findings of the Association for Computa- rional Linguistics: EMNLP 2020, pages 3592-3603.
Tanya Goyal, Jiacheng Xu, Junyi Jesay Li, and Greg Durrett. 2022. Training dynamies for text summa- rization models. In Findings of the Association for Computational Linguisties: ACL 2022, pages 2061- 2073.
Shiqi Chen, Siyang Gao, and Junxian He. 2023. Eval- uating factual consistency of summaries with large language models. arXiv preprint 2rXiv:235..1400
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford. Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. An empirical analy- sis of compute-optimal large language model train- ing. Advances Neural Information Processing Systems, 35:30016-300 630000.
Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing- 2023. Vicuna: An open- source chathot impressing gpt-4 with 90%* chatgpt quality.
Esin Durmus, He He, and Mona Diab. 2020. Feqa: A
Dandan Huang. Leyang Cui, Sen Yang. Guangsheng Bao, Kun Wang, Jun Xie, and Yue Zhang. 2020. What have achieved on text summarization? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages
question answering evaluation framework for faith- fulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5055- 5070.
Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Os- mar Zaiane, Mo Yu, Edoardo M Ponti, and Siva Reddy. 2022. Faithdial: A faithful benchmark for information-seeking dialogue. Transactions of the Association for Computational Linguisties. 10:1473-1499 1490.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallu- cination in natural language generation. ACM Com- puting Surveys, 55(12)11-3.
Daniel Kang and Tatsunori B Hashimoto. 2020. Im- proved natural language generation via loss trun- cation. In Proceedings of the 58th Annual Meet- ing of the associationnfr Computational Linguisties. pages 718-731.
Nicholas Egan, Oleg Vasilyev, and John Bohannon. 2022. Play the shannon game with language mod- els: A human-free approach to summary evaluation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 10599-10607.
Wojeiech Kryšciński, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 9332-9346.
Alexander Richard Fabbri, Wojciech Kryšciński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re- evaluating summarization evaluation. Transactions of the Association for Computational Linguistics. 9:391-409.
Philippe Laban, Tobias Schnabel, Paul Bennett, and Marti A Hearst. 2021. Keep it simple: Unsupervised simplification of multi-paragraph text. In Proceed- ings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 1 1ihh Interna- tional Joins Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pages (6778
Tobias Falke, Leonardo FR Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019, Ranking generated summaries by correctness: An in- teresting but challenging application for natural lan- guage inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 2214-2220.
Philippe Laban, Tobias Schnabel, Paul Bennett, and Marti A Hearst. 2022. Summac: Re-visiting nli- based models for inconsistency detection in summa- rization. Transactions of the Association for Compu- tational Linguistics. 10: 167-177
Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang. and Xiaojun Wan. 2023. Human- like summarization evaluation with chatgpt. arXiv preprint arXiv:2304.0055::
Ben Goodrich, Vinay Rao, Peter J Liu, and Moham- mad Saleh. 2019. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 166-175.
Mike Lewis, Yinhan Liu, Naman Goyal, Mar- jan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising eequence-to-sequenee pre- training for natural language generation, translation,

---

and comprehension. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 7871 1-7880.
the Association for Computational Linguistics: Hu- man Language Technologies, pages 1322-1336
Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. Stereoset: Measuring stereotypical bias in pre- trained language models. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume I: Long Papers), pages 5356-5371.
Chin-Yew Lin. 2004. Rouge: A package for automatic
evaluation of summaries. branches ouf, pages 74-81.
Text summarization
Yizhu Liu, Qi Jia, and Kenny Q. Zhu. 2021. Keyword- aware abstractive summarization by extracting set- level intermediate summaries. In WWW '21: The Web Conference 2021, Viruall Event Ljubljana, Slovenia, April 19-23, 2021, pages 3042-3054. ACM/ / IW3C2.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gulçehre, and Bing Xiang. 2016. Abstrac- tive text summarization using sequence-to-sequenec mns and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Lan- guage Learning, pages 280-290.
Yizhu Liu, Qi Jia, and Kenny Q Zhu. 2022a. Opin- ion summarization by superviinn from mix- structured data. In Proceedings of the 2022 Con- ference Empiricul Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 3086-3096. Association for Computational Linguistics.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1797-1807.
Yizhu Liu, Qi Jia, and Kenny Q Zhu. 2022b. Reference-fre free summarization evaluation via seman- tic correlation and compression ratio. In Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 2109-2115, Association for Computational Linguistics.
OpenAL. 2022. Online chatgpt: Optimizing language models for dialogue. OpenAl Blog.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstrac- tive summarization with frank: A benchmark for fac- tuality metrics. In Proceedings of the 2021 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, pages 4812-4829
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chaigpt as a factual inconsistency evaluator for abstractive text summarization. arXiv preprint arXiv:2303.15621
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic eval- uation of machine translation. In Proceedings of the 40th annual meeting of the Association for Compu- tational Linguistics. pages 311-318.
Kazuki Matsumaru, Sho Takase, and Naoaki Okazaki. 2020. Improving truthfulness of headline genera- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties. pages 1335-1346.
Leonardo Ranaldi, Elena Sofia Ruzzetti Davide Ven- ditti, Dario Onorati, and Fabio Massimo Zanzotto. 2023. A trip towards faimesss: Bias and de- biasing in large language models. arXiv preprint arXiv:2305.1 13862.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano. Alex Wang, and Patrick Gallinari. 2021. Questeval: Summariza- tion asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4594-6604,
Menggi Miao, Fandong Meng, Yijin Liu, Xiao-Hua Zhou, and Jie Zhou. 2021. Prevent the language model from being overconfident in neural machine translation. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference on Nat- ural Language Processing (Volume 1: Long Pupers), pages
Shuaijie She, Xiang Geng, Shujian Huang, and Jiajun Chen. 2023. Cop: Factual inconsistency detection by controlling the preference. In Proceedings of the AAAI Conference on Artificial Intelligence.
Anshuman Mishra, Dhruvesh Patel, Apama Vijayaku- mar, Xiang Lorraine Li, Pavan Kapanipathi, and Kar- tik Talamadupula. 2021. Looking beyond sentence- level natural language inference for question answer- ing and text summarization. In Proceedings of the 2021 Conference of the North American Chapter of
Seonil Simon Son, Junsoo Park. Jeong-in Hwang. Junghwa Lee, Hyungjong Noh and Yeonsoo Lee. 2022. Harim+: Evaluating summary quality with hallucination risk: Evaluating summary quality with hallucination risk. In Proceedings of the 2nd Confer- ence of the Asia- -sia-Pacific Chapter of the Association

---

for Computational Linguistics and the 12th Interna- tional Joint Conference on Natural Language Pro- cessing, pages 895-924.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019a. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang- and Tatsunori B Hashimoto. 2023. Alpaca: strong, replicable instruction-follovingg model. Stanford Center for Research on Foundation Models. https:/erffm stanford. cdu/2023/03/1//llppccc himl, 3(6):7.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang. Kathleen McKeown, and Tatsunori B Hashimoto. 2023. Benchmarking large language models for news summarization. arXiv preprint arXiv:2301.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019b. Paws: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguisties: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298- 1308.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Mane Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 2.Xiv:2302.13977
Prasetya Utama, Joshua Bambrick, Nafise Sadat Moosavi, and Tryma Gurevych. 2022. Falsesum: Generating document-level nli examples for recog- nizing factual inconsistency in summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguisties: Human Language Technologies, pages 2763-2776.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma. Avia Efrat, Ping Yu, Lili Yu, al. 2023. Lima: Less more for align- ment. arXiv preprint 230vv 1206.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the fac- tual consistency of summaries. In Proceedings of the Annual Meeting of the Association for Com- putational Linguistics, pages 5008-5020.
Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is chatgpt good nlg evaluator? a preliminary study, arXiv preprint arXiv:2303.0mm 04048.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoe Le. Denny Zhou, et al. 2023. Chain-of- tf-thughttt prompting elicits rea- soning in large language models. In Advances in Neural Information Processing Systems.
Yuexiang Xie. Fei Sun. Yang Deng, Yaliang Li, and Bolin Ding. 2021. Factual consistency evaluation for text summarization via counterfactual estimation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 100-110.
Wenpeng Yin. Dragomir Radev, and Caiming Xiong. 2021. Docnli: A large-scale dataset for document- level natural language inference. Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 4922.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text text gener- ation. Advances in Neural Information Processing Systems, 34:27263-27271.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In In- lernational Conference Machine Learning, pages 11328-11339, PMLR.

---

A Preliminaries
B Analysis Inconsistency Detection
CoP proposed by (She et al., 2023) is calculated based on A gond, Although they only showed A cond in their paper, they took the logarithm of probabil- ities during implementation without analysis and explanations. Overall, CoP
Ablation studies of FFLLM for inconsistency detec- tion are Table 9. Fig. illustrates the perfor- mances of FFLM on variable sizes of the founda- tion language model. Results with the prompting approach and comparison to the instruction-tuning technique are in Table 10.
CoP
log 228 log pref
CuClenSum
SummFival
m
12
69
HaRiM proposed by (Son et al., 2022) is derived
66
from Apy
83
Podylupe
FaaCC
XSumFaith
HaRiM 1 m
Pye [22)
(p
82
(9)
Our FFLM is different from HaRiM in three ways: First, we propose do faithfulness evaluation with foundation language models while they suggested to evaluate with models fine-tuned on summariza- tion data. Second, HaRiM only considers one of the changes with prior probability, i.e., prior Our metric not only adds a back constraint of PYX additionally to HaRiM, but also considers the prob- ability changes with conditional probability. Third, Specifically for the formula, the intuition of adding weights behind HaRiM and our FFLM are dif- ferent regardless of the function-form variations. HaRiM's weight (1 - p2) was originally intro- duced to adjust the loss scale for training better neural machine translation models by Miao et al. (2021). FFLM adopted the weight c/ and the logarithm to pay more attention to low- www-pobbbbility tokens which generally correspond to unfaithful contents according to (Goyal et al., 2022).
Figure Spearman correlation (%) of FFLM with dif- ferent model sizes on inconsistency detection datasets.
The observations for inconsistency detection are in line with those for faithfulness ranting. An- other finding is that the gap between prompting approaches and FFLM under this setup is much smaller than that under the faithfulness rating setup, This indicates that inconsistency detection easier than faithfulness rating with less number of an- swer choices when doing faithfulness evaluation by prompting large models.
C A Collection of Prompts
The prompts we collected are in Table 11. We made tiny changes to some of the prompts to en- hance their probability of generating an acceptable answer, such adding hint like "Marks:" and moving the answer choices the end of the prompt. We recognize the faithfulness score by analyzing the generations with simple rules. If there isn' 't an acceptable answer, we regard the summary as unfaithful, i.e., "No" for inconsistency detection and "1" for faithfulness rating.
Xie et al. (2021) introduced another similar metric CoCo which also based on probability changes. Instead of using the prior probability of Y in prior they condition Y on the masked doc- ument X' by removing Y -relatedd information. The masking strategy is hard to design since locating the Y -relateed information is uneasy. Their compli- cated operation also largely slows down the infer- ence speed.
D Performance of Different Prompts
We compared these three metrics backed on BART for faithfulness ranting in Table 8. Since CoCo's performances vary a lot with different masking strategies and none of them completely outperform the other two metrics on a single dataset, we didn't compare with it thoroughly in our work.
We show the performances of different prompts with Vicuna-7B for inconsistency detection in Ta- ble 12 and for faithfulness rating in Table 13. The performance with different prompts for the same task varies a lot. And some prompts used previ- ous works with ChatGPT failed with smaller mod- els, showing the high-level language understanding

---

FRANKCNN
QAGSCNN
SummEval
FRANKXSUM
QAGSXSUM
Metric
T
P
7
P
71
P
7
CaP
56.1
31.0
39.4
73.0
65.3
33.2
23.6
22.6
18.0
22.8
20.8
17.0
26.6
25.3
20.7
HaRiM
61.0
53.9
42.1
67.4
58.2
47.1
37.6
29.8
14.3
13.9
11.4
15.8
16.0
13.1
CoCorolaen
55.9
50.1
19.0
64.7
53.1
42.6
41.3
36.8
29.1
8.0
6.8
5.6
21.4
22.8
18.7
CoCoupan
CoCo
60.9
54.1
42.2
71.7
62.0
50.2
3944
35.0
27.6
16.3
15.8
12.9
16.5
14.9
12.2
Table 8: Comelltionns(9) of comparisons among CoP, HaRiM, and CoCo with BART for faithfulness rating. The
highest scores are in bold.
Metric
CoLkenSum
SummEval
FRANK
Polstope
FactOC
XSumPaith
Ablations on the metric components
A
$2.2
64.6
76.2
54,8
55.8
60.5
Frior
A
64.4
82.9
83.1
56.8
79.0
53.5
A Aerior Jelon
Ax
54.7
65.6
77.4
62.0
55.0
$8.9
70.5
83.5
84.4
56.8
77.3
59.8
Arrior
Lond
Ablations on the metric design
69.5
wil log
68.7
78.5
83.5
60.9
74.2
38.1
wlo and
65.3
80.9
83.1
64.1
75.2
56.6
Table 9: Ablations of FFLM for inconsistency detection. The highest scores in bold.
Metric
CuGinnSum
SummEval
FRANK
Polytope
FacCCC
XSumFaith
Prompting Approach
11 aMa-7h
54.3
50.0
$3.6
53.7
51.7
$1.7
7b
Alpaca-7b
57.8
50.0
57.5
52.6
58.8
51.1
Approach
LLaMa-7b
7L.
83.9
84.4
61.5
77.3
$8.9
Vieuna-Th
68.6
83.2
58.3
77.2
58.9
Alpaca 7b
65.2
83.9
59.1
78.5
60.7
Table 10: Comparisons with prompting and instruction instruction-tuning techniques under the same model size for inconsis- tency detection. The highest scores in bold in each column and underlined among each kind of approach.
and generation ability requirements for prompting large language models.

---

Ciation
Prompt
Incomsistency Delection
[Document)
Q: the following
inferred
Chen al. (2023)
the above document? Yes or No?
[Summary]
A:
the
supported the articl??
Article: (Document)
Goo et al. (2023)
Sentenoe: [Summary]
Answer
Decide if the tollowing summary is consistent with
comropunning article.
Luo al. (2023)
Article: (Document)
Summary: [Summary]
Answer (yes or no):
Faihifulness Raning
Evaluate the quality summaries written for articlo. Rate each summary consistency. (warst)
Gan et (2023)
(best).
Article: (Decument)
Summary: [Summary]
Marks:
Score the following summary given the corresponding article with respect to consistency from 1 to Note that consistency measures mach information included in the summary is points indicate
present the sounce
Luo et al. (2023)
summary contains only statements that are entailed by the source document.
Summary: [Summary]
Source Article: [Document) Marks:
Table 11:
collection of prompts."() marks place- holders for corresponding contents.
(2023)
(2023)
CatienSum
56.9
49.8
49.4
SummeEaal
FRANK
69.2
46.9
$7.8
Pulytope
54.6
$1.0
52.8
FacCCC
69.0
51.7
XSumFaith
52.2
49.6
88.5
Table 12: Balanced accuracy(%) of Vicuna-7B with dif- ferent prompts for inconsistency detection. The highest accuracy each dataset is in bold.
(2023)
(2023)
FRANKUNN
17.4
-5.0
QAGSCNN
SummEEal
11.9
-0.9
FRANKXSUM
5.5
-3.6
QAGSXSUM
8.4
3.3
Table 13: Spearman correlation(Te) of Vicuna-7B with different prompts for faithfulness rating. The highest correlation on each dataset is in bold.


Reviews:
[{"review_id": "fCQXwUEKRV", "reviewer": "Reviewer_TWe7", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We profoundly appreciate the hard work and the precious comments.\n\n**Response to \"the advances over the state-of-the-art are small\"**\n\nIn terms of method innovation:\n* The core of this work is to show the feasibility of doing the zero-shot faithfulness evaluation with a foundation language model (instead of a fine-tuned model or prompting methods). \n* Besides, we classify the probability changes into two categories and show that different probability changes are complementary to each other to some extent, which can achieve further improvements when considering them all together. \n\nIn terms of result improvements:\n* With our metric FFLM, a 7B-parameter language model can surprisingly outperform ChatGPT on most of the datasets and perform competitively on the others. \n* FFLM also shows strong generalization ability among different datasets (see Table 1), tasks (inconsistency detection in Table 2 and faithfulness rating in Table 3), and evaluation setups (summary-level in Table 3 and system-level in Table 4), compared with the baselines.\n\n* It should also be noted that the baselines FactCC and SummaC in Table 2 are weakly-supervised methods, while our FFLM is a zero-shot evaluation metric. FFLM can still outperform them on CoGenSum, SummEval, and Frank, while ChatGPT lags behind.\n\nBased on the above two points, we think our work makes considerable improvements over the state-of-the-art.\n\n\n**Response to \"Scores for ChatGPT-CoT on the SummaC benchmark\":** \n\nWe didn't include it because: Chen et al.(2023)[1] found that \"chain-of-thought prompting hurts performances dramatically compared to vanilla prompting in most cases\" as faithfulness evaluation is less reasoning-intensive. Thus, we only listed the results for ChatGPT without CoT in the paper. This has been mentioned in the footnote on Page 3.\n\n[1] Shiqi Chen, Siyang Gao, and Junxian He. 2023. Evaluating factual consistency of summaries with large language models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- the authors advance a recently introduced (and thus small) line of work: faithfulness evaluation with probability changes\n- the proposed FFLM model achieves SotA results on many datasets and is competitive with other approaches on the other evaluated datasets, even though it uses less parameters. Further, their method achieves consistent results across all datasets as opposed to other approaches (even on the harder XSum based datasets), demonstrating good generalization capabilities over dataset.\n- the experiments are extensive. Models are evaluated in two important faithfulness tasks \"Faithfulness Rating\" and \"Inconsistency Detection\". Further, all components of their FFLM approach (which consists of three measurements of probability changes and two token weighting schemes) are analyzed in ablation studies as well as error analyses.\n- the paper is well structured and clearly written, which makes it easy to understand", "reasons_to_reject": "- the advances over the state-of-the-art are small\n(- this work describes incremental work)", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work is on the faithfulness (factual consistency) evaluation of abstractive summaries. It is an important research direction to improve natural language generation models and reduce their hallucinations. In particular, this work considers the task of zero-shot faithfulness evaluation, in contrast to weakly-supervised methods.\n\nThe authors contribute a new evaluation metric FFLM, a zero-shot faithfulness evaluation based on foundation language models (they use Llama). In their experiments, they extensively show that their approach has favorable performance over state-of-the art models, especially ChatGPT-based approaches, even though their model is \"small\" (7B parameters) in comparison.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Scores of ChatGPT with Chain of Though reasoning are available for the SummaC benchmark, but not reported in this paper."}, "extra_text": {}}, {"review_id": "2tipdfLzRh", "reviewer": "Reviewer_cG7n", "summary": "", "questions": "Can you provide the main difference to HaRiM conceptually, though the formula is provided in Appendix (and it varies with the formula for the proposed method) from my understanding it seems exactly similar conceptually with an additional constraint.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much. We will explain the differences between our work and the baselines more clearly.\n\nFirst, we apologize that there is a typo in Eq. 9 in the Appendix and it should be:\n\n$HaRiM = \\frac{1}{m}\\sum_{i=1}^{m}(1-p_{y_i}^{s2s})[1-(p_{y_i}^{s2s} - p_{y_i}^{prior})]$\n\n\nOur metric FFLM is different from HaRiM in three ways:\n1. We propose to do faithfulness evaluation with foundation language models while they suggested to evaluate with models fine-tuned on summarization dataset. HaRiM_LLaMa is an improved metric with our proposal and performs quite differently from the original HaRiM as discussed in Line 380-391: The original HaRiM tends to prefer summaries generated by abstractive models, while HaRiM_LLaMa doesn't.\n\n2. HaRiM only considers one of the changes with prior probability, i.e., $\\Delta_{p_Y}^{prior}$.  Our metric not only adds a back constrained of P(X|Y) additionally to HaRiM, but also considers the probability changes with conditional probability as explained in Line 168-185. \n\n3. Specifically for the formula, the core of HaRiM and one of the three components in our FFLM are both $\\Delta_{p_Y}^{prior}$. Nevertheless, the intuition of adding weights behind HaRiM and our FFLM are different regardless of the function-form variations.\n    * HaRiM's weight $(1-p_{s2s})$ was originally introduced to adjust the loss scale for training better neural machine translation models by Miao et al.(2021)[1]. \n    * In our formula, we adopted the weight $e^{p_{y_i}^{s2s}}$ and the logarithm to pay more attention to low-probability tokens which generally correspond to unfaithful contents according to Goyal et al.(2022)[2].\n\n[1] Mengqi Miao, Fandong Meng, Yijin Liu, Xiao-Hua Zhou, and Jie Zhou. 2021. Prevent the language model from being overconfident in neural machine translation. ACL.\n\n[2] Tanya Goyal, Jiacheng Xu, Junyi Jessy Li, and Greg Durrett. 2022. Training dynamics for text summarization models. findings of ACL 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper does an excellent job in terms of comparing with a large number of models, on various datasets and correlating across multiple (currently adopted) metrics. This includes comparing with just the output of LLM without the proposed metric.  \n\nThe metrics show a strong correlation with human ratings pointing to the usefulness of the proposed methods. Further, the formulation is very intuitive, and all the 3 components of the weighted sum make sense intuitively (along with providing a somewhat naive interpretablity).", "reasons_to_reject": "The technique proposed is extremely similar to HaRiM, though HaRiM ends up with a more complex formula, the proposed metric only seems to add a back constraint of P(X|Y) additionally to HaRiM's method. Further note, HaRiM_LLaMa in fact outperforms the given metric on human correlation in Table 4.", "questions_to_authors": "Can you provide the main difference to HaRiM conceptually, though the formula is provided in Appendix (and it varies with the formula for the proposed method) from my understanding it seems exactly similar conceptually with an additional constraint.", "paper_topic_and_main_contributions": "This paper presents a metric for zero-shot faithfulness evaluation of summaries simply with a foundation language model. The work is very similar to (the same motivation) She et al. (2023) and Son et al. (2022) as cited by the authors with slight changes in the formulation of the actual proposed metric.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LhMHQ7pkzQ", "reviewer": "Reviewer_df1Z", "summary": "", "questions": "Q1: what is the correlation of these metric components in Table 5 experiments?\n\nQ2: Can we adjust the score or weighting parameters by removing the effect of different lengths in the generation prefixes of different component metrics?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your precious comments. We will follow your advice to improve the paper.\n\n**Response to Reject 1:**\n\nYes, it's possible to use smaller-sized models with some prerequisites:\n\nFirst, the model should be a foundation language model without being fine-tuned on task-specific datasets. The drawbacks of models trained on task-specific datasets are explained in Line 380-391. Flan-T5 was fine-tuned on Muffin corpus containing a large amount of summarization data, thus it's not recommended.\n\nSecond, this is highly related to research on investigating the optimal model size and dataset size for training foundation language models as mentioned in Sec. 4.5. \n\n\n\n**Response to Reject 2:**\n\nYes, the combination weights are tuned on the dev set if it exists. The weights are tuned on the dev set for the SummaC benchmark (Table 2), and set empirically for the faithfulness rating datasets (Table 3).\n\n**Response to Q1:**\n\nIn Table 5, we mainly show the ablation study of our metric FFLM, and all of the results are calculated between the faithfulness ratings by human annotators and automatic metrics.\n\nThe Pearson/Spearman/Kendall correlations(%) between the pairs of the metric components are as follows:\n\n| | FRANKCNN | QAGSCNN | SummEval | FRANKXSUM | QAGSXSUM |\n|  ----  | ----  |  ----  | ----  |  ----  | ----  |\n| $\\Delta_{Y}^{prior}$, $\\Delta_{X}^{prior}$ | 44.7 / 46.2 / 32.0 | 32.5 / 35.8 / 23.8 | 29.5 / 34.4 / 23.7 | 28.3 / 31.1 / 20.9 | 28.0 / 24.9 / 17.0 |\n| $\\Delta_{Y}^{prior}$, $\\Delta_{Y}^{cond}$ | 26.5 / 19.9 / 12.9 | 14.7 / 2.3 / 1.2 | 20.1 / 15.6 / 10.1 | 23.4 / 20.1 / 13.4 | -6.2 / -2.4 / -1.5 |\n| $\\Delta_{X}^{prior}$, $\\Delta_{Y}^{cond}$ | 47.2 / 49.9 / 34.6 | 35.2 / 41.2 / 28.0 | 36.5 / 40.6 / 27.9 | 38.7 / 38.3 / 25.9 | -0.4 / 3.8 / 2.5 |\n\nWe can see that the correlations vary among different datasets, and none of the pairs show a high degree of correlation. This also indicates that these components may capture unfaithfulness from different aspects.\n\nWe will add the results of the correlation of these metric components in the Appendix.\n\n\n\n**Response to Q2:**\n\nIf I understand you correctly, when you say \"the generation prefixes\", you mean the generated tokens before each step. For example, \n{$y_0, ..., y_{t-1}$} are the prefix tokens of predicting $y_{t}$ for LM(Y). If that is the case, in this work we simply take the average of them (see Eq. 6) and have achieved favorable results compared with the baselines. It should be noted that we calculate the probability of each token under the teacher-forcing strategy (Line 127-128), which won't lead to the exposure bias issue.\n\nIf what you mean is \"the length of generation contexts\", for example, X is the context of Y in calculating LM(Y|X). We didn't consider the effect of it in the current proposed metric, and will consider it in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The guideline of the metric is clearly motivated: independent of other tasks and dataset-specific expertise.\n\n2. This work contains clear explanation on the concepts and comparison with other work. For example, the authors clarify that extrinsic hallucination is unfaithful in this work, which is often ambiguous in other recent work.\n\n3. The comparison and ablation are nicely designed, for example, the comparison between prompting and instruction-tuning is much appreciated.\n", "reasons_to_reject": "1.\tFFLM is backed on LLaMa 7B. Is it possible to use smaller sized models such as Flan-T5 to conduct the same experiments?\n\n2.\tThe authors mentioned that the combination weights can be tuned, besides naïve 0.25 0.25 0.5 and all 1/3, mentioned in 4.3. Are there results based on tuning on dev set?\n", "questions_to_authors": "Q1: what is the correlation of these metric components in Table 5 experiments?\n\nQ2: Can we adjust the score or weighting parameters by removing the effect of different lengths in the generation prefixes of different component metrics?\n", "paper_topic_and_main_contributions": "This paper presents a novel evaluation metric FFLM to solve zero-shot faithfulness evaluation for summarization. FFLM is designed based on the probability output of foundation models. FFLM is a parametric combination of changes with prior probability and changes with conditional probability. Extensive experiments and ablation studies on various benchmarks compared with many recent methods show the effectiveness of FFLM, given a comparatively small backbone model and no fine-tuning required.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None as far as I know", "Typos_Grammar_Style_and_Presentation_Improvements": "None as far as I know"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The proposed technique to evaluate factuality in a zero-shot manner leveraging probability changes of a moderately sized model (LLama-7B). The reviewers all agree that the proposed method is extensively evaluated and would be useful for the community. The authors should clearly distinguish their approach from prior work (HaRiM) in future versions.
Decision: Accept-Main