Mitigating Intrinsic Named Entity-Related Hallucinations of Abstractive Text Summarization
Authors: Jianbin Shen
Junyu Xuan
Christy Jie Liang
Venue: EMNLP 2023 Findings (2023)
Keywords: Abstractive text summarization
named entity-related hallucinations
adaptive margin ranking loss

Abstract: Abstractive text summarization (ATS) is both important and challenging. Recent studies have shown that ATS still faces various forms of hallucination. Our study also indicates that a significant portion of hallucinations is named entity-related. They might appear in different forms, such as mistaken entities and erroneous entity references. The underlying causes implicit in data are complex: data samples pose varying learning conditions. Despite recent research efforts dedicated to named entity-related hallucinations, the solutions have not adequately addressed the varying learning conditions posed by data. This paper aims to bridge the gap in pursuit of reducing intrinsic named entity-related hallucinations. To do so, we propose an adaptive margin ranking loss to facilitate two entity-alignment learning methods to tackle them. Our experiment results show that our methods improve the used baseline model on automatic evaluation scores. The human evaluation also indicates that our methods jointly reduce the intrinsic named entity-related hallucinations considerably compared to the used baseline model.

Mitigating Intrinsic Named Entity-Related Hallucinations of Abstractive
Text Summarization
Jianbin Shen 2
Jie Liang?
Junyu Xuan
Australian Artificial Intelligence Institute (AAII), University of Technology Sydney, Australia Visualisationn Institute, University of Technology Sydney, Australia "jianbin. shen@student. uts. edt.ed "[jie.liang, xunyu.uuan xuan Jeuts. edu.au au
Abstract
Zhao et al., 2022), some hallucinations remain a challenge ATS. Named entity-related hallucina- tions (NERHs) are among them. Our study has also found that summarization of long text sequences (e.g., CNN MailyMail or CNNDM in short) incurs high occurrences of NERHs compared to other types of factual problems (e.g.. negation related). NERHs can be divided into intrinsic and extrinsic. The former includes the cases where named en- tity hallucinated in a summary is mentioned in the source document, while the latter occurs when a summary includes novel named entity not in the source document. We are interested in the intrinsic NERHs as the main source of NERHs. Among the intrinsic NERHs, entity-entity hallucinations often observed. For example, given the source document from the CNNDM test set below,
Abstractive text summarization (ATS) is both important and challenging. Recent studies have shown that ATS still faces various forms of hallucination. Our study also indicates that significant portion of hallucinations is named entity-related. They might appear differ- ent forms, such mistaken entities and erro- neous entity references. The underlying causes implicit data are complex: data samples pose varying learning conditions. Despite re- cent research efforts dedicated to named entity- related hallucinations, the solutions have not adequately addressed the varying learning con- ditions posed by data. This paper aims to bridge the gap in pursuit of reducing intrinsic named entity-related hallucinations. To do so, pro- pose an adaptive margin ranking loss to fa- cilitate entity-alignment learning methods to tackle them. Our experiment results show our methods improve the adopted baseline model on automatic evaluation scores. The hu- man evaluation also indicates that our methods jointly reduce the intrinsic named entity-related hallucinations considerably compared to the adopted baseline model.
Since civil war ..., 310,000 people have been killed, the Syrian Observatory for Human Rights said Thursday. ... es- timate by the U.N. of at least 220, 000 dead.
A pre-trained model fine-tuned with the CNNDM training data may generate the following summary.
1 Introduction
"U.N.: More than 310,000 people have been killed in Syria..."
Abstractive text summarization (ATS) has im- proved considerably in recent years, altributed the advancement of encoder-decoder ka.k.a. seq2seq) modeling (e.g.. Vaswani al., 2017). The language modeling using innovative pre-training methods (c.g., Lewis et al., 2020; Zhang et al., 2020) has further led to the significant perfor- mance gain on n-gram overlap-based metrics (e.g. ROUGEs). But recent studies (c.g., Ji et al., 2022; Maynez et al., 2020) have found generative sum- maries prone various forms of factual problems known hallucinations.
The summary mistakes the Syrian Observatory for Human Rights for U.N. We believe that the cause is rooted in the model's misaligning of entities with their contexts in terms of sentences.
The other form of intrinsic NERHs is entity- reference hallucinations. For example, given the following source document,
".. -wtth an eight-month-old baby, Savannah Guthrie ... help her precious baby girl Vale drift off. When ... mother- of-one discussed an Australian father's 's tip for getting his baby to sleep..."
Despite the research efforts leading to hallucina- tion reduction and factuality improvement (c.g., Lukasik et al., 2020; Sridhar and Visser, 2022;

---

The fine-tuned model generates the summary,
variant learning conditions tackle the hallu- cinations of interest respectively and holisti- cally.
"The ... mother-of-one discussed ... tips for getting his eight-month-ole daughter to sleep..."
We integrate our methods with a BART encoder-decoder. We also fine-tune the BART alone as a baseline for comparison analysis. We experiment with our methods using CN- NDM and XSum datasets. The experiment results show that our methods improve the adopted bascline model on automatic evalu- ation scores. Further human evaluation also indicates a noticeable reduction of the intrin- sic NERHs.
As seen, "her" is mistaken for "his" in the summary. We think that the cause could lie the reference similarity in the latent space of the model.
Researchers have proposed entity-aware ap- proaches to tackle intrinsic NERHs (e.g.. Xiao and Carenini, 2022; Bi et al., 2021). Meanwhile, the illustrated examples suggest that the models could fail learn from data distinguish positive candi- dates (e.g.. her) from negative ones (e.g.. his) the latent space. Margin ranking loss is well-known for tackling such problems. For example, Chen et al. (2021) incorporate a margin ranking loss to ad- dress extrinsic entity hallucinations. The classical margin ranking loss treats all samples identically, but various forms of intrinsic NERHs are implicit in data with complicated linguistic structures and might interplay at both lexical and contextual levels. Data samples thus pose varying learning conditions (easy or difficult) for a model. And yet it is diffi- cult to predefine and categorize them in modeling. It is desirable for methods adaptive to the variant learning conditions at fine granularity. Perhaps missing such adaptive capacity will have limited the efficacy of the prior works.
2 Related Work
Factual issues or hallucinations are challenging to ATS despite recent progress. Many methods have been proposed in recent years to tackle the chal- lenges. Typically, contrastive learning methods are adopted to address the factual issues caused by exposure bias sample imbalance (e.g., Cao and Wang, 2021; Wu et al., 2022). The recent progress summarization evaluation research also inspires new learning objectives (e.g., Wan and Bansal, 2022; Pernes et al., 2022) and question- answer-based approaches (e.g.. Gunasekara et al., 2021; Nan et al., 2021b). Additionally, hallucina- tions in generated summaries prompt post-editing correction methods such as Cao et al. (2020) and Balachandran et al. (2022).
To bridge the gap in tackling our hypothesized causes of the illustrated examples, we develop an adaptive margin ranking loss to facilitate two learning methods to jointly mitigate the intrinsic NERHs.
Meanwhile, realizing entity-related hallucina- tions as one of the main factual issues, researchers have also developed entity-award methods. One idea is to encode entity-aware representations for expressiveness (e.g., Bi et al., 2021). To im- prove named entity matches between the gener- ated summaries and the source documents, Xiao and Carenini (2022) propose a named entity span copy from source documents based on generative likelihood estimation with a global relevance clas- sification task. Nan et al. (2021a) introduce a summary-worthy entity classification on the en- tities occurring in both the source documents and reference summaries. To address named entity- relation hallucinations, Lyu et al. (2022) introduce entity- relation generators with entity-consistency and caation-consistent learning objectives. Ob- serving the source-summary entity aggregation phe- nomenon in which the named entities are replaced by more general scoped descriptions, González.
In summary, our contributions are:
1. We propose an adaptive margin ranking loss function. The function incorporates a named entity span-based cettance-rguularized inter- section over union (ESpan-DloU) metric to derive udaptive scaling variables of chosen base margin. The ESpan-DioU provides well-defined scalar space permissible for our learning purpose.
2. By utilizing the adaptive margin ranking loss, further propose an entity-sentence align- ment method to tackle entity-entity hallu- cinations and an entity-reference alignment method to mitigate entity-reference hallucina- tions. The methods automatically fit data of

---

Adapiive margin ranking loss module
Adaptive margin ranking loss module
(
[6] 8.)
Fitrty content representation
Fintity context representation
Iddapéive mangin
(Ay)
[Me, Rag D
101
(la R M1-1
Margis usllng vanable fencias
Eikg.
Agl
ESpan DioU
Intity sen gonnd trafs
Nepative samplar
Caniolqqutlll span cleaiffoziaa
BART encoder
BART docoder
Emhedding layer
Embedding layer
MLF
MLL
21
02
Dc-1
yu
Fariey
Fetily onstrst seprosctation
(a) Extended BART encoder-decoder.
(b) Adaptive margin ranking loss module.
Figure 1: (a) Architecture of BART encoder-decoder (blue-ish block) extended with an entity- sentence alignment method (green-ish block) and an entity-reference alignment method (carmine block). The two alignment methods are internally based on adaptive margin ranking loss module (milky block) and entity context representation modulc (purple-ish block). (b) Adaptive margin ranking loss module consists of two key submodules, the margin ranking loss with adaptive capacity (pinkish block) and the margin scaling variable submodule (sky-blue block). MLPs are used to reduce dimensionality.
et al. (2022) introduce aggregation-awarene -wareness to ensure the aggregations are factually aligned with the aggregated entities. Introducing guiding sig- nals, Zhang et al. (2022) propose an entity cover- age control method by prepending an entity cover- age precision value to the encoder's inputs while Narayan et al. (2021) chain named entities from the reference summary and prepend them to the de- coding inputs. Post-editing approaches (e.g., Chen et al., 2021; Lee et al., 2022) are also developed to reduce entity hallucination as part of strategies for improving broader factual consistency.
entity-sentence alignment method (E-Sent AM), and the entity-reference alignment method (E-Ref AM). Both alignment methods utilize adaptive mar- gin ranking loss that has the same modular structure as shown in Fig. lb except that the E-Sent AM uses sumilarity-scorinn function while the E-Ref AM adopts antisymmetric-scont methods also share entity context representation module. We first discuss the generic aspects of the architecture before proceeding to the specifics of our two alignment methods.
function. Two
3.1 Incoder-Decoder Generative Model
Methods discussed so far have not adequately considered adaptiveness to the variant learning con- ditions at fine granularity. In developing our adaptive methods, we desire real-valued margin- scaling variable function to provide the resulting margins with three propertics. That is, they are proportional to the (easy or difficult) learning con- ditions implicit in samples; they are real values within a well-defined scalar space permissible for our learning purpose; and the base margin is pre- served at the low bound of the scalar space.
An encoder takes an input sequence of tokens (I1, ( .2.,.,, 2z) and outputs a latent sequence Z = (21, 22, ... 21)- decoder then auto-regresses its inputs } 1== of m tokens with the Z latent states to generate the Z-attentive outputs (ύn, $2.... ... Fm). The learning objective is maximum likelihood esti- mation (MLE) measured by cross-entropy loss:
Egen
log(U.)
(1)
where ÿi and y/ are the 8th word prediction and the corresponding ground truth respectively.
3 Our Methods
Our I are illustrated Fig. 1. The architec- ture shown Fig. la, consists of the backbone BART encoder-decoder (Lewis et al., 2020), the
3.2 Adaptive Margin Ranking Loss
Margin ranking (or triplet ranking) loss, first pro- posed by Schroff et al. (2015), can be defined as:
As our methods focus solely on named entities, we simply mcan "named entity" whenever we use "entity" from now on.
Emr = max(0, sc - sc2 + m),
(2)

---

where scj and sck are positive and negative sam- ple scores concerning an anchor sample of the kth sample triplet (i.e. anchor, positive and negative samples), and m is a margin hyperparameter. The function aims to distance negative samples from positive ones concerning an anchor. But the scalar margin assumes that samples pose uniform learn- ing condition. The assumption is hardly suitable for many complex problems including ours. This gives rise to our adaptive margin ranking loss defined as:
context representation. Using the contextual sen- tences in Eq. 4 and each sentence's bounding token index pair in Eq. 5, we can gather the token latent states of the contextual sentences from the encoder:
ba (lbg...... h (
(7)
is the token representation quence of the ttth contextual sentence. To learn the context representation of the sequence, we im- plement multi-filter multi-kerna multi-scale con- volutional neural network (M3-CNN) described in Appendix Using the model, we get the resulting representation as follows:
may) where (hag,..., ...
mmr = max (0, 5 80g + ) m),
where 0 is a scaling variable for the hdhh triplet. We now detail its upplications for our two alignment methods
hC: M3-CNN ()).
3.3 Entity-Sentence Alignment Method
Negative Sampler We construct negative sample pairs by randomly drawing the samples from the positive samples of the other entities:
An entity has its contextual sentences where the entity occurs and/or is referenced in document. We express the contextual sentences of an entity as an entity cluster as follows:
(er, (yy))) ((())
(9)
(² CC, i' # i ind jr E [S]-
(es, (sy)) (C0)), (+())= C and j c [S), (4)
We get the sample triplets from Eq. 6 and Eq. 9:
where is the entity i, (i)( is the number of contextual sentences in the entity cluster (*(i), C*
{ee sty $ (c
(10)
is entity cluster set, {sly (ss))==1))) [0)(1) is the ordered set of the contextual sentences, j is the sentence num- ber in the document, and |S] is the total number of sentences in the document. The contextual sen- tences may not be adjacent. Each sentence sij is further expressed by its bounding (i.e. beginning and ending) token index pair in the document as:
Sample Representations For an entity, we may simply use its key token latent state from the en- coder as its representation. For sentence samples, an M3-CNN to learn their representations. The resulting representations of the triplets then be expressed as:
K
(hep h Moto 8,y k=
(11)
Sig = (g)
(5)
where her Moj and h "yy and negative sentence representations respectively. With sample data ready, we can now follow Fig. 1b to instantiate adaptive margin ranking loss. We start with a measure adaptive learning conditions.
are the entity, positive
We construct the entity-sentence positive sam- ples by transforming the entity and sentences into one-to-one pairs as follows:
(es, (e00)) > {le,, st) HC00
(6)
Context-Guided Entity Span Classification Kryscinski et al. (2020), in their FactCCX method, develop a claim (text) span classification to identify supports and mistakes predicted by their evaluation model. Such text span classification may be re- garded test method for Icarning conditions. That is, easy learning conditions are implied if a model converges the classification well. It dif- ficult to learn otherwise. We extend the idea by integrating context guidance. To do so, we concate- nate the entity, positive sentence representations
The superseription + denotes positive samples.
Entity Context Representation Different enti- ties can have their contexts sharing subset of sentences. This presents ambiguous learning condi- tions to ATS models. Instead mutually excluding such sharing to train our model, we take an entity context-guided learning approach discussed later to take into account the impact. We first learn the
Appendix A summarizes key notations used in the meth-
ads.

---

from Eq. 11 and the context representation (Eq. 8),
To preserve base margin at the low bound, we transform the loss be bound by [1.0,3.0]. So, the scaling variable function for the sample j of the entity is defined as:
followed by clamped linear transformation get the entity span prediction logits as follows:
logits.., min(max(O, W Wikee Muy Megl), Imaz);
(12)
Aa 1.0 + LDIOU,
(16)
where W is the learning parameters, [ is the con- catenation operator, lma is the maximum docu- ment length that the encoder-decoder permits, and logitse., E 2x is the predictions of bounding token index pair of the entity.
Scoring Function The aim is to measure the rel- evance of the sentences to the entity. We consider this similarity measure using the cosine func- tion. A positive sample score is thus defined as:
he, h
(17)
Entity Span Distance-loU We now need met- ric on the entity span classification to give rise to scaling variable function that satisfies the afore- mentioned adaptive margin properties. Intersection over union (loU) well-known metric for ob- ject detection in computer vision. It measures the overlapping ratio between a predicted object and the ground truth in terms of bounding boxes. It is ocale-invariant and bound to [0.0, 1.0]. Entity span in ID shares similar overlapping charac- teristic to the bounding box in 2D. We adopt Distance-loU by Zheng al. (2020) for fast learn- ing convergence, and derive an entity span-based based Distance-IoU (ESpan-DIoU) as follows.
The
1h
With the same formula, a negative sample score sery can be acquired with the negative sample rep- resentations h h_ instead. "yy
Entity-Sentence Alignment Learning Objective With the obtained {(sct, soges Ao)) \ learning objective is computed as:
(C*(0)
the
Lee-am
[0)(
C*((i) m0 sc.j se, + Aij m).
(18)
Let the predicted logitse, egg for the entity be split into index pair (b), b,). With the ground truth (b), d)), an entity span IoU is defined as:
3.4 Entity-Reference Alignment Method
This alignment method follows the same process as section 3.3 but with different data and scoring func- tion. We focus our discussion on the difference. Positive samples here consist of the annotated coreferences for each entity, that is, entity refer- ent and its references. We also express the entity references as an entity cluster as follows:
fn min C2 of) - max(0],00 U!), fu (b)² - -8!) + (66- b!) fn +,, (13)
fn
IoU; fo'
where In and fu are the intersection and union of the two text spans respectively, and is a small residue to avert divide-by-zero. A DIoU is then derived as:
(Ci 1) C0 j (CT(1) C E C
(19)
where e is an entity i, rij is a reference j (e.g., a pronoun), (CT(t) is the number of references in the entity cluster (i)),, and C" is entity cluster set.
fo = max (b,) - min( (8),00 8!), fp ((b) + b7) (6) 22)))/4 /4, (14)
Negative Sampler We construct negative sample pairs by randomly drawing the samples from the positive reference samples of the other entities:
DIoU; = loU, - (D + +))
where fa is the smallest width enclosing the two spans, fo is the squared distance of the two span centers, and f is the squared f.-
(( I' E (CT(()) C²(1) e and i' +i.
(20)
Margin Scaling Variable Function We first con- vert the ESpan-DIoU to a loss bound by [0.0, 2.0]:
The sample triplets are then given:
(C)
LDIOU = 1.0 -DIoU,
(15)
((a, r rey)) k=1
(21)

---

Sample Representations As with the approach taken in section 3.3, we may use the key token latent states from the encoder for entity referent and reference representations respectively. So, the representations of the triples are expressed as:
CNNDM Test Set
Model
R-1
R-2
R-L
DA-SSan/BertSumExtAn
41.75
19.27
38.81
(Dong et al., 2020)
ERPGN/BART Base (Lyu al., 2022)
42.28
19,64
38.93
FaciPEG/SUS/Zero-
40.98
18.97
28.90
shoty/BART-hase (Wan and Bansal, 2022)
(C) k=
(he Priy' b
(22)
BART-base
42.81
19.52
29.36
Context-Guided Entity Span Classification
BART-base E-Ref AM
43.10
19.82
29.62
The classification can share the same context repre-
base- E-Sent AM
42.88
19.56
29.35
BAKT-base Dual AMs
42.81
19.50
29.36
sentation guide as in section 3.3 but concatenates it with the entity and positive reference representation pairs from Eq. 22 follows:
Table I: ROUGE Evaluation (with CNNDM). 1. The number test samples from our annotation prepro- cessing is 1 11483 (out of 11490 samples). and Bansal (2022) does not have results on CNNDM. We use their published source code (https: //g/ttttttttttt thub. com/meetdan dwriiwwan/fff Sawlavidwww/aaappggaaae factpegasss)) to train and test model using their settings except that the maximum source length and larget length are changed to 1024 and 142 respectively. 3. Entity- refereere alignment method. 4. Entity-sentence alignment method. 5. Dual AMs consists of both entity-reference and entity-sentence alignment methods.
Wan
lo wimmmimm((())) W Whehe Myt. N. Mo,1), bmax),
(23)
where W is the learning parameters.
Scoring Function An entity referent and its ref- erences are functionally different from a linguistic perspective. So, in general, they are not seman- tically symmetric. We expect scoring function to be able to score the antisymmetric relations en- coded in their representations. To realize this, design the function in complex embedding space as in Trouillon et al. (2016) but with a more simple formula as follows:
Experiment and Analysis
4.1 Dataset
We experiment with CNN DailyMail (CNNDM) (See et al., 2017; Hermann et al., 2015) and XSum (Narayan et al., 2018). Both datasets consist of news article and reference summary pairs. XSum's summaries more abstractive than CNNDM's in that XSum has one-sentence alike summaries while CNNDM tends to have multi-sentence summaries. We use Stanford CoreNLP software" to prepare our datasets (Appendix D.I).
sc Re(E,EE
(24)
where Es the referent representation matrix, and E, is the complex conjugate of the reference repre- sentations E. matrix form. As the dot product of complex vector matrices may not always result in real values, the score se is taken by the real part (expressed by Re operator) for the real balue-based margin ranking loss. One can easily prove that the function possesses the antisymmetric property.
4.2 Implementation
Our methods adopt pre-trained BART-base encoder-decoder". It has 6 layers, 12 attention heads, a filter size of 3072, and hidden state dimen- sions of 768. The key implementation detailed in Appendix D.2. Our source code is also accessible
Entity-Reference Alignment Learning Objec- (Cr()) tive With the obtained {(sclj sCay' Azy) [==1
the learning objective is computed as follows:
Ler-am
(C)
4.3 Results and Analysis
max (0, scl
sGgg + Aig m).
(r(i))
We first evaluate generated summaries using (25) ROUGE metrics (Lin, 2004), followed by recent- developed factuality consistency evaluation met- rics, SummaC (Laban et al., 2022). We then con-
3.5
Total Learning Objective
The total learning objective consists of the three learning objectives Eqq. 1, Eq. 18 and Eq. 25 as:
'nttos://stanforddll .it.ghttt.bb....... io/CoreNLP.
"nttps://hugginggaac watpa/cc..o/fceeeeeeeeeeeeeeeeee to/ffcebook/batt b-ase. "nttps://clousston aawnet. .. edu. wttpaaa/pps/e. BUhKQEoHTULankrr.
L = fyen + Les-am + Ler-am-
(26)

---

XSum Test Set!
SummaCrs
Summa C Conv
Model
R-1
R-2
R-L
Model
x1%)
((%)
Ha-Span/BertSumExtAnn (Dong et al., 2020)
36.86
14.82
29.70
BART-base
68.3
0.255
62.5
0.230
E-Ref AM
71.6
0.222
65.0
0.206
ERPGN/BART BAGN/BBRTTBassS Base (Lyu et al., 2022)
39.60
16.90
31.74
E-Sent AM
68.4
0.239
64.5
0.201
Dual AMs
71.8
0.231
66.8
0.200
FaciPEGASUSIZero
32.97
11.42
25.41
Reference 48.5
0.243
45.6
0.188
shot//BART-base (Wan and
Bansal, 2022)
Table 3: SummaC score distribution statistics over the
BART-base
41.80
18.99
33,90
100 randomly sampled generated summaries from the
BART-base + E-Ref AM
41.74
18.84
33.68
CNNDM test set.
BART-base + E-Sent AM'
41.99
19.06
34,00
BART-buse + Dual AMs*
41.87
18.91
33.77
Model
SummaCzx
Summa
Conv
f(%)
[(X]
Table 2: ROUGE Evaluation (with XSum). 1. The num- ber of test samples from our annotation preprocessing 11328 (out of 1134 samples). 2. Entity-reference align- ment method. 3. Dual AMs consists both entity-reference and entity- sentence alignment methods.
BART-base
I0.2
0.190
23.8
0.036
E-Ref AM
8.9
0.134
23.5
0.026
E-Sent AM
7.0
0.123
23.5
0.029
alignment method. 4.
Dual AMs
9.9
0.185
23.9
0.035
Reference
6.4
0.108
23.3
0.028
Table 4: SummaC score distribution statistics over the 100 randomly sampled generated summaries from the XSum test set.
duct our human evaluation of named entity-related hallucinations (NERHs) and commonly observed syntactic agreement errors. We also extend our evaluations as detailed in Appendix E.1.
consistency as follows.
Automatic Evaluation of Factuality Consistency We use the SummaC for the evaluation. It has two metric flavors, SummaCzs and SummaCcouv. SummaCzs is highly sensitive to extrema pointed out by the authors. SummaCçuuv mitigates the sensitivity by transforming entailment probabil- ities into a histogram to learn a score function.
ROUGE Evaluation Table 1 shows the ROUGE scores evaluated with the CNNDM test set. Sepa- rated by double lines, the top section of the table lists several recent ATS factuality research using the same or similar-sized backbone models, fol- lowed by experiment section containing the high ROUGE scores. Our experiments have evalu- ated the fully configured dual alignment methods alignment
We randomly sample 100 generated summaries, then use the metrics to score each summary con- cerning its source document, followed by comput- ing their score distributions (i.e. mean and standard deviation statisties) shown in Table and Table 4 for both CNNDM and XSum respectively
and two ablations - the method and the entity-sentence alignment method. We have also fine-tuned the baseline BART-base for comparison. Similarly, Table 2 shows the ROUGE scores evaluated with the XSum test set.
As seen, the Dual AMs method scores higher mean values than the alignment ablations with both CNNDM and XSum even though the Dual AMs may have had lower ROUGE scores respectively. Compared the baseline BART-base, our meth- ods achieve better scores with CNNDM. But, with XSum, the bascline has an advantage on SummaCzs scores while the Dual AMs edges ahead on the SummaCCon- Along with the stan- dard deviations, the results for XSum suggest that the baseline produces some summaries with higher probabilities while the Dual AMS generates more summaries in high entailment probability bins.
Compared to prior work, the models trained on our annotated datasets have outperformed mostly R-1 and R-2 metrics on CNNDM while bettering across all metrics on XSum.
Among our experiments, we see that the E-Ref AM produces better scores with CNNDM while the F-Sent AM has the edge with XSum. Would the results also suggest that the alignment methods individually outperform the combined Dual AMs in reducing intrinsic NERHs and improving the overall factuality? Recent ATS factuality evalua- tion studies (e.g., Kryscinski al., 2020; Maynez et al., 2020) have found that n-gram overlap-based metrics (e.g., ROUGEs) are not sufficient for factu- ality assessment. To answer the above question, we first conduct an automatic evaluation of factuality
We also score the reference summaries for anal- ysis. We see that the reference summaries have the lowest scores for both datasets. This could indicate
'Appendix E.2 provides statistical significance assessment.

---

Error Type
Model
hallucinations, covering event, event time, location. number, modifier, and negation. Misspelling errors are also included. Appendix F details the defining rules for counting the hallucinations and errors.
Our
Dual
E-Ref
E-Sent
Baseline
AMs
AM
AM
Entity intrinsic
17
10
Entity extrinsic
9
6
Subtotal
25
14
19
17
The same 100 sampled summaries evaluated by SummaC are used. The erroncous occurrences are shown in Table 5 and Table 6 for both CNNDM and XSum respectively. Separated by double lines, each table contains the statistics of the NERHs, the syntactic agreement issues and misspellings, followed by the sums of all counts.
Modilier
2
0
Event
3
2
4
2
Event-time
2
5
Location
0
0
Negation
0
Number
Misspelling
Subioial
17
17
17
Total
36
37
36
24
As shown, our alignment methods consistently reduce the intrinsic NERHs compared the base- line for both CNNDM and XSum: the Dual AMs method reduces them considerably. Noticed that the models trained with XSum result in much fewer extrinsic entity hallucinations than those trained with CNNDM. We think that the one-sentence con- ciseness of XSum summarization might have lim- ited exposure to the extrinsic entity hallucinations. Meanwhile, Table 5 shows that the Dual AMs and E-Ref AM methods introduce more syntactic agreement and misspelling errors than the base- line while the E-Sent AM method can reduce them. The Dual AMs method has more errors on loca- tion, negation, numbers, and misspellings while the baseline results in more errors on event and event time. In Table 6, the Dual AMs method sults in the least syntactic agreement errors while the baseline has much higher error counts on events. This error-type divergence might occur because the models attend to the respective features more often. Additionally, the models trained with XSum in- cur much more syntactic agreement errors than those trained with CNNDM. It is worth noting that the sampled summaries from all models have few other factual errors outside our categories. So, we believe that the abstractive and extrinsic nature of XSum summaries could have contributed to more errors in events, event time, locations, and num- bers.
Table 5: Human evaluation of factuality on the 100 randomly sampled generated summaries (CNNDM).
Model
Error Type
Our Baseline
Dual AMs
E-Rel
E-Sent
AM
AM
Entity intrinsic
23
12
16
14
Entity extrinsic
0
Subtotal
24
13
18
14
Modifier
12
15
Event
19
9
8
7
Event-time
9
8
6
8
Location
10
10
11
13
Negation
2
Number
13
12
18
13
Misspelling
Subtotal
56
42
56
57
Total
80
55
74
71
Table 6: Human evaluation of factuality on the 100
randomly sampled generated summaries (XSum).
that the reference summaries contain significant extrinsic knowledge, The entailment model trained mainly on intrinsic data might not them well. It is noticeable that the scores for CNNDM are much higher than those for XSum. As studied by Lu et al. (2020), XSum reference summaries have 35.76% and 83.45% novel unigrams and bigrams respectively compared CNNDM's 17.00% and 53.91% The semantic disparity of XSum between source documents and reference summaries might expose the data imbalance in training SummaC for evaluating summaries that tend to be extrinsic. This rationale agrees with the reasoning behind the low reference summary scores.
5 Conclusion
This paper proposes an adaptive margin ranking loss function. We utilize the loss function to facil- itate two entity alignment methods to mitigate in- trinsic named entity-related hallucinations of ATS. Our experiment results and analysis show that our methods improve the adopted baseline model on automatic evaluation scores and reduce the intrinsic named entity-related hallucinations.
We further assess various aspects of factuality issues that may give rise to automatic metric scores. We evaluate how well our methods reduce the NERHs of interest. We also assess a range of commonly observed syntac- tic agreement errors that are often the causes of
Human Evaluation

---

Limitations
in Abstractive Summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6633-6649, Online and Punta Cana, Domimican Republic. Association for Computational Linguistics.
Our research in this paper is confined to news ar- ticle documents. Given the data feature used, our methods are limited to exploring the mitigation of intrinsic named entity-related hallucinations. We have properly chosen the syntactic categories for our human evaluation, and have assessed the gener- ated summaries with due diligence and as objective as possible, but assessment inaccuracy can still oc- cur, considered as low-bound by Bayes error. We hope that the combination of ROUGE scores, Sum- maC metrics, and human evaluation in the paper has strengthened our overall analysis.
Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. 2021. Improving Faithfulness in Abstractive Sum- marization with Contrast Candidate Generation and Selection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5935-5941, Online. Association for Computational Linguistics.
Yen-Chun Chen and Mohit Bansal. 2018. Fast Abstrac- tive Summarization with Seinforce-Selectee Sen- tence Rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675- 686, Melbourne, Australia. Association for Compu- tational Linguistics.
Ethics Statement
To the best of our knowledge, have attributed work to the previous research and implemen- tations on which this paper depends, either in the main script or in the appendices. Any changes to third-party implementations follow their license permits. Our work and experiment results are all genuine.
Yue Dong. Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, and Jingjing Liu. 2020. Multi-Fact Correction in Abstractive Text Summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9320-9331, Online. Association for Computa- tional Linguisties.
Acknowledgements
Adam Ek and Jean-Philippe Bernardy. 2020. Com- posing Byte-Pair Encodings for Morphological Se- quence Classification. In Proceedings of the Fourth Workshop on Universal Dependencies (UDW 2020). pages 76 86, Barcelona, Spain (Online). Association for Computational Linguistics.
This work was supported by the Australian Re- search Council through Discovery Early Career Re- searcher Award DE200100245 and Linkage Project LP210301046.
José Ángel González, Annie Louis, and Jackie Chi Kit Cheung. 2022. Source-summary Entity Aggregation in Abstractive Summarization. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6019-6034, Gyeongju, Republic of Korea, International Committee on Computational Linguistics.
References
Vidhisha Balachandran, Hannaneh Hajishirzi. William Cohen, and Yulia Tsvetkov. 2022. Correcting Di- verse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9818-9830, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Chulaka Gunasekara, Guy Feigenblat, Benjamin Sz- najder. Ranit Aharonov, and Sachindra Joshi. 2021. Using Question Answering Rewards to Improve Abstractive Summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 518-526, Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Qiwei Bi, Haoyuan Li, Kun Lu, and Hanfang Yang. 2021. Augmented Abstractive Summarization with Document-Leww Semantic Graph. Journal Data Science, 19(3)4550-44. Publisher: School of Statis- tics, Remmin University of China.
Benjamin Heinzerling and Michael Strube. 2019. Se- quence Tagging with Contextual and Non-Contextual Subword Representations: A Multilingual Evalua- tion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 273-291 Florence, Italy, Association for Com- putational Linguistics.
Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020. Factual Error Correction for Abstrac- tive Summarization Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6251-6258, Online. Association for Computational Linguistics.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suley- man, and Phil Blunsom. 2015. Teaching Machines
Shuyang Cao and Lu Wang. 2021. CLIFF: Contrastive Learning for Improving Faithfulness and Factuality

---

Read and Comprehend. In Advances in Neural Information Processing Systems, volume 28, Curran Associates, Inc.
Michal Lukasik, Himanshu Jain, Aditya Menon, Se- ungyeon Kim, Srinadh Bhojanapalli, Felix Yu, and Sanjiv Kumar. 2020. Semantic Label Smoothing for Sequence to Sequence Problems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1992-4998, Online. Association for Computational Linguistics.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022. Survey of Hal- lucination in Natural Language Generation. ACM Computing Surveys. Just Accepted.
Yuanjic Lyu, Chen Zhu, Tong Xu, Zikai Yin, and Enhong Chen. 2022. Faithful Abstrac- tive Summarization via Fact-aware Consistency- constrained Transformer. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM '22, pages 1410- 1419, New York, NY, USA. Association for Comput- ing Machinery.
Yoon Kim. 2014. Convolutional Neural Networks for Sentence Classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751, Doha, Qatar. Association for Computational Linguis- tics.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the Factual Consistency of Abstractive Text Summarization. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346, Online. Association for Computa- tional Linguisties.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On Faithfulness and Factu- ality in Abstractive Summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, On- line. Association for Computational Linguistics.
Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. 2021a. Entity- level Factual Consistency of Abstractive Text Sum- marization. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2727-2733, Online. Association for Computational Linguistics.
Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. SummaC: Re- Visiting NLI-based Models for Inconsistency Detection in Summarization. Transactions of the Association for Computational Linguistics, 10:163-177.
Hwanhee Lee, Cheoneum Park, Seunghyum Yoon. Trung Bui, Franck Demonnourt, Juae Kim, and Kyomin Jung. 2022. Factual Error Correction for Abstractive Summaries Using Entity Retrieval. In Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metries (GEM), pages 439 444, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Feng Nan, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng, Kathleen McKeown, Ramesh Nallapati, Dejiao Zhang, Zhiguo Wang, Andrew O. Amold, and Bing Xiang. 202 Improving Factual Consistency of Abstractive Summarization via Question Answer- ing. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume Long Papers), pages 6881-6894, Online. Association for Computa- tional Linguisties.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zeulemoyer. 2020. BART: Denoising seguencc-to-Sqquene Pre- training for Natural Language Generation. Trans- lation, and Comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 7871-7880, On- line. Association for Computational Linguistics.
Shashi Narayan. Shay B. Cohen, and Mirella Lapata. 2018. Don't Give Me the Details, Just the Sum- mary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. In Proceedings of the 2018 Conference on Empirical Methods Natural
Language Processing, pages 1797-1800 Brussels. Belgium. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Asso- ciation for Computational Linguistics.
Shashi Narayan, Yao Zhao, Joshua Maynez, Gonçalo Simões, Vitaly Nikolaev, and Ryan McDonald. 2021. Planning with Learned Entity Prompts for Abstrac- tive Summarization. Transactions of the Association for Computational Linguistics, 9: 9:1475-144 1492. Place: Cambridge, MA Publisher: MIT Press.
Yao Lu, Yue Dong, and Laurent Charlin. 2020. Multi- XScience: A Large-scale Dataset for Extreme Multi- document Summarization of Scientific Articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural I Language Processing (EMNLP), pages 8068 8074, Online, Association for Computa- tional Linguistics.
Diogo Pernes, Afonso Mendes, and André F. T. Mar- tins. 2022. Improving abstractive summarization with energy-based re-ranking. In Proceedings of

---

the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 1-17, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Wen Xiao and Giuseppe Carenini. 2022. Entity-based SpanCopy for Abstractive Summarization to Improve the Factual Consistency. MXXiv:220.033479 [es] ver- sion: 1.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. FaceNet: A unified embedding for face recognition and clustering. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 815- 823. ISSN: 1063- 6919.
Jiaxuan You, Zhitao Ying, and Jure Leskovec. 2020. De- sign Space for Graph Neural Networks. In Advances in Neural Information Processing Systems, vol- ume 33, pages 17009-17021. Curran Associates, Inc.
Haopeng Zhang, Semih Yavuz, Wojciech Kryscinski. Kazuma Hashimoto, and Yingbo Zhou. 2022. Im- proving the Faithfulness of Abstractive Summariza- tion via Entity Coverage Control. Findings of the Association for Computational Linguisties: NAACL 2022, pages 528-535, Seattle, United States. Associ- ation Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Man- ning. 2017. Get To The Point: Summarization with Pointer-Generator Networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 1073-1083, Vancouver, Canada. As- sociation for Computational Linguistics.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. PEGASUS: Pre-training with Extracted Sap-sentences for Abstractive Summarization. In Proceedings the 37th International Conference on Machine Learning, pages 11328-1 1339. PMLR.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 1715-1725, Berlin, Germany. Associ- ation for Computational Linguistics.
ISSN: 2640-3498.
Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J. Liu. 2022. Calibrating Sequence likelihood Improves Condi- tional Language Generation.
Arvind Krishna Sridhar and Erik Visser. 2022. Im- proved Beam Search for Hallucination Mitigation in Abstractive Summarization. ArXiv:2212.0277 12 [cs] version: 1.
Zhaohui Zheng, Ping Wang, Wei Liu, Jinze Li, Rong- guang Ye, and Dongwei Ren. 2020. Distance- loU Loss: Faster and Better Learning for Bound- ing Box Regression. Proceedings of the AAAI Conference on Artificial Intelligence, 14(0)):1 13000. Number: 07.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Eric Gaussier, and Guillaume Bouchard. 2016. Com- plex Embeddings for Simple Link Prediction. In Proceedings of The 33rd International Conference on Machine Learning, pages 2071-2080. PMLR. ISSN: 1938-7228.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
David Wan and Mohit Bansal. 2022. FacdPEGASUS: Factuality-Awart Pre-training and Fine-tuning for Abstractive Summarization. In Proceedings of the 2022 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1010-1028, Seattle, United States. Association Computational Linguistics.
Wenhao Wu, Wei Li, Jiachen Liu, Xinyan Xiao, Ziqiang Cao, Sujian Li, and Hua Wu. 2022. FRSUM: To- wards Faithful Abstractive Summarization via En- hancing Factual Robustness. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 3640-3654, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

---

Notation
Description
tively short sequences while still capturing salient features well. They have been used encode a representation of short text sequences (e.g. Kim, 2014; Chen and Bansal, 2018) with a single-scale max-over-time pooling. We adopt a convolutional neural network (CNN) for encoding a representa- tion of multi-sentential entity context. We think the choice of CNN is sensible because the inputs to the CNN the latent states from the BART encoder- decoder. The lengths of our multi-sentential con- texts are somewhat in-between full-length docu- ments and the short-text sequences seen in the aforementioned prior work. So, we extend single- scale pooling to multi-scale pooling by borrowing the idea from deep learning in computer vision. Given a padded text sequence z to the pre-defined max length and down wown-ssmpling scale factor, we can compute the number of down-sampling scales, We can also compute input and output size along the sequence length dimension at each scale. So, each convolutional block consists 1D convo- lution, a ReL.U, and an adaptive ID max-pool. A fully-connected linear layer is applied to the output of the last dlwn-ssapling scale.
The total number of items samples. An ordered set of items or samples.
{+}
An ordered set indexed by an indexing vari- able up the total number
pair/triplet notation.
sequence notation.
6d
dimensional concatenation operator latent representations.
(4
sentence bounding (i.e. beginning and ending) token indices concerning docu- ment.
An entity i.
Bi (or 8:3)
sentence j in a document where an entity i appears and/or is referenced (as a positive sentence sample).
A negative sentence sample related to an entity i, that is, a sentence j' in a document where an entity does not appear or is ref- erenced.
rij (or rz))
A reference in the reference cluster of an entity i (as positive reference sample).
Caye
A negative reference sample linked to an entity i, that is. reference j" refers to different entity from the entity i.
Xay
margin scaling variable for the sample j of the entity s.
(*)
cluster of sentences where entity i
appears and/or is refereneed.
C*
set of all entity- sentence clusters.
C Super Token Representation Learning
CT (t)
cluster of references to an entity
The BART adopts word-segmennttioon-aasss to- ken encoding method (Sennrich al., 2016) deal with oversized vocabulary issues. But the NLP parsing tools produce coreference resolution on words Researchers have applied the word-level structures either to the leading tokens of words (e.g., Heinzerling and Strube, 2019) or to the aggre- gated token representations (e.g., Ek and Bernardy, 2020). We take the latter approach and apply a GNN-based super token representation learning, detailed as follows.
C"
of all entity-refereee clusters.
|S|
The total number of sentences a ment.
h.
(latent) representation item or sam- ple.
An entity i's (latent) context representa- tion.
Margin ranking scoring function of posi- tive/negative sample with respect to an an- chor entity i.
C.
A learning objective or loss notation.
Table 7: Key notation summary. use place of item (or sample) identities for simplicity without loss of generality.
Given the tokens of word, we construct a token graph g = (V, E, D). V = f"o. "m) is the token node set. E {(vo ez)lū E [O, m]: ("o,
is the edge set between the leading token up and each following tokens. D = (0...., m] is the distance set of the subse- quent tokens the leading token. Note that we add a self-loop edge to the leading token to deal with a single-token word case. Similar to BART's positional embeddings, we build the distance em- beddings of vocabulary size [D] but use them as
A Key Notation Summary
Table summarizes the key used notations for their mathematical meanings and functions in our meth- ods.
B M3 Convolutional Neural Network
Convolutional neural networks with multi-filte and multi-kernels are computationally more eco- nomic than Transformer-based models for rela-
'NLP parsing tools (c.g.. Stanford CoreNIP) may use word segmentation methods (e.g., Penn Treehank tokeniza- tion) to produce words for annotation.

---

Dataset
Size
Dataset
Size
Size
Train
149634
Train
186873
65698
Validation
7782
Validation
10391
10391
Test
11483
Test
11328
11328
Table 8: Preprocessed CNNDM dataset sizes.
Table 9: Preprocessed XSum dataset sizes. 1. Without CR annotations. 2. With CR annotations. 3. We do not annotate CRs for validation and test datasets for inference. So, they remain the same for the two-stage fine-tunings.
edge features.
We adopt generic GNN model proposed by You et al. (2020) for learning the representations. Given the designed graph, a single-layer GNN is sufficient to learn super token representations. In simple form, the message passing is defined as:
and PRONOMINAL). We build the vocabularies of these CR attributes respectively. It is worth noting that CR entities and references are also annotated with their positions concerning their sentences and the associated sentence number. So, our process computes each sentence's 's length and the total num- ber of sentences. These are word-leved annotations.
+ bb), 1 (W by + bb)) + (W h
(27)
u E ((oo)
where {WY, b and {W? b2,) are the linear transformation parameters of the token embedding by and the distance embedding h2 respectively. The message aggregation followed as:
g.
We preprocess the encodings* of the built docu- ments using the model tokenizer". The number of encoded output tokens may exceed the length limit of the model. Truncation of the exceeded tokens would break the application of the CR annotations. We therefore exclude these samples. While build- ing the token document, we have also created a (29) word-token map (graph). The word-token map en- ables runtime mapping between word indices and token indices. The preprocessed encodings also save the model's 's token encoding time during train- ing runs. Table lists the final preprocessed dataset sizes.
h P
Σ
(28)
(VD(
The update is then computed as:
VV
= hy (x)-
up
Experiment Setup
D.1 Data Preparation
We use Stanford CoreNLP to annotate coreference resolution (CR) for the training dataset. Stanford CoreNLP uses Penn Treebank tokenization (a word segmentation method) to produce words for an- notation. So, the annotated training dataset has a slightly different data distribution from the original data. To maintain validation and test with the same data distribution as the annotated training data, we also use the same process to annotate the validation and test datasets without CR.
Annotated XSum Datasets Similar CNNDM preprocessing, we download XSum datasets us- ing Hugging Face's datasets package and annotate them using the Stanford CoreNLP parsing software. We then preprocess the annotated data to create our final train, validation, and test datasets. Noticed that small set of reference summaries in XSum has annotated CRs. It is insufficient fine-tune the pre-trained BART-base model directly with the CR-am anonttttde dataset. So, we create two training sets for two-stage fine-tunings, one without CR an- notations and the other with CR annotations. Note that we only conduct test evaluations on the fine- tuned model in the second stage after the model is fine-tuned with our proposed methods with the CR- derived data. We list the dataset sizes in Table 9,
Annotated CNNDM Datasets We download CNNDM datasets (train, validation, and test) using Hugging Face's datasets Python package. We use the Stanford CoreNLP parsing software to acquire coreference resolution (CR) annotations. We de- velop a tool to transform the CR annotations into training dataset format of entity referents, ref- erences, and their attributes. The attributes include animacy (e.g., ANIMATE and INANIMATE), gen- der (c.g., MALE and FEMALE), number (c.g.. SIN- GULAR and PLURAL), and type (e.g., PROPER
"The tokenization uses byte-puir encoding method. 9 htt htat:::s//huggnnaaaacc hooffacbbook/bbte base.

---

D.2 Key Implementation
tence feature in the concatenated feature vector [he h
D.2.1 Super Token Representation Learning
The GNN model discussed in Appendix Cs im- plemented by Pytorch Geometric package We adopt the model implementation from the package and configure it with our settings.
D.2.5 Boost Coreference Expressiveness by Annotated CR Attributes
As discussed in Appendix D.I, the CR annotation expresses the relations between an entity referent and its reference by several coreference attributes. We implement them as attribute embeddings and combine them with the coreference representations to boost representation expressiveness.
D.2.2
Adaptive Margin Scaling Variables and Base Margin
To determine a base margin, we are interested in the dynamics of the margin scaling variables. So, we add trace logic in the source code to log the maximum and minimum values of the scaling vari- trial
D.2.6 Index Mapping from Word-Level Structure to Token
ables per epoch during training. From runs, find that the value range starts from about [2225, 2.99] and converges towards the range about [1.99, 2.24] when runs are finished on an early stop criterion. Based on the observation, we set our base margin at 25.
The model-encoded tokens and the word-level CRs are not aligned by their sequential position indices. That is, their corresponding indices not the same. To apply the CR-derived data the super tokens, we have developed source codes for index mapping by utilizing the word-token map created in Appendix D.I.
D.2.3 Alignment Methods
Referents and references may consist of multiple words. To simplify the learning methods without loss of representational discriminative power, we take super node representation learning approach similar Appendix C. In short, we center on the keyword of an entity or its reference, and upply a GNN to aggregate its neighboring (up to) n-gram words at each side of the keyword if applicable"! We set n-gram to 2 in this paper based on our ob- servation of the annotated CR data.
D.2.7 Early Stop Training Criterion
We use an early-stop training approach up the configured maximum epoch. The criterion is the ROUGE metric-based evaluation of the validation dataset. The same ROUGE metrics for test time inference evaluation are used. The training stops when the ROUGE scores remain the same for a predefined number of consecutive times. We set the early stop criterion and the max epoch is 50.
Our alignment methods refactor several func- tions from the knowledge graph learning package 12 The changes are related to the com- plex vector based model for the entity-reference alignment method, including both interaction and margin ranking loss functions. We also add a simi- larity interaction interface for the entity-sentence alignment method.
D.2.8 Training on Multi-GPUs
We develop multi-GPU running procedure based on the reference runtime script We train mod- els on two-GPU parallelism. Our GPU cards are NVIDIA A100/80GB cach A configuration Dual AMs has a model size of 581. 1.460B.. fine- tuning session of the Dual AMS with CNNDM on an carly-stop setting takes about 94 hours. For XSum, we first fine-tune the model without CR- derived data (i.e., without using alignment methods). It takes roughly 26 hours an early- stop setting. We then further fine-tune the model using the alignment methods with the CR-derived data on the same early-stop setting for about 3 hours.
D.2.4 Entity Span Classification in Sntity-Sentence AM
The classification in Eq. 12 includes sentence fea- ture hag Given the entity context vector hg encap- sulates the possible salient feature of the sentence, we thus simplify the equation by omitting the sen-
peetts::/pytott geome tric. ri....r readthedocs.. io. 11 "Imposing n-gram constraint because entity annotations can occasionally include restrictive clause (e.g., which clause) as a whole.
/ cww.hhuggingaace/treeeeeee wtwaaaaeeeeetttteeeeeeseeeeeeeee blob/ wewaaateeeeeamlle//tttrrrre http://smmarzzzeee cummmriizaiinn/rnn. summar i ion_no_ wwwoooottaiierr___________ wwt.irrneery..
"hhtpps://e //pp::///itte h/g.tthubbccmm htt.u//......kke//gke//eeeeeeeee src/pykeen.
18 As the used GPUs are shared resources, we are con- strained to fully utilize them for larger pre-trained models.

---

Setting
CNNDM
XSum
Error
CNNDM
XSum
Type
Fact
Maximum Length
142
62
PRGASUS
Dual AMs
PRGASUS
Dual AMs
Minimum Length
56
11
Entity
21
intrinsic
Beam No.
4
6
Entity
0
Length Penalty
2.0
1.0
extrinsic
Subtotal
13
14
21
13
Table 10: Inference settings.
Modilier
2
14
Event
0
2
5
Event-time
0
2
D.2.9
Training Setting Summary
Location
Negation
0
We use AdamW optimizer. The learning rate is 5e-5 with a lincar decay. The weight decay is le
Number
Misspelling
0
5
Broken
44
0
2
sentence
Subtotal
46
17
35
42
D.2.10 Inference Setting
Total
59
31
56
55
We extract the inference settings from the pre- trained BART-base configuration as shown in Ta- ble 10.
Table 11: Human evaluation of FactPEGASUS factual- ity on the 100 random samples (CNNDM and XSum).
D.2.11 Automatic Evaluation Metrics
in the main script. The FactPEGASUS re- sults in SummaCw (p-85.0% a-0.199) and SummaCcone (M-75.9% 0-0.193) on CNNDM while producing Summa (p-18.9% a-0.309) o-0.150) on XSum. The FactPEGASUS has better SummaC scores than ours while, on the other hand, our models present better ROUGE scores as listed in the main script. To better understand what underlies such a disconnected correlation between the ROUGE and SummaC scores, we also further conduct our human evaluation the same 100 summaries as follows.
ROUGE Metrics The ROUGE automatic met- rics (R-1, R-2, and R-L) commonly used for ATS. We use ROUGE metrics implemented by Lin (2004) via wrapper API from Hugging Face's datasets Python package. The implementation pro- duces low, medium, and high scores for each metric. We take the high scores in our experiment evalua- tions.
and SummaCcom (μ-27.1%, CCn
SummaC Metrics We adopt SummaC source "5 to produce our formatted outputs.
E Extended Factuality Assessment
E.1 Comparison to FactPEGASUS
E.1.2 Human Evaluation
During our evaluation, we notice that the FactPEGASUS-generrn summaries from CN- NDM contain many broken sentences. That is, the sentences are suddenly clipped in way that they finish with words such as prepositions, pronouns, or determiners dangling at the end with missing information. We think that considering the phe- nomenon is important because the endings could otherwise avert various hallucination occurrences but can also miss out on important facts and leave the summaries in an incomplete and less compre- hensible state. We thus include the assessment of the phenomenon. The results are shown in Table 11 where we compare the FactPEGASUS with the Dual AMs.
We extend our factuality assessment to compare with FacPPEGASUS/Zeno-hoor by both automatic metric evaluations and manual assessment The extended assessment leads to some interesting findings and conclusions. We detail them in the section as follows.
E.1.1 Automatic Metric Evaluation
We use SummaC to evaluate the generated summaries by the FactPEGASUS on the same 100 random samples as the ones used
Phttps: thub. wtwptttt.....nnggeeeeeeeeeeeeeee cow/tingoourrol/ surma.. "It is worth noting that FaeIPEGASUS/Zero uses FactCC to score important secudo-summary sentence selec- tion for masked pre-training and FactCC was trained with CNNDM-derived data. As we use the authors' published source code and datasct to fine-tune and evaluate Fac-PEGA- SUS zero-shot on CNNDM, the scro-shot requirement is thus relaxed.
Compared to the FactPEGASUS on CNNDM, the Dual AMs method performs better on the intrin- sic named entity-related hallucinations (NERHs)

---

while the FactPEGASUS has much fewer extrin- sic entity hallucinations and syntactic agreement and misspelling errors. But the FactPEGASUS has a significant number of broken sentences. This could explain its lower ROUGE scores but higher SummaC's factuality consisteney scores in that the clipping sentence phenomenon may degenerate the n-gram overlap matches. On the other hand, it could benefit the factual entailment assessment in latent semantic space because of less noise and fewer outliers, even though might lose many facts at the same time. This can also partially explain its few extrinsic entity hallucinations.
Model
SummaCzs
Summa Coar
Conv
((%)
((%)
BART-base
72.5
0.233
68.8
0.214
Dual AMs
73.1
0.231
69.3
0.210
Table 12: SummaC score distribution statistics over the full CNNDM test (11488 samples).
SummaCzs
Summa Coor Cour
Model
((%)
BART-base
9.6
0.177
23.7
Dual AMs
9.8
0.178
23.7
0.048
Table 13: SummaC score distribution statistics over the
full XSum test set (1 (11228 samples).
CNNDM
Model
Summa C2
Summa Cone 0.003 (<0.05)
Comparing the models on XSum, we see that the Dual AMs also performs much better on the reduc- tion of intrinsic NERHs than the FactPEGASUS while the FactPEGASUS results in fewer syntac- tic agreement errors. Meanwhile, the FactPEGA- SUS has much higher errors in the modifier cate- gory. Given XSum has much shorter summaries discussed in the main script, the FactPEGASUS creates far fewer broken sentences than happens on CNNDM. Overall, the Dual AMs method shows outperformance on factuality consistency.
BART-base vs BART-
0.007 (<0.05)
base/Dual AMs
Table 14: Statistic significance (paired t-test) Sum- maC scores resulted from the full test set (CNNDM).
XSum
Model
SummaC2s
SummaCCom
BART-base vs BART-
0.304 (>0.05)
(50.05)
base/Dual AMs
Table 15: Statistic significance (paired t-test) Sum- maC scores resulted from the full test set (XSum).
E.2 Factuality Evaluation Statistical Significance Assessment
E.1.3 Discussion
As devised to reduce intrinsic entity hallucinations, methods have different objectives from n-gram matching seen in pre-training language modeling. So, we expect that our methods improve scores slightly on n-gram overlapping -baed metrics (e.g. ROUGEs) by matching the de-hallucinated entities and references in summaries. Given entities and references are sparse and small sets in long doc- uments, we also expect that the improvement in overall automatic factuality scores would be small. But would nonetheless like evaluate Sum- maC on the full test sets and examine the statistical significance of SummaC scores between the BART- base backbone (trained with the Dual AMs) and the BART-base baseline. We first compute SummaC scores on the full test sets. Their score distributions are shown in Table 12 on CNNDM and Table 13 on XSum respectively. We then, using paired t7 compute statistical significance scores between the BART-base backbone and the BART-base baseline, as shown in Table 14 on CN- NDM and Table 15 on XSum respectively. Our null
The analysis of the ROUGE results, SummaC scores, and human evaluation in main script has indicated that there is correlational disparity be- tween ROUGE scores and factuality consistency results. The assessment in Appendix E. provides us further insights into how various factual and syntactic errors might underpin the ROUGE and SummaC scores. We see how the generated sum- maries with the clipping phenomenon low n-gram overlap-based metric scores but high factual entailment metrie results. indicates to us that the automatic metrics can be a 'bias' in the model's generative capacity by not knowing the underlying conditions and phenomena of the generated summaries. We believe that the ATS factuality research going forward needs systemic approach to understand the correlations of various quantitative metric scores concerning wide range of factual and syntactic phenomena. Gaining such a holistic understanding of the correlations go- ing to open us up to new approaches to factuality modeling and evaluation metric development.
result in
the SummaC
Le use ttest rel API from Python's 's scipy.stats package.

---

hypothesis is that there is no significant difference between scores on the generated summaries from the two models. We set the significance level at 0.05 as standard.
Package
Version
Python
3.8.10
Pytorch
111.0+cul13 13
Hugging Face
4.9.0
By rejecting the null hypothesis, Table 14 indi- cates that the scores on the generated summaries from the Dual AMs-trained backbone are signifi- cantly different from those from the baseline CNNDM. Given Table 12 have shown that the backbone achieves better factuality scores than the baseline, this confirms our confidence that the sum- maries generated from the Dual AMs-trained back- bone achieve significantly better SummaC scores than those from the baseline. On the other hand, Table 15 on XSum accepts the null hypothesis, and indicates that there is no significant difference in scores the generated summaries between the Dual AMs-trained backbone and the baseline. We think that the results on XSum may be partly due to the one-sentence conciseness of XSum's 's sum- maries. Nonetheless, agrees with the results in Table 13.
(Transformers)
Pytorch Geometric (PyG)
2.0.4
PyKeen
1.9.0
Rouge Metrics
0.0.4
Stanford CoreNLP
4.4.0
Table 16: Key package versions.
coreference, for example, mistaking 'she" for 'he'.
5. It is intrinsic NERH if named entity is missing altogether when necessary its place. For example, no named entity is present when a coreference (e.g., he or she) is men- tioned.
We do not mark hallucinations for entity aggre- gations if their logical containment relation with the aggregated named entity is correct given their context.
F Defining Rules for Entity-Related Hallucination Analysis
Named entity-related hallucinations (NERHs) can have various forms. Therefore, necessary to define rules to identify those hallucinations for our statistical analysis. Although the focus of this paper is the intrinsic NERHs, the identifying rules also cover the extrinsic NERHs. So, categorize assessment rules for NERH follows,
As the pre-trained BART model has encoded a large amount prior knowledge of general infor- mation, there are considerable amounts of extrinsic hallucinations that may be factual. Thus, we do not consider the cases as hallucinations or eπrors if we can verify them using external knowledge bases such Wikipedia, online news, and/or Google Maps though they cannot be deduced from the source document. We do not mark cases errors if they can be reasonably deduced from the source documents. We extend the negation category cover different forms of meaning contradiction at the phrasal level.
1. It is an intrinsic NERH if named entity is mistaken for the other named entity within the context of the source document.
2. It is an intrinsic NERH if named entity is mistaken concerning the context of the source document where there may not be any other named entity present in the source document.
G Versions and Licenses
G.1
Key Packages
3. A named entity has its name incorrect, for example, surname and given name. If the full name is wrong, categorize it extrinsic NERH. the partial name (either surname or given name) is wrong, we categorize it as an intrinsic NERH. However, if there are one or two wrong characters a partial name, we categorize it as a misspelling instead.
G.1.1 Versions
We summarize the versions of key third-party pack- ages used our work in Table 16.
G.1.2 Licenses
The licenses related to these package versions are described follows.
Python Python license accessible 18
4. It is an intrinsic NERH named entity's coreference is mistaken for another entity's
https://///occ..ttoo.ooo 0tt//..///icee e/wicense..eee html.

---

Dataset
Version
CNNDM
3.0.0
XSum
1.0.0
Table 17: Dataset versions.
Pytorch Pytorch uses a collective license
Hugging Face Hugging Face uses Apache-2.0 License, and covers its packages including Trans- former derived models (c.g., BART), pre-trained BART-base tokenizer, datasets package (incl. CN- NDM and XSum), and Accelerate package.
PyG Pytorch Geometric2l is graph neural net- work modeling package and uses MIT License.
KeKeen PyKeen is knowledge graph learning package and uses MIT License.
Rouge Metrics Rouge Metrics uses Apache-2.0 License.
Stanford CoreNLP CoreNLP21 uses GNU Gen- eral Public License v3.0.
FactPEGASUS FactPEGASUS uses MIT Li- cense.
SummaC 2umma uses Apache-2.0 License.
G.2 Datasets
Table 17 lists version information of both CNNDM and XSum datasets used our work.
master/LICENSE.
hhttps:://gttuub..mm/pygg geametric.
v4.4.0.
2 http:////....ttteeee//eee//////s
wgg.ttubb..mm/inngfuuooo//sseeae 25 ht


Reviews:
[{"review_id": "HU8rl1eocj", "reviewer": "Reviewer_VoaE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your efforts to review our paper and for providing us with your expert feedback.\n\n*Unfair comparison to zero-shot FactPEGASUS as one of the baselines.*\n- We included FactPEGASUS as a baseline because\n    a) We use the authors' published source code and dataset to train and evaluate FactPEGASUS zero-shot on CNNDM. As FactPEGASUS uses FactCC to score important pseudo-summary sentence selection for masked pre-training and FactCC was trained with CNNDM-derived data, the zero-shot requirement is relaxed. The subsequently observed broken sentence phenomenon from the evaluation has ramifications for factuality evaluation metric research. We think that including it could be beneficial since the phenomenon has not been discussed elsewhere to the best of our knowledge.\n    b) The authors also pointed out in footnote 5 that the zero-shot requirement was relaxed to learn the connector used in their zero-shot evaluation.\n\n*The results on SummaCx are confusing since references have lower scores than the optimization object.*\n- Thank you for pointing out the less intuitive SummaC score disparity between gold references and the generated summaries. We think that it is a phenomenon mainly attributed to the extrinsic information in gold references ([gehrmann_repairing_2023]). Such a phenomenon is also observed in other factuality metric evaluations. For example, factuality evaluation scores on XSum are commonly much lower than those on CNNDM as seen in several prior works (e.g., [lyu_faithful_2022]; [dong_multi-fact_2020]; [yuan_bartscore_2021]) because XSum’s gold references contain much more extrinsic information than CNNDM’s ([lu_multi-xscience_2020]).\n\n*Need more factuality evaluation metrics.*\n- We strived to use evaluation tools from more recent research which often improve on several prior works. SummaC has taken advantage of several benchmark datasets of prior factuality evaluation metrics including FactCC. It has shown better performance than several prior works including DAE and FactCC-CLS on the benchmark datasets. A second factor in choosing evaluation tools is their package availability or their published source code working.\n\n*BART model is over 4 years old at this point, and BART-base is a weaker one.*\n- BART-derived encoder-decoders remained a popular choice for abstractive text summarization research. Depending on computational resource availability, recent prior works including the cited baselines in this paper may utilize BART-base. The GPUs/80GB used in this paper are shared resources such that it is infeasible for us to fully utilize them to train our methods with a large model (e.g., BART-large) over several days at any given time.\n\n*Why not conduct the methods on FactPEGASUS and use it as another strong baseline?*\n- We backbone a pre-trained model that has not been fine-tuned with any other downstream task. This allows us to assess the results of our methods (good or bad) knowing that there is little interference by a different downstream task of a prior work. Meanwhile, it is a great idea to use FactPEGASUS as a backbone to train a stronger FactPEGASUS with our methods in our future work. We can also follow this idea for other prior works.\n\n*Highly recommend evaluating the full test sets with different factuality evaluation metrics.*\n- We update the SummaC evaluation on the full tests.\n    On CNNDM (11488 samples), BART-base baseline results in 72.5\\%($\\mu$) and 0.233($\\sigma$) on SummaCzs, and 68.8\\%($\\mu$) and 0.214($\\sigma$) on SummaCconv. Dual AMs results in 73.1\\%($\\mu$) and 0.231($\\sigma$) on SummaCzs, and 69.3\\%($\\mu$) and 0.210($\\sigma$) on SummaCconv.\n    On XSum (11328 samples), BART-base baseline results in 9.6\\%($\\mu$) and 0.177($\\sigma$) on SummaCzs, and 23.7\\%($\\mu$) and 0.048($\\sigma$) on SummaCconv. Dual AMs results in 9.8\\%($\\mu$) and 0.178($\\sigma$) on SummaCzs, and 23.7\\%($\\mu$) and 0.048 ($\\sigma$) on SummaCconv.\n    Following your advice, we also managed to use QA-based QuestEval to evaluate summaries in time.\n    On CNNDM (11488 samples), BART-base baseline results in 57.9\\%($\\mu$) and 0.080($\\sigma$). Dual AMs results in 58.0\\%($\\mu$) and 0.081($\\sigma$).\n    On XSum (11328 samples), BART-base baseline results in 44.8\\%($\\mu$) and 0.128($\\sigma$). Dual AMs results in 44.8\\%($\\mu$) and 0.127($\\sigma$).\n    Given entities and references are sparse and small sets in long documents, we expect that the improvement in overall automatic factuality scores would be small. It is a factor that motivated us to conduct the manual evaluation of the effectiveness of our objectives on entity-related hallucinations and other categories. The manual evaluation indicated that our objectives are complementarily effective on their goals. Following prior works (e.g., [lyu_faithful_2022]; [kouris_abstractive_2021]; [zhu_enhancing_2021]) which may or may not improve automatic metric scores given the merits of their objectives, we would highlight our results with respect to our objectives.\n\n\n[gehrmann_repairing_2023]: Gehrmann, Sebastian and Clark, Elizabeth and Sellam, Thibault, \"Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text\", {Journal of Artificial Intelligence Research}, vol.77, pp.103--166, May 2023.\n\n[lyu_faithful_2022]: Lyu, Yuanjie and Zhu, Chen and Xu, Tong and Yin, Zikai and Chen, Enhong, \"Faithful Abstractive Summarization via Fact-aware Consistency-constrained Transformer\", {Proceedings of the 31st ACM International Conference on Information \\& Knowledge Management}, pp.1410--1419, October 2022.\n\n[dong_multi-fact_2020]: Dong, Yue and Wang, Shuohang and Gan, Zhe and Cheng, Yu and Cheung, Jackie Chi Kit and Liu, Jingjing, \"Multi-Fact Correction in Abstractive Text Summarization\", {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)}, pp.9320--9331, November 2020.\n\n[yuan_bartscore_2021]: Yuan, Weizhe and Neubig, Graham and Liu, Pengfei, \"BARTScore: Evaluating Generated Text as Text Generation\", {Advances in Neural Information Processing Systems}, vol.34, pp.7263--27277, 2021.\n\n[lu_multi-xscience_2020]: Lu, Yao and Dong, Yue and Charlin, Laurent, \"Multi-XScience: A Large-scale Dataset for Extreme Multi-document Summarization of Scientific Articles\", {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)}, pp.8068--8074, November 2020.\n\n[kouris_abstractive_2021]: Kouris, Panagiotis and Alexandridis, Georgios and Stafylopatis, Andreas, \"Abstractive Text Summarization: Enhancing Sequence-to-Sequence Models Using Word Sense Disambiguation and Semantic Content Generalization\", {Computational Linguistics}, vol.47, pp.813--859, December 2021.\n\n[zhu_enhancing_2021]: Zhu, Chenguang and Hinthorn, William and Xu, Ruochen and Zeng, Qingkai and Zeng, Michael and Huang, Xuedong and Jiang, Meng, \"Enhancing Factual Consistency of Abstractive Summarization\", {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies}, pp.718--733, June 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* These authors propose entity-sentence alignment adaptive margin ranking loss to help the model find more precise sentence spans of a certain entity.\n* The authors propose an entity-reference alignment adaptive margin ranking loss for better entity-reference alignment.\n* The results show the proposed methods achieve higher scores on SummaCzs and SummaCconv.\n", "reasons_to_reject": "BART model is the only baseline on SummaCzs and SummaCconv, which is over 4 years old at this point, the model used BART-base is a weaker one. In addition, the improvements are relatively small and based on only 100 samples. It's not clear if these gains would hold up if the experiments are conducted on the full test sets on CNN/DM and XSum.\n\nIn detail:\n1. Unfair Comparison. I don't understand why the authors use zero-shot FactPEGASUS as one of the baselines in Table 2. ROUGE evaluation.\n2. Need more factuality evaluation metrics. The results on SummaCx are confusing since references have lower scores but the optimization object is higher scores. Therefore, I suggest authors use more factuality evaluation metrics such as FactCC and DAE.\n3. The experimental results may be influenced by random samples. Only 100 samples are evaluated, and the gain is sight. I highly recommend evaluating the full test sets with different factuality evaluation metrics.\n4. Original FactPEGASUS has significantly higher SummaCx scores (in Appendix E.1) than the proposed methods. Why not conduct the methods on FactPEGASUS and use it as another strong baseline?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work introduces an adaptive margin ranking that facilitates two entity-alignment learning methods for reducing intrinsic named entity-related hallucinations on abstraction text summarization tasks. Compared with BART-base, the experiment results show their proposed method with BART-base works on automatic and human evaluation metrics.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ifETzQK9kb", "reviewer": "Reviewer_sW6c", "summary": "", "questions": "How does this method generalize to other domains (dialogue, scientific documents, etc) and model types (BART-large, decoder-only models, small (7B) foundation models)?\nSome of the terminology and explanations seem unnecessarily complex, could the writing be simplified?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your efforts to review our paper and for providing us with your expert feedback.\n\n*Limited scope of exploration.*\n- We follow the lead of recent prior works to focus solely on abstractive text summarization based on the encoder-decoder architecture with the two most used datasets (CNNDM and XSum) since abstractive text summarization is a relatively big topic.\n    Meanwhile, it is a very valid suggestion to investigate the generalizability of our methods for other domains and architectures in our future work, such as abstractive text summarization using decoder-only architectures and abstractive dialogue summarization.\n\n*With GPUs with 80GB memory, the choice of baselines seems insufficient.*\n- The used GPUs/80GB in this paper are shared resources. It is infeasible for us to fully utilize them for training over several days at any given time. Meanwhile, small-sized pre-trained models such as pre-trained BART-base are used and accepted for abstractive text summarization research in several recent prior works including the cited prior works in this paper. \n    As our objectives in this paper are to address specific hallucinations instead of competing overall performance, we think that the BART-base is a reasonable choice for the purpose.\n\n*Some of the terminology and explanations could be simplified.*\n- Thank you for suggesting an improvement to some of the terminology and explanations. We are to go through a second round of proofreading to identify and simplify terminology and explanations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Work touches on a very relevant and important topic of improving factuality\n- Analysis showing promising results\n- Well documented experiment details (Appendix)", "reasons_to_reject": "- Limited scope of exploration leaves the question of generalizability of the method open:\na) Datasets used in study cover the same domain (news), does the method work for other domains, such as dialogue?\nb) Model used in study is small by todays standards, does the method bring improvements to other architectures, such as decoder-only?\n- Given that the authors had access to state-of-the-art compute (GPUs with 80GB memory) the choice of baselines seems insufficient", "questions_to_authors": "How does this method generalize to other domains (dialogue, scientific documents, etc) and model types (BART-large, decoder-only models, small (7B) foundation models)?\nSome of the terminology and explanations seem unnecessarily complex, could the writing be simplified?", "paper_topic_and_main_contributions": "The paper studies named-entity related hallucinations (NERHs) in the context of abstractive text summarization. The authors propose two extensions to a traditional Seq2Seq model trained using XE loss: 1) adaptive margin ranking loss function which includes adaptive scaling variable of the base margin, 2) entity-sentence and entity-reference alignments methods which mitigate the entity-context and entity-reference hallucinations respectively. Experiments are conducted two datasets CNNDM and XSum, using a BART-base architecture.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uC6n7o8uhJ", "reviewer": "Reviewer_oSii", "summary": "", "questions": "A. It is unclear how we learn the token representation in eq 7?\n\nB. Please bold the highest scores in Table 1 and 2.\n\nC. In Table 1 and 2, compared to the BART-base, can you please verify if the improvements attained in the proposed methods are statistically significant?\n\nD. The evaluation needs to be done on the whole test set to accurately reflect the performance of the models. It would be better to re-compute the Table 3 and 4 to verify if the trend correlates for the whole test set. \n\nE. In Table 4, $SummaC_{Conv}$ scores, the scores barely change in the third decimal point. 0.233 to 0.239. Please verify if its statistically significant.\n\nMight increase the scores after re-computing them for Table 3 and 4. (answer to Question D).", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your efforts to review our paper and for providing us with your expert feedback.\n\n*Unclear why the authors disregard mutually excluding context-guided learning approach while performing a negative sampler.*\n- Thank you for raising the valid question. We design the context guidance approach instead of mutual exclusion sampling for the following considerations, a) We aim to maintain sufficient data samples to generalize trained models better. b) The context guidance approach also aims at helping models learn differences from similarities. It is analogous to human abstraction.\n\n*Provide explanation on how the entity clusters and their corresponding references are constructed.*\n- We use the Stanford CoreNLP parsing tool to produce conference resolution and sentence construction.\n    Conference resolution consists of named entities and their references. Each named entity is clustered with its references. Both entities and references are annotated with their positions concerning their sentences, the associated sentence number, and several relation attributes. \n    We also compute each sentence's length and the total number of sentences.\n    These are word-level annotations. \n    We also use the BART model's tokenizer (using BPE tokenization) to construct the token sequence of the document, and word-token map during the preprocessing.\n    The word-token map allows runtime mapping between word indices and token indices.\n    So, using data from the preprocessing, we build training triplet samples of entity, reference, and relation along with their positions from runtime mini-batches.\n    The related underlying implementations are mainly detailed in Appendix D.1, D.2.1, D.2.6, and C.\n    We are to improve on these appendices following your suggestion.\n\n*It is unclear how we learn the token representation in eq 7?*\n- With data from the point above, we can recover sentence token indices with respect to a document concerning equation 4 and equation 5. We then use the token indices to gather (i.e. index) the corresponding latent outputs of the encoder for equation 7.\n\n*The evaluation needs to be done on the whole test set.*\n- Following your advice, we updated SummaC evaluation on the full tests.\n    On CNNDM (11488 samples), BART-base baseline results in 72.5\\%($\\mu$) and 0.233($\\sigma$) on SummaCzs, and 68.8\\%($\\mu$) and 0.214($\\sigma$) on SummaCconv. Dual AMs results in 73.1\\%($\\mu$) and 0.231($\\sigma$) on SummaCzs, and 69.3\\%($\\mu$) and 0.210($\\sigma$) on SummaCconv.\n    On XSum (11328 samples), BART-base baseline results in 9.6\\%($\\mu$) and 0.177($\\sigma$) on SummaCzs, and 23.7\\%($\\mu$) and 0.048($\\sigma$) on SummaCconv. Dual AMs results in 9.8\\%($\\mu$) and 0.178($\\sigma$) on SummaCzs, and 23.7\\%($\\mu$) and 0.048 ($\\sigma$) on SummaCconv.\n    The results are consistent with those evaluated on 100 random samples in the paper.\n\n*Verify statistical significance of SummaCconv in Table 4 and Rouges in Tables 1 and 2.*\n- We computed statistical significance on SummaC scores using a paired t-test between BART-base baseline and Dual AMs following your suggestion. The null hypothesis is no significant difference between them.\n    On CNNDM, we got a p-value of 0.007 (<0.05) for SummaCzs, and 0.003 (<0.05) for SummaCconv.\n    On XSum, we got a p-value of 0.304 (>0.05) for SummaCzs, and 0.912 (>0.05) for SummaCconv.\n    Results for CNNDM reject the null hypothesis while the results for XSum accept it. This may be partly due to the one-sentence conciseness of XSum's summaries.\n    Following your advice, we further computed a paired t-test on ROUGE f-scores between BART-base baseline and Dual AMs.\n    On CNNDM, p-value is 0.9443 for rouge-1, 0.9820 for rouge-2, and 0.8172 for rouge-L.\n    On XSum, p-value is 0.7119 for rouge-1, 0.3968 for rouge-2, and 0.3281 for rouge-L.\n    As devised to reduce intrinsic entity hallucinations, our methods have different objectives from n-gram matching seen in pre-training language modeling. So, we expect that our methods improve scores slightly on n-gram overlapping-based metrics (e.g. ROUGEs) by matching the de-hallucinated entities and references in summaries.\n    Given entities and references are sparse and small sets in long documents, we also expect that the improvement in overall automatic factuality scores would be small. These factors motivated us to conduct the manual evaluation of the effectiveness of our objectives on entity-related hallucinations and other categories. The manual evaluation indicated that our objectives are complementarily effective on their goals.\n    Following prior works (e.g., [lyu_faithful_2022]; [kouris_abstractive_2021]; [zhu_enhancing_2021]) which may or may not improve automatic metric scores given the merits of their objectives, we would highlight our results with respect to our objectives.\n\n*Typos, and bold the highest scores in Table 1 and 2.*\n- Thank you for pointing out the typos. We are to fix them. We are also to bold the highest scores in Table 1 and Table 2 following your advice.\n\n[lyu_faithful_2022]: Lyu, Yuanjie and Zhu, Chen and Xu, Tong and Yin, Zikai and Chen, Enhong, \"Faithful Abstractive Summarization via Fact-aware Consistency-constrained Transformer\", {Proceedings of the 31st ACM International Conference on Information \\& Knowledge Management}, pp.1410--1419, October 2022.\n\n[kouris_abstractive_2021]: Kouris, Panagiotis and Alexandridis, Georgios and Stafylopatis, Andreas, \"Abstractive Text Summarization: Enhancing Sequence-to-Sequence Models Using Word Sense Disambiguation and Semantic Content Generalization\", {Computational Linguistics}, vol.47, pp.813--859, December 2021.\n\n[zhu_enhancing_2021]: Zhu, Chenguang and Hinthorn, William and Xu, Ruochen and Zeng, Qingkai and Zeng, Michael and Huang, Xuedong and Jiang, Meng, \"Enhancing Factual Consistency of Abstractive Summarization\", {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies}, pp.718--733, June 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper has novel contribution for entity based intrinsic hallucination area.\n2. Human evaluation of the randomly sampled generated summaries provide more insights into the type of entity hallucination errors the baseline makes compared to the proposed method. \n3. Compares the proposed model's performance based multiple baseline SOTA.", "reasons_to_reject": "1. It is unclear why the authors disregard mutually excluding context-guided learning approach while performing negative sampler. Providing intuition for this might benefit the overall paper.\n2. It might be easier to understand the methodology by providing explanation on how the entity clusters and their corresponding references are constructed.\n3. It is unclear whether the results obtained in Table 1-4 are significant compared to the baseline.\n4. The paper randomly samples 100 examples from the test set to evaluate their faithfulness. The relatively high standard deviation around 0.22 (where score range is 0 to 1), tells that sampling 100 instances is not enough to conclude of the performance. It might be better to re-compute the Table 3 and 4 with whole test set to verify whether the trend still stands. ", "questions_to_authors": "A. It is unclear how we learn the token representation in eq 7?\n\nB. Please bold the highest scores in Table 1 and 2.\n\nC. In Table 1 and 2, compared to the BART-base, can you please verify if the improvements attained in the proposed methods are statistically significant?\n\nD. The evaluation needs to be done on the whole test set to accurately reflect the performance of the models. It would be better to re-compute the Table 3 and 4 to verify if the trend correlates for the whole test set. \n\nE. In Table 4, $SummaC_{Conv}$ scores, the scores barely change in the third decimal point. 0.233 to 0.239. Please verify if its statistically significant.\n\nMight increase the scores after re-computing them for Table 3 and 4. (answer to Question D).", "paper_topic_and_main_contributions": "The paper addresses the problem of hallucination in abstractive summarization. Hallucination refers to the spans in the generated output that either are not supported (extrinsic) or contradict(intrinsic) the input context. Particularly, the authors propose a loss function named adaptive margin ranking loss to reduce entity based intrinsic hallucination. They propose two loss functions, namely, entity-sentence alignment and entity-reference alignment methods. The Entity-sentence alignment method takes into account the entity names while the latter takes into account its reference pronouns in the remainder of the article. They use contrastive learning by sampling positive and negative samples from the entity clusters. They adapt the Distance-IoU from [1] and propose Entity span Distance-IoU for the loss function. \n\nReferences:\n1. Zhaohui Zheng, Ping Wang, Wei Liu, Jinze Li, Rong guang Ye, and Dongwei Ren. 2020. Distance- IoU Loss: Faster and Better Learning for Bounding Box Regression. Proceedings of the AAAI Conference on Artificial Intelligence, 34(07):12993 13000. Number: 07. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 20,25 ~used\n2. In line 256, denoted by tp and tq\n3. In line 587, \"Although, \"\n4. Remove empty acknowledgements section"}, "extra_text": {}}, {"review_id": "Ly5Wds0j17", "reviewer": "Reviewer_n18X", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your efforts to review our paper and for providing us with your expert feedback. \n\n*The improvement of the proposed method over the baseline in terms of automatic evaluation metrics is not obvious.*\n- As devised to reduce intrinsic entity hallucinations, our methods have different objectives from n-gram matching seen in pre-training language modeling. So, we expect that our methods improve scores slightly on n-gram overlapping-based metrics (e.g. ROUGEs) by matching the de-hallucinated entities and references in summaries.\n    Given entities and references are sparse and small sets in long documents, we also expect that the improvement in overall automatic factuality scores would be small. These factors motivated us to conduct the manual evaluation of the effectiveness of our objectives on entity-related hallucinations and other categories. The manual evaluation indicated that our objectives are complementarily effective on their goals.\n    Following prior works (e.g., [lyu_faithful_2022]; [kouris_abstractive_2021]; [zhu_enhancing_2021]) which may or may not improve automatic metric scores given the merits of their objectives, we would highlight our results with respect to our objectives.\n\n*Why not further classify the intrinsic entity hallucinations more finely into the two proposed methods?*\n- This paper did not separately evaluate entity-entity hallucinations and entity-reference hallucinations because the evaluation of 100 random samples did not see obvious entity-reference hallucinations. Additionally, the results from the manual evaluation showed that two alignment methods are complementary to reduce intrinsic entity-related hallucinations more than individual methods working in isolation.\n\n*Several observations on the CNNDM and XSum datasets in Tables 5 and 6 are not discussed in detail.*\n- We strived to reveal the most valuable aspects of our work, including observations and insights, within the 8-page limit. So, we may omit some discussions that may be less important. In this case, we focus on entity-related hallucinations, most error occurrences of other categories, and their plausible causes that we might be able to explain from Tables 5 and 6.\n\n\n[lyu_faithful_2022]: Lyu, Yuanjie and Zhu, Chen and Xu, Tong and Yin, Zikai and Chen, Enhong, \"Faithful Abstractive Summarization via Fact-aware Consistency-constrained Transformer\", {Proceedings of the 31st ACM International Conference on Information \\& Knowledge Management}, pp.1410--1419, October 2022.\n\n[kouris_abstractive_2021]: Kouris, Panagiotis and Alexandridis, Georgios and Stafylopatis, Andreas, \"Abstractive Text Summarization: Enhancing Sequence-to-Sequence Models Using Word Sense Disambiguation and Semantic Content Generalization\", {Computational Linguistics}, vol.47, pp.813--859, December 2021.\n\n[zhu_enhancing_2021]: Zhu, Chenguang and Hinthorn, William and Xu, Ruochen and Zeng, Qingkai and Zeng, Michael and Huang, Xuedong and Jiang, Meng, \"Enhancing Factual Consistency of Abstractive Summarization\", {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies}, pp.718--733, June 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is generally well written and the proposed method is easy to follow and understand.", "reasons_to_reject": "1. The improvement of the proposed method over the baseline in terms of automatic evaluation metrics is not obvious, and further validation of the effectiveness of the proposed method in terms of producing informative summaries is needed.\n2. In the manual evaluation, the authors counted the entity hallucinations, syntactic agreement errors, and misspelling errors respectively, why not further classify the intrinsic entity hallucinations more finely into the two intrinsic hallucinations proposed to be solved in this paper: i.e. the entity-entity hallucinations and the entity-reference hallucinations, in order to further argue whether the two alignment methods proposed in this paper are effective or not in mitigating the entity hallucinations.\n3. For the analyses in Tables 5 and 6, there are several observations on the CNNDM and XSum datasets that are not discussed in detail by the authors.", "questions_to_authors": "", "paper_topic_and_main_contributions": "To mitigate the implicit entity-related hallucinations, this paper proposes an adaptive margin ranking loss function and adopts an entity-sentence alignment method (E-Sent AM) to tackle entity-entity hallucinations and an entity-reference alignment method (E-Ref AM) to mitigate entity-reference hallucinations. Experiments on the CNNDM and XSum datasets show that it improves the baseline model on automatic evaluation scores.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to address named entity-related hallucinations through an adaptive margin ranking loss function and two alignment methods 1)  entity-sentence alignment and 2) entity-reference alignment to reduce halluciation in abstractive text summarization. The results seem to be promising based on BART-base model evaluated on CNNDM and XSum. 

Pros: 
- Work touches on a very relevant and important topic of improving factuality
- Introduces novel adaptive margin ranking loss function and alignment methods.
- Analysis showing promising results
- Well-documented experiment details (Appendix)

Cons: 
- The choice of baselines and datasets raises concerns about the generalizability of the proposed methods to other domains and larger, modern architectures.
- The performance improvements seen are relatively small and based on a limited number of samples (100), leading to questions about the significance of the results.
- "Given that the authors had access to state-of-the-art compute (GPUs with 80GB memory) the choice of baselines seems insufficient"

In general, reviewers express enthusiasm regarding the proposed methods. However, they pinpoint two significant concerns: the perceived weakness in the evaluation process and the unjustified choice of using BART-base as the backbone for the system. In its current form, while the submission retains some element of excitement, it does not meet the threshold for acceptance at the main conference. Addressing these concerns, possibly by incorporating other baseline models could potentially elevate the work to the required standard. On the other hand, I think the work meets the standard to be accepted in the findings. 

I also found the justification of BART-base is not fully convincing to me, even bigger models does not work out, models like T5-small could be alternatives compared to test on a single model:

"With GPUs with 80GB memory, the choice of baselines seems insufficient.

The used GPUs/80GB in this paper are shared resources. It is infeasible for us to fully utilize them for training over several days at any given time. Meanwhile, small-sized pre-trained models such as pre-trained BART-base are used and accepted for abstractive text summarization research in several recent prior works including the cited prior works in this paper. As our objectives in this paper are to address specific hallucinations instead of competing overall performance, we think that the BART-base is a reasonable choice for the purpose.
"
Decision: Accept-Findings