Factual Relation Discrimination for Factuality-oriented Abstractive Summarization
Authors: Zhiguang Gao
PEIFENG LI
Feng Jiang
Xiaomin Chu
Qiaoming Zhu
Venue: EMNLP 2023 Findings (2023)
Keywords: Factuality-oriented Abstractive Summarization
Factual Relation Discrimination
TL;DR: Using factual relation discrimination for factuality-oriented abstractive summarization

Abstract: Most neural abstractive summarization models are capable of producing high-quality summaries. However, they still frequently contain factual errors. Existing factuality-oriented abstractive summarization models only consider the integration of factual information and ignore the causes of factual errors. To address this issue, we propose a factuality-oriented abstractive summarization model DASum, which is based on a new task factual relation discrimination that is able to identify the causes of factual errors. First, we use data augmentation methods to construct counterfactual summaries (i. e., negative samples), and build a factual summarization dataset. Then, we propose the factual relation discrimination task, which determines the factuality of the dependency relations in summaries during summary generation and guides our DASum to generate factual relations, thereby improving the factuality of summaries. Experimental results on the CNN/DM and XSUM datasets show that our DASum outperforms several state-of-the-art benchmarks in terms of the factual metrics.

Factual Relation Discrimination for Factuality-oriented Abstractive
Summarization
Zhiguang Gao Peifeng Li Feng Jiang Xiaomin Chu', Qiaoming Zhu'
"School of Computer Science and Technology, Soochow University, Suzhou, China School of Data Science, The Chinese University of Hong Kong, Shenzhen, China
Shenzhen Research Institute of Big Data, Shenzhen, China 4 University of Science and Technology of China, Hefei, China 1503022536@qq com, pfl ofli@@udda.em edu. cn, jeffrey wii.acnhk.eeeuu..ommm cn {xmchu, qmzhu)@suda edu. cn
Abstract
approaches can partially deepen the model's per- ception of facts. However, they still cannot find the cause of factual errors. One the reasons for this that the available summarization corpora contain only factual summaries and ignore counterfactual summaries. If counterfactual summary has the annotated causes, it will help the model to identify the causes of the factual errors and then improve the performance of factuality-orientedd abstractive summarization. For example, there is a document whose summary is "Tom stole the cheese". If its counterfactual summary is "Jerry stole the cheese" and the annotated cause entity swapping (i.e., re- placing "Tom" with "Jerry"), it is possible to train model to deal with different factual errors based on their causes.
Most neural abstractive summarization mod- els are capable of producing high-quality sum- maries. However, they still frequently contain factual errors. Existing factuality-oriente ab- stractive summarization models only consider the integration factual information and ig- nore the causes of factual errors. To address this issue, we propose a factuality-oriented ab- stractive summarization model DASum, which is based on a new task factual relation discrim- ination that is able to identify the causes of factual errors. First, methods to construct counterfactual summaries (i.e., negative samples), and build a factual sum- marization dataset. Then, propose the fac- tual relation discrimination task, which deter- mines the factuality of the dependency rela- tions in summaries during summary generation and guides our DASum to generate factual rela- tions, thereby improving the factuality of sum- maries. Experimental results the CNN/DM and XSUM datasets show that our DASum out- performs several oafe-off-the-ar benchmarks in terms of the factual metries.
data augmentation
In this paper, we introduce factual relation dis- crimination task factuality-oriented abstractive summarization, which can make the model aware of the causes of factual errors and learn to avoid factual errors during the summary generation pro- cess. Specifically, we propose DASum model based on factual relation discrimination to study factuality-oriented abstractive summarization.
1 Introduction
We first data augmentation methods (e.g., pronoun swapping, sentence negation, time and date entity swapping, quantifier swapping, and named entity swapping) to construct counterfac- tual summarics, which, together with the annotated factual summaries, form the factual summarization dataset. Experimental results on the CNN/DM and XSUM datasets show that our DASum outperforms several state-of-the- the----rtth-ttttt benchmarks the factual mtriis (i.e., FactC, DAE and SENT). We summa- rize our contributions as follows:
Using pre-trained models, neural abstractive sum- marization (Lewis et al., 2020; Zhang et al.. 2020) produce fluent and informative summaries. However, the facts in the generated summaries often inconsistent with the source documents. For example, Cao al. (2021) and Goyal and Durrett (2021) show that nearly 30% the summaries gen- erated by recent neural models have factual errors. Thus, factuality-oriented abstractive summariza- tion is intended to improve the factuality of sum- maries and make summarization models applicable in practice.
We a series of data augmentation methods construct the factual summarization dataset to simulate the distribution of factual errors the model-generated summaries;
Recent work improving the factuality of gen- erated summaries relies heavily on extracting facts from the source documents and integrating these facts into abstractive summarization models. These
We propose the factual relation discrimination task, which determines the correctness of the

---

Factual Signal
Data
Article of Positive Sample
Target Summary Positive Sample
Summarization
Augmentation
Factual
Training
Training
Summarization
Swapped Target
DASum
Datasel
Dataset
Article of Negalive Sample
False
Summary Negative Sumple
Summarization
Testing
Article of Positive Sample
Testing
Seq2seg of DAmum
Generated Summary
Dataset
Figure Overall process of our model DASum.
facts in the summary when generating, guid- ing the model to generate correct factual rela- tionships, and thereby enhance the factuality of summaries.
reward for RL. However, because the accuracy of the factual evaluation metric is not good enough, RL doesn't it really make the model understand the facts in the source document.
2 Related Work
Methodology
Previous studies on improving the factuality of sum- maries mainly fall into two categories, i.c., fact integration and reinforcement learning.
Fig. I shows the overall process of our model DA- Sum. In the training phase, the factual summariza- tion dataset is first constructed through data aug- mentation strategies, and then the DASum model is trained together with factual signals; in the testing phase, the correct factual signal "true" added all samples to generate summaries.
Fact integration is based on integrating factual knowledge into the summarization model. Cao al. (2018) extract relational triples from the source document as internal facts, and use a dual encoder to encode the source document and inter- nal facts simultaneously. Zhu et al. (2021) also take relation triples as internal facts and build the relation triple graph by connecting different triples. They then encode the relation triples graph with graph neural network and feed the graph represen- tation into the decoder of seq2seq model. Gao al. (2022b) encode internal facts and external Knowledge Graph information simultaneously to alleviate internal and external factual errors. Gao et al. (2022a) analyse the fine- mme-grained categories of factual errors and create fact graphs to repre- sent the factual information in the source document and target summary. They then employ adversarial learning to integrate the fact graph into the summa- rization model. Balachandran al. (2022) propose a factual error correction model, which improves the factuality by modifying summaries.
Fig. shows the architecture of DASum. DA- Sum has different training modes for positive and negative samples on the factual summarization dataset. For positive samples, shown in Fig. 2(b), the correct factual signal (True) is added the end of the source document. At this point, all facctull relations in the summary are factually cor- rect; for negative samples, as shown in Fig. 2(a), the counterfactual signal (False) is added, and there are both factual and counterfactual relations in the summary.
In this section, we first introduce the construc- tion process of the factual summarization dataset. then explore the representation methods of core words and dependent words in factual relations. and finally elaborate on the factual relation discrim- ination.
3.1 Construction of Factual Summarization Dataset
Recent studies (Choubey et al., 2021; ; Cao et al., 2021) also improve the factuality of summaries through reinforcement learning (RL). Choubey et al. (2021) use two factuality metrics, entity over- entailment metric,
The main effort of previous research to enhance the factuality of generated summaries is to better inte- grate factual information into the model, ignoring the causes of factual errors. To address this issue, train both the abstractive summarization task and the factual relation discrimination task using the factual summarization dataset on both factual and counterfactual summaries.
lap metric and dependency compute the factuality score and then it as reward for RL. Cao et al. (2021) focus on exter- nal factual errors and annotate the external factual dataset XENT, a factual discriminator is trained on XENT, and the discriminator score used as a

---

Factual Instance Discrimination
1. right are
wrong
(u) Negative sample
Encoder
Decoder
Tom died 2007, the 13th year after mom Andy passed away in
Andy died
2007
1996.12.1 [SEP] False
right
(b) Positive sample
Factual Guidance
case
Signal
Encoder
Decoder
Tom died in 2007, the 13th year after his mom Andy passed away in
Andy died
1996.12. 12.1
1996.12.1 [SEP] True
Figure 2: The architecture of DASum.
The factual summarization dataset not only con- tains factual summaries (positive samples), but also contains counterfactual summaries (negative sam- ples). It is oriented from the existing summariza- tion datasets (e.g., CNN/DM and XSUM) which only have factual summaries. In this paper, various kinds of dependency relation-basee transformation rules construct counterfactual summaries. That is, by modifying the dependency words of some dependency relations summary, the modified summary is counterfactual. For exam- plc. the time in factual summary "Tom died in 2023.6.20" is modified to "2007", we can obtain a counterfactual summary "Tom died in 2007". The negative samples constructed as follows.
use the operation of dependent word swapping to construct negative samples.
Since summary may contain more than one dependency relation, mary to many counterfactual summaries by modify- ing one or more dependency words. To balance the rate of positive and negative samples, we only con- struct one counterfactual summary for each sum- mary in NE. Particularly, only a part of dependency relations are used to construct the negative samples. That is, only the selected pronouns, auxiliary verbs, time and date entities, quantifiers, and named enti- ties are transformed, while the rest kept in their original state. This train the model to discover the reasons for factual errors, rather than coarsely discriminating the factuality the entire summary. The transformation rules for dependent words the summary are as follows.
transfer factual sum-
First, we divide the training set the original summarization dataset into two parts PO and NE, where the original summaries in PO are positive samples of the new factual summarization dataset and its negative samples are constructed based on NE by transformation rules. The discussion of different portion of PO and NE can be found in Subsection 4.4.
Pronoun swapping replaces pronouns in the summary with other pronouns of the same type. Pronouns refer to words that have the function of replacing or indicating people or objects, and they can be divided into subject pronouns, object pronouns, possessive pronouns, and reflexive pro- nouns. Gao et al. (2022a) classify factual errors at fine- ine-gnained level and point out that coreference errors common type of factual errors in the generated summaries. The cause of coreference rors is mainly due the ambiguity and confusion of the model regarding the objects referred to by pronouns. To solve this problem, we replace the pronouns the summary to deepen the model's
Then, we use the Spacy (Altinok, 2021) tool to obtain the dependency relations from each sum- mary in NE. A dependency relation is an are that points from the core word to the dependent word. According to the analysis the causes of the fac- tual emrrr in previous work (Gao et al., 2022b), we remove the relations whose dependency words are not one pronouns, auxiliary verbs, time, dates, quantifiers, or named entities. Here, we simply

---

connection the pronouns and the objects they refer to. Specifically, for pronouns that exist in the summary, we replace them with other pronouns of the same type. An example is shown as follows.
"may".
E2: Oringinal summary sentence: Peter may have enough to spread around.
Modified summary sentence: Peter may not have enough to spread around.
E1: Original summary sentence: Her sister, shaneah, was dating Iloyd.
Time and date entity, quantifier, and named entity swapping respectively refer to replacing time and date entities, quantifiers, and named enti- ties that exist in the summary with other different entities of the same type. The cause of factual errors in generated summaries is usually due to the model's hallucination with the entity objects in the source document. To alleviate this situation, replace the time and date entities, quantifiers, and named entities that exist in the summary with other entities of the same type to train the model to identify whether the generated entities are correct. First, we use the Spacy (Altinok, 2021) tool to iden- tify entity types and select representative time and date entities, quantifiers, and named entities in the source document and the gold summary. Then, we replace the entities presenting in the summary with entities of the same type in the source document, and an example is shown as follows.
Modified summary sentence: His sister, shaneah, was dating lloyd.
In E1, the pronoun "her" in the original sum- mary sentence belongs to the possessive pronoun and is a denendency word. We randomly select another possessive pronoun (e.g., his) to replace "her", It is worth noting that some pronouns have multiple types, such as "her", which is both an ob- ject pronoun and a possessive pronoun. Since part of speech parsing tools are unable to further clas- sify pronoun types, we randomly assign type to pronouns with multiple types, such as calling "her" a possessive pronoun.
Sentence negation is reversal of the seman- ties expressed in the summary, with the positive state changing to the negative state or the negative state changing to the positive state. When doing sentence negation, we focus on the auxiliary verbs in English (c.g., are, is, was, were). The summary generated by the abstractive summarization model sometimes confuses the negative form of sentences, for example, the negative statement that appears in the source document while the content in the sum- mary is a positive statement. To solve this problem, we deepen the model's understanding of the posi- tive and negative states of sentences by changing the negative form of the auxiliary verbs. Specif- ically, if the summary sentence has an auxiliary word, changing the negative form of the sentence can be divided into the following three steps:
E3: Original summary sentence: Andy died in 12.11.
Set of DATA type entities in the source docu- ment: {1996.12.1, 2007, this week, Friday}
Modified summary sentence: Andy died in 2007.
In E3, "199.. 12.1" is an entity of the type DATE. In this case, we use other entities of the type DATA, such "2007" selected from the source document, to replace "1996. 12.1", thus forming a factual sam- ple with factual errors. Specifically, we do not directly select an entity from the summary, mainly because the content of the summary is not as rich as the source document. By selecting an entity from the source document, the model can be further trained to discriminate the correctness of factual samples.
(i) We determine the positive or negative status of the summary sentence. If the auxiliary verb is followed by "not", "n't", or "not", the summary sentence is in a negative state. Otherwise, it is in positive state;
(ii) If the state of the summary sentence is pos- itive, we add the negative words "not", "n't", or "not" after the auxiliary verb;
Finally, when the rate of PO to NE is set to 1:1, the positive and negative samples in the factual summarization dataset are 143,223 and 143,890 based on the CNN/DM's training set, 102,018 and 101,999 based on the XSUM's training set. Be- sides, our task of factual relation discrimination is to judge whether a dependency relation is true or false in the corresponding summary and we also must construct a training set to learn a model to discriminate factual and counterfactual samples.
(iii) If the state of the summary sentence is nega- tive, we remove the negative word after the auxil- iary verb.
E2 shows an example of sentence negation. The summary sentence has the auxiliary verb "may" and is in an positive state. Then, the sentence is changed to a negative state by adding " "ot"" after

---

Hence, we randomly select 10 factual dependency relations and no more than counterfactual rela- tions from a summary in the CNN/DM dataset to construct the training set of factual relation discrim- ination. For the XSUM dataset, these figures are 4 and 2, since the summaries in CNN/DM is long than those in XSUM. Finally, there are 1,831,627 positive relations and 640,541 negative relations in the training set of the factual relation discrimi- nation task on CNN/DM, while these figures are 653,012 and 214, 21,,677 on XSUM.
words and core words are sd - [zd; #5] and sy [ax 2*1, respectively.
3.3 Factual Relation Discrimination
DASum uses the task of factual relation discrim- ination to guide in mining the causes of fac- tual errors. As mentioned in Subsection 3.1, there are multiple factual relations S { 11)....).. (Dm- Hmm} in summary SUM. Each factual relation is a dependency relation, where D; is a dependent word and H; is a core word. Factual relation discrimination is to deter- mine the correctness of a given dependency relation in the generated summary.
3.2 Representation of Factual Relations
When constructing the factual summarization dataset, each dependency relation has two core el- ements: the dependent word and core word. The dependent word and core word are composed of words phrases and can be considered a span of text. This subsection describes the representation of the span text.
For factual relation (D,, we first obtain the feature representation st [=d: =*), s) = [zx; *]
of the dependent and core word through section 3.2. Then, we splice s sf. to obtain the representation of the factual relation s² [s s/.. Finally, we do the factual binary classification of factual relations through linear regression, as shown in Eq. 1, where is the sigmoid activation function, is the factual label, and W, b are the parameters to learn.
When the summary is decoded through the seq2seq (Lewis et al., 2020) model, it is tok- enized to obtain its token sequence. We extract the token sequences D d2, ... ,de and = (h1. h2,..................... h corresponding to the de- pendent words and core words, where and refer to the number of tokens in D, H. The represen- tation of dependent words and core words can be calculated as follows:
logils oWW'' b)
Lifett 1
(1)
ylogg(lggiss))
Considering that the factual relation discrimina- tion task is to assist the abstractive summarization model in better generating summary, we add the contrastive language loss Liang. as shown in Eq. 2, where 0 is the parameter of the seq2seq model, P Positive and Negative are the positive and nega- tive sets constructed in Subsection 3.1, and X, Y represent the source document and summary, spectively. Liang based the idea that for positive instance, the model should minimize its MLE loss, while for a negative instance, since the negative instance has undergone phrase replace- ment, to prevent the language model from learning the wrong posterior probability, it is necessary to maximize its MLE loss.
(i) Boundary Embedding. In this paper, we use BART (Lewis et al., 2020) as the basic architecture of the abstractive summarization model, and the output of the last layer of the hidden layers of the BART decoder will be used as the feature represen- tation of the summary. The feature representation corresponding to the token sequence of the depen- dent word D is Dj concatenate the first and last token features of the dependent word as the final feature representation the dependent word, i.c. 2d = [fa, fa.]- We use the same representation for the core word yields 2h [fn J.
[fd.. Jdz: J. We
(ii) Length Embedding. We introduce length embedding (Fu et al., 2021) to expand the fea- ture representation of dependent and core words. Specifically, we will first establish a length feature query table and then query the length feature ta- ble based on the length of dependent words and core words to obtain the length vectors z1, & of dependent words and core words.
Liong = t + L.om
Ltm Ex Powitiiie tog (Yl)))
(2)
L.o - Ex Negatiooo - pe (Y (Y(X))
The final loss is as follow.
The final feature representations of dependent
L Ljact + Liang
(3)

---

3.4 Factual Signal
Model
R1
R2
RL.
DAE
SENT FactC
CNN/DM
DASum is trained to understand the causes of fac- tual errors by factual relation discrimination. How- ever, DASum is not explicitly guided to gener- ate factual summaries during the validation pro- cess. To address this issue, add factual sig- nal to control the direction of model-generated content. Specifically, for source document X {II,... ,T:). after adding factual signal, be- comes X = [-1,,... I,, sep: flag), where flag is a factual signal. During the training process, when the instance is positive, flag = True; oth- erwise, flag False; During the validation pro- cess, flag is always True.
BertSum
41.43 19.05 38.55 71.74 87.51 53.93
Unilm
43.33 20.21 40.31 65.21 82.32 36.43
FASum
40.53 17.84 37.40 73.57 79.12 50.14
FASumpe
40.38 17.67 37.23 73.80 80.36
51.17
BART
43.86 20.92 40.64 72.75 84.94
49.60
CONSEQ
44.40 21.37 41.17
72.83
CLIFF
41.01
50.05
FACTEDIT
42.17 20.22 39.37 75.71
75.49
LASum
43.25 20.21 40.09 82.51 87.61
82.40
DASum
44.27 21.18 40.99 77.64 88.24
60.24
DASum(L)
44.55 21.42 41.33 84.16 90.43
82.92
XSUM
BertSum
38.76 16.33 31.15 28.70 61.38 23.56
4 Experimentation
Unilm
42.14 19.53 34.13 30.54 64.32 22.54
FASum
30.28 10.03 23.76 12.66 65.74 26.20
In this section, we first introduce the experimental settings, and then report the experimental results. Finally, we give the ablation study and analysis.
FASumpc
30.20 9.97 23.68 12.03 67.13 26.09
BART
45.52 22.48 37.29 34.83 65.45
22.98
CONSEQ
44.67 21.66 36.47
22.42
CLIFF
36.19
25,47
4.1 Experimental Settings
FACTEDIT
33.58 14.68 26.71 20.13
23.91
Following previous work (Pagnoni et al., 2021; Nan et al., 2021), we evaluate our DASum on the popular CNN/DM (Hermann et al., 2015) and XSUM (Narayan et al., 2018) datasets, and em- ploy ROUGE (Lin, 2004) evaluate the informa- tiveness of the summary and the automatic fac- tual evaluation metric: DAE (Goyal and Durrett, 2021), SENT (Goyal and Durrett, 2021) and FactC (Kryscinski et al., 2020) to evaluate the factuality. In the data processing phase, we set the maxi- mum number of tokens for the source document and gold summary of CNN/DM 1024 and 200, and 1024 and 80 for XSUM, to deal with the incon- sistency in the distribution of the summary lengths of the two datasets. In the training phase, or- der to fully utilize the advantages of abstractive summarization pre-training, we use facebook/bart- large-cnn and training parameter for the DASum model the CNN/DM dataset and XSUM dataset, respectively.
LASum
44.59 21.48 36.17 39.02 68.10
26.33
DASum 44.89 21.54 36.25 37.62 67.80 27.31
DASum(L) 44.49 21.06 35.74 39.74 69.15 27.43
Table I: ROUGE and factual evaluation scores on CNN/DM and XSUM, where R1, R2, RL refer ROUGE-I, ROUGE-2, ROUGE-L, respectively. DA- Sum takes BART as backbone, while DASum(L) takes trained LASum model as hackbone.
3) FASum (Zhu et al., 2021) builds fact graph by relation triples and feeds the graph representation to the decoder when training. Based FASum, Zhu et al. (2021) propose corrector FC modify errors in summary, represented by FASumfe here. 4) BART (Lewis et al., 2020) is an auto- regressive pre-trained model with denoising strat- egy.
the pre-
5) CONSEQ (Nan et al., 2021) divides the mnner--gne summries into positive and neg- ative set and improves factuality by maximizing the likelihood probability of positive instances and minimizing the likelihood probability negative instances.
4.2 Experimental Results
To verify the effectiveness of our DASum, we con- duct the following baselines.
1) BERTSum (Liu and Lapata, 2019) takes BERT the encoder and initializes the parameters of the decoder randomly.
6) CLIFF (Cao and Wang, 2021) also constructs a collection of positive and negative summaries, and utilizes comparative learning to enhance the factuality.
2) Unilm (Dong et al., 2019) employs special mask strategies for pre-training and applicable to both NLU and NLG task.
7) FACTEDIT (Balachandran et al., 2022) is

---

factual error correction model for summaries, which improves the factuality by modifying sum- maries.
CNN/DM
XSUM
Model
DAE SENT FactC DAE SENT FactC
Baseline
74.75 85.34 56. 16..163..10 16 5.03 23.47
8) LASum (Gao al., 2022a) employs adver- sarial learning to integrate the fact graph into the summarization model.
Cha 77.64 88.24 60.2437.66 62 67.80 27.31 Emb 76.15 87.72 37.56 66.12 27.45
+
+ignal
Table 2: The ablation results of factual signals.
Table shows the ROUGE and factual evalua- tion scores on two datasets. On CNN/DM, DASum has improved in both the informativeness evalu- ation metric ROUGE and the factual evaluation metric: DAE, SENT, and FactC.
The emergence of advanced instruction-tuni large language models (LL.ss, e.g.. ChatGPT !). has presented exciting possibilities for factuality- oriented abstractive summarization. Zhang et al. (2023) report that ChatGPT can achieve 28.00 (w/o knowledge extractor) and 36.00 (w/ knowledge extractor) in FactC and these figures are larger than ours. However, the informativeness metrics ROUGEs of ChatGPT are very lower than the ex- isting fine-tuning models and ours. It only obtains 21.92, 5.98 and 17.62 in R1, R2 and RL, respec- tively Zhang et al. (2023). The gap between the factual metrics and informativeness metrics show ChatGPT is still an preliminary model for abstrac- tive summarization. By contrast, our DASum can achieve both the high factual and informativeness values.
Compared with BART, our ROUGE value of DASum has increased by 0.34 on average, while the three factual evaluation metric has increased by 6.3 on average; On XSUM, DASum achieves comparable ROUGE scores. However, in terms of factual metrics, the introduction of the factual relation discrimination task helps improve the fac- tuality of summaries and outperforms BART in all three factual metrics. This indicates that the factual relation discrimination task is helpful for factuality- oriented abstractive summarization.
Compared with LASum, DASum(L)) also im- proves all the factual metrics on both the CNN/DM and XSUM datasets, especially on CNN/DM. This result also indicates that the factual relation discrim- ination task not only can help factuality oriented stractive summarization, but also has the generaliza- tion ability to boost different models of factuality- oriented abstractive summarization.
voctualityorrientedd ab-
4.3 Ablation Study
The factual signal ablation experiments are de- signed to test whether the factual signal has a guid- ing effect on the model's generation of factual sum- maries. The experimental results are presented in Table Baseline refers the DASum model with- out factual signals, while Cha refers to the model using special characters as factual signals and Emb refers to the model using embedding vectors as guidance signals. Specifically, when adding Posi- tion Embedding, a factual label embedding should also be added to indicate whether it is factually correct or not.
CONSEQ has very similar Rouge scores to our DASum variations. The reason for the similar Rouge scores is that both CONSEQ and our DA- Sum variations utilize Bart-large as the backbone. CONSEQ specifically employs the Fairseq imple- mentation of BART-large as the summarization model and fine-tunes the BART-large model on CNN/DM and XSUM datasets. On the other hand, and
our work makes use of bacebook/bart-large-ee facebook/bart-largee Darbarrttar--summ xsum as the pre-training pa- rameter for the DASum model on the CNN/DM and XSUM datasets, respectively. Moreover, in com- parison to CONSEQ, our DASum incorporates the factual relation discrimination task to enhance the factuality of abstractive summarization by address- ing diverse factual errors. Hence, our DASum can improve the factuality metrics FaCTSETDAE/SENT due to the factual relation discrimination and both CONSEQ and our DASum achieve the similar R1/R2/RL because both of them take Bart-large as the backbone.
Table 2 indicates that the two models, Cha and Emb, with the addition of factual signals, outper- form Baseline in terms of factuality, which proves that the factual signals can guide the model to gen- erate summarics with strong factuality. In addition, Cha outperforms Emb in factual metrics due to that Emb requires the model to learn the embedding of the label. Cha uses special characters as guiding signals, which is simpler and easier for the model to understand.
https///chatoppmaaaicomcccc

---

CNN/DM
CNN/DM
XSUM
20
16
14
15
13
9
9
75
10
70
40
PredE
EntE
CircE
CorefE
XSUM
40
30
30
0%
10%
30%
50%
70%
90%
30
23
25
19
Figure 3: The DAE scores under different proportions of negative samples.
20
13
10
1
PredE
EntE
CircE
CorefE
4.4 Proportion of Positive and Negative Instances
BART
DASum
The comparative experiments the proportion of positive and negative samples in the factual summa- rization dataset is used to verify the impact of the proportion of negative samples on model perfor- mance. The experimental results are shown Fig. 3, where the horizontal axis represents the propor- tion of negative samples the whole dataset, and the vertical axis represents the score of the DAE (the experiments on the other factual metrics show the similar results).
Figure The number of instances with different cate- gories of factual errors.
ated by DASum and BART.
Consistent with Gao et al. (2022a), the human evaluation is also conducted by three experts with NL.P experience, who independently provide fac- tual labels for each instance according to the cat- egory of factual errors defined by Pagnoni et al. (2021). Pagnoni et al. (2021) find that 69% factual errors are Semantic Frame errors (PredE, EntE, CircE) and Coreference error (CorefE). Se- mantic Frame errors mean the semantic frame of the generated summary sentence is wrong. Seman- tic frame consists of predicates and frame elements, which can be divided into core and non-core frame elements. The former is crucial for the mcaning of the semantic frame, such as subject and object, while the latter yields additional descriptive infor- mation, such as location and date. Predicate Error (PredE), Entity Error (EntE) and Circumstance Er- (CireE) indicates factual errors of predicates, core and non-core frame elements, respectively, Coreference error interprets pronouns and other types of references to previously mentioned enti- ties that are either incorrect or have no explicit antecedents, making them disputable. Specifically, the evaluator will first assess the factuality of the entire summary. If marked factually incorrect. they will determine the specific category of factual error.
We can find that as the proportion of negative samples in the factual summarization dataset in- creases, the DAE score tends to increase and then decrease. For the CNN/DM and XSUM datasets, the factual summarization dataset with negative sample proportion of 50% has the highest DAE score. When the negative proportion is less than 50%, the factuality of the model gradually im- proves, indicating that the construction of negative samples is beneficial for the model to enhance its understanding of facts; When the proportion of neg- ative samples exceeds 50%, the factuality of the model shows a downward trend. This is mainly because when there are too many negative samples, the factual guidance signal tends to be biased to- wards the negative sample side, and the guidance effect on the positive sample side becomes weaker.
4.5 Human Evaluation on Differrnt Errors
DASum enhances the factuality of summaries through factual relation discrimination. To further verify the effectiveness factual relation discrimi- nation, we manually evaluate the summaries gener-
We randomly select 100 samples from the CNN/DM and XSUM test sets for human eval-

---

London Welsh have announced contract extension for their former England international back Olly Barkley. The 33-year-old joined the Exiles last year, having previously played for Bath, Gloucester, Racing Metro, Grenoble and the Scarlets. He won 23 Test caps, the last of which against New Zealand in 2008
Article-1:
Target Summary: Olly Barkley has signed a contract extension at London Welsh. The Exiles were relegated from the Aviva Premiership this season. Fly half Barkley is keen to fire the Welsh back up to the top flight. CLICK HERE for all the latest rugby unions news.
BART: Olly Barkley has extended his stay at London Welsh.... ... Barkley won 23 England caps, the last of which in 2008.
DASum: Olly Barkley has extended his stay at London Welsh. The 33-year-old has previously played for Bath, Gloucester, Racing Metro, Grenoble and the Scarlets. Welsh are set to return to the Championship next season after finishing bottom of the Aviva Premiership.
Table 3: Example human evaluation.
uation. For CNN/DM, there are 30 instances with factual errors for BART, while 17 for DASum; For XSUM, there are 46 instances with factual errors for BART, while 37 for DASum.
Limitations
Although our proposed DASum model has made strides in improving factual accuracy, there remains a disparity between its information and that state- of-the-art models. This presents an area for future improvement. Additionally, in subsection 3.1, we strive to simulate the distribution of factual errors to artificially construct those with factual errors. However, these constrained construction strategies cannot always encompass all types of errors. There- fore, our proposed model cannot address all types of factual errors in summaries.
Fig 4 shows the different categories of factual errors contained in the summaries generated by BART and DASum, as well their corresponding numbers. We can see that DASum significantly al- leviates semantic framework errors and coreference errors. For example, on CNN/DM, the number of EntE decreases from 16 (BART) to 9 (DASum); On XSUM, the number of EntE decreased from 30 (BART) to 23 (DASum).
Table 3 shows an example of human evaluation, in which the generated summary of BART has en- tity error. It confuses "Test caps" with "England caps", while our DASum model generates the sum- mary with the right dependency relations.
Furthermore, solely relying on our factual rela- tion discrimination task is insufficient to effectively address all factual inconsistency issues. It may be beneficial to incorporate additional discrimina- tion tasks in order to enhance performance. Ad- ditionally, each assessment metric pertaining abstractive summarization possesses strengths and weaknesses. Thus, it is important to incorporate a wide range of automated evaluation metrics (e.g. SummAC for entailment and FactEval for QA) or propose more inclusive evaluation metrics.
5 Conclusion
In this paper, we employ the factual relation dis- crimination task deepen the model's understand- ing of factual errors and guide it to generate factual summaries. Specifically, we construct factual summarization dataset with positive and negative samples through data augmentation. Then, we train the model to identify factual samples, discover the locations and reasons for factual errors, and guide the model to generate summaries through factual guidance signals. Experimental results on CNN/DM and XSUM show that our DASum out- performs the baselines on factual metries. Our future work will focus on how to construct more accurate negative samples to further identify the fine-graing causes of factual errors.
Acknowledgements
The authors would like to thank the three anony- mous reviewers for their comments on this paper. This research supported by the National Natu- ral Science Foundation of China (Nos. 62276177. 61836007, and 62376181), and Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions (PAPD).

---

References
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequemn pre-training for natural language generation, translation, and com- prehension. In ACL., pages 7871-7888
Duygu Altinok. 2021. Mastering spaCy: An end-to-end practical guide implementing NLP applications using the Python ecosystem. Packt Publishing Ltd.
Vidhisha Balachandran, Hannaneh Hajishirzi, William Cohen, and Yulia Tsvetkov, 2022. Correcting diverse factual errors in abstractive summarization via post- editing and language model infilling. In EMNLP, pages 18-800
Chin-Yew Lin. 2004. Rouge: evaluation of summaries. In Text summarization branches out, pages 74-81.
A package for automatic
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In EMNLP, pages 3730 30 3740.
Meng Cao, Yue Dong, and Jackie Chi Cheung. 2021. Inspecting the factuality of hallucinated en- tities in abstractive summarization. arXiv preprint arXiv:2109.09784.
Feng Nan, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng, Kathleen McKeown, Ramesh Nallapati, Dejiao Zhang, Zhiguo Wang. Andrew O. Amold, and Bing Xiang. 2021. Improving factual consistency of abstractive summarization via question answering. In ACL, pages
Shuyang Cao and Lu Wang. 2021. CLIFF: Contrastive learning for improving faithfulness and factuality in abstractive summarization. In EMNLP, pages 66333 6649.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In AAAI, volume 32.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In EMNLP, pages 1797-1807.
Prafulla Kumar Choubey, Jesse Vig, Wenhao Liu, and Nazneen Fatema Rajani. 2021. MoFE: Mix- ture of factual experts for controlling hallucina- tions in abstractive summarization. arXiv preprint arXiv:21
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstrac- tive summarization with FRANK: A benchmark for factuality metrics. In NAACL, pages 4812 4829, On- line.
Li Dong, Nan Yang, Wenhui Wang. Furu Wei, Xi- aodong Liu, Yu Wang. Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Pro- cessing Systems, volume 32.
Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023. Summit: Iterative text summarization via ChatGPT. arXiv preprint 2rXiv:2305.144. 14835.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre- Pre-training with extracted gap- sentences for abstractive summarization. In ICML, pages 11328-11339.
Jinlan Fu, Xuanjing Huang, and Pengfei Liu. 2021. SpanNER: Named entity re-recognition as span pre- diction. In ACL, pages 7183-7195.
Zhiguang Gao, Feng Jiang. Xiaomin Chu, and Peifeng Li. 2022a. Adversarial fine-grained fact graph for factuality-orientedl abstractive summarization. In NLPCC, pages 339-351.
Chenguang Zhu, William Hinthorn. Ruochen Xu. Qingkai Zeng, Michael Zeng. Xuedong Huang, and Meng Jiang. 2021. Enhancing factual consistency of abstractive summarization. In NAACL, pages 718- 733, Online.
Zhiguang Gao, Feng Jiang, Xiaomin Chu, and Peifeng Li. 2022b. Employing internal and external knowl- edge to factuality-orrentedd abstractive summarization. In NLPCC, pages 797-809.
Tanya Goyal and Greg Durrett. 2021. Annotating and modeling fine-grained factuality in summarization. In NAACL, pages 1462, Online.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette. Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances neural information processing systems, 18.11.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In EMNLP, pages 9332-9346.


Reviews:
[{"review_id": "uZuSGiykVP", "reviewer": "Reviewer_FEBY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable comments.\n\n**Q1: What are the main differences compared with CLIFF (2021)?**\n\n**A1**: The primary differences between CLIFF and our work stem from two aspects. First, while CLIFF only takes into account entity substitution when building factual summarization datasets, we not only consider entity substitution, but also carry out pronoun substitution to address coreference errors and perform sentence negation to enhance the model's comprehension of sentence polarities. Additionally, both CLIFF and our work employ the contrastive approach. The distinction lies in the fact that while CLIFF aims to maximize the cosine similarity of positive pairs, our approach assigns higher probabilities to positive samples and lower probabilities to negative samples, which is more intuitive and facilitates model learning. Finally, the human evaluation on different errors shows that our model DASum can reduce coreference, circumstance and predicate errors, in comparison with CLIFF.\n\n**Q2: More discrimination tasks can be introduced to further improve the performance.**\n\n**A2**: Yes. There are several kinds of factual errors found in abstractive summaries, such as entity errors, entity relation errors, predicate errors, coreference errors, and commonsense errors. To improve the factuality of abstractive summarization, it is recommended to include additional discrimination tasks. Currently, most existing methods concentrate on only a few kinds of factual errors. In our paper, we present the factual relation discrimination task to tackle five types of factual errors, including pronoun errors, sentence negation errors, time and date errors, quantifier errors, and entity errors. \n\nBesides, the term \"relation\" in \"factual relation discrimination\" refers to the use of the dependency relation to model the aforementioned factual errors, rather than solely focusing on relation errors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method is clearly explained and is easy to follow.\n- Comprehensive experiments and significant improvements compared with other counterparts.", "reasons_to_reject": "- I think the novelty of constructing factual summarization dataset seems a little limited. Similar construction methods have already been proposed in other factual summarization works (e.g., CLIFF). What are the main differences compared with those works?\n- I think the designed factual relation discrimination task alone is not enough to solve the factual inconsistency problem well, and the factuality problem in summarization not only contains the incorrect factual relations. More discrimination tasks can be introduced to further improve the performance.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a factuality-oriented abstractive summarization model based on a new task factual relation discrimination to mine the causes of factual errors. A factual summarization dataset is first built, and then a factual relation discrimination task is proposed to improve the factuality of dependency relations in summaries. Experiments demonstrate the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tG4ALcFIYZ", "reviewer": "Reviewer_QpAJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable comments.\n\n**Q1: It would be important to include more diverse automated evaluation metrics like SummAC (entailment) or FactEval (QA).**\n\n**A1**: Yes. A wider variety of automated evaluation metrics can provide a more comprehensive evaluation of the proposed model. However, most previous studies reported FactCC, DAE, and SENT, while a few reported only FactCC (e.g., CONSEQ, CLIFF, and FACTEDIT). For comparison with previous studies, we report the above three metrics.\n\n**Q2: The method chosen to obtain feature representations of the relations seems overly complex and no ablation is reported.**\n\n**A2**: In Subsection 3.2, only the boundary embedding and length embedding are introduced for representing factual relations. BART is used to produce the boundary embedding, which is superior to other options such as BERT and RoBERTa. The length embedding is determined by the length of dependent and core words. Our experimental results indicate that the enhancements primarily arise from the boundary embedding. Using BERT/RoBERTa/XLNET instead of BART results in significant drops in various metrics, including FactCC, DAE, and SENT (by about 2.0-5.0). In addition, removing the length embedding results in a decrease of 0.3 for FactCC, DAE, and a decrease of 1.4 for SENT.\n\n**Q3: Lacking a more thorough limitations section.**\n\n**A3**: We revised this section as follows:\n\nAlthough our proposed DASum model has made strides in improving factual accuracy, there remains a disparity between its information and that of state-of-the-art models. This presents an area for future improvement. Additionally, in subsection 3.1, we strive to simulate the distribution of factual errors to artificially construct those with factual errors. However, these constrained construction strategies cannot always encompass all types of errors. Therefore, our proposed model cannot address all types of factual errors in summaries. Furthermore, solely relying on our factual relation discrimination task is insufficient to effectively address all factual inconsistency issues. It may be beneficial to incorporate additional discrimination tasks in order to enhance performance. Additionally, each assessment metric pertaining to abstractive summarization possesses strengths and weaknesses. Thus, it is important to incorporate a wide range of automated evaluation metrics (e.g. SummAC for entailment and FactEval for QA) or propose more inclusive evaluation metrics.\n\n**Q4: The improvements in terms of factuality are slightly lower than LASum.**\n\n**A4**: We have evaluated the effectiveness of our proposed model, DASum and DASum(L), using BART and LASum as backbones. DASum outperforms BART, improving FactCC, DAE, and SENT by 10.64, 4.89, and 3.30 on CNN/DM and 4.33, 2.79, and 2.35 on XSUM, respectively. Additionally, DASum(L) outperforms LASum, improving FactCC, DAE, and SENT by 0.52, 1.65, and 2.82 on CNN/DM and 1.10, 0.72, and 1.05 on XSUM, respectively.\n\n**Q5: Like in other data augmentations methods introducing factual errors in summaries (ex: FactCC), there are limitations in the type of errors that can be modeled. These tend to lack the distribution of naturally occurring errors.**\n\n**A5**: There are numerous categories of factual errors present in abstractive summaries, including entity errors, entity relation errors, predicate errors, coreference errors, and commonsense errors. It would be advantageous for a model to address all of these types of errors, rather than focusing solely on a few. Presently, nearly all current methods concentrate on only a limited number of factual errors. \nAs explained in Appendix A, Pagnoni et al. (2021) classify factual errors and find that 69% of these errors consist of predicate, entity, circumstance, and coreference errors. Therefore, we present the factual relation discrimination task to tackle five types of factual errors: pronoun errors, sentence negation errors, time and date errors, quantifier errors, and entity errors. This improves the accuracy of abstractive summaries.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The problem of factuality is a practical concern of abstractive summarization systems and this work provides a novel approach to improve the factuality of abstractive summarization systems. This approach can be easily incorporated into new autoregressive models. The evaluation is thorough (both automatic and human).\n", "reasons_to_reject": "- It would be important to include more diverse automated evaluation metrics like SummAC (entailment) or FactEval (QA). DAE and FactCC might are similar in spirit to the training paradigm in this paper and there could be confounding factors. \n- The method chosen to obtain feature representations of the relations seems overly complex and no ablation is reported. \n- Like in other data augmentations methods introducing factual errors in summaries (ex: FactCC), there are limitations in the type of errors that can be modeled. These tend to lack the distribution of naturally occurring errors.\n- The improvements in terms of factuality are slightly lower than LASum. \n- Lacking a more thorough limitations section.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new method for factuality oriented abstractive summarization.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Please restructure the section where you present the types of errors introduced as the format does not make it easy to understand which types of errors are considered. The term \"swapped\" is also confusing. \n\nPlease ensure the paper goes through grammar and spell checking."}}, {"review_id": "4vOKp72CSs", "reviewer": "Reviewer_FTBC", "summary": "", "questions": "L.073 mtrics => metrics\n\nTable 1, part 1: CONSEQ has very similar Rouge scores to DASum variations. Is there an explanation for this, maybe an intuitive one? What are the similarities between the two methods that cause this phenomenon?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable comments.\n\n**Q1: Some explanation about the results would be nice - no intuition of why the method works better is provided.**\n\n**A1**: In this paper, we use data augmentation methods to introduce negative samples to the task of abstractive summarization. Specifically, we use five operations (i.e., pronoun swapping, sentence negation, time and date entity swapping, quantifier swapping, and named entity swapping) to construct counterfactual summaries. These negative samples with the error causes (i.e., the above five operations) can help the factual relation discrimination to identify counterfactual relations and then guide the abstractive summarization model to generate factual summaries.\n\nIn the section “Human Evaluation on Different Errors”, we randomly select 100 samples from the CNN/DM and XSUM test sets for human evaluation. For CNN/DM, BART has 30 instances of factual errors while DASum has 17; for XSUM, BART has 46 instances of factual errors while DASum has 37. As shown in Fig.4, DASum can considerably reduce Predicate Error (PredE), Entity Error (EntE), and Circumstance Error (CircE). These results confirm that the factual relation discrimination is effective for abstractive summarization.\n\n**Q2: Table 1, part 1: CONSEQ has very similar Rouge scores to DASum variations. Is there an explanation for this, maybe an intuitive one? What are the similarities between the two methods that cause this phenomenon?**\n\n**A2**: The reason for the similar Rouge scores is that CONSEQ and our DASum both utilize Bart-large as the backbone. CONSEQ specifically employs the Fairseq implementation of BART-large as the summarization model and fine-tunes the BART-large model on CNN/DM and XSUM datasets. On the other hand, our work makes use of facebook/bart-large-cnn and facebook/bart-large-xsum as the pre-training parameter for the DASum model on the CNN/DM and XSUM datasets, respectively. Moreover, in comparison to CONSEQ, our DASum incorporates the factual relation discrimination task to enhance the factuality of abstractive summarization by addressing diverse factual errors. Hence, our DASum can improve the factuality metrics FactCC/DAE/SENT due to the factual relation discrimination and both CONSEQ and our DASum achieve the similar R1/R2/RL because both of them take Bart-large as the backbone.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea is nice and well described, and the results are slightly above current state-of-the-art.", "reasons_to_reject": "Some explanation about the results would be nice - no intuition of why the method works better is provided.", "questions_to_authors": "L.073 mtrics => metrics\n\nTable 1, part 1: CONSEQ has very similar Rouge scores to DASum variations. Is there an explanation for this, maybe an intuitive one? What are the similarities between the two methods that cause this phenomenon?", "paper_topic_and_main_contributions": "In order to identify the root causes of factual errors, the paper suggests an abstractive summarization model that is factuality-oriented and is based on factual relation discrimination. The authors provide negative examples of summaries using data augmentation techniques and create a dataset for factual summarizing. Following that, students complete the factual relation discrimination task, which assesses the veracity of the dependencies in summaries during summary generation, enhancing the factuality of summaries.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None that I can see.", "Typos_Grammar_Style_and_Presentation_Improvements": "There are some typos. Please proof-read the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a factuality-oriented abstractive summarization model DASum. The model is based on the factual relation discrimination task, guiding DASum to generate true factual relations. Also, the authors construct a new factual summarization dataset using data augmentation to generate negative samples. Experimental results on two datasets demonstrate the DASum's superiority over several SOTA benchmarks on factual metrics. 

The reviewers mentioned several pros of this work, such as:
1. The proposed method is novel,
2. The method is clearly explained and is easy to follow,
3. Comprehensive experiments show significant improvements compared with other counterparts.
4. The evaluation is thorough and contains both automatic and human parts.

I also want to mention the contribution to the data resources for the task of factual summarization. 

The authors addressed all comments (including missing evaluation metrics, motivating the choice of augmentation methods, lacking a more thorough limitations section, etc.) in their rebuttal.
Decision: Accept-Findings