FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge
Authors: Shangbin Feng
Vidhisha Balachandran
Yuyang Bai
Yulia Tsvetkov
Venue: EMNLP 2023 Main (2023)
Keywords: factuality evaluation
knowledge bases
summarization

Abstract: Evaluating the factual consistency of automatically generated summaries is essential for the progress and adoption of reliable summarization systems. Despite recent advances, existing factuality evaluation models are not robust, being especially prone to entity and relation errors in new domains. We propose FactKB---a simple new approach to factuality evaluation that is generalizable across domains, in particular with respect to entities and relations. FactKB is based on language models pretrained using facts extracted from external knowledge bases. We introduce three types of complementary factuality pretraining objectives based on entity-specific facts, facts extracted from auxiliary knowledge about entities, and facts constructed compositionally through knowledge base walks. 
The resulting factuality evaluation model achieves state-of-the-art performance on two in-domain news summarization benchmarks as well as on three out-of-domain scientific literature datasets. Further analysis of FactKB shows improved ability to detect erroneous entities and relations in summaries and is robust and easily generalizable across domains.

FACTKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge
Shangbin Feng'
Vidhisha Balachandran
Yuyang Bai
Yulia Tsvetkov'
!University of Washington Carnegie Mellon University
X an Jiaotong University
(shangbin, yuliats]ecs washington.edu vbalacha@cs.c cmu.edu 12869446338stu. xjtu. edu.cm
Abstract
Article:
The stone got past elephant's fence and a ditch separating animal visitors... Scientific Director of the Amboseli Trust Elephants, says that targeted throwing of stones and branches by elephants very unusual. 'It can happen when elephants are frustrated or bored... The moments after the girl was struck at Rabat Zoo on Tuesday were filmed by bystander and uploaded onto YouTube
Evaluating the factual consistency of automati- cally generated summaries is essential for the progress and adoption of reliable summariza- tion systems. Despite recent advances, exist- ing factuality evaluation models are not ro- bust, being especially prone to entity and re- lation errors in new domains. We propose FAC- TKB-a simple new approach to factuality evaluation that is generalizable across domains, particular with respect to entities and re- lations. FACTKB is based language mod- els pretrained using facts extracted from exter- nal knowledge bases. We introduce three types of complementary factuality pretraining objec- tives based on entity-specific facts, facts ex- tracted from auxiliary knowledge about entities, and facts constructed compositionally through knowledge base walks. The resulting factual- ity evaluation model achieves state-of-the-art performance on two in- nn-domain news summa- rization benchmarks as well as on three out- of-domain scientific literature datasets. Further analysis of FACTKB shows improved ability to detect erroneous entities and relations sum- maries and is robust and casily generalizable across domains. Code and data are available at https://github.co httieidb..ooomuuuuseeeeg//oi/iie
Summary:
Gold: Not Factual
elephant has been by stone western France after it was hit by tree.
QAGS: Factual
DAE: Factual
FactCC: Factual
Figure 1: Existing factuality models struggle to identify semantic frume errory encompassing entities and rela- tions. In the example. they fail identify error in the generated summary about who was hit by the stone.
classifiers are thus not robust to ever-growing infor- mation, in which the distribution of entities, events, and their relations changes greatly across time and domains (Elsahar and Gallé, 2019; Laparra et al., 2020). Pagnoni et al. (2021) highlighted this limi- tation, finding that over 50% of factuality errors in the XSUM (Narayan et al., 2018) summarization dataset stem from semantic frame errors, namely entities, events, and relations between them, as il- lustrated in Figure 1.
1 Introduction
Generating factually accurate document summaries in addition to fluent and informative ones critical to the adoption of summarization models (Kryš- ciñski et al., 2020; Goyal and Durrett, 2020). How- ever, evaluating the factual consistency of sum- maries is still challenging, especially in special- ized domains like scientific or legal (Cachola et al., 2020; Goldsack et al., 2022; Polsley et al., 2016; Kanapala et al., 2019). The key reason is that the majority of existing approaches employ neural clas- sifiers trained on synthetic data constructed from a relatively small set of documents (Kryšcinski et al., 2020; Goyal and Durrett, 2020). These factuality
To address these issues, we develop a new fac- tuality evaluation model with improved factual knowledge representation, specifically focusing entities and relations. Entity-oriented pretraining objectives have been shown to improve QA and reasoning tasks (Yasunaga et al., 2022: Liu et al., 2022b); we thus hypothesize that similar objectives can aid factuality evaluation in better detecting se- mantic frame errors in generated summaries.
We propose FACTKB, B, a novel factuality eval- uation model built upon language models (LMs) augmented with factual knowledge ($2). The LMs

---

abroor Miche NEREERS
Kaptar eEUUSS
Plet
de Sook wnter Servean
Boss
Base
pildpedia
Bise
Tarpe
Juhurno Kapki we ligut Thttoeerryy ava plane planstyy mosion-
Entity Wiki
Evidence Extraction
Knowledge Walk
Eeyler howiir the der So -80454 a countiy Europe.
Factuality Evaluation Model
FactCollect
Language Model
Fact-Enhanced LM
FactKB
Factuelity Pretraining
Figure 2: Overview of FACTKB. FACTKB pretrains LMs using three entity-centric pretraining strategies improve fact representations. The objectives are designed to fill masked entities/relations in KB facts using i) Entity Wiki- direct facts about entities ii) Evidence Extraction auxiliary knowledge about entities and iii) Knowledge Walk compositional knowledge from the KB. The pretrained LMs then fine-tuned for robust factuality evaluation.
are pretrained with knowledge-focused objectives using text synthesized from external knowledge bases (KBs) which store high-quality facts about entities and relations. We propose three types of complementary pretraining strategies: (1) entity wiki, with a focus on improving entity understand- ing; (2) evidence extraction, with focus in- corporating supporting evidence from surrounding context; and (3) knowledge walks, with focus on augmenting compositional reasoning about en- tities. For factuality evaluation, we first pretrain a language model using these three entity-centric pre- training strategies, and then fine-tune the enhanced LM on factual error detection dataset.
FACTKB Methodology
FACTKB aims improve the robustness and gen- eralizability of factuality evaluation by a simple factuality pretraining, which improves entity and relation representations in LMs. We first propose three pretraining strategies ($2.1). We then describe the training process to (1) pretrain an LM using the proposed strategies and (2) fine-tune the fact- enhanced LM factuality error detection dataset, resulting in FACTKB ($2.2). Figure 2 presents overview of our approach.
2.1 Factuality Pretraining
Knowledge bases are rich reservoirs of facts about entities and relations (Vrandečié and Krötzsch, 2014; Pellissier Tanon et al., 2020), and we ex- plore the possibility of leveraging external KBs as "fact teachers" to enhance an LM's representation of entities and relations.
We evaluate FACTKB's 's correlation with human factuality judgments across three settings ($3). In in-domain (news) summarization, FACTKB sig- nificantly outperforms baselines by 2-7 balanced accuracy (BACC) points on the FactCollect dataset (Ribeiro et al., 2022) and 10-12 correlation points the FRANK benchmark (Pagnoni et al., 2021), particularly showing marked improvements in se- mantic frame errors. out-o--domain experiments, FACTKB consistently outperforms existing ap- proaches by 3-5 BACC points on three datasets in biomedical and scientific domains (Saakyan et al., 2021: Sarrouti et al., 2021; Wadden et al., 2020), demonstrating stronger generalizability to unseen documents in new domains. Further analysis shows that FACTKB compatible with different LMs and KBs while presenting lightweight and easy-to-use approach factuality evaluation. Code, data, and trained factuality evaluation models are publicly available.
(E, A, where E {e fe eNr represents the entities in the KB. R = (ri...., r'M) denotes the relations in the KB, A denotes the adjacency matrix where aij = k in- dicates relation 7'k connecting entities e; and ej (e,, Tk, ej) KB, e(-) : E + str and pp-)) R + str map the entities and relations to their textual names. We propose three novel types of factuality pretraining strategies that leverage the KB.
Let KB
Strategy 1: Entity Wiki Entities in KBs often have multiple edges connecting them other enti- ties via relations, each representing a distinct but related fact about the entity. Inspired by the task of knowledge base completion (Bordes et al., 2013; Vashishth et al., 2019) to predict missing connec-

---

Factuality Pretraining
Corpus Size Bound
# Tokens
Example
ENTITY WIKI
x |8]
5.4M
Johannes Kepler is horn in Itall, Jobannes
Kepler is [MASK], [SEP] Johannes Ke- pler the author
EVIDENCE EXTRACTION
0c ||A||o
12.2M
Hillary Clinton purty atfiliation [MASK] Hillary Diane Rodham Clinton is an Amer- ican politician. Member the Demo-
Party, she
KNOWLEDGE WALK
[E]( | y
2.7M
University of Edinburgh located in Scot- land located [MASK]
Table 1: Summary of the three factuality pretraining strategies.
tions in KBs based on available KB facts, we pro- pose the entity wiki factuality pretraining, where an LM is pretrained with the task of predicting masked entities relations in KB facts. Specifically, for each entity G &, retrieve its one-hop neigh- borhood in the KB as & {ej | E r s.t. aij k). We then synthesize a sentence using entity 6 and its connected one-hop facts:
the To norm of the adjacency matrix ||/4/o- We use this corpus for the evidence extraction factuality pretraining and train the LM to predict the mask by using relevant evidence in the auxiliary paragraph. Through this, aim to augment FACTKB's ' ability to implicitly select evidence from the document to support its factuality evaluation.
Strategy 3: Knowledge Walk Natural language documents often include compositional statements about entities and relations (Feldman and El- Yaniv, 2019; Wang and Pan, 2022), but pretrained LMs struggle with such compositional reasoning (Press et al., 2022). To improve FACTKB's abil- ity to understand multi-hop claims, we propose the knowledge walk factuality pretraining strat- egy. Specifically, we randomly select a starting en- tity c(o) and randomly select an entity e(1) from its direct neighborhood Eggoy resulting in one- hop triple {e(o), T(0.1), (()) where r(0,1) denotes the relation between e(g) and e(1). Now, from C(1), we randomly select an entity from it's di- reet neighborhood to take the next step. We re- peat this process for K. times, and obtain a K-- hop random walk of triples beginning at e(o): {E(o), "(o,1): E(I), "., T(c--, K)), EOC) J. We then produce a sentence based on the KC-hop walk:
d, concatu,es., [e(e:))((rxlo); = [S{(]]
where concat denotes string concatenation and [SEP] denotes the special token. Repeating this generation process for all E E, we produce corpus of entity facts as { {d,) !8] with the max size being the number of entities ( We use this entity wiki corpus to pretrain an LM for better factual rea- soning by randomly masking entities and relations in it and training the LM to predict the mask given the surrounding facts about an entity. We randomly mask the corpora with probability p and pretrain LMs with the masked language modeling objective. We expect this objective to train LMS to infer facts from surrounding knowledge and penalize unsup- ported hallucinations about entities and relations.
Strategy 2: Evidence Extraction The goal of this pretraining strategy is to enhance the model's s ability to evaluate facts based on relevant evi- dence. We begin by randomly selecting triple (ee, rke eÿ) E KB and use the first paragraph of the Wikipedia deseription of e; as the auxiliary knowl- edge. We synthesize a sentence using the two as:
d, = (((o)) concat K-1 (+(((s+1) d(((+)) ]
Repeating this KC-hop walk N times with differ- ent randomly selected starting entities, we obtain [ as the corpus for the knowledge walk fac- tuality pretraining, whose size is bounded by the number of all possible K--hop walks as |&1( [Lel|l In this corpus, tions in each group of facts with probability p and train an LM to predict the masked element using the compositional facts around using the masked language model objective. Through this pretraining, we expect FACTKB to improve in compositional
dj E(ei) ((rk) [MASK] Wikipediaaes)
where we mask out <(ez) and [MASK] denotes the special token, Wikipedia(-) E str maps entities the first paragraph of their Wikipedia de- scription. Repeating this process N times with ran- domly selected triples, we obtain a corpus of triples paired with auxiliary knowledge [d The cor- pus size is bounded by all KB triples represented as
El ya. randomly mask entities or rela-

---

All Data
CNN/DM
XSUM
Model
BACC
BACC
F1
BACC
QAGS
79.8
79.7
64.2
76.2
59.3
85.2
QUALS
78.3
78.5
60.8
76.2
57.5
82.2
ROBERTA
76.1
76.5
62.5
76.2
62.1
78.3
FALSESUM
78.9
78.2
53.7
34.6
G1.1
64.3
FALSESUM+
84.2
83.7
64.2
77.1
67.4
82.1
SUMMAC
86.6
86.2
75.4
83.5
71-9
9U.4
FACTCC
76.0
76.3
69.0
77.8
55.9
73.9
FACTCC+
83.9 (#00.4)
84.2 (40044
68.0 (+11.))
83.7 (+-0.5)
58.3 (4:2.2
84.0 (+1.0)
FACTGRAPH
86.3 (#11.3)
86.7 (+1.1)
73.0 (4:2.)
86.8 (+-0.8)
68.6 (4:2.3)
86.6 (+22.0)
FACTGRAPI-ADAPTERR
87.6 (#:0.7)
87.8 (#00.7
76.0 (4:2.8)
87.5 (#:0.4
69.9 (4:2.8)
88.4 (#:1.2
FACTKB-WII
89.3 (=:0.4)
89.5 (=00.))
77.3 (40.3))
88.2 (#0.6))
77.3 (::1.3)"
91.8 (+:1.))
FACTKи B-EVIDENCE
89.4 (4:0.2)
89.5 (+0.3))
77.7 (4:14)
87.9 (4-0.7)
76.8 (#11.))
90.8 (+-08))
FACTKB-WAR
89.1 (=:04))
89.3 (#00.5)
78.3 (=1.2)
87.7 (40.4)
76.4 (#:0.3)
90.4 (+1.4)
Table Performance FACTKB on the FactCollect dataset. We report average performance and standard deviation across random seeds. Best performance is shown in bold, while * indicates statistical significance. FACTKB significantly outperforms existing factuality evaluation approaches on in-domain evaluation.
fact understanding about entities and relations ap- pearing in the summary and the input document. We briefly summarize the three factuality pre- training strategies and provide examples Table I,
2020; Pagnoni et al., 2021) and consolidates them into single dataset. It mainly focuses the news media domain, covering summaries and ticles from CNN, Daily Mail, and BBC. FactCol- lect follows binary classification setting where each (SUMMARY, ARTICLE) pair has a FACTUAL or NON-FACTUAL label. We present more details about the FactCollect dataset in Appendix C.
2.2 FACTKB Training
We initialize FACTKB with encoder-based LMs and pretrain FACTKB separately with each of the three factuality pretraining corpora using the masked language modeling objective to study the effectiveness of each strategy. This results in fact- enhanced LMs with the ability better represent facts, entities, and relations. Finally, we fine-tune FACTKB on human--nnotatee factual error detec- tion datasets with the sequence classification set- ting, taking SUMMARY [SEP] DOCUMENT as input and produce FACTUAL or NON-FACTUAL. labels. The [CLS] token is adopted for classification. As entailment-basee factuality evaluation model that classifies machine- generated summaries as factual or non-factual.
We use ROBERTA-BASH (Liu et al., 2019) checkpoint and continue pretraining sepa- rately on each of the three factuality pretraining corpora. We discuss FACTKB's s compatibility with different LM initializations in Section $5.2. We
Settings
assign corpus size parameter N 1e5, masking probability p 0. 0.15, and knowledge walk length K. in the experiments, while we discuss the effect of corpus size and knowledge walk length in Appendix 5.4. We use a learning rate of 2e 5 for pretraining, 1e 4 for fine-tuning, a batch size of 32, and the RAdam optimizer. Pretraining is conducted for 5 epochs and fine-tuning has 50 max- imum epochs with early stopping. More hyperpa- rameter settings presented in Appendix 3.1.
result, obtain FACTKB,
3 Data and Experiment Settings
3.1 Training
Data We use YAGO (Pellissier Tanon et al., 2020), an encyclopedic knowledge base based on Wikidata (Vrandečić and Krötzsch, 2014), to con- struct the three types of factuality pretraining cor- pora, while discuss FACTKB's ' compatibility with different KBs in Section 5.2. For finetuning, we use the FactCollect dataset (Ribeiro et al., 2022), a dataset for factual error detection that gathers human annotations from different sources (Wang et al., 2020a; Kryšcińási et al., 2020; Maynez et al.,
We propose to further pre- train LM checkpoints with three types of factuality pretraining and fine-tune on factuality evaluation datasets. We present hyperparameters for the pre- training and fine-tuning stage in Table 4. We mostly follow the hyperparameters in Gururangan et al. (2020) for the pretraining stage. The default hy- perparameters on Huggingface Transformers are adopted if not included in Table
Hyperparameters

---

All Data
CNN/DM
XSUM
Model
p
p-val
p-val
p-val
p-val
p
p-val
p-val
QAGS
.22
.00
.23
.00
.34
.00
.27
.00
.07
.05
.06
.09
QUALS
.22
.00
.19
.00
.31
.00
.27
.00
.14
.00
.07
.03
DAE
.17
.20
.00
.27
.00
.22
.00
.03
.38
.33
.00
ROBERTA
.35
.00
.00
.31
.00
.23
.15
.00
FALSESUM
.05
.00
.04
.11
.07
.05
.07
.03
.04
.28
.04
.35
FALSESUM+
.22
.00
.26
.00
.27
.00
.33
.00
.24
.00
.27
.00
SUMMAC
.33
.00
.35
.00
.42
.00
.36
.00
.24
.00
.25
.00
FACTCC
-20
.29
.00
.00
.30
.06
.07
.19
.00
FACTCC+
.32
.38
.00
.00
.28
.24
.00
FACTGRAPH
.35
.00
.42
.00
.45
.00
.34
.00
.30
.00
.49
.00
FACTKB- WIKI
.46
.52
.00
.00
.49
.29
.39
.00
.43
.00
.49
.00
.53
.00
.45
.00
.31
.00
.37
.00
FACTKB-WALK
.47
.00
.52
.00
.57
.00
.45
.00
.35
.00
.36
.00
Table Correlation of FACTKB with human judgments of factuality on the FRANK benchmark. Best performance
is shown in bold. FACTKB has the highest correlation with human judgments across five of the six settings.
Pretraining Stage
Fine-Tuning Stage
on the FactCollect dataset without the FRANK sub- Value
Hyperparameter
Value
Hyperparameter
set for the FRANK evaluation.
LEARNING RATE
2c-5
LEARNING RATE
1e-1
WEIGHT DECAY
le-5
WEIGHT DECAY
le-5
Generalizable Factuality Evaluation Summa-
EPOCHS
50
BATCH SIZE
32
BATCH SIZE
32
rization systems are used in diverse domains in the real world, including but not limited to news media (Liu et al., 2022e; Eyal et al., 2019; Li et al., 2016), social media (Syed et al.. 2019; Kano al., 2018: He al., 2020), and scientific literature (Cachola et al., 2020; Lev et al., 2019). Consequently, factu- ality metrics should also provide reliable factuality scores the face of shifting domains. To study this, we perform an out-of- evaluation using un- seen documents and summaries from the scientific domain. To establish test bed for generalizable factuality evaluation, make use of three datasets in the scientific literature domain:
OPTIMIZER
ADAM
OPTIMIZER
RADAM
EPSILON
ADAM BETA
0.9. 0.98
WARMUP RATIO
0.06
EVIDENCE:
WALK: N
le5
WALK:
Table 4: Hyperparameter settings of FACTKB.
3.2 Evaluation
To study the robustness of FACTKB, we perform both in-domain and out-of-domain evaluation.
In-Domain Evaluation Since most research and resources on summarization and factuality are in the news media domain, we leverage the FactCol- lect dataset (Ribeiro et al., 2022) and the FRANK benchmark (Pagnoni et al., 2021) for in-domain factuality evaluation. We evaluate FACTKB on the held-out test set of the FactCollect dataset. FRANK (Pagnoni et al., 2021) is factuality eval- uation benchmark with human judgments on the factual consistency of model-generated summaries collected across 9 summarization models along with human annotations on the category of factual errors. Following the FRANK benchmark guide- lines, use two correlation measures (Pearson (Benesty et al., 2009) and Spearman (Myers and Sirois, 2004)). We present more details about the FRANK benchmark in Appendix C. Following pre- vious work (Ribeiro et al., 2022), we train FACTKB
CovidFact (Saakyan et al., 2021) collects claims from the r/COVID19 subreddit and verifies them against relevant scientific literature and Google search results, resulting in a binary classification setting that is similar to the FactCollect dataset.
Health Ver (Sarrouti et al., 2021) consists claims sourced from TREC-COVID (Voorhees et al., 2021) 1) and verified against the CORD-19 (Wang et al., 2020b) corpus. While Health Ver originally follows threc-way classification set- ting (SUPPORT, REFUTE, NOT ENOUGH INFOR- MATION), remove the examples in the "NoT ENOUGH INFORMATION, category evaluate models as they are trained on the binary classifi- cation setting (factual, non-factual..

---

CovidFact
Health Ver
SciFact
Model
BACC
F1
BACC
F1
BACC
F1
RANDOM
52.7
41.3
49.0
57.5
FACTCC
52.3
49.2
51.9
42.7
45.9
FACTCC+
51.1
50.5
49.5
51.6
48.6
55.2
FACTGRAPH
57.6
53.5
55.1
24.3
61.0
42.2
FACTGRAPH-EDEE
50.6
48.4
50.6
53.5
56.7
68.2
FALSESUM
50.6
41.6
56.8
51.2
45.7
65.3
FALSESUM+
50.1
41.2
37.3
51.6
51.9
65.4
SUMMAC
57.6
53.4
52.5
41.2
59.8
46.9
ROBERTA
59.0
46.4 (2:4.3
55.0 (12.2)
50.0 (13.9)
58.1 (1:40)
71.3 (1335)
FACTKB- WIKI
64.8 (=0.3)"
54.4 (#0.7))
60.1 (#0.4)"
71.6 (#2.9))
62.9 (#0.4)"
72.3 (#1.1))
FACTKB-EVIDENCE
63.9 (#:0.6)"
53.3 (=1.7
59.0 (+1)0))
70.8 (+-0.))
61.4 (+0.5)
74.1 (#16))
FACTKB- WALK
63.7 (#1.0)"
53.1 (=1.6)
58.5 (#0.5))
68.7 (#1.7)
63.1 (#:.1)
67.6 (#111
Table Performance of FACTKB out-of-domain scientific datasets. We report average performance and standard deviation across 5 random seeds. Best performance is shown in bold, while indicates statistical significance. FACTKB exhibits better generalization to new domains across three datasets.
SciFact (Wadden et al., 2020) includes claims sourced from citation sentences in biomedical literature and verified against the cited paper's abstract. While SciFact uses three-way classifi- cation that includes "NOT ENOUGH INFORMA- TION", we similarly remove them in this work.
tire held-out test data, the CNN/DM subset and the XSUM (BBC) subset, and report balanced ac- curacy scores and miero Fl scores. We each method five times with different random seeds and report the average performance as well as the stan- dard deviation. Table demonstrates that FACTKB significantly (*) outperforms all bascline factuality evaluation methods by 3.8 BACC points on aver- age across the three dataset settings. This demon- strates that the introduction of KBs and factuality pretraining is beneficial for factuality evaluation. Among the three factuality pretraining strategies, all of them outperform baseline models, suggesting that FACTKB's general methodology compatible with different types of KB utilization.
We leverage the well-organized version of the three datasets in Wadden et al. (2022). We train and validate FACTKB with the FactCollect dataset from the news domain and evaluate on the test set of these datasets for zero-shot transfer learning.
Baselines We compare FACTKB with differ- ent types of existing factuality evaluation mod- cls: QAGS (Wang et al., 2020a), QUALS (Nan et al., 2021), DAE (Goyal and Durrett, 2020), FalseSum (Utama et al., 2022), SummaC (Laban et al., 2022), FactCC (Kryściáski et al., 2020), and FactGraph (Ribeiro et al., 2022). Since training data is key factor in factuality evaluation mod- els and they are often used off-the-shelf, we in- clude factuality evaluation measures trained on both synthetic data (QAGS, QUALS, DAE, Sum- maC, FalseSum, FactCC) and human-annotated data (RoBERTa, FalseSum+, FactCC+, FactGraph, FnetGraph-edgel We follow the same train/dev/test dataset split and experiment settings so that the results directly comparable. We present more details about the baselines in Appendix D.
Human Correlation We evaluate FACTKB and baselines on the FRANK benchmark to study how well FACTKB correlates with human judgments. We use the official script 2 to report the Pearson (p) and Spearman (r) correlation and p-values. Results in Table show that classification-bases metrics (FactCC, FactGraph, and FACTKB) generally out- perform QA-based metries (QAGS and QUALS). FACTKB significantly advances the state-of-the-an on the FRANK benchmark, resulting in the im- provement of 5-15 correlation points across mul- tiple settings. Our results show that FAC is highly correlated with human judgments, making it practical approach for evaluating the factual consistency of generated news summaries.
4 Results
in-Domain Results We evaluate FACTKB and baselines on the FactCollect dataset using the en-
We evaluate FACTKB and existing factuality evaluation models out-of-
Out-of-Domain Results
Datasct statistics are presented in Table
Phttps://gis http://ggttttubtcoo httpl//cc///ttoo/fraaaaa htttess//tttha...corrrriiirrrrr/

---

vanilla
FactKB-wiki
FactK B-evid-eeee
FaciKB-walk
85.3
89.4
87.6
86.7
87.5 89.1
87.7
89.4
88.3
87.6 88.0 88,7
87.6
89.1
87.8 81.4 87.4 86.6 88.4
Eletta 85.8
$7.0
86.8
84.4
86.1
87.8
87.3
87.1
87.1
N6.2
86.1 86.2 86.9
85.5
87.0 86.3 86.3 86.1 86.2
BART 84.0
84.55
85.4
83.8
85.1
84.9
85.9
85.9
84.1
84.9
85.6
83.3 84,9
83.9
84.3
85.0 85.1
84.9 84.9
DeBERTu 87.5
.87.9
87.9
86.6
85.8
90.1
90.1
89.0
89.0
89.0
87.5
88.4 88.0
87.7
87.7 88.2
87.0
89.1 89.2
ALBERT 85,5
88.6
87.1
86.2
85.8
K8.0
87.4
87.5
89.1
87.9
88.6
87.7 87.7
87.7
87.0
86.2
87.6
86.2
87.1
dsS.RABUKTT 85,0
87.22
86.7
85.6
85.3
87.4
86.8
86.7
87.1 87.6 86.2 87.1 86.9
86.2
86.
86, 4 86.3
86.6
86.7
KJB
YAGO WIKI CPNe Aoonic KGAP UMES
YAGO WIKT CPNer Anomic KGAP UMES
YAGO WIKI CPSer KGAP UMEES
Figure 3: Compatibility of FACTKB across various LMs and K.B.. We report BACC scores of different setups on the FactCollect dataset. FACTKB is general method compatible with various LMM and KB settings.
GAGS
SAI
2021) and examine FACTKB's performance on the three error categories: semantic frame, discourse, and content verifiability errors. Using the official script in the FRANK benchmark, remove each category of errors and report changes in correlation scores. Higher variation indicates a greater influ- ence on model's 's ability handle certain type of error. Figure demonstrates that FACTKB is significantly better at identifying semantic frame errors, which focus on entities and relations. This indicates that our KB-based factuality pretraining strategies successfully result in better understand- ing of the facts regarding entities and relations. FACTKB also has good performance in other cat- egories, resulting in a factuality evaluation model that captures diverse types of errors and advances the state-of-the- across the board. We conduct further qualitative analysis in Appendix B.
0.4
Pearson Correlation
0.2
0.1
discourse
verifiability
Figure 4: Correlation of FACTKB and baselines with hu- man judgments across error categories, FACTKB shows significant improvement capturing semantic frame eπrоrs and has slightly better par performance on discourse and content verifiability errors.
domain scientific literature datasets zero-shot manner. Results are presented Table 5, which demonstrate that while existing factuality evalua- tion models previously achieve good performance in the in-domain setting, they exhibit severe perfor- mance drops on the three out-of-domait datasets, performing only slightly better than random factu- ality scores (RANDOM). This suggests that existing approaches are not generalizable to other domains, limiting their applicability. On the contrary, FAC- TKB significantly (*) outperforms existing factual- ity metrics by 4.1 BACC points on average across datasets. Our results sug-
5.2 KB and LM Compatibility
FACTKB uses pretrained LMs for initialization, leverages external KBs for factuality pretraining, and trains on factuality evaluation datasets to result in factuality metric. Our general methodology to leverage knowledge bases as fact teachers for gen- eralizable factuality evaluation could work with dif- ferent LM and KB combinations. To study whether out approach works across different settings, we ap- ply the FACTKB methodology to six different LMs (RoBERTa, Electra, BART, DeBERTa, ALBERT, and distilRoBERTaa) and pretrain the LMs on fac- tuality pretraining corpora constructed based on six different KBs (YAGO, Wikidata, ConceptNet, Atomic, KGAP, and UMLS). For each combination, initialize particular LM and pretrain it using the proposed three pretraining strategies based on a particular KB. For each setting, we evaluate the re- sulting model using the FactCollect dataset and re- port the BACC scores. We present the performance
the three gest that the factuality pretraining strategies enable FACTKB to better represent facts (entities and re- lations) in a new domain, making the factuality evaluation model more robust shifting domains.
5 Analysis and Discussion
5.1 Where did FACTКB Improve?
To better understand F'CTKB's improvement over existing approaches, we leverage the factual error typology the FRANK benchmark (Pagnoni et al.,

---

Metric
Pearson
Spearman
Usage Steps
QAGS
.22
.23
extruct answer candidates generate the questions 3) unswer the generated questions 4) compare the answers to obtain the QAGS factuality score
QUALS
.22
.19
1) generating question and answer pairs from summaries 2) filter the generated question and answer for high-quality pairs 3) evaluate the generated question and answer pains using the source document input, compute QUALS scores
summary
DAE
.17
.20
1) preprocess summarics and documents with dependency parsing 2) the pretruined model get DAE
FACTCC .20
.29
the pretrained model to get FacCCC
FACTGRAPH .35
.42
1) build abstract meaning representation graphs 2) run the pretrained model to FactGrruhh scores
FACTKB
.47
.52
the peetrained model get FACI scores
Table 6: Usage steps of factuality metries and their performance on the FRANK benchmark. FACTKB (WALK) presents state-of-the-art factuality metric with minimum hassle when evaluating new summaries and articles.
of different settings in Figure 3, which illustrates that regardless of which LM and KB, FACTKB generally results in improved factual error detec- tion capabilities compared to the vanilla LM check- points without factuality pretraining. In addition, certain LMs (RoBERTa and DeBERTa) and KBs (YAGO, KGAP, and UMLS) are better than oth- ers, suggesting that the choice of the base LM and external KB warrants further research. Our results demonstrate that FACTKB is a general pretraining approach that can be applied to various LM-KB combinations to improve fact representations and develop better factuality evaluation models.
we further explore the effect of factuality pretrain- ing corpus size in Figure 5. is illustrated that N = 1e4 or N = 1e5 are generally desirable set- tings, while factuality pretraining with too large Ns might be counterproductive. This could in part be attributed to catastrophic forgetting (Ramasesh et al., 2021), which warrants further research.
Pretraining epoch. FACTKB further pretrains LM checkpoints on the three factuality pretrain- ing corpora, while the training epoch governs the intensity of such exercises. We adopted 5 epochs of continued pretraining in the main experiments, while we further explore the effect of pretraining epochs in Figure 5. it is demonstrated that to 10 epochs are generally desirable while exercising too much might be counterproductive.
5.3 Simplicity Study
While existing factuality evaluation approaches re- quire additional processing (such as computing the dependency structure (Goyal and Durrett, 2020) and AMR graphs (Ribeiro et al., 2022) or running multiple iterations of question generation (Fabbri et al., 2022)) in the face of new data, FACTKB re- quires no preprocessing and only uses fine-tuned RoBERTa for sequence classification. We summa- rize the steps involved in using existing approaches and their performance on the FRANK benchmark in Table 6, which demonstrates that FACTKB not only has state-of-the-art performance but is also a lightweight and simple factuality evaluation model.
Knowledge walk length. An important aspect of the knowledge walk factuality pretraining is the generated walk length K., which governs the de- gree of compositionality in the pretraining corpus. While we adopted = 5 in the main experiments, we further explore the effect of IC in Figure 5. It is illustrated that K. = 5 performs best by provid- ing moderate amount of compositionality in the factuality pretraining corpora.
6 Related Work
5.4 Parameter Analysis
Factuality Evaluation Recent advances in text summarization have presented models and systems that are capable of generating increasingly fluent, controllable, and informative summaries of docu- ments (Liu and Lapata, 2019; Balachandran et al., 2021; Meng et al., 2022: Tang et al., 2022; Gold- sack et al., 2022; Peng et al., 2021; Aharoni et al.,
Corpus size. For evidence extraction and knowl- edge walk, the pretraining corpus size N is con- trollable and governs the amount of information towards augmenting FACTKB's s ability towards fac- tual errors regarding entities and relations. While we adopted N = 1e5 in the main experiments,

---

2023; Liu al., 2022d; Rothe al., 2021; Narayan al., 2021; Bhattacharjee al., 2023; Chen et al., 2023b; He et al.. 2023; Liu et al., 2023b; Chen et al., 2023a). However, they suffer from hallucina- tion and might not be factually faithful towards the source document (Cao et al., 2018 Pagnoni et al.,
BACC
0.90-
inder
miker
0.85
0.50
1k
10lk
100k
1m
10m
0.80
1k
10k 100k
Tim
10m
corpus size
corpus size
0.95
0.95
wilk
wit
2021; Balachandran et al., 2022; Tang et al., 2023; Liu et al., 2023a; Luo et al., 2023), leading to in- creased research in factuality evaluation. QA-based approaches (Wang et al., 2020a; Nan et al., 2021; Scialom et al., 2021; Fabbri et al., 2022) attempt to generate and answer questions based on sum- maries and documents and judge the factuality by comparing answers. Later approaches are generally entailment-based (Kryšciński et al., 2020; Goyal and Durrett, 2020, 2021: Laban et al., 2022; Ribeiro et al., 2022), proposing to classify (summary, docu- ment) pairs into FACTUAL or NON-FACTUAL labels. Among them, FactCC (Kryšciñski al., 2020) is one of the first entailment-based metrics and trained on synthetic data; DAE (Goyal and Durrett, 2020, 2021) proposes to leverage the dependency structure of summaries and documents; FactGraph (Ribeiro et al., 2022) builds abstract meaning repre- sentation graphs and adopts graph neural networks for joint representation learning along the textual content. In addition, hypothesis re-ranking (Gar- neau and Lamontagne, 2021), counterfactual esti- mation (Xie et al., 2021), NLI models ( (Utama et al., 2022), phrase-level localization (Takatsuka et al., 2022), and weighting facts in the source document (Xu et al., 2020) were also explored in factuality evaluation. Moving beyond a binary concept of fac- tuality, FRANK (Pagnoni et al., 2021) promotes fine-gra fine-nrained understanding of factuality and pro- poses typology of factuality errors. Inspired by its analysis that semantic frume errors, errors re- garding entities and relations, are a major source of factuality errors yet under-explored by existing fac- tuality metrics, we propose FACTKB to leverage external KBs for factuality pretraining and help en- force better factuality towards entities and relations discussed in summaries and documents.
BACC
0.50
5
10
50
100
0.80
10
50
100
pre-training epoch
pre-training epoch
0.95
0.95
vok
vide
ВАCC
0.90
0.85
2
10
50
2
10
walk length
walk length
Figure 5: Parameter analysis of pretraining corpus size, epoch, and knowledge walk length.
et al., 2021; Zhang et al., 2022b), and long docu- ments (Feng et al., 2023). KB KB-enhanced approaches also advanced numerous NLP tasks, ranging from question answering (Mitra et al., 2022; Bosselut et al., 2021; Oguz et al., 2022; Feng et al., 2022; Heo et al., 2022; Ma et al., 2022), text generation (Rony et al., 2022; Dognin et al., 2021; Yu et al., 2021). and commonsense reasoning (Kim et al., 2022: Jung et al., 2022: Amayuelas et al., 2021; Liu et al., 2022a). In this work, we tap into KBs' nature as high-quality reservoirs of factual informa- tion and construct factuality pretraining objectives to augment factuality evaluation.
7 Conclusion
We propose FACTKB, a simple and novel approach to factuality evaluation using language models pre- trained facts from external KBs to improve tity and relation representations. Specifically, we leverage KBs construct three factuality pretrain- ing objectives: entity wiki, evidence extraction, and knowledge walk. FACTKB pretrains an LM using the three objectives and fine-tunes the resulting model on factuality evaluation datasets. Extensive experiments demonstrate that FACTKB advances the statc-of-the-an in both in-domain and out-of- domain factuality evaluation, better correlates with human factuality annotations, and better detects se- mantic frame errors. FACTKB presents an easy-to- use and generalizable factuality metric, facilitating research on factually-come conssssten summarization.
Knowledge Bases in NLP Knowledge base is a standard format for structured knowledge represen- tation. One application of KBs in NLP is to inject knowledge and augment LMs, where different ap- proaches focused aspects such as pretraining (Chen al., 2020; Agarwal et al., 2021; Rosset et al., 2020; Li et al., 2022), document graphs (Hu et al., 2021; Zhang et al., 2022a), KB structure (Yasunaga

---

Limitations
alizable factuality metric. However, in this work. we mainly focused on the additional domain of scientific literature, while other potential domains remain underexplored such as social media (Syed et al., 2019; Kano et al., 20 He et al., 2020). We leave it to future work the exploration of FACTKB and existing factuality metrics more document domains that are present in summarization systems.
LM and KB Selection While Section 5.2 offers empirical evidence that FACTKB is compatible with 6 language models and 6 external knowledge bases, remains unclear upfront which LM and KB combination would be most desirable. While empirical performance could be good guide, there are several unaddressed possibilities: For language models, it is possible to leverage an ensemble of FACTK Bs seeded with different LM checkpoints and architectures. This might result in better fac- tuality evaluation, but would also dramatically in- crease the computation costs when evaluating new data. For knowledge bases, it is possible to leverage domain expertise and select an external knowledge base that would be most helpful for the domain adaptation of factuality evaluation. It is also possible to leverage a combination of existing knowledge bases for FACTKB's 's factuality pretrain- ing, while the specific combination and how to apply different factuality pretraining to different KBs hard to determine. All in all, F presents general KB-enhanced factuality metric with numerous possibilities, while we leave some these considerations future work.
Tradeoff between Performance and Granularity Existing approaches (Kryšciñiski al., 2020; Takat- suka et al., 2022) struggle with semantic frame rors and involve heavy preprocessing, while they provide fine-grained analysis and specific localiza- tion of summarization errors. FACTKB achieves significantly better factuality evaluation results and is casier to use while lacking the ability of error localization. We argue that this tradeoff should be considered with the use case in mind: for LLM evaluation, it is better have accurate metric for benchmarking efforts and efficient metric for handling large-scale LM generation. As a result, FACTKB provides a valuable tool for factuality evaluation and LLM rescarch.
Ethics Statement
FACTKB training is not end-to-end. FACTKB has two-step training process: pretraining with KB-based factuality pretraining and fine-tuning on factuality evaluation datasets. This creates several limitations, among which is the difficulty of hy- perparameter tuning. Appendix 3.1 presents the study of hyperparameters in the factuality pretrain- ing stage, which demonstrates that FACTKB works best with a moderate but not excessive amount of factuality pretraining. This reliance on certain hy- perparameter configurations is further complicated by more hyperparameter choices in the fine-tuning stage. While the current hyperparameter setting in Appendix 3.1 achieves statc-of-the -frtc-r empiri- cal performance, we acknowledge the difficulty in FACTKB hyperparameter tuning.
LM and KB Bias FACTKB is initialized with pretrained language model checkpoints and lever- ages knowledge-basee base- factuality pretraining. Consequently, FACTKB might pick up the biases of the adopted language models (Liang et al., 2021; Nadeem et al., 2021; Shaikh et al., 2023; Tan and Celis, 2019) and knowledge bases (Fisher et al., 2020; Mehrabi et al., 2021). As a result, FACTKB might leverage these biases in judging the factu- ality of summaries, further reinforcing the bias in text summarization systems. We leave it to future work on understanding and mitigating the bias of factuality metrics.
Misuse Potential FACTKB leverages high- quality and factual knowledge bases to generate fac- tuality pretraining corpora and augment LM's abil- ity to stay factual with respect to entities and rela- tions discussed in the summary and document. On the contrary, if non-factual and misleading knowl- edge is leveraged for the three factuality pretraining strategies, it might jeopardize the factuality of FAC- TKB and make it insensitive misinformation and falsehoods in summaries and documents. As a re- sult, we encourage the responsible use of FACTKB and the factuality pretraining methodology.
Out-of-Domain Factuality Evaluation. An im- portant focus of this work is out-of of-domain factual- ity evaluation: Summarization systems face input documents from varying domains, which requires factuality metrics also generalize different document domains. Existing metrics struggle with semantic frame errors and such struggle exacer- bated by the domain shift of entities and relations, while FACTKE KC offers a stronger and more gener-

---

Acknowledgements
Jacob Benesty, Jingdong Chen, Yiteng Huang, and Is- rael Cohen. 2009. Pearson correlation coefficient. In Noise reduction in speech processing, pages 1-4. Springer.
We thank the reviewers, the area chair, members of Tsvetshop, and the UW NLP Group for their feed- back. This research was supported by This research is supported in part by the Office of the Director of National Intelligence (ODNT), Intelligence Ad- vanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022-2202 00122-220072000044 This material is also funded by the DARPA Grant under Contract No. HR001 120C0124. We also gratefully acknowledge support from NSF CA- REER Grant No. IIS2142739 and the Alfred P. Sloan Foundation Fellowship. The views and conclusions contained herein are those of the au- thors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, TARPA, or the U.S. Gov- crnment. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
Abhik Bhattacharjee, Tahmid Hasan, Wasi Uddin Ah- mad, Yuan-Fang Li, Yong- Bin Kang, and Rifat Shahriyar. 2023. CrossSum: Beyond English-centrie cross-fingual summarization for 1,500+ language pairs. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume I: Long Papers), pages 2541-2564.
Steven Bird, Ewan Klein, and Edward Loper. 2009, Nat- ural language processing with Python; analyzing text with the natural language toolkit. O'Reilly Media, Inc.".
Antoine Bordes, Nicolas Usumier, Alberto Gareia- Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data. Advances in neural information pro- cessing systems, 26.
Antoine Bosselut, Ronan Le Bras, and Yejin Choi. 2021. Dynamic neuro neuro-symbon symbolic knowledge graph construc- tion for zero-shot commonsense question answering. In Proceedings of the 35th AAAI Conference on Arti- ficial Intelligence (AAAI).
References
Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. 2021. Knowledge graph based synthetic corpus generation for ppowledge-enhaneed language model pre-training. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 3554-3565.
Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel S Weld. 2020. Tidr: Extreme summarization of sci- entific documents. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4766-4777
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural ahstractive summarization. In second AAAI conference on artificial intelligence.
Roee Aharoni, Shashi Narayan, Joshua Maynez, Jonathan Herzig, Elizabeth Clark, and Mirella La- pata. 2023. Multilingual summarization with factual consistency evaluation. In Findings of the Associa- tion for Computational Linguisties: ACL 2023, pages 3562-3591.
Wenhu Chen, Yu Su, Xifeng Yan, and William Yang Wang. 2020. Kgpt: knowledge-groundde pre- training for data-to-text generation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 8635- 8648.
Alfonso Amayuelas, Shuai Zhang, Xi Susie Rao, and Ce Zhang. 2021. Neural methods for logical reason- ing over knowledge graphs. In International Confer- ence on Learning Representations.
Xiuying Chen, Guodong Long, Chongyang Tao, Mingzhe Li, Xin Gao, Chengqi Zhang, and Xian- gliang Zhang. 2023a. Improving the robustness of summarization systems with dual augmentation. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), ). pages 6846-6857.
Vidhisha Balachandran, Hannaneh Hajishirzi, William Cohen, and Yulia Tsvetkov. 2022. Correcting diverse factual errors in abstractive summarization via post- editing and language model infilling. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.
Vidhisha Balachandran, Artidoro Pagnoni, Jay Yoon Lee. Dheeraj Rajagopal. Jaime Carbonell, and Yu- lia Tsvetkov. 2021. StructSum: Summarization via structured representations. In Proceedings of the 1 16th Conference of the European Chapter of the Associ- ation for Compulational Linguisties: Main Volume,
Yulong Chen, Yang Liu, Ruochen Xu, Ziyi Yang, Chen- guang Zhu, Michael Zeng, and Yue Zhang. 20236. UniSumm and SummZoo: Unified model and diverse benchmark for few-shot summarization. In Proceed- ings of the 61 Annual Meeting of the Association for Computational Linguisties (Volame Long Papers). pages 12833-12855.
pages 2575-2585.

---

Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguisties: Human Language Technolo- gies, Volume 1 (Long and Short Papers), pages 4171- 4186.
Joseph Fisher, Arpit Mittal, Dave Palfrey, and Chris- tos Christodoulopoulls. 2020. Debiasing knowledge graph embeddings. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 7332-7345.
Nicolas Garneau and Luc Lamontagne. 2021. Trainable ranking models evaluate the semantic accuracy of data-to-text neural generator. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 51-61.
Pierre Dognin, Inkit Padhi, Igor Melnyk, and Payel Das. 2021. ReGen: Reinforcement learning for text and knowledge base generation using pretrained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1084-1099.
Tomas Goldsack, Zhihao Zhang, Chenghua Lin, and Carolina Scarton. 2022. Making science simple: Cor- pora for the lay summarisation of scientific literature. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.
Hady Elsahar and Matthias Gallé. 2019. To annotate or not? predicting performance drop under domain shift. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing JEMNL.--JCNIA pages 2163-2173.
Tanya Goyal and Greg Durrett. 2020. Evaluating f'actu- ality in generation with dependency- entailment. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3592-3603.
Matan Eyal, Tal Baumel, and Michael Elhadad. 2019. Question answering as an automatic evaluation met- ric for news article summarization. In Proceedings of the 2019 Conference of the North American Chap- ler of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pages 9938-3948.
Tanya Goyal and Greg Durrett. 2021. Annotating and modeling ame-grained factuality in summarization. In P Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies.
Suchin Gururangan, Ana Marasović. Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't 't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 8342-8360.
Alexander Fabbri, Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. QAFactFival: Improved QA- based factual consistency evaluation for summariza- tion. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, pages 2587-2601.
Charles R Harris, K Jarrod Millman, Stéfan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cour- napeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, al. 2020. Array programming with numpy. Nature, 585(7825)3557-333 362.
William Falcon and The Py Lightning team. 2019. PoToh Lightning.
Yair Feldman and Ran El- Yaniv. 2019. Multi-hop para- graph retrieval for open-domain question answering. In Proceedings of the 57th Annual Meeting of the As- sociationfor Computational Linguistics, pages 2296- 2309.
Pengeheng He, Baolin Peng. Song Wang. Yang Liu, Ruochen Xu, Hany Hassan, Yu Shi, Chenguang Zhu, Wayne Xiong, Michael Zeng, Jianfeng Gao, and Xue- dong Huang. 2023. Z-code++: A pre-trace rreitra lan- guage model optimized for abstractive summariza- tion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Pupers), pages 5095-5112 12.
Shangbin Feng, Zilong Chen, Wenqian Zhang, Qingyao Li. Qinghua Zheng, Xiaojun Chang, and Minnan Luo. 2021. Kgap: Knowledge graph augmented political perspective detection in news media. arXiv preprint arXiv:2108.00 100.0666..
Ruifang He, Liangliang Zhao, and Huanyu Liu. 2020. TWEETSUM: Event oriented social summarization dataset. In Proceedings of the 28th International Conference on Compulational Linguistics, pages 5731-5736.
Shangbin Feng, Zhaoxuan Tan. Wenqian Zhang. Zhenyu Lei, and Yulia Tsvetkov. 2023. KALM: Knowledge- aware integration of local, document, and global con- texts for long document understanding. In Proceed- ings of ACL 2023, pages 2116 -138.
Yu-Jung Heo, Eun-Sol Kim, Woo Suk Choi, and Byoung Tak Zhang. 2022. Hypergraph trans- former: Weakly-superviser multi-hop reasoning for knowledge-based visual question answering. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), pages 373-390.
Yue Feng, Zhen Han, Mingming Sun, and Ping Li. 2022. Multi-hop open-domain question answering over structured and unstructured knowledge. In Find- ings of the Association for Computational Linguisties: NAACL 2022, pages 151-156, Seattle, United States.

---

Linmei Hu, Tianchi Yang, Luhao Zhang, Wanjun Zhong, Duyu Tang, Chuan Shi. Nan Duan, and Ming Zhou. 2021. Compare to the knowledge: Graph neural fake news detection with external knowledge. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 754-763.
scientific paper summarization based on conference talks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2125-2131.
Chen Li, Zhongyu Wei, Yang Liu, Yang Jin, and Fei Huang. 2016. Using relevant public posts to en- hance news article summarization. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 557-566.
Yong-Ho Ho Jung, Jun-Hyung Park, Joon- Young Choi, Mingyu Lee, Junho Kim. Kang-Min Kim, and SangKeun Lee. 2022. Learning from missing rela- tions: Contrastive learning with commonsense knowl- edge graphs for commonsense inference. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1514-1523.
Shaobo Li, Xiaoguang Li, Lifeng Shang, Chengjie Sun, Bingquan Liu, Zhenzhou Ji, Xin Jiang, and Qun Liu. 2022. Pre-training language models with determin- istic factual knowledge. In Proceedings of the 2022 Conference Empirical Methods in Natural Lan- guage Processing.
Ambedkar Kanapala, Sukomal Pal, and Rajendra Pa- mula. 2019. Text summarization from legal doc- uments: a survey. Artificial Intelligence Review, 51(3):371-402.
Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021. Towards understand- ing and mitigating social biases in language models. In International Conference on Machine Learning, pages 6565-657 6576. PMLR.
Ryuji Kano, Yasuhide Miura, Motoki Taniguchi, Yan- Ying Chen, Francine Chen, and Tomoko Ohkuma. 2018. Harnessing popularity in social media for extractive summarization of online conversations. In Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pages 1139-1145.
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Pe- ter West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishirzi. 2022a. Generated knowledge prompting for commonsense reasoning. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 3154-3169.
Yu Jin Kim, Beong-woo Kwak, Youngwook Kim, Reinald Kim Amplayo, Seung-wor won Hwang, and Jiny- oung Yeo. 2022. Modularized transfer learning with multiple knowledge graphs for zero-shot common- sense reasoning. In Proceedings of the 2022 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages
Xiao Liu. Shiyu Zhao. Kai Su, Yukuo Cen, Jiezhong Qiu, Mengdi Zhang, Wei Wu, Yuxiao Dong, and Jie Tang. 2022b. Mask and reason: Pre-training knowl- edge graph transformers for complex logical queries. In Proceedings of the 28th ACM SIGKDD Confer- ence Knowledge Discovery and Data Mining, pages 120-1130.
Wojeiech Kryšciński. Bryan McCann, Caiming Xiong. and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pages 9332-9346.
Yang Liu and Mirella Lapata. 2019. Text summariza- tion with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740.
Philippe Laban, Tobias Schnabel. Paul N. Bennett, and Marti Hearst. 2022. SummaC: Re- Visiting NLI- based Models for Inconsistency Detection in Summa-
Yang Liu, Chenguang Zhu, and Michael Zeng. 2022c. End-ta-end regmentation-hasee news summarization. In Findings of the Association for Computational Linguistics: ACL 2022, pages 544-554.
rization. Transactions of the Association for Compu-
tational Linguistics. 10:163-177.
Timothée Lacroix, Guillaume Obozinski, and Nicolas Usunier. 2019. Tensor decompositions for temporal knowledge base completion. In International Con- ference on Learning Representations.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 1rXiv:1907.1169.
Egoitz Laparra, Steven Bethard, and Timothy A Miller. 2020. Rethinking domain adaptation for machine learning over clinical language. JAMIA open, 3(2):146-150.
Yixin Liu, Budhaditya Deb, Milagro Teruel, Aaron Hal- faker, Dragomir Radev, and Ahmed Hassan Awadal- lah. 2023a. On improving summarization factual consistency from natural language feedback. In Pro- ceedings of the 61 Annual Meeting of the Associa- tion for Computational Linguistics ( (Volume I: Long Papers), pages 15144-15161.
Guy Lev, Michal Chmueli-Scheues Jonathan Herzig, Achiya Jerbi, and David Konopnicki. 2019. Talk- Summ: A dataset and scalable annotation method for

---

Yixin Liu, Alex Fabbri, Pengfei Liu, Yilum Zhao, Liny- ong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. 2023b. Revisiting the gold standard: Ground- ing summarization evaluation with robust human evaluation. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Pupers). pages 4140-4170.
Feng Nan, Cicero dos Santos, Henghui Zhu, Patrick Ng. Kathleen Mckeown, Ramesh Nallapati, Dejiao Zhang, Zhiguo Wang, Andrew O Arnold, and Bing Xiang. 2021. Improving factual consistency of ab- stractive summarization via question answering. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 6881- 6894.
Yongtai Liu, Joshua Maynez, Gonçalo Simöes, and Shashi Narayan. 2022d. Data augmentation for low- resource dialogue summarization. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 703-710.
Shashi Narayan, Shay Cohen, and Maria Lapata. 2018. Don't give me the details, just the summary! topic- aware convolutional neural networks for extreme summarization. In 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807. Association for Computational Linguis- tics.
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as a factual inconsistency evaluator for abstractive text summarization. arXiv preprint w.Xiv:2303.1621..
Shashi Narayan, Yao Zhao, Joshua Maynez, Gonçalo Simdes, Vitaly Nikolaev, and Ryan McDonald. 2021. Planning with learned entity prompts for abstractive summarization. Transactions of the Association for Computational Linguistics, 1.1475--492
Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, and Jianfeng Gao. 2022. Open domain question answering with a unified knowledge interface. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 1605-1620, Dublin, Ireland.
Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. 2022. UniK- -AI: Unified representations of structured and unstructured knowledge for open-domain question answering. In Findings of the Association for Compu- tational Linguistics: NAACL 2022, pages Seattle, United States.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 1906-1919.
Ninareh Mehrabi, Pei Zhou, Fred Morstatter, Jay Pu- jara, Xiang Ren, and Aram Galstyan. 2021. Lawyers are dishonest? quantifying representational harms in commonsense knowledge resources. In Proceedings of the 202] Conference on Empirical Methods in Natural Language Processing, pages 5016-5033.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstrac- tive summarization with frank: A henchmark for fac- tuality metrics. In Proceedings of the 2021 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, pages 1829..
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual asso- ciations in gpt. In Advances in Neural Information Processing Systems.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, nigh-performance deep learning library. Advances in neural information processing systems, 32.
Sayantan Mitra, Roshni Ramnani, and Shubhashis Sen- gupta. 2022. Constraint- banstr multi-hop question answering with knowledge graph. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies: Industry Track, pages 280-288.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer. R. Weiss, V. Dubourg. J. Vanderplas, A. Passos. D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 22:882528330
Leann Myers and Maria J Sirois. 2004. Spearman cor- relation coefficients, differences between. Encyclo- pedia of statistical sciences, 12.
Thomas Pellissier Tanon, Gerhard Weikum, and Fabian Suchanek. 2020. Yago 4: A reason-able knowledge base. In European Semantic Web Conference, pages 583-596. Springer.
Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. Stereoset: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 1ith International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 5356-5371.
Xutan Peng, Yi Zheng, Chenghua Lin, and Advaith Siddharthan. 2021. Summarising historical text in modern languages. Proceedings of the 16th Con- ference of the European Chapter of the Association

---

for Computational Linguistics: Main Volume, pages 3123-3142.
Omar Shaikh, Hongxin Zhang. William Held, Michael Bernstein, and Diyi Yang. 2023. On second thought, let's not think step by step! bias and toxicity in zero- shot reasoning. In Proceedings of the 6 Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Pupers), pages 4454 4470.
Seth Polsley, Pooja Jhunjhunwala, and Ruihong Huang. 2016. Casesummarizer: a system for automated sum- marization of legal texts. In Proceedings of COLING 2016, the 26th international conference on Compu- tational Linguistics: System Demonstrations, pages 258 258-262.
Siamak Shakeri, Cicero dos Santos, Henghui Zhu, Patrick Ng, Feng Nan, Zhiguo Wang. Ramesh Nal- lapati, and Bing Xiang. 2020. End-to-end synthetic data generation for domain adaptation of question answering systems. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 5445-5460.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350.
Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. 2021. Effect of scale on catastrophic forgetting in neural networks. In International Con- ference Learning Representations.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge, In Thirty-first AAAI conference on artificial intelligence.
Leonardo F. R. Ribeiro, Mengwen Liu, Tryna Gurevych, Markus Dreyer, and Mohit Bansal. 2022. FactGraph: Evaluating factuality in summarization with semantic graph representations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages
Shahbaz Syed, Michael Völske, Nedim Lipka, Benno Stein, Hinrich Schütze, and Martin Potthast. 2019. Towards summarization for social media- results of the TL:DR challenge. In Proceedings of the 12th International Conference Natural Language Gen- eration, pages 523-528.
Md Rashad AI Hasan Rony, Ricardo Usbeck, and Jens Lehmann. 2022. DialoKG: Knowledge-structure aware task-oriented dialogue generation. In Find- ings of the Association for Computational Linguis- tics: NAACI. 2022, pages 2557-2571, Seattle, United States.
Masato Takatsuka. Tetsunori Kobayashi, and Yoshihiko Hayashi. 2022. Phrase-level localization of inconsis- tency errors in summarization by weak supervision. In Proceedings of the 29th International Conference on Computational Linguistics. pages 6151-6164.
Corby Rosset, Chenyan Xiong, Minh Hieu Phan, Xia Song, Paul Bennett, and Saurabh Tiwary. 2020. Knowledge-aware language model pretraining. ArXiv, abs/2007.00653
Yi Chem Tan and L Elisa Celis. 2019. Assessing so- cial and intersectional biases contextualized word representations. Advances in Neural Information Processing Systems, 32.
Sascha Rothe, Joshua Maynez, and Shashi Narayan. 2021. A thorough evaluation of task-specific pre- training for summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 140-145.
Liyan Tang. Tanya Goyal, Alex Fabbri, Philippe La- ban. Jiacheng Xu, Semih Yavuz, Wojciech Kryscin- ski, Justin Rousseau, and Greg Durreu. 2023. Un- derstanding factual errors in summarization: Errors, summarizers, datasets, error detectors. In Proceed- ings of the 61 Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers). pages 11666-11644
Arkadiy Saakyan, Tuhin Chakrabarty, and Smaranda Muresan. 2021. Covid-fact: Fact extraction and veri- fication of real-world claims on covid- 19 pandemic. In Proceedings of the 59th Annual Meeting of the A Association for Computational Linguistics and the 1 1ith International Joint Conference on Natural Lan- guage Processing (Volume I: Long Papers), pages 21 16-29
Xiangru Tang, Arjun Nair, Borui Wang, Bingyao Wang, Jai Desai, Aaron Wade, Haoran Li, Asli Celikyilmaz. Yashar Mehdad, and Dragomir Radev. 2022. CON- FIT: Toward faithful dialogue summarization with mingustticlly-inoomme
contrastive fine-tuning, In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 5657-5668.
Mourad Sarrouti, Asma Ben Abacha, Yassine M' d'rabet, and Dina Fainnerr-- 2021. Evidence- based fact-checking of health-related claims. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3499-3512.
Prasctya Utama, Joshua Bambrick, Nafise Moosavi, and Iryna Gurevych. 2022. Falsesum: Generating document- NLI examples for recognizing fac- tual inconsistency in summarization. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 2763-2776.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang. and Patrick Gallinari. 2021. Questeval: Summariza-
tion asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6594-6604.

---

Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar. 2019. Composition-basee multi- relational graph convolutional networks. In Interna- tional Conference on Learning Representations.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Authony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empiricul methods in natural language processing: system demonstrations, pages 38-45.
Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021. Trec-covid: constructing pandemic information re- trieval test collection. In ACM SIGIR Forum, vol- ume 54, pages 1- ACM New York, NY, USA.
Yuexiang Xie, Fei Sun, Yang Deng, Yaliang Li, and Bolin Ding. 2021. Factual consistency evaluation for text summarization via counterfactual estimation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 100-110.
Denny Vrandečić and Markus Krötzsch. 2014. Wiki- data: a free collaborative knowledgebase. Communi- cations of the ACM, 57(10) 85.
Xinnuo Xu, Ondiej Dušek, Jingyi Li, Verena Rieser, and Joannis Konstas. 2020. Fact-based content weighting for evaluating abstractive summarisation. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5071-5081.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534-755 45500.
Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher Manning, Percy S Liang, and Jure Leskovec. 2022. Deep bidirectional imguage-knowlekde graph pretraining. Advances in Neural Information Processing Systems, 35:37309- 37323.
David Wadden, Kyle Lo, Lucy Lu Wang, Arman Cohan, Iz Beltagy, and Hannanch Hajishirzi. 2022. Mul- VerS: Improving scientific claim verification with weak supervision and full-document context. In Find- ings of the Association for Computational Linguistics: NAACL 2022, pages 61-76.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang. and Jure Leskovec. 2021. Qa-gnn: Rea- soning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 535-546.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020a. Asking and answering questions to evaluate the fac- tual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics. pages 5008-5020.
Wenhao Yu, Meng Jiang, Zhiting Hu, Qingyun Wang. Heng Ji. and Nazneen Rajani. 2021. Knowledge- enriched natural language generation. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, pages 11-16.
Lucy Lu Wang. Kyle Lo. Yoganand Chandrasekhar, Rus- sell Reas, Jiangjiang Yang. Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Michael Kinney, et al. 2020b. Cord-19; The 1ovid-19 open research dataset. In Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger. and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In International Confer- ence on Learning Representations.
Wenya Wang and Sinno Pan. 2022. Deep inductive logic reasoning for multi-hop hop reading comprehension. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 4999-5009.
Wenqian Zhang, Shangbin Feng, Zilong Chen, Zhenyu Lei. Jundong Li, and Minnan Luo. 2022a. KCD: Knowledge walks and textual cues enhanced political perspective detection in news media. In Proceedings of NAACL 2022, pages 4129-4140.
Xiaozhi Wang. Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transac- tions of the Association for Computational Linguis- tics, 9:176-194.
X Zhang, A Bosselut, M Yasunaga. H Ren, P Liang. C Manning, and J Leskovec. 2022b. Greaselm: Graph reasoning enhanced language models for ques- tion answering. In International Conference on Rep- resentation Learning (ICLR).
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language mod- els to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistiex: Hu- man Language Technologies, pages 4602-4625.

---

A Merging the three strategies
DAE (Goyal and Durrett, 2020) leverages the dependency structure of the summary and article design a factuality metric.
We also tried combining the three factuality pre- training strategies to obtain FACTKB BROCCK--COMBNETI We evaluate it on the FactCollect dataset and present results in Table 7. It is demonstrated that BACT-B-COMBRR
SummaC (Laban et al., 2022) proposes to revisit and repurpose NLI models for detecting factual inconsistencies in text summarization.
is not significantly better than using single factuality pretraining strategy, while we will make all versions of FACTKB pub- licly available.
FalseSum (Utama et al., 2022) augments NLI training data with controllable text generation for better factuality evaluation.
B Qualitative Analysis
We present examples (summary, article) pairs and their factuality scores Table and 10, where BACTKB is significantly closer to human judgment than existing factuality metrics. It is demonstrated that while existing factuality metrics are insensitive to major errors in entities and relations, FACTKB is capable of identifying inconsistencies and enforc- ing strict factuality standards.
FactCC (Kryšciński et al., 2020) is an entailment-based factuality metric trained syn- thetic data evaluating factuality with binary clas- sification. FactCC+ is variant of FactCC pro- viding explanations. FactCC+ an enhanced version trained with human-annotated data.
FactGraph (Ribeiro et al., 2022) is entailment-based factuality metric based jointly analyzing the textual content and AMR graphs of the summary and article. FactGraph- adapters an enhanced version with pretrained adapters for both the text and graph modules.
Dataset Details
We present details about the adopted datasets in Table There might be minor differences in certain numbers with the original dataset as re- sult of data preprocessing. FRANK (Pagnoni et al.. 2021) does not explicitly have binary labels such as [FACTUAL, NOT FACTUAL It also does not have training set due to its nature as an evaluation bench- mark. Health Ver (Sarrouti et al., 2021) and SciFact (Wadden al., 2020) originally had NoT ENOUGH INFORMATION labels, while we removed such ex- amples in the out-of-domain factuality evaluation ensure their compatibility with FactCollect.
E LM and KB Details
In Section 5.2, we explored whether FACTKB is compatible with different language models and knowledge bases. For LMs, we used the GOOGLE/BLECTRA-BASE:
ROBERTA-BASE DISCRIMINATOR, FACHBOOK/BART--ASE. ALBERT-BASF--2, MICROSOFT/DEERRTTA.Y..
BASE, DISTILROBERTA- BASE LM checkpoints on Huggingface Transformers. For the six KBs, we used their organized versions: YAGO15k at Lacroix et al. (2019), Wikidata5M at Wang et al. (2021), Atomic West et al. (2022), ConceptNet at Zhang et al. (2022h), KGAP at Feng et al. (2021), and UMLS at Zhang al. (2022b).
D Baseline Details
We present more details about baseline factuality metrics in the following:
BERTScore (Zhang et al., 2019) is general metric for text generation evaluation based on pretrained BERT (Devlin et al., 2019).
F
Statistical Significance Test Details
QAGS (Wang et al., 2020a) is QA-based factu- ality metric, asking questions about summaries and articles while examining whether the answers are consistent.
We use the student (-test for statistical significance analysis throughout the paper. Specifically, the t-test calculator for 2 independent means was adopted for the calculations. We use (*) to denote statistical significance in Tables 2 and 5.
QUALS (Nan et al.. 2021) is QA-based factual- ity metric that uses QAGen (Shakeri et al., 2020) to generate both questions and answers from the summary.
http:///www.occccssatiittss.ss.s studentteest/defalt2..cc

---

All Data
CNN/DM
XSUM
Model
BACC
BACC
BACC
FACTKB-WIKI
89.3 (+0.4)
89.5 (+0.5)
77.3 (=0.3)
88.2 (+0.6)
77.3 (+133)
91.8 (+1.2)
FACTKB-EVIDENCE
89.4 (10.2)
89.5 (40.3)
77.7 (=14)
87.9 (40.7)
76.8 (=19)
90.8 (10.8)
FACTKB-WALK
89.1 (+0.4)
89.3 (+0.5)
78.3 (=1)
87.7 (+0.4)
76.4 (+0.3)
90.4 (+1.4)
BACTKB-COMBING
89.0
89.7
76.0
88.1
74.2
89.1
Table Performance of various FACTKB settings on FactCollect dataset.
Dataset
Datapoint
Class
Class Distribution
Train/Dev/Test Split
Proposed In
FACTCOLLECT
9,567
4994 4573
8667 300 600
Ribeiro al. (2022)
FRANK
2,246
0/671/ 1575
Pagnoni et al. (2021)
COVIDFACT
1,257
401 856
846 94 / 317
Saakyan et al. (2021)
HEALTHVER
4,447
3 -
2,758 1,689
3,340 / 508 / 599
Sarrouti et al. (2021)
SCIFACT
773
3 3 2
508 265
508 56 209
Wadden et al. (2020)
Table Statisties of the datasets and benchmarks adopted in this work.
G Computational Resources
We used GPU cluster with 16 NVIDIA A40 GPUs, 1988G memory. and 104 CPU cores for the experiments. Factuality pretraining with the default hyperparameters takes around 1.5 hours, while fine- tuning language models on the FactCollect dataset takes around 30 minutes.
Scientific Artifacts
FACTKB would be possible without many open-source scientific artifacts, including pytorch (Paszke et al., 2019), pytorch lightning (Falcon and The Toroh Lightning team, 2019), transform- ers (Wolf et al., 2020), sklearn (Pedregosa et al., 2011), numpy (Harris et al., 2020), nltk (Bird et al., 2009), and the six adopted knowledge bases (Pel- lissier Tanon et al., 2020; Vrandečié and Krötzsch, 2014; West al., 2022: Speer al., 2017; Feng al., 2021; Zhang al., 2022b). We commit to making our code and data publicly available upon acceptance to facilitate reproduction and further research.

---

QAGS
DAE
FactCC B Gold
Summary
Article
0.3000 0.9990
1.0000
0.0035 0
plans to build a new gener- ation of royal navy frigates on the isle of wight have heen suhmit- ted to the gov- erument.
The decoemieetoond Type 22 figans HMS Cunberland, HMS Camphchown. HMS Caotaan and HMS Contwall careauty mocred Portumooth Hus- Bidders unil 23 Jnmarr register intersst the forner Deverpprr- ships The BSCC undemttnil propueals to preueree ships submatted These have registered as inerest are ltzalsing their bids vice- ings set tike place in Rebeaary and Murch A decision not expected miil uprng The govemment' nn Diquoal Servioes Autheriyy which handling the sak, wanss to award at least one of the trigates to U.K ship recycler to deter mine the capacity of the UK induery in the heeld Penny Mondamn Comerraaive MP foe Foctsmouth North. said was impoctinn UK recyclers had the chunce to prove Bensdlivs in the field but she was also keen so at lesst one of them sood inen the sempoadd She silied: Ter that has servdd you've licerity been through noble life "My prefareace go the seef and you get to get vahur the budgs the country through touism "The Ministry Defesce has presioudly ssid will "cumaddrr all opiions for the frigxies to ensure "bes frromcial retum for the uapuyer A spoleswoman would not comment on the aumber nature of the bids received due to "commneeidl semuiiiity DDiiinalll designed is a specialist ant-- sibroetna ship. the Type 22 tinae meblvd into powerful eirface combetont with substaaiial anti surfee. ant submanine and ane sircraft eapons systems They were also knowa for having excellent commtard and connoo, and commutication tacilites, making them ideal fagships deptoyments, with complemens of about crew your. sirenft cariee HMS Ark Royal sold scrap for
it's yourr ve
would gmnrate
0.5333 0.9296 1.0000 0.0043 0
an elephant
The stote got past eleptast's fauce and disch sepanating the uninal visturs, the aoo said in atvemmnt. The gil was taken hosptad and within a few houn, the zon addedd The zoo sinement said the enclosure met imemational
has been hit by stone zoo
staidards and said this kind of accsdent is met, ungredictalle and unusaal AAlica Live: More on this and other stortes the statement went on (nn Prenchi to point out two odies recent incidents in the US Plyllis Les, Scientifee Director of the Amboaali Trust fir Flephanss,
in western france after it was hit by tree.
thee tangtted thrownng of venns and beanches by when exepharts irsstnned bored In echilsting You can't prodict animals capisity da The after girl
elephants very unuurll "I elephast direclly targeting the
opisios,
Rabat Zou Tuesday were lilmed a bystander uplonded onto YouTuhee The video shows dlephure waving its trink hehind lence and swerves round to show a sione on the ground Metres sway people are gathered around the gal, hokinng her head and saoking he: leg.
0.6000 0.9991 1.0000 0.0037 0
a woman has been arrested after a fire broke out in a restaurant greater manchester city centre, police have said.
The victim was gatuing for food on the brunch in George's 's Sineet, Contebbrr about 02:15 GMT on Frday when the assault occured accured Investigating officess said three men entered the restuuratt and begun being oaisy and bumping into people believed images relessad Bany said: "Roeturnaely fine wus pui quickly the voctim was not scriussly bur, things could cleatyy have tumedd out much wass: "This was staay and extermely dangeroox thing to dis and urge anyone who mongrises the men in the OCTV imagas in cxmrac me soon de possibie"
the group thes set light
hairOfffcess
they
speak regunding
0.8000 0.9974 1.0000 0.0044 0
tata steel has confirmed it is in talks with the company about selling its long products division.
The tirm said had signed a Letter of Insent to eneer into exclusive negocations with libett House Group Morn than 1,700 people employai in the division, which
Rotherhum Stocksbridge Sted
wekomed news aegotiatioss following "mondis of usim's ginnil
South Yorkshin
try Rey Rickius said "This is postiive sep the UK steel industyy however there remain hage challenges which govemusest must addresi 'The union said
would be sooking urpent talls their plass were fox investmers, protecting jobs sad providin desert persints for membrr in mninemem: Tha Stral's a UI lus Bimbondr lkk ssid the ammunnement
libert Huuse Gruup and wonld toking what
was "an important step forwand". "We now look forward to working with Liberty the due diligence and other wock steams that the sale can be successfully conchsindd." " he said. The Spaciality Steals unit makes high-end components for the autommtivve arrospace and oil industnes Apri. Taia sold divisio,, based Sounthagee Greybull Captall. UK based investmeat tinu.
products
0.3000 0.9990 1.0000 0.0058 0
the site of new burial site in oxford has been approved by the city council.
Oxford City Council said the money mostly been used for "goound investi- pations possible sitee" but nonchere suitalle had been found. Two cometerios still have space. in Walveroote and Botley. but they are expected to full by 2018 and 2021 The comeiil said it had not gaven xp and was 'sall exploring op- tinns" Linds Smith, boand member for leivure, parks and sport, said the council has been searching for a suttalle new hural site for many years She sadeed: "But uhimandly, as with new housing sites, have run out of sunable land within Oxford SS the council owied
identited have, following
investigations and surveys, had discoutted "Bither due reericting the council and
the the groand conditions. high uater talle "After reopening fumily plots, use of dow reserved plons and the inserment of salos would be proable The last incresse burial space Diffnd was in 1952.
remaining cemeteries
Table 9: Qualitative analysis of FACTKB and existing factuality metrics, part 1.

---

QAGS
DAE
FactCC FactKB Gold
Summary
Article
0.6000 0.9886 1.0000 0.0037 0
plans to
Three Victorin lotels will go to make way sitoateryy, four stur hotel twor seiseei-linng spatiment Nlccks, a Fast Chill in Boumemusth Fnglish imnge stoolgly objecsed to the scale of the developmen: what a designated consevation area But, conncillos voted sevan three fuvour saying would help turriam Chair of the plaming besrd and Conservative word cuuncillee David Kebsy, said the buidings carmarked for demolition were nice but no longer "nec maarlly functional" "They's 've come the and of their working lives, we to while improing living regietable
demolish and demolish parts of seaside resort and build more than 1. 000 old buildings have been approved.
the toutie said The loss of buildiag along with stoeey seven sorey buildng, compriing 122 auistd--liing apecttients Applicats East Project will demollish
people the he staad
"The
ger so
Ray View Cuun. The Cuntoneeal and the Oenan batele The council recrival
246 leners supporting the plans Focty-iiee reiden and the Anciont Masumeuts
Sucisty wnete to objic to the demulition eating that deoqin bring aberad they still "gave sense of the histoeic character of the arto' Linglish Heriaage saad the scale of the develogment would came "evvrr hara" the couservation crea.
0.6000 0.9928 1.0000 0.0059 0
russiaš new president has called for new law to allow russian citizens to be harred from leaving the country.
Just Russa pit forward both hill, and linked them directly sitution Slanait,Seppaaiii and pro Russia Crimes eegion which
Pro Krenlin
strong Ukzaite's facus the Rusisnn say reficcidum a plea from conitsy'ss leaders simes There aee shrady many Ruscin citiens Crimes Sevtstope. hase of the Rusian Black Sea Fleei majonty hold Russia passpons Under Russia's exintig law, a naighooning state would have so siga a treary with Russia to al- low past of iis tarittry to become a now "ubjooe" of the Rossin Felentioo But Mikhai Venelyunov, depony leader of A Just Rassia, said the hiw had bern drafied
eoough singer
grovi-
for pezcetul times, and did not far enough for sttutions where state conditions
failing whose geople expresed clese beeskawoy Georgion temiories Grorpta
neighbouring
dhsnttgraring accept
Russiun Fedes atton tinild restikted doiie Rusin"" Since
and South Ossotia have come under Moscow's costoLRussia pozsed trogs
ims segions to help peo-Bassian sepanaisss who not rocognise Goorgia's authority The other will in he considened by the Dama- Ruoda's i lover homse would agead up the proceduncs for soaing Russian prosports Prosport apelicants would not have to a sone tas, and presioss residence Rascia would no linger be required la oddition, they would not have have sufficieut funds to suppoct themsclees and uould not have to give their Urrainian ciizznnhig The bill's preambl says it's simed "sl aupperinng who are
geosper
frstemal people Ukraine, espocially the 'bown threat" uniforn. bill would
Russan
who were Ukrainaas apply for Rusiaa passports at Russian diplomatic missious before August, and they could hecome citinns ifler twa months, instead of watling a year, isis cammiiy the numn The plm hive a now fast-track procodure for iscaing Russian passport was amnounced in Sevasooel on Thunday by A Just Russia leader Sergei Minooo, clacing Russian celebrities furmer Olympic for skating champion Irisa Redrins, torner cosmenaut Valentia Itrestiova and hearyweight boser Nikoln Valuev.
refarenes World Two
Russian MPs have also gone to Critica, in-
0.7000 0.9984 1.0000 0.0047 0
19 ye-yearoold man has been arrested in connection with the fatal shooting of an 18 y8-year- old student in the southern indian state of
The shooting occures bostel atached
Pragudi Residential School
Bargalose city say illegel gumman, dlanifeed Mabesh, working school Iniskmis gin crime schooli colligos are very nes. It not clear what prompoed the shooting Police said on Thussay Maheila had semanded witll 12 April Mabesh alleged to luve hurped into the of B-yoe-oll Cantami and
in the head with pisol on on Tuesday evening He then shot another student, Sinish, who suffered sever injerias hut is belisveal to out of dangee, soy polise He was arosted on Wednesday after manbunt India has stiic control laws, altooigh large number achoothoy was shat doad
feuds are scttled with firea In 2007, 14 twn fellow students school compus near the expisal, Delli.
0.5000 0.9995 1.0000 0.0036 0
police have appealed for help to trace two men who threatened woman with a knife at quarry in fife.
The enteted Office Quarywaad Avence. the Barmulloch 07.55 on Frilay They thecttene monsy have been buill shaken otherwise wnhurnedd the ondeal are while, and oe of them was about 4000 yemm old with short brown hair wearing black jumper Dei Sgi Raymond Hunter said officers had beea carrying out doa-te- to-dooe inquirizs and were in the process of collecting CCTV imiges from the crnunding seca. He wided:: "There ane number of other shups in this area and people may have seen the two men price after the incident. "I am therefore appealing azyonz who was die seea local cesidents connact any anformation you could asist enquiy."
with kulfe and demanded
esciping
cads 27-yooe-old weeker
by police
suspects
0.5000 0.9999
1. 1.0000 0.0063 0
the numher of people using plastic carrier bags in england has reached record high.
The Depertment for Fnvinumment, Food and Rumal Affains found sumber gone up by 200 million since 2013.There been big problem with plastic carterr bags the last fow yeurs many of thess can't recycled
oftes
thewnn mvey after they love herm used bugs end up in mibbich chmps and even rven causing big pribtems for the enstronment. From October people in England will love io pay Sp for their placric bugs in bid to encruragg them to masse the ones that they already have Supermariats
Wales, Scotland and Northern licland, where geople clurged for cartir bags. have all sees deccesse bags mool Compuiggees ard hoping the change Fagland will looan the amouni hags being away,
the envinonment.
Table 10: Qualitative analysis of FACTKB and existing factuality metrics, part 2.


Reviews:
[{"review_id": "GR5mIOFxzL", "reviewer": "Reviewer_ttnU", "summary": "", "questions": "A: Were the baseline results in Tab2-Tab4 obtained directly from the reported results in previous research? If so, it is necessary to indicate this in the report.\n\nB: Regarding the significant test in Tab2 and Tab4, was it conducted to compare Vanilla RoBERTa with FactKB-*? Additionally, were there any significant differences observed among the three models: FactKB-Wiki, FactKB-Evidence, and FactKB-Walk?\n\nC: In Appendix B, it is mentioned that combining the three strategies did not yield better results than using single strategies. Could you provide any explanations for this finding, considering that each of the three strategies has its own specific focus?\n\nD:  In Figure 3, how was FactKB-evidence being trained on other knowledge graphs that are not aligned with Wikipedia? How are the training data constructed?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Particularly, it needs to provide more details about the experimental setup, such as the training data used for each baseline model, to convince readers that the comparison is fair.\n\nBelow we provide details on our baselines and experimental setup. We will expand the experimental setup in the baseline paragraph of Section 3 in the revised version to include these details.\n\nIn Table 2, FactKB and baselines are first pre-trained on respective synthetic data (if any), then fine-tuned on the FactCollect training set, and finally evaluated on the FactCollect test set. We report results on the full test set (all data) and the two subsets (CNN/DM and XSUM).\n\nIn Table 3, the above trained models are evaluated on the test set of the FRANK benchmark.\n\nIn Table 4, FactKB and baselines are first pre-trained on respective synthetic data (if any), then fine-tuned on the training sets of CovidFact, HealthVer, or SciFact, and then evaluated on their respective test splits.\n\nIn Figure 4 and Table 5, we use the FactKB and baselines trained on the FactCollect benchmark (Table 2) and evaluate with the official FRANK script.\n\nIn short, the above experimental setups and training/test data splits are designed for fair comparisons against strong baselines.\n\n> Were the baseline results in Tab2-Tab4 obtained directly from the reported results in previous research? If so, it is necessary to indicate this in the report.\n\nFor Tables 2 and 3, the results for QAGS, QUALS, FactCC, FactCC+, FactGraph, and FactGraph-adapters are taken from the FactGraph work [1] as the datasets and splits (Fact collect and FRANK) are identical. For other baselines in Tables 2 and 3 and all of Table 4, we conducted experiments to evaluate existing model checkpoints if available (e.g., FactCC and FactGraph) or train from scratch.\n\n[1] Ribeiro, Leonardo, et al. \"FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations.\" NAACL 2022.\n\n> Regarding the significant test in Tab2 and Tab4, was it conducted to compare Vanilla RoBERTa with FactKB-*? Additionally, were there any significant differences observed among the three models: FactKB-Wiki, FactKB-Evidence, and FactKB-Walk?\n\nThe significance tests and asterisks indicate whether the FactKB variant significantly outperforms the best-performing baseline model. In some experiments, we observed statistical differences among the three FactKB variants, such as FactKB-walk significantly outperforming the other two on CNN/DM in Table 2 in terms of BACC scores, but in most experiments, they all have similar performance.\n\n> In Appendix B, it is mentioned that combining the three strategies did not yield better results than using single strategies. Could you provide any explanations for this finding, considering that each of the three strategies has its own specific focus?\n\nWe hypothesize that this is because the advantages from the different knowledge reasoning augmentations could be more nuanced and therefore on a general benchmark like FactCollect their difference is marginal (as similarly indicated in Table 2). However, in more reasoning-intensive factuality scenarios like fact-checking complex claims [1], combining the three strategies might yield greater improvement over individual approaches, which we leave for future work.\n\n[1] Pan, Liangming, et al. \"Fact-Checking Complex Claims with Program-Guided Reasoning.\" arXiv preprint arXiv:2305.12744 (2023).\n\n> In Figure 3, how was FactKB-evidence being trained on other knowledge graphs that are not aligned with Wikipedia? How are the training data constructed?\n\nFor knowledge graphs other than Wikipedia, we continue to use the Wikipedia Search API to identify entities that are linkable to Wikipedia entries and use its Wikipedia description to construct the training data. The success rate for entity linking varies from 62% to 98% for the five other knowledge graphs, and we use the identifiable entities for dataset construction. One orthogonal improvement is to adopt in-domain retrieval corpora for dataset construction in the evidence approach and we leave it for future work.\n\n> In Figure 3, is the color used to represent the gap between the vanilla and FactKB-*? If this is the case, it would be helpful to include this information in the caption to ensure clarity.\n\nYes, that’s the case. We thank the reviewer for this suggestion and we will definitely include this in the caption.\n\n> In lines 430-431, the citations for the knowledge graphs used are not provided. While they are cited in the appendix, it is also expected that they are cited in the main body of the text.\n\nWe will add knowledge graph references in lines 430-431 for the revised version.\n\nWe thank the reviewer for their insightful questions. We will add all the clarifications above, especially experimental details about baselines, in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper conducts robust experiments, which include three pre-training objectives and an examination of compatibility across multiple knowledge graphs. \n\nIt provides a fine-grained analysis, demonstrating that FactKB mitigates semantic frame errors. Furthermore, the paper reveals that FACTKB enhances performance on both in-domain and out-of-domain scientific datasets.", "reasons_to_reject": "The paper shows comprehensive experimental results, but it lacks sufficient emphasis on highlighting the differences between FactKB and other existing models. Particularly, it needs to provide more details about the experimental setup, such as the training data used for each baseline model, to convince readers that the comparison is fair. This lack of comparison to existing methods can be considered a weakness and a reason for potential rejection.", "questions_to_authors": "A: Were the baseline results in Tab2-Tab4 obtained directly from the reported results in previous research? If so, it is necessary to indicate this in the report.\n\nB: Regarding the significant test in Tab2 and Tab4, was it conducted to compare Vanilla RoBERTa with FactKB-*? Additionally, were there any significant differences observed among the three models: FactKB-Wiki, FactKB-Evidence, and FactKB-Walk?\n\nC: In Appendix B, it is mentioned that combining the three strategies did not yield better results than using single strategies. Could you provide any explanations for this finding, considering that each of the three strategies has its own specific focus?\n\nD:  In Figure 3, how was FactKB-evidence being trained on other knowledge graphs that are not aligned with Wikipedia? How are the training data constructed?", "paper_topic_and_main_contributions": "The objective of this paper is to address the issue of factuality detection, which involves determining whether a summary generated from an article accurately consistent with the original content. Previous research has highlighted that a significant error in factuality evaluation is related to semantic frame errors, specifically concerning entities and their relationships. \n\nTo address this, the main contribution of this paper is the proposal of FactKB, containing three pre-training objectives aimed at improving a model's ability to better represent entities and relationships. These objectives involve further training the pre-trained language models using external knowledge graphs such as Yago by prediction missing entities in a triplet or random walk path. The experimental results demonstrate that models trained on external knowledge graphs exhibit enhanced performance on the overall fact validation dataset, particularly in relation to the semantic frame type. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In Figure 3, is the color used to represent the gap between the vanilla and FactKB-*? If this is the case, it would be helpful to include this information in the caption to ensure clarity.\n2. In lines 430-431, the citations for the knowledge graphs used are not provided. While they are cited in the appendix, it is also expected that they are cited in the main body of the text."}}, {"review_id": "K3oZiKsN2u", "reviewer": "Reviewer_S3Ep", "summary": "", "questions": "1. Evidential fact verification tasks are also research hotspots. Is it possible to use a single model (e.g., evidence extraction, instead of three strategies)  for the current task? By the way, the motivations of Entity and Knowledge walk are not so clear.\n2. Can you give interpretability of the model, e.g, presenting examples to illustrate which tokens or sentences are selected or assigned more attention weights for factuality evaluation?\n3. According to Table 3, 4, it seems that FACTKB-EVIDENCE is not so effective. Maybe you can neglect FACTKB-EVIDENCE?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Evidential fact verification tasks are also research hotspots. Is it possible to use a single model (e.g., evidence extraction, instead of three strategies) for the current task? By the way, the motivations of Entity and Knowledge walk are not so clear.\n\nThe three proposed strategies are orthogonal approaches that could each be used individually for factuality evaluation. In addition, the three strategies have respective pros and cons, evident in the analysis in Figures 3 and 4. The entity wiki strategy is motivated by the task of knowledge graph completion which helps models understand relationships between entities. We discuss this in detail in lines 123-132 for elaboration. The knowledge walk strategy is motivated by the need for multi-hop knowledge reasoning in understanding factuality which enables models to reason across multiple entity chains.  We elaborate this in lines 174-182.\n\n> Can you give interpretability of the model, e.g, presenting examples to illustrate which tokens or sentences are selected or assigned more attention weights for factuality evaluation?\n\nWe refer the Reviewer to the “Tradeoff between Performance and Granularity” section in the limitations discussion. In short, FactKB aims at presenting an efficient and low-burden approach for factuality evaluation, which is especially valuable in the context of large-scale generated text from large language models. Consequently, the interpretability aspect is not emphasized in FactKB by design. FactKB can be used in combination with more interpretable approaches to factuality evaluation [1-2], which are much more compute-intensive. \n\nIn addition, whether the attention weights from a neural language model such as RoBERTa could be considered “interpretability” is subject to debate [3-4], thus we did not include such results and did not claim for interpretability.\n\n[1] Kryściński, Wojciech, et al. \"Evaluating the Factual Consistency of Abstractive Text Summarization.\" EMNLP 2020.\n[2] Takatsuka, Masato, Tetsunori Kobayashi, and Yoshihiko Hayashi. \"Phrase-Level Localization of Inconsistency Errors in Summarization by Weak Supervision.\" COLING 2022.\n\n[3] Jain, Sarthak, and Byron C. Wallace. \"Attention is not Explanation.\" NAACL 2019.\n\n[4] Wiegreffe, Sarah, and Yuval Pinter. \"Attention is not not Explanation.\" EMNLP 2019.\n\n> According to Table 3, 4, it seems that FACTKB-EVIDENCE is not so effective. Maybe you can neglect FACTKB-EVIDENCE?\n\nEmpirically, FactKB-evidence is actually the best-performing model on the FactCollect benchmark in Table 2 (the All data columns). Indeed, it does not perform most strongly in the settings of Tables 3 and 4, showing that there are advantages and disadvantages of all three strategies, which is why we included all the results to best inform readers. Conceptually, FactKB-evidence enables the integration of external information such as Wikipedia and retrieval corpora to augment language models for factuality evaluation. As a result, we kept the FactKB-evidence approach in the paper.\n\nWe thank the reviewer for their insightful questions. We will include the above clarifications in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-organized and well-written.\n2. This paper offeres a comprehensive analysis of limitations.\n3. This paper proposes three strategies of factuality pretraining: Entity Wiki, Evidence Extraction, KnowledgeWalk.\n4. The descriptions of the proposed model and training process are clear and easy to understand.\n5. This paper considers both in-domain evaluation and generalizable factuality evaluation.", "reasons_to_reject": "1. The proposed method is somewhat simple and coarse-grained, and lacks interpretability.\n2. The experimental analysis is expected to be more detailed.", "questions_to_authors": "1. Evidential fact verification tasks are also research hotspots. Is it possible to use a single model (e.g., evidence extraction, instead of three strategies)  for the current task? By the way, the motivations of Entity and Knowledge walk are not so clear.\n2. Can you give interpretability of the model, e.g, presenting examples to illustrate which tokens or sentences are selected or assigned more attention weights for factuality evaluation?\n3. According to Table 3, 4, it seems that FACTKB-EVIDENCE is not so effective. Maybe you can neglect FACTKB-EVIDENCE?", "paper_topic_and_main_contributions": "This paper studies on evaluating the factual consistency of automatically generated summaries. Previous factuality classifiers are not robust to ever-growing information, where the distribution of entities, events, and relations vary greatly across time and domains. To address these issues, this paper develops a novel factuality evaluation model FACTKB with improved factual knowledge representation, focusing on entities and relations. To be more concrete, FACTKB is built on LMs augmented with factual knowledge (Section 2).", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VfgaH3ajpW", "reviewer": "Reviewer_BoeU", "summary": "", "questions": "Could the authors provide some examples of semantic frame errors that the model is able to capture (e.g., from Frank benchmark) where other systems (such as FactGraph-edge) fails in capture? There are some examples of evaluations in the Appendix but they are not fine-grained explained.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Differently to previous approaches such as FactGraph-edge and DAE, the proposed method is not able to detect fine-grained factuality assessment which is an important feature to indicate specific factuality issues compared to a binary label for the entire summary.\n\nWe refer the Reviewer to the “Tradeoff between Performance and Granularity” section in the limitations discussion. In short, FactKB aims at presenting an efficient and low-burden approach for factuality evaluation, which is especially valuable in the context of large-scale generated text from large language models. Consequently, the interpretability aspect is not emphasized in FactKB by design. FactKB can be used in combination with more interpretable approaches to factuality evaluation [1-2], which are much more compute-intensive.\n\nFactKB and more interpretable approaches could also be combined to form a pipeline: given huge quantities of machine-generated text, we could first use the much more efficient FactKB for a round of quick and large-scale detection and evaluation, while leaving the more nuanced instances with low confidence scores for enhanced and interpretable analysis. This pipeline would bring out the best of both worlds, while we leave it for future work.\n\n[1] Kryściński, Wojciech, et al. \"Evaluating the Factual Consistency of Abstractive Text Summarization.\" EMNLP 2020.\n\n[2] Takatsuka, Masato, Tetsunori Kobayashi, and Yoshihiko Hayashi. \"Phrase-Level Localization of Inconsistency Errors in Summarization by Weak Supervision.\" COLING 2022.\n\n\n> The paper focuses on improvements on semantic frame errors (related to entities, events and their relations) but lacks a deeper exploration of this improvement. Some use cases / qualitative analysis could bring more insights to be discussed.\n\n> Could the authors provide some examples of semantic frame errors that the model is able to capture (e.g., from Frank benchmark) where other systems (such as FactGraph-edge) fails in capture? There are some examples of evaluations in the Appendix but they are not fine-grained explained.\n\nConsider the following example:\n\n**Summary**: an elephant has been hit by a stone at a zoo in western france after it was hit by a tree.\n\n**Document**: The stone got past the elephant’s fence and a ditch separating the animal and visitors, the zoo said in a statement.The girl was taken to hospital and died within a few hours, the zoo added.The zoo statement said the enclosure met international standards and said \"this kind of accident is rare, unpredictable and unusual\".Africa Live: More on this and other stories. The statement went on (in French) to point out two other recent incidents in the US:Phyllis Lee, Scientific Director of the Amboseli Trust for Elephants, says that targeted throwing of stones and branches by elephants is very unusual.\"It can happen when elephants are frustrated or bored. In my opinion, it’s unlikely the elephant was directly targeting the girl - but exhibiting\nfrustration. You can’t predict what animals in captivity will do.\"The moments after the girl was struck at Rabat Zoo on Tuesday were filmed by a bystander and uploaded onto YouTube.The video shows the elephant waving its trunk behind a fence and swerves round to show a stone on the ground.Metres away people are gathered around the girl, holding her head and stroking her leg.\n\n**Analysis**: While it was the girl who was hit by a stone, the summary incorrectly claimed that the elephant was hit by a stone, confusing the entities and relations between stone, girl and elephant and making a semantic frame error. While existing approaches (QAGS, DAE, FactCC, FactGraph) evaluate this summary as factual (giving > 0.5 factuality scores), FactKB gives a score of 0.0035, indicating that the summary is not factual.\n\nWe will include a qualitative analysis section in the additional page, dedicated to fine-grained analysis of summaries and documents such as the above example and how FactKB improves over existing models.\n\n> Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference (Falke et al., ACL 2019)\n\nWe will include this work in the related work discussion.\n\nWe thank the reviewer for their insightful questions. We will include the above clarifications and a qualitative analysis section in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper leverages the structure of external KB for proposing three factuality pretraining objectives: entity wiki, evidence extraction, and knowledge walk, with the goal of enabling model capabilities such as understanding multi-hop claims and evaluate facts from evidences.\n- The paper provides improvements over existing factuality evaluation on in- and out-of-domain settings.\n- While previous approaches such as FactGraph and DAE rely on parsers as an additional step for evaluation (which increases the inference complexity), the model does not require additional steps which makes the inference easier and faster (but required a pretraining step).", "reasons_to_reject": "- Differently to previous approaches such as FactGraph-edge and DAE, the proposed method is not able to detect fine-grained factuality assessment which is an important feature to indicate specific factuality issues compared to a binary label for the entire summary.\n- The paper focuses on improvements on semantic frame errors (related to entities, events and their relations) but lacks a deeper exploration of this improvement. Some use cases / qualitative analysis could bring more insights to be discussed.", "questions_to_authors": "Could the authors provide some examples of semantic frame errors that the model is able to capture (e.g., from Frank benchmark) where other systems (such as FactGraph-edge) fails in capture? There are some examples of evaluations in the Appendix but they are not fine-grained explained.", "paper_topic_and_main_contributions": "This paper studies the evaluation of factual consistency in automatic summarization. It discusses the challenges of evaluating factual consistency, and proposes a new approach that uses an external knowledge base (KB) to pre-train a classification model to evaluate factual inconsistencies in summaries. \n\nIn particular, it introduces three pretraining objectives that leverages connections/relations between entities in the KG which are based on (i) entity-specific facts (ii) facts aligned with relevant evidence – e.g. a paragraph – and (iii) random walks with traverse the KG in a multi-hop setting. The model is pretrained using an encyclopedic KG based on Wikidata.\n\nThe model is further fine-tuned in a human-curated dataset for factual consistency assessment and evaluated in-domain (news summarization) and on out-of-domain (scientific literature) datasets. The results indicate that the model improves over previous work for factual consistency evaluation, improving correlations with human annotations and metrics such as BACC and F1.\n\nThe paper further investigates the performance of the method using other KGs indicating that such pretraining can help to improve factuality evaluation compared to vanilla pretrained models.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference (Falke et al., ACL 2019)"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses an important issue of factuality, and propose a novel approach to encode factual knowledge during pre-training. It convincingly shows the merits of their method on both in- and out-of-domain datasets. As suggested by reviewers, I hope that the authors include more qualitative analysis of FactKB in the next version of the paper.
Decision: Accept-Main