ALIGNSCORE: Evaluating Factual Consistency with A Unified Alignment Function
Year: 2023

Abstract:
Many text generation applications require the generated text to be factually consistent with input information. Automatic evaluation of factual consistency is challenging. Previous work has developed various metrics that often depend on specific functions, such as natural language inference (NLI) or question answering (QA), trained on limited data. Those metrics thus can hardly assess diverse factual inconsistencies (e.g., contradictions, hallucinations) that occur in varying inputs/outputs (e.g., sentences, documents) from different tasks. In this paper, we propose ALIGNSCORE, a new holistic metric that applies to a variety of factual inconsistency scenarios as above. ALIGNSCORE is based on a general function of information alignment between two arbitrary text pieces. Crucially, we develop a unified training framework of the alignment function by integrating a large diversity of data sources, resulting in 4.7M training examples from 7 well-established tasks (NLI, QA, paraphrasing, fact verification, information retrieval, semantic similarity, and summarization). We conduct extensive experiments on large-scale benchmarks including 22 evaluation datasets, where 19 of the datasets were never seen in the alignment training. ALIGNSCORE achieves substantial improvement over a wide range of previous metrics. Moreover, ALIGNSCORE (355M parameters) matches or even outperforms metrics based on ChatGPT and GPT-4 that are orders of magnitude larger.1

heading: 1 Introduction
text: Recent systems for natural language generation, such as summarization and dialogue systems, can produce fluent and coherent text. However, studies show the generated text can often contain factual consistency errors, such as contradictions with input information or hallucinations irrelevant to the
1Our code is available at https://github.com/ yuh-zha/AlignScore.
context (Cao et al., 2018; Kryscinski et al., 2019; Nie et al., 2019a; Tan et al., 2020; Maynez et al., 2020; Deng et al., 2021).
It is thus crucial to develop automatic metrics that evaluate factual consistency of a claim (e.g., generated text) with regard to a context (e.g., model input). The evaluation, however, has long been a challenge. Recent work has devised various metrics based on specific pretrained functions, such as natural language inference (NLI) (Honovich et al., 2022a; Mishra et al., 2021; Kryscinski et al., 2020; Utama et al., 2022; Laban et al., 2022) and question answering (QA) (Durmus et al., 2020; Fabbri et al., 2022; Honovich et al., 2021; Fabbri et al., 2022). Specifically, an NLI-based metric measures if the claim is entailed by the context; while a QA-based metric first creates (question, answer) pairs from the claim and then checks if answering the questions with a QA model conditioning on the context will lead to the same answers.
However, by relying on specific functions trained with only narrow data (i.e., NLI or QA datasets), previous metrics have limited generalizability and fail to apply to diverse evaluation scenarios, including different types of factual consistency errors and varying lengths and characteristics of contexts/claims from different tasks and domains. For instance, a metric trained exclusively with NLI data of sentences in a certain domain tends to have difficulty in evaluating summaries of long documents in a different domain (Mishra et al., 2021; Laban et al., 2022). The limitations motivate a more holistic metric that develops a general understanding of factual consistency and generalizes to diverse evaluation scenarios.
In this paper, we propose ALIGNSCORE, a new general factual consistency metric based on a unified text-to-text information alignment function. In particular, we unify a wide range of data sources, and use the massive diverse data to train a general information alignment model that estimates
11328
an alignment score given two arbitrary text pieces. More specifically, we reformat and aggregate 15 datasets from 7 popular language tasks, including NLI, QA, paraphrasing, fact verification, information retrieval, semantic similarity, and summarization. This results in a total of 4.7M training examples with diverse characteristics, and yields an alignment function with great generalizability. We then build ALIGNSCORE using the alignment function as a building block. In particular, to handle long text and accommodate the different roles of context and claim, we develop a splitting strategy that breaks a context into coarse-grained chunks and a claim into fine-grained sentences. Aggregating the alignment scores between context-chunks and claim-sentences leads to the final factual consistency score.
In our experiments, we build ALIGNSCORE by finetuning the lightweight RoBERTa models (125M and 355M) for alignment. We evaluate ALIGNSCORE on the latest large-scale evaluation benchmarks, including SummaC (Laban et al., 2022), TRUE (Honovich et al., 2022b), and other testbeds, which contain a total of 22 challenging evaluation datasets. Our approach substantially outperforms previous state-of-the-art metrics in terms of different quality measures. Notably, our metric (355M) is on par with, and sometimes even much better than latest metrics based on orders-ofmagnitude larger language models (e.g., ChatGPT and GPT-4). In particular, ALIGNSCORE shows strong generalizability on the 19 zero-shot datasets that were never seen during the alignment function training. We also conduct extensive ablation studies to demonstrate the effectiveness of the context splitting strategy and other modeling choices.

heading: 2 Related Work
text: Factual Consistency Metrics Traditionally, generative systems are evaluated using n-gram based metrics (Papineni et al., 2002; Lin, 2004; Banerjee and Lavie, 2005; Popović, 2015). Recently, factual consistency metrics are often use task-specific language understanding capabilities, such as NLI and QA. To improve performance when evaluating generative tasks with long texts, NLI-based metrics adopt training sets with long premises (Honovich et al., 2022a; Mishra et al., 2021), use large synthetic datasets (Kryscinski et al., 2020; Utama et al., 2022), or use sentence level evaluation (Laban et al., 2022). A separate line of research formu-
lates factual consistency evaluation as QA (Durmus et al., 2020; Fabbri et al., 2022; Honovich et al., 2021; Fabbri et al., 2022). Other consistency evaluation methods that use pretrained language models (LMs) include embedding matching (Zhang et al., 2020; Deng et al., 2021), finetuning LMs to directly regress human evaluation scores (Sellam et al., 2020), and using LMs to score candidates based on weighted log probability (Yuan et al., 2021; Liu et al., 2022). CTC (Deng et al., 2021) develops a suite of text generation evaluation metrics based on the similar concept of alignment. Yet we define alignment in a more general way to enable integration of diverse training data, and deliver ALIGNSCORE as a more effective metric focusing on factual consistency. Concurrent work proposes to combine large language models (LLMs) with prompting to evaluate different aspects of generated text, including factual consistency (Fu et al., 2023; Liu et al., 2023; Gao et al., 2023). Our proposed ALIGNSCORE shows stronger performance with a much smaller model size.
Unified Training Recent work converts related but different tasks into the same input-output format to train unified models. Raffel et al. (2020) propose to unify text generation tasks into a text-to-text conditional generation problem. Sanh et al. (2022) further show that the text-to-text generation framework, combined with natural language prompting, improves zero-shot task generalization to unseen tasks. Zhong et al. (2022) develop a unified automatic evaluation metric by framing different aspects of NLG evaluation as a Boolean Question Answering problem. Recent studies also present task unification as an effective approach to improve model performance and generalizability in multimodal tasks (Xie et al., 2022; Zhang et al., 2021; Wang et al., 2022).

heading: 3 Methods
text: We introduce the ALIGNSCORE metric built on top of a unified alignment function. We first train the alignment function by unifying a large diversity of data sources (Section 3.1). We then define ALIGNSCORE by combining the alignment function with a new context/claim splitting and aggregation strategy (Section 3.2).

heading: 3.1 Unified Alignment Function
text: Given two pieces of text a and b, we consider b to be aligned with a if all information in b is present

heading: Unified Alignment
text: in a and does not contradict a. Conceptually, we model information alignment as a function that maps the text pair (a, b) to a label y that characterizes the level of alignment:
f : (a, b) → y . (1)
A holistic and generalizable alignment function must account for all types of consistency errors, domains, and data distributions. Therefore, in order to learn the alignment function, we want to adapt and aggregate diverse language tasks to form a unified alignment training corpus (Figure 1). In this work, we collect 15 datasets spanning 7 wellestablished tasks, including NLI, fact verification, paraphrase, semantic textual similarity, QA, information retrieval, and summarization. We present an overview of these datasets in Table 1 and include more details in Section A.1 and A.2 in the appendix.
The vast diversity of input/output formats across the above tasks poses significant challenge for unifying them into a uniform alignment training corpus. To unify input formats, we convert each sample into a text pair (a, b). For tasks that do not cleanly fit into the text pair format, such as QA (where each sample contains a question, an answer, and a context) and information retrieval (where each sample contains a query, an answer, and a supporting document), we use a sequence-to-sequence model (Song, 2022) to convert the question answer
pair into a single declarative sentence (underlined items in Figure 1; See Section C.1 for examples).
To unify output formats, while it is possible to transform all tasks into binary classification, instead we convert them into a set of related alignment problems to preserve as much information as possible from the original datasets (Figure 1). Specifically, we devise 3 options for the alignment label y:
ybin ∈ {ALIGNED, NOT-ALIGNED}, y3way ∈ {ALIGNED, CONTRADICT, NEUTRAL}, yreg ∈ [0, 1].
More concretely, for tasks that come with discrete labels, depending on their setup, the alignment function predicts either the binary classification label ybin (paraphrase, QA, information retrieval, and summarization) or the 3-way classification label y3way (NLI, and fact verification); for tasks with continuous labels (semantic textual similarity), the alignment function predicts the regression label yreg. Here a higher yreg indicates that more information in b is supported by a.
We build the alignment model consisting of a language model (e.g., RoBERTa; Liu et al., 2019) and 3 individual linear layers as the 3-way classification (y3way), binary classification (ybin), and regression (yreg) heads. First, we feed into the language model the concatenation of the text pair (a, b) and use the contextual embedding of the special begin-of-
sentence token as the encoded representation, h. Then, the classification and regression heads map h into an estimation of y3way, ybin, and yreg through logistic regression and linear regression, respectively. We use cross entropy loss for both 3-way and binary classification, and mean squared error loss for regression. The joint loss function is:
Ltotal = λ1L3way + λ2Lbin + λ3Lreg, (2) where λ1, λ2, λ3 are scalar weights. In our experiments, we set λ1 = λ2 = λ3 = 1.

heading: 3.2 The ALIGNSCORE Metric
text: As the definition of factual consistency is closely related to the information alignment problem, one naive way of building a factual consistency metric is simply using the alignment model to estimate the alignment score of the text pair (context, claim). However, this approach (also referred to as "document level evaluation"; Laban et al., 2022) has several drawbacks.
First, generative tasks often contain long inputs, especially long contexts, that go beyond the input length limit of a language model (e.g., source documents in summarization tasks can easily exceed the 512-token limit of a RoBERTa model). Consequently, if long inputs are not explicitly handled (Kryscinski et al., 2020; Mishra et al., 2021), language-model-based metrics could silently drop important information because of truncation.
Second, information contained in a claim often spreads across multiple sentences in the context. To verify the factual consistency of a claim, a metric needs access to long context spans. Therefore, evaluating the claim against individual context sentences (as in previous sentence level evaluation; Laban et al., 2022; Amplayo et al., 2022) can degrade metric performance as paragraph- and documentlevel semantic information is lost.
Third, humans typically assign consistency
scores in a continuous spectrum that reflect the amount of consistency errors in the samples. Similarly, good metrics should produce fine-grained scores. Unfortunately, as classification tasks make up most of the training data (only semantic textual similarity datasets provide continuous labels), our alignment model tends to assign scores close to the two extremes, limiting its effectiveness if used directly as a factual consistency metric.
Conceptually, to resolve the first challenge, we need to split the context into chunks such that when concatenated with a claim, the resulting sequence does not exceed the input length limit. By picking a large enough chunk size, we allow the model to reason over longer context spans, mitigating the second issue. Since sentences in a claim tend to be self-contained statements, an effective way to make the metric produce more fine-grained scores is to evaluate claim sentences independently of each other (Laban et al., 2022). Specifically, for each sentence in the claim (green rectangles in Figure 2), we evaluate it against all context chunks (yellow rectangles in Figure 2) using the alignment function. Then, we select the highest alignment score (lines labeled with numbers in Figure 2) for each claim sentence. Intuitively, this step identifies the context chunk that most strongly supports each claim sentence, and the highest score reflects how well the claim sentence is supported. Finally, we use the average value of all highest scores as the factual consistency score. This addresses the third challenge, as taking the average prevents a single inconsistent claim sentence from dominating the final score. Alternatively, the average value of highest scores can be roughly interpreted as "the proportion of the claim that are factually consistent with respect to the context", which naturally leads to a more fine-grained metric. As we show in experiments, our novel chunk level evaluation method consistently outperforms document level (which risks truncation) and sentence level evaluation.
We formally define ALIGNSCORE as:
ALIGNSCORE(o, l)
= mean j max i
alignment(o′i, l ′ j) , (3)
where o is the context, l is the claim, {o′i} is the set of context chunks, {l′j} is the set of claim sentences, and alignment(·) is the probability of the model predicting the ALIGNED label in the 3-way classification setting. In practice, for RoBERTa
models (that have an input length limit of 512 tokens) we split the context into chunks at sentence boundaries such that each chunk contains roughly 350 tokens. We use the output of the 3-way classification head, our ablation studies reveal that it performs better than the binary classification head and the regression head (Section 4.5).

heading: 4 Experiments
text: In this section, we evaluate ALIGNSCORE on a wide range of benchmarks and show it consistently outperforms existing metrics (Section 4.1- 4.4). We also conduct extensive ablation study in Section 4.5.

heading: 4.1 Implementation
text: We use RoBERTa (Liu et al., 2019) to implement the alignment model. We denote ALIGNSCORE based on RoBERTa-base/large as ALIGNSCOREbase/large.
We follow common practice (Liu et al., 2019; Devlin et al., 2019) and train the model for 3 epochs with a batch size of 32 in all the experiments. Training samples are randomly sampled across the converted upstream NLP tasks. Due to resource constraints we only use the first 500k samples in each dataset for training, resulting in a total of 4.7 million training samples. Training details are listed in Appendix A.3.

heading: 4.2 Benchmarks
text: Following Deng et al. (2021), Fabbri et al. (2022), Zhong et al. (2022) and Gabriel et al. (2021), we evaluate factual consistency metrics using TRUE benchmark (Honovich et al., 2022a) (consists of 11 datasets in diverse domains), SummaC benchmark (Laban et al., 2022) (includes 6 large summarization datasets), and a set of other latest datasets including XSumFaith (Maynez et al., 2020), SummEval (Fabbri et al., 2021), QAGS-XSum (Wang et al., 2020), QAGS-CNNDM (Wang et al., 2020), FRANK (Pagnoni et al., 2021) and SamSum (Gliwa et al., 2019).
SummaC benchmark standardizes the task of summary inconsistency detection by casting it as a binary classification problem. Following Laban et al. (2022), we 1) tune the threshold of metrics on the validation sets, and then compute the balanced accuracy (Brodersen et al., 2010) on the test sets, 2) report the AUC-ROC (Bradley, 1997) of each metric. TRUE benchmark covers summa-
rization, dialogue, paraphrase and fact verification tasks. It also assigns binary labels to samples based on whether the entire claim is factually consistent with the context. We report AUC-ROC of each metric following Honovich et al. (2022a). We also collect 6 popular factual consistency evaluation datasets, namely XSumFaith, SummEval, QAGSXSum, QAGS-CNNDM, FRANK and SamSum. We compute instance-level Pearson, Spearman, and Kendall’s tau correlation coefficients between metric scores and human annotated consistency scores.

heading: 4.3 Baselines
text: We compare ALIGNSCORE with state-of-the-art metrics, which we categorize into question answering (QA), similarity matching, regression, NLI, and miscellaneous. We use open-source code and models released by authors. Additionally, we also compare with latest LLM-based metrics.
QA Based Metrics adapt question generation (QG) and question answering (QA) models to automatically evaluate factual consistency. We include the latest QAFactEval (Fabbri et al., 2022), QuestEval (Scialom et al., 2021), and FEQA (Durmus et al., 2020) as our baselines.
Similarity Matching Based Metrics vary in their granularity and matching functions. We re-
port BLEU (Papineni et al., 2002) and ROUGE1/2/L (Lin, 2004), which compute token-level string matching scores. We also include the named-entity level metric NER-Overlap introduced in Laban et al. (2022). BERTScore (Zhang et al., 2020) uses token-level embedding to compute scores, for which we use the best variant (microsoft/deberta-xlarge-mnli) recommended by the authors2. We also use SimCSE (Gao et al., 2021) as sentence-level embedding matching function, with the best released model sup-simcse-roberta-large3.
Regression Based Metrics learn to estimate ground truth scores directly. We use BLEURT (Sellam et al., 2020) with its recommended checkpoint (BLEURT-20)4 as our baseline.
NLI Based Metrics methods also vary in their granularity. We use a RoBERTa-large (Liu et al., 2019) model finetuned5 on MultiNLI (Williams et al., 2018b) as a baseline for document-level evaluation, where the model evaluates a candidate against the entire context. Our baselines also include the DAE (Goyal and Durrett, 2020) met-
2https://github.com/Tiiiger/bert_score 3https://github.com/princeton-nlp/SimCSE 4https://github.com/google-research/bleurt 5https://huggingface.co/roberta-large-mnli
ric, which decomposes text at the level of dependency arcs. For sentence-level baseline, we use SummaC-ZeroShot and SummaC-Conv introduced in the SummaC Benchmark (Laban et al., 2022) and FactCC (Kryscinski et al., 2020) which is trained on synthetic data.
Miscellaneous Besides the above metrics, we also use competitive metrics including UniEval (Zhong et al., 2022), CTC (Deng et al., 2021), BARTScore (Yuan et al., 2021) and BLANC (Vasilyev et al., 2020) as baselines.
UniEval is a unified multi-dimensional metric, capable of evaluating different aspects of text generation. We use the Consistency variant as the baseline. Deng et al. (2021) propose CTC, which is based on token-level information alignment. We use its discriminative variant trained on synthetic CNN/DailyMail (See et al., 2017) (D-CNNDM) as our baseline. For BARTScore, we use the pretrained BART-Large-CNN6 checkpoint.
LLM-Based Metrics Concurrent work proposes to utilize LLMs for NLG evaluation. GPTScore uses the log probability of an LLM generating the target text conditioned on the prompt as the metric score (Fu et al., 2023). G-EVAL first augments its prompts with chain-of-thoughts and then evaluates texts by form-filling (Liu et al.,
6https://github.com/neulab/BARTScore
2023). Gao et al. (2023) uses ChatGPT in place of human annotators in four popular human evaluation setups (ChatGPT in Table 5). As we directly compare with correlation coefficients reported by Fu et al. (2023); Liu et al. (2023); Gao et al. (2023), results on some datasets are not available.

heading: 4.4 Results

heading: 4.4.1 Results on SummaC Benchmark
text: We report AUC-ROC on the test set of the SummaC Benchmark in Table 2. A higher AUC-ROC score indicates the metric is better at detecting factual consistency errors. Our ALIGNSCORE-large achieves the best average performance on the SummaC benchmark, scoring the highest in 4 out of 6 datasets. We also present the balanced accuracy in Appendix (Table 9), where ALIGNSCORE-large also establishes new state-of-the-art results.

heading: 4.4.2 Results on TRUE Benchmark
text: The results on the TRUE benchmark are shown in Table 3, where ALIGNSCORE-large gets the highest average AUC-ROC score. It outperforms baselines on 7 out of 11 tasks while staying competitive on the rest. For a fair comparison, we also report the average AUC-ROC (denoted as AVG-ZS) excluding datasets that the alignment function is trained on (PAWS, VitaminC and FEVER). The per-
formance of ALIGNSCORE remains to be on top, outperforming strong baselines like QAFactEval, UniEval, and SummaC-CONV. This demonstrates ALIGNSCORE generalizes well to unseen data (e.g., DialFact dataset in the dialogue domain).

heading: 4.4.3 Results on Other Datasets
text: We present Pearson correlation coefficients of various metrics on other factual consistency datasets in Table 4. We also report Spearman correlation and Kendall’s tau coefficients in Appendix (Table 10 and 11). The ALIGNSCORE-large metric outper-
Metric Backbone Datasets
SE Q-X Q-C
G-EVAL-3.5 GPT3.5-d03 38.6 40.6 51.6 G-EVAL-4 GPT4 50.7 53.7 68.5 GPTScore GPT3.5-d03 47.5 / / ChatGPT GPT3.5-turbo 43.3 / /
ALIGNSCORE-base RoBERTa (125M) 43.4 51.9 69.0 ALIGNSCORE-large RoBERTa (355M) 46.6 57.2 73.9
Table 5: The Spearman correlation coefficients of ALIGNSCORE and LLM-based metrics on SummEval (SE), QAGS-XSum (Q-X) and QAGS-CNNDM (Q-C). The best models are shown in bold. The results of G-EVAL, GPTScore and ChatGPT are from Liu et al. (2023), Fu et al. (2023), and Gao et al. (2023).
forms previous metrics in terms of overall performance, including the competitive QAFactEval and UniEval metrics, dominating 6 out of 7 datasets. We note that DAE and QuestEval perform better on XSumFaith dataset. Similar to Fabbri et al. (2022), we speculate it is because the relatedness between the token-level annotation of XSumFaith and the fine-grained metrics.
We also compare our metric with LLM-based metrics in Table 5. Result shows ALIGNSCORE has comparable performance with LLM-based metrics on SummEval. And it outperforms LLM-based
metrics on QAGS-XSum and QAGS-CNNDM, showing the capability and efficiency of our proposed metric.

heading: 4.5 Ablation Study
text: To understand 1) which classification head is more suitable for factual consistency evaluation, 2) which splitting method is more effective, and 3) which upstream NLP task contributes the most to the superior performance of ALIGNSCORE, we conduct 3 ablation studies. The experiments in this section are all based on ALIGNSCORE-base.
Classification Head We keep the same splitting method as in Section 3.2 and change the heads that generate alignment scores. We first use the regression head (ALIGNSCORE-base-REG) and the binary classification head (ALIGNSCORE-base-BIN). Then, we compare these two heads with our proposed ALIGNSCORE-base, which adopts the 3-way classification head. We present the results in Figure 3, which shows the 3-way classification head consistently performs better than the regression head and the binary classification head.
Splitting Method Then, we keep the 3-way classification head and change the splitting method. Following Amplayo et al. (2022), we implement SMART-L and SMART-N, and use our alignment model as the sentence matching function. SMARTL uses sentence-level evaluation and aggregates the alignment scores through a soft version of Longest Common Subsequence (LCS), while SMART-N aggregates using greedy matching between Nsentences. In our experiments, we set N=1. We
also implement ALIGNSCORE without any splitting (denoted as ALIGNSCORE-base-DOC) where the inputs are directly fed into the model. The result in Figure 4 shows that our chunk level splitting method performs best compared to the other 3 methods. It demonstrates that our splitting method helps ALIGNSCORE capture salient information from long contexts.
Upstream NLP Task We study the contribution of each upstream NLP task by excluding one task at a time to train the alignment model. The results are shown in Figure 5. When the QA task is removed, the performance of the metric is the worst, indicating QA datasets make the biggest contribution to metric performance. Similarly, fact verification task has the second largest contribution. Surprisingly, with the removal of the NLI task, the model performs better on a majority of benchmarks, showing the NLI task plays a negative role in the training. We speculate that it is because 1) premises and hypothesises in NLI datasets are generally shorter, which differs from most factual consistency benchmarks and datasets, 2) other NLP tasks have larger-scale and higher quality datasets.

heading: 5 Conclusion
text: We propose ALIGNSCORE, a holistic factual consistency metric based on a unified alignment function. To learn the alignment function, we adapt 7 well established language understanding tasks into a unified alignment task , resulting in 4.7M diverse training samples. Experiments show ALIGNSCORE achieves state-of-the-art performance on SummaC and TRUE Benchmark, has higher correlation with human judgements than competing metrics, and generalizes well to unseen data.

heading: Limitations
text: Interpretability. Although ALIGNSCORE shows high correlation with human judgments, it is hard to interpret the reasoning behind its predictions. Therefore, an interesting future research direction is to develop interpretable factual consistency metrics that can accurately identify words or spans in the input that contain factual consistency errors and (or) produce human readable explanations justifying its predictions.
Synthetic data. Our alignment training data contains datasets augmented with synthetic data. While ablation studies show that synthetic data helps improve metric performance, our rule-based method for generating synthetic data could generate noisy data that may not accurately model the error types and distributions produced by real world generative systems. Thus, analyzing the quality of synthetic data and developing more effective ways to generate synthetic data is an interesting research topic.
Language coverage. While we show ALIGNSCORE generalize well to unseen data, it only covers a single language, English. Undoubtedly, factual consistency evaluation is also important for more resource-constrained languages or in a multilingual setting. Consequently, future research could focus on extending the Align metric to multiple languages, including resource-constrained languages.

heading: Ethics Statement
text: ALIGNSCORE is intended as an automatic metric to be used in NLP research. While it has state-ofthe-art performance, it can produce false positives and false negatives, and may not be appropriate for applications other than its intended use. As it is trained on publicly available datasets, the metric might be affected by biases inherent to those datasets.

heading: A.1 Unifying Language Understanding Tasks
text: We adapt datasets from 7 NLP tasks into the information alignment format. An overview of our unified training sets is shown in Table 1.
Tasks that cleanly fit into the form of the alignment problem, including NLI, fact verification, and paraphrase datasets are adapted by mapping the original labels into either binary or 3-way classification alignment labels. Next, we discuss how we adapt semantic textual similarity (STS), QA, and information retrieval (IR) tasks.
STS STS datasets contain pairs of sentences labeled with semantic similarity scores. We use STS datasets in the regression task by normalizing the score to between 0 and 1.
QA A QA sample consists of a context paragraph, a question, and a ground truth answer. One can derive the ground truth answer given the context and the question. To convert QA samples into a format suitable for binary classification, we use a pretrained sequence-to-sequence model to convert question-answer pairs into declarative sentences (Song, 2022; Demszky et al., 2018). Sentences generated from ground truth answers form ALIGNED pairs with corresponding contexts, while sentences generated from wrong options form NOT-ALIGNED
samples. For samples with unanswerable questions, we first use a QA model7 to generate wrong answers, and then turn them into NOT-ALIGNED samples using the above method.
See Section C.1 for converted samples.
IR A sample in an information retrieval dataset consists of a query-answer pair and a list of passages, some of which can be used to answer the query. Similar to QA datasets, we adapt information retrieval datasets for binary classification by converting query-answer pairs into declarative sentences and then pairing them with passages. If a passage can be used to answer the corresponding query, we consider the sample to have ALIGNED label. Otherwise it is assigned NOT-ALIGNED.

heading: A.2 Synthetic Data
text: We further augment our training set with synthetic data based on the WikiText-103 corpus (Merity et al., 2017) and the WikiHow summarization dataset (Koupaee and Wang, 2018).
To generate ALIGNED samples, we create a paraphrase of each sentence in WikiText-103 through back translation using a neural machine translation model (Junczys-Dowmunt et al., 2018). For the WikiHow dataset, we use source documents as text a, and the ground truth summaries together with extractive summaries generated by an extractive summarizer (Barrios et al., 2016) as text b to form ALIGNED samples.
Inspired by recent work in creating factually inconsistent samples (Deng et al., 2021; Kryscinski et al., 2020), we randomly mask 25% of the tokens in text b from the ALIGNED samples and infill with a masked language modeling model (Sanh et al., 2019). The resulting sentences are semantically different from the originals and are used in NOTALIGNED samples.

heading: A.3 Training the Alignment Model
text: We use the Transformers8 library to implement the proposed model, and the PyTorch Lightning framework to train our model.
The alignment model is optimized with AdamW (Loshchilov and Hutter, 2019). The learning rate is first warmed up to a peak of 1e-5, and then linearly decayed. The hyperparameters used to train
7https://huggingface.co/valhalla/ t5-base-qa-qg-hl
8https://huggingface.co/docs/transformers/ index
ALIGNSCORE-base and ALIGNSCORE-large are shown in Table 6.
We don’t split the context and claims into chunks in the training for simplicity.

heading: Hyperparameter ALIGNSCORE-base ALIGNSCORE-large

heading: A.4 Cleaning Evaluation Datasets
text: Certain datasets we use for evaluation contain artifacts that could hurt model performance. Notable issues include claims having escape sequences (-LRB- and -RRB- instead of parentheses) and being uncased (all lower case) while contexts do not have escape sequences and are cased.
We use rule-based methods to remove these artifacts. Specifically, we replace escape sequences in claims with the original characters, capitalize the first letter of the first word in a sentence, and for words that appear in contexts, we fix their capitalization in the corresponding claims according to their occurrences in the contexts.

heading: A.5 Computing Correlations
text: We first split the inputs to sentences with NLTK sentenizer. Then ALIGNSCORE computes the instancelevel factual consistency score as stated in Section 3.2. We use scipy to compute Pearson correlation, Spearman correlation and Kendall’s tau correlation.

heading: B Additional Experiment Details/Results

heading: B.1 SummaC Benchmark
text: SummaC benchmark consists of 6 summarization datasets: CogenSum (Falke et al., 2019), XSumFaith (Maynez et al., 2020), Polytope (Huang et al., 2020), FactCC (Kryscinski et al., 2020), SummEval (Fabbri et al., 2021) and FRANK (Pagnoni et al., 2021). The datasets are standardized by binarizing each labels. Metrics are evaluated as classifiers on SummaC benchmark.
The SummaC Benchmark considers samples in PolyTope with Addition9, Omission10, Inaccuracy Intrinsic11, Inaccuracy Extrinsic12 and Positive-Negative Aspect13 errors to be negative samples. However, Addition and Omission do not imply factual consistency errors. Thus, we only consider samples with Inaccuracy Intrinsic, Inaccuracy Extrinsic and Positive-Negative Aspect errors to be factually incorrect. The reported PolyTope result uses this definition of errors.
We also report balanced accuracy, which deals with imbalanced datasets, in Table 9.

heading: B.2 TRUE Benchmark
text: TRUE benchmark is for evaluating factual consistency metrics in summarization, dialogue, factverification and paraphrasing tasks. There are totally 11 datasets in this benchmark: FRANK (Pagnoni et al., 2021), SummEval (Fabbri et al.,
9Defined as: Unnecessary and irrelevant snippets from the source are included in the summary
10Defined as: Key point is missing from the output 11Defined as: Terms or concepts from the source are misrepresented and thus unfaithful. 12Defined as: The summary has content not presented in the source and factually incorrect 13Defined as: The output summary represents positive statements whereas the source segment is negative, and vice versa.
2021), MNBM (Maynez et al., 2020), QAGSCNNDM (Wang et al., 2020), QAGS-XSum (Wang et al., 2020), BEGIN (Dziri et al., 2022), Q2dataset (Honovich et al., 2021), DialFact (Gupta et al., 2022), PAWS (Zhang et al., 2019), FEVER (Nie et al., 2019b; Thorne et al., 2018) and VitaminC (Schuster et al., 2021). TRUE also treats factual consistency evaluation as a binary classification task and reports AUC-ROC.
The full names of the datasets in Table 3 are listed in Table 7.

heading: B.3 Other Datasets
text: In addition to the Pearson correlation reported in Table 4, we also report the Spearman correlation and Kendall’s tau correlation on 9 datasets in Table 10 and 11, respectively. The full names of the abbreviations in Table 4, Table 10 and Table 11 are listed in Table 8.

heading: B.3.1 Why BLEU Metric Performs Relatively Well?
text: We notice that the BLEU metric has comparable performance with some neural model based methods, which seems to contradict some previous findings. We attribute it to the case matching in the pre-processing, since BLEU is case sensitive.

heading: C Sample Training Data

heading: C.1 Converted QA Samples
text: We show converted SQuAD v2 (Rajpurkar et al., 2018) samples below to illustrate the process of converting QA samples into the alignment format (discussed in Section A.1). Concretely, questions and answers are combined into declarative claims using a sequence-to-sequence model (Song, 2022; Demszky et al., 2018).
Context: The Times Literary Supplement (TLS) first appeared in 1902 as a supplement to The Times, becoming a separately paid-for weekly literature and society magazine in 1914. The Times and the TLS have continued to be coowned, and as of 2012 the TLS is also published by News International and cooperates closely with The Times, with its online version hosted on The Times website, and its editorial offices based in Times House, Pennington Street, London.
Question: The editorial offices of The Times Literary Supplement is based in what location in London?
Answer: Times House, Pennington Street Generated claim: The editorial offices of The
Times Literary Supplement is based in Times House, Pennington Street in London.
Label: ALIGNED
Context: The 25,000 cotton growers in the United States of America are heavily subsidized at the rate of $2 billion per year although China now provides the highest overall level of cotton sector support. The future of these subsidies is uncertain and has led to anticipatory expansion of cotton brokers’ operations in Africa. Dunavant expanded in Africa by buying out local operations. This is only possible in former British colonies and Mozambique; former French colonies continue to maintain tight monopolies, inherited from their former colonialist masters, on cotton purchases at low fixed prices.
Question: How many subsidized cotton growers are in the US?
Answer: 25,000
Generated claim: 25,000 subsidized cotton growers are in the US.
Label: ALIGNED
Context: On October 28, 2015, IBM announced its acquisition of digital assets from The Weather Company—a holding company of Bain Capital, The Blackstone Group and NBCUniversal which owns The Weather Channel, including its weather data platforms (such as Weather Services International), websites (Weather.com and Weather Underground) and mobile apps. The acquisition seeks to use Watson for weather analytics and predictions. The acquisition does not include The Weather Channel itself, which will enter into a long-term licensing agreement with IBM for use of its data. The sale closed on January 29, 2016
Question: When did the sale of Weather Company assets close?
Answer: January 29, 2016
Generated claim: The sale of Weather Company assets closed on January 29, 2016.
Label: ALIGNED
Context: The dipole component of the magnetic field at the magnetic equator of Neptune is about 14 microteslas (0.14 G). The dipole magnetic moment of Neptune is about 2.2 × 1017 T·m3 (14 µT·RN3, where RN is the radius of Neptune). Neptune’s magnetic field has a complex geometry that includes relatively large contributions from non-dipolar components, including a strong quadrupole moment that may exceed the dipole moment in strength. By contrast, Earth, Jupiter and Saturn have only relatively small quadrupole moments, and their fields are less tilted from the polar axis. The large quadrupole moment of Neptune may be the result of offset from the planet’s centre and geometrical constraints of the field’s dynamo generator.
Question: What is the dipole component of the magnetic field at the magnetic equator of neptune?
Answer: 14 microteslas (0.14 G) Generated claim: The dipole component of the
magnetic field at the magnetic equator of neptune is 14 microteslas (0.14 G).
Label: ALIGNED
Context: Qing dynasty rule in Tibet began with their 1720 expedition to the country when they expelled the invading Dzungars. Amdo came under Qing control in 1724, and eastern Kham was incorporated into neighbouring Chinese provinces in 1728. Meanwhile, the Qing government sent resident commissioners called Ambans to Lhasa. In 1750 the Ambans and the majority of the Han Chinese and Manchus living in Lhasa were killed in a riot, and Qing troops arrived quickly and suppressed the rebels in the next year. Like the preceding Yuan dynasty, the Manchus of the Qing dynasty exerted military and administrative control of the region, while granting it a degree of political autonomy. The Qing commander publicly executed a number of supporters of the rebels and, as in 1723 and 1728, made changes in the political structure and drew up a formal organization plan. The Qing now restored the Dalai Lama as ruler, leading the governing council called Kashag, but elevated the role of Ambans to include more direct involvement in Tibetan internal
affairs. At the same time the Qing took steps to counterbalance the power of the aristocracy by adding officials recruited from the clergy to key posts.
Question: What did the Qing commander do in 1732 and 1728?
Answer: Unanswerable
Generated claim: The Qing commander publicly executed a number of supporters of the rebels in 1732 and 1728.
Label: NOT-ALIGNED
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: In the Limitations section

heading: 3 A2. Did you discuss any potential risks of your work?
text: In Limitation, Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract, 1. Introduction
7 A4. Have you used AI writing assistants when working on this paper? Not used
B 3 Did you use or create scientific artifacts?

heading: 3. Method, 4. Experiments

heading: 3 B1. Did you cite the creators of artifacts you used?
text: 3. Method, 4. Experiments

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: 3. Method, 4. Experiments

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 3. Method, 4. Experiments
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The used data are from publicated

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 3. Method, 4. Experiments, Appendix A
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3. Method, 4. Experiments
C 3 Did you run computational experiments?

heading: 4. Experiments

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 3. Method, 4. Experiments, Appendix

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 3. Method, 4. Experiments, Appendix

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: Left blank.
text: D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: The future of these subsidies is uncertain and has led to anticipatory expansion of cotton brokers’ operations in Africa. Dunavant expanded in Africa by buying out local operations. This is only possible in former British colonies and Mozambique; former French colonies continue to maintain tight monopolies, inherited from their former colonialist masters, on cotton purchases at low fixed prices. Question: How many subsidized cotton growers are in the US?
Answer: 25,000
Generated claim: 25,000 subsidized cotton growers are in the US. Label: ALIGNED
Context: On October 28, 2015, IBM announced its acquisition of digital assets from The Weather Company—a holding company of Bain Capital, The Blackstone Group and NBCUniversal which owns The Weather Channel, including its weather data platforms (such as Weather Services International), websites (Weather. com and Weather Underground) and mobile apps. The acquisition seeks to use Watson for weather analytics and predictions. The acquisition does not include The Weather Channel itself, which will enter into a long-term licensing agreement with IBM for use of its data. The sale closed on January 29, 2016
Question: When did the sale of Weather Company assets close?
Answer: January 29, 2016
Generated claim: The sale of Weather Company assets closed on January 29, 2016. Label: ALIGNED
Context: The dipole component of the magnetic field at the magnetic equator of Neptune is about 14 microteslas (0. 14 G). The dipole magnetic moment of Neptune is about 2. 2 × 1017 T·m3 (14 µT·RN3, where RN is the radius of Neptune). Neptune’s magnetic field has a complex geometry that includes relatively large contributions from non-dipolar components, including a strong quadrupole moment that may exceed the dipole moment in strength. By contrast, Earth, Jupiter and Saturn have only relatively small quadrupole moments, and their fields are less tilted from the polar axis. The large quadrupole moment of Neptune may be the result of offset from the planet’s centre and geometrical constraints of the field’s dynamo generator. Question: What is the dipole component of the magnetic field at the magnetic equator of neptune?
Answer: 14 microteslas (0. 14 G) Generated claim: The dipole component of the
magnetic field at the magnetic equator of neptune is 14 microteslas (0. 14 G). Label: ALIGNED
Context: Qing dynasty rule in Tibet began with their 1720 expedition to the country when they expelled the invading Dzungars. Amdo came under Qing control in 1724, and eastern Kham was incorporated into neighbouring Chinese provinces in 1728. Meanwhile, the Qing government sent resident commissioners called Ambans to Lhasa. In 1750 the Ambans and the majority of the Han Chinese and Manchus living in Lhasa were killed in a riot, and Qing troops arrived quickly and suppressed the rebels in the next year. Like the preceding Yuan dynasty, the Manchus of the Qing dynasty exerted military and administrative control of the region, while granting it a degree of political autonomy. The Qing commander publicly executed a number of supporters of the rebels and, as in 1723 and 1728, made changes in the political structure and drew up a formal organization plan. The Qing now restored the Dalai Lama as ruler, leading the governing council called Kashag, but elevated the role of Ambans to include more direct involvement in Tibetan internal
affairs. At the same time the Qing took steps to counterbalance the power of the aristocracy by adding officials recruited from the clergy to key posts. Question: What did the Qing commander do in 1732 and 1728?
Answer: Unanswerable
Generated claim: The Qing commander publicly executed a number of supporters of the rebels in 1732 and 1728. Label: NOT-ALIGNED
ACL 2023 Responsible NLP Checklist