SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dataset Following Zha et al. [11], we use two popular factual consistency evaluation benchmarks, TRUE (containing 11 datasets, including dialog, fact verification, and paraphrase detection) [59] and SummaC (consisting of 6 summarization datasets) [31]. We also include Other popular meta-evaluation datasets, namely XSumFaith [58], SummEval [60], QAGS-XSum [61], QAGS-CNNDM [61], FRANK [62] and SamSum [63]. This results in 23 datasets in total for our study.
Baselines We compare ALIGN with the latest LLM based automatic metrics: GPTScore [64], G-EVAL [65] and a ChatGPT-based metric [66]. These metrics achieve the best performance when using the GPT family of LLMs, which are significantly larger than our alignment model (e.g., GPT-3 has 175B parameters). GPTScore evaluates texts based on the probability of a LLM generating the target text, while G-EVAL augments its prompt using chain-of-thoughts techniques and asks the LLM to score the input by form-filling. Liu et al. [65] design a prompt that asks ChatGPT to score the faithfulness of the summary on a five point scale. Additionally, we include strong, smaller-scale (similar with our alignment model) baselines, including BERTScore [23], BLEURT [67], BARTScore [68], CTC [37], UniEval [69] and QAFactEval [70], following Zha et al. [11].
Metrics Both the TRUE and SummaC benchmarks formulates factual consistency evaluation as binary classification (i.e., identifying factual consistency errors). Following the common practice, we report ROC AUC [71], treating each model as a classifier. For the rest of datasets, we report instance-level Pearson, Spearman, and Kendall-  $\tau$  correlation coefficients between automatic metric scores and human-annotated scores.
For the LLMs-based metrics, we use the results reported by Fu et al. [64], Liu et al. [65], Gao et al. [66], and consequently results for some model-dataset combinations are unavailable. Despite being much smaller than ChatGPT/GPT-3.5 or GPT-4, our alignment model achieves comparable performance on SummEval (see Table 3). When evaluated on the QAGS-XSum and QAGS-CNNDM datasets, even our 125M alignment model outperforms both G-EVAL and GPTScore based on
GPT-3.5, while the 355M alignment model beats G-EVAL based on GPT-4. When compared with similarly sized metrics, our method consistently outperforms the strong baselines on factual consistency benchmarks and datasets (see Figure 4). We include detailed results in Appendix D.
Table 3: The Spearman Correlation coefficient of ALIGN and GPT-based models on SummEval, QAGS-XSUM and QAGS-CNNDM datasets. Bold number shows the best model on a specific dataset.
G-EVAL-3.5 GPT3.5-d03	G-EVAL-4 GPT4	GPTScore GPT3.5-d03	ChatGPT GPT3.5-turbo	ALIGN-base (Ours)	ALIGN-large (Ours)
Model Parameters	—	—	—	—	125M	355M
Datasets	SummEval	38.6	50.7	47.5	43.3	42.0	47.9
QAGS-XSUM	40.6	53.7	22.0	—	52.7	57.4
QAGS-CNNDM	51.6	68.5	—	—	56.1	71.6
Figure 4: The performance of different models on diverse factual consistency benchmarks and datasets. The left figure includes performance on the SummaC and TRUE benchmark. The right figure shows models measured by correlation coefficients on other datasets (Section 4.2.1).
4.3 Question Answering with Unanswerable Question
In question answering tasks, a system must find the correct answer to a question from a context. When the question cannot be answered with information in the context, the system must indicate the question is not answerable. Despite being a well-studied task, predicting whether a question is answerable remains challenging, especially in a zero-shot setting.
A common approach to improve a system's ability to handle unanswerable questions is to introduce a verifier model in addition to the QA model [72, 73]. Given a context and question pair, the QA model first predicts a candidate answer. Then, the verifier model independently predicts whether the question is answerable by comparing the candidate answer to the question and the context. Lastly, the outputs of the two models are aggregated to form the final prediction. In our experiments, we use the alignment model as the verifier.
4.3.1 Experiment Setup
Datasets We experiment with two existing QA datasets with unanswerable questions, SQuAD v2 [29] and ACE-whQA [74]. Additionally, we construct a third dataset, Simplified Natural Questions (Simplified NQ), base on Natural Questions [75]. To build the dataset, for samples in Natural Questions with both short and long answers, we use the long answer as the context, and the short answer as the ground truth answer; for samples without short and long answers, we select random paragraphs from the articles as contexts and consider them to be unanswerable. Both ACE-whQA and Simplified NQ are not seen by the alignment model during training (i.e., a zero-shot experiment). We use the validation split of SQuAD v2 and Simplified NQ as their test splits are not publicly available.
Baselines We include FLAN T5 [14] and GPT-3.5 $^{4}$  to represent large sequence-to-sequence language models. In addition, we experiment with using ALIGN as a verifier add-on for FLAN T5 and GPT-3.5. Here, we use  $1 - \operatorname*{Pr}(y_{bin} = \text{ALIGNED})$  as the unanswerable probability and use the SQuAD v2 validation split to find the best unanswerable threshold that maximizes the F1 score. The prompts we use and other experiment details are discussed in the appendix (Section D).
Metrics We follow Rajpurkar et al. [76] and report exact match and macro-averaged F1 score. To evaluate each model's performance at identifying unanswerable questions, we also formulate the problem as a binary classification task (predicting whether the sample is answerable) and report the ROC AUC [71]. A higher ROC AUC indicates the model is better at identifying unanswerable questions. For GPT-3.5 and FLAN T5, we consider the unanswerable classifier output to be 0 if the model predicts an answer, or 1 otherwise.
4.3.2 Results
As shown in Table 4, using ALIGN as a verifier add-on significantly improves GPT-3.5 and FLAN T5 in most cases (increases exact match score by 17.94 on average and F1 score by 15.05), suggesting that it is effective at identifying unanswerable questions. For the Simplified NQ dataset, adding the alignment verifier to GPT-3.5 degrades exact match and F1 score, but improves AUC. This indicates that while ALIGN produces meaningful unanswerable probabilities on the Simplified NQ dataset, the threshold found on the SQuAD v2 validation split is not ideal for Simplified NQ. Repeating the experiment with the best threshold selected on the Simplified NQ validation split (see numbers in parenthesis in Table 4) shows the potential for improvements in exact match and F1 scores, albeit this can no longer be considered a zero-shot setting.
Table 4: QA experiment results. Cases where adding the ALIGN verifier improves performance are highlighted in green. Best model for each dataset is shown in **bold**. For the combination of GPT-3.5 + Verifier and Simplified NQ, we also report the exact match and F1 scores with the best unanswerable threshold selected on the Simplified NQ validation split in parenthesis.
SQuAD v2	ACE-whQA	Simplified NQ
EM	F1	AUC	EM	F1	AUC	EM	F1	AUC
GPT-3.5	52.53	63.96	0.76	67.98	71.98	0.77	58.37	68.61	0.81
FLAN T5	75.72	79.01	0.83	26.29	29.24	0.51	38.24	44.98	0.58
GPT-3.5 + Verifier (Ours)	67.19	77.63	0.93	79.02	80.91	0.84	56.16	57.40	0.86
FLAN T5 + Verifier (Ours)	83.72	86.55	0.95	75.75	77.60	0.90	64.93	67.99	0.83
Table 5: Ablation results on in-domain natural language understanding tasks. Each row corresponds with a model trained on data adapted from incrementally more types (+) of tasks from scratch, or from fewer types (-) of tasks from the entire alignment training dataset. For example, the model on the second row is trained with NLI, Fact Verification and Paraphrase tasks. +QA (ALIGN-base) refers to the model trained on the same data as ALIGN-base. We report the average performance for each evaluation tasks. The last column shows the overall average for all the evaluated tasks. The best model for each evaluated task is shown in bold.
Training Tasks	Evaluation Tasks
NLI	Fact Verification	STS	Paraphrase	QA	Coreference	Average
+NLI	69.0	65.3	53.7	69.3	56.0	70.6	64.0
+FV, Para	70.1	83.0	51.6	92.4	53.6	67.9	69.8
+Coref, Sum, IR, STS	69.9	82.9	90.3	91.9	51.8	83.2	78.3
+QA (ALIGN-base)	70.9	83.3	89.9	91.4	78.2	81.4	82.5
-Synthetic	70.4	83.1	90.1	92.0	78.6	83.1	82.9
4gpt-3.5-turbo,see https://platform.openai.com/docs/models/gpt-3-5
As discussed in Section 3.1, ALIGN is trained on datasets from a wide set of language understand tasks. To understand their contributions to the performance of the alignment model, we conduct an ablation study by incrementally adding subsets of tasks to the training set. Specifically, we start with only NLI dataset, and then add the remaining tasks in the following order: 1) paraphrase detection (para) and fact verification (FV) datasets; 2) coreference resolution (coref), summarization (sum), information retrieval (IR), and STS datasets, and lastly 3) QA datasets. Additionally, we train an alignment model without synthetic data to measure the contribution of such data. For simplicity, we use RoBERTa-base as the backbone in this experiment. As shown in Table 5, each added subset improves the overall performance of the alignment model, suggesting our training tasks are compatible and contribute to the model performance. We notice that removing the synthetic data could slightly improve the overall performance, possibly due to the quality of the synthetic data. We will leave this for future study.
We propose to unify diverse language tasks into a text pair alignment problem. This framework yields an alignment model (ALIGN) that, despite being less versatile than LLMs, solves a wide range of language problems efficiently with superior performance. We show that ALIGN outperforms task-specific models finetuned on several NLU tasks while having performance comparable to LLMs that are orders of magnitude larger. Additionally, ALIGN excels in factual consistency evaluation, and can be used as an add-on to augment LLMs in QA tasks by identifying unanswerable questions.
Limitations Our alignment framework uses splitting and aggregation to handle long inputs (see Section 3), with the assumption that text  $x_{2}$  is short and its sentences are self-contained. While we empirically show this method works well on diverse datasets, violating this assumption has a few implications. First, if text  $x_{2}$  sentences are highly interrelated, splitting them discards document-level semantic information, which could degrade performance. Second, as we need to evaluate all text  $x_{2}$  sentences individually, doing so will be slow for long text  $x_{2}$ .
We use a wide collection of NLU datasets to learn the alignment function, with the assumption that these dataset, after being adapted into the text pair alignment format, accurately reflect our definition of alignment. However, as with all datasets, they could contain biases that are subsequently learned by our alignment model. Additionally, we augment the training set with synthetic data. While it proves to improve performance in our experiments, synthetic data likely do not perfectly model real-world data distributions.
[1] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openreview.net/forum?id=yzkSU5zdwD. Survey Certification.
[2] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings.neurips.cc/paper/2020/bit/1457c0d6bcbd4967418bf8ac142f64a-AAbstract.html.
[3] Matúš Pikuliak. Chatgpt survey: Performance on NLP datasets, Mar 2023. URL http://opensamizdat.com/posts/chatgpt_surveys/.
[4] Wenpeng Yin, Nazneen Fatema Rajani, Dragomir Radev, Richard Socher, and Caiming Xiong. Universal natural language processing with limited annotations: Try few-shot textual entailment as a start. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),
[82] Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191-200, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-2623. URL https://aclanthology.org/W17-2623.
[83] Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, and Matt Gardner. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5925-5932, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1606. URL https://aclanthology.org/D19-1606.
[84] Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. Reasoning over paragraph effects in situations. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 58-62, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-5808. URL https://aclanthology.org/D19-5808.
[85] Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94-106, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-4413. URL https://aclanthology.org/W17-4413.
[86] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361, 2021. doi: 10.1162/tacl_a_00370. URL https://aclanthology.org/2021.tacl-1.21.
[87] Mahnaz Koupae and William Yang Wang. Wikihow: A large scale text summarization dataset. CoRR, abs/1810.09305, 2018. URL http://arxiv.org/abs/1810.09305.
[88] Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: pre-training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview.net/forum?id=r1xMH1BtvB.
While our text pair alignment model achieves state-of-the-art performance on many downstream tasks, like all models, it does make mistakes. For example, when used for fact verification or factual consistency evaluation, it could misidentify factually correct statements as incorrect and vice versa. Additionally, as we use publicly available datasets to train the alignment model, it might have learned biases inherent to those datasets. Thus, one should proceed with caution when using ALIGN for purposes other than NLP research.
B Comparison with Other Model Types
We illustrate the major differences between our approach, LLMs, multitask learning models, and task-specific finetuned models in Table 6. Compared with LLMs, our alignment function is more efficient but less versatile. In contrast to task-specific finetuned models, the alignment function is more general and can handle more types of tasks. Unlike multitask learning models, we unify language tasks into a single text pair alignment problem, and share model components across multiple tasks (as apposed to using dataset-specific prediction heads). As a result, our alignment function can be directly applied to a wide range of tasks out-of-the-box, without any finetuning.
Table 6: Comparison between the alignment function and other types of language models.
Type	Model Example	Efficient	Out of the box	General
LLM	T5, PALM, UL2, GPT	×	✓	✓
Multitask learning	MT-DNN, MUPPET	✓	×	✓
Task specific LM	Finetuned RoBERTa/BERT	✓	×	×
Text pair alignment	ALIGN (Ours)	✓	✓	✓
C.1 Trainig Setup
We choose RoBERTa [12] as the backbone for the alignment model. ALIGN-base and ALIGN-large are based on RoBERTa-base and RoBERTa-large, respectively. For the experiments in Section 4, we train ALIGN for 3 epochs with a batch size of 32, following common practice [12, 16]. Other hyperparameters are listed in Table 7. For the finetuned RoBERTa and RoBERTa-NLI model in Section 4.1, we set batch size to 16 and 32, respectively.
Table 7: The hyperparameters for training the alignment model.
Hyperparameter	ALIGN-base	ALIGN-large
Parameters	125M	355M
Batch Size	32	32
Epochs	3	3
Optimizer	AdamW	AdamW
Learning Rate	1e-5	1e-5
Weight Decay	0.1	0.1
Adam ε	1e-6	1e-6
Warmup Ratio	0.06	0.06
Random Seed	2022	2022
GPU	2×3090	4×A5000
GPU Hours	152h	620h
C.2 Training Datasets
We collect datasets that falls into the scope of alignment as mentioned in Section 3. Table 8 lists the datasets we use for training the alignment model. The size of these datasets ranges from 4k samples to 5M. Most of the datasets are used for binary classification task except some NLI, fact verification and STS datasets.
We only use the first 500k samples in each dataset due to limited computation resource, which results in 5.9M training samples in total. During training, the samples are randomly sampled from the entire adapted training sets.
Table 8: The training datasets of ALIGN. Note due to resource constraints, we only use at most  ${500}\mathrm{k}$  samples from each dataset to train the alignment model.
NLP Task	Dataset	Training Task	Sample Count
NLI	SNLI [40]	3-way classification	550k
MultiNLI [7]	3-way classification	393k
Adversarial NLI [39]	3-way classification	163k
DocNLI [77]	binary classification	942k
Fact Verification	NLI-style FEVER [41, 39]	3-way classification	208k
VitaminC [24]	3-way classification	371k
Paraphrase	QQP [43]	binary classification	364k
PAWS-Wiki [25]	binary classification	695k
PAWS-QQP [25]	binary classification	12k
WikiText-103 [33]	binary classification	8M
STS	SICK [78]	regression	4k
STSB [42]	regression	6k
QA	SQuAD v2 [29]	binary classification	130k
RACE [28]	binary classification	351k
Adversarial QA [79]	binary classification	60k
BoolQ [45]	binary classification	19k
DROP [80]	binary classification	155k
MultiRC [44]	binary classification	24k
HotpotQA [81]	binary classification	362k
NewsQA [82]	binary classification	161k
QuAIL [46]	binary classification	41k
Quoref [83]	binary classification	39k
ROPES [84]	binary classification	22k
SciQ [85]	binary classification	47k
StrategyQA [86]	binary classification	5k
Information Retrieval	MS MARCO [27]	binary classification	5M
Summarization	WikiHow [87]	binary classification	157k
Coreference	GAP [30]	binary classification	4k
D Additional Experiment Details
D.1 Natural Language Understanding Tasks
**Prompts** For FLAN models, we use the same prompts as Longpre et al. [48]. For datasets that do not appear in Longpre et al. [48], we use prompts of similar tasks. The prompt used for each dataset is listed below.
MNLI, NLI-FEVER, VitaminC:
"Premise: {premise}\n\nHypothesis: {hypothesis}\n\nDoes the premise entail the hypothesis?\n\nA yes\nB it is not possible to tell\nC no"
ANLI:
{"context}\n\nBased on the paragraph above can we conclude that \{"\{\hypothesis\}\"?\\n\nA Yes\nB It's impossible to say\nC No"
SNLI:
"If \\"{premise}\\"", does this mean that \\"{hypothesis}\\"?\\n\nA yes\nB it is not possible to tell\nC no"
SICK, STSB:
{"sentence1}\n{sentence2}\n\nRate the textual similarity of these two sentences on a scale from 0 to 5, where 0 is "no meaning overlap" and 5 is "means the same thing".\nA0\nB1\nC2\nD3\nE4\nF5"
PAWS, PAWS-QQP:
"\{sentence1\}\n{sentence2}\n\nDo these sentences mean the same thing?\nA no\nB yes"
QQP:
{"question1}\n{question2}\nWould you say that these questions are the same? \nA no\nB yes"
RACE, QuAIL, SciQ:
{"fact}\n{question}\n\nA {option 1}\nB {option 2}\nC {option 3} ...
Multi-RC:
{"paragraph}\n\nQuestion: {"question}\\"n\nResponse:
\"{response}\\"\n\nDoes the response correctly answer the question?\\nA no\nB yes"
BoolQ:
{"text}\n\nCan we conclude that {question}?\n\nA no\nB yes"
GAP:
"Context: {context}\n Given the context, which option is true? \n\nA{option 1}\nB {option 2}\nC {option 3} ... "
Multitask RoBERTa Baseline To obtain the multitask-learning RoBERTa model, we follow the popular multi-task learning work [9], and train the same base model with the same set of tasks (datasets) as our alignment model. Notably, different from our alignment model that uses a unified interface to accommodate all diverse tasks, the conventional multitask-learning model learns separate prediction heads for different tasks.
Instruction Finetuned T5 Baseline In order to understand the performance difference between our text alignment formulation and instruction fine-tuning, we instruction-finetune T5-base (250M parameters) on the same datasets as our alignment model. We do not convert QA tasks since T5 naturally supports sequence generation. We follow the prompts mentioned in Chung et al. [14], Longpre et al. [48] and format the datasets to train the T5 model.
Results We show the performance of finetuned RoBERTa and FLAN-Alpaca on these datasets in Table 9, while the result of the multi-task RoBERTA is in Table 10 and 11. The comparisons with the T5 model instruction finetuned on alignment datasets are in Table 12 and 13. We have compared ALIGN with finetuned RoBERTa on these datasets in Figure 2. When comparing FLAN-T5 and FLAN-Alpaca, we notice FLAN-T5 consistently outperforms FLAN-Alpaca on all scales. Thus, we compare our alignment model with FLAN-T5 in Table 1.
D.2 Factual Consistency Evaluation for Language Generation
Detailed Result In this section, we report the detailed results associated with Figure 4. We list the performance of each metric on SummaC Benchmark and TRUE Benchmark in Table 15 and Table 16, respectively. We also show the Pearson Correlation, Spearman Correlation and Kendall's tau Correlation coefficients on other datasets in Table 17, 18 and 19, respectively.
Table 9: The performance of finetuned RoBERTa and FLAN-Alpaca on the in-domain datasets. We report the average performance of each model and we also include the average without RACE and QuAIL.
Finetuned RoBERTa	FLAN-Alpaca
base	large	base	large	xlarge
Model Parameters	125M	355M	220M	770M	3B
NLI	MNLI-mm	87.2	90.3	79.9	86.4	89.3
MNLI-m	87.9	90.6	80.0	87.2	89.4
ANLI-1	62.8	72.7	47.4	65.7	74.8
ANLI-2	44.5	48.3	38.2	46.6	57.6
ANLI-3	42.8	47.0	37.7	46.4	54.6
SNLI	91.0	91.4	82.9	88.1	90.2
Fact Verification	NLI-FEVER	76.1	77.7	69.6	73.0	72.1
VitaminC	89.3	91.6	63.3	72.5	77.4
STS	SICK	88.9	84.7	37.7	66.4	70.1
STSB	89.8	90.6	33.4	52.5	79.5
Paraphrase	PAWS	92.3	92.5	68.1	92.0	93.0
PAWS-QQP	94.7	94.2	57.6	85.1	87.1
QQP	91.1	92.0	75.5	81.6	86.5
QA	RACE-m	74.6	24.0	64.3	78.5	87.8
RACE-h	67.8	23.9	57.3	71.9	82.9
Multi-RC	77.5	85.5	64.2	84.3	87.1
BoolQ	79.1	85.7	71.7	82.0	87.2
QuAIL	57.7	27.0	56.7	78.2	84.2
SciQ	93.4	95.5	90.8	83.1	95.6
Coreference	GAP	74.3	89.8	58.4	65.6	80.7
Average	78.1	74.7	61.7	74.4	81.4
Average w/o RACE, QuAIL	80.2	83.5	62.1	74.0	80.7