Text Alignment Is An Efficient Unified Model for Massive NLP Tasks
Authors: Yuheng Zha
Yichi Yang
Ruichen Li
Zhiting Hu
Venue: NeurIPS 2023 poster (2023)
Keywords: Text Alignment
Efficient Unified Model
NLU Tasks
Factual Consistency Evaluation
QA with Unanswerable Question

Abstract: Large language models (LLMs), typically designed as a function of next-word prediction, have excelled across extensive NLP tasks. Despite the generality, next-word prediction is often not an efficient formulation for many of the tasks, demanding an extreme scale of model parameters (10s or 100s of billions) and sometimes yielding suboptimal performance.
In practice, it is often desirable to build more efficient models---despite being less versatile, they still apply to a substantial subset of problems, delivering on par or even superior performance with much smaller model sizes.
In this paper, we propose text alignment as an efficient unified model for a wide range of crucial tasks involving text entailment, similarity, question answering (and answerability), factual consistency, and so forth. Given a pair of texts, the model measures the degree of alignment between their information. We instantiate an alignment model through lightweight finetuning of RoBERTa (355M parameters) using 5.9M examples from 28 datasets. Despite its compact size, extensive experiments show the model's efficiency and strong performance: (1) On over 20 datasets of aforementioned diverse tasks, the model matches or surpasses FLAN-T5 models that have around 2x or 10x more parameters; the single unified model also outperforms task-specific models finetuned on individual datasets; (2) When applied to evaluate factual consistency of language generation on 23 datasets, our model improves over various baselines, including the much larger GPT-3.5 (ChatGPT) and sometimes even GPT-4; (3) The lightweight model can also serve as an add-on component for LLMs such as GPT-3.5 in question answering tasks, improving the average exact match (EM) score by 17.94 and F1 score by 15.05 through identifying unanswerable questions.

Text Alignment Is An Efficient Unified Model for Massive NLP Tasks
Yuheng Zha" Yichi Yang"
Ruichen Li
Zhiting Hu
UC San Diego
{yzha, yiy067, ru1014, zhho19) edu
Abstract
Large language models (LLMs), typically designed as a function of next-word prediction, have excelled across extensive NLP tasks. Despite the generality, next-word prediction is often an efficient formulation for many of the tasks, demanding an extreme scale of model parameters (10s or 100s of billions) and sometimes yielding suboptimal performance. In practice, it is often desirable build more efficient models models-desplnne despite being less versatile, they still apply to a substantial subset of problems, delivering on par or even superior performance with much smaller model sizes. In this paper, we propose text alignment as an efficient unified model for wide range of crucial tasks involving text entailment, similarity, question answering (and answerability). factual consistency, and so forth. Given pair of texts, the model measures the degree of alignment between their information. We instantiate alignment model (ALIGN) through lightweight finetuning of RoBERTa (355M parameters) using 5.9M examples from 28 datasets. Despite its compact size, extensive experiments show the model's efficiency and strong performance: (1) On over 20 datasets of aforementioned diverse tasks, the model matches surpasses FLAN-T5 models that have around 2x 10x more parameters; the single unified model also outperforms task-specific models finetuned individual datasets; (2) When applied to evaluate factual consistency of language generation on 23 datasets, our model improves over various baselines, including the much larger GPT-3.5 (ChatGPT) and sometimes even GPT-4; (3) The lightweight model can also serve as an add-on component for LLMs such as GPT-3.5 in question answering tasks, improving the average match (EM) score by 17.94 and Fl score by 15.05 through identifying unanswerable questions."
Introduction
Recent large language models (LLMs) have demonstrated exceptional generalizability in wide range of natural language processing (NLP) tasks. the underlying formulation of these LLMs, next-word prediction is proven to be general function applicable to diverse language problems. However, often not being an efficient solution for many tasks. I.LMs often need to scale up to over tens of billions of parameters to achieve meaningful performance [1], with popular models like GPT-3 boasting as many as 175B parameters [2]. Additionally, even with their extreme scale, LLMS sometimes still find themselves outperformed by smaller models. For example, ChatGPT/GPT-3.5 falls behind existing finetuned baselines on most classical natural language understanding tasks [3].
As result, in many cases it is desirable to navigate the spectrum of inberality-vs-efffieenn tradeoff, for example, by developing smaller but general-purpose models that excel substantial subset of tasks. Despite being less versatile than the extreme-scale LLMs, these models are more efficient and
"Equal contribution.
"Code made available https: httpaas////ttttttteeeee httpps:/ggtthab.ccmmmmmuhahhhah
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Language Understanding
Consistency Evaluation
Answering
FLANT
GPT-3.5
Alinn (ouns)
1.00
Perfrmance
04
90-
66
40-
-o- TLANT5
-0- FLANTS
TLA44Aipec
38-
GPTS
Algn
10
101 1072
10
S0.ND2
ACE WHOA Datasets
Smpitted NO
Model Parameters
Model Parameters
Figure 1: Our alignment model (125M and 355M) achieves substantially better efficiency and performance compared to much larger models on wide range of tasks, including (left) diverse text pair understanding tasks on over 20 datasets and (middle) factual consistency evaluation, and (right) improves existing LLMs question answering by detecting unanswerable questions. See Section 4 for more details.
provide superior performance, making them more usable the set of tasks that they are designed to handle. Previous work has attempted to build natural language inference (NLI) models as an efficient solution for broad tasks 64 But with limited NLI data (e.g., MNLI [7]) for training, the models exhibit limited performance and applicability across diverse domains. Another related line of research trains general text representation models with pretraining and multi-task learning [8-10]. Those models need to be specifically finetuned (with task-specific head) for each downstream task, instead of functioning as ready-to-un modu-tt-use solutions.
In this paper, we investigate the underlying commonalities among a broad range of NLP tasks that concern the relationship between two texts, and propose text alignment model (ALIGN) efficient unified solution, following Zha et al. [11]. Given an arbitrary pair of texts, ALIGN measures the degree of alignment between the content in the texts. We show the formulation subsumes substantial set of popular tasks, ranging from NLI, fact verification, semantic textual similarity, question answering, coreference resolution, paraphrase detection, factual consistency evaluation of diverse language generation tasks, question answerability verification, and so forth (Figure 1). use diverse data learn the alignment
The generality, in turn, presents opportunity for model. Specifically, we adapt and aggregate 28 datasets from the above tasks, resulting in 5.9M training samples with diverse characteristics. We then use the data finetune small-scale LM (e.g.. RoBERTa [12]), yielding model that directly applies to and excels in diverse problems and domains.
We evaluate ALIGN with extensive experiments. First, we test 25 seen and unseen datasets of the aforementioned tasks, and show our alignment model based on RoBERTa (355M parameters) achieves on par even better performance than the FLAN-T5 models (780M and 3B) that are 2x or 8.5x as large and trained with substantially more data. In addition, the single alignment model outperforms RoBERTa specifically finetuned each of the datasets. Second, we ALIGN eval- uate the factual consistency of natural language generation (NI.) systems (e.g., for summarization, dialog, paraphrasing, etc.). On 23 datasets, the small alignment model achieves substantially higher correlation with human judgements than recent metrics, including those based on GPT-3.5 and even GPT-4. Third, we use ALIGN component to existing LLMs (e.g., GPT-3.5 and FLAN-TS), It significantly enhances the LLMs' performance in three question answering datasets, improving the average exact match score by 17.94 and F1 score by 15.05.
question answerability verifier and incorporate it as an add-on
2 Related Work
Recent work has shown that LLMs are few-shot learners capable of generalizing to diverse tasks [13, 2, 14]. These LLMs are designed based on the principle of next-word word-prediction. where the joint probability distribution of text sequences are factored into product of conditional probabilities [15]. The performance of L.Mss is highly correlated with their scales. Wei et al. [1] show that scale of more than 10 training FL.PPs (around 10B model parameters) is required for several different L.LM designs to achieve above-random performance many language tasks.

---

Another line of research has tried to design models that can either learn from multiple tasks or handle many downstream tasks. Liu et al. [8] propose to use BERT model [16] with task-specific heads to learn four types of tasks, including single-sentence classification, pairwise text classification, text similarity scoring, and relevance ranking. Aghajanyan et al. [9] pre-finetune language models on 50 dataset to encourage learning more general representations, and show that the process improves model performance and data-efficiency in the finetuning stage. Yin et al. [4], Wang et al. [5] and Mishra et al. [6] explore the application of model trained with NLI datasets to multiple downstream tasks, in an effort to find more general yet still efficient methods. Some research explores the use of smaller models for enhancing LLMs. For example, Cappy [17], small scorer model trained on the same datasets as in TO [18], functions well on natural language classification tasks while boosting the performance of LLMs as an add-on. While we also use diverse datasets to train our model, we 1) unify language tasks into a single text pair alignment problem, 2) share all model components across multiple tasks and do not dataset-specifie heads, and 3) our model can be directly applied to wide range of tasks without additional finetuning. The work demonstrates the strong potential of unified modeling and learning with all diverse relevant forms of experience [19].
Text alignment has long been used for measuring the correspondence of information between two pieces of text. For example, in machine translation, Brown et al. [20] propose a model that learns alignment between two languages. Papineni et al. [21] use n- n-gram overlap to compare translated sentences with their references. Gao et al. [22] train sentence-level embedding model that compares similarity between two sentences while BERTScore [23] utilizes token-level aligmment for evaluating text generation. Zha al. [11] also propose building an automatic factual consistency metric for NL.G systems through a text alignment framework. Expanding on the idea of text alignment, we explore how the formulation enables training a single alignment model that excels a wide variety of tasks, including natural language understanding, factual consistency evaluation and answerability verification.
3 Text Alignment Model
In this section, we introduce the text pair alignment formulation. We first formally define the concept of text pair alignment, and then discuss how the alignment function can be used to solve set of popular language tasks. Additionally, we cover the pplitthhn-aggregatt method used to handle long inputs. In Section 3.1, we discuss the training process of the alignment model (ALigN).
Given a text pair (x 22), we define text x2 to be aligned with text æ if all information in 22 is supported by information in 11 following Zha et al. [11]. For example, let "I have been in Kentucky, Kirby. " be text 21. Then, "I have been in the US. " is aligned with 2 In con- trast, both "I have been in Europe. and "Kentucky has the best fried chicken. are not aligned with æ1. as the former is contradicted by 31, and the latter cannot be inferred from x1. Formally, we model alignment as function that maps the text pair (x1, x2) to a label describing the level of alignment:
(x1, y
In practice, the language tasks we wish to solve with the alignment function can be broadly categorized into two groups: one that uses discrete labels, and the other that uses continuous labels (e.g., semantic textual similarity). More specifically, tasks with discrete labels are typically formulated as either binary classification (e.g., paraphrase detection) or three way classification (e.g., fact verification). In order to make the alignment function more general, such that it accommodates all the above cases, our alignment model produces three outputs: Pr(yon). Pr(ywway), ), and Yreg- Here, Pr(Mbna and Pr(yyway) are probability distributions over the binary (ALIGNED, NOT-AL (OT-ALIGNED) and 3-way (ALIGNED, CONTRADICT, NEUTRAL) classification labels, respectively; Breg E [0, 1] is real-valued score for regression tasks.
This formulation allows to apply the alignment function to diverse tasks:
For tasks that naturally fit into the text pair alignment format, such as [L] fact verification [24], paraphrase detection [25]. and semantic textual similarity [26], depending on the nature of the task, we simply map one of the alignment labels the desired label. For example, for most NLI tasks, we interpret the corresponding V/way labels as "entailment", "iaiieeeet.."oonaiiiiiii "contradiction" and "neutral".
3

---

In information retrieval tasks [27], the goal is to find documents that can answer given query from large set of candidate documents. Since relevant documents contain information related to respective queries, we use candidate documents as text 11, and queries as text 222 Then, higher Pr(yhin ALIGNED) indicates the candidate document is more likely to be useful in answering the query.
In multiple choice QA tasks [28], the inputs are a context, question, and several choices (with one of them being the correct answer). In extractive QA tasks (including ones with unanswerable questions [29]), the inputs only consist of context and question. In either case, the expected output (the correct answer) can be inferred from the question and the context, while a wrong answer either contradicts the context or is not supported by the context. Therefore, we use the context as text 2 and the concatenation of the question a candidate answer as text 22. Here, higher Pr(yhin ALIGNED) indicates the candidate answer more likely to be correct.
In coreference resolution tasks [30], each sample includes a context containing a pronoun, and list of candidate entities. The goal is find the correct entity that the pronoun is referring to. As the pronoun and the associated entity is equivalent in this context, we consider the context with the pronoun replaced with the correct entity to be aligned with the original context. To solve coreference resolution problems, we simply replace the pronoun with candidate entities and compare the resulting contexts (22) with the original context (æ1). We pick the candidate that produces the highest Pr(yywa = ALIGNED) or Pr( = ALIGNED) as the correct answer.
For generative tasks like machine summarization, dialog, and paraphrasing, the alignment function can be used to evaluate the factual consistency of generated outputs. We use the generation context (e.g., input document) text 21, and candidate system output (e.g., generated summary) as text Ø2. In this case, the probability of Pr(yyway = ALIGNED) or Pr(yhin = ALIGNED) indicates if the candidate output faithfully reflects information in the context, without introducing hallucinations or contradictions.
One specific challenge of applying the alignment function to downstream tasks is that text æ1 in some datasets (e.g., contexts in QA or summarization datasets) tends to be significantly longer than the input length limit of typical language models (c.g.. 512 tokens for RoBERTa). As result, naively truncating oversized inputs could throw away important information. To alleviate this problem, inspired by Laban al. [31], Amplayo al. [32], inference time, instead of truncating the inputs, we split text 11 into a set of chunks { and text 22 into a set of sentences
{x } such that the combined length of chunk-sentened pair is slightly below that length limit. Then, evaluate each pair and aggregate the results as
aligument(2), x2) = mcan max f(x (0) 222 ())
where the max operation selects the output with the highest ALIGNED probability or regression score. Since in most downstream applications, text 22 tends to be succinet (e.g., summaries) and consists of self-contained sentences, this aggregation method can be interpreted as first finding the text æ1 chunk that most strongly supports each text ø2 "fact", and then taking the average across all text 222 "facts".
3.1 Training
Our formulation not only allows us to solve the above tasks with a single alignment function, but also learn the alignment function from these tasks. By adapting text pair understanding tasks into a uniform alignment format as above, we can naturally model these tasks as simple classification and regression, allowing us train a small model while achieving strong performance. Specifically, we use RoBERTa [12] as a lightweight backbone language model, and attach three individual linear layers to predict the three types of alignment outputs, Pr(ybin). Pr(ysway) ), and Vreg. respectively. The two classification heads are trained with cross entropy loss, while the regression head is trained with mean squared error loss. The losses are aggregated as a weighted sum,
Liotal = X Laway + Xz + +xgGreg
where we set A = 1/lug3, 3, A = 1/log 2, and Ag = 1, following Aghajanyan et al. [9].
Besides the aforementioned downstream datasets, we also include synthetic data to increase the diversity of the training set. Specifically, for QA datasets without wrong options (e.g. extractive QA datascts like SQuAD v2 [29]), we first remove the ground truth answer from the context, and then

---

Table I: Performance of ALIGN and the much larger FLAN-T5 on in-domain datasets. Bold number
indicates our performance is better than either FLAN-TT5-arge or xlarge.
ALIGN (Ours)
FLAN-TS
base
large
large
xlarge
Model Parameters
125M
355M
780M
3B
MNLI- mm [7]
87.5
90.3
88.7
90.4
MNI I-mm [7]
87.8
90.3
88.8
90.5
ANLI-1 [39]
65.3
75.8
68.1
77.0
NLI
ANLI-2 [39]
48.7
48.7
60.6
ANLI-3 [39]
45.5
52.3
49.8
56.6
SNLI [40]
90.8
91.8
89.7
90.7
NLI-FEVER [41, 39)
76.8
77.8
72.0
71.9
Fact Verification
VitaminC [24]
89.8
91.8
72.9
73.9
SICK [26]
90.7
91.5
79.3
79.1
STS
STSB [42]
89.0
89.8
83.9
88.2
PAWS [25]
92.3
92.6
94.0
94.6
Paraphrase
PAWS-QQP [25]
91.9
93.8
88.3
90.1
QQP [43]
90.1
86.8
87.4
RACE-m [28]
76.9
86.8
84.8
87.6
RACE-h [28]
68.8
81.6
78.3
84.6
Multi-RC [44]
82.2
87.8
84.7
88.2
QA
BoolQ [45]
81.1
87.7
84.9
89.6
QuAIL [46]
67.8
78.6
79.1
86.3
SciQ [47]
92.4
94.9
95.7
Coreference
GAP [30]
81.4
88.6
73.8
81.5
Average
79.8
84.3
79.6
83.2
use QA model [13] to generate wrong answers that can be used to create NOT-ALIGNED samples. Additionally, we create synthetic paraphrase samples by back translating the WikiText-103 corpus [33] using neural machine translation model [34]. For the WikiHow summarization dataset, use an extractive summarizer [35] to generate synthetic summaries additional to ground truth summaries. Following Kryscinski et al. [36], Deng et al. [37], we create negative samples for both WikiText- 103 and WikiHow samples by randomly masking 25%
tokens text and infilling with small masked language modeling model [38]. In total, we collect 5.9M examples from 28 datasets to train our alignment model ALIGN. We include more details our training setup and data in Appendix C.
Experiments
In this section, experiment with applying ALIGN multiple downstream tasks, including language pair understanding tasks (Section 4.1), factual consistency evaluation (Section 4.2), and question answering with unanswerable questions (Section 4.3). We discuss experiment details and include data contamination analysis in the Appendix (Section D).
4.1 Natural Language Understanding Tasks
Natural Language Understanding (NLU) is major category of tasks for language models, and our formulation allows us directly use ALIGN solve these tasks. Specifically, include NLI fact verification, paraphrase detection, multiple-choic QA, STS, and coreference resolution datasets the experiments. We also include unseen datasets to demonstrate the generalizability of ALIGN. Experiments show the alignment model is par with FLAN T5 that has 8.5x as many parameters. Additionally, without further task-specific finetuning, our model outperforms finetuned language models of similar size.
5

---

Aage
100
Muslarge
Average
ANU
ANU
190
000
Average
1977
620
NU
FV
STS
Para
QA
Coref
Para
Q2A
NLI
Figure 2: The performance of the ALiGe-large and the task-specific finetuned RoBERTa-large on in-domain tasks.
Figure 3: The performance of the ALIGN-large and RoBERTa-NLI-lage zero-shot tasks.
4.1.1 Experiment Setup
sets of the 20 datasets used during training (in-domain setting; see Table 1). Then, we use 9 unseen datasets for evaluation (zero-shot setting; see Table 2). For more details about the datasets, please refer to appendix C.2. If dataset does have public test set, we use its validation set instead. For datasets that require binary, 3-way classification or regression, we use the associated output heads, respectively, as discussed in Section 3.
Datasets We first evaluate ALIGN
Baselines To demonstrate the efficiency of ALIGN, we compare it with FLAN-T5 [14] and FLAN- Alpaca' with model size ranging from 220M (FLAN-Alpaca- FLAN-TS-xlarge). For both models, we use the same prompts as in Longpre et al. [48]. We include task-specific RoBERTa models that individually finetuned each training and evaluated on the corresponding test set to show that our alignment model works well out-of-the-box without further finetuning. We also compare with multi-task RoBERTa model trained all the original datasets (before converting to the alignment- format data), and a T5-base model instruction finetuned on our alignment datasets show the effectiveness of our alignment formulation. Lastly, we compare with RoBERTa model finetuned on MNLI, ANLI and SNLI datasets (RoBERTa-NLD) in the zero-shot setting, to show the generalizability of our proposed formulation.
to 3B PLAN-Alpace xlarge and
4.1.2 Results
We report average Pearson Correlation coefficient for the STS tasks [26, 42], and average accuracy for the other tasks.
In the in-domain setting, show in Table ALIGN outperforms FLAN-T5 that is 2x large (780M) and has comparable performance to the version 8.5x as large (3B). Similar performance gain also observed when comparing with FLAN-Alpaca in Table 9, Furthermore, ALIGN is par with the task-specific finetuned RoBERTa models (Figure 2) and the multi-task RoBERTa model (Table 10). ALIGN outperforms the the instruction finetuned T5 model (Table 12), showing the effectiveness of our unified alignment formulation.
In the zero-shot setting, ALIGN achieves comparable performance with similarly sized variants of FLAN-TS (Table 2), even on datasets that exist in FLAN-T5's training set. also shows superiority to the multi-task RoBERTa in Table 1I it eliminates the need to choose task heads while outperforming the average performance of the heads in the multi-task RoBERTa model. Additionally, ALIGN has stronger performance average than the RoBERTa-NLA model (Figure 3) and the instruction finetuned T5 model (Table 13), indicating that our formulation leads to better generalizability.
alpaca

---

Table 2: Zero-shot setting results from ALIGN. The bold numbers indicates a better performance of ALIGN when comparing with similarly sized FLAN-T5. The gray number shows the specific dataset is appeared in the training set of FLAN-T5.
ALIGN (Ours)
FLAN-TS
base (125M)
large (355M)
base (250M)
large (780M)
AXB [49]
75.1
79.6
71.7
76.2
AXG [49]
59.8
72.5
53.4
73.6
CB [49]
76.8
89.3
82.1
87.5
NLT
RTE [50]
83.4
89,9
81.6
87.0
WNLI [50]
52.1
54.9
46.5
62.0
SE14T1 [51]
90.7
86.9
69.6
69.9
Paraphrase
MRPC [52]
66.0
67.9
74.8
80.1
DREAM [53]
71.3
81.1
69.9
79.0
QA
Quartz [54]
59.7
79.2
74.4
90.2
Average
70.5
77.9
69.3
78.4
4.2 Factual Consistency Evaluation for Language Generation
Studies have shown that natural generation systems (NLG) are prone to generating text that is consistent with the source material [55-59]. As result, many automatic metrics have been developed with the goal of detecting factual consistency errors. As factual consistency is closely related definition of text pair alignment, we directly apply ALIGN for this purpose, using the NLG input context as 22, and system outputs as 22- We consider a system output with higher Pr(yyway ALIGNED) to be more factually consistent.
4.2.1 Experiment Setup
Dataset Following Zha et al. [11], two popular factual consistency evaluation benchmarks, TRUE (containing 11 datasets, including dialog, fact verification, and paraphrase detection [59] and SummaC (consisting of 6 summarization datasets) [31]. We also include Other popular meta- evaluation datasets, namely XSumFaith [58], SummEval [60], QAGS-XSum [61]. QAGS-CNNDS [61], FRANK [62] and SamSum [63]. This results in 23 datasets in total for our study.
Baselines We compare ALIGN with the latest LLM based automatic metrics: GPTScore [64], G-EVAL [65] and a ChatGPT-based metric [66]. These metrics achieve the best performance when using the GPT family of LLMs, which are significantly larger than our alignment model (e.g., GPT-3 has 175B parameters). GPTScore evaluates texts based on the probability of a LLM generating the target text. while G-EVAL augments its prompt using chain-of of-thoughtss techniques and asks the LLM to score the input by form-fill form-filing. Liu et al. [65] design a prompt that asks ChatGPT to score the faithfulness of the summary (similar with our alignment model) baselines, including BERTScore [23], BLEURT [67], BARTScore [68], CTC [37], UniEval [69] and QAFactEval [70], following Zha al. [11].
five point scale. Additionally, we include strong, smaller-scale
Metrics Both the TRUE and SummaC benchmarks formulates factual consistency evaluation as binary classification (i.e., identifying factual consistency errors). Following the common practice, we report ROC AUC [71], treating each model as a classifier. For the rest of datasets, we report instance-level Pearson, Spearman, and Kendall-T correlation coefficients between automatic metric scores and human-annotatee scores.
4.2.2 Results
For the LLMs-based metrics, we use the results reported by Fu et al. [64], Liu et al. [65]. Gao al. [66], and consequently results for some model-dataset combinations are unavailable. Despite being much smaller than ChatGPT/GPT-3.5 or GPT-4, our alignment model achieves comparable performance on SummEval (see Table 3). When evaluated on the QAGS-XSum and MAGS-CNNDM datasets, even 125M alignment model outperforms both G-EVAL and GPTScore based on
7

---

GPT-3.5, while the 355M alignment model beats G-EVAL based on GPT-4. When compared with similarly sized metries, our method consistently outperform the strong baselines on factual consistency benchmarks and datasets (see Figure 4). We include detailed results in Appendix D.
Table 3: The Spearman Correlation coefficient of ALIGN and GPT--ased models SummEval, QAGS-XSUM and QAGS-CNNDM datasets. Bold number shows the best model on specific dataset.
G-EVAL-3.5 GPT3.5-003
G-EVAL-4 GPT4
GPTScore GPT3.5-003
CharGPT GPT3,5-urrb
ALIGN-base (Ours)
ALIGN-large
(Ours)
Model Parameters
125M
355M
SummEval
38.6
50.7
47.5
43.3
42.0
47,9
Datasets
QAGS-X -SG-
40.6
53.7
22.0
52.7
57.4
ALGS-CNNDM
51.6
68.5
56.1
71.6
1.0
Unibval
UniEval @Anactboui
@Atactiboal
ScOre
Aigribase Aignilarge
Algn-base Align-argg
Condidin Coeffiiiont
0.5
N0C AJC
0.4
0.8
02-
Summac
TRUE
Othen-Speaman. Bondhmank Datasett
Other-annaal
Benchmani: Danscets
Figure 4: The performance of different models on diverse factual consistency benchmarks and datasets. The left figure includes performance on the SummaC and TRUE benchmark. The right figure shows models measured by correlation coefficients on other datasets (Section 4.2.
4.3 Question Answering with Unanswerable Question
In question answering tasks, a system must find the correct answer to question from context. When the question cannot be answered with information the context, the system must indicate the question is not answerable. Despite being well-studied task, predicting whether a question is answerable remains challenging, especially in a zero-shot setting.
A common approach to improve system's ability to handle unanswerable questions is to introduce verifier model addition the QA model [72, 73]. Given context and question pair, the QA model first predicts candidate answer. Then, the verifier model independently predicts whether the question answerable by comparing the candidate answer the question and the context. Lastly, the outputs of the two models are aggregated to form the final prediction. In our experiments, we use alignment model as the verifier.
4.3.1 Experiment Setup
Datasets We experiment with two existing QA datasets with unanswerable questions, SQuAD v2 [29] and ACE-whQA [74]. Additionally, we construct a third dataset, Simplified Natural Questions (Simplified NQ), base on Natural Questions [75]. To build the dataset, for samples in Nat- ural Questions with both short and long answers, we use the long answer the context, and the short answer as the ground truth answer; for samples without short and long answers, we select random paragraphs from the articles contexts and consider them to be unanswerable. Both ACE-whQA and Simplified NQ are not seen by the alignment model during training (i.e., a zero-shot experiment). We the validation split of SQuAD v2 and Simplified NQ their test splits are not publicly available.

---

Baselines We include FLAN T5 [14] and GPT-3.5* to represent large sequence-tor to toreecettt--qqueeee sequence lan- guage models. addition, we experiment with using ALIGN as a verifier add-on for FLAN T5 and GPT-3.5. Here, we use - Pr(Uhin ALIGNED) as the unanswerable probability and use the SQuAD v2 validation split find the best unanswerable threshold that maximizes the F1 score. The prompts we use and other experiment details are discussed in the appendix (Section D).
Metrics We follow Rajpurkar et al. [76] and report exact match and macro-averaged F1 score. To evaluate each model's performance at identifying unanswerable questions, we also formulate the problem binary classification task (predicting whether the sample is answerable) and report the ROC AUC [71]. A higher ROC AUC indicates the model is better at identifying unanswerable questions. For GPT-3.5 and FLAN T5, we consider the unanswerable classifier output to be 0 if the model predicts an answer, or 1 otherwise.
4.3.2 Results
As shown in Table 4, using ALIGN as a verifier add-on significantly improves GPT-3.5 and FLAN T5 in most cases (increases exact match score by 17.94 on average and Fl score by 15.05), suggesting that it is effective at identifying unanswerable questions. For the Simplified NQ dataset, adding the alignment verifier to GPT-3.5 degrades exact match and Fl score, but improves AUC. This indicates that while ALIGN produces meaningful unanswerable probabilities on the Simplified NQ dataset, the threshold found on the SQuAD v2 validation split not ideal for Simplified NQ. Repeating the experiment with the best threshold selected on the Simplified NQ validation split (see numbers in parenthesis in Table 4) shows the potential for improvements in exact match and F1 scores, albeit this can no longer be considered a zero-shot setting.
Table 4: QA experiment results. Cases where adding the ALIGN verifier improves performance highlighted in green. Best model for each dataset is shown in bold. For the combination GPT-3.5 + Verifier and Simplified NQ, also report the exact match and F1 scores with the best unanswerable threshold selected on the Simplified NQ validation split in parenthesis.
SQuAD
ACE-whQA
Simplified NQ
FM
F1
AUC
FM
F1
AUC
FM
F1
AUC
GPT-3.5
52.53
63.96
0.76
67.98
71.98
0.77
58.37
68.61
0.81
FLAN
75.72
79.01
0.83
26.29
29.24
0.51
38.24
44.98
0.58
GPT-3.5 + Verifier (Ours)
67.19
77.63
0.93
79.02
80.91
0.84
56.16
$7.40
0.86
(63.51)
(71.8)
FLAN Verifier (Ours)
83.72
86.55
0.95
75.75
77.60
64.93
67.99
0.83
Table 5: Ablation results on in-domain natural language understanding tasks. Each row corresponds with model trained on data adapted from incrementally more types (+) of tasks from scratch, or from fewer types (-) of tasks from the entire aligmment training dataset. For example, the model on the second row is trained with NLI, Fact Verification and Paraphrase tasks. +QA (ALIGN- base) refers to the model trained on the same data as ALIGN-base. We report the average performance for each evaluation tasks. The last column shows the overall average for all the evaluated tasks. The best model for each evaluated task is shown in bold.
Evaluation Tasks
Training Tasks
NL.
Fact Verification
STS
Paraphrase
QA
Coreference
Average
+NLI
69.0
65.3
53.7
69.3
56.0
70.6
64.0
+FV, Para
70.1
83.0
51.6
92.4
53.6
67.9
69.8
+Coref, Sum, IR, STS
69.9
82.9
90.3
91.9
51.8
83.2
78.3
+QA (ALIGN-base)
70.9
83.3
89.9
91.4
78.2
81.4
82.5
Synthetic
70.4
83.1
90.1
92.0
78.6
83.1
82.9
-pt-3.5-00 5.t.-rrb.. https:: https::/platforr openai com//ocs/nodels//ee

---

4.4 Ablation Study
As discussed in Section 3.1, ALIGN trained on datasets from a wide set of language understand tasks. To understand their contributions to the performance of the alignment model, we conduct an ablation study by incrementally adding subsets of tasks to the training set. Specifically, we start with only NLI dataset, and then add the remaining tasks in the following order: 1) paraphrase detection (para) and fact verification (FV) datasets; 2) coreference resolution (coref), summarization (sum), information retrieval (IR), and STS datasets, and lastly 3) QA datasets. Additionally, we train an alignment model without synthetic data to measure the contribution of such data. For simplicity, we use RoBERTa- as the backbone in this experiment. As shown in Table 5, each added subset improves the overall performance of the alignment model, suggesting our training tasks are compatible and contribute to the model performance. We notice that removing the synthetic data could slightly improve the overall performance, possibly due to the quality of the synthetic data. We will leave this for future study.
Conclusion
We propose unify diverse language tasks into text pair alignment problem. This framework yields an alignment model (ALigN) that, despite being less versatile than LLMs, solves a wide range of language problems efficiently with superior performance. We show that ALIGN outperforms task-specific models finetuned on several NLU tasks while having performance comparable to L.MSs that are orders of magnitude larger. Additionally, ALIGN excels in factual consistency evaluation, and can be used as an add-on to augment LLMs in QA tasks by identifying unanswerable questions.
Limitations Our alignment framework uses splitting and aggregation to handle long inputs (see Section 3). with the assumption that text 22 is short and its sentences are self-contained. While we empirically show this method works well on diverse datasets, violating this assumption has a few implications. First, if text 2 sentences are highly interrelated, splitting them discards document-lews semantic information, which could degrade performance. Second, as we need to evaluate all text 272 sentences individually, doing so will be slow for long text 22
We use wide collection of NLU datasets to learn the alignment function, with the assumption that these dataset, after being adapted into the text pair alignment format, accurately reflect our definition of alignment. However, as with all datasets, they could contain biases that are subsequently learned by our alignment model. Additionally, we augment the training set with synthetic data. While it proves to improve performance in our experiments, synthetic data likely do not perfectly model real-world data distributions.
References
[1] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals. Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL URI.http::/// Survey Certification.
aet//forurP.d-yyakSU5Zddoo.
[2] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger. Tom Henighan, Rewon Child. Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Process- ing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6- 62, 2020, viriual, 2020. URL https:/ httpe::/pooceedeieeeee wwwpeeeiinnnnnnniis...eemee wwwrips.cc/paper//// /tteee///2/////// 202h/hash/ (t4.:00060..b0044404444444444441
42f6c4a-14444-a Abstract htmml
[3] Maníš Pikuliak. Chatgpt survey: Performance on NI.P datasets, Mar 2023, URR.http:://openaamiie com/ cww/posts///ccccccccccoccoos http://sssssattttt..tttt survey//
[4] Wenpeng Yin, Nazneen Fatema Rajani, Dragomir Radev, Richard Socher, and Caiming Xiong. Universal natural language processing with limited annotations: Try few-shot textual entailment as start. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP).
10

---

pages 8229-8239, Online. November 2020. Association for Computational Linguisties. doi: 18.18653/v/ 2020.emnlp-mmt waaamnllllmmimeoom m66..660.. URL htsps:://aclaathhooogy www.ogg........e.o.. :ttp/2000...mmmmmmm...... 660.
[5] Sinong Wang. Han Fang. Madian Khabsa, Hanzi Mao, and Hao Ma. Entailment as few-shot learner. CoRR, abs/2104.14699 2021. URL https:///axiis http::/:///a.ivooog/aa//////////
14690.
[6] Anshuman Mishra, Dhruvesh Patel, Aparna Vijayakumar, Xiang Lorraine Li, Pavan Kapanipathi, and Kartik Talamadupula. Looking beyond sentence-level natural language inference for question answering and text summarization. In Proceedings the 2021 Conference of the North American Chapter of the Asxociation for Computational Linguistics: Human Language Technologies, pages 1322-1336, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v 1/2021 Lnaacl-mait main. 104. URL https://aclant lttps:://aclanthooogg org/2021.
104.
[7] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 1/N18 1101. URL
werg/N18-1100
[8] Xiaodong Liu, Pengcheng Hc, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4487 4496, Florence, Italy, July 2019. Association for Computational Linguistics. doi: aalpaaaaooattooooyooooooomo httpp://aclaanthoolgg
10. N..18653/1//11- 1441. URL https:
org/P19-1441
[9] Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. Muppet: Massive multi-task representations with pre- finetuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5799-5811, Online and Punta Cana, Dominican Republic, Novemher 2021. Association for Computational I Linguistics. doi: 1tt5333/1/2211.mmppmmo
468. URL https: battps://aclin htt////acaaannloo////// org/ 22211 emnlp main. 468.
[10] Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. MVP: multi-task supervised pre-training for natural language generation. CoRR, abs/2206. 12131, 2022. doi: 10.485550a22iv22266.122 12131. URL
https: //doi. org/ 10. 48550/ aar.v..rrvv.v.. 12131.
[11] Yuheng Zha, Yichi Yang. Ruichen Li, and Zhiting Hu. AlignScore: Evaluating factual consistency with a unified alignment function. In Proceedings the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 11 11328-1133 1348, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 120533//12233..llloomg 2023. acl-long. 634.
URL. http:ttta://// stttppa//aclaanthllogooo org/
[12] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.1 1692, 2019. URL http://arxiv..ot orgg 1907. 11692.
[13] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21:140:111111 1040:67, 2020. URL /ttp://jmlr org/ happp/pper////e/e (apprss/v1//220 /2////111222/22///2 html.
[14] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghami, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetueed language models. CoRR, aabs2210.111 1416, 2022. doi: 20..5000arii.2200.1444.. URL https: bttppa://daa //to::/doooooooooo .togiioog/111. 2rg/10..o 48550/ arXiv. 2210 11416.
[15] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin, A neural probabilistic language model. J. Mach. Learn Res., 3:1137-1155, 155,, 2003. URL attp://jmlz: .oria...gggg org. wwtgg/apprs//vve hwtps//////b/ee/........ html.
[16] Jacob Devlin, Ming-Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 4171-48 4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/NI33 I/N19. 1423. URL https://aclantho/ org/
N19-1444 1423.

---

[17] Bowen Tan. Yun Zhu, Lijuan Liu, Eric Xing, Zhiting Hu, and Jindong Chen. Cappy: Outperforming and boosting large multi-task Ims with a small scorer. NeurIPS, 2023.
[18] Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, 7aid Alyafcai, An- toine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Tacwoon Kim, Gunjan Chhablani, Nihal V. Nayak, Dehajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang. Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey. Rachel Bawden, Thomas Wang. Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan. Teven Le Scao, Stella Biderman, Leo Gao. Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL betps://opense
[19] Zhiting Hu and Eric P. Xing. Toward 'Standard Model' of Machine Learning. Harvard Data Science Review, 4(4), oct 27 2022. httpss//ndss.miiprss...i http::/ss...iiddddd/////kbbbbb..
[20] Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistiess 19(2):263-311 1993. org/I93 293-2003.
URI. https: hacppp///l/la
[21] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Anmual Meeting on Association for Computational Linguistics, ACL '02. page 311-318 USA, 2002. Association for Computational Linguistics. doi: 10. 015311007003.07733333 URL htt.a:tp//// //doi. org/10. 10. 31110..1.11111111111 3115/1073083. 1073135.
[22] Tianyu Gao, Xingeheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning sentence embed- dings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894 6910, Online and Punta Cana, Dominican Republic, November 2021. Association for Com- putational Linguistics. doi: 10.18853//1/2021.. .empl/ main.552. URL httpss://cclaattooolo 2021. emnlp- mommp-mmin........ 552.
org/
[23] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.ne, 2020. URL id Cd-SkeiuuVFDD.
[24] Tal Schuster, Adam Fisch, and Regina Barzilay. Get your vitamin C! robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Asxociation for Computational Linguistics: Human Language Technologies, pages 624-643, Online, June 2021. Association for Computational Linguisties. doi: 10.18653/v1/2221 1000:3//1/222..macllmoo m2. URL https: /tttps://allanthooogg org/2021 srrelennaccl-am naacl-main.52
[25] Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase adversarics from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Papers). pages 1298-1308. Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653/v1/ N19-1131, URL https: /cc aclantollogy www//oggg/i//eeooo 1131.
[26] Kai Sheng Tai, Richard Socher, and Christopher D. Manning. Improved semantic representations from tree- structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1556-1566, Beijing, China, July 2015. Association for Computational Linguistics. doi: 113111//1//P151111111
URL htttp::/aacaatthllolo. org/P15-1150.
[27] Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurahh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated machine reading comprehension dataset. CoRR, abs/1611 0111666119992666 1092688 2016. URL http:/ //arivv///ivv//// org/ abs/1611 09268.
[28] Guokun Lai, Qizhe Xie, Hanxiao Liu. Yiming Yang. and Eduard Hovy, RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785-794 Copenhagen, Denmark. September 2017. Association .or//
for Computational Linguistics. doi: 18.11863/v11 1/D11//1111111822
URI.
D17-1082.
[29] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't oon't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784 789, Melbourne, Australia, July 2018 Association for Computational org/P18- 21224
Linguistics. doi: 10. M/6633/11/P18 2124. URL
12

---

[30] Kellie Webster. Marta Recasens. Vera Axelrod, and Jason Baldridge. Mind the GAP: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605-617,
2018, doi: 10.1162/tacl 00240. URL https::/aclaathollaatt
org/Q18- 10442.
[31] Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. SummaC: Re-visiting NIJI-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10: 163-177, 2022. doi: 10.1 (0.1162/tac_aa_000 a__00433. URL. https: http:///aaaaaaaaaaaacc/ .wgg//oogg/gg/ 2022. tac1-1.10.
[32] Reinald Kim Amplayo, Peter J. Liu, Yao Zhao, and Shashi Narayan. SMART: sentences as basic units for text evaluation. CoRR, abs/2208.0103 2022. doi: 10445550)axx..22200.0000 010000 URL https:
/ //doi. org. 0w1.448550/aaa/. 22..20202000 01030.
[33] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations ICLR 2017, Toulon, France, April 24-26, 2017. Conference Truck Pruceedings. OpenReview.net, 2017. URL https: / http////ppp//eeeeeeeeeeeeeeeeeee wwt.rviwwwneeeeeee www.:nneefoommmmeccc Byj72dxxe.
[34] Marcin Junczys-Dower Mowzyy--Dommunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. Marian: Fast neural machine translation in C++. In Proceedings of ACI. 2018, System Demonstrations, pages 116-121, Melbourne, Australia, July 2018. Association for Computational URL https:://cclaataaattt
Linguistics. doi: 10. 10.18833/11/11-44000
org/P18-4020
[35] Federico Barrios, Federico López, Luis Argerich, and Rosa Wachenchauzer. Variations of the similarity function of textrank for automated summarization. CoRR, abx/1602.03606, 2016. URL http: httt:://rxxiv... /arxiv. arg/abs/ten 11///66022.00 1302006006.
[36] Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the fac- tual consistency of abstractive text summarization. In Proceedings the 2020 Conference Em- pirical Methods in Natural Language Processing (EMNLP), pages 9332-9346, Online, November 2020. Association for Computational Linguistics. doi: 10. 10018633/112220.cmm main.750. URL org/2020. emmlp 750.
[37] Mingkai Deng. Bowen Tan, Zhengzhong Liu, Eric Xing. and Zhiting Hu. Compression, transduction, and creation: A unified framework for evaluating natural language generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7580-7605, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 186138611/2221 .mnlp- m99n URL httph:tt///// htth://aaaaooooo..ooo.ooooo org/2021. emnlp main. 599.
[38] Victor Sanh, Lysandre Dehut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs//110.00 108, 2019. URL http: http://aaavvv//o///e .rg/ haaaaba/199 /9900. 01108.
[39] Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 1885-4901, Online, July 2020. Association for Computational Linguistics. doi: 10. 1031 441. URL https: http::://alaaannn /awaa/tacaaantggcccccacaa org/ 2020. acl-main 441.
[40] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/v//D15-. 1075. URL https://aclanthe ogy.ooo....oo.oo...... org/ D15-1075.
[41] James Thorne, Andreas Vlachos. Christos Christodoulopoulos. and Arpit Mittal. FEVER: large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Lingwistics: Human Language Technologies, Volume I (Long Papers), pages 809-819, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18. 1074. URL httpt:://cclaataaaaaae hwwpchhoooogoooog/o...ee wrg/NN8--107 1074.
[42] Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task
I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-22 2017) pages 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 18.18653/v 10.18653/v//S7722200 //aclanthooloe org/S17-2000
URL https:
13

---

[43] Kornl Csernai. First quora dataset release: Question pairs. URL httph:tttt//ueee htpps://uuuurraaaaaaaau com/ -utaaaaataaaee-e---aa Rewe-eeeleeeeeeeeeeeeeeeee Question-Pairs
[44] Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (I.ong Papers), pages 252-262, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10..1663/v11/ 10//33/11/111111 1023. // //aclanthology org/ttt arg/W18-1 0118-102...
URL https:
[45] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2924-2936, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.1 10.18653/v 36653/v1/119 1300. URL https://aclanthhol.og org/N19- 1300.
[46] A Rogers, Kovaleva, M Downey, and A Rumshisky. Getting closer to ai complete question answering:
A set of prerequisite real tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
[47] Johannes Welbl, Nelson Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User geeerraed Text, pages 94 106, 2017.
[48] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. CoRR, abs/2301.13m 13688, 2023. doi: 10.485550arXiv.2301.. 2301. 13688. URL attps://dos:
org/10. 20048550//rxii 2301 13688.
[49] Alex Wang. Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems. 32, 2019.
[50] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNL.P: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. 1118653/11/188 1/446-
URL https: http::::///caanhologyy org/W18-5445
[51] SemEval-200 2014. Evaluation compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. URL https://alt..cri.. org/ semeval20eeeeeee wwwpraa111/aaaaaa 4/task1/.
[52] William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (/WP2005), 2005. URL https: / /aclanthologyy owg/Ior-----. 5002.
[53] Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. DREAM: A challenge data set and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics, 7:217-231, 2019. org/ htt/Q19--144
2019. doi: 10.1162/tac1 a 00264. URL battps://ache aclhooolonnnogoo.oo
[54] Oyvind Tafford, Matt Gardner, Kevin Lin, and Peter Clark. QuaRTz: An woen-ddmmane dataset of qualitative relationship questions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing FEMNLP-IJCNI pages 5941-5946, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/DU 1/D19/ 1608. URL /aatttss:/aalaattoolll
org/D19-1608
[55] Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. Faithful to the original: Fact aware neural abstractive summarization. In Sheila A. Mcllraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Secom AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (1AAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiona, USA, February 2-7, 2018, pages 4784-4791. AAAI Press, 2018. URI. https: //wpp. aaai. org/ocs/ /iwee/iidx...../
[56] Wojeiech Kryseinski. Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. Neural text summarization: critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNI) pages 540- 551, Hong Kong. China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/DD 1/D19- 1051. URL https::/accaatth////
org/D19-110
14

---

[57] Feng Nie, Jin-Ge Yao. Jinpeng Wang, Rong Pan, and Chin-Yew Lin. A simple recipe towards reducing hallucination in neural surface realisation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2673-2679, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.188811//111-1 1256. URL https: htt.ttt:: http::/:///aaaaahllogg.oo org/P19- 1256.
[58] Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, Online, July 2020. Association for Computational Linguistics. doi: 10. malmmmiin... 173.
18653/v1/0020.acl manel. 173. URL
[59] Or Honovich, Roce Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. TRUE: Re-evaluating factual consis- teney evaluation. In Proceedings of the Second DialDoc Workshop on Document-groundee Dialogue and Conversational Question Answering, pages 161-175, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/11/22222ialle 1.19. URL httpa://///// / ht/p//aaaaaatoogyoy...o.. org/ 2022. 1e222ddaalocc11111
[60] Alexander R. Fabbri. Wojciech Kryšcińskii Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. SummEval: Re-evaluating summarization evaluation. Transactions of the Asso- ciation for Computational Linguistics, 9:391-409, 2021. doi: 10.1 10.1166/tacl___M033 URL https:
// //aclanthollogy.ozz wtg/2021. tacl 1.
[61] Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008-5020, Online, July 2020. Association for Computational Linguistics. doi: 10. 18653/v1/2020.acl m-inmm4ii0 URL https: https////aaaaathho///oo
. org
acl-main main.main0 450.
[62] Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics. In Proceedings of the 202/ Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4812-4829, Online, June 2021. Association for Computational Linguistics. doi: moneel-mmin..380
10. 10.18653/v12021. .aaallnnaaaaaaae 383n.383, URL https: https::/aclantthlll.oeeoo hatpa/acaanntoogg...ooooo . org/2021
[63] Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human- annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79, 79, Hong Kong, China, November 2019. Association for Computa- tional Linguistics. doi: 10. 1011665311//11954499 URL. http::/ awlaaaaaaoooooooor///oaaaa
[64] Jinlan Fu. See-Kiong Ng. Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. CoRR. abs/2302.04166, 2023. doi: 10.48550WarXi..2222.4446 URL https: http:///doi.oo////// arg/1..48550/arr 48550/arXiv
2302. 04166.
[65] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using GPT-4 with better human alignment. CoRR, 1ts/2303.166344 2023, doi: 10.48550/arXiv. 1663.16634. URL https: //doi. org/10. 48550/ w.w.a................. 16634.
[66] Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. Human-like summarization evaluation with chatgpt. CoRR, abs/2304.02553 0225544 2023. doi: 00445500arXi..234400222
URL https://doi. org/ 10. 48550/ arriv./ 2304. 02554.
[67] Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metries for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881- 7892, Online, July 2020. Association for Computational Linguistics. doi: 10. 18653/ 18653//1/2020.ac/// w/w.//..ac.mmaiin..... 704. org/2020 acl-main.700
URL https: hatpp:://alaatttao/
[68] Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text text generation. M. Ranzato, A. Beygelzimer, Dauphin. P.S. Liang. and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 27263-27277. Curran Associates, Inc., 2021. URL attps://pprot https://proceedings www.iee..ccpeeeeeeeeeee wwtppppppee//2//iiiii. eld2266e6fdecca366600ff watpeefffe22eeeeeeee wwwoddddpappr.p...e
[69] Ming Zhong, Yang Liu, Da Yin. Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. Towards unified muuti-dimmnnionna evaluator for text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2023-200 2038, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https: // //aclanthology . arg/2022,oo cemlp-mairg 131.
15

---

[70] Alexander Fabbri, Chien-Sheng Wu, Weahao Liu, and Caiming Xiong. QAFactEval: Improved QA-based factual consistency evaluation for summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 2587-2601, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 11/33//12022.maa/ main. 187. URL https: http:///aaaaaahhoooo weggg..rg/2222. naacc main. 187.
[71] Andrew Bradley. The use of the area under the roc curve the evaluation of machine learning algorithms. Pattern recognition, 30(7):1 145-1111111 1159, 1997.
[72] Minghao Hu, Furu Wei, Yuxing Peng. Zhen Huang, Nan Yang. and Dongsheng Li. Read verify: Machine reading comprehension with unanswerable questions. In The Thirty-Third Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-Firsi Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposiun on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawati, USA, January 27- February 1, 2019, pages 10. 16099aaai.v33110 w33.01.3333000000 330166,,,,, URL https:///oi..... org/
AAAI Press, 2019, doi: /arg/..0./ 10. 1609 aaai. v33101. 3301652....
[73] Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, and Rui Wang. Sg-net: guided machine reading comprehension. In The Thirry- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY,
USA, February 7-12, 2020, pages 9636-9643. AAAI Press, 2020. URL https: /ojs aaai /6111.
php/
[74] Elior Sulem, Jamaal Hay, and Dan Roth. Do we know what we don't know? studying unanswerable questions heyond SQuAD 2.0. In Findings of the Association for Computational Linguisties: EMNLP 2021, pages
Punta Cana, Dominican Republic, November 1021. Association for Computational Linguistics. doi: 10. 18653vv 1/202/1112222l2/2 .indingss--mmnll...... URL https: htttps://ccaantoolllo aclattoooontttococoooooooooo.ooe or2/2021. findings-emnlp.55 385.
[75] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alherti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang. Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452-466, 2019. doi: 10. 1211622aclaaa URL https: //doi. org/ 10. /111//tacllaa./a a_ a_0276.
[76] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. 118833/11/D166 1220466664...00111 URL https://aclanii: althooooy. org/D16- 1264.
[77] Wenpeng Yin. Dragomir Radev, and Caiming Xiong. DocNLI: A large-scale dataset for document- level natural language inference. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 4913-4922, Online, August 2021. Association for Computational I Linguisties. doi: 10. 10.1866 18663/v1/22222 wwiniings-all.433... URL https: // accl anthhoooggyyyo.o.. .aggyy.or. prg/2021... findingss-ach acl. 435.
[78] Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zam- parelli. A SICK cure for the evaluation of compositional distributional semantic models. In Proceed- ings of the Ninth International Conference on Language Resources and Evaluation (LREC' pages 216-223, Reykjavik, Iceland, May 2014. European I Language Resources Association (ELRA). URI. http:/ //tt::///ww... farec-conf-.. wttp://pooeeedggg/eeee////////// /363 /36__Paper_______
[79] Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. Beat the Al: Investigating adversarial human annotation for reading comprehension. Transactions of the Association for Computational Linguistics, 8:662-678, 2020. doi: 10.1 162/tacl a 00338. URI.https. httpa////ccaaalolooooa/o
org/2020. tac1-1.43.
[80] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabricl Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.11853//1/N19... 1246. URL attps://aclanthol/
org/N19--221 1246.
[81] Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369 2380, Brussels, Belgium. October-November 2018. Association for Computational Linguistics. doi: 10. 18653/v1/D18. 1259. URL
hww.ooolloyyooggggee. org/D18- 1259.
16

---

[82] Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learming for NLP, pages 191-200, Vancouver, Canada, August 2017. Association org/
for Computational Linguistics. doi: 10.18655/1//177262255 URL httpe://aclantholl W17-2623.
[83] Pradeep Dasigi, Nelson Liu, Ana Marasović, Noah A. Smith, and Matt Gardner. Quoref: A reading comprehension datasct with questions requiring corcferential reasoning. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pages 3925-5932. Hong Kong, China, November 2019. Association Computational Linguisties. doi: 10.18653/1/D19 1606. URL https: //aclanthology. org/D19-1606.
[84] Kevin Lin, Oyvind Tafjord, Peter Clark, Matt Gardner. Reasoning over paragraph effects situations. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 58-62, Hong Kong. China, November 2019. Association for Computational Linguistics. doi: URL attpa://aclanttoo/
[85] Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings the 3rd Workshop on Noisy User-generated Text, pages 94-106, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653//1/W11-4411, URL https: //aclanthology org/org////.. 21///117-00 W17-4413.
[86] Mor Geva. Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361 3611, 2021. doi: 1111622tacl_a000377 URL
https://aclanthholog" org/2021. 1.21.
[87] Mahnaz Koupace and William Yang Wang. Wikihow: A large scale text summarization datasct. CoRR, abs/1810.09305, 2018. URL
wrtp/bs//////////ooo 09305.
[88] Kevin Clark. Minh-Thang Luong, Quoe V. Le. and Christopher D. Manning. ELECTRA: pre-training text encoders discriminators rather than generators. In 8th International Conference on Learning Representations, ICIR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL. https://openreviiee net/forumm werermmmiddrii BMV....V..
17

---

Appendix
A Ethics Statement
While our text pair alignment model achieves state-of-the-art performance on many downstream tasks, like all models, does make mistakes. For example, when used for fact verification factual consistency evaluation, it could misidentify factually correct statements as incorrect and vice versa. Additionally, as we use publicly available datasets train the alignment model, might have learned biases inherent to those datasets. Thus, one should proceed with caution when using ALIGN for purposes other than NLP research.
Comparison with Other Model Types
We illustrate the major differences between our approach, LLMs, multitask learning models, and task-specific finetuned models in Table 6. Compared with LLMs, our alignment function is more efficient but less versatile. contrast task-specific finetuned models, the alignment function is more general and can handle more types of tasks. Unlike multitask learning models, we unify language tasks into a single text pair alignment problem, and share model components across multiple tasks (as apposed to using pataset-specifi prediction heads). As result, our alignment function can be directly applied wide range of tasks out-of-the-box: without any finetuning.
Table Comparison between the alignment function and other types of language models.
Type
Model Example
Efficient
Out of the box
General
LLM
T5, PALM, U1.2, GPT MT-DNN, MUPPET
Multitask learning
XX
Task specific I.M
Finetuned RoBERTa/BERT
V
X
Text pair alignment
ALIGN (Ours)
C Training Details
C.1 Trainig Setup
choose RoBERTa [12] as backbone the alignment model. ALIGN-base and ALIGN-large based RoBERTa-bee and RoBERTa-large. respectively. For the experiments in Section 4,
we train ALIGN for epochs with a batch size of 32, following common practice [12, 16). Other hyperparameters are listed in Table 7. For the finetuned RoBERTa and RoBERTa-NLI model in Section 4.1, we set batch size to 16 and 32, respectively.
Table The hyperparameters for training the alignment model.
Hyperparameter
ALIGN-base
ALIGN-large
Parameters
125M
355M
Batch Size
32
32
Epochs
3
3
Optimizer
AdamW
AdamW
Learning Rate
le-5
le-5
Weight Decay
0.1
0.1
Adam e
lc-6
le-6
Warmup Ratio
0.06
0.06
Random Seed
2022
2022
GPU
2x3090
4xA5000
GPU Hours
152h
620h
18

---

C.2 Training Datasets
collect datasets that falls into the scope alignment as mentioned Section 3. Table 8 lists datasets we use for training the alignment model. The size of these datasets ranges from 4k samples to 5M. Most the datasets are used for binary classification task except some NLI, fact verification and STS datasets.
We only use the first 500k samples in each dataset due to limited computation resource, which results in 5.9M training samples in total. During training, the samples are randomly sampled from the entire adapted training sets.
Table The training datasets ALIGN. Note due to resource constraints, we only use at most 500k
samples from each dataset to train the alignment model.
NLP Task
Dataset
Training Task
Sample Count
SNLI [40]
3-way classification
550k
MultiNLI [7]
3-way classification
393k
NLI
Adversarial NLI [39]
3-way classification
163k
DocNLI [77]
binary classficiation
942k
NLI-style FEVER [41, 39]
3-way classification
208k
Fact Verification
VitaminC [24]
3-way classification
371k
QQP [43]
binary classficiation
364k
PAWS-Wiki [25]
binary classficiation
695k
Paraphrase
PAWS-QQP [25]
binary classficiation
12k
WikiText-103 [33]
binary classficiation
8M
SICK [78]
regression
4k
STS
STSB [42]
regression
6k
SQuAD v2 [29]
binary classficiation
130k
RACE [28]
binary classficiation
35lk
Adversarial QA [79]
binary classficiation
60k
BoolQ [45]
binary classficiation
19k
DROP [80]
binary classficiation
155k
MultiRC [44]
binary classficiation
24k
QA
HotpotQA [81]
binary classficiation
362k
NewsQA [82]
binary classficiation
161k
QuAIL. [46]
binary classficiation
41k
Quoref [83]
binary classficiation
39k
ROPES [84]
binary classficiation
22k
SciQ [85]
binary classficiation
47k
StrategyQA [86]
binary classficiation
5k
Information Retrieval
MS MARCO [27]
binary classficiation
5M
Summarization
WikiHow [87]
binary classficiation
157k
Coreference
GAP [30]
binary classficiation
4k
D Additional Experiment Details
D.1 Natural Language Understanding Tasks
Prompts For FLAN models, we use the same prompts as Longpre et al. [48]. For datasets that do
not appear in Longpre et al. [48], prompts of similar tasks. The prompt used for each dataset listed below.
MNLI, NLI-FEVER, VitaminC:
"Premise: {premise) \n hnynHyothhesss:: Sapptthesis//nNnDeeer the premise entail the hypothesis?An yes AnB it is not possible to tellinG no"
19

---

ANLI:
you say that these questions are the "[context) An (naeddd on the paragraph above can we conclude that \ " YnB It's impossible to say\nC No
SNLI:
"If \"'premise>\" does this mean that \"dbypothesiss yes it is not possible to tell no"
SICK. STSB:
"(sentencel) wwtpr//annnnnnnneennaaaaaaa AnnnRaee the textual similarity of these sentences on a scale from 0 to 5, where 0 is \"no meaning overlap)" and
is \mmaan the same thing)"..me AnAnA O\nB 1n 2nnD 3 4\nF 5"
PAWS, PAWS-QQP:
"(sentence1) wewieeeeeenssseeteeeeeenneeeeeee AnnDo these sentences mean the same thing?\nA no AnB yes"
QQP:
(questiion//nnquusttio21111 same?\nA no no\nB yes"
RACE, QuAIL, SciQ:
(aaoct))n(quettioo)uMMa Annd (option 1MnB Coption 2} ncC {option 3} ..."
Multi-RC:
\"[question] "nnna""nnnnnnnnnn" "Reeoosssp
\"[response>\ \ "nnnDoe the response correctly answer the question? AA noBnB yes"
BoolQ:
"(text) AnnCan we conclude that (question)7\n\nA nolnB yes"
GAP:
"Context: {context]ln Given the context, which option is true? \n\nA foption 1mnB {option 2) (option 3}
Multitask RoBERTa Baseline To obtain the multitask-learning RoBERTa model, we follow the popular multi-task learning work [9], and train the same base model with the same set of tasks (datasets) as our alignment model. Notably, different from our alignment model that uses unified interface accommodate all diverse tasks, the conventional multitask-leaming model learns separate prediction heads for different tasks.
Instruction Finetuned T5 Baseline In order to understand the performance difference between our text alignment formulation and instruction fine-tuning, we instruction-finetus T5-base (250M parameters) on the same datasets as our alignment model. We do not convert QA tasks since T5 naturally supports sequence generation. We follow the prompts mentioned Chung et al. [14], Longpre al. [48] and format the datasets to train the T5 model.
Results We show the performance of finetuned RoBERTa and FLAN-Alpaca these datasets in Table 9, while the result of the multi-task RoBERTA is in Table 10 and 11. The comparisons with the T5 model instruction finetuned on alignment datasets are in Table 12 and 13. We have compared ALIGN with finetuned RoBERTa on these datasets in Figure 2. When comparing FLAN-TS and FLAN-Alpaca, notice FLAN-TS consistently outperforms FLAN-Alpaca all scales. Thus, compare our alignment model with FLAN-T5 in Table 1.
D.2 Factual Consistency Evaluation for Language Generation
Detailed Result In this section, we report the detailed results associated with Figure We list the performance of each metric on SummaC Benchmark and TRUE Benchmark in Table 15 and Table 16, respectively. We also show the Pearson Correlation, Spearman Correlation and Kendall's tau Correlation coefficients on other datasets in Table 17, 18 and 19, respectively.
20

---

Table 9: The performance of finetuned RoBERTa and FLAN-Alpaca on the in-domain datasets. We report the average performance of each model and we also include the average without RACE and QuAlL.
Finetumed RoBERTa
FLAN-Alpaca
base
large
base
large
xlarge
Model Parameters
125M
355M
220M
770M
3B
MNLI-mm
87.2
90.3
79.9
86.4
89.3
MNLI-m
87.9
90.6
80.0
87.2
89.4
ANLI-1
62.8
72.7
47.4
65.7
74.8
NLI
ANLI-2
44.5
48.3
38.2
46.6
57.6
ANLI-3
42.8
47.0
37.7
46.4
54.6
SNLI
91.0
91.4
82.9
88.1
90.2
NLI-FEVER
76.1
77.7
69.6
73.0
72.1
Fact Verification
VitaminC
89.3
91.6
63.3
72.5
77.4
SICK
88.9
84.7
37.7
66.4
70.1
STS
STSB
89.8
90.6
33.4
52.5
79.5
PAWS
92.3
92.5
68.1
92.0
93.0
Paraphrase
PAWS-QQP
94.7
94.2
57.6
85.1
87.1
QQP
91.1
92.0
75.5
81.6
86.5
RACE-m
74.6
24.0
64.3
78.5
87.8
RACE-h
67.8
23.9
57.3
71.9
82.9
Multi-RC
77.5
85.5
64.2
84.3
87.1
QA
BoolQ
79.1
85.7
71.7
82.0
87.2
QuAIL
57.7
27.0
56.7
78.2
84.2
SciQ
93.4
95.5
90.8
83.1
95.6
Coreference
GAP
74.3
89.8
58.4
65.6
80.7
Average
78.1
74.7
61.7
74.4
81.4
Average w/o RACE, QuAIL
80.2
83.5
62.1
74.0
80.7
Ablation Study In Equation 1, we propose to aggregate chunk-sentence scores by taking the maximum score over context chunks and then average over claim sentences (mean-max). Another reasonable choice is to take the minimum score over claim sentences instead of the average (min-max). evaluate the two aggregation methods on factual consistency evaluation tasks and report the results in Table 20. Overall, the mean-max method leads to better correlation with human annotation. We speculate this is because taking the average helps remove some of the noise introduced by the alignment estimator, while the min min-max setup would be easily affected by underestimation.
D.3 Question Answering with Unanswerable Question
Simplified Natural Questions For this experiment, we construct new SQuAD-style QA dataset with unanswerable questions, Simplified Natural Questions (Simplified NQ), base on Natural Ques- tions [75]. For each sample (an search query and Wikipedia article) in Natural Questions, [75] ask five annotators to find 1) an HTML bounding box (the long answer: e.g., paragraphs, tables, list items, whole list) containing enough information to infer the answer to the query (long answer), and 2) short span within the long answer that answers the question (the short answer). For both short and long answers, the annotators can alternatively indicate that answer could not be found.
For the purpose of constructing Simplified NQ, we consider a sample be answerable if at least annotators identified both long and short answers. In this case, we use the most popular (among the annotators) short answer as the ground truth answer, and the most popular long answer containing selected short answer as the context. Conversely, if less than 2 annotators identified any long answer, and less than 2 annotators identified any short answer, we consider the sample be unanswerable and use random paragraph from the article the context. We discard all remaining samples to
21

---

Table 10: The performance of multi-task RoBERTa-baed and ALIGN-base on the in-domain datasets. The last row is the averaged performance each model on these datasets.
Task
Dataset
Multitask-ban
AL.GGN-bas
MNLI-mm
87.61
87.54
MNLI-m
87.68
87.82
ANLI-I
65.10
65.30
NLI
ANLI-2
48.20
48.70
ANLI-3
46.17
45.50
SNLI
91.33
90.78
NLI-FEVER
76.50
76.78
Fact Verification
VitaminC
89.82
89.79
SICK
89.01
90.71
STS
STSB
87.86
89.03
PAWS
93.94
92.33
Paraphrase
PAWS-QQP
92.32
91.88
QQP
90.66
90.07
RACE-m
78.34
76.95
RACE-h
71.56
68.84
QA
Multi-RC
83.83
82.20
BoolQ
81.32
81.07
SciQ
92.10
92.40
Coreference
GAP
81.65
81.35
Average
80.79
80.48
Table 11: The performance of multi-task RoBERTa-base and ALIGN-base on the zero-shot datasets. Multi-task RoBERTa uses musk-specific prediction heads for each its training sets. To test on zero-shot datasets. we evaluate a set of "reasonable" heads obtained during training (c.g,, we use heads trained on NLI datasets for NLI evaluation tasks) and report the average (avg. of heads), best (best head), and worst (worst head) performance of these heads. "Best head" assumes oracle access evaluation results, and is thus effectively an unrealistic upper bound. The last row the averaged performance of each configuration on these datasets.
Multitask-bass (avg. of heads)
Multitask-bass (best head)
Multitask-bass (worst head)
Task
Dataset
ALIGN-base
AXB
74.94
76.18
72.37
75.09
AXG
63.58
65.73
60.39
59.83
CB
80.00
83.93
76.79
76.79
NLI
RTE
81.41
81.95
81.23
83.39
WNLI
56.34
60.56
47.89
52.11
SE14T1
57.86
58.45
57.17
90.72
Paraphrase
MRPC
69.60
71.19
68.23
65.97
DREAM
67.04
73.98
57.03
71.34
QA
Quartz
59.09
63.52
54.46
59.69
Average
67.76
70.61
63.95
70.55
Table 12: The performance of ALIGN-base and the instruction finetuned T5-base (IF-T5-base) on the in-domain tasks. The last column shows the average performance on the in-domain tasks.
Model (Size)
NLI
Fact Verification
STS
Paraphrase
QA
Coreference
Average
ALIGN-base (125M)
70.9
83.3
89.9
91.4
78.2
81.4
82.5
IF- IF-T5-base base (222M)
53.7
68.7
62.0
83.2
30.3
42.9
56.8
22

---

Table 13: The performance of ALIGN-base and the instruction finetuned T5-base (IF-T5-base) on the zero-shot tasks. The last row shows the average performance on the in-domain tasks.
Dataset
Alignment-ben base
IF-TS-base
AXB
75.1
65.3
AXG
59.8
50.8
CB
76.8
58.9
RTE
83.4
59,9
WNLI
52.1
47.9
SEI4TI
90.7
48.5
MRPC
66.0
66.1
DREAM
71.3
35.8
Quartz
59.7
50.6
AVG
70.5
53.8
Table 14: The performance of RoBERTa-NLI and FLAN-Alpaca on the zero-shot datasets (of alignment model). The gray number shows the specific dataset is appeared in the training set of FLAN-TS. report the average performance of each model the last row.
RoBERTa-NLI
FLAN-Alpaca
base
large
base
large
xlarge
Model Parameters
125M
355M
220M
770M
3B
AXB
75.2
79.2
53.6
72.3
77.2
AXG
59.6
73.6
49.4
72.5
88.8
CB
85.7
87.5
78.6
78.6
87.5
NLI
RTE
81.2
88.1
72.9
79.8
87.0
WNLI
52.1
50.7
40.8
62.0
71.8
SE14T1
91.2
93.1
65.0
72.4
77.3
Paraphrase
MRPC
38.7
42.3
66.7
75.4
83.1
DREAM
63.9
73.0
63.5
76.8
89.5
QA
Quartz
54.6
65.6
68.1
87.4
90.2
Average
66.9
72.6
62.1
75.2
83.6
Table 15: The ROC AUC of each metric on the SummaC benchmark. CGS and XSF are abbreviations of CogenSumm and XSumFaith, respectively. The strongest performance on each dataset is shown in bold. The last column shows the average performance each dataset in the SummaC benchmark.
SummaC Benchmark
CGS
XSF
PolyTope
FactCC
SummEval
FRANK
AVG
BERTScore
63.1
49.0
85.3
70.9
79.6
84.9
72.1
BLEURT
60.8
64.7
76.7
59.7
71.1
82.5
69.2
BARTScore
74.3
62.6
91.7
82.3
85.9
88.5
80.9
CTC
76.5
65.9
89.5
82.6
85.6
87.3
81.2
UniEval
84.7
65.5
93.4
89.9
86.3
88.0
84.6
QAFactEval
83.4
66.1
86.4
89.2
88.1
89.4
83.8
ALIGN-base (Ours)
80.6
76.1
87.5
93.1
88.6
89.5
85.9
ALige-large (Ours)
88,4
74.6
92.5
94.9
92.3
91.3
89.0
23

---

Table 16:
The ROC AUC of each metric the TRUE benchmark. The datasets with asterisks(*) appear in the training set the alignment model. compute both the overall average on all datasets (Average) and average without PAWS, FEVER, VitaminC datasets (Average -S). The latter shows the zero-shot performance ALIGN. Bold indicates the best performance on dataset.
ALIGN -base (Ours)
ALIGN
BeRTScore
BLEURT
BARTScore
CTC
UniEval
QAFactFval
-large
(Ours)
FRANK
84.0
81.6
87.8
87.1
88.1
88.5
90.4
91.4
SummEval
72.3
68.0
78.9
79.8
79.5
MNBM
52.5
65.5
63.5
65.0
66.8
67.3
76.4
74.4
QAGS-C
70.6
71.2
83.9
77.3
86.5
83.9
80.3
89.0
QAGS-X
44.3
56.2
60.2
67.7
76.7
76.1
79.6
83,2
BEGIN
86.4
86.6
86.7
72.0
73.6
81.0
81.3
81.1
TRUE
Q2
70.2
72.9
65.1
66.8
70.4
75.8
77.2
Benchmark
DialFact
68.6
73.0
60.8
63.7
80.4
81.8
83.3
85.1
PAWS#
78.6
68.4
77.1
63.1
80.1
86.1
97.6
98.4
FEVER*
54.2
59,5
66.1
72.5
92.1
86.0
94.7
94.9
VitaminC
58.2
61.8
64.2
65.0
79.1
73.6
97.8
98.3
Average
67.2
69.5
72.2
70.9
79.5
80.1
85,3
87.2
Average-ZS
68.6
71.9
73.4
72.4
78.0
79.4
81.0
83.4
Table 17: The Pearson correlation coefficients of various metrics other datasets mentioned in Section 4.2.1. Q-XSum and Q-CNNDM are abbreviations of QAGS- -AGS-XSum and QAGS-CNNDM, respectively. F-XSum and F-CNNDM are abbreviations of TRANK-XSSu and FRANK-CNNDM, respectively. The last column shows the average performance on each dataset. The best performance shown in bold.
Other Datasets Pearson
XSumFaith
SummEval
Q-Xsum
Q-CNNDM
F-Xsum
F-CNNDM
SamSum
AVG
BERTScore
13.0
33.1
51.7
13.0
$1.7
BLEURT
38.7
23.8
13.2
45.2
15.6
37.5
8.1
26.0
BARTScore
29.3
35.5
16.3
71.5
23.7
51.9
15.0
34.7
CTC
27.2
54.7
30.6
64.5
20.0
54.5
16.9
38.3
Unilval
23.9
57.8
45.5
66.7
27.2
58.3
23.2
43.2
QAFactEval
30.3
61.6
44.2
68.4
32.1
64.6
38,9
48.6
ALIGN-base (Ours)
33.2
57.8
51.1
60.9
31.2
61.8
21.1
45.3
ALIGN-large (Ours)
28.8
66.7
53.9
76.1
38.9
68.9
47.7
54.4
Table 18: The Pearson correlation coefficients of various metrics on other datasets mentioned in Section 4.2.1. The format in this table follows Table 17.
Other Datasets Spearman
XSumFaith
SummEval
Q-Xsum
Q-CNNDM
F-Xsum
F-CNNDM
SamSum
AVG
BERTScooe
13.4
31.5
-8.9
46.2
12.7
45.1
13.1
21.9
BLEURT
37.0
23.6
12.4
43.4
13.9
37.6
6.7
24,9
BARTScore
29.8
39.1
17.0
68.1
20.0
53.3
16.3
34.8
CTC
29.8
41.7
57.3
20.4
49.4
17.7
35.3
Uniliaa
25.3
44.3
67.6
26.7
22.8
QAFactEval
31.9
42.8
44.1
63.1
25.5
53.7
35.9
42.4
ALIGN-base (Ours)
38.8
42.0
$2.7
56.1
25.5
56.4
22.3
42.0
ALIGN-large (Ours)
32.1
47.9
57.4
71.6
30.0
49.7

---

Table 19: The Kendall's tau correlation coefficients of various metrics on other datasets mentioned in Section 4.2.1. The format in this table follows Table 17.
Other Datasets Kendall's
XSumFFaith
SummEval
Q-Xsum
Q-CNNDM
F-Xsum
F-CNNDM
SamSum
AVG
BERTScore
9.2
24.9
36.3
10.4
34.7
10.7
17.0
BLEURT
25.3
18.6
10.1
33.9
11.4
28.8
5.5
19.1
BARTScore
20.2
31,0
13.9
55.6
16.3
41.4
13.3
27.4
CTC
20.2
33.2
25.1
45.7
16.6
38.2
14.4
27.6
UniEval
17.0
35.3
40.9
54.4
21.8
42.4
18.7
32,9
QAFactEval
23.2
34.0
36.2
50.5
22.4
42.2
30.1
34.1
ALIGN-base (Ours)
26.6
33.4
43.1
45.5
20.8
44.4
18.2
33.1
ALIGN-large (Ours)
21.9
38.4
47.0
59.6
24.5
49.5
38.2
39,9
Table 20: The comparison between the mean aggregation method and the min aggregation method.
The bold number represents better performance when comparing the aggregation method.
ALIGN-base mean-max
ALIGN-base min-max
ALIGN- mean-max
ALIGN-large min-max
SummaC
85.9
86.1
89.0
89.1
TRUE
85.3
84.2
87.2
86.2
Other-Pearson
45.3
43.7
54.4
52.1
Other-Spearman
42.0
42.3
49.7
49.2
Other-Kendall
33.1
33.1
39.9
38.8
avoid ambiguity (c.g., some samples might only have long answers but not short answers). This results in total of 3336 answerable samples and 3222 unanswerable samples in the validation set.
Prompts and QA Inference For FLAN T5, follow [48] and use the following prompt: Context: {context] AnQuestion: <context?!uquestion:::
{question]\nAnan AnAnswr:
For GPT-3.5, we use prompt with additional instructions:
Find the answer to the question from the given context. When the question cannot be answered with the given context, say "umanswerable" Just say the answer without repeating the question. \nContext: [oontex)]\n[uesttete [question]\ans (nAsswer:
At inference time, we truncate the context necessary such that the entire input is at most around 2000 tokens long (2000 for FLAN T5, 2040 for GPT-3.5 to account for the longer prompt). We use greedy decoding for FLAN T5, the default chat completion settings for GPT-3.5. When FLAN T5 outputs "unanswerable" Similarly, if GPT-3.5's output contains any of "unanswerable" "no answer", "context does not provide answer", we consider the prediction to be unanswerable.
interpret it predicting the sample to be not answerable.
In addition to FLAN T5 and GPT-3.5, we also experiment with Flectra [88], one of the top performing single models on the SQuAD v2 leaderboard, for reference. Specifically, we reproduce Clark et al. [88]'s design that use QA prediction head to jointly predict the answer span and unanswerable probability. As shown in Table 21, while Electra is strong performer on SQuAD v2 and Simplified NQ, adding the alignment verifier to GPT-3.5 and FLAN T5 greatly reduces the performance gap. Additionally, on ACE-whQA, our design (both FLAN T5 and GPT-3.5 with alignment verifiers) outperforms Electra.
Additional Results
D.4 Ablation Study
We present the additional ablation result on factual consistency evaluation tasks in Table 22. This part follows Section 4.4, where we use the same checkpoints that are trained on incrementally added tasks. Result shows the training tasks are generally compatible and effective, though we notice adding fact verification and paraphrase detection tasks lead to a slightly performance drop. We speculate is due to the paraphrase detection task, where text pair is expected to have exactly the same information.
25

---

Table 21: Additional experiment results on QA with unanswerable questions including Electra. The best model for each task/metric is shown in bold.
SQuAD v2
ACE-whQA
Simplified NQ
EM
AUC
EM
F1
AUC
EM
AUC
Electra GPT-3.5 Flan T5
86.47
89.37
0.97
52.32
55.59
0.87
70.81
74.13
0.88
52.53
63.96
0.76
67.98
71.98
0.77
58.37
68.61
0.81
75.72
79.01
0.83
26.29
29.24
0.51
38.24
44.98
0.58
GPT-3.5 + Verifier (Ours)
67.19
77.63
0.93
79.02
80,91
0.84
56.16
57.40
0.86
FLAN T5 Verifier (Ours)
83.72
86.55
0.95
75.75
77.60
0.90
64.93
67.99
0.83
The ALIGN-base model, which uses all the possible training data, gets the best performance on every factual consistency evaluation task.
Table 22:
Ablation results on factual consistency evaluation tasks, Each row corresponds with a model trained with data adapted from incrementally more types of tasks. For example, the model on the second row is trained with NLI, Fact Verification and Paraphrase tasks. The model on the last row is the same Aligmment-base We report the average performance for each evaluation tasks. The last column shows the overall average for the factual consistency evaluation tasks. The best performance for each task is shown in bold.
Factual Consistency Evaluation Tasks
Training Tasks
SummaC
TRUE
Other-Pearson
Other-Spearrann Other-Kendal
Average
+NLI
78.1
77.5
32.6
33.6
26.3
49.6
+FV, Para
74.9
80.3
27.6
27.2
21.1
46.2
+Coref, Sum, IR, STS
84.2
83.77
39.4
36.8
28.8
54.6
+QA (Alignment-base)
85.9
85.3
45.3
42.0
33.1
58.3
D.5 Evaluation Data Contamination Analysis
As we train and evaluate our models by adapting existing datasets, a subset of our evaluation data could be contaminated with training data. To understand the impact of data contamination, we perform a post hoc overlap analysis. Following [2]. for each evaluation example, we check any of its grams exist in the training set, where is the 5th percentile example length for the evaluation dataset words (in practice we clamp n to between and 13 to avoid spurious collisions and limit run time). We consider an evaluation example "dirty" if there is at least one overlapping n-gram or "clean" otherwise. Due to resource constraints (e.g., available RAM), we sample up 1000 examples from each NLU (Section 4.1) and factual consistency evaluation (Section 4.2) dataset for analysis.
The results are shown in Table 23. Overall, the SummaC benchmark and the other datasets we use in the factual consistency evaluation experiment have the least number of dirty examples, with other datasets having varying level of overlap with training data. One notable case is ANLI, where almost all examples are marked as dirty. We manually examine a small selection of ANLI examples, and find that most of the overlapping n-grams come from the premise portion of the examples (they overlap with examples from DocNLI and HotpotQA in the training set). As noted by [2], this n-gram overlap metric tends to show a large number of false positives for datasets constructed from common data sources. For instance, one possible explanation for a high dirty count is that training and evaluation examples share texts sourced from the Internet (e.g., Wikipedia) as supporting information (e-g.. HotpotQA contexts and ANLI premises), which does not necessarily Icak evaluation set answers. Thus, a high number of dirty examples alone does not meaningfully indicate contamination.
A more reliable indicator of data contamination is the difference in metric scores when evaluating using the clean subset compared to the full datasets. If removing the dirty samples (i.e., using the clean subset) leads to significantly worse metric scores, then the model might have overfit to the overlapping training data. As show in Table 23, NLI-FEVER and RACE-h have the biggest drop in metric scores. However, overall, removing dirty examples does not meaningfully reduce metric scores (the average difference is -0. Thus, the data contamination does not have a significant impact on our evaluation results and conclusions.
26

---

Data contamination analysis results for evaluation datasets. For each evaluation dataset, we report the percentage of examples containing n-grams that also exist in training set ("dirty % %"); the performance of ALIGN-large subset, and the dirty subset; and lastly, the difference between the clean subset and the entire dataset ("Clean vs. Full"). We exclude subsets with less than 100 examples to reduce variance.
Table 23:
the entire dataset ("full eval set"; see Section 4 for details), the clean
Accuracy Pearson Correlation AUC
Experiment
Dataset
Dirty
Full Eval Set
Clean Subset
Dirty Subset
Clean vs. Full
MNLI-mm
2.8
90.3
90.0
-0.3
MNLI-m
5.4
90.3
88.6
-1.8
ANLI-1
99.1
75.8
75.9
ANLI-2
99.4
52.4
52.3
ANLI-3
99.9
52.3
51.7
SNLI
0.5
91.8
91.4
-0.4
NLI-FEVER
68.9
77.8
73.6
79.1
4.1
VitaminC:
18.5
91.8
90.4
94.6
-1,4
SICK
52.4
91.5
93.2
90.0
1.7
Language
STSB
12.2
89.8
88.8
91.3
-1.0
Understanding
PAWS
16.5
92.6
91.7
92.7
-0,9
In-Domain
PAWS-QQP
78.4
93.8
97.3
92.8
3.5
QQP
34.1
91.3
93.2
91.8
1.9
RACE-m
34.0
86.8
88.0
82.6
1.3
RACE-h
28.8
81.6
78.7
83.7
-3.0
Multi-RC
44.0
87.8
87.0
85.7
00,9
BoolQ
49.5
87.7
85.9
90.9
-1.8
Quail
0.4
78.6
78.5
0011
SciQ
20.1
93.7
92.4
99.0
-1.3
GAP
1.6
88.6
87.5
-1.1
AXB
1.2
79.6
79.8
0.1
AXG
0.0
72.5
72.5
0.0
CB
12.5
89.3
87.8
-1.5
Language
RTE
6.5
89.9
89.2
-0.7
Understanding
WNLI
0.0
54.9
54.9
0.0
Zero-Shot
SE14T1
50.6
86.9
90.5
84.8
3.5
MRPC
32.0
67.9
68.7
66.9
0.8
DREAM
0.0
81.1
81.4
0.3
Quartz
24.5
79.2
77.7
83.9
-1.5
CogenSumm
2.0
88.4
88.6
0.2
Factual
XSumFaith
0.7
74.6
75.2
0.5
Consistency
PolyTope
1.3
92.5
91.9
-0.6
Evaluation
FactCC
9.3
94.9
95.3
0.5
SummaC:
SummEval
4.0
92.3
92.6
0.3
FRANK
2.0
91.3
90.2
-1.0
FRANK
2.8
91.4
91.4
0.0
SummEval
4.6
83.8
84.4
0.6
MNBM
0.5
74.4
76.5
2.1
QAGS-C
0.4
89.0
89.0
0.0
Factual
QAGS-X
0.4
83.2
83.1
-0.1
Consistency
BEGIN
27.9
81.1
81.0
82.3
-0.1
Evaluation
Q2
41.6
79.2
81.3
77.8
2.1
TRUE
DialFact
28.9
85.1
85.2
89.0
0.1
*AWS*
19.4
98.4
98.8
97.1
0.4
FEVER*
54.4
94.9
96.2
94.9
1.3
VitaminC*
17.5
98.3
98.3
99.6
0.11
XSumFaith
0.8
28.8
30.1
1.3
SummFyal
3.3
66.7
64.9
-1.8
Factual
Q-Xsum
0.4
53.9
53.6
-0.2
Consistency
Q
0-CNNDM
0.4
76.1
76.1
0.0
Evaluation
F-Xsum
2.6
38.9
45.7
6.8
Other
F-CNNDM
2.6
68.9
67.9
-1.0
SamSum
0.0
47.7
47.7
0.0
Average
-0.1
27


Reviews:
[{"review_id": "widznirz6f", "reviewer": "Reviewer_ayzf", "summary": "This paper proposes a way to cast a variety of classification tasks into a single text alignment task. The authors found out that using the text alignment task could generate better results on certain downstream tasks, compared to Flan-T5 and GPT-3.5.", "questions": "Also, more baseline should be compared to. If the model just train the the same subset of the task without casting them into alignment model, will the model still perform well?", "limitations": "The proposed model cannot handle generative tasks.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper presents a novel approach by framing everything as an alignment task. The results are good, and I think the verifier results are interesting.", "weaknesses": "I understand that the text-alignment model is not well-suited for generative tasks. Therefore, I believe that the conclusion of the paper is a bit unfair to Flan T5 and Gpt-3, which can be used for generative tasks. It is well-known that sometimes specialized small models can be performed better than by much larger general models. Somehow, the proposed method sounds like another trade-off. I think the proposed method is reasonable, but I believe that further research into different trade-offs would be beneficial.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**More baseline - multitask learning**\n\nWe add comparison with multitask learning as suggested, where the model just trains with exactly the same set of tasks without casting into alignment format. Please see the General Response for results. The results show our alignment model has better performance than the conventional multi-task learning, along with other advantages.  \n\n**Trade-offs: small vs large & general, discriminative vs generative**\n\nOur decision to use a small discriminative model is intentional, and we have explicitly discussed this generality-vs-efficiency tradeoff in the introduction section. One of our contributions is to propose a concrete approach in the spectrum of the generality-vs-efficiency tradeoff. We have shown through experiments that this trade-off is desirable as it achieves better performance on a significant set of tasks, with smaller parameter counts than other more general models, leading to a model suitable for many interesting applications (its usefulness is also acknowledged by reviewers Bk1x and 6Rac). The ability to handle generative tasks (while interesting in its own right) is outside of the scope of our definition of alignment."}, {"review_id": "mi22caPtS3", "reviewer": "Reviewer_6RaC", "summary": "This paper leverages the fact that a lot of popular comparison based NLP tasks like entailment, paraphrase detection, semantic similarity judgement, multiple choice passage based QA etc. amount to learning a specific similarity function between two sets of sequences that is a proxy for “information alignment” between the two sets. Hence, this paper gathers together public datasets for many such tasks and with some light rule-based data augmentation results in 5.9 training examples from 28 datasets. Then a moderately sized RoBERTa model is finetuned on this big comparison-based dataset which is then compared to task specific models and larger models like FLAN T5 on many such datasets. This model has also been used as a metric for measuring factual consistency of NLG models like summarization models. The authors also use this model to detect questions that are unanswerable from the accompanying context to boost performance of systems on some QA datasets.\n", "questions": "See above", "limitations": "See above", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "– The paper is well-motivated and the large aggregated dataset and the model trained on it will be useful to the community for further study.\n\n– This model outperforms larger general models like FLAN T5 and is competitive with task specific finetuned RoBERTa models on various semantic comparison tasks (some of them unseen during training) which shows the effectiveness of similarity between various such tasks.\n\n– The results on summarization evaluation are promising and the usage of this model for identifying unanswerable questions is interesting.\n\n– The ablation study shows an interesting trend indicating that various comparison-based datasets and tasks are very similar and compatible with each other. This raises interesting questions related to the nature of these tasks and datasets.\n", "weaknesses": "– No comparison is made against specialized task-specific models. While fine-tuning RoBERTa on task-specific datasets is informative, a deeper insight into how the proposed model fares in comparison to more focused task-specific models will strengthen the comparison.\n\n– While this paper focuses on comparison and understanding tasks, it is compared to larger generative models that are specially designed for natural language generation. While these models show impressive performance on these understanding/comparison tasks, a more informative comparison would be against larger encoder-based models that are specifically trained to do well on these datasets and benchmarks like SuperGLUE. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comparison with task-specific models**\n\nThe goal of our work is to design a model that can perform well on a range of tasks without further task-specific fine-tuning. As a result, we compare with LLM that can be used in a similar way (do not require fine-tuning) and include the task-specific fine-tuned RoBERTa baseline as a sanity check. To the best of our knowledge, the results reported on leaderboards such as SuperGLUE are for models specifically fine-tuned on each of the datasets in the benchmark, making them not suitable for a direct comparison with our method. \n\n**Comparison with large encoder-based models**\n\nThere are few large encoder-only language models, and we compare with the most well known large encoder-based model Megatron BERT using results reported by Shoeybi et al. (2020). Our alignment model has similar performance compared to Megatron BERT of a similar size, but the alignment model does **not** require further fine-tuning, while Megatron BERT does. We also note as the parameter count of Megatron BERT increases, its performance improvement is limited. This diminishing return is also one of the reasons that we opt to use a smaller model when it comes to the generality-vs-efficiency tradeoff.\n\n|         | Megatron | Megatron | Megatron | Alignment base | Alignment large |\n|---------|----------|----------|----------|----------------|-----------------|\n| size    | 336M     | 1.3B     | 3.9B     | 125M           | 355M            |\n| MNLI-m  | 89.7     | 90.9     | 91.4     | 87.82          | 90.34           |\n| MNLI-mm | 90.0     | 91.0     | 91.4     | 87.54          | 90.31           |\n| QQP     | 92.3     | 92.6     | 92.7     | 90.07          | 91.27           |"}, {"review_id": "ulY0pdM5kG", "reviewer": "Reviewer_zhhb", "summary": "This work proposed a text alignment model for a wide range of tasks that aims to measure the degree of alignment between their information. To be more specific, 5.9M examples from 28 datasets are used to fine-tune RoBERTa model. Experimental results show that the text alignment-enhanced model delivers comparable or superior performance compared to larger LMs, validating the effectiveness of the proposed method.", "questions": "As for Flan models, there is a potential concern regarding contamination of the training set and test set, especially when it comes to the in-domain test, as shown in Table 1 and Figure 2. I understand you have seen and unseen datasets. However, the training and test sets may not have the exact same data points, but they share similar patterns or characteristics. In this case, things become very trivial: the improvement is just because your 5.9M contains similar data instances as in the test set. Given this context, I have two questions:\n \n1. Did you experiment with an out-of-domain test setting for the alignment-based RoBERTa model? I'd like to know if you cluster the tasks to determine the task boundary, as many tasks have a high overlap.\n2. Line 634 in the appendix: you mentioned for each dataset, we only use the first 500k samples. How did you determine the number \"500k,\" and how will performance change if you vary this number?", "limitations": "The authors state the limitation regarding biases in the constructed dataset, and it is an inevitable issue in the training data.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. 5.9M examples from 28 datasets are extracted for LMs fine-tuning, and the experimental result on in-domain datasets (Table 1) and zero-shot setup (Table 2) demonstrate the effectiveness of Alignment-RoBERTa.\n2. Experimental code has been submitted, benefitting future pre-fine-tuning research.", "weaknesses": "1. The previous pre-fine-tuning work proposed combining multiple losses with different weights [1], which diminishes the technical contribution of this work in terms of pre-fine-tuning.\n2. Additional synthetic data is needed, increasing the complexity of the proposed pre-fine-tuning method. Also, the ablation study and corresponding are insufficient in terms of how much contribution synthetic data makes to the overall performance.\n3. Experimental setup is not convincing because of the contamination of the training set (seen task) and test set (unseen task). Task clustering is needed to remove the concern.\n\n\n[1] https://aclanthology.org/2021.emnlp-main.468.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Previous Pre-finetuning Works**\n\nWe’d like to clarify that our text alignment method is **not** pre-fine-tuning. Instead, we develop a unified alignment model that is directly applied to a diverse set of tasks **without any additional finetuning**. As we pointed out in the related works section, our work differs from MUPPET (Aghajanyan et al.) in that we 1) use a unified definition of alignment instead of multitask learning , 2) share all model components across tasks and do not use dataset-specific heads, and 3) our model can be directly applied to different tasks without additional finetuning.\n\n**Contribution of synthetic data** \n\nThe method we use to create synthetic data is easy and commonly used by many pretrained models (Kryściński et al., 2019; Deng et al., 2021). We also note that some popular datasets we use already contain synthetic samples (e.g., PAWS, DocNLI). \n\nWe acknowledge the need for further ablation study on the synthetic data and we thank the reviewer for the suggestion. The experiment results for removing the synthetic data are shown below.\n\nTable 1: Performance comparison on NLU tasks\n| Training \\ Evaluation Tasks | NLI      | Fact Verification | STS      | Paraphrase | QA       | Coreference | Average  |\n| --------------------------- | -------- | ----------------- | -------- | ---------- | -------- | ----------- | -------- |\n| All tasks                   | **70.9** | **83.3**          | 89.9     | 91.4       | 78.2     | 81.4        | 82.5     |\n| -Synthetic                  | 70.4     | 83.1              | **90.1** | **92.0**   | **78.6** | **83.1**    | **82.9** |\n\n \nThe result shows that the synthetic data is not critical as it does not make a big difference to the model performance. The synthetic samples are included to provide a more comprehensive representation of the data that the alignment model may encounter. Therefore, the synthetic datasets are not a fundamental part in our alignment framework and they will not significantly increase the complexity of it (the synthetic samples are minor in the whole training set), which in turn shows the efficiency and simplicity of our proposed method.\n\n**Test set contamination**\n\nEven for tasks of the same type (e.g., QA), over evaluation datasets can have very different patterns from the training datasets. For instance, the DREAM question answering dataset we use for zero-shot evaluation includes dialogues as the QA context, while training QA datasets contain general QA samples (e.g., SQuAD v2). In our factual consistency evaluation use case, the evaluation benchmarks also contain task types not present in training data. For example, TRUE contains dialogue datasets that have very different characteristics compared to NLI, QA, etc. data used for training. In both cases, our experiment results indicate that our model generalizes better than the baselines.\n\n**First 500k samples**\n\nWe limit the number of samples per training dataset to avoid extreme data imbalance between datasets as some of them are extremely large (please refer to Table 8 in the appendix for statistics). Our practice follows previous work (e.g. FLAN-T5 (Chung et al., 2022)) that also uses similar methods to mitigate the dominance of large datasets. Additionally this limit helps us save computational resources. It’s certainly possible that including more training data could lead to better performance, and we are happy to explore more in the future\n\nReference  \n[1] Kryściński, Wojciech, et al. \"Evaluating the factual consistency of abstractive text summarization.\" arXiv preprint arXiv:1910.12840 (2019).  \n[2] Deng, Mingkai, et al. \"Compression, transduction, and creation: A unified framework for evaluating natural language generation.\" arXiv preprint arXiv:2109.06379 (2021).  \n[3] Chung, Hyung Won, et al. \"Scaling instruction-finetuned language models.\" arXiv preprint arXiv:2210.11416 (2022). https://arxiv.org/pdf/2210.11416.pdf, page 46  "}, {"review_id": "NZKYTDWimO", "reviewer": "Reviewer_Bk1X", "summary": "While next word prediction produces task-general models that can do a wide variety of tasks when prompted, it is not an efficient formulation in that it requires very large models. On the other hand, fine-tuned models achieve higher performance at smaller sizes but are specific to a few tasks. This paper proposes text alignment as a middle ground that encompasses a wide range of tasks while allowing for smaller models than next word prediction. Concretely, they convert 28 datasets (encompassing tasks like entailment, IR, QA, coref, and consistency) into the text alignment format and fine-tune RoBERTa on them. The resulting model outperforms much larger models (that are instruction-finetuned), as well as RoBERTa with task-specific fine-tuning.", "questions": "(1) For handling longer contexts, I wonder if it would make sense to take min_j max_i f(x_1^i, x_2^j) instead of mean_j max_i f(x_1^i, x_2^j) as in the paper, with the interpretation that all of the facts in x_2 should be supported. The mean could fail if x_2 contains mostly supported statements, except for an egregiously unsupported statement in a single chunk.", "limitations": "Addressed", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "(1) The idea of using text alignment as a task-general interface is interesting and seems useful for producing useful task-general models at smaller sizes.\n\n(2) The experiments are extremely thorough, and the method performs well across the board.\n\n(3) The paper is well-written and clear.", "weaknesses": "While this section contains some suggested experiments, I support the acceptance of this paper regardless of whether or not they are run during the rebuttal period.\n\n(1) While the paper frames alignment as being more general than multi-task finetuning, they only evaluate the model on tasks seen during alignment finetuning. Indeed, as shown in the task ablations in Table 5, it seems that the model can only do tasks that are included during training. Therefore, I wonder how different the model is from simply doing multi-task fine-tuning, and what clear advantages are provided by using a unified interface. While already strong, I think the results would be even stronger if there were examples of the model doing unseen tasks.\n\n(2) Related to questions of how general alignment is, the tasks included in the training and evaluation feel very close to entailment, which might be a key factor in enabling them all to use the same interface without task-specific heads. Therefore, I wonder how performance would be affected if tasks very far from entailment were included. (For example, you could include POS tagging, where x_1 is the original sentence and x_2 is the sentence with some or all of the words replaced by their part of speech.)\n\n(3) Related to the above, while the paper claims that changing the interface from next word prediction to alignment allows for smaller models, I wonder if the smaller model sizes are simply a result of considering a narrower set of tasks than the instruction-finetuned models. I suspect that FLAN-T5 needs larger model sizes simply because it needs more capacity to do more tasks. One relevant ablation testing this question would be to take the same RoBERTa model and instruction-finetune it on the same 28 datasets as the alignment model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. Your recognition of our ALIGN framework, experiments and paper writing is truly encouraging. We also appreciate your list of suggested experiments.\n\n**Performance on unseen tasks**\n\nDespite we group some training and evaluation datasets into the same type, they can have significantly different formats. For example, the DREAM dataset (used for evaluation) contains QA samples with dialogues contexts, while in our training datasets the QA contexts are usually articles (no dialogues). The TRUE benchmark (factual consistency evaluation, Figure 4) also contains dialogue datasets and we do not have similar (dialogue) datasets in our training data.\n\nAs correctly pointed out by Bk1X, our ablation study indicates the inclusion of tasks (e.g., NLI, QA, etc.) during alignment training significantly boosts model performance on datasets of that task. However, as shown in the factual consistency evaluation use case (see Table 3 and Figure 4), our method outperforms existing approaches when good quality training data for the task (e.g., automated machine summarization evaluation) is limited. \n\nWe also included an additional comparison between our method and multitask learning in the general response. The experiment results suggest that our unified interface allows the model to better generalize to new datasets in the zero-shot setting. \n\n**Alignment in relation to entailment**\n\nIndeed, we consider entailment to be a special case of the more general definition of alignment. We note we don’t claim that alignment can be used to solve all NLP problems. We intentionally choose a set of related tasks such that they can be effectively modeled and learned using a single interface. Our experiments show this is a good tradeoff. That said, we are excited to explore if alignment can be generalized to more distant tasks such as POS tagging.\n\n**smaller model, narrower tasks**\n\nWe would like to thank the reviewer for bringing up the ablation experiment. However, in this experiment, we decide to instruction fine-tune a a T5 model due to the following reasons: 1) Roberta is an encoder-based transformer model, and it does not undergo training involving next word prediction,  2) existing instruct finetuning methods are designed for decoder-based models such as T5 and PALM, 3) this choice will enable us to effectively compare the distinctions between the two training objectives: next word prediction and alignment. We use the T5-base (250M parameters) version to show whether a smaller instruction fine-tuned model could still perform well on the alignment tasks, compared with the alignment-based model.\n\nWe instruction-finetune the T5-base model on the same datasets as our alignment model. We don’t convert QA tasks since T5 naturally supports sequence generation. We follow the prompts mentioned in [1, 2] and format the datasets to feed the T5 model. The following tables are the result for the instruction finetuned T5-base.\n\nTable 1. Comparison with the instruction finetuned next word prediction model on in-domain tasks\n|Model\\EvaluationTasks|NLI|FactVerification|STS|Paraphrase|QA|Coreference|Average|\n|-|--|-|--|-|--|-|--|\n|Alignment-base(125M)|**70.9**|**83.3**|**89.9**|**91.4**|**78.2**|**81.4**|**82.5**|\n|Instruction-finetuned-T5(222M)|53.7|68.7|62.0|83.2|30.3|42.9|56.8|\n\nTable 2. Comparison with the instruction finetuned next word prediction model on zero-shot tasks\n|Dataset\\Model|Alignment-base|Inst-FT-T5-base|\n|-|-|-|\n|AXB|**75.1**|65.3|\n|AXG|**59.8**|50.8|\n|CB|**76.8**|58.9|\n|RTE|**83.4**|59.9|\n|WNLI|**52.1**|47.9|\n|SE14T1|**90.7**|48.5|\n|MRPC|66.0|**66.1**|\n|DREAM|**71.3**|35.8|\n|Quartz|**59.7**|50.6|\n|AVG|**70.5**|53.8|\n\nThe result shows the alignment-based model achieves better performance than the instruction finetuned model on the in-domain and zero-shot tasks, showing the effectiveness of the alignment model when training on a set of alignment tasks. Therefore, the good performance does not come from narrowing the tasks as the small instruction finetuned T5-base performs worse than the alignment model.\n\nReference  \n[1] Chung, Hyung Won, et al. \"Scaling instruction-finetuned language models.\" arXiv preprint arXiv:2210.11416 (2022).  \n[2] Longpre, Shayne, et al. \"The flan collection: Designing data and methods for effective instruction tuning.\" arXiv preprint arXiv:2301.13688 (2023).\n\n**Aggregation method (min-max vs. mean-max)**\n\nWe agree that, if the model used for alignment estimation is perfect, using a min-max formulation as suggested by the reviewer would more accurately reflect the definition of alignment. However, in practice, the alignment estimator can be noisy, and we speculate taking the average helps remove some of that noise. In comparison, the min-max setup would be easily affected by underestimation. We also note mean-max aggregation is widely used in previous automatic evaluation work, such as SummaC [1] and SMART [2].\n\nWe have therefore experimented with the min-max aggregation method on the factual consistency tasks. The results are shown as follows:\n\n||Alignment-base|-base-minmax|Alignment-large|-large-minmax|\n|:-|:-:|:-:|:-:|:-:|\n|SummaC|85.9|86.1|89.0|89.1|\n|TRUE|**85.3**|84.2|**87.2**|86.2|\n|Other-Pearson|**45.3**|43.7|**54.4**|52.1|\n|Other-Spearman|42.0|**42.3**|**49.7**|49.2|\n|Other-Kendall|33.1|33.1|**39.9**|38.8|\n\nThe bold number indicates a better performance among the same sized alignment models. It shows our mean-max aggregation outperforms min-max in most cases.\n\nReference  \n[1] Laban, Philippe, et al. \"SummaC: Re-visiting NLI-based models for inconsistency detection in summarization.\" Transactions of the Association for Computational Linguistics 10 (2022): 163-177.  \n[2] Amplayo, Reinald Kim, et al. \"SMART: sentences as basic units for text evaluation.\" arXiv preprint arXiv:2208.01030 (2022).\n"}]

Author rebuttal:
We thank all reviewers for your thoughtful and positive comments. We're encouraged by your appreciation that our text alignment framework is well-motivated (6RaC), novel (ayzf), and interesting (Bk1X); the resulting model has strong performance (ayzf, 6RaC, zhhb), enables interesting use cases (6RaC), and is useful to the community (6RaC, zhhb, Bk1X); the experiments are extremely thorough (Bk1X), and reveal insights about the nature of comparison-based tasks (6RaC); and lastly, the paper is well-written and easy to follow (Bk1X).

&nbsp;

**Here we address the concern of comparison with multitask learning**:

### Comparison with multitask learning
We add comparison with multitask learning and show the advantage of our unified alignment framework (we thank the reviewers for the suggestion). Specifically, to obtain the multitask-learning model, we follow the popular multi-task learning work (Muppet, Aghajanyan et al., 2021), and train the same base model with the same set of tasks/datasets as for our alignment model. Notably, different from our alignment model that uses a unified interface to accommodate all diverse tasks, the conventional multitask-learning model learns separate prediction heads for different tasks.

Results show our alignment model outperforms the multitask learning model on zero-shot datasets, while having similar performance on in-domain datasets. This suggests that our definition of alignment indeed helps the model better generalize to out-of-domain problems.

**Experiment results**

**(1)** The table below shows the **results for in-domain datasets**. The alignment model and the multitask learning model have roughly the same performance.
| Dataset | Multitask base | Alignment base |
|---|---:|---:|
| MNLI-mm | 87.61 | 87.54 |
| MNLI-m | 87.68 | 87.82 |
| ANLI-1 | 65.10 | 65.30 |
| ANLI-2 | 48.20 | 48.70 |
| ANLI-3 | 46.17 | 45.50 |
| SNLI | 91.33 | 90.78 |
| NLI-FEVER | 76.50 | 76.78 |
| VitaminC | 89.82 | 89.79 |
| SICK | 89.01 | 90.71 |
| STSB | 87.86 | 89.03 |
| PAWS | 93.94 | 92.33 |
| PAWS-QQP | 92.32 | 91.88 |
| QQP | 90.66 | 90.07 |
| Multi-RC | 83.83 | 82.20 |
| BoolQ | 81.32 | 81.07 |
| SciQ | 92.10 | 92.40 |
| GAP | 81.65 | 81.35 |
| **Average** | **81.48** | **81.37** |

**(2)** The table below shows the **results in the zero-shot setting**. For the multitask-learning model and each zero-shot evaluation task, we use a set of “reasonable” heads obtained during training for prediction, and we report the best, worst, and average performance among these heads. For NLI, QA, and paraphrase evaluation datasets, the “reasonable” heads are all heads trained with NLI, QA, and paraphrase detection datasets, respectively.

As shown in the table below, the alignment model outperforms the average performance of the “reasonable” multitask learning prediction heads, and the multitask learning model only slightly outperforms the alignment model if we **cherry-pick** the best performing head for each zero-shot dataset. In a strict zero-shot setting, as we can’t use evaluation data for head selection, and the best performing head varies across evaluation datasets, the reported “best head” performance is an unrealistic upper bound. Thus the results indicate the alignment model has an advantage over multitask learning when applied to new datasets.

| Dataset | Multitask base (avg. of heads) | Multitask base (best head) | Multitask base (worst head) | Alignment base |
|---|:---:|:---:|:---:|:---:|
| AXB | 74.94 | 76.18 | 72.37 | 75.09 |
| AXG | 63.58 | 65.73 | 60.39 | 59.83 |
| CB | 80.00 | 83.93 | 76.79 | 76.79 |
| RTE | 81.41 | 81.95 | 81.23 | 83.39 |
| WNLI | 56.34 | 60.56 | 47.89 | 52.11 |
| SE14T1 | 57.86 | 58.45 | 57.17 | 90.72 |
| MRPC | 69.60 | 71.19 | 68.23 | 65.97 |
| DREAM | 67.04 | 73.98 | 57.03 | 71.34 |
| Quartz | 59.09 | 63.52 | 54.46 | 59.69 |
| **Average** | **67.76** | **70.61** | **63.95** | **70.55** |



**In addition** to the above stronger performance in the zero-shot setting, there are two notable advantages of our proposed method over multitask learning:
1. Our approach casts all the diverse tasks into a unified alignment format, which allows us to leverage these datasets naturally. In comparison, it’s not straightforward for a discriminative multitask-learning model to train on some of the datasets in our alignment data. For example, a multitask-learning model cannot easily train on open-ended QA datasets where the ground truth answers do not exist in the context. For fair comparison, we reuse our converted alignment version of those datasets in multitask learning.
2. As the multitask-learning model uses task-specific heads, it cannot be straightforwardly applied to new out-of-domain datasets. In the out-of-domain (zero-shot) setting, for each of the new dataset we have to find the most “similar” training task and use the associated head for inference, which can be difficult when data is limited. In comparison, our alignment model has unified  prediction heads across tasks and datasets, making it easy to use the model for new datasets.

We’ll include the above results and articulate all experimental details in the revised version of the paper.

Meta-review: Meta Review:

This paper proposes a text alignment model for a wide range of tasks that aims to measure the degree of alignment between their information. The authors leverage 5.9M examples from 28 datasets to fine-tune a RoBERTa model, which delivers comparable or superior performance compared to larger language models.

Strengths:
- The novel approach of framing everything as an alignment task shows promising results.
- The model outperforms larger general models like FLAN T5 and is competitive with task-specific fine-tuned RoBERTa models on various semantic comparison tasks.

Weaknesses:
- The proposed model cannot handle generative tasks, and the conclusion may be a bit unfair to Flan T5 and GPT-3, which can be used for generative tasks.
- No comparison is made against specialized task-specific models, which would strengthen the comparison.
- The experimental setup may have contamination between the training set and test set, raising concerns about the generalizability of the model.

Overall, while there remain some concerns of this work, the strengths outweigh the limitations. But I hope in the revised version, the author could do further research into different trade-offs, comparisons with specialized task-specific models. Also, I hope the author can address the concerns about contamination between the training set and test set (something like a 6-gram overlap detection will make the results much more convincing).
Decision: Accept (poster)