SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuheng Zha* Yichi Yang* Ruichen Li Zhiting Hu
{yzha, yiy067, rul014, zhh019}@ucsd.edu
Large language models (LLMs), typically designed as a function of next-word prediction, have excelled across extensive NLP tasks. Despite the generality, next-word prediction is often not an efficient formulation for many of the tasks, demanding an extreme scale of model parameters (10s or 100s of billions) and sometimes yielding suboptimal performance. In practice, it is often desirable to build more efficient models—despite being less versatile, they still apply to a substantial subset of problems, delivering on par or even superior performance with much smaller model sizes. In this paper, we propose text alignment as an efficient unified model for a wide range of crucial tasks involving text entailment, similarity, question answering (and answerability), factual consistency, and so forth. Given a pair of texts, the model measures the degree of alignment between their information. We instantiate an alignment model (ALIGN) through lightweight finetuning of RoBERTa (355M parameters) using 5.9M examples from 28 datasets. Despite its compact size, extensive experiments show the model's efficiency and strong performance: (1) On over 20 datasets of aforementioned diverse tasks, the model matches or surpasses FLAN-T5 models that have around 2x or 10x more parameters; the single unified model also outperforms task-specific models finetuned on individual datasets; (2) When applied to evaluate factual consistency of language generation on 23 datasets, our model improves over various baselines, including the much larger GPT-3.5 (ChatGPT) and sometimes even GPT-4; (3) The lightweight model can also serve as an add-on component for LLMs such as GPT-3.5 in question answering tasks, improving the average exact match (EM) score by 17.94 and F1 score by 15.05 through identifying unanswerable questions.
Recent large language models (LLMs) have demonstrated exceptional generalizability in a wide range of natural language processing (NLP) tasks. As the underlying formulation of these LLMs, next-word prediction is proven to be a general function applicable to diverse language problems. However, it is often not being an efficient solution for many tasks. LLMs often need to scale up to over tens of billions of parameters to achieve meaningful performance [1], with popular models like GPT-3 boasting as many as 175B parameters [2]. Additionally, even with their extreme scale, LLMs sometimes still find themselves outperformed by smaller models. For example, ChatGPT/GPT-3.5 falls behind existing finetuned baselines on most classical natural language understanding tasks [3].
As a result, in many cases it is desirable to navigate the spectrum of generality-vs-efficiency tradeoff, for example, by developing smaller but general-purpose models that excel in a substantial subset of tasks. Despite being less versatile than the extreme-scale LLMs, these models are more efficient and
$^{2}$ Code is made available at https://github.com/yuh-zha/Align
Figure 1: Our alignment model (125M and 355M) achieves substantially better efficiency and performance compared to much larger models on a wide range of tasks, including (left) diverse text pair understanding tasks on over 20 datasets and (middle) factual consistency evaluation, and (right) improves existing LLMs on question answering by detecting unanswerable questions. See Section 4 for more details.
provide superior performance, making them more usable on the set of tasks that they are designed to handle. Previous work has attempted to build natural language inference (NLI) models as an efficient solution for broad tasks [4-6]. But with limited NLI data (e.g., MNLI [7]) for training, the models exhibit limited performance and applicability across diverse domains. Another related line of research trains general text representation models with pretraining and multi-task learning [8-10]. Those models need to be specifically finetuned (with task-specific head) for each downstream task, instead of functioning as ready-to-use solutions.
In this paper, we investigate the underlying commonalities among a broad range of NLP tasks that concern the relationship between two texts, and propose a text alignment model (ALIGN) as an efficient unified solution, following Zha et al. [11]. Given an arbitrary pair of texts, ALIGN measures the degree of alignment between the content in the texts. We show the formulation subsumes a substantial set of popular tasks, ranging from NLI, fact verification, semantic textual similarity, question answering, coreference resolution, paraphrase detection, to factual consistency evaluation of diverse language generation tasks, question answerability verification, and so forth (Figure 1). The generality, in turn, presents an opportunity for us to use diverse data to learn the alignment model. Specifically, we adapt and aggregate 28 datasets from the above tasks, resulting in 5.9M training samples with diverse characteristics. We then use the data to finetune a small-scale LM (e.g., RoBERTa [12]), yielding a model that directly applies to and excels in diverse problems and domains.
We evaluate ALIGN with extensive experiments. First, we test on 25 seen and unseen datasets of the aforementioned tasks, and show our alignment model based on RoBERTa (355M parameters) achieves on par or even better performance than the FLAN-T5 models (780M and 3B) that are 2x or  $8.5\mathrm{x}$  as large and trained with substantially more data. In addition, the single alignment model outperforms RoBERTa specifically finetuned on each of the datasets. Second, we use ALIGN to evaluate the factual consistency of natural language generation (NLG) systems (e.g., for summarization, dialog, paraphrasing, etc.). On 23 datasets, the small alignment model achieves substantially higher correlation with human judgements than recent metrics, including those based on GPT-3.5 and even GPT-4. Third, we use ALIGN as a question answerability verifier and incorporate it as an add-on component to existing LLMs (e.g., GPT-3.5 and FLAN-T5). It significantly enhances the LLMs' performance in three question answering datasets, improving the average exact match score by 17.94 and F1 score by 15.05.
Recent work has shown that LLMs are few-shot learners capable of generalizing to diverse tasks [13, 2, 14]. These LLMs are designed based on the principle of next-word-prediction, where the joint probability distribution of text sequences are factored into a product of conditional probabilities [15]. The performance of LLMs is highly correlated with their scales. Wei et al. [1] show that a scale of more than  $10^{22}$  training FLOPs (around 10B model parameters) is required for several different LLM designs to achieve above-random performance on many language tasks.
Another line of research has tried to design models that can either learn from multiple tasks or handle many downstream tasks. Liu et al. [8] propose to use a BERT model [16] with task-specific heads to learn four types of tasks, including single-sentence classification, pairwise text classification, text similarity scoring, and relevance ranking. Aghajanyan et al. [9] pre-finetune language models on 50 dataset to encourage learning more general representations, and show that the process improves model performance and data-efficiency in the finetuning stage. Yin et al. [4], Wang et al. [5] and Mishra et al. [6] explore the application of a model trained with NLI datasets to multiple downstream tasks, in an effort to find more general yet still efficient methods. Some research explores the use of smaller models for enhancing LLMs. For example, Cappy [17], a small scorer model trained on the same datasets as in T0 [18], functions well on natural language classification tasks while boosting the performance of LLMs as an add-on. While we also use diverse datasets to train our model, we 1) unify language tasks into a single text pair alignment problem, 2) share all model components across multiple tasks and do not use dataset-specific heads, and 3) our model can be directly applied to a wide range of tasks without additional finetuning. The work demonstrates the strong potential of unified modeling and learning with all diverse relevant forms of experience [19].
Text alignment has long been used for measuring the correspondence of information between two pieces of text. For example, in machine translation, Brown et al. [20] propose a model that learns alignment between two languages. Papineni et al. [21] use n-gram overlap to compare translated sentences with their references. Gao et al. [22] train a sentence-level embedding model that compares similarity between two sentences while BERTScore [23] utilizes token-level alignment for evaluating text generation. Zha et al. [11] also propose building an automatic factual consistency metric for NLG systems through a text alignment framework. Expanding on the idea of text alignment, we explore how the formulation enables training a single alignment model that excels at a wide variety of tasks, including natural language understanding, factual consistency evaluation and answerability verification.
3 Text Alignment Model
In this section, we introduce the text pair alignment formulation. We first formally define the concept of text pair alignment, and then discuss how the alignment function can be used to solve a set of popular language tasks. Additionally, we cover the split-then-aggregate method used to handle long inputs. In Section 3.1, we discuss the training process of the alignment model (ALIGN).
Given a text pair  $(x_{1}, x_{2})$ , we define text  $x_{2}$  to be aligned with text  $x_{1}$  if all information in  $x_{2}$  is supported by information in  $x_{1}$ , following Zha et al. [11]. For example, let "I have been in Kentucky, Kirby." be text  $x_{1}$ . Then, "I have been in the US." is aligned with  $x_{1}$ . In contrast, both "I have been in Europe." and "Kentucky has the best fried chicken." are not aligned with  $x_{1}$ , as the former is contradicted by  $x_{1}$ , and the latter cannot be inferred from  $x_{1}$ . Formally, we model alignment as a function that maps the text pair  $(x_{1}, x_{2})$  to a label  $y$  describing the level of alignment:
$$
f: (\boldsymbol {x} _ {1}, \boldsymbol {x} _ {2}) \to y.
$$
In practice, the language tasks we wish to solve with the alignment function can be broadly categorized into two groups: one that uses discrete labels, and the other that uses continuous labels (e.g., semantic textual similarity). More specifically, tasks with discrete labels are typically formulated as either binary classification (e.g., paraphrase detection) or three way classification (e.g., fact verification). In order to make the alignment function more general, such that it accommodates all the above cases, our alignment model produces three outputs:  $\operatorname{Pr}(y_{\mathrm{bin}})$ ,  $\operatorname{Pr}(y_{3\mathrm{way}})$ , and  $y_{\mathrm{reg}}$ . Here,  $\operatorname{Pr}(y_{\mathrm{bin}})$  and  $\operatorname{Pr}(y_{3\mathrm{way}})$  are probability distributions over the binary (ALIGNED, NOT-ALIGNED) and 3-way (ALIGNED, CONTRADICT, NEUTRAL) classification labels, respectively;  $y_{\mathrm{reg}} \in [0,1]$  is a real-valued score for regression tasks.
This formulation allows us to apply the alignment function to diverse tasks:
- For tasks that naturally fit into the text pair alignment format, such as NLI [7], fact verification [24], paraphrase detection [25], and semantic textual similarity [26], depending on the nature of the task, we simply map one of the alignment labels to the desired label. For example, for most NLI tasks, we interpret the corresponding  $y_{3\text{way}}$  labels as "entailment", "contradiction", and "neutral".
- In information retrieval tasks [27], the goal is to find documents that can answer a given query from a large set of candidate documents. Since relevant documents contain information related to respective queries, we use candidate documents as text  $x_{1}$ , and queries as text  $x_{2}$ . Then, a higher  $\operatorname{Pr}(y_{\mathrm{bin}} = \mathrm{ALIGNED})$  indicates the candidate document is more likely to be useful in answering the query.
- In multiple choice QA tasks [28], the inputs are a context, a question, and several choices (with one of them being the correct answer). In extractive QA tasks (including ones with unanswerable questions [29]), the inputs only consist of a context and a question. In either case, the expected output (the correct answer) can be inferred from the question and the context, while a wrong answer either contradicts the context or is not supported by the context. Therefore, we use the context as text  $x_{1}$  and the concatenation of the question a candidate answer as text  $x_{2}$ . Here, a higher  $\operatorname{Pr}(y_{\mathrm{bin}} = \mathrm{ALIGNED})$  indicates the candidate answer is more likely to be correct.
- In coreference resolution tasks [30], each sample includes a context containing a pronoun, and a list of candidate entities. The goal is to find the correct entity that the pronoun is referring to. As the pronoun and the associated entity is equivalent in this context, we consider the context with the pronoun replaced with the correct entity to be aligned with the original context. To solve coreference resolution problems, we simply replace the pronoun with candidate entities and compare the resulting contexts  $(x_{2})$  with the original context  $(x_{1})$ . We pick the candidate that produces the highest  $\operatorname{Pr}(y_{3\text{way}} = \text{ALIGNED})$  or  $\operatorname{Pr}(y_{\text{bin}} = \text{ALIGNED})$  as the correct answer.
- For generative tasks like machine summarization, dialog, and paraphrasing, the alignment function can be used to evaluate the factual consistency of generated outputs. We use the generation context (e.g., input document) as text  $x_{1}$ , and candidate system output (e.g., generated summary) as text  $x_{2}$ . In this case, the probability of  $\operatorname{Pr}(y_{3\text{way}} = \text{ALIGNED})$  or  $\operatorname{Pr}(y_{\text{bin}} = \text{ALIGNED})$  indicates if the candidate output faithfully reflects information in the context, without introducing hallucinations or contradictions.
One specific challenge of applying the alignment function to downstream tasks is that text  $x_{1}$  in some datasets (e.g., contexts in QA or summarization datasets) tends to be significantly longer than the input length limit of typical language models (e.g., 512 tokens for RoBERTa). As a result, naively truncating oversized inputs could throw away important information. To alleviate this problem, inspired by Laban et al. [31], Amplayo et al. [32], at inference time, instead of truncating the inputs, we split text  $x_{1}$  into a set of chunks  $\{x_{1}^{(i)}\}$  and text  $x_{2}$  into a set of sentences  $\{x_{2}^{(j)}\}$  such that the combined length of a chunk-sentence pair is slightly below that length limit. Then, we evaluate each pair and aggregate the results as
$$
\operatorname {a l i g n m e n t} \left(\boldsymbol {x} _ {1}, \boldsymbol {x} _ {2}\right) = \underset {j} {\operatorname {m e a n}} \underset {i} {\operatorname * {m a x}} f \left(\boldsymbol {x} _ {1} ^ {(i)}, \boldsymbol {x} _ {2} ^ {(j)}\right), \tag {1}
$$
where the max operation selects the output with the highest ALIGNED probability or regression score. Since in most downstream applications, text  $x_{2}$  tends to be succinct (e.g., summaries) and consists of self-contained sentences, this aggregation method can be interpreted as first finding the text  $x_{1}$  chunk that most strongly supports each text  $x_{2}$  "fact", and then taking the average across all text  $x_{2}$  "facts".
Our formulation not only allows us to solve the above tasks with a single alignment function, but also learn the alignment function from these tasks. By adapting text pair understanding tasks into a uniform alignment format as above, we can naturally model these tasks as simple classification and regression, allowing us to train a small model while achieving strong performance. Specifically, we use RoBERTa [12] as a lightweight backbone language model, and attach three individual linear layers to predict the three types of alignment outputs,  $\operatorname{Pr}(y_{\mathrm{bin}})$ ,  $\operatorname{Pr}(y_{3\mathrm{way}})$ , and  $y_{\mathrm{reg}}$ , respectively. The two classification heads are trained with cross entropy loss, while the regression head is trained with mean squared error loss. The losses are aggregated as a weighted sum,
$$
\mathcal {L} _ {\text {t o t a l}} = \lambda_ {1} \mathcal {L} _ {3 \text {w a y}} + \lambda_ {2} \mathcal {L} _ {\text {b i n}} + \lambda_ {3} \mathcal {L} _ {\text {r e g}},
$$
where we set  $\lambda_1 = 1 / \log 3$ ,  $\lambda_2 = 1 / \log 2$ , and  $\lambda_3 = 1$ , following Aghajanyan et al. [9].
Besides the aforementioned downstream datasets, we also include synthetic data to increase the diversity of the training set. Specifically, for QA datasets without wrong options (e.g., extractive QA datasets like SQuAD v2 [29]), we first remove the ground truth answer from the context, and then
Table 1: Performance of ALIGN and the much larger FLAN-T5 on in-domain datasets. Bold number indicates our performance is better than either FLAN-T5-large or xlarge.
ALIGN (Ours)	FLAN-T5
base	large	large	xlarge
Model Parameters	125M	355M	780M	3B
NLI	MNLI-mm [7]	87.5	90.3	88.7	90.4
MNLI-m [7]	87.8	90.3	88.8	90.5
ANLI-1 [39]	65.3	75.8	68.1	77.0
ANLI-2 [39]	48.7	52.4	48.7	60.6
ANLI-3 [39]	45.5	52.3	49.8	56.6
SNLI [40]	90.8	91.8	89.7	90.7
Fact Verification	NLI-FEVER [41, 39]	76.8	77.8	72.0	71.9
VitaminC [24]	89.8	91.8	72.9	73.9
STS	SICK [26]	90.7	91.5	79.3	79.1
STSB [42]	89.0	89.8	83.9	88.2
Paraphrase	PAWS [25]	92.3	92.6	94.0	94.6
PAWS-QQP [25]	91.9	93.8	88.3	90.1
QQP [43]	90.1	91.3	86.8	87.4
QA	RACE-m [28]	76.9	86.8	84.8	87.6
RACE-h [28]	68.8	81.6	78.3	84.6
Multi-RC [44]	82.2	87.8	84.7	88.2
BoolQ [45]	81.1	87.7	84.9	89.6
QuAIL [46]	67.8	78.6	79.1	86.3
SciQ [47]	92.4	93.7	94.9	95.7
Coreference	GAP [30]	81.4	88.6	73.8	81.5
Average	79.8	84.3	79.6	83.2
use a QA model [13] to generate wrong answers that can be used to create NOT-ALIGNED samples. Additionally, we create synthetic paraphrase samples by back translating the WikiText-103 corpus [33] using a neural machine translation model [34]. For the WikiHow summarization dataset, we use an extractive summarizer [35] to generate synthetic summaries in additional to ground truth summaries. Following Kryscinski et al. [36], Deng et al. [37], we create negative samples for both WikiText-103 and WikiHow samples by randomly masking  $25\%$  of the tokens in text  $\pmb{b}$  and infilling with a small masked language modeling model [38]. In total, we collect 5.9M examples from 28 datasets to train our alignment model ALIGN. We include more details of our training setup and data in Appendix C.
In this section, we experiment with applying ALIGN to multiple downstream tasks, including language pair understanding tasks (Section 4.1), factual consistency evaluation (Section 4.2), and question answering with unanswerable questions (Section 4.3). We discuss experiment details and include a data contamination analysis in the Appendix (Section D).
4.1 Natural Language Understanding Tasks
Natural Language Understanding (NLU) is a major category of tasks for language models, and our formulation allows us to directly use ALIGN to solve these tasks. Specifically, we include NLI, fact verification, paraphrase detection, multiple-choice QA, STS, and coreference resolution datasets in the experiments. We also include unseen datasets to demonstrate the generalizability of ALIGN. Experiments show the alignment model is on par with FLAN T5 that has  $8.5\mathrm{x}$  as many parameters. Additionally, without further task-specific finetuning, our model outperforms finetuned language models of a similar size.
Figure 2: The performance of the ALIGN-large and the task-specific finetuned RoBERTa-large on in-domain tasks.
Figure 3: The performance of the ALIGN-large and RoBERTa-NLI-large on zero-shot tasks.
4.1.1 Experiment Setup
Datasets We first evaluate ALIGN on test sets of the 20 datasets used during training (in-domain setting; see Table 1). Then, we use 9 unseen datasets for evaluation (zero-shot setting; see Table 2). For more details about the datasets, please refer to appendix C.2. If a dataset does not have a public test set, we use its validation set instead. For datasets that require binary, 3-way classification or regression, we use the associated output heads, respectively, as discussed in Section 3.
Baselines To demonstrate the efficiency of ALIGN, we compare it with FLAN-T5 [14] and FLAN-Alpaca<sup>3</sup> with model size ranging from 220M (FLAN-Alpaca-base) to 3B (FLAN-Alpaca-xlarge and FLAN-T5-xlarge). For both models, we use the same prompts as in Longpre et al. [48]. We include task-specific RoBERTa models that are individually finetuned on each training set and evaluated on the corresponding test set to show that our alignment model works well out-of-the-box without further finetuning. We also compare with a multi-task RoBERTa model trained on all the original datasets (before converting to the alignment-format data), and a T5-base model instruction finetuned on our alignment datasets to show the effectiveness of our alignment formulation. Lastly, we compare with RoBERTa model finetuned on MNLI, ANLI and SNLI datasets (RoBERTa-NLI) in the zero-shot setting, to show the generalizability of our proposed formulation.
4.1.2 Results
We report average Pearson Correlation coefficient for the STS tasks [26, 42], and average accuracy for the other tasks.
In the in-domain setting, as show in Table 1, ALIGN outperforms FLAN-T5 that is  $2\mathrm{x}$  as large (780M) and has comparable performance to the version  $8.5\mathrm{x}$  as large (3B). Similar performance gain is also observed when comparing with FLAN-Alpaca in Table 9. Furthermore, ALIGN is on par with the task-specific finetuned RoBERTa models (Figure 2) and the multi-task RoBERTa model (Table 10). ALIGN outperforms the instruction finetuned T5 model (Table 12), showing the effectiveness of our unified alignment formulation.
In the zero-shot setting, ALIGN achieves comparable performance with similarly sized variants of FLAN-T5 (Table 2), even on datasets that exist in FLAN-T5's training set. It also shows superiority to the multi-task RoBERTa in Table 11 as it eliminates the need to choose task heads while outperforming the average performance of the heads in the multi-task RoBERTa model. Additionally, ALIGN has stronger performance on average than the RoBERTa-NLI model (Figure 3) and the instruction finetuned T5 model (Table 13), indicating that our formulation leads to better generalizability.
<sup>3</sup>https://github.com/declare-lab/flan-alpaca
Table 2: Zero-shot setting results from ALIGN. The bold numbers indicate a better performance of ALIGN when comparing with similarly sized FLAN-T5. The gray number shows the specific dataset is appeared in the training set of FLAN-T5.
ALIGN (Ours)	FLAN-T5
base (125M)	large (355M)	base (250M)	large (780M)
NLI	AXB [49]	75.1	79.6	71.7	76.2
AXG [49]	59.8	72.5	53.4	73.6
CB [49]	76.8	89.3	82.1	87.5
RTE [50]	83.4	89.9	81.6	87.0
WNLI [50]	52.1	54.9	46.5	62.0
SE14T1 [51]	90.7	86.9	69.6	69.9
Paraphrase	MRPC [52]	66.0	67.9	74.8	80.1
QA	DREAM [53]	71.3	81.1	69.9	79.0
Quartz [54]	59.7	79.2	74.4	90.2
Average	70.5	77.9	69.3	78.4
4.2 Factual Consistency Evaluation for Language Generation
Studies have shown that natural generation systems (NLG) are prone to generating text that is not consistent with the source material [55-59]. As a result, many automatic metrics have been developed with the goal of detecting factual consistency errors. As factual consistency is closely related to our definition of text pair alignment, we can directly apply ALIGN for this purpose, using the NLG input context as  $x_{1}$ , and system outputs as  $x_{2}$ . We consider a system output with higher  $\operatorname{Pr}(y_{3\text{way}} = \text{ALIGNED})$  to be more factually consistent.
4.2.1 Experiment Setup