SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bowen Tan $^{1*}$ ; Yun Zhu $^{2}$ , Lijuan Liu $^{2}$ , Eric Xing $^{1,3,5}$ , Zhiting Hu $^{4}$ , Jindong Chen $^{2}$
$^{1}$ Carnegie Mellon University,  $^{2}$ Google Research,  $^{3}$ Petuum Inc.,  $^{4}$ UC San Diego,
Mohamed bin Zayed University of Artificial Intelligence
{btan2, epxing}@andrew.cmu.edu, zhh019@ucsd.edu,
{yunzhu, lijuanliu, jdchen}@google.com
Large language models (LLMs) such as T0, FLAN, and OPT-IML, excel in multitasking under a unified instruction-following paradigm, where they also exhibit remarkable generalization abilities to unseen tasks. Despite their impressive performance, these LLMs, with sizes ranging from several billion to hundreds of billions of parameters, demand substantial computational resources, making their training and inference expensive and inefficient. Furthermore, adapting these models to downstream applications, particularly complex tasks, is often unfeasible due to the extensive hardware requirements for finetuning, even when utilizing parameter-efficient approaches such as prompt tuning. Additionally, the most powerful multi-task LLMs, such as OPT-IML-175B and FLAN-PaLM-540B, are not publicly accessible, severely limiting their customization potential. To address these challenges, we introduce a pretrained small scorer, Cappy, designed to enhance the performance and efficiency of multi-task LLMs. With merely 360 million parameters, Cappy functions either independently on classification tasks or serve as an auxiliary component for LLMs, boosting their performance. Moreover, Cappy enables efficiently integrating downstream supervision without requiring LLM finetuning nor the access to their parameters. Our experiments demonstrate that, when working independently on 11 language understanding tasks from PromptSource, Cappy outperforms LLMs that are several orders of magnitude larger. Besides, on 45 complex tasks from BIG-Bench, Cappy boosts the performance of the advanced multi-task LLM, FLAN-T5, by a large margin. Furthermore, Cappy is flexible to cooperate with other LLM adaptations, including finetuning and in-context learning, offering additional performance enhancement.
Large language models (LLMs) have led to a new paradigm that seeks to unify various natural language processing (NLP) tasks within an instruction-following framework. This paradigm is exemplified by the recent multi-task LLMs, such as T0 [23], FLAN [30, 4], and OPT-IML [10]. These models are trained with data from many tasks: for each task, following a task-specific template, each labeled example is converted into an instruction (e.g., "Put the concepts together to form a sentence: ski, mountain, skier.") and a corresponding response (e.g., "Skier skis down the mountain"). Such (instruction, response) pairs are then used to train the LLM, resulting in a conditional generation model that takes input of a data example as an instruction
*Work done during an internship at Google.
$^{2}$ Code and model available at https://github.com/tanyuqian/cappy and https://huggingface.co/btan2/cappy-large, respectively.
Figure 1: Cappy outperforms multi-task LLMs: The overall accuracy averaged over 11 test tasks from PromptSource. Every dashed line connects different sizes of the same model. Lines positioned more towards the upper left denote models that are more efficient and yield superior performance.
Figure 2: Cappy boosts multi-task LLMs: The averaged Rouge-L score over 45 complex tasks within BIG-Bench. Every dashed line represents an approach working on LLMs of various sizes. Self-scoring refers to using the cross-entropy of LLM to select responses.
and generates a response. Moreover, these multi-task LLMs have exhibited remarkable task-wise generalization capabilities. That is, they can address unseen tasks by understanding and solving brand-new instructions.
Due to the complexity of understanding and resolving various tasks solely via instructions, the sizes of these multi-task LLMs typically span from several billion parameters to hundreds of billions, such as T0-11B [23] and OPT-IML-175B [10]. As a result, operating such sizable models poses significant challenges to the majority of LLM users, because they demand considerable computational power and impose substantial requirements on the memory capacities of GPUs/TPUs, making their training and inference expensive and inefficient.
In practical applications, harnessing a single multi-task LLM to manage all conceivable tasks in a zero-shot manner remains challenging, particularly when dealing with complex tasks, personalized tasks and those that cannot be succinctly defined using instructions. On the other hand, the size of downstream training data is usually insufficient to well train a model without incorporating rich prior knowledge. Hence, it is long desired to adapt LLMs with downstream supervision. Yet, the adaptation process presents three significant obstacles: the extensive storage to maintain a unique LLM copy for each downstream task; the considerable memory demands on GPUs/TPUs; and the unavailability of the most powerful multi-task LLMs, such as OPT-IML-175B [10] and FLAN-PaLM-540B [4]. Certain parameter-efficient tuning strategies, including prompt tuning [18] and adapters [8], substantially diminish storage requirements, but they still perform back-propagation through the LLM parameters during the tuning process, thereby their memory demands keep high. Additionally, some in-context learning techniques [5] circumvent parameter tuning by integrating a limited number of supervised examples into the instruction. However, these techniques are constrained by the model's maximum input length, which permits only a few samples to guide task resolution.
In this work, we propose a novel approach to enhance the performance and efficiency of multi-task LLMs. Specifically, we introduce a lightweight pretrained scorer, Cappy, based on a continual pretraining on top of RoBERTa [20], with merely 360 million parameters. Cappy takes in an instruction and a candidate response as input, and produces a score between 0 and 1, indicating an estimated correctness of the response with respect to the instruction. Naturally, we formulate Cappy's pretraining as a regression problem. This anticipates training data in the form of (instruction, response) pairs that correspond to various correctness score annotations. To generate the desired training data from multiple pretrain datasets that solely contain instructions and their ground truth responses, we propose a weakly-supervised approach with data augmentation through the use of existing multi-task LLMs. As a result, we obtain a large and effective regression pretraining dataset with diverse correctness score annotations ranging from 0 to 1.
To apply Cappy to practical problem-solving scenarios, we suggest an intuitive approach in an candidate selection style. Specifically, Cappy works independently on classification tasks by selecting
Figure 3: (left) The modeling of Cappy. (right) Illustration of Cappy's application in enhancing multi-task LLMs, and the comparison between downstream adaptation through Cappy and approaches that rely on LLM's parameters, such as finetuning and prompt tuning.
the answer choice that produces the highest score. Furthermore, beyond the standalone use, Cappy serves as an auxiliary component of existing multi-task LLMs, choosing the most appropriate output from a set of candidates generated by the LLM. In this case, Cappy allows for effective and efficient adaptation to complex tasks through the incorporation of downstream supervision, without requiring finetuning the multi-task LLM or the access to its parameters. Remarkably, Cappy exhibits flexibility in collaboration with other LLM adaptations, such as finetuning and in-context learning.
We validate Cappy through an extensive suite of held-out tasks distinct from those incorporated in its pretraining. The overall performance is as shown in Fig. 1 and Fig. 2. Specifically, on 11 language understanding tasks drawn from PromptSource [1], Cappy, with 360 million parameters, outperforms OPT-IML-30B and OPT-175B significantly, and matches the best ones among previous multi-task LLMs. Besides, on 45 diverse complex tasks from BIG-Bench [25], Cappy consistently boosts the performance of the advanced multi-task LLM, FLAN-T5, by a large margin. Furthermore, Cappy offers additional performance enhancement when applied together with finetuning or in-context learning. Our subsequent ablation study proves the significance of our proposed pretraining and data augmentation strategies.
LLMs for Instruction Following and Multi-task Prompted Training The scaling up of language models brings them increasingly strong capabilities, culminating in a general paradigm of addressing diverse problems in a unified instruction-following manner. There are two primary approaches of such LLMs, each distinguished by the purpose of their instructions. The first approach emphasizes compliance with arbitrary human instructions, often in a question-and-answer or dialogue format (e.g., "I have to make a difficult decision. What should I do?"). Models such as GPT-4 [21] and Vicuna [3] are designed to respond to these instructions with the goal of maximizing user satisfaction. These models are typically trained through Reinforcement Learning with Human Feedback (RLHF) [13], leveraging extensive human annotations. Their quantitative evaluation also heavily depends on human judgment [33]. The second approach, however, is primarily devoted to resolving well-defined NLP tasks. In this context, each data instance adheres to a task-specific template, and is transformed into an instruction (e.g., "Put the concepts together to form a sentence: ski, mountain, skier.") and a corresponding response (e.g., "A skier skis down the mountain."). Multi-task LLMs, such as OPT-IML [10], FLAN [30, 4], and T0 [23], are pretrained via multi-task prompted training. This process trains models as a unified conditional generation task using pairs of instructions and responses from multiple upstream pretraining tasks. These models are typically assessed based on performance on held-out test tasks, utilizing traditional evaluation metrics such as accuracy, Rouge scores [17], and so forth. In this study, our primary focus is on the second approach, i.e., multi-task LLMs, given its more straightforward evaluation. Nonetheless, we posit that there is no significant obstacle to apply our proposed methodologies to more humanish instructions, which we leave as our future direction.
Figure 4: Data augmentation with a multi-task LLM to construct weakly supervised regression dataset for Cappy's pretraining and finetuning, as described in Sec. 3.2.
Adaptation of LLMs The size of LLMs makes their finetuning for downstream tasks particularly challenging, primarily due to three issues. Firstly, finetuning necessitates the creation of a new copy of an LLM for each specific downstream task. This is unacceptable for many applications. Secondly, fine-tuning an LLM demands considerable device memory because of the back-propagation through the LLMs, which is achievable only with high-end GPU/TPU clusters. Thirdly, the most powerful LLMs, such as FLAN-PaLM-540B [4] and GPT-4 [21], are closed-source and thus inaccessible for fine-tuning. A collection of parameter-efficient LLM adaptation techniques, including prompt tuning [18] and adapters [8], like prefix tuning [15] and LoRA [7], have largely mitigated the storage issue by decreasing the number of tunable parameters. However, these methods still require back propagation through the original LLM weights to update the prompt or the adapter, leaving the second and third issues remain significant barriers in LLM adaptation. Certain in-context learning techniques [5] circumvent LLM's parameter tuning by appending training examples to the instruction. However, the instruction length is limited by the model's maximum input length, such in-context learning techniques allow for only a finite number of samples to guide the task-solving process. In this work, we propose the adaptation of multi-task LLMs by employing Cappy to incorporate downstream supervision. This approach enables any number of training examples without necessitating LLM finetuning or access to its parameters. Hence, the LLM serves merely as a black box, and Cappy is even compatible with WebAPIs of LLMs. Importantly, Cappy can also be deployed in conjunction with other LLM adaptations, such as finetuning and in-context learning.
Ranking-based Models Ranking is a pivotal component of information retrieval systems, notably in search engines and recommendation systems [19]. It involves sorting vast numbers of documents to find content pertinent to a specific query. Recently, ranking has been adapted for NLP tasks to aggregate answers [14], and cater to human preferences [2, 6]. Furthermore, in the emergent domain of reinforcement learning from human feedback (RLHF) [26], ranking models trained on human-ranked model outputs serve as reward providers for the training RL agents. Concurrently with this work, [32] proposes a unified ranking model solving information alignment style tasks, such as natural language inference and paraphrase detection. In this work, Cappy is conceptually a ranking model for multi-task learning. Unlike methodologies specifically designed for question answering [12] or summarization [22], Cappy offers extensive generalizability across multi-task scenarios. Additionally, in contrast to RLHF reward models, Cappy doesn't rely on expensive human-annotated data, enabling large-scale pretraining.
3 The Cappy Scorer
3.1 Modeling
Cappy adopts the architecture of RoBERTa [20] with a linear layer on the top as a regression head. The input of Cappy is a pair of text, comprising an instruction and a response, and the output is a scalar score ranging from 0 to 1. This score indicates an estimation of the correctness of the response with regard to the task instance described in the instruction.
3.2 Pretraining
Cappy's pretraining uses the same dataset collection that is utilized by T0 [23]. This collection is comprised of 39 diverse datasets from PromptSource [1], encompassing a wide range of task types, such as question answering, sentiment analysis, and summarization, etc. Each dataset is associated with one or multiple templates, converting each instance from the original datasets into a instruction
paired with its ground truth response. Following the pretraining configuration of T0, the size of each dataset is limited to a maximum of 500,000 examples.
In light of Cappy's regression modeling, each data instance during pretraining must have a text pair (instruction, response), coupled with an correctness annotation for the response in relation to the instruction. A diverse array of score annotations is a pivotal aspect of a regression dataset. However, the text pairs in our pretraining datasets merely contain instructions with their ground truth responses, hence each text pair invariably has a correctness score of 1.0. This could culminate in a critical lack of label diversity throughout Cappy's pretraining. To address this issue, we propose a data construction approach to produce Cappy's pretraining dataset with correctness annotations that diversely range from 0 to 1. The data construction consists of three components:
Ground Truth (score 1.0) This component encompasses all ground truth instruction-response pairs from the pretraining dataset. Each pair is assigned a correctness annotation of 1.0.
Incorrect Responses (score 0.0) We engineer incorrect data points by creating mismatched instruction-response pairs from the original datasets. For classification datasets, each instruction is paired with all incorrect answer choices. For generation datasets, each instruction is arbitrarily paired with a ground truth response from a distinct data point within the dataset.
Data Augmentation (score within [0, 1]) In addition to purely correct or incorrect data samples, we fabricate instruction-response pairs with scores ranging between 0 and 1. This is achieved through data augmentation applied across all generation task instances. For every instance within a generation task, we leverage an existing multi-task LLM to generate multiple responses by sampling conditioned on the given instruction. Subsequently, we assign an annotation to the pair formed by the instruction and every response, using the similarity between the response and the ground truth response of the instance. Specifically, we employ Rouge-L [17] to calculate this similarity as a form of weak supervision. as it has been widely recognized as a reliable metric for overall performance in multi-task environments and has demonstrated a strong alignment with human evaluation [29]. In practice, our augmented samples are generated by two multi-task LLMs, BART0 [16] and T0-3B [23]. For each instance within a generation task, both these models generate two samples using the top-k and top-p sampling, respectively.
Consequently, we collect a pretraining dataset comprised of 160 million instances, each in the format of (instruction, response, score). Cappy is initialized as RoBERTa and optimized using the AdamW optimizer with an L2 regression loss. The optimization process involves a learning rate of  $10^{-6}$ , a warmup rate of 0.1, and an effective batch size of 1024. In alignment with the RoBERTa variants, Cappy is also offered in two distinct versions: the smaller Cappy<sub>BASE</sub>(120M parameters), and the Cappy<sub>LARGE</sub>(360M parameters).
3.3 Applying Cappy
Cappy solves practical tasks within a candidate-selection mechanism. More specifically, given an instruction and a set of candidate responses, Cappy produces a score for each candidate. This is achieved by inputting the instruction alongside each individual response, and then assigning the response with the highest score as its prediction. In classification tasks, all candidate responses are inherently predefined. For example, the options are {positive, negative} in a sentiment classification task. In such scenarios, Cappy functions independently. On the other hand, in generation tasks, candidate responses are not pre-defined, requiring an existing multi-task LLM to yield the candidate responses. In this case, Cappy serves as an auxiliary component of the multi-task LLM, enhancing its decoding.
3.4 Adapting Multi-task LLMs
When there is available downstream training data, Cappy enables effective and efficient adaptation of multi-task LLMs on downstream tasks. Specifically, we propose to integrate downstream task information into LLM's predictions through the finetuning of Cappy. To elaborate, a downstream regression dataset can be acquired through a data annotation process same as the approach utilized during the pretraining data construction (§3.2). Then, Cappy can be finetuned on this regression dataset. As a result, the finetuned Cappy collaborates with a multi-task LLM, boosting the LLM's performance on the downstream task.
Figure 5: The performance of Cappy and multi-task LLMs on various test datasets. A series of dashed lines are used to connect different sizes of the same model, such as OPT-30B and OPT-175B. Lines or points positioned more towards the upper left of the diagram denote models that are more efficient and yield superior performance. Each diagram corresponds to a specific test task, with the exception of ANLI that represents three different tasks (ANLI-R1/R2/R3).
In contrast to other LLM tuning strategies such as finetuning and prompt tuning, adapting LLMs with Cappy on downstream tasks avoids the need for back-propagation through LLM parameters. Therefore, it significantly reduces the high demand for device memory. Besides, the adaptation with Cappy does not rely on the access to the LLM parameters, making it compatible with closed-source multi-task LLMs, such as the ones only accessible via WebAPIs. Compared with in-context learning approaches which circumvent model tuning by attaching training examples to the instruction prefix, Cappy is not restricted by the LLM's maximum input length. Thus, Cappy can incorporate an unlimited number of downstream training examples. Moreover, Cappy is flexible to work together with other adaptation methods, such as finetuning and in-context learning, further boosting their overall performance, as we demonstrate in experiments.
All the experiments of this work, including the pretraining of Cappy and downstream adaptations, are conducted on Google TPU-v4 [11], and all the code is implemented with Redco [27], a lightweight toolkit for automating distributed training.
4.1 Zero-shot Performance on PromptSource
Our evaluation aligns with the ones used by OPT-IML and T0. We assess performance on 11 held-out language understanding tasks in PromptSource [1], all of which are in classification style. These tasks are categorically distinct from those utilized in the pretraining datasets. Our baselines include multi-task LLMs, i.e., OPT, OPT-IML, T0, and BART0, and a RLHF reward model trained with human feedback data, released by LAION-AI  $^{3}$ . Following the answer selection strategy employed
<sup>3</sup>https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2
Model	Accuracy
BART0BASE-140M	45.7
BART0LARGE-400M	50.2
RLHF-RM-185M	43.6
RLHF-RM-435M	53.3
OPT-30B	47.6
OPT-IML-30B	51.3
OPT-175B	49.3
OPT-IML-175B	56.5
T0-11B	58.2
CappyBASE-120M	49.9
CappyLARGE-360M	56.6
Table 1: The overall accuracy averaged over 11 held-out test tasks from PromptSource in a zero-shot setting. "RLHF-RM" refers to the RLHF reward model mentioned in Section 4.1.
Frozen FLAN-T5
Small	Base	Large	XL	XXL
Sampling	11.43	15.79	19.62	23.22	25.73
Temperature	12.01	17.06	20.05	24.27	27.10
Top-K	11.52	15.75	19.76	22.67	25.82
Nucleus	11.92	16.62	20.20	24.17	26.90
Beam Search	16.40	19.86	23.48	26.12	29.66
Self-scoring	15.08	20.71	24.12	28.47	32.02
CappyBASE	23.36	27.26	29.83	32.79	36.63
CappyLARGE	24.45	28.25	30.75	33.97	36.93
ICL + Nucleus	16.37	20.46	23.65	28.64	32.70
ICL + Self-scoring	20.61	24.42	27.00	32.56	36.37
ICL + CappyLARGE	26.18	28.65	31.84	36.41	38.48
Table 2: The averaged Rouge-L score over 45 BIG-Bench tasks. The backbone FLAN-T5 models are frozen. "ICL" refers to in-context learning, i.e., attaching training examples in the prefix of instruction. We include more prompt-tuning related results in the appendix.
by T0 and OPT-IML, predictions from these multi-task LLMs are determined by the answer choice that yields the highest model likelihood. FLAN is not considered among the baselines, as the test tasks are included in its pretraining tasks. We calculate the performance for each task by averaging the results across all associated prompts.
The outcomes of the 11 tasks are presented in Figure 5, with the averaged overall performance of each model summarized in Table 1. From the per-task figures, Cappy consistently outperforms BART0 models, which are comparable in size to Cappy, and also surpasses OPT and OPT-IML in the majority of tasks. From the overall accuracy in Table 1, we can summarize that our Cappy<sub>BASE</sub> model yields performance comparable to that of OPT-30B, OPT-IML-30B, and OPT-175B. Furthermore, our larger 360M Cappy<sub>LARGE</sub> model performs at a level consistent with T0-11B and OPT-IML-175B. These findings highlight Cappy's superior performance and parameter efficiency in comparison to existing multi-task LLMs. This improved performance can be credited to Cappy's scoring-based pretraining strategy, which integrates contrastive information by differentiating between high-quality and low-quality responses. On the contrary, previous multi-task LLMs depend exclusively on teacher-forcing training that utilizes only the ground truth responses.
4.2 Adaptation to BIG-Bench