Title: Cappy: Outperforming and Boosting Large Multi-Task LMs with a Small Scorer
Venue: NeurIPS 2023
Accepted: yes
Keywords: multi-task

large language models

pretrain model
Abstract: Large language models (LLMs) such as T0, FLAN, and OPT-IML excel in multi-tasking under a unified instruction-following paradigm, where they also exhibit remarkable generalization abilities to unseen tasks. Despite their impressive performance, these LLMs, with sizes ranging from several billion to hundreds of billions of parameters, demand substantial computational resources, making their training and inference expensive and inefficient. Furthermore, adapting these models to downstream applications, particularly complex tasks, is often unfeasible due to the extensive hardware requirements for finetuning, even when utilizing parameter-efficient approaches such as prompt tuning. Additionally, the most powerful multi-task LLMs, such as OPT-IML-175B and FLAN-PaLM-540B, are not publicly accessible, severely limiting their customization potential. To address these challenges, we introduce a pretrained small scorer, \textit{Cappy}, designed to enhance the performance and efficiency of multi-task LLMs. With merely 360 million parameters, Cappy functions either independently on classification tasks or serve as an auxiliary component for LLMs, boosting their performance. Moreover, Cappy enables efficiently integrating downstream supervision without requiring LLM finetuning nor the access to their parameters. Our experiments demonstrate that, when working independently on 11 language understanding tasks from PromptSource, Cappy outperforms LLMs that are several orders of magnitude larger. Besides, on 45 complex tasks from BIG-Bench, Cappy boosts the performance of the advanced multi-task LLM, FLAN-T5, by a large margin. Furthermore, Cappy is flexible to cooperate with other LLM adaptations, including finetuning and in-context learning, offering additional performance enhancement.

Cappy: Outperforming and Boosting Large

Abstract

Large language models (LLMs) such as T0, FLAN, and OPT-IML, excel in multitasking under a unified instruction-following paradigm, where they also exhibit remarkable generalization abilities to unseen tasks. Despite their impressive performance, these LLMs, with sizes ranging from several billion to hundreds of billions of parameters, demand substantial computational resources, making their training and inference expensive and inefficient. Furthermore, adapting these models to downstream applications, particularly complex tasks, is often unfeasible due to the extensive hardware requirements for finetuning, even when utilizing parameter-efficient approaches such as prompt tuning. Additionally, the most powerful multi-task LLMs, such as OPT-IML-175B and FLAN-PaLM-540B, are not publicly accessible, severely limiting their customization potential. To address these challenges, we introduce a pretrained small scorer, Cappy , designed to enhance the performance and efficiency of multi-task LLMs. With merely 360 million parameters, Cappy functions either independently on classification tasks or serve as an auxiliary component for LLMs, boosting their performance. Moreover, Cappy enables efficiently integrating downstream supervision without requiring LLM finetuning nor the access to their parameters. Our experiments demonstrate that, when working independently on 11 language understanding tasks from PromptSource, Cappy outperforms LLMs that are several orders of magnitude larger. Besides, on 45 complex tasks from BIG-Bench, Cappy boosts the performance of the advanced multi-task LLM, FLAN-T5, by a large margin. Furthermore, Cappy is flexible to cooperate with other LLM adaptations, including finetuning and in-context learning, offering additional performance enhancement. 2

1 Introduction

Large language models (LLMs) have led to a new paradigm that seeks to unify various natural language processing (NLP) tasks within an instruction-following framework. This paradigm is exemplified by the recent multi-task LLMs, such as T0 [23], FLAN [30, 4], and OPT-IML [10]. These models are trained with data from many tasks: for each task, following a task-specific template, each labeled example is converted into an instruction (e.g., "Put the concepts together to form a sentence: ski, mountain, skier." ) and a corresponding response (e.g., "Skier skis down the mountain" ). Such (instruction, response) pairs are then used to train the LLM, resulting in a conditional generation model that takes input of a data example as an instruction

10

1

10 0

10 1

10 2

46

48

50

52

54

56

OPT

OPT-IML

BART0

Cappy (ours)

10

1

10 0

10 1

15

20

25

30

35

Rouge-L

Nucleus

Beam Search

Self-scoring

Cappy

BASE

(ours)

Cappy

LARGE

(ours)

and generates a response. Moreover, these multi-task LLMs have exhibited remarkable task-wise generalization capabilities. That is, they can address unseen tasks by understanding and solving brand-new instructions.

Due to the complexity of understanding and resolving various tasks solely via instructions, the sizes of these multi-task LLMs typically span from several billion parameters to hundreds of billions, such as T0-11B [23] and OPT-IML-175B [10]. As a result, operating such sizable models poses significant challenges to the majority of LLM users, because they demand considerable computational power and impose substantial requirements on the memory capacities of GPUs/TPUs, making their training and inference expensive and inefficient.

In practical applications, harnessing a single multi-task LLM to manage all conceivable tasks in a zero-shot manner remains challenging, particularly when dealing with complex tasks, personalized tasks and those that cannot be succinctly defined using instructions. On the other hand, the size of downstream training data is usually insufficient to well train a model without incorporating rich prior knowledge. Hence, it is long desired to adapt LLMs with downstream supervision. Yet, the adaptation process presents three significant obstacles: the extensive storage to maintain a unique LLM copy for each downstream task; the considerable memory demands on GPUs/TPUs; and the unavailability of the most powerful multi-task LLMs, such as OPT-IML-175B [10] and FLAN-PaLM540B [4]. Certain parameter-efficient tuning strategies, including prompt tuning [18] and adapters [8], substantially diminish storage requirements, but they still perform back-propagation through the LLM parameters during the tuning process, thereby their memory demands keep high. Additionally, some in-context learning techniques [5] circumvent parameter tuning by integrating a limited number of supervised examples into the instruction. However, these techniques are constrained by the model's maximum input length, which permits only a few samples to guide task resolution.

In this work, we propose a novel approach to enhance the performance and efficiency of multi-task LLMs. Specifically, we introduce a lightweight pretrained scorer, Cappy , based on a continual pretraining on top of RoBERTa [20], with merely 360 million parameters. Cappy takes in an instruction and a candidate response as input, and produces a score between 0 and 1, indicating an estimated correctness of the response with respect to the instruction. Naturally, we formulate Cappy's pretraining as a regression problem. This anticipates training data in the form of (instruction, response) pairs that correspond to various correctness score annotations. To generate the desired training data from multiple pretrain datasets that solely contain instructions and their ground truth responses, we propose a weakly-supervised approach with data augmentation through the use of existing multi-task LLMs. As a result, we obtain a large and effective regression pretraining dataset with diverse correctness score annotations ranging from 0 to 1.

To apply Cappy to practical problem-solving scenarios, we suggest an intuitive approach in an candidate selection style. Specifically, Cappy works independently on classification tasks by selecting

Multi-task LLM

e.g., FLAN, OPT-IML

Cappy

Downstream training data

Enhanced prediction

Adaptation via Cappy

•

Efficient

•

Any amount of training data

•

Enhance whatever LLMs

The	picture	appeared	on	the	wall	of	a

Poundland store	on	Whymark Avenue

[...]	How	would	you	rephrase	that	in

a	few	words?

Graffiti	artist	Banksy	is	believed	to	be

behind	[...]

Score: 0.86

Instruction:

Response:

Cappy

Adaptation through LLM

•

Expensive

•

Demanding to hardware

•

Accessibility issue

samples

argmax

the answer choice that produces the highest score. Furthermore, beyond the standalone use, Cappy serves as an auxiliary component of existing multi-task LLMs, choosing the most appropriate output from a set of candidates generated by the LLM. In this case, Cappy allows for effective and efficient adaptation to complex tasks through the incorporation of downstream supervision, without requiring finetuning the multi-task LLM or the access to its parameters. Remarkably, Cappy exhibits flexibility in collaboration with other LLM adaptations, such as finetuning and in-context learning.

We validate Cappy through an extensive suite of held-out tasks distinct from those incorporated in its pretraining. The overall performance is as shown in Fig. 1 and Fig. 2. Specifically, on 11 language understanding tasks drawn from PromptSource [1], Cappy, with 360 million parameters, outperforms OPT-IML-30B and OPT-175B significantly, and matches the best ones among previous multi-task LLMs. Besides, on 45 diverse complex tasks from BIG-Bench [25], Cappy consistently boosts the performance of the advanced multi-task LLM, FLAN-T5, by a large margin. Furthermore, Cappy offers additional performance enhancement when applied together with finetuning or in-context learning. Our subsequent ablation study proves the significance of our proposed pretraining and data augmentation strategies.

2 Related Work

LLMs for Instruction Following and Multi-task Prompted Training The scaling up of language models brings them increasingly strong capabilities, culminating in a general paradigm of addressing diverse problems in a unified instruction-following manner. There are two primary approaches of such LLMs, each distinguished by the purpose of their instructions. The first approach emphasizes compliance with arbitrary human instructions, often in a question-and-answer or dialogue format (e.g., "I have to make a difficult decision. What should I do?" ). Models such as GPT-4 [21] and Vicuna [3] are designed to respond to these instructions with the goal of maximizing user satisfaction. These models are typically trained through Reinforcement Learning with Human Feedback (RLHF) [13], leveraging extensive human annotations. Their quantitative evaluation also heavily depends on human judgment [33]. The second approach, however, is primarily devoted to resolving well-defined NLP tasks. In this context, each data instance adheres to a task-specific template, and is transformed into an instruction (e.g., "Put the concepts together to form a sentence: ski, mountain, skier." ) and a corresponding response (e.g., "A skier skis down the mountain." ). Multi-task LLMs, such as OPT-IML [10], FLAN [30, 4], and T0 [23], are pretrained via multi-task prompted training. This process trains models as a unified conditional generation task using pairs of instructions and responses from multiple upstream pretraining tasks. These models are typically assessed based on performance on held-out test tasks, utilizing traditional evaluation metrics such as accuracy, Rouge scores [17], and so forth. In this study, our primary focus is on the second approach, i.e., multi-task LLMs, given its more straightforward evaluation. Nonetheless, we posit that there is no significant obstacle to apply our proposed methodologies to more humanish instructions, which we leave as our future direction.

The	picture	appeared	on	the	wall	of	a

Poundland store	on	Whymark Avenue	[...]

How	would	you	rephrase	that	in	a	few

words?

Instruction

:

Graffiti	artist	Banksy	is	believed	to	be

behind	[...]

Target (ground truth):

Multi-Task LLM

e.g., FLAN

Sample 1

Sample 2

Sample 3

Sample 4

Regression Training Data for Cappy

Ground-truth instruction-response pair

Input text pair

Score annotation

(instruction, sample 1)

Rouge-L

(sample 1, target)

(instruction, sample 2)

Rouge-L

(sample 2, target)

(instruction, sample 3)

Rouge-L

(sample 3, target)

(instruction, sample 4)

Rouge-L

(sample 4, target)

Adaptation of LLMs The size of LLMs makes their finetuning for downstream tasks particularly challenging, primarily due to three issues. Firstly, finetuning necessitates the creation of a new copy of an LLM for each specific downstream task. This is unacceptable for many applications. Secondly, fine-tuning an LLM demands considerable device memory because of the back-propagation through the LLMs, which is achievable only with high-end GPU/TPU clusters. Thirdly, the most powerful LLMs, such as FLAN-PaLM-540B [4] and GPT-4 [21], are closed-source and thus inaccessible for fine-tuning. A collection of parameter-efficient LLM adaptation techniques, including prompt tuning [18] and adapters [8], like prefix tuning [15] and LoRA [7], have largely mitigated the storage issue by decreasing the number of tunable parameters. However, these methods still require back propagation through the original LLM weights to update the prompt or the adapter, leaving the second and third issues remain significant barriers in LLM adaptation. Certain in-context learning techniques [5] circumvent LLM's parameter tuning by appending training examples to the instruction. However, the instruction length is limited by the model's maximum input length, such in-context learning techniques allow for only a finite number of samples to guide the task-solving process. In this work, we propose the adaptation of multi-task LLMs by employing Cappy to incorporate downstream supervision. This approach enables any number of training examples without necessitating LLM finetuning or access to its parameters. Hence, the LLM serves merely as a black box, and Cappy is even compatible with WebAPIs of LLMs. Importantly, Cappy can also be deployed in conjunction with other LLM adaptations, such as finetuning and in-context learning.

Ranking-based Models Ranking is a pivotal component of information retrieval systems, notably in search engines and recommendation systems [19]. It involves sorting vast numbers of documents to find content pertinent to a specific query. Recently, ranking has been adapted for NLP tasks to aggregate answers [14], and cater to human preferences [2, 6]. Furthermore, in the emergent domain of reinforcement learning from human feedback (RLHF) [26], ranking models trained on human-ranked model outputs serve as reward providers for the training RL agents. Concurrently with this work, [32] proposes a unified ranking model solving information alignment style tasks, such as natural language inference and paraphrase detection. In this work, Cappy is conceptually a ranking model for multi-task learning. Unlike methodologies specifically designed for question answering [12] or summarization [22], Cappy offers extensive generalizability across multi-task scenarios. Additionally, in contrast to RLHF reward models, Cappy doesn't rely on expensive human-annotated data, enabling large-scale pretraining.

3 The Cappy Scorer

3.1 Modeling

Cappy adopts the architecture of RoBERTa [20] with a linear layer on the top as a regression head. The input of Cappy is a pair of text, comprising an instruction and a response, and the output is a scalar score ranging from 0 to 1. This score indicates an estimation of the correctness of the response with regard to the task instance described in the instruction.

3.2 Pretraining

Cappy's pretraining uses the same dataset collection that is utilized by T0 [23]. This collection is comprised of 39 diverse datasets from PromptSource [1], encompassing a wide range of task types, such as question answering, sentiment analysis, and summarization, etc. Each dataset is associated with one or multiple templates, converting each instance from the original datasets into a instruction

paired with its ground truth response. Following the pretraining configuration of T0, the size of each dataset is limited to a maximum of 500,000 examples.

In light of Cappy's regression modeling, each data instance during pretraining must have a text pair (instruction, response) , coupled with an correctness annotation for the response in relation to the instruction. A diverse array of score annotations is a pivotal aspect of a regression dataset. However, the text pairs in our pretraining datasets merely contain instructions with their ground truth responses, hence each text pair invariably has a correctness score of 1.0. This could culminate in a critical lack of label diversity throughout Cappy's pretraining. To address this issue, we propose a data construction approach to produce Cappy's pretraining dataset with correctness annotations that diversely range from 0 to 1. The data construction consists of three components:

Ground Truth (score 1.0) This component encompasses all ground truth instruction-response pairs from the pretraining dataset. Each pair is assigned a correctness annotation of 1.0.

Incorrect Responses (score 0.0) We engineer incorrect data points by creating mismatched instruction-response pairs from the original datasets. For classification datasets, each instruction is paired with all incorrect answer choices. For generation datasets, each instruction is arbitrarily paired with a ground truth response from a distinct data point within the dataset.

Data Augmentation (score within [0 , 1] ) In addition to purely correct or incorrect data samples, we fabricate instruction-response pairs with scores ranging between 0 and 1. This is achieved through data augmentation applied across all generation task instances. For every instance within a generation task, we leverage an existing multi-task LLM to generate multiple responses by sampling conditioned on the given instruction. Subsequently, we assign an annotation to the pair formed by the instruction and every response, using the similarity between the response and the ground truth response of the instance. Specifically, we employ Rouge-L [17] to calculate this similarity as a form of weak supervision. as it has been widely recognized as a reliable metric for overall performance in multi-task environments and has demonstrated a strong alignment with human evaluation [29]. In practice, our augmented samples are generated by two multi-task LLMs, BART0 [16] and T0-3B [23]. For each instance within a generation task, both these models generate two samples using the top-k and top-p sampling, respectively.

Consequently, we collect a pretraining dataset comprised of 160 million instances, each in the format of (instruction, response, score) . Cappy is initialized as RoBERTa and optimized using the AdamW optimizer with an L2 regression loss. The optimization process involves a learning rate of 10 -6 , a warmup rate of 0.1, and an effective batch size of 1024. In alignment with the RoBERTa variants, Cappy is also offered in two distinct versions: the smaller Cappy BASE (120M parameters), and the Cappy LARGE (360M parameters).

3.3 Applying Cappy

Cappy solves practical tasks within a candidate-selection mechanism. More specifically, given an instruction and a set of candidate responses, Cappy produces a score for each candidate. This is achieved by inputting the instruction alongside each individual response, and then assigning the response with the highest score as its prediction. In classification tasks, all candidate responses are inherently predefined. For example, the options are {positive, negative} in a sentiment classification task. In such scenarios, Cappy functions independently. On the other hand, in generation tasks, candidate responses are not pre-defined, requiring an existing multi-task LLM to yield the candidate responses. In this case, Cappy serves as an auxiliary component of the multi-task LLM, enhancing its decoding.

3.4 Adapting Multi-task LLMs

When there is available downstream training data, Cappy enables effective and efficient adaptation of multi-task LLMs on downstream tasks. Specifically, we propose to integrate downstream task information into LLM's predictions through the finetuning of Cappy. To elaborate, a downstream regression dataset can be acquired through a data annotation process same as the approach utilized during the pretraining data construction (§3.2). Then, Cappy can be finetuned on this regression dataset. As a result, the finetuned Cappy collaborates with a multi-task LLM, boosting the LLM's performance on the downstream task.

60

70

80

90

StoryCloze

30

40

50

60

CB

50

60

70

80

WinoGender

32

34

36

38

40

ANLI (R1/R2/R3)

45

50

55

60

WSC

50

52

54

56

WIC

10

1

10 0

10 1

10 2

50

55

60

WinoGrande

10

1

10 0

10 1

10 2

20

40

60

80

Crows-Pairs

10

1

10 0

10 1

10 2

50

60

70

80

RTE

BART0

OPT

OPT-IML

T0

Cappy (ours)

In contrast to other LLM tuning strategies such as finetuning and prompt tuning, adapting LLMs with Cappy on downstream tasks avoids the need for back-propagation through LLM parameters. Therefore, it significantly reduces the high demand for device memory. Besides, the adaptation with Cappy does not rely on the access to the LLM parameters, making it compatable with closed-source multi-task LLMs, such as the ones only accessible via WebAPIs. Compared with in-context learning approaches which circumvent model tuning by attaching training examples to the instruction prefix, Cappy is not restricted by the LLM's maximum input length. Thus, Cappy can incorporate an unlimited number of downstream training examples. Moreover, Cappy is flexible to work together with other adaptation methods, such as finetuning and in-context learning, further boosting their overall performance, as we demonstrate in experiments.

4 Experiments

All the experiments of this work, including the pretraining of Cappy and downstream adaptations, are conducted on Google TPU-v4 [11], and all the code is implemented with Redco [27], a lightweight toolkit for automating distributed training.

4.1 Zero-shot Performance on PromptSource

Our evaluation aligns with the ones used by OPT-IML and T0. We assess performance on 11 held-out language understanding tasks in PromptSource [1], all of which are in classification style. These tasks are categorically distinct from those utilized in the pretraining datasets. Our baselines include multi-task LLMs, i.e., OPT, OPT-IML, T0, and BART0, and a RLHF reward model trained with human feedback data, released by LAION-AI 3 . Following the answer selection strategy employed

by T0 and OPT-IML, predictions from these multi-task LLMs are determined by the answer choice that yields the highest model likelihood. FLAN is not considered among the baselines, as the the test tasks are included in its pretraining tasks. We calculate the performance for each task by averaging the results across all associated prompts.

The outcomes of the 11 tasks are presented in Figure 5, with the averaged overall performance of each model summarized in Table 1. From the per-task figures, Cappy consistently outperforms BART0 models, which are comparable in size to Cappy, and also surpasses OPT and OPT-IML in the majority of tasks. From the overall accurary in Table 1, we can summarize that our Cappy BASE model yields performance comparable to that of OPT-30B, OPT-IML-30B, and OPT-175B. Furthermore, our larger 360M CappyLARGEmodel performs at a level consistent with T0-11B and OPT-IML-175B. These findings highlight Cappy's superior performance and parameter efficiency in comparison to existing multi-task LLMs. This improved performance can be credited to Cappy's scoring-based pretraining strategy, which integrates contrastive information by differentiating between high-quality and lowquality responses. On the contrary, previous multi-task LLMs depend exclusively on teacher-forcing training that utilizes only the ground truth responses.

4.2 Adaptation to BIG-Bench

We examine the adaptation of multi-task LLMs with Cappy on complex tasks from BIG-Bench [25], a set of manually curated, challenging tasks that are considered beyond the capability of many LLMs. We focus on all the 45 generation tasks within BIG-Bench, specifically those that do not offer pre-established answer choices. The train/test split is provided by TaskSource [24]. The training sizes of these tasks are variable, with a minimum of 14 and a maximum of 50,000 instances. The median training size is 876. For each task, we finetune Cappy with an AdamW optimizer for 400 steps with a learning rate of 2 × 10 -5 and an effective batch size of 256. We evaluate the performance using the Rouge-L score on every test set, reporting the average score across 45 tests. In this experiment, all variants of FLAN-T5 serve as the backbone LLMs.

We incorporate these approaches in comparison, including: Sampling : Standard token-by-token sampling; Temperature : Sampling every token with a distribution temperature of 0.9; Top-K : Top-k sampling with k=40; Nucleus : Nucleus sampling with top-p=0.95; Beam Search : Beam search with a width of 4; Self-scoring : We collect four generated samples using all the sampling-based decoding strategies above, plus the top sample from beam search 4 , in total 4 × 4 + 1 = 17 samples. With all the samples, self-scoring selects the best one as prediction based on the model log-likelihood; Cappy : conducting sample selection on the same set of samples as in self-scoring, but based on Cappy's

Advantage (Rouge-L)

-20

-10

0

10

20

30

ascii_word_recognition

auto_categorization

auto_debugging

bridging_anaphora_resolution_barqa

chess_state_tracking

codenames

conlang_translation

cryptonite

disfl_qa

few_shot_nlg

gem

gender_inclusive_sentences_german

hindi_question_answering

international_phonetic_alphabet_tr

language_games

linguistic_mappings

list_functions

matrixshapes

minute_mysteries_qa

modified_arithmetic

mult_data_wrangling

object_counting

operators

paragraph_segmentation

parsinlu_reading_comprehension

physics_questions

polish_sequence_labeling

qa_wikidata

repeat_copy_logic

rephrase

scientific_press_release

semantic_parsing_spider

simp_turing_concept

simple_arithmetic_json

simple_text_editing

sufficient_information

tellmewhy

tense

topical_chat

unnatural_in_context_learning

word_sorting

word_unscrambling

scoring. Besides, we include a setting of in-context learning (ICL) by attaching training examples at the beginning of the instruction until the model's sequence length limit is reached. 5

The foundational FLAN-T5 models are frozen, that is, not finetuned. The results are displayed in Table 2. They suggests that Cappy enhances the performance of FLAN-T5 models by a large margin, consistently outperforming the most effective baseline achieved through sample selection using self-scoring of the LLM itself.

As mentioned in Section 3.4, Cappy exhibits flexibility by enabling collaboration with other LLM adaptations. The performance of Cappy working together with finetuning and in-context learning, is presented in Table 2 and Table 3, respectively. The result demonstrate that Cappy keeps boosting performance on top of other adaptations. This can be attributed to the unique downstream knowledge that Cappy acquires from downstream training data. More precisely, while other LLM adaptations predominantly depend on traditional ground truth instructionresponse pairs for learning, Cappy extracts and harnesses the contrastive information by its regression training data constructed with our proposed method.

4.2.1 Analysis on Cappy's Scoring

To further understand Cappy's scoring, we conducted a task-by-task analysis for all the 45 BIG-Bench tasks, comparing the performance between Cappy's scoring with the self-scoring of the multi-task LLM - FLAN-T5-XXL (11B). Figure 6 displays the performance advantage of Cappy over FLANT5-XXL's self-scoring (with negative values indicating Cappy's disadvantage). The results reveal that, for most tasks, Cappy consistently maintains a Rouge-L approximately 5 points higher than the LLM's self-scoring. However, there are 3 tasks on which Cappy exhibits a significant disadvantage: codenames , simple_arithmetic_json , and sufficient_information .

We showcase examples of these 3 tasks in Table 4. Upon examining the codenames examples, we find that the instructions often contain disjointed words with diverse meanings without syntactic

connections. This might presents a considerable challenge to a model's "memory" capability. In the case of simple_arithmetic_json and sufficient_information tasks, the focus is on testing mathematical and commonsense logical abilities. The abilities of momorizing, math, and commonsense, have been demonstrated to be key advantages of LLMs [31, 9], and they are difficult to acquire through downstream training alone. Consequently, it is not surprising that the LLM's self-scoring outperforms Cappy's scoring in these tasks.

4.2.2 Performance Scaling with the Number of Samples

We study the relation between the adaptation performance and number of model generation samples. Specifically, we conduct this on a frozen FLAN-T5XXL model, with three different numbers of samples, including 1 or 4 neucleus samples, or all the 17 samples as described in Section 4.2. Results are shown in Table 5, we can see that as the number of samples increases, Cappy consistently enhances task performance significantly, but the baseline Selfscoring doesn't provide significant performance boost increasing from 4 to 17 samples.

4.2.3 Ablation Study

To verify the importance of the two key components in our proposed methodology, we carry out an ablation study, utilizing FLANT5-XL and -XXL's adaptation on BIG-Bench tasks. Specifically, in the downstream adapation, we ablate the pretraining of Cappy

Table 6: Performance of BIG-Bench adaptation, before and after the ablation of Cappy's pretraining and data augmentation using LLM, numbers in the brackets are the performance drop.

that we described in Sec. 3.2, by using RoBERTa as the model initialization instead of Cappy. We also ablate the data augmentation using LLM, described in Sec. 3.2 during the downstream regression data construction for Cappy.

Table 6 shows the results. The ablation of either pretraining or data augmentation using LLM results in a noticeable decline in performance, thereby highlighting the significance of both these components within our proposed approach. Further, the performance decline scale reveals that the impact of data augmentation is more significant than the pretraining in the downstream adaptation of LLM.

5 Conclusion and Discussions

We deliver a lightweight pretrained scorer, Cappy , to enhance the performance and efficiency of multi-task LLMs. Cappy takes an instruction and a candidate response as input, and produces a score between 0 and 1. The score indicates an estimated correctness of the response with regard to the instruction. A weakly-supervised approach is proposed for the construction of Cappy's pretraining data in regression style. We suggest a candidate selection manner to apply Cappy into practical task-solving. Specifically, Cappy can be utilized either independently or in collaboration with an existing multi-task LLM, serving as an auxiliary component. Our experiments demonstrate that Cappy outperforms much larger multi-task LLMs, on 11 language understanding tasks. Besides, Cappy boosts FLAN-T5's performance on the adaptation to 45 complex tasks drawn from BIG-Bench, without requiring to finetune the LLM. Moreover, Cappy can effectively collaborate with other LLM adaptation strategies such as finetuning and in-context learning, providing further performance enhancement.

Limitations and Future Directions

In Section 4.2.1, we have analyzed certain limitations of Cappy, specifically in the realm of mathematics and complex logical problems. Here, We detail some other limitations and future directions.

Rouge-L Score for Weak Supervision In the construction of Cappy's pretraining data, Rouge-L serves as the metric to evaluate the correctness of model generations. However, Rouge-L may not be the optimal proxy for correctness of model generations, and there is no consensus across the ML and NLP community on the best metric across all the tasks. Presently, Rouge-L is commonly used in multi-task scenarios to report model performance for generation-style tasks [10]. Although Cappy's performance in our experiments demonstrate Rouge-L to be a reasonable design choice, investigating the most suitable metric for multi-task applications remains a highly valuable research direction.

Answer Aggregation Across Multiple Generations The primary contribution of this work is the development and application of the pretrained model, Cappy, for multi-task applications. For sample selection from multiple model generations, we use a straightforward argmax manner that picks the sample with the largest score. However, recent research with nicely designed answer aggregation techniques [28] suggests potential avenues for refining the answer aggregation with Cappy, to further improve the performance of multi-task learning.

Not Handling Tasks Outside the LLM's Expertise The aim of Cappy is to enhance the performance of tasks where the backbone LLM has a fundamental understanding of data inputs. However, Cappy doesn't impact the intrinsic ability of the LLM. It is also worth noticing that many multi-task LLMs, such as FLAN-T5 used in our experiments, already exhibit proficiency across a wide range of domains, encompassing areas like medicine, law, and coding.

Single LLM Adaptation In the experiments of this work, we adapt a single LLM to several domains with Cappy. In the future, Cappy as a pretrained model can potentially be used in other creative ways beyond on single LLMs. For example, Cappy may work as a filter for generations from multiple LLMs. In this case, Cappy plays a role that selects the best LLM regarding a specific input.

Resolving More Human-like Instructions and Leveraging Human Feedback In this work, our focus is multi-task learning where tasks are well-defined. In the future, Cappy can be potentially applied to resolve more "human-like" instructions where the tasks are often vaguely defined. To this end, it would be highly benefitial to leverage costly but high-quality human feedback data, which would require further algorithmic design. This is worth our further exploration.

References

S. H. Bach, V. Sanh, Z. X. Yong, A. Webson, C. Raffel, N. V. Nayak, A. Sharma, T. Kim, M. S. Bari, T. Févry, Z. Alyafeai, M. Dey, A. Santilli, Z. Sun, S. Ben-David, C. Xu, G. Chhablani, H. Wang, J. A. Fries, M. S. Al-shaibani, S. Sharma, U. Thakker, K. Almubarak, X. Tang, M. T.-J. Jiang, and A. M. Rush. Promptsource: An integrated development environment and repository for natural language prompts. ArXiv , abs/2202.01279, 2022.

M. Bakker, M. Chadwick, H. Sheahan, M. Tessler, L. Campbell-Gillingham, J. Balaguer, N. McAleese, A. Glaese, J. Aslanides, M. Botvinick, et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems , 35:38176-38189, 2022.

W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.

H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, D. Valter, S. Narang, G. Mishra, A. W. Yu, V. Zhao, Y. Huang, A. M. Dai, H. Yu, S. Petrov, E. H. hsin Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q. V. Le, and J. Wei. Scaling instruction-finetuned language models. ArXiv , abs/2210.11416, 2022.

Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui. A survey for in-context learning. ArXiv , abs/2301.00234, 2022.

A. Glaese, N. McAleese, M. Tr˛ ebacz, J. Aslanides, V. Firoiu, T. Ewalds, M. Rauh, L. Weidinger, M. Chadwick, P. Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375 , 2022.

J. E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. ArXiv , abs/2106.09685, 2021.

Z. Hu, Y. Lan, L. Wang, W. Xu, E.-P. Lim, R. K.-W. Lee, L. Bing, and S. Poria. Llmadapters: An adapter family for parameter-efficient fine-tuning of large language models. ArXiv , abs/2304.01933, 2023.

J. Huang and K. C.-C. Chang. Towards reasoning in large language models: A survey. ArXiv , abs/2212.10403, 2022.

S. Iyer, X. Lin, R. Pasunuru, T. Mihaylov, D. Simig, P. Yu, K. Shuster, T. Wang, Q. Liu, P. S. Koura, X. Li, B. O'Horo, G. Pereyra, J. Wang, C. Dewan, A. Celikyilmaz, L. Zettlemoyer, and V. Stoyanov. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. ArXiv , abs/2212.12017, 2022.

N. Jouppi, G. Kurian, S. Li, P. Ma, R. Nagarajan, L. Nai, N. Patil, S. Subramanian, A. Swing, B. Towles, et al. Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings. In Proceedings of the 50th Annual International Symposium on Computer Architecture , pages 1-14, 2023.

B. Kratzwald, A. Eigenmann, and S. Feuerriegel. Rankqa: Neural question answering with answer re-ranking. ArXiv , abs/1906.03008, 2019.

N. Lambert, L. Castricato, L. von Werra, and A. Havrilla. Illustrating reinforcement learning from human feedback (rlhf). Hugging Face Blog , 2022. https://huggingface.co/blog/rlhf.

J. Lee, S. Yun, H. Kim, M. Ko, and J. Kang. Ranking paragraphs for improving answer recall in open-domain question answering. ArXiv , abs/1810.00494, 2018.

X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , abs/2101.00190, 2021.

B. Y. Lin, K. Tan, C. Miller, B. Tian, and X. Ren. Unsupervised cross-task generalization via retrieval augmentation. NeurIPS , 2022.

C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out , pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.

P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys , 55:1 - 35, 2021.

T.-Y. Liu et al. Learning to rank for information retrieval. Foundations and Trends® in Information Retrieval , 3(3):225-331, 2009.

Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv , abs/1907.11692, 2019.

OpenAI. Gpt-4 technical report. ArXiv , abs/2303.08774, 2023.

M. Ravaut, S. R. Joty, and N. F. Chen. Summareranker: A multi-task mixture-of-experts re-ranking framework for abstractive summarization. In Annual Meeting of the Association for Computational Linguistics , 2022.

V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Scao, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. Sharma, E. Szczechla, T. Kim, G. Chhablani, N. V. Nayak, D. Datta, J. Chang, M. T.-J. Jiang, H. Wang, M. Manica, S. Shen, Z. X. Yong, H. Pandey, R. Bawden, T. Wang, T. Neeraj, J. Rozen, A. Sharma, A. Santilli, T. Févry, J. A. Fries, R. Teehan, S. R. Biderman, L. Gao, T. Bers, T. Wolf, and A. M. Rush. Multitask prompted training enables zero-shot task generalization. ICLR , 2022.

D. Sileo. tasksource: Structured dataset preprocessing annotations for frictionless extreme multi-task learning and evaluation. arXiv preprint arXiv:2301.05948 , 2023.

A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, A. Kluska, A. Lewkowycz, A. Agarwal, A. Power, A. Ray, A. Warstadt, A. W. Kocurek, A. Safaya, A. Tazarv, A. Xiang, A. Parrish, A. Nie, A. Hussain, A. Askell, A. Dsouza, A. A. Rahane, A. S. Iyer, A. Andreassen, A. Santilli, A. Stuhlmuller, A. M. Dai, A. D. La, A. K. Lampinen, A. Zou, A. Jiang, A. Chen, A. Vuong, A. Gupta, A. Gottardi, A. Norelli, A. Venkatesh, A. Gholamidavoodi, A. Tabassum, A. Menezes, A. Kirubarajan, A. Mullokandov, A. Sabharwal, A. Herrick, A. Efrat, A. Erdem, A. Karakacs, B. R. Roberts, B. S. Loe, B. Zoph, B. Bojanowski, B. Ozyurt, B. Hedayatnia, B. Neyshabur, B. Inden, B. Stein, B. Ekmekci, B. Y. Lin, B. S. Howald, C. Diao, C. Dour, C. Stinson, C. Argueta, C. F. Ram'irez, C. Singh, C. Rathkopf, C. Meng, C. Baral, C. Wu, C. Callison-Burch, C. Waites, C. Voigt, C. D. Manning, C. Potts, C. T. Ramirez, C. Rivera, C. Siro, C. Raffel, C. Ashcraft, C. Garbacea, D. Sileo, D. H. Garrette, D. Hendrycks, D. Kilman, D. Roth, D. Freeman, D. Khashabi, D. Levy, D. Gonz'alez, D. Hernandez, D. Chen, D. Ippolito, D. Gilboa, D. Dohan, D. Drakard, D. Jurgens, D. Datta, D. Ganguli, D. Emelin, D. Kleyko, D. Yuret, D. Chen, D. Tam, D. Hupkes, D. Misra, D. Buzan, D. C. Mollo, D. Yang, D.-H. Lee, E. Shutova, E. D. Cubuk, E. Segal, E. Hagerman, E. Barnes, E. P. Donoway, E. Pavlick, E. Rodolà, E. F. Lam, E. Chu, E. Tang, E. Erdem, E. Chang, E. A. Chi, E. Dyer, E. J. Jerzak, E. Kim, E. E. Manyasi, E. Zheltonozhskii, F. Xia, F. Siar, F. Mart'inez-Plumed, F. Happ'e, F. Chollet, F. Rong, G. Mishra, G. I. Winata, G. de Melo, G. Kruszewski, G. Parascandolo, G. Mariani, G. Wang, G. Jaimovitch-L'opez, G. Betz, G. Gur-Ari, H. Galijasevic, H. S. Kim, H. Rashkin, H. Hajishirzi, H. Mehta, H. Bogar,

H. Shevlin, H. Schütze, H. Yakura, H. Zhang, H. Wong, I. A.-S. Ng, I. Noble, J. Jumelet, J. Geissinger, J. Kernion, J. Hilton, J. Lee, J. F. Fisac, J. B. Simon, J. Koppel, J. Zheng, J. Zou, J. Koco'n, J. Thompson, J. Kaplan, J. Radom, J. N. Sohl-Dickstein, J. Phang, J. Wei, J. Yosinski, J. Novikova, J. Bosscher, J. Marsh, J. Kim, J. Taal, J. Engel, J. O. Alabi, J. Xu, J. Song, J. Tang, J. W. Waweru, J. Burden, J. Miller, J. U. Balis, J. Berant, J. Frohberg, J. Rozen, J. Hernández-Orallo, J. Boudeman, J. Jones, J. B. Tenenbaum, J. S. Rule, J. Chua, K. Kanclerz, K. Livescu, K. Krauth, K. Gopalakrishnan, K. Ignatyeva, K. Markert, K. D. Dhole, K. Gimpel, K. O. Omondi, K. W. Mathewson, K. Chiafullo, K. Shkaruta, K. Shridhar, K. McDonell, K. Richardson, L. Reynolds, L. Gao, L. Zhang, L. Dugan, L. Qin, L. Contreras-Ochando, L.-P. Morency, L. Moschella, L. Lam, L. Noble, L. Schmidt, L. He, L. O. Col'on, L. Metz, L. K. cSenel, M. Bosma, M. Sap, M. ter Hoeve, M. Andrea, M. S. Farooqi, M. Faruqui, M. Mazeika, M. Baturan, M. Marelli, M. Maru, M. Quintana, M. Tolkiehn, M. Giulianelli, M. Lewis, M. Potthast, M. Leavitt, M. Hagen, M. Schubert, M. Baitemirova, M. Arnaud, M. A. McElrath, M. A. Yee, M. Cohen, M. Gu, M. I. Ivanitskiy, M. Starritt, M. Strube, M. Swkedrowski, M. Bevilacqua, M. Yasunaga, M. Kale, M. Cain, M. Xu, M. Suzgun, M. Tiwari, M. Bansal, M. Aminnaseri, M. Geva, M. Gheini, T. MukundVarma, N. Peng, N. Chi, N. Lee, N. G.-A. Krakover, N. Cameron, N. S. Roberts, N. Doiron, N. Nangia, N. Deckers, N. Muennighoff, N. S. Keskar, N. Iyer, N. Constant, N. Fiedel, N. Wen, O. Zhang, O. Agha, O. Elbaghdadi, O. Levy, O. Evans, P. A. M. Casares, P. Doshi, P. Fung, P. P. Liang, P. Vicol, P. Alipoormolabashi, P. Liao, P. Liang, P. W. Chang, P. Eckersley, P. M. Htut, P.-B. Hwang, P. Milkowski, P. S. Patil, P. Pezeshkpour, P. Oli, Q. Mei, Q. LYU, Q. Chen, R. Banjade, R. E. Rudolph, R. Gabriel, R. Habacker, R. R. Delgado, R. Millière, R. Garg, R. Barnes, R. A. Saurous, R. Arakawa, R. Raymaekers, R. Frank, R. Sikand, R. Novak, R. Sitelew, R. L. Bras, R. Liu, R. Jacobs, R. Zhang, R. Salakhutdinov, R. Chi, R. Lee, R. Stovall, R. Teehan, R. Yang, S. J. Singh, S. M. Mohammad, S. Anand, S. Dillavou, S. Shleifer, S. Wiseman, S. Gruetter, S. Bowman, S. S. Schoenholz, S. Han, S. Kwatra, S. A. Rous, S. Ghazarian, S. Ghosh, S. Casey, S. Bischoff, S. Gehrmann, S. Schuster, S. Sadeghi, S. S. Hamdan, S. Zhou, S. Srivastava, S. Shi, S. Singh, S. Asaadi, S. S. Gu, S. Pachchigar, S. Toshniwal, S. Upadhyay, S. Debnath, S. Shakeri, S. Thormeyer, S. Melzi, S. Reddy, S. P. Makini, S. hwan Lee, S. B. Torene, S. Hatwar, S. Dehaene, S. Divic, S. Ermon, S. R. Biderman, S. C. Lin, S. Prasad, S. T. Piantadosi, S. M. Shieber, S. Misherghi, S. Kiritchenko, S. Mishra, T. Linzen, T. Schuster, T. Li, T. Yu, T. A. Ali, T. Hashimoto, T.-L. Wu, T. Desbordes, T. Rothschild, T. Phan, T. Wang, T. Nkinyili, T. Schick, T. N. Kornev, T. Telleen-Lawton, T. Tunduny, T. Gerstenberg, T. Chang, T. Neeraj, T. Khot, T. O. Shultz, U. Shaham, V. Misra, V. Demberg, V. Nyamai, V. Raunak, V. V. Ramasesh, V. U. Prabhu, V. Padmakumar, V. Srikumar, W. Fedus, W. Saunders, W. Zhang, W. Vossen, X. Ren, X. Tong, X. Wu, X. Shen, Y. Yaghoobzadeh, Y. Lakretz, Y. Song, Y. Bahri, Y. J. Choi, Y. Yang, Y. Hao, Y. Chen, Y. Belinkov, Y. Hou, Y. Hou, Y. Bai, Z. Seid, Z. Xinran, Z. Zhao, Z. F. Wang, Z. J. Wang, Z. Wang, Z. Wu, S. Singh, and U. Shaham. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. ArXiv , abs/2206.04615, 2022.

N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008-3021, 2020.

B. Tan, Y. Zhu, L. Liu, H. Wang, Y. Zhuang, J. Chen, E. Xing, and Z. Hu. Redco: A Lightweight Tool to Automate Distributed Training of LLMs on Any GPU/TPUs. arXiv e-prints , page arXiv:2310.16355, Oct. 2023.

X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 , 2022.

Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Arunkumar, A. Ashok, A. S. Dhanasekaran, A. Naik, D. Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. EMNLP , 2022.

J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. ICLR , 2022.

Z. Yuan, H. Yuan, C. Tan, W. Wang, and S. Huang. How well do large language models perform in arithmetic tasks? ArXiv , abs/2304.02015, 2023.

Y. Zha, Y. Yang, R. Li, and Z. Hu. Text alignment is an efficient unified model for massive nlp tasks. arXiv preprint arXiv:2307.02729 , 2023.

L. Zheng, Y. Sheng, W.-L. Chiang, H. Zhang, J. E. Gonzalez, and I. Stoica. Chatbot arena: Benchmarking llms in the wild with elo ratings, 2023.

A Comparison of Cappy with other adaptation methods

Cappy doesn't mean to beat other adaptation methods such as finetuning, in-context learning, and prompt tuning. Compared with these approaches, adaptation with Cappy is an alternative that is free from the constraints associated with storage, device memory, model accessibility, and training sample limitations. Moreover, Cappy makes no assumption about the backbone model, enabling seamless integration with other adaptations and providing steady and significant performance promotion at little additional cost. To illustrate this, we include an experiment below that combines Cappy with in-context learning and prompt-tuning, respectively.

Specifically, we add more comparison with prompt tuning and in-context learning in our BIG-Bench adaptation experiment with FLAN-T5-Large as the backbone model. For prompt tuning, we apply prefix tuning, which is usually considered suitable for generation tasks, with 20 virtual tokens. As demonstrated by the results presented in Table 7, Cappy offers further performance boost on top of both in-context learning and prompt tuning.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose training a small(er) classifier model (‘Cappy’) to predict an answer given a set of possibilities (if the answer set is closed) or multiple generations from an LLM (in the case of open-ended/generative tasks). The model is trained on data from T0, using LLMs to generate partially correct responses alongside positive and negative examples. They evaluate performance on T0 evaluation tasks (classification) and big-bench (generative) and find that Cappy outperforms same-size models, and consistently provides improvements over a base model when picking from generations.\n\nSTRENGTHS:\nThe proposed approach is straightforward, using a trained classifier as a reranker for model generations. The improvements seem significant, and the fact that a classifier outperforms a similar-size generative model is interesting to see.\n\nThe data gathering scheme for Cappy’s training data is interesting, and using rouge-l as a proxy for gold labels is interesting and seems to work well in ablations.\n\nThe writing is clear and the paper is easy to follow.\n\nWEAKNESSES:\n**Comparisons to Best-of-N** - I’m somewhat concerned about the novelty of this work. The proposed method is very similar to the best-of-N sampling originally proposed in [1] (“Best-of-N sampling”) and discussed (sometimes under ‘reranking’) in [2] [3], [9], inter alia. These methods work by sampling multiple LM generations and then picking the answer with the highest reward as chosen by a trained reward model.\nThe three core differences between these prior works and Cappy I can detect are:\n1. Different use in classification - Cappy directly uses the set of possible answers for classification tasks, while these prior works do not.\n2. Training data - Cappy is trained on prompt source-based data with augmentation, which is interesting and seems to help performance, while (afaik) best-of-n uses trained reward models, which are often trained using preference datasets.\n3. Application - best-of-n approaches are often applied and focussed on how they aid with aligning to human preferences, while the focus in this work is on improving over benchmarks.\n\nWhile these are interesting differences, the authors do not compare to these prior works and explicitly make these comparisons. At the very least, I would expect these methods to be discussed in a related work section, and the decisions made in training Cappy contrasted to them. Ideally, they would be integrated as baselines/ablations to justify differences, highlighting the merits of Cappy over these approaches and covering what is needed to adapt these approaches to Cappy’s chosen setting.\n\n**Missing prior work for reranking** - Cappy effectively is serving as a reranker in the generative case. There is a long history of work in answer reranking, that would be useful to discuss as related work (e.g., [4] for multi-choice qa, [5,7] for summarisation, [5,6] for open-domain qa). \n\n**Further baselines and ablations** - There are other methods for training and applying rerankers, including contrastive training (as used to train a reranker in [7], with rouge as a proxy for human judgements), or encoding multiple samples at once as opposed to one-at-a-time (as done in [4]). It would be useful to test these changes, especially given the authors suggest that making use of contrastive information is helpful for Cappy in line 202. It would also be interesting to compare against self-consistency [8], another approach for improving models that makes use of multiple generations.\n\nGiven all of these things, I am inclined to recommend rejection for this paper in its current state. Further changes differentiating this approach against best-of-n sampling and further experiments exploring the full space of answer reranking would greatly improve this work, and may sway my opinion, but require significant extra work. I think the overall idea is interesting and evidently effective, but requires further work to make a substantial contribution. I hope the authors make these changes and are successful in the future!\n\n[1] Stiennon et al. (2020). Learning to summarize from human feedback. NeurIPS. https://arxiv.org/pdf/2009.01325.pdf\n\n[2] Bakker et al. (2022). Fine-tuning language models to find agreement among humans with diverse preferences. ArXiv. https://arxiv.org/pdf/2211.15006.pdf\n\n[3] Glaese et al. (2022). Improving alignment of dialogue agents via targeted human judgements. ArXiv. https://arxiv.org/abs/2209.14375\n\n[4] Kratzwald et al. (2019). RankQA: Neural Question Answering with Answer Re-Ranking. ACL. http://aclanthology.lst.uni-saarland.de/P19-1611.pdf\n\n[5] Revaut et al. (2022). SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization. ACL. https://aclanthology.org/2022.acl-long.309.pdf\n\n[6] Lee et al. (2018). Ranking Paragraphs for Improving Answer Recall in Open-Domain Question Answering. EMNLP. https://aclanthology.org/D18-1053.pdf\n\n[7] Liu et al. (2021) SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization. ACL. https://aclanthology.org/2021.acl-short.135/\n\n[8] Wang et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR. https://arxiv.org/pdf/2203.11171.pdf\n\n[9] Dubois et al. (2023). AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. ArXiv. https://arxiv.org/abs/2305.14387\n\nEdit: The authors have responded with some of the experiments and comparisons I asked for, with positive results, and so I'm happy to raise my score.\n\nQUESTIONS:\n1. How is cappy different to best-of-n techniques proposed in the papers mentioned above? Do you think using preference data already available could help or augment your data collection strategy?\n2. Did you explore contrastive objectives in pretraining cappy? How about encoding multiple answers at once?\n3. Do you have further details on the distribution of rouge-l scores you used in the data augmentation setup? It would be interesting to see this, and do you think there are any limitations in using rouge-l against something like cosine similarity for this?\n\nLIMITATIONS:\nI think the authors effectively discuss limitations. They miss some elements such as Cappy requiring multiple generations (thus increasing its inference cost over using a single-generation) and complicating the model pipeline, but cover Cappy’s weakness in the realm of complex logical problem-solving / mathematics, and its current reliance on supervised datasets.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

comments: SUMMARY:
This paper introduces an auxiliary module called Cappy, which aims to enhance the performance of large language models. Cappy operates by taking a task instruction and a proposed response as input, and it estimates the quality score for the response. The training process involves creating a dataset that combines correct and mismatched answers with various instructions, and the Cappy scorer is then trained using a regression objective. In downstream adaptation, the small scorer, Cappy, is fine-tuned on a small dataset that follows the same approach as the training set. During testing, predictions are determined by selecting the answer choice with the highest score. In essence, Cappy acts as a valuable filter that enhances the overall quality of generated responses.

STRENGTHS:
The idea of leveraging a lightweight scorer as a ‘’filter’’ is interesting. Fine-tuning the filters on downstream tasks can be efficient and practical.

WEAKNESSES:
Cappy itself is only a filter for in-domain responses and can not impact the intrinsic ability of a pre-trained LLM. This method cannot handle tasks outside the LLM's expertise, emphasizing the need for a sufficiently general and powerful LLM. Alternatively, fine-tuning LLMs, as shown in Table 3, may be necessary but costly. By comparing able 2 and Table 3, fine-tuning is still very essential for task adaptation.

QUESTIONS:
Please refer to the weaknesses.

LIMITATIONS:
No additional limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAdapting SotA LLMs to novel tasks is difficult given their extreme size, yet is generally more effective than in-context learning alone. To address this, the authors propose “Cappy”, a scoring model (leveraging RoBERTa as a backbone) which is trained to score pairs of (instruction, response) for any task instruction and corresponding response. Cappy is pre-trained on many examples of ((instruction, response) -> score); after pre-training, Cappy can be easily fine-tuned (because of it’s small size) to a given task and then used as an augmentation to LLM generation by acting as a ranking mechanism atop several LLM generated responses.\n\nTo create pre-training data for Cappy, the authors use the PromptSource dataset, training the model to predict 1 for correct pairs of (prompt, response) and 0 for mismatched pairs. Additionally, the authors generate additional responses using a large LLM (e.g. FLAN) and use the Rouge-L between the generated response and gold response as the regression target for the pair (prompt, generated response). To test the validity of Cappy, the authors evaluate Cappy in isolation, with no additional fine-tuning, on 11 held-out classification tasks from PromptSource. They show that Cappy consistently ranks the correct answer higher than many instruction-tuned LLMs, including OPT-IML-175B.\n\nNext, the authors evaluate Cappy on LLM augmentation using the BIG-Bench benchmark. For each task, Cappy is first fine-tuned, then used as a scoring function for multiple generations from a frozen FLAN-T5 LLM. Cappy consistently outperforms the proposed sampling methods, most notably improving upon the LLMs own self-scoring and In-Context Learning. Moreover, the authors show that Cappy can improve upon a FLAN-T5 model’s performance even when the LLM parameters are fine-tuned as well.\n\nFinally, the authors present an ablation study of their proposed training strategy for Cappy, showing that both the overall pre-training strategy and using Rouge-L regression targets from LLM generations are important to Cappy’s success.\n\nSTRENGTHS:\n- Cappy represents a light-weight option to adapt LLMs to a given target task which requires no updating of parameters to the LLM nor any back-propagation through the LLM whatsoever (e.g. this is not true for adapter layers or prompt tuning).\n- Moreover, even without adapting to a given task, Cappy presents some benefits over generic LLM sampling methods for task-specific predictions.\n-  Additionally, Cappy can ostensibly be added on top of any LLM without additional training (i.e. train once, use anywhere).\n- Cappy is a very simple idea, and is easy to understand, yet the usage of a scorer in this capacity, as well as the training and data augmentation scheme, is novel to the best of my knowledge.\n- The ablation study is helpful in demonstrating the benefits of the proposed pre-training and data-augmentation scheme.\n\nWEAKNESSES:\n- Cappy is not compared to other parameter-efficient adaptation methods. While there is some justification given, because other methods require storing forward activations and back-propagating through LLM parameters, it nevertheless makes judging the effectiveness of Cappy difficult.\n- Moreover, Cappy’s improvements over a frozen LLM are comparatively small, e.g. while Cappy improves Flan-T5 Large performance by ~7%, fine-tuning alone increases performance by ~22%. It’s therefore unclear how significant Cappy’s benefit is without comparison to other adaptation methods.\n- In the zero-shot experiments (4.1) it is claimed that the improved performance of Cappy over much larger LLMs can be attributed to Cappy’s scoring-based learning strategy. However, T0 outperforms Cappy and T0 is significantly smaller than the OPT models; moreover, T0 is also trained on PromptSource only, similar to Cappy, and unlike the other LLMs. Thus, it seems that at least some of Cappy’s performance benefits compared to other LLMs may arise from domain mismatch between training and test domains, rather than the proposed scoring function.\n- While Cappy does only require a small number of parameters to train, inference requires a number of decoder forward passes to generate multiple candidate responses from an LLM (it's not clear how Cappy scales with the number of samples provided, also); to my knowledge, other adaptation methods such as Adapter Layers do not have the same limitation. So while it saves computation on one end, it does slightly increase computation on another end, and exploring this trade-off would be useful.\n\nQUESTIONS:\n- I think it would be really helpful to understand how Cappy compares to other parameter efficient adaptation methods. Do you have any experiments that indicate that Cappy is e.g. comparable to prompt-tuning in terms of it's benefits?\n- Can you provide an analysis of how much additional computation is required to sample many generations for Cappy to score?\n- Also, do you know how Cappy's performance scales with the number of generated samples? e.g. does it's performance flatten out after only 4 samples, or can it's performance increase significantly if several generations are considered?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors tackle the challenge of computational requirements and memory constraints in fine-tuning Large Language Models (LLMs) by introducing an innovative approach that enhances LLM performance without the need for backpropagation through the LLM or access to its parameters. They propose Cappy, a pre-trained scorer that evaluates texts generated by LLMs based on downstream instructions and assigns them a score ranging from 0 to 1.\n\nTo create Cappy, the authors curate a diverse collection of 39 samples from the PromptSource dataset, including good (ground truth), bad (random), and intermediate examples. The intermediate examples are generated using top-k and top-p sampling from BART0 and T0-3B models. Cappy assigns a score to each example based on the ROUGE-L metric compared to the ground truth response. This dataset is used to create a regression task and train a RoBERTa model that serves as the foundation for Cappy.\n\nThe authors demonstrate improved accuracy across 11 classification held-out tasks from PromptSource compared to OPT, OPT-IML, T0, and BART0. They also showcase Cappy's effectiveness as an LLM booster for downstream adaptation on the BIG-Bench generative tasks, outperforming other selection strategies and achieving higher ROUGE-L scores on both frozen and fine-tuned FLAN-T5 models.\n\nAdditionally, the paper discusses scenarios where Cappy performs less favorably than the \"self-scoring\" strategy on certain BIG-Bench tasks, suggesting that the lack of \"memory\" may contribute to this performance difference. Furthermore, the authors conduct an ablation study, revealing that data augmentation using LLMs is more crucial for improved performance than pre-training Cappy, although pre-training still offers benefits.\n\nOverall, the paper presents an innovative approach, Cappy, for boosting LLM performance without backpropagation or parameter access. The experiments showcase its superiority in classification tasks and downstream adaptation, while highlighting the importance of data augmentation and the potential limitations related to memory.\n\nSTRENGTHS:\n- The idea of an auxiliary performance booster is an intriguing concept. The demonstrated improvements over the selected baselines highlight the effectiveness of this approach in enhancing language model performance.\n- The utilization of LLM generations for data augmentation and the regression-based evaluation using ROUGE-L scores is a novel and innovative methodology employed in this work.\n- Cappy introduces a unique integration of samples from LLMs and ground-truth information, leveraging correctness scores to enhance the quality of text generations. This ability to distinguish between good and poor responses is a novel idea that has the potential to enhance the performance of all language models.\n- Despite having significantly fewer parameters compared to large language models used in zero-shot baselines, pre-trained Cappy achieves comparable performance. This contribution is valuable as it demonstrates that Cappy can achieve performance on par with larger models, offering a more efficient and resource-friendly solution.\n- The versatility of Cappy is showcased by its potential for downstream adaptation and further fine-tuning to improve task-specific performance. This adaptability highlights the broad applicability and utility of Cappy as a solution in various contexts.\n\nWEAKNESSES:\n- The performance of Cappy, trained on BART0 and T0 generations, does not consistently outperform T0 and OPT-IML. While the zero-shot performance is better than BART0 (a relatively smaller model), it falls short compared to T0 in most cases. The evidence supporting the claim of boosting language model performance is not sufficiently strong, and the improvements are not consistently observed.\n- The training of Cappy to optimize ROUGE-L scores against ground truth data raises concerns about the choice of this metric. ROUGE-L may not necessarily be the most optimal metric for evaluation, as alternative responses with low ROUGE-L scores may be valid and informative. Including additional metrics for evaluation would provide a more comprehensive understanding of the overall increase in performance.\n- The discussion on the tasks in BIG-Bench where Cappy performs worse than the \"self-scoring\" strategy is insufficient. Given the expectations for Cappy to perform at least as well, if not better, further analysis and exploration of these instances would strengthen the paper and provide a clearer understanding of the limitations of Cappy's performance.\n\nQUESTIONS:\n- Could you provide an example illustrating how the limitation of instruction lengths affects in-context learning? Adding a specific example would enhance the understanding of this limitation and its implications.\n- In the context of downstream adaptation, it is not entirely clear whether the same LLM that is being augmented is used to generate the synthetic data. Clarifying this aspect in the paper would help readers better comprehend the process of downstream adaptation and the relationship between the augmented data and the LLM.\n- The reasoning behind the requirement for more \"memory\" in tasks like \"sufficient_information\" is unclear. It would be beneficial to provide a more detailed explanation either in the paper or in the appendix to shed light on the specific aspects that require increased memory in such tasks.\n- The results show that removing the Cappy pre-training step does not significantly impact the downstream score on BIG-Bench, implying that fine-tuning with augmented data from LLMs is sufficient. Have you explored the performance of RoBERTa initialization with data augmentation using LLMs? It would be insightful to investigate and report the results of this comparison to further validate the efficacy of the proposed approach.\n\nMinor Issues:\n- Line 186: PromprSource -> PromptSource\n- Line 285: 360 -> 360 M\n- Line 222: freezed -> frozen - This issue exists at several places in the paper.\n\nLIMITATIONS:\nThe authors thoroughly discuss various limitations, including mathematical complexities, reliance on supervised datasets, absence of multi-lingual extensions, and other important concerns. However, it would be beneficial to further address limitations related to the reliance on generations from existing LLMs and potential biases that may arise as a result. Additionally, the use of ROUGE-L as the sole metric for regression and evaluation may not provide a comprehensive and accurate portrayal of the overall performance. It would be valuable to acknowledge this limitation and discuss the potential impact on the interpretation and generalization of the results.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}