Cappy: Outperforming and Boosting Large Multi-Task LMs with a Small Scorer
Authors: Bowen Tan
Yun Zhu
Lijuan Liu
Eric Xing
Zhiting Hu
Jindong Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: multi-task
large language models
pretrain model
TL;DR: we introduce a pretrained small scorer, Cappy, designed to enhance the performance and efficiency of multi-task LLMs.

Abstract: Large language models (LLMs) such as T0, FLAN, and OPT-IML excel in multi-tasking under a unified instruction-following paradigm, where they also exhibit remarkable generalization abilities to unseen tasks. Despite their impressive performance, these LLMs, with sizes ranging from several billion to hundreds of billions of parameters, demand substantial computational resources, making their training and inference expensive and inefficient. Furthermore, adapting these models to downstream applications, particularly complex tasks, is often unfeasible due to the extensive hardware requirements for finetuning, even when utilizing parameter-efficient approaches such as prompt tuning. Additionally, the most powerful multi-task LLMs, such as OPT-IML-175B and FLAN-PaLM-540B, are not publicly accessible, severely limiting their customization potential. To address these challenges, we introduce a pretrained small scorer, \textit{Cappy}, designed to enhance the performance and efficiency of multi-task LLMs. With merely 360 million parameters, Cappy functions either independently on classification tasks or serve as an auxiliary component for LLMs, boosting their performance. Moreover, Cappy enables efficiently integrating downstream supervision without requiring LLM finetuning nor the access to their parameters. Our experiments demonstrate that, when working independently on 11 language understanding tasks from PromptSource, Cappy outperforms LLMs that are several orders of magnitude larger. Besides, on 45 complex tasks from BIG-Bench, Cappy boosts the performance of the advanced multi-task LLM, FLAN-T5, by a large margin. Furthermore, Cappy is flexible to cooperate with other LLM adaptations, including finetuning and in-context learning, offering additional performance enhancement.

Cappy: Outperforming and Boosting Large ea
Multi-Tast LMs with a Small Scorer
Bowen Tan'; Yun Zhu?, Lijuan Liu", Eric Xing!.3,5, Zhiting Hu Jindong Chen
Caanegie Mellon University. *Google Research, *Petuum Inc., "UC San Diego,
A MMohamed bin Zayed University of Artificial Intelligence
{btan2, epxing)@andren cmu. edu, zhh01 1..00999csddddd.. {yunzhu, lijuanliu, jdchen)@google.. com
Abstract
Large language models (LLMs) such as TO, FLAN, and OPT-IML, excel in multi- tasking under unified instruction-following paradigm, where they also exhibit remarkable generalization abilities to unseen tasks. Despite their impressive per- formance, these LLMs, with sizes ranging from several billion hundreds of billions of parameters, demand substantial computational resources, making their training and inference expensive and inefficient. Furthermore, adapting these mod- els to downstream applications, particularly complex tasks, is often unfcasible due to the extensive hardware requirements for finetuning, even when utilizing parameter-efficient approaches such prompt tuning. Additionally, the most powerful multi-task LLMs, such as OPT-IML- and BLLAN-PaLM-5444 not publicly accessible, severely limiting their customization potential. To address these challenges, we introduce a pretrained small scorer, Cappy, designed to en- hance the performance and efficiency of multi-task LLMs. With merely 360 million parameters, Cappy functions either independently on classification tasks or serve as auxiliary component for LLMs, boosting their performance. Moreover, Cappy enables efficiently integrating downstream supervision without requiring LLM fine- tuning nor the access to their parameters. Our experiments demonstrate that, when working independently on language understanding tasks from PromptSource, Cappy outperforms LLMs that are several orders of magnitude larger. Besides, 45 complex tasks from BIG GeGBBecch, Cappy boosts the performance of the advanced multi-task LLM, FLAN-TS, by large margin. Furthermore, Cappy is flexible to cooperate with other LLM adaptations, including finetuning and in-context learning, offering additional performance enhancement,
are
Introduction
Large language models (LLMs) have led to a new paradigm that seeks to unify various natural language processing (NLP) tasks within an instruction-oolloinnn framework. This paradigm is exemplified by the recent multi-task LLMs, such as TO 233, FLAN [30] 4. and OPT-IML. (10]. These models trained with data from many tasks: for each task, following task-specific template, each labeled example is converted into an instruction (e.g., "Put the concepts together to form sentence: ski, mountain, skier. and corresponding response (e.g., "Skier skis down the mountain"). Such (instruction, response) pairs are then used to train the LLM, resulting in conditional generation model that takes input of data example as instruction
'Work done during an intershhi at Google, "Code and model available at co/ co/btan2//capy--arg respectively.
and https: /ttgg/gggggggcceeeee//
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

56
35-
54
30-
Accuracy
Rouge-L
52-
50
20:
Nadeus
48
OPT
Beam Search
15-
Selfacoring
46
Cappy
Caperuacee
10
10a
10
10a
10
10
10a
Model Size (log scale)
Model Size (log scale)
Figure Cappy outperforms multi-task LLMs: The overall accuracy averaged over 11 test tasks from PromptSource. Every dashed line connects different sizes of the same model. Lines positioned more towards the upper left denote models that are more efficient and yield superior performance.
Figure 2: Cappy boosts multi-task LLMs: The averaged Rouge-L score over 45 complex tasks within BIG-Bench. Every dashed line represents an approach working on LLMs of various sizes. Self-scoring refers to using the cross-entropy of LLM to select responses.
and generates a response. Moreover, these multi-task LLMs have exhibited remarkable task-wise generalization capabilities. That is, they can address unseen tasks by understanding and solving brand-new instructions.
Due to the complexity of understanding and resolving various tasks solely via instructions, the sizes of these multi-task LLMs typically span from several billion parameters hundreds billions, such as TO- 1B 23 and OPT-IML-177 -15B [10] As result, operating such sizable models poses significant challenges to majority LLM users, because they demand considerable computational power and impose substantial requirements on the memory capacities of GPUs/TPUs, making their training and inference expensive and inefficient.
In practical applications, harnessing a single multi-task LLM to manage all conceivable tasks in zero-shot manner remains challenging, particularly when dealing with complex tasks, personalized tasks and those that cannot be succinctly defined using instructions. On the other hand, the size of downstream training data is usually insufficient well train model without incorporating rich prior knowledge. Hence, it is long desired to adapt LLMs with downstream supervision. Yet, the adaptation process presents three significant obstacles: the extensive storage to maintain a unique LLM copy for each downstream task; the considerable memory demands GPUs/TPUs; and the unavailability of the most powerful multi-task L.LMs, such as OPT-IML-I75B [T0 and FLAN--PLMM 540B 4 Certain parameter-ee efficiient tuning strategies, including prompt tuning [18] and adapters (8), substantially diminish storage requirements, but they still perform back pao--propggttion through the LLM parameters during the tuning process, thereby their memory demands keep high. Additionally, some in-context learning techniques (S circumvent parameter tuning by integrating a limited number of supervised examples into the instruction. However, these techniques are constrained by the model's maximum input length, which permits only a few samples to guide task resolution.
In this work, propose novel approach to enhance the performance and efficiency of multi-task LLMs. Specifically, we introduce a lightweight pretrained scorer, Cappy, based on continual pretraining top of RoBERTa [20]. with merely 360 million parameters. Cappy takes in an instruction and candidate response as input, and produces a score between 0 and 1, indicating an estimated correctness of the response with respect the instruction. Naturally, formulate Cappy's 's pretraining as a regression problem. This anticipates training data in the form of finstruction response) pairs that correspond to various correctness score annotations. To generate the desired training data from multiple pretrain datasets that solely contain instructions and their ground truth responses, we propose
approach with data augmentation through the of existing multi-task LLMs. As a result, we obtain large and effective regression pretraining dataset with diverse correctness score annotations ranging from 0 to 1.
To apply Cappy to practical problem-solving scenarios, we suggest an intuitive approach in an candidate selection style. Specifically, Cappy works independently classification tasks by selecting

---

Score: 0.86
Downstream training data
Adaptation through LLM Expensive
Cappy
Demanding to hardware Accessibility issue
Adaptation Cappy
Efficient
amount of training datta whatever LUMS
The picture appeared on the wall of Pounciana rellnnd store Whymaak Avenue (..] Hov would you rephrass in words?
*g, FLAN, OPT:MM.
Enhanced prediction
Cappy
argmax
Response:
samples
Graffii artist Bankey beloved behind
Figure 3: (left) The modeling of Cappy. (right) Illustration of Cappy's application in enhancing multi-task task LLMs, and the comparison between downstream adaptation through Cappy and approaches that rely on LLM's parameters, such as finetuning and prompt tuning.
the answer choice that produces the highest score. Furthermore, beyond the standalone use, Cappy serves as an auxiliary component of existing multi-task LLMs, choosing the most appropriate output from a set of candidates generated by the LLM. In this case, Cappy allows for effective and efficient adaptation to complex tasks through the incorporation of downstream supervision, without requiring finetuning the multi-task LLM or the access to its parameters. Remarkably, Cappy exhibits flexibility in collaboration with other LLM adaptations, such as finetuning and in-context learning.
We validate Cappy through an extensive suite of held-out tasks distinct from those incorporated in its pretraining. The overall performance is shown in Fig. [ and Fig. 2 Specifically, on 11 language understanding tasks drawn from PromptSource [1] Cappy. with 360 million parameters, outperforms OPT-IML-30B and OPT-175B significantly, and matches the best ones among previous multi-tass -aas LLMs. Besides, on 45 diverse complex tasks from BIG-Bench [25]. Cappy consistently boosts the performance of the advanced multi-task LLM. FLAN-TS, by a large margin. Furthermore, Cappy offers additional performance enhancement when applied together with finetuning or in-context learning. Our subsequent ablation study proves the significance of our proposed pretraining and data augmentation strategies.
Related Work
LLMs for Instruction Following and Multi-task Prompted Training The scaling up of language models brings them increasingly strong capabilities, culminating in general paradigm of addressing diverse problems in a unified instruction-followe such LLMs, each distinguished by the purpose of their instructions. The first approach emphasizes compliance with arbitrary human instructions, often in a question-and-ansven (e.g., "I have to make difficult decision. What should I do?"). Models such as GPT-4 21 and Vicuna 3 are designed to respond to these instructions with the goal of maximizing user satisfaction. These models are typically trained through Reinforcement Learning with Human Feedback (RLHF) 133, leveraging extensive human annotations. Their quantitative evaluation also heavily depends human judgment { The second approach, however, is primarily devoted to resolving well-defined NI.P tasks. In this context, each data instance adheres to a task-specific template, and is transformed into an instruction (c.g., "Put the concepts together to form sentence: ski, mountain, skier. ") and corresponding response (c.g.. "A skier skis down the mountain. "). Multi-task LLMs, such as OPT-IML [10], FLAN [30 4 and TO [23] are pretrained via multi-tass task prompted training. This process trains models unified conditional generation task using pairs of instructions and responses from multiple upstream pretraining tasks. These models are typically assessed based on performance on held-out test tasks, utilizing traditional evaluation metrics such as accuracy, Rouge scores 17 and so forth. In this study, our primary focus is on the second approach, i.e., multi-task LLMs, given its more straightforward evaluation. Nonetheless, we posit that there is no significant obstacle apply our proposed methodologies to more humanish instructions, which leave as our future direction.
manner. There are two primary approaches of
or dialogue format
3

---

Ground-truth instractioo-ressoone pair
Input text pair
Score annotation
Instruction:
Multi-Task L.M eg.
Sample
(eetrutiion corgle
Rouge Leamle target
skoure
wall
Sample
Orstructinn semple 7
Soage-/uamlle targe)
Poundiand store Whymark Avenue (oi) woou you costrase thei in Iow worrh2
Sample
Sample d
(entrurtion
Rouge Upample taggeo
Target (ground truth): Graffiii behnd
billovel
Regression Training Data Cappy
Figure 4: Data augmentation with multi-task LLM to construct weakly supervised regression dataset for Cappy's s pretraining and finetuning, as described in Sec. 3.2
The size of LLMs makes their finetuning for downstream tasks particularly challenging, primarily due to three issues. Firstly, finetuning necessitates the creation of new copy of an LLM for each specific downstream task. This is unacceptable for many applications. Secondly, fine-tuning LLM demands considerable device memory because of the pack-propagattoon through the LLMs, which is achievable only with high-end GPU/TPU clusters. Thirdly, the most powerful LLMs, such as FLAN-PaLM-5444 PaLM-540B 4 and GPT-4 [2 are closed-source and thus inaccessible for fine-tuning, collection of parameter-efficien L.M adaptation techniques, including prompt tuning 18] and adapters & like prefix tuning [15] and LoRA (7), have largely mitigated the storage issue by decreasing the number of tunable parameters. However, these methods still require back propagation through the original LLM weights to update the prompt or the adapter, leaving the second and third issues remain significant barriers in LLM adaptation. Certain in-context learning techniques (S] circumvent LLM's parameter tuning by appending training examples the instruction. However, the instruction length is limited by the model's maximum input length, such in-context learning techniques allow for only finite number of samples to guide the task-solving process. In this work, we propose the adaptation of multi-task LLMs by employing Cappy to incorporate downstream supervision. This approach enables any number of training examples without necessitating LLM finetuning or access to its parameters. Hence, the LLM serves merely as black box, and Cappy is even compatible with WebAPIs of L.MSs. Importantly, Cappy can also be deployed in conjunction with other LLM adaptations, such as finetuning and in-context learning.
Adaptation of LLMs
Ranking- -baee Models Ranking is pivotal component of information retrieval systems, notably in search engines and recommendation systems [19]. It involves sorting vast numbers of documents to find content pertinent to a specific query. Recently, ranking has been adapted for NI.P tasks to aggregate answers [14. and cater to human preferences [2] 6 Furthermore, in the emergent domain of reinforcement learning from human feedback (RLHF) (26). ranking models trained on human-ranked model outputs
as reward providers for the training RL agents. Concurrently with this work, [22 proposes a unified ranking model solving information alignment style tasks, such as natural language inference and paraphrase detection. In this work, Cappy is conceptually a ranking model for multi-task learning. Unlike methodologies specifically designed for question answering
[2] summarization [22]. Cappy offers extensive generalizability across multi-task scenarios. Additionally, in contrast to RLHF reward models, Cappy doesn't 't rely on expensive human-annotated data, enabling large-scale pretraining.
3
The Cappy Scorer
3.1
Modeling
Cappy adopts the architecture of RoBERTa 20 with a linear layer on the top as a regression head. The input of Cappy is pair of text, comprising an instruction and response, and the output is scalar score ranging from to This score indicates an estimation of the correctness of the response with regard to the task instance described in the instruction.
3.2 Pretraining
Cappy's pretraining uses the same dataset collection that is utilized by TO (23 This collection is comprised of 39 diverse datasets from PromptSource D1, encompassing a wide range of task types, such as question answering, sentiment analysis, and summarization, etc. Each dataset is associated with one multiple templates, converting each instance from the original datasets into instruction

---

paired with its ground truth response. Following the pretraining configuration of TO, the size of each dataset is limited a maximum of 500,000 examples.
In light of Cappy's s regression modeling, each data instance during pretraining must have a text pair (instruction, response), coupled with correctness annotation for the response in relation to the instruction. A diverse array of score annotations is a pivotal aspect of a regression dataset. However, the text pairs in our pretraining datasets merely contain instructions with their ground truth responses, hence each text pair invariably has a correctness score of 1.0. This could culminate in a critical lack of label diversity throughout Cappy's 's pretraining. To address this issue, we propose a data construction approach to produce Cappy's s pretraining dataset with correctness annotations that diversely range from 0 1. The data construction consists of three components:
Ground Truth (score 1.0) This component encompasses all ground truth instruction-response pairs from the pretraining dataset. Each pair is assigned correctness annotation of 1.0.
Incorrect Responses (score 0.0) We engineer incorrect data points by creating mismatched instruction-responsee pairs from the original datasets. For classification datasets, each instruction is paired with all incorrect answer choices. For generation datasets, each instruction is arbitrarily paired with ground truth response from a distinct data point within the dataset.
Data Augmentation (score within [0. 1) In addition to purely correct or incorrect data samples, we fabricate instruction-respee response pairs with scores ranging between 0 and 1. This is achieved through data augmentation applied across all generation task instances. For every instance within generation task, we leverage an existing multi-task LLM to generate multiple responses by sampling conditioned on the given instruction. Subsequently, we assign an annotation the pair formed by the instruction and every response, using the similarity between the response and the ground truth response of the instance. Specifically, we employ Rouge-L [77 to calculate this similarity as a form of weak supervision. as it has been widely recognized as a reliable metric for overall performance in multi-task environments and has demonstrated a strong alignment with human evaluation (2].. In practice, our augmented samples are generated by two multi task LLMs, BARTO [16] and TO-3B [23]. For each instance within a generation task, both these models generate two samples using the top-k and top-p sampling, respectively.
Consequently, we collect a pretraining dataset comprised of 160 million instances, each in the format of (instruction, response, score). Cappy is initialized as RoBERTa and optimized using the AdamW optimizer with an I.2 regression loss. The optimization process involves a learning rate of 10-6, 6, warmup rate of 0, and an effective batch size of 1024. In alignment with the RoBERTa variants, Cappy also offered in two distinct versions: the smaller (appyBAsE(1200)) 20M parameters), and the (360 parameters).
3.3 Applying Cappy
Cappy solves practical tasks within a candidate-selection mechanism. More specifically, given an instruction and a set of candidate responses, Cappy produces a score for each candidate. This is achieved by inputting the instruction alongside each individual response, and then assigning the response with the highest score as its prediction. classification tasks, all candidate responses are inherently predefined. For example, the options are (positive, negative) in a sentiment classification task. In such scenarios, Cappy functions independently. On the other hand, in generation tasks, candidate responses are not pre-defined, requiring existing multi-task LLM to yield the candidate responses. In this case, Cappy serves as an auxiliary component of the multi-task LLM, enhancing its decoding.
3.4 Adapting Multi-task LLMs
When there is available downstream training data, Cappy enables effective and efficient adaptation of multi-task LLMs on downstream tasks. Specifically, we propose to integrate downstream task information into LLM's predictions through the finetuning of Cappy. To elaborate, a downstream regression dataset can be acquired through a data annotation process same as the approach utilized during the pretraining data construction ($3.2). Then, Cappy can be finetuned on this regression dataset. As result, the finetuned Cappy collaborates with a multi-tas L.M,, boosting the L.MM's performance on the downstream task.
5

---

StoryCloze
CB
WinoGender
60
90
80
80
70
40
70
50
ANLI (R1/R2/R3)
WSC
WIC
40
--
56
Accuracy
54
36-
55
34
50
45
WinoGrande
Crows-Pairs
80
60
60
70
BARTO
OPT
55
60
OPT-ML
50 1
50
Cappy (ours)
20
10-1
10
10
10a
10-1
10
10
10a
10-4 10"
10a
10a
Model Size (log scale)
Figure 5: The performance of Cappy and multi-task LLMs various test datasets. series of dashed lines are used to connect different sizes of the same model, such as OРT-30B and OPT-175B. Lines points positioned more towards the upper left of the diagram denote models that are more efficient and yield superior performance. Each diagram corresponds to a specific test task, with the exception of ANLI that represents three different tasks (ANLI-RI/R2/R3),
In contrast to other LLM tuning strategies such as finetuning and prompt tuning, adapting LLMs with Cappy on downstream tasks avoids the need for pack-propagttion through LLM parameters. Therefore, it significantly reduces the high demand for device memory. Besides, the adaptation with Cappy does not rely on the access to the LLM parameters, making it compatable with closed-source multi-task L.Mss, such as the ones only accessible via WebAPls. Compared with in-context learning approaches which circumvent model tuning by attaching training examples to the instruction prefix, Cappy is not restricted by the LLM's maximum input length. Thus, Cappy can incorporate an unlimited number of downstream training examples. Morcover, Cappy flexible to work together with other adaptation methods, such as finetuning and in-context learning, further boosting their overall performance, as we demonstrate in experiments.
Experiments
All the experiments of this work, including the pretraining of Cappy and downstream adaptations, are conducted on Google TPU-v4 (IT). and all the code is implemented with Redco (27), lightweight toolkit for automating distributed training.
4.1 Zero-shot Performance on PromptSource
Our evaluation aligns with the ones used by OPT-IML and TO. We assess performance held-out language understanding tasks in PromptSource [[]. all of which are in classification style. These tasks are categorically distinct from those utilized the pretraining datasets. Our baselines include multi-task LLMss, i.e., OPT, OPT-IMI, TO, and BARTO, and RLHF reward model trained with human feedback data, released by LAION-AI Following the answer selection strategy employed

---

Model
Accuracy
Frozen FLAN-TS
Small
Base
Large
XL
XXL
BARTOBASE- 140M
45.7
BARTOLARGE
50.2
Sampling
11.43
15.79
19.62
23.22
25.73
RLHF-RM-I85M
43.6
Temperature
12.01
17.06
20.05
24.27
27.10
Top-K
11.52
15.75
19.76
22.67
25.82
RLHF-RM-33M
53.3
Nucleus
11.92
16.62
20.20
24.17
26,90
OPT-30B
47.6
Beam Search
16.40
19.86
23.48
26.12
29.66
OPT-IML-30B
51.3
scorisc
15.08
20.71
24.12
28.47
32.02
OPT-175B
49.3
OPT-IML-175B
56.5
CAPPyBASE
23.36
27.26
29.83
32.79
36.63
TO-11B
58.2
CapPYLARGE
24.45
28.25
30.75
33.97
36.93
CappyBASE-120N
49.9
ICL Nucleus 16.37
20.46
23.65
28.64
32.70
CapPYLARGE-3660
56.6
ICL Self-scoring
20.61
24.42
27.00
32.56
36.37
ICL + CapPyLARGE
26.18
28.65
31.84
36.41
38.48
Table 1:
The overall accuracy av- eraged over 11 held-out test tasks from PromptSource in zero-shot setting. "RLHF-RM" refers to the RLHF reward model mentioned in Section 4.1
Table 2:
The averaged Rouge-L score over 45 BIG-Bench tasks. The backbone FLAN-TS models are frozen. "ICL" refers to in- context learning, i.e., attaching training examples in the prefix of instruction. We include more prompt-tuning related results in the appendix.
by TO and OPT-IML, predictions from these multi-task LLMs are determined by the answer choice that yields the highest model likelihood. FLAN not considered among the baselines, as the the test tasks are included in its pretraining tasks. We calculate the performance for each task by averaging the results across all associated prompts.
The outcomes of the 11 tasks are presented in Figure 5 with the averaged overall performance of each model summarized in Table I From the per-task figures, Cappy consistently outperforms BARTO models, which are comparable in size to Cappy, and also surpasses OPT and OPT-IML in the majority of tasks. From the overall accurary Table we can summarize that our Cappy yields performance comparable to that of OPT-30B, OPT-IML-30B, and DPT-175B. Furthermore, our larger 360M CappyLARd LoGE performs at level consistent with TO-1 and OPT-IML-I75B. These findings highlight Cappy's superior performance and parameter efficiency in comparison to existing multi-task L.Ms. This improved performance can credited to Cappy's 's scoring-based pretraining strategy, which integrates contrastive information by differentiating between high-quality and low- quality responses. On the contrary, previous multi-task LLMs depend exclusively on macher-forcing training that utilizes only the ground truth responses.
4.2 Adaptation BIG-Bench
We examine the adaptation of multi-task LLMs with Cappy on complex tasks from BIG-Benel (5), a set of manually curated, challenging tasks that are considered beyond the capability of many LLMs. We focus all the 45 generation tasks within BIG-Bench, specifically those that do not offer pre-established answer choices. The train/test split is provided by TaskSource [24). The training sizes of these tasks variable, with minimum of 14 and a maximum of 50,000 instances. The median training size is 876. For each task, we finetune Cappy with an Adam optimizer for 400 steps with a learning rate of 10-5 and an effective batch size of 256. We evaluate the performance using the Rouge-L score on every test set, reporting the average score across 45 tests. In this experiment, all variants of FLAN-T5 serve the backbone LLMs.
We incorporate these approaches in comparison, including: Sampling: Standard token-by-token sampling; Temperature: Sampling every token with distribution temperature of 0.9; Top-K: Top-k sampling with k=40; Nucleus: Nucleus sampling with top-p=0.95; Beam Search: Beam search with width of 4; Self-scoring: We collect four generated samples using all the sampling-based decoding strategies above, plus the top sample from beam search
4 in total 4 x 4 + 1 17 samples. With all the samples, self-scoring selects the best as prediction based on the model log-likelihood; Cappy: conducting sample selection on the same set of samples as in self-scoring, but based on Cappy's
*We don't I collect multiple samples by beam search, because the Jax API for beam search in huggingface- transformers only returns the top sample.
7

---

Advantage (Rouge-L) L)
Figure 6: Advantage of Cappy scoring over FLAN-T5-XXL'S self-scoring, on 45 BIG-Beneh tasks.
The x-axis is the names of the tasks.
scoring. Besides, we include setting of in-context learning (ICL. by attaching training examples at the beginning of the instruction until the model's sequence length limit reached.
The foundational FLAN-T5 models are frozen, that is, not finetuned. The results are displayed in Table 2 They suggests that Cappy enhances the performance of FLAN-T5 models by large margin, consistently outperforming the most effective baseline achieved through sample selection using self-scoring of the LLM itself.
As mentioned in Section 3.4 Cappy exhibits flex- ibility by enabling collaboration with other LLM adaptations. The performance of Cappy working together with finetuning and in-context learning, presented in Table 2land Table 3] respectively. The result demonstrate that Cappy keeps boosting per- formance on top of other adaptations. This can be attributed to the unique downstream knowledge that Cappy acquires from downstream training data. More precisely, while other LLM adaptations predomi- nantly depend on traditional ground truth instruction- response pairs for learning, Cappy extracts and har- nesses the contrastive information by its regression training data constructed with our proposed method.
Finetuned FLAN-T5
Small
Base
Large
Sampling
29.34
37.93
43.45
Temperature
29.83
38.21
43.65
Top-K
30.06
37.48
43,86
Nucleus
30.12
37.87
44.35
Beam Search
32.00
39.21
44.52
Self-scoring
33.95
41.00
46.49
CappyRASE CapPYL.RRGE
37.79
43.73
47.22
39.74
45.18
48.98
Table 3: The averaged Rouge-L score over BIG-Bench tasks, when the backbone FLAN- TS models are also finetuned.
4.2.1 Analysis on Cappy's Scoring
To further understand Cappy's 's scoring, we conducted task-by-task analysis for all the 45 BIG-Bench tasks, comparing the performance between Cappy's scoring with the self-scoring of the multi-task LLM FLAN-TA--SS--EEE
(11B). dispplass the performance advantage of Cappy over FLAN- T5-XXL's self-scoring (with negative values indicating Cappy's disadvantage). The results reveal that, for most tasks, Cappy consistently maintains Rouge-L. approximately 5 points higher than the L.MM's self-scoring. However, there are 3 tasks on which Cappy exhibits significant disadvantage: codenames, simple_arithmeee arithmetic_jsseee
and mufficient_informette
We showcase examples of these tasks in Table Upon examining the codenames examples, we find that the instructions often contain disjointed words with diverse meanings without syntactic
We also conduct prompt tuning for the adaptation of Big- Hig-Bench. Results are discussed the Appendix.

---

Task name:
codenames
Try to identify the 3 words best associated with the word PAJAMAS from the following list: nude, judge, sleep, einstein, groom, troll, wish, sun, quarter, halloween, brain, stamp, wedding, slipper, minotaur, pad, tip, crusader, helmet. Give your answer in alphabetical order. Target: nude, sleep, slipper
Instruction:
Try to identify the word best associated with the word PREHISTORIC from the following list: boom, new york, cotton, green, ball, pumpkin, force, suit, board, jet, mug, head, mammoth, seal, day, engine. Give your answer in alphabetical order.
Instruction:
Target: mammoth
Task name: simple, ara trittmetic..jee
Instruction: 5 = Target: 5
Instruction: 348 + 227 Target: 575
Task name: sufficient_ inrrieentinffomaiiinnn
Instruction: Ed, Jeff, E-Jay, and Michael are in a circle. Ed is on Jeff's left. Is Mike on Ed's left? Target:
I do not know
Instruction: Jake is ten feet away from me. Bryan is one hundred feet from Jake. Am closer to Jake or Brynn?
Target: Jake
Table 4: Tasks on which Cappy's score shows obvious disadvantage compared with FLAN-TS-XXL'S self-scoring.
connections. This might presents considerable challenge to model's "memory" capability. In case of simple_aar aeeeerrrtttteeee thmetic.json and ierriiiienn_nnformreeeeeeeee testing mathematical and commonsense logical abilities. The abilities of momorizing, math, and commonsense, have been demonstrated to be key advantages of LLMs BI and they are difficult to acquire through downstream training alone. Consequently, it is not surprising that the LLM's self-scoring outperforms Cappy's 's scoring in these tasks.
tasks, the focus on
4.2.2 Performance Scaling with the Number of Samples
We study the relation between the adaptation per- formance and number of model generation samples. Specifically, we conduct this on frozen FLAN-T5- XXL. model, with three different numbers of sam- ples, including 1 or 4 neucleus samples, or all the 17 samples as described in Section 4.2] Results are shown in Table 5 we see that as the number of samples increases, Cappy consistently enhances task performance significantly, but the baseline Self- scoring doesn't iI provide significant performance boost increasing from 4 to 17 samples.
Samples
4
17
Self-scoring
26.90
31.15
32.02
CapPyLARGE
26.90
33.64
36.93
Table 5: Performance of BIG-Bench adapta- tion frozen FLAN-T5-XXL with differ- ent numbers of samples.
4.2.3 Ablation Study
To verify the importance of the two key components in our proposed methodol- ogy, we carry out an abla- tion study, utilizing FLAN- T5-XL and XXL's adapta- tion on BIG-Bench tasks. Specifically, in the down- stream adapation, we ablate the pretraining of Cappy
Frozen FLAN-T5
XI.
XXL.
CappyLARGE
33.38
36.56
w/o Cappy pretraining
32.03 (-1.35)
35.01 (-1.55)
w/o Data augmentation using LLM
28.66 (-4.72)
32.88 (-3.67)
Table 6: Performance of BIG-Bench adaptation, before and after the ablation of Cappy's 's pretraining and data augmentation using LLM, numbers in the brackets are the performance drop.

---

that we described in Sec. 3.2 by using RoBERTa as the model initialization instead of Cappy. We also ablate the data augmentation using LLM, described in Sec. 3.2 during the downstream regression data construction for Cappy.
Table shows the results. The ablation of either pretraining data augmentation using LLM results in noticeable decline in performance, thereby highlighting the significance of both these components within our proposed approach. Further, the performance decline scale reveals that the impact of data augmentation is more significant than the pretraining in the downstream adaptation of LLM.
5 Conclusion and Discussions
We deliver a lightweight pretrained scorer, Cappy, to enhance the performance and efficiency of multi-task LLMs. Cappy takes an instruction and a candidate response as input, and produces a score between 0 and 1. The score indicates an estimated correctness of the response with regard to the instruction. weakly-superviser approach is proposed for the construction of Cappy's pretraining data in regression style. We suggest candidate selection manner to apply Cappy into practical task-solving. Specifically, Cappy can be utilized either independently or in collaboration with an existing multi-task LLM, serving as auxiliary component. Our experiments demonstrate that Cappy outperforms much larger multi-lask LLMs, on 11 language understanding tasks. Besides, Cappy boosts FLAN-T5's performance without requiring to finetune the LLM. Moreover, Cappy can effectively collaborate with other LLM adaptation strategies such as finetuming and in-context learning, providing further performance enhancement.
the adaptation to 45 complex tasks drawn from BIG-Bench,
Limitations and Future Directions
In Section 1. we have analyzed certain limitations of Cappy, specifically in the realm of mathe- matics and complex logical problems. Here, We detail some other limitations and future directions.
Rouge-L Score for Weak Supervision In the construction of Cappy's 's pretraining data, Rouge-L serves as the metric evaluate the correctness of model generations. However, Rouge-L, may not be the optimal proxy for correctness of model generations, and there is no consensus across the ML and NLP community on the best metric across all the tasks. Presently, Rouge-L is commonly used in multi-task scenarios to report model performance for generation-style tasks [10]. Although Cappy's ss performance in our experiments demonstrate Rouge-L to be a reasonable design choice, investigating the most suitable metric for multi-task applications remains highly valuable research direction.
Answer Aggregation Across Multiple Generations The primary contribution of this work is the development and application of the pretrained model, Cappy, for multi-task applications. For sample selection from multiple model generations, we use straightforward argmax manner that picks the sample with the largest score. However, recent research with nicely designed answer aggregation techniques (88 suggests potential avenues for refining the answer aggregation with Cappy, to further improve the performance of multi-task learning.
Not Handling Tasks Outside the LLM's Expertise The aim of Cappy is to enhance the perfor- mance of tasks where the backbone LLM has a fundamental understanding of data inputs. However, Cappy doesn't impact the intrinsic ability of the LLM. It is also worth noticing that many multi-task LLMs, such as FLAN-T5 used our experiments, already exhibit proficiency across a wide range of domains, encompassing areas like medicine, law, and coding.
Single LLM Adaptation In the experiments of this work, we adapt single LLM to several domains with Cappy. In the future, Cappy as a pretrained model can potentially be used in other creative ways beyond on single LLMs. For example, Cappy may work filter for generations from multiple L.MSs. In this case, Cappy plays a role that selects the best LLM regarding a specific input.
Resolving More Human-like Instructions and Leveraging Human Feedback In this work, our focus multi-task learning where tasks are well-defined. In the future, Cappy can be potentially applied to resolve more "human-like" instructions where the tasks are often vaguely defined. To this end, it would be highly benefitial to leverage costly but high-quality human feedback data, which would require further algorithmic design. This is worth our further exploration.
10

---

Acknowledgements
We thank Google Research for supporting Bowen Tan working as a student researcher in an internship. Eric Xing and Bowen Tan has also been graciously supported by NGA HM04762010002, NSF IIS1955532, NSF CNS2008248, NIGMS RO1GM140467, NSF IIS2123952, NSF BCS2040381, an Amazon Research Award, NSF IIS2311990, and DARPA ECOLE 12390063, Zhiting Hu is partically supported by DARPA ECOLE HR001 12390063.
References
[1] S. H. Bach, V. Sanh, Z.X. Yong. A. Webson, C. Raffel, N. V. Nayak, A. Sharma, T. Kim, M. S. Bari, T. Févry. Z. Alyafeai, M. Dey, A. Santilli, Z. Sun, S. Ben-David, C. Xu, G. Chhablani, H. Wang, J. A. Fries, M. S. Al-shaibani, S. Sharma, U. Thakker, K. Almubarak, X. Tang, M. T.-J. Jiang, and A. M. Rush. Promptsource: An integrated development environment and repository for natural language prompts. ArXiv, abs/2202.01 279, 2022.
[2] M. Bakker, M. Chadwick, H. Sheahan, M. Tessler, L. -iilingam J. Balaguer, N. McAleese, A. Glaese, J. Aslanides, M. Botvinick, et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems, 35:38 18888166388888 2022.
[3] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang. J. E. Gonzalez, Stoica, and E. P. Xing. Vieuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023.
[4] H. W. Chung, L. Hou, S. Longpre, B. Zoph. Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, D. Valter, S. Narang, G. Mishra, A. W. Yu, V. Zhao, Y. Huang, A. M. Dai, H. Yu, S. Petrov, E. H. hsin Chi,
J. Dean, J. Devlin, A. Roberts, D. Zhou, Q. V. Le, and J. Wei. Scaling -entuttion-finetunedd language models. ArXiv, abs/2210.11416 2022.
[5] Q. Dong, I. Li, D. Dai, C. Zheng, 7. Wu, B. Chang, X. Sun, J. Xu, and 7. Sui. A survey for in-context learning. ArXiv, abs/2301.000 .00244, 2022.
[6] A. Glaese, N. McAleese, M. Trębacz, J. Aslanides, V. Firoiu, T. Ewalds. M. Rauh, L. Weidinger, M. Chadwick, P. Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209. 14375, 2022.
[7] J. E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, and W. Chen. Lora: Low-rank 2021.
adaptation of large language models. ArXiv,
[8] Z. Hu, Y. Lan, L. Wang, W. Xu, E.-P. Lim, R. K.-W. Lee, L. Bing, and S. Poria. Llm- adapters: An adapter family for parameter-efficient fine-tuning of large language models. ArXiv, abs/2304.01933. 2023.
[9] J. Huang and K. C.-C. Chang. Towards reasoning in large language models: A survey, ArXiv, 1os/2212.10403. 2022.
[10] S. Iyer, X. Lin, R. Pasunuru, T. Mihaylov, D. Simig, P. Yu, K. Shuster, T. Wang, Q. Liu, P. S. Koura, X. Li, B. O'Horo, G. Pereyra, J. Wang, C. Dewan, A. Celikyilmaz, Zettlemoyer, and V. Stoyanov. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. ArXiv,
2022.
[11] N. Jouppi, G. Kurian, Li, P. Ma, R. Nagarajan, L. Nai, N. Patil, S. Subramanian, A. Swing, B. Towles, et al. Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings. In Proceedings of the 50th Annual International Symposium on Computer Architecture, pages 1-14, 2023.
[12] B. Kratzwald, A. Eigenmann, and S. Feuerriegel. Rankqa: Neural question answering with answer re-ranking. ArXiv, abs/1906.03008 2019.
[13] N. Lambert, Castricato, L. von Werra, and A. Havrilla. Illustrating reinforcement learning from human feedback (rlhf). Hugging Face Blog, 2022. https//tuuginngfac..oobbogttt
11

---

[14] J. Lee, S. Yun, H. Kim, M. Ko, and J. Kang. Ranking paragraphs for improving answer recall in open-domain question answering. ArXiv, 110.0494, 2018.
[15] X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), abs/2101 000900, 2021.
[16] B. Y. Lin, K. Tan, C. Miller, B. Tian, and X. Ren. Unsupervised cross-task generalization via retrieval augmentation. NeurlPS, 2022.
[17] C.-Y. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguisties.
[18] Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, ACM Computing Surveys, 55:1 - 35, 2021.
[19] T-Y. Liu et al. Learning to rank for information retrieval. Foundations and Trends@) in Information Retrieval, 3(3):225-333 331, 2009.
[20] Y. Liu, M. Ott, N. Goyal, Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692 2019.
[21] OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
[22] M. Ravaut, S. R. Joty, and N. F. Chen. Summareranker: A multi-task mixture-of ofxperr-of-eepeers re-ranking framework for abstractive summarization. In Annual Meeting of the Association for Computational Linguistics, 2022.
[23] V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Scao, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. Sharma, E. Szczechla, T. Kim, Datta, Chang, M. T.-J. Jiang. H. Wang, M. Manica, S. Shen, Z. X. Yong, H. Pandey, R. Bawden, T. Wang, T. Neeraj. J. Rozen, A. Sharma, A. Santilli, Févry, J. A. Fries, R. Techan, S. R. Biderman, L. Gao, T. Bers, T. Wolf, and A. M. Rush. Multitask prompted training enables zero-shot task generalization. ICLR, 2022.
G. Chhablani, N. V. Nayak,
[24] D. Sileo. tasksource: Structured dataset preprocessing annotations for frictionless extreme multi-task learning and evaluation. arXiv preprint wtX.i:2301105944 2023.
[25] A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro,
A. Gupta, A. Cturiga-Alonse A. Kluska, A. Lewkowyez, A. Agarwal, A. Power, A. Ray, A. Warstadt, A. W. Kocurek, A. Safaya, A. Tazarv, A. Xiang, A. Parrish, A. Nie, A. Hussain, A. Askslll,, A. Dsouza, A. A. Rahane, S. Iyer, A. Andreassen, A. Santilli, A. Stuhlmuller, A. M. Dai, A. D. La, A. K. Lampinen, A. Zou, A. Jiang, A. Chen, A. Vuong, A. Gupta, A. Gottardi, A. Norelli, A. Venkatesh, A. Gholamidavoodi, A. Tabassum, A. Menezes, A. Kirubarajan, A. Mullokandov, A. Sabharwal, A. Herrick, A. Efrat, A. Erdem, A. Karakacs, B. R. Roberts, B. S. Loe, B. Zoph, B. Bojanowski. B. Ozyurt, B. Hedayatnia, B. Neyshabur, B. Inden, B. Stein, B. Ekmekci, B. Y. Lin, B. S. Howald, C. Diao, C. Dour, C. Stinson, C. Argueta, C. F. Ram 'rez, C. Singh, C. Rathkopf, C. Meng, C. Baral, C. Wu, C. Callison-Burch, C. Waites, C. Voigt, C. D. Manning, C. Potts, C. T. Ramirez, C. Rivera, C. Siro, C. Raffel, C. Asheraft, C. Garbacea. D. Sileo, D. H. Garrette. D. Hendrycks, D. Kilman, D. Roth, D. Freeman, D. Khashabi, D. Levy, D. Gonz' alez, D. Hernandez, D. Chen, D. Ippolito, D. Gilboa, D. Dohan, D. Drakard, D. Ju- rgens, D. Datta, D. Ganguli, D. Emelin, D. Kleyko, D. Yuret, D. Chen, D. Tam, D. Hupkes, D. Misra, D. Buzan, D. C. Mollo, D. Yang, D.-H. Lee, E. Shutova, E. D. Cubuk, E. Segal, E. Hagerman, E. Barnes, E. P. Donoway, E. Pavlick, E. Rodolà, E. F. Lam, E. Chu, E. Tang, E. Erdem, E. Chang, E. A. Chi, E. Dyer, E. J. Jerzak, E. Kim, E. E. Manyasi, E. Zheltonozhskii, F. Xia, F. Siar, F. Mart' inezeP, F. Happ's e, F. Chollet, F. Rong, G. Mishra, G. I. Winata, G. de Melo, G. Kruszewski, G. Parascandolo, G. Mariani, G. Wang, G. Jaimovitch-- oope,, G. Betz, G. Gur-Ari Ari, H. Galijasevic, H. S. Kim, H. Rashkin, H. Hajishirzi, H. Mchta, H. Bogar,
12

---

H. Shevlin, H. Schütze, H. Yakura, H. Zhang, H. Wong, I. A.-S. Ng, I. Noble, J. Jumelet,
J. Geissinger, J. Kernion, J. Hilton, Lee, J. Fisac, J. B. Simon, J. Koppel, J. Zheng, J. Zou, J. Koco'n, n, J. Thompson, J. Kaplan, J. Radom, J. N. Sohl-Dickstein, J. Phang. J. Wei, Yosinski, Novikova, J. Bosscher, J. Marsh, J. Kim, J. Taal, J. Engel, O. Alabi, J. Xu, J. Song, J. Tang. J. W. Waweru, J. Burden, J. Miller, J. U. Balis, J. Berant, J. Frohberg, J. Rozen, J. Hemández-Orallo, J. Boudeman, J. Jones, B. Tenenbaum, J. Rule, J. Chua, K. Kanclerz, K. Livescu. K. Krauth, K. Gopalakrishnan, K. Ignatyeva, K. Markert, K. D. Dhole, K. Gim- pel, K. O. Omondi, K. W. Mathewson, K. Chiafullo, K. Shkaruta, K. Shridhar, K. McDonell, K. Richardson, L. Reynolds, L. Gao, L. Zhang. L. Dugan, L. Qin, L. Contreras-Ochant Morency, L. Moschella, L. Lam, L. Noble, L. Schmidt, L. He. L. O. Col on, L. Metz. L. K. eSenel, M. Bosma, M. Sap, M. ter Hoeve, M. Andrea, M. S. Farooqi, M. Faruqui, M. Mazeika, M. Baturan, M. Marelli, M. Maru, M. Quintana, M. Tolkiehn, M. Giulianelli, M. Lewis, M. Potthast, M. Leavitt, M. Hagen, M. Schubert, M. Baiteminova, M. Arnaud, M. A. McEI- rath, M. A. Yee, M. Cohen, M. Gu, M. I. Ivanitskiy, M. Starritt, M. Strube, M. Swkedrowski, M. Bevilacqua, M. Yasunaga, M. Kale, M. Cain, M. Xu, M. Suzgun, M. Tiwari, M. Bansal, M. Aminnaseri, M. Geva, M. Gheini, T. Mukund N. Peng, N. Chi, N. Lee, N. G.-A. Krakover, N. Cameron, N. S. Roberts, N. Doiron, N. Nangia, N. Deckers. N. Muennighoff, N. S. Keskar, N. Iyer, N. Constant, N. Fiedel, N. Wen, O. Zhang, O. Agha, O. Elbaghdadi, O. Levy, Doshi, P. Fung, P. P. Liang, P. Vicol, P. Alipoormolabashi, P. Liao, P. Liang, P. W. Chang. P. Eckersley. P. M. Htut, P.-B. Hwang. P. Milkowski, P. S. Patil, P. Pezeshkpour, Oli, Q. Mei, Q. LYU, Q Chen, R. Banjade, R. E. Rudolph, R. Gabriel, R. Habacker, R. R. Delgado, R. Millière, R. Garg, R. Barnes, R. A. Saurous, R. Arakawa, R. Ray- maekers, R. Frank, R. Sikand, R. Novak, R. Sitelew, R. I.. Bras, R. Liu, R. Jacobs, R. Zhang, R. Salakhutdinov, R. Chi, R. Lee, R. Stovall, R. Teehan, R. Yang, S. Singh, S. M. Mohammad, S. Anand, S. Dillavou, S. Shleifer, S. Wiseman, S. Gruetter, S. Bowman, S. S. Schoenholz, S. Han, S. Kwatra, S. A. Rous, S. Ghazarian, S. Ghosh, S. Casey, S. Bischoff, S. Gehrmann, Schuster, S. Sadeghi, S. S. Hamdan, S. Zhou, S. Srivastava, S. Shi, S. Singh, S. Asaadi, S. S. Gu, S. Pachchigar, S. Toshniwal, S. Upadhyay, S. Debnath, S. Shakeri, S. Thormeyer, S. Melzi, S. Reddy, S. P. Makini, S. hwan Lee, S. B. Torene, S. Hatwar, S. Dehaene, S. Divic, S. Ermon, S. R. Biderman, S. C. Lin, S. Prasad, S. T. Piantadosi, S. M. Shicber, S. Misherghi, S. Kiritchenko, S. Mishra, T. Linzen, T. Schuster, T. Li, T. Yu, T. A. Ali, T. Hashimoto, T.-L. Wu, T. Desbordes, T. Rothschild, TT Phan, T. Wang, T. Nkinyili, T. Schick, T. N. Kornev, T. Telleen-Lawton, T. Tunduny, T. Gerstenberg, T. Chang, T. Neeraj. T. Khot, T. O. Shultz, U. Shaham, V. Misra, V. Demberg, V. Nyamai, V. Raunak, V. V. Ramasesh, V. U. Prabhu, V. Padmakumar, V. Srikumar, W. Fedus, W. Saunders, W. Zhang, W. Vossen, X. Ren, X. Tong, X. Wu, X. Shen, Y. Yaghoobzadeh, Y. Lakretz, Y. Song, Y. Bahri, Y. J. Choi, Y. Yang, Y. Hao, Y. Chen, Y. Belinkov, Y. Hou, Y. Hou, Y. Bai, Z. Seid, Z. Xinran, Z. Zhao, Z. F. Wang, Z. J. Wang, Z. Wang, Z. Wu, S. Singh, and U. Shaham. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. ArXiv, abs/2206.046
L.-P.
O. Evans, P. A. M. Casares,
2022.
[26] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
[27] B. Tan, Y. Zhu, L. Liu, H. Wang, Y. Zhuang, J. Chen, E. Xing, and Z. Hu. Redco: A Lightweight Tool to Automate Distributed Training of LLMs on Any GPU/TPUs. arXiv e-prints, page arXiv:2310 10. 16355, Oct. 2023.
[28] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11177 2022.
[29] Y. Wang. S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaci, A. Arunkumar, A. Ashok, A. S. Dhanasekaran, A. Naik, D. Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. EMNLP, 2022.
[30] J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. ICLR, 2022.
[31] Z. Yuan, Yuan, C. Tan, W. Wang, and S. Huang. How well do large language models perform in arithmetic tasks? ArXiv, abs/2304.02015, 2023.
13

---

[32] Y. Zha, Y. Yang, R. Li, and Z. Hu. Text alignment is an efficient unified model for massive nlp tasks, arXiv preprint arXiv:2307.02729 2023.
[33] L. Zheng, Y. Sheng, W.-L. Chiang, H. Zhang, J. E. Gonzalez, and I. Stoica. Chatbot arena: Benchmarking llms in the wild with elo ratings, 2023.
14

---

A Comparison of Cappy with other adaptation methods
Cappy doesn't mean beat other adaptation methods such as finetuning, in-context learning, and prompt tuning. Compared with these approaches, adaptation with Cappy is an alternative that is free from the constraints associated with storage, device memory, model accessibility, and training sample limitations. Moreover, Cappy makes no assumption about the backbone model, enabling seamless integration with other adaptations and providing steady and significant performance promotion at little additional cost. To illustrate this, we include an experiment below that combines Cappy with in-context learning and prompt-tuning respectively.
Specifically, we add more compari- son with prompt tuning and in-context learning in our BIG-Bench adaptation experiment with FLAN-T5-Land the backbone model. For prompt tun- ing, we apply prefix tuning, which is usually considered suitable for gener- ation tasks, with virtual tokens. As demonstrated by the results presented T Cappy offers further perfor- mance boost top of both in-context learning and prompt tuning.
Setting
Rouge-L
Frozen FLAN T5 La-Lagee Cappy
30.75
as
In-context learning + Nucleus
23.65
In-context learning + Self-scoring
27.00
In-context learning (ApPYLARGEEQUSS)
31.84
Prompt-tuning + Nueleus
34.00
Prompt-tuning Self-scoring
38.43
Prompt-tuning + CappyL.ARGr(ours)
42.71
Table 7: Big- Big-Bench performance under multiple adaptation settings with FLAN-TS-Large
15


Reviews:
[{"review_id": "olEvhStJWA", "reviewer": "Reviewer_uzD3", "summary": "In this paper, the authors tackle the challenge of computational requirements and memory constraints in fine-tuning Large Language Models (LLMs) by introducing an innovative approach that enhances LLM performance without the need for backpropagation through the LLM or access to its parameters. They propose Cappy, a pre-trained scorer that evaluates texts generated by LLMs based on downstream instructions and assigns them a score ranging from 0 to 1.\n\nTo create Cappy, the authors curate a diverse collection of 39 samples from the PromptSource dataset, including good (ground truth), bad (random), and intermediate examples. The intermediate examples are generated using top-k and top-p sampling from BART0 and T0-3B models. Cappy assigns a score to each example based on the ROUGE-L metric compared to the ground truth response. This dataset is used to create a regression task and train a RoBERTa model that serves as the foundation for Cappy.\n\nThe authors demonstrate improved accuracy across 11 classification held-out tasks from PromptSource compared to OPT, OPT-IML, T0, and BART0. They also showcase Cappy's effectiveness as an LLM booster for downstream adaptation on the BIG-Bench generative tasks, outperforming other selection strategies and achieving higher ROUGE-L scores on both frozen and fine-tuned FLAN-T5 models.\n\nAdditionally, the paper discusses scenarios where Cappy performs less favorably than the \"self-scoring\" strategy on certain BIG-Bench tasks, suggesting that the lack of \"memory\" may contribute to this performance difference. Furthermore, the authors conduct an ablation study, revealing that data augmentation using LLMs is more crucial for improved performance than pre-training Cappy, although pre-training still offers benefits.\n\nOverall, the paper presents an innovative approach, Cappy, for boosting LLM performance without backpropagation or parameter access. The experiments showcase its superiority in classification tasks and downstream adaptation, while highlighting the importance of data augmentation and the potential limitations related to memory.", "questions": "- Could you provide an example illustrating how the limitation of instruction lengths affects in-context learning? Adding a specific example would enhance the understanding of this limitation and its implications.\n- In the context of downstream adaptation, it is not entirely clear whether the same LLM that is being augmented is used to generate the synthetic data. Clarifying this aspect in the paper would help readers better comprehend the process of downstream adaptation and the relationship between the augmented data and the LLM.\n- The reasoning behind the requirement for more \"memory\" in tasks like \"sufficient_information\" is unclear. It would be beneficial to provide a more detailed explanation either in the paper or in the appendix to shed light on the specific aspects that require increased memory in such tasks.\n- The results show that removing the Cappy pre-training step does not significantly impact the downstream score on BIG-Bench, implying that fine-tuning with augmented data from LLMs is sufficient. Have you explored the performance of RoBERTa initialization with data augmentation using LLMs? It would be insightful to investigate and report the results of this comparison to further validate the efficacy of the proposed approach.\n\nMinor Issues:\n- Line 186: PromprSource -> PromptSource\n- Line 285: 360 -> 360 M\n- Line 222: freezed -> frozen - This issue exists at several places in the paper.", "limitations": "The authors thoroughly discuss various limitations, including mathematical complexities, reliance on supervised datasets, absence of multi-lingual extensions, and other important concerns. However, it would be beneficial to further address limitations related to the reliance on generations from existing LLMs and potential biases that may arise as a result. Additionally, the use of ROUGE-L as the sole metric for regression and evaluation may not provide a comprehensive and accurate portrayal of the overall performance. It would be valuable to acknowledge this limitation and discuss the potential impact on the interpretation and generalization of the results.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The idea of an auxiliary performance booster is an intriguing concept. The demonstrated improvements over the selected baselines highlight the effectiveness of this approach in enhancing language model performance.\n- The utilization of LLM generations for data augmentation and the regression-based evaluation using ROUGE-L scores is a novel and innovative methodology employed in this work.\n- Cappy introduces a unique integration of samples from LLMs and ground-truth information, leveraging correctness scores to enhance the quality of text generations. This ability to distinguish between good and poor responses is a novel idea that has the potential to enhance the performance of all language models.\n- Despite having significantly fewer parameters compared to large language models used in zero-shot baselines, pre-trained Cappy achieves comparable performance. This contribution is valuable as it demonstrates that Cappy can achieve performance on par with larger models, offering a more efficient and resource-friendly solution.\n- The versatility of Cappy is showcased by its potential for downstream adaptation and further fine-tuning to improve task-specific performance. This adaptability highlights the broad applicability and utility of Cappy as a solution in various contexts.", "weaknesses": "- The performance of Cappy, trained on BART0 and T0 generations, does not consistently outperform T0 and OPT-IML. While the zero-shot performance is better than BART0 (a relatively smaller model), it falls short compared to T0 in most cases. The evidence supporting the claim of boosting language model performance is not sufficiently strong, and the improvements are not consistently observed.\n- The training of Cappy to optimize ROUGE-L scores against ground truth data raises concerns about the choice of this metric. ROUGE-L may not necessarily be the most optimal metric for evaluation, as alternative responses with low ROUGE-L scores may be valid and informative. Including additional metrics for evaluation would provide a more comprehensive understanding of the overall increase in performance.\n- The discussion on the tasks in BIG-Bench where Cappy performs worse than the \"self-scoring\" strategy is insufficient. Given the expectations for Cappy to perform at least as well, if not better, further analysis and exploration of these instances would strengthen the paper and provide a clearer understanding of the limitations of Cappy's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive feedback that our idea of auxiliary performance booster is intriguing, our methodology is innovative, our delivered model Cappy is versatile, and our multi-task application with Cappy is valuable, efficient and resource-friendly.\n\n**Boosting language model performance** \\\nIndeed, Cappy doesn’t beat T0 in the zero-shot setting. However, it actually gets an accuracy very close to that of T0 (Cappy 56.6 v.s. T0 58.2, as illustrated in Table 1), considering the substantial difference of their model sizes (Cappy 360M v.s. T0 11B). Furthermore, our BIG-Bench adaptation experiments demonstrate that Cappy provides steady performance improvement for LLMs of varied sizes, and either frozen or finetuned. Based on all these observations above, we overall conclude that Cappy boosts multi-task language models.\n\n**Using Rouge-L score as a weak supervision**\\\nWe agree that Rouge-L may not be the optimal proxy for correctness of model generations. Our choice of Rouge-L is primarily based on three considerations as below: (1) The community has not reached a consensus on the best metric across all the tasks. Nonetheless, Rouge-L is commonly used in multi-task scenarios to report model performance for generation-style tasks, such as in OPT-IML paper. (2) As a weak supervision for a subset of data in our large-scale pretraining, we acknowledge and accept that some data examples might not be perfectly labeled. (3) Moreover, Cappy’s performance in our experiments is further evidence for Rouge-L to be a reasonable design choice. \n\nThat being said, investigating the most suitable metric for multi-task applications is a highly valuable area of research. Thank you for pointing this out, and we will keep exploring this in the future.\n\n\n**Further analysis on BIG-Bench experiments**\\\nWe appreciate the suggestion and we will deeply dive into more tasks on which Cappy doesn't perform better than Self-scoring. \n\nHere we present another two such tasks as below – “operators” and “physics_questions”. In line with the tasks we elaborated in our paper, they also demand heavy math and commonsense/physics knowledge, respectively. \n\n“operators”:\\\n*Instruction: Given the definition of the op operator, compute the result. op n1 n2 ... nn extracts the last multiple of 8 from the n listed numbers. op 4 32 128 132 =*\\\n*Target: 128*\n\n“physics_questions”\\\n*Instruction: Q: The historic Stanley Center for the Arts in Utica, New York is the proud owner of the world’s largest LED chandelier. The chandelier is 35 feet wide, 17 feet tall and has a mass of 2900 kg. It is directly supported by four cables which make an angle of 63° with the horizontal. Determine the tension in the cables. A:*\\\n*Target: 7974 N*\n\n\n**Response to Questions**\n* For example, consider an LLM with a large context length of 2048, and a task such as NLI, where the average length of each training example is approximately 32. In this case, in-context learning can accomondate about 2048 / 32 = 64 demonstrations within the input context at its maximum. However, many complex downstream tasks have tens of thousands of training examples, such as many tasks in BIG-Bench. \n* Yes, in the downstream adaptation, the augmented data for Cappy’s finetuning is from the model to be enhanced. \n* The task requiring memory ability is actually “codenames”, where a long list of words is given and the model needs to remember and identify a couple of them with a given association.\n* Our ablation of Cappy pretraining is already using RoBERTa initialization with data augmentation. We will make this clearer in our next paper update.\n\nThanks for pointing out our typos and suggesting more potential limitations! We will fix/add them in the next version of our paper.\n\n"}, {"review_id": "lPjoH9UESq", "reviewer": "Reviewer_XRdH", "summary": "Adapting SotA LLMs to novel tasks is difficult given their extreme size, yet is generally more effective than in-context learning alone. To address this, the authors propose “Cappy”, a scoring model (leveraging RoBERTa as a backbone) which is trained to score pairs of (instruction, response) for any task instruction and corresponding response. Cappy is pre-trained on many examples of ((instruction, response) -> score); after pre-training, Cappy can be easily fine-tuned (because of it’s small size) to a given task and then used as an augmentation to LLM generation by acting as a ranking mechanism atop several LLM generated responses.\n\nTo create pre-training data for Cappy, the authors use the PromptSource dataset, training the model to predict 1 for correct pairs of (prompt, response) and 0 for mismatched pairs. Additionally, the authors generate additional responses using a large LLM (e.g. FLAN) and use the Rouge-L between the generated response and gold response as the regression target for the pair (prompt, generated response). To test the validity of Cappy, the authors evaluate Cappy in isolation, with no additional fine-tuning, on 11 held-out classification tasks from PromptSource. They show that Cappy consistently ranks the correct answer higher than many instruction-tuned LLMs, including OPT-IML-175B.\n\nNext, the authors evaluate Cappy on LLM augmentation using the BIG-Bench benchmark. For each task, Cappy is first fine-tuned, then used as a scoring function for multiple generations from a frozen FLAN-T5 LLM. Cappy consistently outperforms the proposed sampling methods, most notably improving upon the LLMs own self-scoring and In-Context Learning. Moreover, the authors show that Cappy can improve upon a FLAN-T5 model’s performance even when the LLM parameters are fine-tuned as well.\n\nFinally, the authors present an ablation study of their proposed training strategy for Cappy, showing that both the overall pre-training strategy and using Rouge-L regression targets from LLM generations are important to Cappy’s success.", "questions": "- I think it would be really helpful to understand how Cappy compares to other parameter efficient adaptation methods. Do you have any experiments that indicate that Cappy is e.g. comparable to prompt-tuning in terms of it's benefits?\n- Can you provide an analysis of how much additional computation is required to sample many generations for Cappy to score?\n- Also, do you know how Cappy's performance scales with the number of generated samples? e.g. does it's performance flatten out after only 4 samples, or can it's performance increase significantly if several generations are considered?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Cappy represents a light-weight option to adapt LLMs to a given target task which requires no updating of parameters to the LLM nor any back-propagation through the LLM whatsoever (e.g. this is not true for adapter layers or prompt tuning).\n- Moreover, even without adapting to a given task, Cappy presents some benefits over generic LLM sampling methods for task-specific predictions.\n-  Additionally, Cappy can ostensibly be added on top of any LLM without additional training (i.e. train once, use anywhere).\n- Cappy is a very simple idea, and is easy to understand, yet the usage of a scorer in this capacity, as well as the training and data augmentation scheme, is novel to the best of my knowledge.\n- The ablation study is helpful in demonstrating the benefits of the proposed pre-training and data-augmentation scheme.", "weaknesses": "- Cappy is not compared to other parameter-efficient adaptation methods. While there is some justification given, because other methods require storing forward activations and back-propagating through LLM parameters, it nevertheless makes judging the effectiveness of Cappy difficult.\n- Moreover, Cappy’s improvements over a frozen LLM are comparatively small, e.g. while Cappy improves Flan-T5 Large performance by ~7%, fine-tuning alone increases performance by ~22%. It’s therefore unclear how significant Cappy’s benefit is without comparison to other adaptation methods.\n- In the zero-shot experiments (4.1) it is claimed that the improved performance of Cappy over much larger LLMs can be attributed to Cappy’s scoring-based learning strategy. However, T0 outperforms Cappy and T0 is significantly smaller than the OPT models; moreover, T0 is also trained on PromptSource only, similar to Cappy, and unlike the other LLMs. Thus, it seems that at least some of Cappy’s performance benefits compared to other LLMs may arise from domain mismatch between training and test domains, rather than the proposed scoring function.\n- While Cappy does only require a small number of parameters to train, inference requires a number of decoder forward passes to generate multiple candidate responses from an LLM (it's not clear how Cappy scales with the number of samples provided, also); to my knowledge, other adaptation methods such as Adapter Layers do not have the same limitation. So while it saves computation on one end, it does slightly increase computation on another end, and exploring this trade-off would be useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive comments, including that our approach is novel, our delivered model Cappy is light-weight and beneficial, and our ablation study is well conducted.\n\n**Comparison of Cappy with other adaptation methods**\\\nWe would like to clarify that Cappy doesn’t mean to beat other adaptation methods such as finetuning, in-context learning, and prompt tuning. Compared with these approaches, adaptation with Cappy is an alternative free from the constraints associated with storage, device memory, model accessibility, and training sample limitations. Moreover, Cappy doesn’t have any assumption about the backbone model, enabling seamless integration with other adaptations, where Cappy also provides steady and significant performance promotion with little additional cost. To illustrate this, we add an experiment below that combines Cappy with in-context learning and prompt-tuning.\n\nSpecifically, we add more comparison with prompt tuning and in-context learning in our BIG-Bench adaptation experiment with FLAN-T5-Large as the backbone model. For prompt tuning, we apply prefix tuning, which is usually considered suitable for generation tasks, with 20 virtual tokens. As demonstrated by the results presented below, Cappy offers further performance boost on top of in-context learning and prompt tuning.\n\n| Setting                                   | Rouge-L |\n|-------------------------------------------|---------|\n| frozen FLAN-T5-Large + Cappy-Large (ours) | 30.08   |\n| In-context learning + Nucleus             | 22.59   |\n| In-context learning + Self-scoring        | 27.00   |\n| In-context learning + Cappy-Large (ours)  | **31.84**   |\n| Prompt-tuning + Nucleus                   | 34.00   |\n| Prompt-tuning + Self-scoring              | 38.43   |\n| Prompt-tuning + Cappy-Large (ours)        | **42.71**   |\n\n\n**Train/test domain mismatch**\\\nActually, PromptSource is part of OPT-IML pretrain data, and OPT-IML encompasses even more training tasks compared with Cappy, so that the domain mismatch might not be a benefit of Cappy. Thanks for asking this! We will make this more clear in our next paper update.\n\n**Computational cost of multiple generations**\\\nGiven Cappy’s small size, the predominant computational overhead arises from generating multiple samples using the backbone LLM. That is, if N samples are generated, the computational effort would be N times that of a standard single-sample generation, assuming the samples are not batch-processed.\n\nHowever, collecting multiple samples for a single prediction is a common operation in many algorithms, such as self-consistency [1], and some re-ranking techniques designed for QA [2] and summarization [3]. Furthermore, by leveraging the batch processing capabilities of GPUs/TPUs, the actual time overhead, when compared with single-sample generation, can be significantly less than N-fold. This efficiency potentially explains the continued popularity of these techniques. \n\n\n[1] Self-Consistency Improves Chain of Thought Reasoning in Language Models\\\n[2] RankQA: Neural Question Answering with Answer Re-Ranking\\\n[3] SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization\n\n\n\n**How performance scales with the number of samples**\n\nWe appreciate such a meaningful suggestion. We conduct this on a frozen FLAN-T5-11B model, with three settings as below:\n* 1 sample:    a single nucleus sample (top-p=0.95)\n* 4 samples:  4 nucleus samples \n* 20 samples: 4 samples from 5 different decoding methods (Random Sampling, Temperature, Top-K, Nucleus, Beam Search). \n\nResults are shown as below, as the number of samples increases, Cappy consistently enhances task performance significantly, in contrast with the baseline Self-scoring.\n\n|                    | 1 sample | 4 samples | 20 samples |\n|--------------------|----------|-----------|------------|\n| Self-scoring       | 27.33    | 31.15     | 32.62      |\n| Cappy-Large (ours) | 27.33    | **33.64**     | **36.56**      |\n\n\nThanks for the suggestions and meaningful points above! We will add all the discussion above to the next version of our paper.\n\n\n"}, {"review_id": "oHTMLTEdwf", "reviewer": "Reviewer_hbCr", "summary": "This paper introduces an auxiliary module called Cappy, which aims to enhance the performance of large language models. Cappy operates by taking a task instruction and a proposed response as input, and it estimates the quality score for the response. The training process involves creating a dataset that combines correct and mismatched answers with various instructions, and the Cappy scorer is then trained using a regression objective. In downstream adaptation, the small scorer, Cappy, is fine-tuned on a small dataset that follows the same approach as the training set. During testing, predictions are determined by selecting the answer choice with the highest score. In essence, Cappy acts as a valuable filter that enhances the overall quality of generated responses.", "questions": "Please refer to the weaknesses.", "limitations": "No additional limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The idea of leveraging a lightweight scorer as a ‘’filter’’ is interesting. Fine-tuning the filters on downstream tasks can be efficient and practical.", "weaknesses": "Cappy itself is only a filter for in-domain responses and can not impact the intrinsic ability of a pre-trained LLM. This method cannot handle tasks outside the LLM's expertise, emphasizing the need for a sufficiently general and powerful LLM. Alternatively, fine-tuning LLMs, as shown in Table 3, may be necessary but costly. By comparing able 2 and Table 3, fine-tuning is still very essential for task adaptation. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments that our idea is interesting, and our proposed downstream adaptation approach is efficient and practical.\n\n**Not handling tasks outside the LLM's expertise**\\\nIndeed, the primary objective of Cappy is to enhance performance of tasks where the backbone LLM possesses a fundamental understanding of the data input. Notably, many multi-task LLMs, including FLAN-T5 utilized in our experiments, exhibit proficiency across a wide range of domains, encompassing areas like medicine, law, and coding.\n\n\n**Comparison of Cappy with finetuning**\\\nWe would like to clarify that Cappy doesn’t mean to beat other adaptation methods, especially finetuning. Compared with other adaptation approaches, Cappy is an alternative free from the constraints associated with storage, device memory, model accessibility, and training sample limitations.\n\nMoreover, Cappy doesn’t have any assumption about the backbone model, enabling seamless integration with other adaptations. As we show in Table 3, Cappy provides steady and significant performance promotion with little additional cost, \n"}, {"review_id": "QrzkKpclrH", "reviewer": "Reviewer_K4oy", "summary": "The authors propose training a small(er) classifier model (‘Cappy’) to predict an answer given a set of possibilities (if the answer set is closed) or multiple generations from an LLM (in the case of open-ended/generative tasks). The model is trained on data from T0, using LLMs to generate partially correct responses alongside positive and negative examples. They evaluate performance on T0 evaluation tasks (classification) and big-bench (generative) and find that Cappy outperforms same-size models, and consistently provides improvements over a base model when picking from generations.", "questions": "1. How is cappy different to best-of-n techniques proposed in the papers mentioned above? Do you think using preference data already available could help or augment your data collection strategy?\n2. Did you explore contrastive objectives in pretraining cappy? How about encoding multiple answers at once?\n3. Do you have further details on the distribution of rouge-l scores you used in the data augmentation setup? It would be interesting to see this, and do you think there are any limitations in using rouge-l against something like cosine similarity for this?", "limitations": "I think the authors effectively discuss limitations. They miss some elements such as Cappy requiring multiple generations (thus increasing its inference cost over using a single-generation) and complicating the model pipeline, but cover Cappy’s weakness in the realm of complex logical problem-solving / mathematics, and its current reliance on supervised datasets.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed approach is straightforward, using a trained classifier as a reranker for model generations. The improvements seem significant, and the fact that a classifier outperforms a similar-size generative model is interesting to see.\n\nThe data gathering scheme for Cappy’s training data is interesting, and using rouge-l as a proxy for gold labels is interesting and seems to work well in ablations.\n\nThe writing is clear and the paper is easy to follow.", "weaknesses": "**Comparisons to Best-of-N** - I’m somewhat concerned about the novelty of this work. The proposed method is very similar to the best-of-N sampling originally proposed in [1] (“Best-of-N sampling”) and discussed (sometimes under ‘reranking’) in [2] [3], [9], inter alia. These methods work by sampling multiple LM generations and then picking the answer with the highest reward as chosen by a trained reward model.\nThe three core differences between these prior works and Cappy I can detect are:\n1. Different use in classification - Cappy directly uses the set of possible answers for classification tasks, while these prior works do not.\n2. Training data - Cappy is trained on prompt source-based data with augmentation, which is interesting and seems to help performance, while (afaik) best-of-n uses trained reward models, which are often trained using preference datasets.\n3. Application - best-of-n approaches are often applied and focussed on how they aid with aligning to human preferences, while the focus in this work is on improving over benchmarks.\n\nWhile these are interesting differences, the authors do not compare to these prior works and explicitly make these comparisons. At the very least, I would expect these methods to be discussed in a related work section, and the decisions made in training Cappy contrasted to them. Ideally, they would be integrated as baselines/ablations to justify differences, highlighting the merits of Cappy over these approaches and covering what is needed to adapt these approaches to Cappy’s chosen setting.\n\n**Missing prior work for reranking** - Cappy effectively is serving as a reranker in the generative case. There is a long history of work in answer reranking, that would be useful to discuss as related work (e.g., [4] for multi-choice qa, [5,7] for summarisation, [5,6] for open-domain qa). \n\n**Further baselines and ablations** - There are other methods for training and applying rerankers, including contrastive training (as used to train a reranker in [7], with rouge as a proxy for human judgements), or encoding multiple samples at once as opposed to one-at-a-time (as done in [4]). It would be useful to test these changes, especially given the authors suggest that making use of contrastive information is helpful for Cappy in line 202. It would also be interesting to compare against self-consistency [8], another approach for improving models that makes use of multiple generations.\n\nGiven all of these things, I am inclined to recommend rejection for this paper in its current state. Further changes differentiating this approach against best-of-n sampling and further experiments exploring the full space of answer reranking would greatly improve this work, and may sway my opinion, but require significant extra work. I think the overall idea is interesting and evidently effective, but requires further work to make a substantial contribution. I hope the authors make these changes and are successful in the future!\n\n[1] Stiennon et al. (2020). Learning to summarize from human feedback. NeurIPS. https://arxiv.org/pdf/2009.01325.pdf\n\n[2] Bakker et al. (2022). Fine-tuning language models to find agreement among humans with diverse preferences. ArXiv. https://arxiv.org/pdf/2211.15006.pdf\n\n[3] Glaese et al. (2022). Improving alignment of dialogue agents via targeted human judgements. ArXiv. https://arxiv.org/abs/2209.14375\n\n[4] Kratzwald et al. (2019). RankQA: Neural Question Answering with Answer Re-Ranking. ACL. http://aclanthology.lst.uni-saarland.de/P19-1611.pdf\n\n[5] Revaut et al. (2022). SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization. ACL. https://aclanthology.org/2022.acl-long.309.pdf\n\n[6] Lee et al. (2018). Ranking Paragraphs for Improving Answer Recall in Open-Domain Question Answering. EMNLP. https://aclanthology.org/D18-1053.pdf\n\n[7] Liu et al. (2021) SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization. ACL. https://aclanthology.org/2021.acl-short.135/\n\n[8] Wang et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR. https://arxiv.org/pdf/2203.11171.pdf\n\n[9] Dubois et al. (2023). AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. ArXiv. https://arxiv.org/abs/2305.14387\n\nEdit: The authors have responded with some of the experiments and comparisons I asked for, with positive results, and so I'm happy to raise my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback that our data construction method is interesting, that Cappy’s improvements over other methods are significant, and that our writing is clear and easy to follow. \n\n\n**Comparison with Reranking and best-of-N sampling**\\\nWe appreciate the suggestions on various sample selection methodologies and your insights on the distinctions between Cappy and these techniques.\n\nTo summarize and extend your valuable observations, the key difference between Cappy and previous reranking and best-of-N sampling methods lies in two aspects: (1) Compared with methods that conduct ranking based on reward models [1, 2, 3, 9], Cappy doesn’t rely on expensive human-annotated data, which enables our large-scale pretraining; (2) Unlike reranking methods specifically tailored for QA [4, 6] or summarization [5, 7], Cappy exhibits broader generalizability across multi-task settings. \n\nHowever, we would also like to clarify that our key contribution is constructing a large-scale dataset, delivering the pretrained model Cappy, and applying Cappy into multi-task applications. In terms of sample selection, we actually use a rather trivial argmax manner that picks the sample with the largest score. This sample selection is also used in our Self-scoring baseline. As of now, we have not incorporated any fancy sample selection techniques such as best-of-N rejection sampling [1] or answer aggregation [8]. That being said, we will definitely explore more suitable sample selection strategies for Cappy in our future research.\n\n\n\n**Contrastive objective, encoding multiple answers at once, human preference data**\\\nWe do incorporate contrastive information during the training of our model. However, this is not achieved through an explicit contrastive loss function or a modeling to encoding multiple answers at once. Instead, the contrastive information is sourced from our pretrain data, where there are several examples sharing the same instruction but have different responses and score annotations. We have demonstrated the effectiveness of this incorporation, by Cappy’s performance in the multi-task scenarios presented in our experiments. \n\nAs for the use of human preference data, we acknowledge this can be very beneficial. However, integrating human-annotated data would require further algorithmic design. It's also worth noticing that obtaining human preference data can be costly, and the volume might be insufficient for large-scale pretraining. We remain committed to exploring strategies for effectively integrating human feedback in our future work.\n\n\n**Using Rouge-L score as a weak supervision**\\\nThanks for the suggestion to analyze the Rouge-L distribution. Among all of our pretrain data with 160M data points, there are 30M samples are labeled 1.0, 90M samples labeled 0.0, and the other 40M has score annotations between 0 to 1. Specifically, the average of the third part is 0.3 and the whole distribution can be visualized in our author rebuttal pdf. From this analysis, we can see that all the scores ranging from 0 to 1 have significant number of samples, which demonstrates the effectiveness of our data construction.\n\nWe agree that Rouge-L may not be the optimal proxy of correctness of model generations. Cosine similarity may measure the semantic information better than the n-gram based Rouge-L. However, using Rouge-L could be a more suitable design choice for our large-scale pretraining, because (1) cosine similarity is less efficient: it would be time-consuming to get an embedding and run a similarity score for every example in the large pretrain dataset. (2) besides, it would require specific embedding models for tasks in a specific domain, while Rouge-L doesn’t have this limitation. (3) The community has not reached a consensus on the best metric across all the tasks. Nonetheless, Rouge-L is commonly used in multi-task scenarios to report model performance for generation-style tasks, such as in OPT-IML paper. (4) As a weak supervision for a subset of data in our large-scale pretraining, we acknowledge and accept that some data examples might not be perfectly labeled. (5) Moreover, Cappy’s performance in our experiments is further evidence for Rouge-L to be a reasonable design choice. \n\nInvestigating the most suitable metric for multi-tasking is a highly valuable research direction. Thank you for pointing this out, and we will keep exploring this in the future.\n\n\nThanks for suggesting more potential limitations of our work. We will add our discussion to all these points in our next paper update.\n\n"}]

Author rebuttal:
We would like to express our gratitude to all the reviewers for their insightful comments. We are encouraged by the reviewers' appreciation that our idea of auxiliary performance booster is intriguing (uzD3), that our methodology is novel and innovative (uzD3, XRdH), that our delivered model Cappy is valuable, efficient, resource-friendly, versatile and practical (uzD3, hbCr), that our ablation study is well conducted (XRdH, K4oy), and that our paper writing is clear and easy to follow (K4oy).



Meta-review: The paper introduces a new method for improving instruction following in language models, by training a small scoring model to rate the quality of a language model generation. Rouge-L is used to automatically judge the quality of a large set of LM generations, and then Roberta is finetuned as a scoring model. Experiments demonstrate that a small ranking model can still lead to improvements on Big Bench. Reviewers appreciated that the technique is simple, effective and practical, and I recommend this paper for acceptance.
Decision: Accept (poster)