SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We examine the adaptation of multi-task LLMs with Cappy on complex tasks from BIG-Bench [25], a set of manually curated, challenging tasks that are considered beyond the capability of many LLMs. We focus on all the 45 generation tasks within BIG-Bench, specifically those that do not offer pre-established answer choices. The train/test split is provided by TaskSource [24]. The training sizes of these tasks are variable, with a minimum of 14 and a maximum of 50,000 instances. The median training size is 876. For each task, we finetune Cappy with an AdamW optimizer for 400 steps with a learning rate of  $2 \times 10^{-5}$  and an effective batch size of 256. We evaluate the performance using the Rouge-L score on every test set, reporting the average score across 45 tests. In this experiment, all variants of FLAN-T5 serve as the backbone LLMs.
We incorporate these approaches in comparison, including: Sampling: Standard token-by-token sampling; Temperature: Sampling every token with a distribution temperature of 0.9; Top-K: Top-k sampling with  $k = 40$ ; Nucleus: Nucleus sampling with top-p=0.95; Beam Search: Beam search with a width of 4; Self-scoring: We collect four generated samples using all the sampling-based decoding strategies above, plus the top sample from beam search  $^4$ , in total  $4 \times 4 + 1 = 17$  samples. With all the samples, self-scoring selects the best one as prediction based on the model log-likelihood; Cappy: conducting sample selection on the same set of samples as in self-scoring, but based on Cappy's
<sup>4</sup>We don't collect multiple samples by beam search, because the Jax API for beam search in huggingface-transformers only returns the top sample.
Figure 6: Advantage of Cappy scoring over FLAN-T5-XXL's self-scoring, on 45 BIG-Bench tasks. The x-axis is the names of the tasks.
scoring. Besides, we include a setting of in-context learning (ICL) by attaching training examples at the beginning of the instruction until the model's sequence length limit is reached.
The foundational FLAN-T5 models are frozen, that is, not finetuned. The results are displayed in Table 2. They suggest that Cappy enhances the performance of FLAN-T5 models by a large margin, consistently outperforming the most effective baseline achieved through sample selection using self-scoring of the LLM itself.
As mentioned in Section 3.4, Cappy exhibits flexibility by enabling collaboration with other LLM adaptations. The performance of Cappy working together with finetuning and in-context learning, is presented in Table 2 and Table 3, respectively. The result demonstrate that Cappy keeps boosting performance on top of other adaptations. This can be attributed to the unique downstream knowledge that Cappy acquires from downstream training data. More precisely, while other LLM adaptations predominantly depend on traditional ground truth instruction-response pairs for learning, Cappy extracts and harnesses the contrastive information by its regression training data constructed with our proposed method.
Finetuned FLAN-T5
Small	Base	Large
Sampling	29.34	37.93	43.45
Temperature	29.83	38.21	43.65
Top-K	30.06	37.48	43.86
Nucleus	30.12	37.87	44.35
Beam Search	32.00	39.21	44.52
Self-scoring	33.95	41.00	46.49
CappyBASE	37.79	43.73	47.22
CappyLARGE	39.74	45.18	48.98
Table 3: The averaged Rouge-L score over BIG-Bench tasks, when the backbone FLAN-T5 models are also finetuned.
4.2.1 Analysis on Cappy's Scoring
To further understand Cappy's scoring, we conducted a task-by-task analysis for all the 45 BIG-Bench tasks, comparing the performance between Cappy's scoring with the self-scoring of the multi-task LLM - FLAN-T5-XXL (11B). Figure 6 displays the performance advantage of Cappy over FLAN-T5-XXL's self-scoring (with negative values indicating Cappy's disadvantage). The results reveal that, for most tasks, Cappy consistently maintains a Rouge-L approximately 5 points higher than the LLM's self-scoring. However, there are 3 tasks on which Cappy exhibits a significant disadvantage: codenames, simple_arithmetic_json, and sufficient_information.
We showcase examples of these 3 tasks in Table 4. Upon examining the codenames examples, we find that the instructions often contain disjointed words with diverse meanings without syntactic
We also conduct prompt tuning for the adaptation of Big-Bench. Results are discussed in the Appendix.
Task name: codenames
Instruction: Try to identify the 3 words best associated with the word PAJAMAS from the following list: nude, judge, sleep, einstein, groom, troll, wish, sun, quarter, halloween, brain, stamp, wedding, slipper, minotaur, pad, tip, crusader, helmet. Give your answer in alphabetical order.Target: nude, sleep, slipper
Instruction: Try to identify the 1 word best associated with the word PREHISTORIC from the following list: boom, new york, cotton, green, ball, pumpkin, force, suit, board, jet, mug, head, mammoth, seal, day, engine.Give your answer in alphabetical order.Target: mammoth
Task name: simple_arithmetic_json
Instruction: 5 + 0 =Target: 5
Instruction: 348 + 227 =Target: 575
Task name: sufficient_information
Instruction: Ed, Jeff, E-Jay, and Michael are in a circle. Ed is on Jeff's left. Is Mike on Ed's left?Target: I do not know
Instruction: Jake is ten feet away from me. Brynn is one hundred feet from Jake. Am I closer to Jake or Brynn?Target: Jake
connections. This might presents a considerable challenge to a model's "memory" capability. In the case of simple_arithmetic_json and sufficient_information tasks, the focus is on testing mathematical and commonsense logical abilities. The abilities of memorizing, math, and commonsense, have been demonstrated to be key advantages of LLMs [31, 9], and they are difficult to acquire through downstream training alone. Consequently, it is not surprising that the LLM's self-scoring outperforms Cappy's scoring in these tasks.
4.2.2 Performance Scaling with the Number of Samples
We study the relation between the adaptation performance and number of model generation samples. Specifically, we conduct this on a frozen FLAN-T5-XXL model, with three different numbers of samples, including 1 or 4 neucleus samples, or all the 17 samples as described in Section 4.2. Results are shown in Table 5, we can see that as the number of samples increases, Cappy consistently enhances task performance significantly, but the baseline Self-scoring doesn't provide significant performance boost increasing from 4 to 17 samples.
Table 4: Tasks on which Cappy's score shows obvious disadvantage compared with FLAN-T5-XXL's self-scoring.
Samples	1	4	17
Self-scoring	26.90	31.15	32.02
CappyLARGE(ours)	26.90	33.64	36.93
4.2.3 Ablation Study
To verify the importance of the two key components in our proposed methodology, we carry out an ablation study, utilizing FLANT5-XL and -XXL's adaptation on BIG-Bench tasks. Specifically, in the downstream adapation, we ablate the pretraining of Cappy
Table 5: Performance of BIG-Bench adaptation on a frozen FLAN-T5-XXL with different numbers of samples.
Frozen FLAN-T5
XL	XXL
CappyLARGE	33.38	36.56
- w/o Cappy pretraining	32.03 (-1.35)	35.01 (-1.55)
- w/o Data augmentation using LLM	28.66 (-4.72)	32.88 (-3.67)
Table 6: Performance of BIG-Bench adaptation, before and after the ablation of Cappy's pretraining and data augmentation using LLM, numbers in the brackets are the performance drop.
that we described in Sec. 3.2, by using RoBERTa as the model initialization instead of Cappy. We also ablate the data augmentation using LLM, described in Sec. 3.2 during the downstream regression data construction for Cappy.
Table 6 shows the results. The ablation of either pretraining or data augmentation using LLM results in a noticeable decline in performance, thereby highlighting the significance of both these components within our proposed approach. Further, the performance decline scale reveals that the impact of data augmentation is more significant than the pretraining in the downstream adaptation of LLM.
We deliver a lightweight pretrained scorer, Cappy, to enhance the performance and efficiency of multi-task LLMs. Cappy takes an instruction and a candidate response as input, and produces a score between 0 and 1. The score indicates an estimated correctness of the response with regard to the instruction. A weakly-supervised approach is proposed for the construction of Cappy's pretraining data in regression style. We suggest a candidate selection manner to apply Cappy into practical task-solving. Specifically, Cappy can be utilized either independently or in collaboration with an existing multi-task LLM, serving as an auxiliary component. Our experiments demonstrate that Cappy outperforms much larger multi-task LLMs, on 11 language understanding tasks. Besides, Cappy boosts FLAN-T5's performance on the adaptation to 45 complex tasks drawn from BIG-Bench, without requiring to finetune the LLM. Moreover, Cappy can effectively collaborate with other LLM adaptation strategies such as finetuning and in-context learning, providing further performance enhancement.
Limitations and Future Directions
In Section 4.2.1, we have analyzed certain limitations of Cappy, specifically in the realm of mathematics and complex logical problems. Here, We detail some other limitations and future directions.
Rouge-L Score for Weak Supervision In the construction of Cappy's pretraining data, Rouge-L serves as the metric to evaluate the correctness of model generations. However, Rouge-L may not be the optimal proxy for correctness of model generations, and there is no consensus across the ML and NLP community on the best metric across all the tasks. Presently, Rouge-L is commonly used in multi-task scenarios to report model performance for generation-style tasks [10]. Although Cappy's performance in our experiments demonstrate Rouge-L to be a reasonable design choice, investigating the most suitable metric for multi-task applications remains a highly valuable research direction.
Answer Aggregation Across Multiple Generations The primary contribution of this work is the development and application of the pretrained model, Cappy, for multi-task applications. For sample selection from multiple model generations, we use a straightforward argmax manner that picks the sample with the largest score. However, recent research with nicely designed answer aggregation techniques [28] suggests potential avenues for refining the answer aggregation with Cappy, to further improve the performance of multi-task learning.
Not Handling Tasks Outside the LLM's Expertise The aim of Cappy is to enhance the performance of tasks where the backbone LLM has a fundamental understanding of data inputs. However, Cappy doesn't impact the intrinsic ability of the LLM. It is also worth noticing that many multi-task LLMs, such as FLAN-T5 used in our experiments, already exhibit proficiency across a wide range of domains, encompassing areas like medicine, law, and coding.
Single LLM Adaptation In the experiments of this work, we adapt a single LLM to several domains with Cappy. In the future, Cappy as a pretrained model can potentially be used in other creative ways beyond on single LLMs. For example, Cappy may work as a filter for generations from multiple LLMs. In this case, Cappy plays a role that selects the best LLM regarding a specific input.
**Resolving More Human-like Instructions and Leveraging Human Feedback** In this work, our focus is multi-task learning where tasks are well-defined. In the future, Cappy can be potentially applied to resolve more "human-like" instructions where the tasks are often vaguely defined. To this end, it would be highly beneficial to leverage costly but high-quality human feedback data, which would require further algorithmic design. This is worth our further exploration.
We thank Google Research for supporting Bowen Tan working as a student researcher in an internship. Eric Xing and Bowen Tan has also been graciously supported by NGA HM04762010002, NSF IIS1955532, NSF CNS2008248, NIGMS R01GM140467, NSF IIS2123952, NSF BCS2040381, an Amazon Research Award, NSF IIS2311990, and DARPA ECOLE HR00112390063. Zhiting Hu is partially supported by DARPA ECOLE HR00112390063.
H. Shevlin, H. Schütze, H. Yakura, H. Zhang, H. Wong, I. A.-S. Ng, I. Noble, J. Jumelet, J. Geissinger, J. Kernion, J. Hilton, J. Lee, J. F. Fisac, J. B. Simon, J. Koppel, J. Zheng, J. Zou, J. Koco'n, J. Thompson, J. Kaplan, J. Radom, J. N. Sohl-Dickstein, J. Phang, J. Wei, J. Yosinski, J. Novikova, J. Bosscher, J. Marsh, J. Kim, J. Taal, J. Engel, J. O. Alabi, J. Xu, J. Song, J. Tang, J. W. Waweru, J. Burden, J. Miller, J. U. Balis, J. Berant, J. Frohberg, J. Rozen, J. Hernandez-Orallo, J. Boudeman, J. Jones, J. B. Tenenbaum, J. S. Rule, J. Chua, K. Kanclerz, K. Livescu, K. Krauth, K. Gopalakrishnan, K. Ignatyeva, K. Markert, K. D. Dhole, K. Gimpel, K. O. Omondi, K. W. Mathewson, K. Chiafullo, K. Shkaruta, K. Shridhar, K. McDonell, K. Richardson, L. Reynolds, L. Gao, L. Zhang, L. Dugan, L. Qin, L. Contreras-Ochando, L.-P. Morency, L. Moschella, L. Lam, L. Noble, L. Schmidt, L. He, L. O. Col'on, L. Metz, L. K. cSenel, M. Bosma, M. Sap, M. ter Hoeve, M. Andrea, M. S. Farooqi, M. Faruqui, M. Mazeika, M. Baturan, M. Marelli, M. Maru, M. Quintana, M. Tolkiehn, M. Giulianelli, M. Lewis, M. Potthast, M. Leavitt, M. Hagen, M. Schubert, M. Baitemirova, M. Arnaud, M. A. McElrath, M. A. Yee, M. Cohen, M. Gu, M. I. Ivanitskiy, M. Starritt, M. Strube, M. Swkedrowski, M. Bevilacqua, M. Yasunaga, M. Kale, M. Cain, M. Xu, M. Suzgun, M. Tiwari, M. Bansal, M. Aminnaseri, M. Geva, M. Gheini, T. MukundVarma, N. Peng, N. Chi, N. Lee, N.G.-A Krakover,N.CameronN.S.RobertsN.DoironN.NangiaN.DeckersN.MuennighoffN.S Keskar,N.Iyer,NCONSTANTN.FiedelN.WenO.ZhangO.AghaO.ElbaghdadiO.Levy O.EvansP.A.M.CasaresP.DoshiP.FungP.P.LiangP.VicolP.Alipoormolabashi P.LiaoP.LiangP.W.ChangP.EckersleyP.M.HtutP.B.HwangP.MilkowskiP.S Patil,P.Pezeshkpor,P.OliQ.MeQ.LYUQ.ChenR.BanjadeR.E.RudolphR.Gabriel R.Habacker,R.R.Delgado,R.Milliere,R.Garg,R.Barnes,R.A.Saurous,R.Arakawa,R.Raymaekers,R.Frank,R.Sikand,R.Novak,R.Sitelew,R.L.Bras,R.LiuR.Jacobs,R.Zhang R.Salakhutdinov,R.ChoR.LeeR.StovallR.TeehanR.YangS.J.SinghS.M.Mohammad S.Anand,S.Dillavou,S.Shleifer,S.Wiseman,S.Gruetter,S.Bowman,S.S.Schoenholz S.HanS.KwatraS.A.RousS.GhazarianS.GhoshS.CaseyS.BischoffS.Gehrmann S.SchusterS.SadeghiS.SHamdanS.ZhouS.SrivastavaS.ShiS.SinghS.Asaadi S.S.GuS.PachchigarS.ToshniwalS.UpadhyayS.DebnathS.ShakeriS.Thormeyer S.MelziS.ReddyS.P.MakiniS.hwan LeeS.B.ToreneS.HatwarS.DehaeneS.Divic S.ErmonS.R.BidermanS.C.LinS.PrasadS.T.PiantadosiS.M.Shieber,S.Misherghi S.Kritchenko,S.MishraT.LinzenT.SchusterT.LiT.YuT.A.AliT HashimotoT-L Wu,T.Desbordes,T.Rothschild,T.PhanT.WangT.Nkinyili,T.SchickT.N.Kornev T.Telleen-LawtonT.TundunY.GerstenbergT.ChangT.NeerajT.KhotT.O.Shultz U.ShahamV.MisraV.Demberg,V.NyamaiV.RaunakV.V.RamaseshV.U.Prabhu V.PadmakumarV.SrikumarW.FedusW.SaundersW.ZhangW.VossenX.RenX.Tong X.WuX.ShenY.YaghoobzadehY.LakretzY.SongY.BahriY.J.ChoiY.YangY.Hao Y.ChenY.BelinkovY.HouY.HouY.BaiZ.SeidZ.XinranZ.ZhaoZ.F.WangZ.J WangZ.WangZ.WuS.Singh and U.Shaham Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.ArXivabs/2206.046152022
[26] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
[27] B. Tan, Y. Zhu, L. Liu, H. Wang, Y. Zhuang, J. Chen, E. Xing, and Z. Hu. Redco: A Lightweight Tool to Automate Distributed Training of LLMs on Any GPU/TPUs. arXiv e-prints, page arXiv:2310.16355, Oct. 2023.
[28] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022.
[29] Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Arunkumar, A. Ashok, A. S. Dhanasekaran, A. Naik, D. Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. EMNLP, 2022.
[30] J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. ICLR, 2022.
[31] Z. Yuan, H. Yuan, C. Tan, W. Wang, and S. Huang. How well do large language models perform in arithmetic tasks? ArXiv, abs/2304.02015, 2023.
[32] Y. Zha, Y. Yang, R. Li, and Z. Hu. Text alignment is an efficient unified model for massive nlp tasks. arXiv preprint arXiv:2307.02729, 2023.
[33] L. Zheng, Y. Sheng, W.-L. Chiang, H. Zhang, J. E. Gonzalez, and I. Stoica. Chatbot arena: Benchmarking llms in the wild with elo ratings, 2023.
A Comparison of Cappy with other adaptation methods
Cappy doesn't mean to beat other adaptation methods such as finetuning, in-context learning, and prompt tuning. Compared with these approaches, adaptation with Cappy is an alternative that is free from the constraints associated with storage, device memory, model accessibility, and training sample limitations. Moreover, Cappy makes no assumption about the backbone model, enabling seamless integration with other adaptations and providing steady and significant performance promotion at little additional cost. To illustrate this, we include an experiment below that combines Cappy with in-context learning and prompt-tuning, respectively.
Specifically, we add more comparison with prompt tuning and in-context learning in our BIG-Bench adaptation experiment with FLAN-T5-Large as the backbone model. For prompt tuning, we apply prefix tuning, which is usually considered suitable for generation tasks, with 20 virtual tokens. As demonstrated by the results presented in Table 7, Cappy offers further performance boost on top of both in-context learning and prompt tuning.
Setting	Rouge-L
Frozen FLAN-T5-Large + CappyLARGE(ours)	30.75
In-context learning + Nucleus	23.65
In-context learning + Self-scoring	27.00
In-context learning + CappyLARGE(ours)	31.84
Prompt-tuning + Nucleus	34.00
Prompt-tuning + Self-scoring	38.43
Prompt-tuning + CappyLARGE(ours)	42.71
Table 7: Big-Bench performance under multiple adaptation settings with FLAN-T5-Large