Ensembles and Cocktails: Robust Finetuning for Natural Language Generation
Authors: John Hewitt
Xiang Lisa Li
Sang Michael Xie
Benjamin Newman
Percy Liang
Venue: ICLR 2022 Submitted (2022)
Keywords: finetuning
pretrained language models
natural language generation
robustness
prefix-tuning

Abstract: When finetuning a pretrained language model for natural language generation tasks, one is currently faced with a tradeoff. Lightweight finetuning (e.g., prefix-tuning, adapters), which freezes all or most of the parameters of the pretrained model, has been shown to achieve stronger out-of-distribution (OOD) performance than full finetuning, which tunes all of the parameters. However, lightweight finetuning can underperform full finetuning in-distribution (ID). In this work, we present methods to combine the benefits of full and lightweight finetuning, achieving strong performance both ID and OOD. First, we show that an ensemble of the lightweight and full finetuning models achieves the best of both worlds: performance matching the better of full and lightweight finetuning, both ID and OOD. Second, we show that we can achieve similar improvements using a single model instead of two with our proposed cocktail finetuning, which augments full finetuning via distillation from a lightweight model. Finally, we provide some explanatory theory in a multiclass logistic regression setting with a large number of classes, describing how distillation on ID data can transfer the OOD behavior of one model to another.

Under review as a conference paper at ICLR 2022
ENSEMBLES AND COCKTAILS: ROBUST FINETUNING FOR NATURAL LANGUAGE GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
When finetuning a pretrained language model for natural language generation tasks, one is currently faced with a tradeoff. Lightweight finetuming (c.g.. prefix- tuning, adapters), which freezes all or most of the parameters of the pretrained model, has been shown to achieve stronger out-of-distribution (OOD) performance than full finetuning, which tunes all of the parameters. However, lightweight finetuning can underperform full finetuning in-distribution (ID). In this work, we present methods to combine the benefits of full and lightweight finetuning, achieving strong performance both ID and OOD. First, we show that ensemble the lightweight and full finetuning models achieves the best of both worlds: performance matching the better of full and lightweight finetuning, both II) and OOD. Second, we show that we can achieve similar improvements using single model instead of two with our proposed cocktail finetuning, which augments full finetuning via distillation from lightweight model. Finally, we provide some explanatory theory multiclass logistic regression setting with a large number of classes, describing how distillation on ID data can transfer the OOD behavior of one model another.
INTRODUCTION
When finetuning a pretrained language model for natural language generation tasks like summariza- tion (Narayanan et al. 2018) and table-to-text (Gar dent et al. 2017). one currently faced with a tradeoff. One can achieve strong in in-distribution (ID) performance or strong out-of-distribution (OOD)
Ensembles and Cocktail Finetuning: Best of both worlds
Lightweight Finetuning
Out-of-distribution performance
Weight A=1
performance-but not both-by choosing one of two
families of finetuning approaches. Finetuning all pa-
Zero-sum tradeoff
rameters of the pretrained language model achieves strong ID performance (Howard & Ruder 2018).. We
A=0
call this full finetuning. Alternatively, freezing most
Full Finetuning
of the pretrained parameters during finetuning, in e.g.
adapters (Rebuffi et al.) 2017 Houlsby al. 2019a).
In-distribution performance
prefix-tuning (Li & Liang] 2021 Lester et al. 2021
and bitfit (Zaken et al. (2021) achieves stronger OOD
performance than full finetuning (Li & Liang, 2021
Figure Instead a zero-sum tradeoff between the ID and OOD performance of lightweight and full finetuning, ensembling and cocktail finetuning (purple square) ap- proximately achieve the best performance of each, both ID and OOD.
Lester al.] (02T). but worse ID performance. We call these methods lightweight finetuning.
In this work, we are the first to demonstrate that this tradeoff is not necessary in natural language genera- tion: we can achieve the best of full and lightweight finetuning both ID and OOD, We test this across three
natural language generation tasks in English: summarization (Narayanan et al. 2018). table-to to textex (Gardent et al. 2017), and open-domain question answering (Roberts et al. 2020 Lee al. 2019a), and two lightweight finetuning methods: adapters and prefix-tuning. Achieving the best of both worlds is important since across our 6 settings (2 lightweight finetuning methods and datasets),

---

Under review as a conference paper at ICLR 2022
full finetuning significantly outperforms lightweight finetuning ID in 5 settings, and lightweight finetuning significantly outperforms full finetuning OOD in 6 settings.
We present two methods for achieving the best of full and lightweight finetuning. We first show that a weighted ensemble of one full finetuning model and one lightweight finctuning model achieves performance comparable to the best of the two, both ID and OOD. To avoid running two models at once, we propose cocktail finetuning: first train a lightweight finetuning model, and then train a full finetuning model on a mixture of student-teacher distillation loss (Bucilà et al.) 2006: Hinton et al.) 2015) (with the lightweight model as teacher) and cross-entropy loss, on the original ID data The ensemble weight (resp. the cocktail mixture loss weight) is chosen optimize for ID performance, and we observe that the resultant model achieves comparable OOD performance to lightweight finetuning.
Our work contributes to an ongoing realization in machine learning: robust models that deviate minimally from a pretrained model can be gainfully combined with model adapted more precisely to a specific dataset. Most saliently, in concurrent work, Wortsman al. (202I) find that zero- shot image recognition through CLIP (Radford et al.] (021) can be combined with a finetuned network to perform well ID and OOD. Our work demonstrates that similar results hold true in natural language generation as well. We draw from simple, well-known methods: ensembles, student-teacher distillation (Abnar et al. 2020 Hinton et al. 2015 but our application achieving the best of ID and OOD performance in natural language generation is novel.
To better understand how cocktail finetuning can transfer the out-of-distribution behavior of the lightweight finetuning model when distilling only on in-distribution data, we theoretically study multiclass logistic regression with large number of output classes, viewing language generation as a classification problem over a large output space. We study distribution shift setting where not all classes are seen in the ID training data (as true in language generation due to Zipf's law (Zipf) 1949),, and getting good OOD accuracy requires generalizing to unseen outputs. First, we prove that model trained with gradient descent cannot distinguish the unseen output classes, leading to poor OOD accuracy. Second, we prove that distilling a teacher model with good OOD accuracy on only ID training data can transfer the OOD behavior of the teacher to the student, such that the student and teacher make the same prediction on any new input. As result, the student also has good OOD accuracy.
In summary, we present two simple sethhods-essebbling and cocktail finetuning for achieving strong ID and OOD natural language generation with only the original ID data, achieving the "best of both worlds" of full and lightweight finetuning methods.
RELATED WORK
Lightweight finetuning. Recently, as pretrained models scale up in the number of parameters, lightweight finetuning techniques have been developed to avoid updating all of the parameters. These techniques update a subset of the parameters (Zaken et al., (021). introduce a small number of new parameters between layers of the model (called "adapters" ") (Rebuffi al. 2017: Houulby al. 2019b Pfeitfer et al. 2020), or learn auxiliary inputs that are prepended to the models inputs (called "prompts") ") (Shin et al.) 2020 Li & Liang] 2021 (in & Eisner 2021: Liu et al. 2021b Logan IV cet al. 2021; and see Liu et al. 202 for a survey). These techniques empirically have the benefit of being able to achieve strong -fiss--dittiibution extrapolation performance at the cost of worse in-distribution performance compared to full finetuning models both in classification and generation settings (Li & Liang| 202T; Lester et al., 202T Zhou et al.) 2021] Utama al., (02T). In our work, we aim to overcome this trade-off by proposing techniques for achieving the best of both worlds, with focus on language generation.
Student-teacher distillation. Buciluă et al. (2006) introduce distillation for compressing large ensembles into smaller, faster, more efficient models, and Hinton et al. (2013) generalize their method and call it knowledge distillation. Kasai et al. (202)) distill Transformers into RNNs for linear-time inference. Abnar et al. (2020) show that knowledge distillation can transfer the inductive bias of one architecture to another: for example, distilling an LSTM to a Transformer Icads to a Transformer with
We use the term cocktail as it is refreshing mixture involving distillation.
2

---

Under review as a conference paper at ICLR 2022
representations that are more similar to independently trained LSTMs than independently trained Transformers. The efficiacy of our proposed cocktail finetuning is surprising even in context of these results, it empirically transfers just the strong OOD performance of the teacher lightweight model to the student, while not transferring the weak ID performance.
Robust finetuning in vision. In concurrent work, Wortsman et al. (2021) propose a similar ap- proach ours, but focus on image classification: they combine full finetuning model with lightweight finetuning (linear probe) model by ensembling the two models in weight space, and show that their approach maintains both good ID and OOD performance. is yet surprising that in the structured prediction setting of natural language generation, we further show that we can achieve high OOD performance even when weight-space ensembling is impossible (for instance, since there are weights not held in common between the full finetuning model and an optimized prompt). Moreover, we further this line of work by providing a theoretical study into how distillation can transfer OOD) behavior while only using ID data.
Self-training for robustness. In adversarial robustness, standard models achieve good accuracy on clean examples but are not robust to adversarial examples, while adversarially trained models more robust but are less accurate on clean examples. Raghunathan et al. (2020) show that the benefits these two models can be combined using robust self-training (RST) CCarmon et al. 2019; Uesato et al. 2019: Najafi et al. 2019) unlabeled data improve clean and robust accuracy. Similarly, Khani & Liang 2222) show that RST can be used to combine models with spurious features removed (which are more robust) and without removing spurious features (which are more accurate). In general OOD robustness, In-N-Out (Xie et al. (202]) uses self-training to transfer the robustness benefits of model pre-trained with auxiliary suffssupevviioo to a model trained with extra auxiliary features. Berthelot et al. (202I) use unlabeled data and a consistency regularization loss for domain adaptation. While many of these methods combine the benefits of two models or use self-training eelf-trainingddisiillaaiio loss to improve robustness, cocktail finetuning differs in that it performs distillation only on ID training data without any additional ID or OOD unlabeled examples.
3
PROBLEM STATEMENT
3.1
SETUP
We consider a conditional generation task from an input space X' (e.g.. a news article) to an output space y (e.g., a summary of the article), where both spaces consist sequences of tokens from shared vocabulary V.
Data. Let Pa and Poot denote the distribution of (x, v) pairs in-distribution and nut-of-distribution. respectively, where x E X and y E y. In summarization, Pa may consist of articles about world and business news while Pood consists of health and technology news. The training data Drain and validation data D consist of n and noya in-distribution (z, y) pairs respectively, drawn from Pa-
(3: xr, Meÿ) be a pretrained neural autoregressive sequence model with parameters 7. Finetuning (full or lightweight) produces finetuned model (v(v 2²) from g.y where 0 denotes the set of trainable parameters. In full finetuning, 0 is the same set of parameters as 7; in lightweight finetuning, often refers to the parameters inserted modules (e.g.. adapters modules prefixes). The final predictor is fel (z) which approximates arg max, Po(y) a)) through, e.g., beam search. The goal is to learn parameters f such that the predictor fo obtains good ID and OOD performance:
Model and metrics. Let (v x) = kon(s)
Sa((fo = Er.y nu (score fo()))
(1)
Soodffe = Ex,y P.m. (e.
(2)
where score may be defined differently for each task (e.g., exact match score or BLEU (Papineni et al. 2002h))
3

---

Under review as a conference paper at ICLR 2022
WalN ID
[108 Row Jehn Madie), (John Madin BirthPiice Birmingham) (Binninghem |eaderrNmee Androw Micheell
WeeNLG ID (g)
John was Dirmingham (with Andrew Mischell as key leader) became architect, designing 100 Colmore
WaiNL OOD (r)
[Alhennic Jones game Rhythmn
[Alheeniee Jones [NirthPacc]] Fretta Missisipei] [Rhythm bluex [dervaatvv] Errato, Mississippi is performer of rhythe
| Disen]
WoNNLG OOD ())
Allissic Jones,
bluss, of which disco derivsiive.
XSUM (D)
The country's ss consumer watchdog has taken Apple to cour: for false advertsinng because the table: computer does not work on Australia's 4G netwock..A Apple's lawyers said they were willing to publish a clarifiaaionn However the company daes not accopt that misled castomers. The Australian Competiioon and Cuncumer Commission (ACOC) said on Tuesday: [... 170 words] comreiive adverising and refamds in consmmes. On its website. Apple does state that 4G LTE only suppocted on selected networks the US and Canada. US technology firm Apple his offered refund Austmllin customers who felt mislad about the 46 expubiiitiss of the new iPad.
XSUM (g)
XSUM OOD (t)
The 200n (65m) smuctare. becutiing panomsmic views of London, post of 2200 rocapp of the wordd-fomuus fomous art muscum It being hillod the UK's impuriani cultural building since the Briiih Library More than of solo displays dedicated to women artiis.AL. lamach Tate Exchange "opan experiment" accapying entire the across UK display their week.
'husday's launc I...
Late: this House building. will enable invited orpanisations The Modern has unvelled new extension, pyramid like tower housing coremous gullery spaces, shead official opening.
XSUM ODD (
OpenOA (2)
medical term
cause discuse
OpeQA (y)
dtology
OpenAA OOD (r)
who the newly elected governoe california?
OpenQA (g)
Jony Brown
Table 1: Examples of in-distribution and out-of-distribution inputs and outputs from the WebNLG, XSUM and OpenQA tasks, respectively. In WebNLG, the distribution shift is due to non-overlapping topics; in XSUM, the distribution shift is due non-overlapping news categories; in OpenQA the distribution shift is due to different data collection processes.
3.2 BASELINE FINE-TUNING METHODS
Full finetuning has been the defacto approach to adapt pretrained language models for downstream
generation tasks (Lewis et al. 2020 Kale & Rastogi 2020). Under this framework, the model pe is
initialized with its pretrained parameters y, and we optimize the following log-likelihood objective.
L(0)= = E(r.)))De... log Po x=]]
(3)
where the set of trainable parameters refer to the set of pretrained parameters 7.
Lightweight finetuning is another family of adaptation approaches, which freezes most of the pretrained parameters and augments the model with small trainable modules. Lightweight finetuning shares the same training objective as full finetuning, but the trainable parameters 0 are different. Within the lightweight finetuning family, we focus on adapter-tuning (Houlsby et al., 2019a) and prefix-tuning (Li & Liang 2021). Both approaches keep the pretrained parameters intact. Adapter- tuning inserts task-speeific MLP layers between each layer of the pretrained language models, and the trainable parameter & consists of these inserted MLP parameters. trefix-tuning prepends a sequence of trainable, task-specific prefix vectors to the input, and consists of these prefix parameters ?
4 EXPERIMENT SETUP
4.1 DATASETS AND METRICS
Table-to-text. We evaluate on the WebNL.G dataset (Gardent et al. (017), where the input is a sequence of (entity, predicate, entity) triples and the output is a natural language description that covers the input information. Each example in this dataset is labeled with one of 14 topic (e.g., airports, comic characters). We select 9 topics as in-distribution and the remaining 5 topics as dittr---iitribution. We evaluate the performance using the official evaluation scripts and report the BLEU scores (Papineni et al. 2002a).
Summarization. We use the XSUM Narayan al. 2018) dataset, which an abstractive sum- marization dataset on BBC news articles. Each example is labeled with a news category. We select world, UK and business categories to be in-distribution and let the remaining news categories such as health and technology to be out- -f--iiiibiiittion. We report the ROUGE-2 scores (Lin) 2004).
Open-domain closed-book QA (Open-QA). We use the open-domain variant (Lee et al.) 2019b) of the Natural Questions dataset (Kwiatkowski et al. 2019) as the in-distribution data and the Web Questions dataset (Berant et al. 2013) as the -uttt--distribution data. The distribution shift is induced by different data collection process and their year of collection (roughly 2018 versus 2013). We
For specifics of these methods, see their respective papers.
4

---

Under review as a conference paper at ICLR 2022
report the exact match accuracy. Note that this task, unlike most QA (e.g., SQuAD (Rajpurkar et al. 2016)), does not provide systems with a text from which to extract answers. Roberts et al. (2020) demonstrated that could finetune an LM to generate answers from knowledge stored during pretraining.
4.2 MODEL. ARCHITECTURES AND HYPERPARAMETER
We use GPT-2-medium (Radford et al. 2019) as the pretrained model for table-to-text and open- domain QA tasks. GPT-2 is an autoregressive language model; therefore, we parametrize po(y x) by concatenating x, separator token, and y (e.g., [x; SEP; y)). We define the loss function to sum over the token losses that correspond to y.
We use BART-large (ewiis et al. 2020) as the pretrained model for the summarization task. BART uses an encoder-decoder architecture and parametrizes po(y) x)) by first feeding r to the encoder, and decoding conditioned on the encoder representation.
For open-domain closed-book QA, this is to our knowledge the first time GPT-2 has been finetuned for the task. We used separate development OOD data to test a handful (<10) of hyperparameter settings determine whether we'd ' observe a similar ID/OOD tradeoff to the other tasks. We note this because, in general, one cannot do hyperparameter optimization on OOD data; we did this to provide another testbed for our methods.
Our implementation is based on the Hugging Face Transformers package (Wolf et al. 2020). We use most of the hyperparameters suggested by the Hugging Face default setup. The hyperparameters we tune include the number of epochs, batch size, learning rate, and prefix length (for prefix-tuninge all reported in Table 3 in the appendix.
5 RESULTS OF FULL AND LIGHTWEIGHT FINETUNING
There is trade-off in ID/OOD performance when choosing between full finetuning and lightweight finetuning: full finetuning consistently outperforms in distributiion, whereas lightweight finetuning consistently outperforms dist-diittribttionn (Li & Liang 202 l Lester et al. 2021b. To quantify this
trade-off, we examine the generation performance of both methods on ID and OOD data.
As shown in Table 2 when evaluating in-distribution, full finetuning achieves significantly stronger results than lightweight finetuning in five out of six settings (across datasets and two lightweight finetuning methods. Only prefix-tuning WebNLG achieves comparable accuracy to full finetuning. When evaluating -itto---isttibutionn lightweight finetuning achieves significantly stronger results than full finetuning in five of six settings. Only adapter-tuning on XSUM fails to outperform full finetuning
COMBINING MODELS WITH WEIGHTED ENSEMBLE
As observe in Section 5 full finetuning models attain better ID performance but worse OOD performance, whereas lightweight finetuning models attain better OOD performance at the cost of good ID performance. Is possible achieve both good ID and OOD performance simultaneously? In this section, we show that simple ensemble of the two methods accomplishes this.
First, we train a lightweight finetuning model p(+) and a full finetuning model nD on D At decoding time, we combine the prediction of the p()) and p by interpolating them at each token.
(En) yet) X (0) (ye x. Ver) + A) " (y, | NI,)))
(4)
At each time step I, both models output next token distribution conditioned on the common history Yee. and the two distributions are interpolated with a mixture weight The output is generated by beam searching the ensemble distribution (lins) (ya x, vct).
'Toough it is worth noting again that for open-domain QA, we performed a small amount of manual, exploratory hyperparameter optimization to optimize the ID of full finetuning and the OOD of lightweight finetuning.
5

---

Under review as a conference paper at ICLR 2022
WebNLG (BLEU)
XSUM (ROUGE-2)
OpenQA (EM)
ID
0OD
ID
0OD
ID
0OD
Prefix Prefix-Tuning Tuning
Full fine-tuning (FT)
63.25
30.39 0.64
21.22 0.11
15.47 0.09
16.8 + 0.1
9.8 0.2
Prefix
63.18 0.37
43.75 + 0.33
19.75 + 0.07
16.27 0.03
8.2
11.0 0.1
Ensemble (Prefix)
65.04
0.37
21.99 0.04
16.50
17.2 0.3
11.4
Cocktail (Prefix)
65.35 0.31
43.96 0.53
21.80 0.20
0.09 16.20
17.8 + 0.4
10.6 +
Adapter- Tuning
Full fine-tuning (FT)
63.25 +: 0.42
30.39 + 0.64
21.22 # 0.11
15.47 + 0.09
16.8 + 0.1
9.8 0.2
Adapters
60.43 -: 0.24
48.04 + 0.45
19.04 + 1.06
15.63 + 0.59
9.7 + 0.4
11.1 + 0.4
Ensemble (Adapters)
64.37 1 0.17
46.78 1 0.13
21.72 1 0.35
16.54 0.19
17.3 1 0.2
11.4 0.2
Cocktail (Adapters)
64.61 + 0.17
44.91 + 0.18
21.08 + 0.19
15.73 + 0.08
17.8 + 0.2
10.6 + 0.2
Table 2: Performance of full finetuning. lightweight finetuning, our ensemble, and our cocktail finetuning model, for prefix-tuning (first half) and for adapter-tuning (second half). Cells are highlighted blue if they are statistically significantly better than lightweight finetuning (for ID columns) or full finetuning (for OOD columns) and red otherwise.
6.1 CHOOSING THE MIXTURE WEIGHT X
We start with a set { where A, {O., 0.1, 0.25, 0.5, 0.75, 0.9, 1.0] in this experiment. We choose the mixture weight X that attains the best score on val-
[]. For example we use E
This method of choosing À is useful because (1) it crucially does not use OOD data, which is typically unaccessible at model selection time, and (2) it guarantees that ID performance is not sacrificed, because match II) performance by setting À 0.
6.2 RESULTS
As shown in Table the ensemble matches and slightly outperforms both the ID performance skyline (achieved by full finetuning) and the OOD performance skyline (achieved by lightweight finetuning). Across all datasets, ensembling achieves significantly improved performance over full finetuning OOD, and lightweight finetuning ID. Further, in eleven out of twelve settings (across 3 datasets, methods, and both ID and OOD), ensembling does not perform significantly worse than the better of full and lightweight finetuning.
6.3 RUNTIME AND MEMORY
Despite its ability to combine the best of both worlds, ensembling is expensive inference time because two models are run simultaneously. doubling the amount of GPU memory required. Fur- thermore, forward passes are required, doubling the runtime for decoding For deployment efficiency, it's better to have a single model at inference time.
7
COCKTAIL FINETUNING: COMBINING WITH DISTILLATION
The goal of cocktail finetuning is to approximate the ensemble in Section 6 with a single model
7.1
METHOD
Train lightweight finetuning model (9), Fix mixture weight [0, 1]. Train all parameters of the pretrained model g, (y x) on the following objective. First, define a token-level student-teacher
'The two forward passes parallelizable. but the cost of another 2x GPU memory.
'One might wish to approximate the two with a single lightweight model, a direction we explored but eventually discarded due to the difficulty of approximating a full finetuning model with lightweight finetuning model. Anecdotally, this is evidence towards the speculation of Lester et al. (2021), I), that lightweight finetuning limited in its ability alter pretrained L.M.

---

Under review as a conference paper at ICLR 2022
Input
output
full
prefix
cocktail
In-distribution (NQ)
medical term for the cause of diseese who the newly elected governor of califoria?
edoloogy
cause a disease
etiology
Out-of-distriuttio (WQ)
John H. Cox
Jenyy Brown
Jerry Brown
Figure 2: Examples in the open-domain closed-book QA (wherein text is given from which to extract the answer). The full finetuning model performs well in-distribution but not dustof-distiiuution, the lightweight finetuning model performs well out-of- but not in-distribution, and the cocktail finetuning model performs well on both. The distribution shift (Natural Questions to WebQuestions) is due temporal and data collection differences.
distillation loss at token j:
distill pio i (z. v.j) KL (p)) (ey vea)))
(5)
The objective the sum over the length of of the X-weighted mixture of distillation and log- likelihood loss:
len(y)
(x = B(a.a))~ - Davin
Σ
Adistill pro) v.J) - 1 log xx Mes)
(6)
Let Je.A be the predictor resulting from optimizing (x and estimate the ID score Sja(Jo,)) on D Perform this across a set of mixture weights {Ax h- i- The cocktail finetuning model is the predictor fe,,, that maximizes the in-distribution score Sia fe, A,) across the mixture weights X, [
7.2 RESULTS
As shown in T cocktail finetuning matches and slightly outperforms the ID performance skyline (achieved by full finetuning) and achieves comparable to or slightly worse than the OOD performance skyline (achieved by lightweight finetuning). Across all datasets, cocktail finetuning achieves significantly improved performance over full finetuning OOD, and over lightweight finetuning ID. Further, in seven out of twelve settings (across 3 datasets, 2 methods, and both ID and OOD), cocktail finetuning does not perform significantly worse than the better of full and lightweight finetuning.
ANALYSIS OF THE WEIGHTING OF FULL AND LIGHTWEIGHT FINETUNING
Trade-off' curves. Our ensembling and cock- tail finetuning methods are useful practice because one can choose the mixture coefficient without OOD data. Now, we explore how ID and OOD performance vary function of X. Figure 3 shows results across five random seeds of ensembling and cocktail finetuning our three datasets, each dot colored according smooth
XSUM Distillation: Full FT Teacher
A=D
000 00D ROUGE?
16.5
D.1
16.0
A D.5
A 0.75
15.5
0,9
15.0
its À value. In each plot, we tradeoff curve, in which a range of improve on the OOD, ID, or both of each constituent model. Interestingly, in two of three tasks, there This
14.19.
20.0
20.5 21.0
21.5 22.0 22.5
ID ROUGE2
Figure 4: Ablation on XSUM in which a full finetuning model is distilled into full finetuning model. No benefits are observed.
effectively only one pareto-optimal means the À is best for both for ID and OOD performance, which is why optimizing for ID performance also leads good OOD performance.
Foor cocktail finetuning, use the hyperparameters reported for full finetuning; no hyperparameter opti- mization performed.
7

---

Under review as a conference paper at ICLR 2022
WebNLG; Ensemble (prefix)
XSUM: Ensemble (prefix)
QA: Ensemble (prefix)
46-
17.0
000 performence
92.
16.
000 perfomence
12.0
28.5
R0.
164
11.5
31.0
0.5
16.0
215
6.25
15.8
10.0
32
15.6
9.5
02.5
63.0
63.5
140
64.5
15.0
25.0
20.5
21.3
21.5
22.0
10
ID performance
ID performance
ID performance
WebNLG: cocktail (prefix)
XSUM: cocktail (prefix)
QA: cocktail (prefix)
16.50
12.0
42
performance
16:00
0OD perfomance
40-
15775
11.0
1550-
AJ5
15:25
035
10.0
32-
30
9.5
42.5
53.0 03.5
64.0 64.5
15.5
22.0 21.3 22.0 performance
3.2
performance
performance
Figure 3: For WebNLG and XSUM, ensembles and cocktails both achieve the best of both prefix- tuning and full finetuning for some X; for OpenQA, one can achieve approximately the best of both.
Ablations. One hypothesis to explain the strong performance of ensembles is the known benefits of ensembling two models trained with different random seeds. Likewise for cocktail finctuming, one could point to the effectiveness of born-again networks (Furlanello al., 2018): distilling teacher model to an identically parameterized student with the same model architecture. As an ablation to test this hypothesis, we distill from a full finetuning model into another full finetuning model, again using A to mix the distillation loss with the label loss. The results, in Figure 4 show that this distillation does not improve over a single full finetuning model, for any of the X.
WHY DOES DISTILLATION ON ID DATA IMPROVE OOD?
Prior works have used unlabeled OOD data and self-training (similar distillation) transfer good OOD performance from another model (Raghunathan et al. 2020 Xie et al. 2021). In contrast, cocktail finetuning performs distillation only on ID data. In this section, we provide an explanation for why this can occur in multiclass logistic regression with a large number of classes.
Prediction problem and distribution shift. Motivated by the large output space in language generation, we consider a prediction problem from inputs X' C Rd to labels y = f1,...,kk kb where large. The labeled training data is {(x,, y;)] For convenience, we also let yi be the one-hot vector of length & denoting the correct class. The labels in the training data do not cover all the possible labels, and we let the subset of seen labels be S.
In this problem, the class marginals of the ID and OOD distributions Pa and Pood are different. To have good OOD accuracy, the model must generalize to unseen classes.
Model and losses. We consider training a multi-class logistic regression model using full batch gradient descent where the learned parameters 0 E Rexd In full fine-tuning, run gradient descent on the negative log-likelihood.
In cocktail finetuning, also add distillation loss using soft labels (a probability vector) from the prefix-tuned model, which denote p = P a)) R with parameters 0, given an input Similarly, the predicted class probability vector of the cocktail model given an input æ is denoted
8

---

Under review as a conference paper at ICLR 2022
Ph po(- (x) E R In this setting, the cocktail loss (Equation 6 with parameter [0,
a (x = Σ ((1 A)yk + xp log
The cocktail loss encourages the model to Iearn output mixture of the hard label and the soft label on the training data. The final score for the classification task is computed with respect to the accuracy metric, where score(y. 9) = fo-1 (v. 5) and fo-1 is the 0-1 loss.
Poor OOD performance with full finetuning. Our first result shows that when training logistic regression model using gradient descent (initialized 0) the only labeled training set (as in full finetuning), the learned parameters 03 for an unseen label j & are all the same.
Proposition Let g()) be the parameters iteration t of gradient descent and let (0)) 0. any iteration t and for any unseen label index & S. 0g = 9t where 9t is the same for all j $ S.
As a result, the model cannot distinguish between any of the unseen classes, resulting in poor OOD performance. Suppose that the model outputs fixed, arbitrary unseen class j when the probabilities of the unseen classes are the largest. Then, assuming that the OOD distribution Pood is such that the probability of drawing an example from an unseen class (excluding j) Paca(y & (U)))) is at least [0, 1]. then the OOD accuracy is bounded as
Sood S (y & UU (i)) 1 -
(8)
This OOD accuracy can be low when e is large, which can occur with very large output spaces.
Disillation transfers OOD behavior through soft logits. We investigate how distillation on ID training data can affect OOD behavior. In the following, let X E Raxd be the training data matrix.
Proposition 2. Assume the data matrix X has full column rank. Let 0 be the minimizer of the cocktail loss when À 1, subject to the constraint that 1 exp(0X)) al exp(0X) for some constant > O, such that the model learns the same normalizing constants (up to scaling) as the prefix-tuned model on the training data only. Then
1=(-1x) pa(-1x)
(9)
for any new input x, such that the cocktail model has the same OOD accuracy as the prefix model.
Therefore, if the prefix-tuning model has high OOD accuracy, then Sood is high. The proof proceeds by showing that the extra information the soft logits can help the model recover the parameters of the prefix-tuned model (up to scaling), which in turn inherits its OOD behavior. Thus, in generation problems where the output space is large, distillation on soft logits can transfer information about OOD behavior even through ID data.
10 CONCLUSION
Considered together, our work and the work of Wortsman al. (2021) provide strong cross-modal evidence that simple ensembles of models that have minimally deviated from pretraining (for them, zero-shot; for us. lightweight finetuning models) with full finetuning models provides strong ID and OOD accuracy. The reasons for this "best of both worlds" behavior are still poorly understood, motivating future work. Since ensemble models effectively weight by model confidence (though modulated by A), one explanation may be that lightweight finetuning models are less confident than full finetuning models in-distribution, yet more confident out-of-distribution.
In summary, we provide simple, effective methods for achieving the best of lightweight and full finetuning- strong OOD and ID performance-2i in language generation. Our cocktail finetuning provides system builders with the opportunity optimize for in-distribution performance (in choosing lambda), often beating full finetuning, while resting assured that the out-of- -ittof-disttibution performance (i.e., what they observe with real users) will likely be higher than had they performed full finetuning.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
The methods presented this work have the potential to improve a wide range of natural language generation tasks. Natural language generation has known dual-use issues, like scaling misinformation generation and generating naturalistic text in scams. The WebNLG XSUM, Natural Questions, and WebQuestions datasets we evaluate contain personal information (though public and published) information about real people and organizations.
REPRODUCIBILITY STATEMENT
All code (including experiment scripts and plot creation scripts) will be released upon publication. All datasets used in this paper are publicly available and will be linked in our released code. All reported numbers represent the average of results from random seeds.
REFERENCES
Samira Abnar, Mostafa Dehghani, and Willem Zuidema. Transferring inductive biases through knowledge distillation. arXiv preprint arXiv:2006. 00555, 2020.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1533-1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL / wwwp:/ho..ooy..oo/1111111111
David Berthelot, Rebecca Roclofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin. Adamatch: A unified approach superviiee learning and domain adaptation. CoRR, Abs/2106604432. 2021. URL htw.https://// /tattps://axivviio www.wawwwww....ooaaaaaaaaaa www.oorgggaao.ooo abs/ / /21//216.... 04732
Cristian Bucilä, Rich Caruana, and Alexandru Siculescu-Mizi Model compression. International
Conference on Knowledge Discovery and Data Mining (KDD). 2006.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizii, Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541, 2006.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C. Duchi. Unlabeled data improves adversarial robustness. In Advances Neural Information Processing Systems (NeurIPS), 2019.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference Machine Learning, pp. 1607-1616. PMLR, 2018.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. The WebNLG challenge: Generating text from RDF data. In Proceedings of the 10th International Conference
on Natural Language Generation, pp. 124-133, Santiago de Compostela, Spain, September 2017. Association for Computational Linguisties. doi: 10. 10.18853/11/117-3511 URL https
w/www. www..aclwed. owg. hwtp/a/ttoogggo//o/oooooooootttt
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efffcient transfer learning for NI.P In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning mlr
Research, pp. 2790-2799. PMLR, 09-15 Jun 2019a. URL
press/v97 www.////ooloeo ssy 9a. html
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruma Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for NLP. arXiv, 2019b.
10

---

Under review as a conference paper at ICLR 2022
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 228-339, Mclbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 33653/11/P14 1031. (tl.https:/// ac www.thoooy...oo.......
Mihir Kale and Abhinav Rastogi. Text-to-text pre-training for data-to-text tasks. In Proceedings of the 13th International Conference on Natural Language Generation, pp. 97-102, 2020.
Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen, and Noah Smith. Finetuning pretrained transformers into rans. arXiv preprint arXiv:2103.13076, 2021.
Fereshte Khani and Percy Liang. Removing spurious features can hurt accuracy and affect groups disproportionately. In ACM Conference on Fairness, Accouniability, and Transparency (FAccT), 2021.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming- Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
Kenton Lee, Ming-Wor Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Association for Computational Linguistics (ACL), 2019a.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching. arXiv, 2019b.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficien prompt tuning. 2021.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequencc pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881-,880, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2022.aa -al.aainl-m-mnnmm 703. URL 703.
https: www www.wwlwwbbb...............www
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1tth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582 4597, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021 acI-- URLhttps ac antho org.
353
Chin-Yes Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out. pp. 74 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
URL https: www acl web org/ http://wwww...b...///aaaaooooo//
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ArXiv, abs/2107.13586 2021a.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv: 103.10385. 2021b.
Robert L Logan IV, Ivana Balažević, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. Cutting down on prompts and parameters: Simple few-shot learning with language models. arXiv preprint arXiv:2 13353, 2021.
Amir Najafi, Shin ichi Maeda, Masanori Koyama, and Takeru Miyato. Robustness to adversarial perturbations in learning from incomplete data. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
11

---

Under review as a conference paper at ICLR 2022
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! Topic-aware aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, Brussels, Belgium, 2018.
Deepak Narayanan, Keshav Santhanam, Amar Phanishayee, and Matci Zaharia. Accelerating Deep Learning Workloads through Efficient Multi-Model Execution. In NeurIPS Workshop on Systems for Machine Learning, pp- 20, 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computationnl Linguistics, ACL '02, pp. 311-318, Stroudsburg, PA, USA, 2002a. Association for Computational Linguistics. doi: 10.3115/1073083.10773 URL /ttpsssss/////.oo////////
3115/1073083 1073135
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A method for automatic cvaluation of machine translation. In Association for Computational Linguistics (ACL), 2002b.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning, 2020.
Guanghui Qin and Jason Eisner. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Techmologies (NAACL-HLT), pp. 5203-5212, Online, June 2021, URL puttp://cst http./eccccsssssoopppperrrr/rrrp
2021
A. Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John C. Duchi, and Percy Liang. Understanding and mitigating the tradeoff between robustness and accuracy. In International Conference on Machine Learning (ICML). 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. Empirical Methods in Natural Language Processing (EMNLP), 2016.
Sylvestre-Alvise Rebuffi Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, pp. 506- 516. Curran Associates, Inc., 2017. URL
neurips. cc/paper/ pdf
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418-5422 2020.
Taylor Shin, Yasaman Razeghi, Robert I.. Logan IV au2, Eric Wallace, and Sameer Singh. Au- toprompt: Eliciting knowledge from language models with automatically generated prompts, 2020.
Jonathan Uesato, Lean-Baptiste Alayrac, Po-Sen Huang, Robert Stanforth, Alhussein Fawzi, and Pushmeet Kohli. Are labels required for improving adversarial robustness? In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Prasetya Ajie Utama, Nafise Sadat Moosavi, Victor Sanh, and Iryna Gurevych. Avoiding inference heuristics in few-shot prsept-based finetuning. 2021.
12

---

Under review as a conference paper at ICLR 2022
WehNi (BLEU)
XSUM (ROUGG:2)
OpenQQ (FM)
epuch
Ir
batch size
other
spoch
Ir
batch size
other
epach
butch size
other
Full fine mee-tunnng (FT)
5 10-a
5 10-
5
5 10-
16
Adapters
Prefix
5 10
1-20
15
5 10
64
1-40
2
10
16
1-20
Table Hyperparameters, refers to the dimension of the adapter MLP's middle layer, and refers to the length of the prefix vector in prefix-tuning.
WebNLG (BLEU)
XSUM (ROUGE-2)
OpenQA (EM)
Ensemble (Adapters)
0.5
0.5
Ensemble (Prefix)
0.5
0.25
0.5
Cocktail (Adapters)
0.5
0.25
0,25
Cocktail (Prefix)
0.5
0,25
0.25
Table 4: The selected values of À for the ensembles and cocktail experiments.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-an the-art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Pro- cessing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics. URL (tttttt:: www. web wwwgg//nnoooogyy///oooooooo
emnlp demos 6.
Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. arXiv preprint arXiv:2109.01900 2021.
Sang Michael Xie, Ananya Kumar, Robert Jones, Fereshte Khani, Tengyu Ma, and Percy Liang. In- N-out: Pre-training and self self-training using auxiliary information for outttf-distribution robustness. In International Conference on Learning Representations (ICLR), 2021.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficint fine- tuning for transformer-based masked language-models. CoRR, abs/2106. 101999 2021. URL
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models, 2021.
George Kingsley Zipf. Human behavior and the principle of least effort. 1949.
A HYPERPARAMETERS AND DATASET DETAILS
training
val
test (id)
test (ood)
WehNLG XSUM
18K
2.2K
1.0K
0.9 K
129K
7k
7k
20k
OpenQA
88K
1.8K
1.8K
2k
Table 5: Number of examples in each dataset.
A.1 HYPERPARAMETERS
Relevant hyperparameters beyond the defaults of Huggingface Transformers Wolf et al. (0200) are reported in Table
13

---

Under review as a conference paper at ICLR 2022
WebNLG: Ensemble (adaoter)
XSUM: Ensemble (adapter)
QA: Ensemble (adapter)
45-
13.0
18.3
12.5
000 performence
92.
performnn
000 perfommnnc
12.0
53
R0.
28.0
11.5
25.5
000
20.5
0.55
15.0
32
9.5
02.5
63.0
63.5
140
64.5
15.0
18.5 19.0 19.5 20.0 20.5
21.5 22.0
10
14
ID perforannc
ID perforaanc
ID performance
WebNLG: cocktail (adapter)
XSUM: cocktail (adapter)
QA: cocktail (adapter)
47.5
152
gerfomiiae
a.
15.8
16.0
00D 00D performance
11.5-
42.5-
15.
11.0
40.0-
15.4
10.5
000
035
10.0
30.0
9.5
15.5 39.5 20.0 20.5 performance
performance
performance
Figure 5: For WebNLG and XSUM, Ensembles and cocktails both achieve the best both prefix and full finetuning for some X; for OpenQA, one can achieve approximately the best of both.
A.2 DATASET DETAILS
Table the number of examples in each dataset split.
For the WebNLG dataset, the ID topics include [Astronaut, University, Monument, Building, Comic- sCharacter, Food, Airport, SpporsTeant City, and WrittenWork] and the OOD topics include [Athlete, Artist, MeanofTrransooraaiion, CelestialBody, Politician).
For the XSUM dataset, the ID topics include fuk, business, world news and the OOD topics includes entertainment, election, technology, science, health, education, explainers, live, uk- politics, world-us-canale world-europe, uk-england, uk-scotland, world-asia, science-environment, Mlssootland-scootaaddpliiii disability).
B ADDITIONAL RESULTS
Due to space constraints, we did not include in Figure 3 the tradeoff plots using adapters lightweight finetuning method (instead reporting only those for prefi--tum -tnnig. The plots for adapters are provided in Figure 5
C
ANALYSIS
We restate and expand upon some of the setup for the analysis here.
Prediction problem. We consider a prediction problem from inputs X' C Rd to labels y (I....,k) where is large. The labeled training data is ((x, v:))= For convenience, we also let ys be the one-hot vector of length & denoting the correct class, and will make this clear. The labels in the training data do not cover the possible labels, and we let the subset of seen labels be
Model. We consider training a multi-class logistic regression model using full batch gradient descent. The learned parameters are E kixd and the model prediction given input x is
0_ E, 0
fo(x) = arg max po(x), arg max
(10)
j
14

---

Under review as a conference paper at ICLR 2022
where 8, is the j-th row of 6. In standard fine-tuning, run gradient descent the negative log-likelihood
((A) Σ Vi 1o
(11)
where ys RR is a one-hot vector of the class label.
Distillation loss. In the Cocktail model, we also add distillation loss using soft labels from the prefix-tuned model, which we denote Pá with parameters 6. The Cocktail loss with parameter [0, 1]
(a(A)= = Σ (1 A)u, logrø" + logr5.)
(12)
i=1
Σ ((1 A)и - ) log P
(13)
The Cocktail loss encourages the model to learn to output a mixture of the hard label and the soft label on the training data.
C.I PROOF OF PROPOSITION
Proof. By induction on By zero initialization, the statement holds with gu = 0.
Suppose for iterations up to t, we have 8.0) = 9r for all j & S. We focus on some j & S WLOG. The gradient for 8 is
n
V
00y)
V log Pe
(14)
exp( ) TTx))
Vi +
v exp(pp (Opp TTx)) 24
(15)
Σ
exp((0) xx1)
Itb
(16)
3 exp( ) Tare)
exp(.g) xz)
It since Viş = for as unseen. This gradient does not vary with j. which shows the result.
(17)
Σ, exp((8 (8()) Tx;) xz)
j'
C.2 PROOF PROPOSITION
Proof. Let Ya | denote the predicted class probabilities of the model on the training data, and YG be the class probabilities the prefix-tuned model on training data. With À the gradient of the loss is
Vola()) =
P
x + Ph xx
(18)
im1
= Ygx X + YoX
(19)
Since is the minimizer, this gradient is 0 and thus
YgX Y = Y/ ===Yo Yo for with full column rank (right-invertibll). We the right-inverse of X be Xt, This implies that for all i,
(20)
exp(@x;)
exp(fr:))
(21)
Σ, exp(((,)
exp(0]
15

---

Under review as a conference paper at ICLR 2022
In matrix form, this is equivalent to
exp(0 diag(1 L exp(oxX)) exp(0X) diag(1' exp(OX)))
(22)
= 0x 1log 1 exp((x)) = =6x 1 T exp(eX))
(23)
= 0 = 0 + log T exp(0X) log 1 exp(ox) x
(24)
On a new input x,
Or Or log 1 T exp(0XX log 1 exp(ox)) xtx
(25)
Ar (log(a) + 1 X1x)1
(26)
by using the assumption on normalizing constants. Therefore the probability of the j-th class under the model is
exp(6f.x) x))
(p6), Σj exp(0Ja) exp(0] 2) exp(log(a) +1
(27)
(28)
Σg exp(0]ar exp(log(a) (oclloog@(++100 x
exp(0] 2)
E exp(0].x) (pt)
(29)
16


Reviews:
[{"review_id": "wMzCrD-Obgk", "reviewer": "Reviewer_ioou", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The current paper looks perfect up to section 4. It properly discusses different fine-tuning schema as well as the trade-off between OOD performance and ID performance with respect to the use of a fune-tuning scheme. It properly defines the problem (i.e., the trade-off) and proposes a simple yet effective way to overcome it. It tests its method on three different NLG tasks: Table2Text Generation, Summarisation, and QA.\n\nHowever, starting from section 5, it makes me hard to recommend an acceptance for this paper in the current format. This is because of two major concerns.\n\nOne is that the interpretations of experimental results are problematic, weight might make the rest discussions and conclusions do not stand anymore. In section 5, regarding Table 2, the authors say \"when evaluating ID, full fine-tuning achieves the stronger results, whereas lightweight fine-tuning achieves only 81% of the skyline on average\". This statement is only true \"on average\" and overlooks the huge differences between different tasks. For example, for ID data of the Data2Text task, full fine-tuning achieves a score of 63.25 while lightweight fine-tuning achieve 63.18. There is no significant difference between the performance on ID of these two schema. Meanwhile, in a similar vein, for OOD samples of the summarisation task, I also found no significant difference between the two schema. The situation described by the author only exists when doing question answering. This suggests that such a trade-off is task-dependent to a large extent and it seems to me that if a task relies on the inputs in a more directed way (e.g., QA is often accomplished by simply copying text from inputs), then the trade-off is more significant and the proposed model is more useful. Whereas, if a task relies on the inputs in a more indirect way (e.g., Data2Text requires the generator to plan what to produce in the first place), then the trade-off is less significant and improvement made by the proposed model is less. Anyway, such a phenomenon should not be overlooked and covered by only saying \"on average\".\n\nThe other is that when evaluating the Data2Text generation, only BLEU is used. However, there has been a bank of work that has proved the BLEU has low validity on NLG tasks (e.g., Reiter, 2018). This makes, at least the results on WebNLG, do not reliable.\n\nReference:\nReiter, E. (2018). A structured review of the validity of BLEU. Computational Linguistics, 44(3), 393-401. ", "summary_of_the_paper": "The present paper first discusses the trade-off between performance for out-of-domain data and in-domain data with respect to whether the model is fully fine-tuned or lightweight fine-tuned on NLG tasks. Second, it argues that such a trade-off is not necessary if one can make use of both of these two fine-tuning schema in a clever way. To this end, it proposes cocktail fine-tuning, which augments full fine-tuning via distillation from a lightweight model and which achieves equal performance as an ensemble of the two fine-tuning schema. At length, this paper also explains the behavior of the cocktail fine-tuning through a toy model. ", "summary_of_the_review": "I generally like the idea of this paper and the paper is perfect up to the place where the experiments are introduced. Discussions of this paper overlook major phenomena in the results, making the discussions and, probably, the conclusions are, in part, wrong.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "C5R_JqgIJd", "reviewer": "Reviewer_rP9V", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* I am not convinced and even surprised by the authors that \"ensembling is one of the methods that we propose as well\". I think the ensemble method is existing and NOT proposed by authors, and authors only apply it to the setting lightweight fine-tuning and full fine-tuning.\n* \"In the experiments we present, we focus on monolingual models, and there isn’t much evidence that prefix-tuning works for translation\". The argument is problematic. This paper's title is \"Ensembles and Cocktails: Robust Finetuning for Natural Language Generation\". Since it is for the general Natural Language Generation (NLG) task, I would like to see these results as MT is one of the most important task in NLG. If the proposed method doesn't work for MT, then I would think the paper overclaims its contribution.\n-----------------------------------------------------------------------\nStrengths\n* The idea of combining adapter-finetuning and full-finetuning to handle both in-domain data and out-of-domain data is interesting and clearly motivated.\n* This paper is well written and presents their motivation clearly and finally also provides theoretical analysis on multi-class logistic regression to help understand why it can work.\n\nWeakness\n* Their methods only perform comparably to the ensemble method but not consistently and significantly better. Although cocktail fine-tuning only does inference once, it's not free and needs a knowledge distillation process before training, which requires inference over whole-training data using the adapter-finetuning model and can be time-consuming. The authors should talk about this limitation in the paper.\n* This paper aims at the natural language generation area and should consider adding experiments on datasets for machine translation (MT).  Adding results for machine translation can make this paper more convincing and solid. ", "summary_of_the_paper": "This paper proposes a simple yet effective method, cocktail fine-tuning, for the natural language generation tasks. Their results show that cocktail fine-tuning can handle both In-domain data and Out-of-domain data effectively by combing adapter-finetuning and full-finetuning through knowledge distillation and overall has comparable performance compared to their ensembles. It also provides theoretical analysis on multi-class logistic regression to explain why it works.", "summary_of_the_review": "This paper combines adapter-finetuning and full-finetuning to handle both in-domain data and out-of-domain data and the idea is interesting. It's also well written and clearly motivated. Further, it provides a theoretical analysis of multi-class logistic regression to explain why it works. However, I think the authors overclaimed their contribution. Although this paper's title is \"Ensembles and Cocktails: Robust Finetuning for Natural Language Generation\", it lacks machine translation results, one of the most important tasks in natural language generation. In addition, the authors mention that \"ensembling is one of the methods that we propose as well\", which I don't agree with. I would recommend declining this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "27tPsVQ6La_", "reviewer": "Reviewer_Sg93", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Strengthens:\n- This paper addressed interesting research topic and the proposed method is interesting and promising.\n- This paper is clear and therefore it is easy to follow and to understand.\n- The analyses are interesting and provide insightful information.\n\n* Weaknesses:\n- The experimental setup of this paper might be problematic. Two out of the three tasks are generation tasks in which BLEU or ROUGE-2 evaluation scores are difficult to interpret, i.e. it is very difficult to judge the effectiveness of the proposed method.\n- The experimental results are mixed, i.e. it is difficult to draw strong conclusions from the results. Furthermore, it is not clear where the results are significant different among systems.\n- The definitions of ID and OOD are not well described in this paper and need to be improved. Furthermore, it is not clear how many examples of ID and OOD were used in the datasets. \n\n", "summary_of_the_paper": "This paper presents interesting an idea of combining lightweight fine-tuning and full fine-tuning to achieve the best of both approaches, i.e. perform best on out-of-domain and in-domain data. The authors proposed two approaches: a simple ensemble method and a so-called cocktail fine-tuning that combines two fine-tuning methods in one single model. They evaluated their tasks in three datasets: WebNLG, XSUM and OpenQA and obtained mixed results. The authors also provided good analyses for more insights. ", "summary_of_the_review": "Overall, I rate this paper as marginally above the acceptance threshold, mainly because the idea is interesting and somewhat novel.\nHowever, there are some weaknesses esp. in the experimental setup and results that make this paper a borderline paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "03xXSi1oFRm", "reviewer": "Reviewer_K9aD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**:\n\n- The writing is generally clear.\n- The paper is addressing an important phenomenon that parameter-efficient tuning which has deficient ID performance, while full fine-tuning has worse OOD performance.\n\n**Weakness**:\n\n- First of all, the presented methodology is not well motivated. Parameter-efficient tuning aims to reduce number of parameters that need to save for each task by only fine-tuning a small number of additional parameters. However, the proposed method produces an ensemble model that fine-tunes all parameters, losing the parameter-efficiency. If the goal of this paper is to improve the OOD performance of full fine-tuning, it should compare with methods that improve OOD performance which has a vast amount of related work. And this literature review is also missing from the paper. On the other hand, if the goal of this paper is to improve the ID performance of parameter-efficient tuning, then it should propose a method for improving parameter-efficient tuning methods rather than combine it with full fine-tuning.\n\n- Second, although it's unfair to mention recent work after submission time, there is recent work on parameter-tuning that shows proper improvements over parameter-efficient tuning methods could match the performance of full fine-tuning. This makes the motivation of this paper less meaningful.\n\n- Third, the experiments do not reflect how and why the hyperparameters (length of prefix vectors, bottleneck dimension) of parameter-efficient tuning are selected, which is important for the discussions and conclusions. For example, if with tuned hyperparameters, the ID results of parameter-efficient tuning methods could be close to full fine-tuning while preserving good OOD performance, then the proposed method makes no sense any more.\n\nAnother minor flaw is that when prefix-tuning is introduces, it's described as \"it prepends a sequence of trainable, task-specific prefix vectors to the input\", which actually is not true. Prefix-tuning prepends tunable vectors to the projected key and value vectors instead of the input.\n\nAnother question is, the reported ROUGE-2 score on XSUM is quite low (21.2) on ID compared to the original reported result in BART paper (22.27) on the full test set. I would guess the ID performance is even higher or at lease to this number. Why is it?", "summary_of_the_paper": "This paper proposes an ensemble model between a full fine-tuning model and a parameter-efficient fine-tuning model to improve the out-of-distribution (OOD) performance of a full fine-tuning model. The proposed method is inspired by the observation that full fine-tuning model achieves good in-distribution (ID) performance while parameter-efficient finetuning model achieves better OOD performance. There are two ensembling methods presented in the paper: linear interpolation between the predictions of the two models; and distill from the predictions of a parameter-efficient model with ID training data. Improved OOD performance is observed with this ensemble method.", "summary_of_the_review": "The motivation of this paper is not well-defined. The goal of this paper is to improve the OOD performance of a full-finetuning model, however, it didn't compare with any method on OOD generalization / domain adaptation. The proposed method also makes parameter-efficient tuning method not attractive any more.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for ensembling light fine-tuning methods and full fine-tuning methods to achieve better performance both in-domain and out-of-domain distributions. As authors agree, similar idea has been explored in the computer vision literature. The reviewers like the overall idea of the paper, but they all had some concerns regarding the experiments. The reviewers provide valuable feedback on how to improve the experiments, potentially running the same idea on more datasets and tasks, provide more analyses and discussions on how to understand the results.
Decision: Reject