Year: 2021

Abstract:
When finetuning a pretrained language model for natural language generation tasks, one is currently faced with a tradeoff. Lightweight finetuning (e.g., prefixtuning, adapters), which freezes all or most of the parameters of the pretrained model, has been shown to achieve stronger out-of-distribution (OOD) performance than full finetuning, which tunes all of the parameters. However, lightweight finetuning can underperform full finetuning in-distribution (ID). In this work, we present methods to combine the benefits of full and lightweight finetuning, achieving strong performance both ID and OOD. First, we show that an ensemble of the lightweight and full finetuning models achieves the best of both worlds: performance matching the better of full and lightweight finetuning, both ID and OOD. Second, we show that we can achieve similar improvements using a single model instead of two with our proposed cocktail finetuning, which augments full finetuning via distillation from a lightweight model. Finally, we provide some explanatory theory in a multiclass logistic regression setting with a large number of classes, describing how distillation on ID data can transfer the OOD behavior of one model to another.

text: 1 INTRODUCTION
When finetuning a pretrained language model for natural language generation tasks like summarization (Narayanan et al., 2018) and table-to-text (Gardent et al., 2017), one is currently faced with a tradeoff. One can achieve strong in-distribution (ID) performance or strong out-of-distribution (OOD) performance—but not both—by choosing one of two families of finetuning approaches. Finetuning all parameters of the pretrained language model achieves strong ID performance (Howard & Ruder, 2018). We call this full finetuning. Alternatively, freezing most of the pretrained parameters during finetuning, in e.g., adapters (Rebuffi et al., 2017; Houlsby et al., 2019a), prefix-tuning (Li & Liang, 2021; Lester et al., 2021) and bitfit (Zaken et al., 2021), achieves stronger OOD performance than full finetuning (Li & Liang, 2021; Lester et al., 2021), but worse ID performance. We call these methods lightweight finetuning.
In this work, we are the first to demonstrate that this tradeoff is not necessary in natural language generation: we can achieve the best of full and lightweight finetuning both ID and OOD. We test this across three
natural language generation tasks in English: summarization (Narayanan et al., 2018), table-to-text (Gardent et al., 2017), and open-domain question answering (Roberts et al., 2020; Lee et al., 2019a), and two lightweight finetuning methods: adapters and prefix-tuning. Achieving the best of both worlds is important since across our 6 settings (2 lightweight finetuning methods and 3 datasets),
full finetuning significantly outperforms lightweight finetuning ID in 5 settings, and lightweight finetuning significantly outperforms full finetuning OOD in 6 settings.
We present two methods for achieving the best of full and lightweight finetuning. We first show that a weighted ensemble of one full finetuning model and one lightweight finetuning model achieves performance comparable to the best of the two, both ID and OOD. To avoid running two models at once, we propose cocktail finetuning: first train a lightweight finetuning model, and then train a full finetuning model on a mixture of student-teacher distillation loss (Bucilă et al., 2006; Hinton et al., 2015) (with the lightweight model as teacher) and cross-entropy loss, on the original ID data.1 The ensemble weight (resp. the cocktail mixture loss weight) is chosen to optimize for ID performance, and we observe that the resultant model achieves comparable OOD performance to lightweight finetuning.
Our work contributes to an ongoing realization in machine learning: robust models that deviate minimally from a pretrained model can be gainfully combined with a model adapted more precisely to a specific dataset. Most saliently, in concurrent work, Wortsman et al. (2021) find that zeroshot image recognition through CLIP (Radford et al., 2021) can be combined with a finetuned network to perform well ID and OOD. Our work demonstrates that similar results hold true in natural language generation as well. We draw from simple, well-known methods: ensembles, student-teacher distillation (Abnar et al., 2020; Hinton et al., 2015)—but our application to achieving the best of ID and OOD performance in natural language generation is novel.
To better understand how cocktail finetuning can transfer the out-of-distribution behavior of the lightweight finetuning model when distilling only on in-distribution data, we theoretically study multiclass logistic regression with a large number of output classes, viewing language generation as a classification problem over a large output space. We study a distribution shift setting where not all classes are seen in the ID training data (as is true in language generation due to Zipf’s law (Zipf, 1949)), and getting good OOD accuracy requires generalizing to unseen outputs. First, we prove that a model trained with gradient descent cannot distinguish the unseen output classes, leading to poor OOD accuracy. Second, we prove that distilling a teacher model with good OOD accuracy on only ID training data can transfer the OOD behavior of the teacher to the student, such that the student and teacher make the same prediction on any new input. As a result, the student also has good OOD accuracy.
In summary, we present two simple methods—ensembling and cocktail finetuning—for achieving strong ID and OOD natural language generation with only the original ID data, achieving the “best of both worlds” of full and lightweight finetuning methods.

heading: 2 RELATED WORK
text: Lightweight finetuning. Recently, as pretrained models scale up in the number of parameters, lightweight finetuning techniques have been developed to avoid updating all of the parameters. These techniques update a subset of the parameters (Zaken et al., 2021), introduce a small number of new parameters between layers of the model (called “adapters”) (Rebuffi et al., 2017; Houlsby et al., 2019b; Pfeiffer et al., 2020), or learn auxiliary inputs that are prepended to the models inputs (called “prompts”) (Shin et al., 2020; Li & Liang, 2021; Qin & Eisner, 2021; Liu et al., 2021b; Logan IV et al., 2021; and see Liu et al., 2021a for a survey). These techniques empirically have the benefit of being able to achieve strong out-of-distribution extrapolation performance at the cost of worse in-distribution performance compared to full finetuning models both in classification and generation settings (Li & Liang, 2021; Lester et al., 2021; Zhou et al., 2021; Utama et al., 2021). In our work, we aim to overcome this trade-off by proposing techniques for achieving the best of both worlds, with a focus on language generation.
Student-teacher distillation. Buciluǎ et al. (2006) introduce distillation for compressing large ensembles into smaller, faster, more efficient models, and Hinton et al. (2015) generalize their method and call it knowledge distillation. Kasai et al. (2021) distill Transformers into RNNs for linear-time inference. Abnar et al. (2020) show that knowledge distillation can transfer the inductive bias of one architecture to another: for example, distilling an LSTM to a Transformer leads to a Transformer with
1We use the term cocktail as it is a refreshing mixture involving distillation.
representations that are more similar to independently trained LSTMs than independently trained Transformers. The efficiacy of our proposed cocktail finetuning is surprising even in context of these results, as it empirically transfers just the strong OOD performance of the teacher lightweight model to the student, while not transferring the weak ID performance.
Robust finetuning in vision. In concurrent work, Wortsman et al. (2021) propose a similar approach to ours, but focus on image classification: they combine a full finetuning model with a lightweight finetuning (linear probe) model by ensembling the two models in weight space, and show that their approach maintains both good ID and OOD performance. It is yet surprising that in the structured prediction setting of natural language generation, we further show that we can achieve high OOD performance even when weight-space ensembling is impossible (for instance, since there are weights not held in common between the full finetuning model and an optimized prompt). Moreover, we further this line of work by providing a theoretical study into how distillation can transfer OOD behavior while only using ID data.
Self-training for robustness. In adversarial robustness, standard models achieve good accuracy on clean examples but are not robust to adversarial examples, while adversarially trained models are more robust but are less accurate on clean examples. Raghunathan et al. (2020) show that the benefits of these two models can be combined using robust self-training (RST) (Carmon et al., 2019; Uesato et al., 2019; Najafi et al., 2019) on unlabeled data to improve clean and robust accuracy. Similarly, Khani & Liang (2021) show that RST can be used to combine models with spurious features removed (which are more robust) and without removing spurious features (which are more accurate). In general OOD robustness, In-N-Out (Xie et al., 2021) uses self-training to transfer the robustness benefits of a model pre-trained with auxiliary self-supervision to a model trained with extra auxiliary features. Berthelot et al. (2021) use unlabeled data and a consistency regularization loss for domain adaptation. While many of these methods combine the benefits of two models or use a self-training/distillation loss to improve robustness, cocktail finetuning differs in that it performs distillation only on ID training data without any additional ID or OOD unlabeled examples.

heading: 3 PROBLEM STATEMENT

heading: 3.1 SETUP
text: We consider a conditional generation task from an input space X (e.g., a news article) to an output space Y (e.g., a summary of the article), where both spaces consist of sequences of tokens from a shared vocabulary V .
Data. Let Pid and Pood denote the distribution of (x, y) pairs in-distribution and out-of-distribution, respectively, where x ∈ X and y ∈ Y . In summarization, Pid may consist of articles about world and business news while Pood consists of health and technology news. The training data Dtrain and validation data Dval consist of n and nval in-distribution (x, y) pairs respectively, drawn from Pid. Model and metrics. Let qγ(y | x) = ∏len(y) j=1 qγ(yj | x, y<j) be a pretrained neural autoregressive sequence model with parameters γ. Finetuning (full or lightweight) produces a finetuned model pθ(y | x) from qγ , where θ denotes the set of trainable parameters. In full finetuning, θ is the same set of parameters as γ; in lightweight finetuning, θ often refers to the parameters of inserted modules (e.g., adapters modules or prefixes). The final predictor is fθ(x) which approximates argmaxy pθ(y | x) through, e.g., beam search. The goal is to learn parameters θ such that the predictor fθ obtains good ID and OOD performance:
Sid(fθ) = Ex,y∼Pid [score(y, fθ(x))] (1) Sood(fθ) = Ex,y∼Pood [score(y, fθ(x))] (2)
where score may be defined differently for each task (e.g., exact match score or BLEU (Papineni et al., 2002b)).

heading: 3.2 BASELINE FINE-TUNING METHODS
text: Full finetuning has been the defacto approach to adapt pretrained language models for downstream generation tasks (Lewis et al., 2020; Kale & Rastogi, 2020). Under this framework, the model pθ is initialized with its pretrained parameters γ, and we optimize the following log-likelihood objective.
L(θ) = E(x,y)∼Dtrain [− log pθ(y | x)] (3)
where the set of trainable parameters θ refer to the set of pretrained parameters γ.
Lightweight finetuning is another family of adaptation approaches, which freezes most of the pretrained parameters and augments the model with small trainable modules. Lightweight finetuning shares the same training objective as full finetuning, but the trainable parameters θ are different. Within the lightweight finetuning family, we focus on adapter-tuning (Houlsby et al., 2019a) and prefix-tuning (Li & Liang, 2021). Both approaches keep the pretrained parameters intact. Adaptertuning inserts task-specific MLP layers between each layer of the pretrained language models, and the trainable parameter θ consists of these inserted MLP parameters. Prefix-tuning prepends a sequence of trainable, task-specific prefix vectors to the input, and θ consists of these prefix parameters.2

heading: 4 EXPERIMENT SETUP

heading: 4.1 DATASETS AND METRICS
text: Table-to-text. We evaluate on the WebNLG dataset (Gardent et al., 2017), where the input x is a sequence of (entity, predicate, entity) triples and the output y is a natural language description that covers the input information. Each example in this dataset is labeled with one of 14 topic (e.g., airports, comic characters). We select 9 topics as in-distribution and the remaining 5 topics as out-of-distribution. We evaluate the performance using the official evaluation scripts and report the BLEU scores (Papineni et al., 2002a).
Summarization. We use the XSUM (Narayan et al., 2018) dataset, which is an abstractive summarization dataset on BBC news articles. Each example is labeled with a news category. We select world, UK and business categories to be in-distribution and let the remaining news categories such as health and technology to be out-of-distribution. We report the ROUGE-2 scores (Lin, 2004).
Open-domain, closed-book QA (Open-QA). We use the open-domain variant (Lee et al., 2019b) of the Natural Questions dataset (Kwiatkowski et al., 2019) as the in-distribution data and the Web Questions dataset (Berant et al., 2013) as the out-of-distribution data. The distribution shift is induced by different data collection process and their year of collection (roughly 2018 versus 2013). We
2For specifics of these methods, see their respective papers.
report the exact match accuracy. Note that this task, unlike most QA (e.g., SQuAD (Rajpurkar et al., 2016)), does not provide systems with a text from which to extract answers. Roberts et al. (2020) demonstrated that one could finetune an LM to generate answers from knowledge stored during pretraining.

heading: 4.2 MODEL ARCHITECTURES AND HYPERPARAMETERS
text: We use GPT-2-medium (Radford et al., 2019) as the pretrained model for table-to-text and opendomain QA tasks. GPT-2 is an autoregressive language model; therefore, we parametrize pθ(y | x) by concatenating x, a separator token, and y (e.g., [x; SEP; y]). We define the loss function to sum over the token losses that correspond to y.
We use BART-large (Lewis et al., 2020) as the pretrained model for the summarization task. BART uses an encoder-decoder architecture and parametrizes pθ(y | x) by first feeding x to the encoder, and decoding y conditioned on the encoder representation.
For open-domain closed-book QA, this is to our knowledge the first time GPT-2 has been finetuned for the task. We used separate development OOD data to test a handful (<10) of hyperparameter settings to determine whether we’d observe a similar ID/OOD tradeoff to the other tasks. We note this because, in general, one cannot do hyperparameter optimization on OOD data; we did this to provide another testbed for our methods.
Our implementation is based on the Hugging Face Transformers package (Wolf et al., 2020). We use most of the hyperparameters suggested by the Hugging Face default setup. The hyperparameters we tune include the number of epochs, batch size, learning rate, and prefix length (for prefix-tuning), all reported in Table 3 in the appendix.

heading: 5 RESULTS OF FULL AND LIGHTWEIGHT FINETUNING
text: There is a trade-off in ID/OOD performance when choosing between full finetuning and lightweight finetuning: full finetuning consistently outperforms in-distribution, whereas lightweight finetuning consistently outperforms out-of-distribution (Li & Liang, 2021; Lester et al., 2021). To quantify this trade-off, we examine the generation performance of both methods on ID and OOD data.
As shown in Table 2, when evaluating in-distribution, full finetuning achieves significantly stronger results than lightweight finetuning in five out of six settings (across 3 datasets and two lightweight finetuning methods. Only prefix-tuning on WebNLG achieves comparable accuracy to full finetuning. When evaluating out-of-distribution, lightweight finetuning achieves significantly stronger results than full finetuning in five of six settings. Only adapter-tuning on XSUM fails to outperform full finetuning.3

heading: 6 COMBINING MODELS WITH A WEIGHTED ENSEMBLE
text: As we observe in Section 5, full finetuning models attain better ID performance but worse OOD performance, whereas lightweight finetuning models attain better OOD performance at the cost of good ID performance. Is it possible to achieve both good ID and OOD performance simultaneously? In this section, we show that a simple ensemble of the two methods accomplishes this.
First, we train a lightweight finetuning model p(`) and a full finetuning model p(f) on Dtrain. At decoding time, we combine the prediction of the p(`) and p(f) by interpolating them at each token.
p(Ens)(yt | x, y<t) = λ · p(`)(yt | x, y<t) + (1− λ) · p(f)(yt | x, y<t) (4)
At each time step t, both models output a next token distribution conditioned on the common history y<t, and the two distributions are interpolated with a mixture weight λ. The output is generated by beam searching the ensemble distribution p(Ens)(yt | x, y<t).
3Though it is worth noting again that for open-domain QA, we performed a small amount of manual, exploratory hyperparameter optimization to optimize the ID of full finetuning and the OOD of lightweight finetuning.
6.1 CHOOSING THE MIXTURE WEIGHT λ
We start with a set of {λi}Ki=1 where λi ∈ [0, 1]. For example we use λ ∈ {0., 0.1, 0.25, 0.5, 0.75, 0.9, 1.0} in this experiment. We choose the mixture weight λi that attains the best score on Dval.
This method of choosing λ is useful because (1) it crucially does not use OOD data, which is typically unaccessible at model selection time, and (2) it guarantees that ID performance is not sacrificed, because we can match ID performance by setting λ = 0.

heading: 6.2 RESULTS
text: As shown in Table 2, the ensemble matches and slightly outperforms both the ID performance skyline (achieved by full finetuning) and the OOD performance skyline (achieved by lightweight finetuning). Across all datasets, ensembling achieves significantly improved performance over full finetuning OOD, and lightweight finetuning ID. Further, in eleven out of twelve settings (across 3 datasets, 2 methods, and both ID and OOD), ensembling does not perform significantly worse than the better of full and lightweight finetuning.

heading: 6.3 RUNTIME AND MEMORY
text: Despite its ability to combine the best of both worlds, ensembling is expensive at inference time because two models are run simultaneously, doubling the amount of GPU memory required. Furthermore, two forward passes are required, doubling the runtime for decoding.4 For deployment efficiency, it’s better to have a single model at inference time.

heading: 7 COCKTAIL FINETUNING: COMBINING WITH DISTILLATION
text: The goal of cocktail finetuning is to approximate the ensemble in Section 6 with a single model.5

heading: 7.1 METHOD
text: Train a lightweight finetuning model p(`). Fix mixture weight λ ∈ [0, 1]. Train all parameters of the pretrained model qγ(y | x) on the following objective. First, define a token-level student-teacher
4The two forward passes are parallelizable, but at the cost of another 2x GPU memory. 5One might wish to approximate the two with a single lightweight model, a direction we explored but eventually discarded due to the difficulty of approximating a full finetuning model with a lightweight finetuning model. Anecdotally, this is evidence towards the speculation of Lester et al. (2021), that lightweight finetuning is limited in its ability to alter a pretrained LM.
distillation loss at token j: distillp(`)(x, y, j) = KL ( p(`)(yj | x, y<j)‖pθ(yj | x, y<j)) ) (5)
The objective is the sum over the length of y of the λ-weighted mixture of distillation and loglikelihood loss:
Lλ(θ) = E(x,y)∼Dtrain [ len(y)∑ j=1 λdistillp(`)(x, y, j)− (1− λ) log pθ(yj | x, y<j) ]
(6)
Let fθ,λ be the predictor resulting from optimizing on Lλ(θ) and estimate the ID score Sid(fθ,λ) on Dval. Perform this across a set of mixture weights {λi}i. The cocktail finetuning model is the predictor fθ,λi that maximizes the in-distribution score Sid(fθ, λi) across the mixture weights λi. 6

heading: 7.2 RESULTS
text: As shown in Table 2, cocktail finetuning matches and slightly outperforms the ID performance skyline (achieved by full finetuning) and achieves comparable to or slightly worse than the OOD performance skyline (achieved by lightweight finetuning). Across all datasets, cocktail finetuning achieves significantly improved performance over full finetuning OOD, and over lightweight finetuning ID. Further, in seven out of twelve settings (across 3 datasets, 2 methods, and both ID and OOD), cocktail finetuning does not perform significantly worse than the better of full and lightweight finetuning.

heading: 8 ANALYSIS OF THE WEIGHTING OF FULL AND LIGHTWEIGHT FINETUNING
text: Trade-off curves. Our ensembling and cocktail finetuning methods are useful in practice because one can choose the mixture coefficient λ without OOD data. Now, we explore how ID and OOD performance vary as a function of λ. Figure 3 shows results across five random seeds of ensembling and cocktail finetuning on our three datasets, each dot colored according to its λ value. In each plot, we see a smooth tradeoff curve, in which a range of λ improve on the OOD, ID, or both of each constituent model. Interestingly, in two of three tasks, there is effectively only one pareto-optimal λ. This means the λ is best for both for ID and OOD
performance, which is why optimizing for ID performance also leads to good OOD performance.
6For cocktail finetuning, we use the hyperparameters reported for full finetuning; no hyperparameter optimization was performed.
Ablations. One hypothesis to explain the strong performance of ensembles is the known benefits of ensembling two models trained with different random seeds. Likewise for cocktail finetuning, one could point to the effectiveness of born-again networks (Furlanello et al., 2018): distilling a teacher model to an identically parameterized student with the same model architecture. As an ablation to test this hypothesis, we distill from a full finetuning model into another full finetuning model, again using λ to mix the distillation loss with the label loss. The results, in Figure 4, show that this distillation does not improve over a single full finetuning model, for any of the λ.

heading: 9 WHY DOES DISTILLATION ON ID DATA IMPROVE OOD?
text: Prior works have used unlabeled OOD data and self-training (similar to distillation) to transfer good OOD performance from another model (Raghunathan et al., 2020; Xie et al., 2021). In contrast, cocktail finetuning performs distillation only on ID data. In this section, we provide an explanation for why this can occur in multiclass logistic regression with a large number of classes.
Prediction problem and distribution shift. Motivated by the large output space in language generation, we consider a prediction problem from inputs X ⊆ Rd to labels Y = {1, . . . , k} where k is large. The labeled training data is {(xi, yi)}ni=1. For convenience, we also let yi be the one-hot vector of length k denoting the correct class. The labels in the training data do not cover all the possible labels, and we let the subset of seen labels be S. In this problem, the class marginals of the ID and OOD distributions Pid and Pood are different. To have good OOD accuracy, the model must generalize to unseen classes.
Model and losses. We consider training a multi-class logistic regression model using full batch gradient descent where the learned parameters are θ ∈ Rk×d. In full fine-tuning, we run gradient descent on the negative log-likelihood.
In cocktail finetuning, we also add a distillation loss using soft labels (a probability vector) from the prefix-tuned model, which we denote px
θ̃ = pθ̃(· | x) ∈ Rk with parameters θ̃, given an input x.
Similarly, the predicted class probability vector of the cocktail model given an input x is denoted
pxθ = pθ(· | x) ∈ Rk. In this setting, the cocktail loss (Equation 6) with parameter λ ∈ [0, 1] is
`ct(θ) = n∑ i=1 ((1− λ)yi + λpxiθ̃ ) > log pxiθ . (7)
The cocktail loss encourages the model to learn to output a mixture of the hard label and the soft label on the training data. The final score for the classification task is computed with respect to the accuracy metric, where score(y, ŷ) = 1− `0−1(y, ŷ) and `0−1 is the 0-1 loss.
Poor OOD performance with full finetuning. Our first result shows that when training a logistic regression model using gradient descent (initialized at 0) on the only labeled training set (as in full finetuning), the learned parameters θj for an unseen label j /∈ S are all the same.
Proposition 1. Let θ(t) be the parameters at iteration t of gradient descent and let θ(0) = 0. At any iteration t and for any unseen label index j /∈ S, θj = gt where gt is the same for all j /∈ S.
As a result, the model cannot distinguish between any of the unseen classes, resulting in poor OOD performance. Suppose that the model outputs a fixed, arbitrary unseen class j ∈ S when the probabilities of the unseen classes are the largest. Then, assuming that the OOD distribution Pood is such that the probability of drawing an example from an unseen class (excluding j) Pood(y /∈ S∪{j}) is at least c ∈ [0, 1], then the OOD accuracy is bounded as
Sood ≤ 1− Pood(y /∈ S ∪ {j}) ≤ 1− c (8)
This OOD accuracy can be low when c is large, which can occur with very large output spaces.
Disillation transfers OOD behavior through soft logits. We investigate how distillation on ID training data can affect OOD behavior. In the following, let X ∈ Rn×d be the training data matrix. Proposition 2. Assume the data matrixX has full column rank. Let θ be the minimizer of the cocktail loss when λ = 1, subject to the constraint that 1> exp(θX) = α1> exp(θ̃X) for some constant α > 0, such that the model learns the same normalizing constants (up to scaling) as the prefix-tuned model on the training data only. Then
pθ(· | x) = pθ̃(· | x) (9)
for any new input x, such that the cocktail model has the same OOD accuracy as the prefix model.
Therefore, if the prefix-tuning model has high OOD accuracy, then Sood is high. The proof proceeds by showing that the extra information in the soft logits can help the model recover the parameters of the prefix-tuned model (up to scaling), which in turn inherits its OOD behavior. Thus, in generation problems where the output space is large, distillation on soft logits can transfer information about OOD behavior even through ID data.

heading: 10 CONCLUSION
text: Considered together, our work and the work of Wortsman et al. (2021) provide strong cross-modal evidence that simple ensembles of models that have minimally deviated from pretraining (for them, zero-shot; for us, lightweight finetuning models) with full finetuning models provides strong ID and OOD accuracy. The reasons for this “best of both worlds” behavior are still poorly understood, motivating future work. Since ensemble models effectively weight by model confidence (though modulated by λ), one explanation may be that lightweight finetuning models are less confident than full finetuning models in-distribution, yet more confident out-of-distribution.
In summary, we provide simple, effective methods for achieving the best of lightweight and full finetuning—strong OOD and ID performance—in language generation. Our cocktail finetuning provides system builders with the opportunity to optimize for in-distribution performance (in choosing lambda), often beating full finetuning, while resting assured that the out-of-distribution performance (i.e., what they observe with real users) will likely be higher than had they performed full finetuning.
ETHICS STATEMENT
The methods presented in this work have the potential to improve a wide range of natural language generation tasks. Natural language generation has known dual-use issues, like scaling misinformation generation and generating naturalistic text in scams. The WebNLG XSUM, Natural Questions, and WebQuestions datasets we evaluate on contain personal information (though public and published) information about real people and organizations.
REPRODUCIBILITY STATEMENT
All code (including experiment scripts and plot creation scripts) will be released upon publication. All datasets used in this paper are publicly available and will be linked in our released code. All reported numbers represent the average of results from 5 random seeds.

heading: A HYPERPARAMETERS AND DATASET DETAILS
text: A.1 HYPERPARAMETERS
Relevant hyperparameters beyond the defaults of Huggingface Transformers Wolf et al. (2020) are reported in Table 3.
A.2 DATASET DETAILS
Table 5 shows the number of examples in each dataset split.
For the WebNLG dataset, the ID topics include {Astronaut, University, Monument, Building, ComicsCharacter, Food, Airport, SportsTeam, City, and WrittenWork} and the OOD topics include {Athlete, Artist, MeanOfTransportation, CelestialBody, Politician}. For the XSUM dataset, the ID topics include {uk, business, world } news and the OOD topics includes { entertainment, election, technology, science, health, education, explainers, live, ukpolitics, world-us-canada, world-europe, uk-england, uk-scotland, world-asia, science-environment, uk-scotland-scotland-politics, disability}.

heading: B ADDITIONAL RESULTS
text: Due to space constraints, we did not include in Figure 3 the tradeoff plots using adapters lightweight finetuning method (instead reporting only those for prefix-tuning.) The plots for adapters are provided in Figure 5.

heading: C ANALYSIS
text: We restate and expand upon some of the setup for the analysis here.
Prediction problem. We consider a prediction problem from inputs X ⊆ Rd to labels Y = {1, . . . , k} where k is large. The labeled training data is {(xi, yi)}ni=1. For convenience, we also let yi be the one-hot vector of length k denoting the correct class, and will make this clear. The labels in the training data do not cover all the possible labels, and we let the subset of seen labels be S.
Model. We consider training a multi-class logistic regression model using full batch gradient descent. The learned parameters are θ ∈ Rk×d and the model prediction given input x is
fθ(x) = argmax j pθ(x)j = argmax j θ>j x∑ j′ θ > j′x
(10)
where θj is the j-th row of θ. In standard fine-tuning, we run gradient descent on the negative log-likelihood
`(A) = − n∑ i=1 y>i log p xi θ (11)
where yi ∈ Rk is a one-hot vector of the class label.
Distillation loss. In the Cocktail model, we also add a distillation loss using soft labels from the prefix-tuned model, which we denote pθ̃ with parameters θ̃. The Cocktail loss with parameter λ ∈ [0, 1] is
`ct(A) = n∑ i=1 ( (1− λ)y>i log p xi θ + λ(p xi θ̃ )> log pxiθ ) (12)
= n∑ i=1 ((1− λ)yi + λpxiθ̃ ) > log pxiθ (13)
The Cocktail loss encourages the model to learn to output a mixture of the hard label and the soft label on the training data.
C.1 PROOF OF PROPOSITION 1
Proof. By induction on t. By zero initialization, the statement holds with g0 = 0.
Suppose for iterations up to t, we have θ(t)j = gt for all j /∈ S . We focus on some j /∈ S WLOG. The gradient for θ(t)j is
∇ θ (t) j `(θ(t)) = ∇ θ (t) j
( −
n∑ i=1 y>i log p xi θ
) (14)
= n∑ i=1 −yijxi + exp((θ (t) j ) >xi)∑ j′ exp((θ (t) j′ ) >xi) xi (15)
= n∑ i=1 exp((θ (t) j ) >xi)∑ j′ exp((θ (t) j′ ) >xi) xi (16)
= n∑ i=1 exp(g>t xi)∑ j′ exp((θ (t) j′ ) >xi) xi (17)
since yij = 0 for all i as j is unseen. This gradient does not vary with j, which shows the result.
C.2 PROOF OF PROPOSITION 2
Proof. Let Yθ ∈ Rn×k denote the predicted class probabilities of the model on the training data, and Yθ̃ be the class probabilities of the prefix-tuned model on training data. With λ = 1, the gradient of the loss is
∇θ`ct(θ) = n∑ i=1 −pxi θ̃ x>i + p xi θ x > i (18)
= −Yθ̃X + YθX (19) Since θ is the minimizer, this gradient is 0 and thus
Yθ̃X = YθX =⇒ Yθ̃ = Yθ (20) for X with full column rank (right-invertible). We let the right-inverse of X be X†. This implies that for all i,
exp(θ̃xi)∑ j exp(θ̃ > j xi) = exp(θxi)∑ j exp(θ > j xi) . (21)
In matrix form, this is equivalent to
exp(θ̃X) · diag(1> exp(θ̃X))−1 = exp(θX) · diag(1> exp(θX))−1 (22) =⇒ θ̃X − 1(log 1> exp(θ̃X))> = θX − 1(log 1> exp(θX))> (23)
=⇒ θ = θ̃ + 1 [ log 1> exp(θX) log 1> exp(θ̃X) ] X†. (24)
On a new input x,
θx = θ̃x+ 1 [ log 1> exp(θX) log 1> exp(θ̃X) ] X†x (25)
= θ̃x+ (log(α) + 1>X†x)1 (26)
by using the assumption on normalizing constants. Therefore the probability of the j-th class under the model is
(pxθ )j = exp(θ>j x)∑ j′ exp(θ > j′x)
(27)
= exp(θ̃>j x) exp(log(α) + 1 >X†x)∑ j′ exp(θ̃ > j′x) exp(log(α) + 1 >X†x) (28) = exp(θ̃>j x)∑ j′ exp(θ̃ > j′x) = (px θ̃ )j (29)

Review summaries:
This paper proposes a simple yet effective method, cocktail fine-tuning, for the natural language generation tasks. Their results show that cocktail fine-tuning can handle both In-domain data and Out-of-domain data effectively by combing adapter-finetuning and full-finetuning through knowledge distillation and overall has comparable performance compared to their ensembles. It also provides theoretical analysis on multi-class logistic regression to explain why it works.

This paper proposes an ensemble model between a full fine-tuning model and a parameter-efficient fine-tuning model to improve the out-of-distribution (OOD) performance of a full fine-tuning model. The proposed method is inspired by the observation that full fine-tuning model achieves good in-distribution (ID) performance while parameter-efficient finetuning model achieves better OOD performance. There are two ensembling methods presented in the paper: linear interpolation between the predictions of the two models; and distill from the predictions of a parameter-efficient model with ID training data. Improved OOD performance is observed with this ensemble method.

This paper presents interesting an idea of combining lightweight fine-tuning and full fine-tuning to achieve the best of both approaches, i.e. perform best on out-of-domain and in-domain data. The authors proposed two approaches: a simple ensemble method and a so-called cocktail fine-tuning that combines two fine-tuning methods in one single model. They evaluated their tasks in three datasets: WebNLG, XSUM and OpenQA and obtained mixed results. The authors also provided good analyses for more insights. 

The present paper first discusses the trade-off between performance for out-of-domain data and in-domain data with respect to whether the model is fully fine-tuned or lightweight fine-tuned on NLG tasks. Second, it argues that such a trade-off is not necessary if one can make use of both of these two fine-tuning schema in a clever way. To this end, it proposes cocktail fine-tuning, which augments full fine-tuning via distillation from a lightweight model and which achieves equal performance as an ensemble of the two fine-tuning schema. At length, this paper also explains the behavior of the cocktail fine-tuning through a toy model.