Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an ensemble model between a full fine-tuning model and a parameter-efficient fine-tuning model to improve the out-of-distribution (OOD) performance of a full fine-tuning model. The proposed method is inspired by the observation that full fine-tuning model achieves good in-distribution (ID) performance while parameter-efficient finetuning model achieves better OOD performance. There are two ensembling methods presented in the paper: linear interpolation between the predictions of the two models; and distill from the predictions of a parameter-efficient model with ID training data. Improved OOD performance is observed with this ensemble method.
main_review: **Strengths**:

- The writing is generally clear.
- The paper is addressing an important phenomenon that parameter-efficient tuning which has deficient ID performance, while full fine-tuning has worse OOD performance.

**Weakness**:

- First of all, the presented methodology is not well motivated. Parameter-efficient tuning aims to reduce number of parameters that need to save for each task by only fine-tuning a small number of additional parameters. However, the proposed method produces an ensemble model that fine-tunes all parameters, losing the parameter-efficiency. If the goal of this paper is to improve the OOD performance of full fine-tuning, it should compare with methods that improve OOD performance which has a vast amount of related work. And this literature review is also missing from the paper. On the other hand, if the goal of this paper is to improve the ID performance of parameter-efficient tuning, then it should propose a method for improving parameter-efficient tuning methods rather than combine it with full fine-tuning.

- Second, although it's unfair to mention recent work after submission time, there is recent work on parameter-tuning that shows proper improvements over parameter-efficient tuning methods could match the performance of full fine-tuning. This makes the motivation of this paper less meaningful.

- Third, the experiments do not reflect how and why the hyperparameters (length of prefix vectors, bottleneck dimension) of parameter-efficient tuning are selected, which is important for the discussions and conclusions. For example, if with tuned hyperparameters, the ID results of parameter-efficient tuning methods could be close to full fine-tuning while preserving good OOD performance, then the proposed method makes no sense any more.

Another minor flaw is that when prefix-tuning is introduces, it's described as "it prepends a sequence of trainable, task-specific prefix vectors to the input", which actually is not true. Prefix-tuning prepends tunable vectors to the projected key and value vectors instead of the input.

Another question is, the reported ROUGE-2 score on XSUM is quite low (21.2) on ID compared to the original reported result in BART paper (22.27) on the full test set. I would guess the ID performance is even higher or at lease to this number. Why is it?
summary_of_the_review: The motivation of this paper is not well-defined. The goal of this paper is to improve the OOD performance of a full-finetuning model, however, it didn't compare with any method on OOD generalization / domain adaptation. The proposed method also makes parameter-efficient tuning method not attractive any more.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents interesting an idea of combining lightweight fine-tuning and full fine-tuning to achieve the best of both approaches, i.e. perform best on out-of-domain and in-domain data. The authors proposed two approaches: a simple ensemble method and a so-called cocktail fine-tuning that combines two fine-tuning methods in one single model. They evaluated their tasks in three datasets: WebNLG, XSUM and OpenQA and obtained mixed results. The authors also provided good analyses for more insights. 
main_review: * Strengthens:
- This paper addressed interesting research topic and the proposed method is interesting and promising.
- This paper is clear and therefore it is easy to follow and to understand.
- The analyses are interesting and provide insightful information.

* Weaknesses:
- The experimental setup of this paper might be problematic. Two out of the three tasks are generation tasks in which BLEU or ROUGE-2 evaluation scores are difficult to interpret, i.e. it is very difficult to judge the effectiveness of the proposed method.
- The experimental results are mixed, i.e. it is difficult to draw strong conclusions from the results. Furthermore, it is not clear where the results are significant different among systems.
- The definitions of ID and OOD are not well described in this paper and need to be improved. Furthermore, it is not clear how many examples of ID and OOD were used in the datasets. 


summary_of_the_review: Overall, I rate this paper as marginally above the acceptance threshold, mainly because the idea is interesting and somewhat novel.
However, there are some weaknesses esp. in the experimental setup and results that make this paper a borderline paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The present paper first discusses the trade-off between performance for out-of-domain data and in-domain data with respect to whether the model is fully fine-tuned or lightweight fine-tuned on NLG tasks. Second, it argues that such a trade-off is not necessary if one can make use of both of these two fine-tuning schema in a clever way. To this end, it proposes cocktail fine-tuning, which augments full fine-tuning via distillation from a lightweight model and which achieves equal performance as an ensemble of the two fine-tuning schema. At length, this paper also explains the behavior of the cocktail fine-tuning through a toy model. 
main_review: The current paper looks perfect up to section 4. It properly discusses different fine-tuning schema as well as the trade-off between OOD performance and ID performance with respect to the use of a fune-tuning scheme. It properly defines the problem (i.e., the trade-off) and proposes a simple yet effective way to overcome it. It tests its method on three different NLG tasks: Table2Text Generation, Summarisation, and QA.

However, starting from section 5, it makes me hard to recommend an acceptance for this paper in the current format. This is because of two major concerns.

One is that the interpretations of experimental results are problematic, weight might make the rest discussions and conclusions do not stand anymore. In section 5, regarding Table 2, the authors say "when evaluating ID, full fine-tuning achieves the stronger results, whereas lightweight fine-tuning achieves only 81% of the skyline on average". This statement is only true "on average" and overlooks the huge differences between different tasks. For example, for ID data of the Data2Text task, full fine-tuning achieves a score of 63.25 while lightweight fine-tuning achieve 63.18. There is no significant difference between the performance on ID of these two schema. Meanwhile, in a similar vein, for OOD samples of the summarisation task, I also found no significant difference between the two schema. The situation described by the author only exists when doing question answering. This suggests that such a trade-off is task-dependent to a large extent and it seems to me that if a task relies on the inputs in a more directed way (e.g., QA is often accomplished by simply copying text from inputs), then the trade-off is more significant and the proposed model is more useful. Whereas, if a task relies on the inputs in a more indirect way (e.g., Data2Text requires the generator to plan what to produce in the first place), then the trade-off is less significant and improvement made by the proposed model is less. Anyway, such a phenomenon should not be overlooked and covered by only saying "on average".

The other is that when evaluating the Data2Text generation, only BLEU is used. However, there has been a bank of work that has proved the BLEU has low validity on NLG tasks (e.g., Reiter, 2018). This makes, at least the results on WebNLG, do not reliable.

Reference:
Reiter, E. (2018). A structured review of the validity of BLEU. Computational Linguistics, 44(3), 393-401. 
summary_of_the_review: I generally like the idea of this paper and the paper is perfect up to the place where the experiments are introduced. Discussions of this paper overlook major phenomena in the results, making the discussions and, probably, the conclusions are, in part, wrong.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a simple yet effective method, cocktail fine-tuning, for the natural language generation tasks. Their results show that cocktail fine-tuning can handle both In-domain data and Out-of-domain data effectively by combing adapter-finetuning and full-finetuning through knowledge distillation and overall has comparable performance compared to their ensembles. It also provides theoretical analysis on multi-class logistic regression to explain why it works.
main_review: * I am not convinced and even surprised by the authors that "ensembling is one of the methods that we propose as well". I think the ensemble method is existing and NOT proposed by authors, and authors only apply it to the setting lightweight fine-tuning and full fine-tuning.
* "In the experiments we present, we focus on monolingual models, and there isn’t much evidence that prefix-tuning works for translation". The argument is problematic. This paper's title is "Ensembles and Cocktails: Robust Finetuning for Natural Language Generation". Since it is for the general Natural Language Generation (NLG) task, I would like to see these results as MT is one of the most important task in NLG. If the proposed method doesn't work for MT, then I would think the paper overclaims its contribution.
-----------------------------------------------------------------------
Strengths
* The idea of combining adapter-finetuning and full-finetuning to handle both in-domain data and out-of-domain data is interesting and clearly motivated.
* This paper is well written and presents their motivation clearly and finally also provides theoretical analysis on multi-class logistic regression to help understand why it can work.

Weakness
* Their methods only perform comparably to the ensemble method but not consistently and significantly better. Although cocktail fine-tuning only does inference once, it's not free and needs a knowledge distillation process before training, which requires inference over whole-training data using the adapter-finetuning model and can be time-consuming. The authors should talk about this limitation in the paper.
* This paper aims at the natural language generation area and should consider adding experiments on datasets for machine translation (MT).  Adding results for machine translation can make this paper more convincing and solid. 
summary_of_the_review: This paper combines adapter-finetuning and full-finetuning to handle both in-domain data and out-of-domain data and the idea is interesting. It's also well written and clearly motivated. Further, it provides a theoretical analysis of multi-class logistic regression to explain why it works. However, I think the authors overclaimed their contribution. Although this paper's title is "Ensembles and Cocktails: Robust Finetuning for Natural Language Generation", it lacks machine translation results, one of the most important tasks in natural language generation. In addition, the authors mention that "ensembling is one of the methods that we propose as well", which I don't agree with. I would recommend declining this paper.
review_title: null

Meta-review: This paper presents a method for ensembling light fine-tuning methods and full fine-tuning methods to achieve better performance both in-domain and out-of-domain distributions. As authors agree, similar idea has been explored in the computer vision literature. The reviewers like the overall idea of the paper, but they all had some concerns regarding the experiments. The reviewers provide valuable feedback on how to improve the experiments, potentially running the same idea on more datasets and tasks, provide more analyses and discussions on how to understand the results.
Decision: Reject