Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors present ReCross, a retrieval augmentation method to improve cross-task generalization of seq2seq models. Starting with a T5-like model ("upstream model") trained on a diverse set of tasks (in seq2seq format), the training data is then embedded using the encoder's top layer and stored in a dense index. To evaluate on an unseen task, the query is used to retrieve training data examples that might be helpful in evaluating this new example (using a two-stage retrieve+rerank process). These helpful examples are used to fine-tune the upstream model, and the model is finally used to perform the target task.
strengths_and_weaknesses: Strengths:
* This is a simple but effective method for allowing a model to retrieve helpful previously seen training data, and re-learn from it. As the authors say, there are probably approaches that would be more effective than continuous fine-tuning. However, it's impressive how well such a simple method already works.
* The authors provide a reasonable set of baselines and an informative choice of ablations (including those in the Appendix). We see the importance of choosing a way of embedding examples that is compatible with the upstream model's internal representation. We also see the contribution of the reranker to the overall performance, and the interactions of various central hyperparameters.

Weaknesses:
* ~One crucial issue I see with this work is that the authors create a new metric (SoftEM) and use it throughout the paper without properly motivating it. They refer to the Appendix, stating "We discuss more on this selection with illustrative examples in Appendix". No such discussion or examples are found in the Appendix, as far as I can tell. It's not clear whether the proposed method would still perform as well using a more standard metric.~ EDIT: Fixed in the latest draft
questions: * How does ReCross compare to the other methods of Table 1 if using exact match as the metric? I see it's implemented in the code, so I assume this was tested?
limitations: The authors adequately address in section 3.5 the fact that their continuous fine-tuning approach is quite simple.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents an approach called ReCross for fine-tuning an LLM by retrieving seen-task data that is particularly relevant to unlabelled inputs from an unseen task. The seen-task examples that are most relevant are retrieved from an index which contains representations from a model that has been fine-tuned on the seen-task data. Following the retrieval, a re-ranker is used to score each retrieved example-query pair, and the top K are used for fine-tuning the model before evaluating on the unseen target task. Experiments on a range of NLU tasks including classification, QA etc. show that ReCross outperforms instruction-tuning baselines.
strengths_and_weaknesses: Strengths:  
This paper shows that carefully selecting data from related tasks for fine-tuning before zero-shot evaluations can help to improve performance. This is sort of a test-time adaptation of the model parameters at a coarser granularity than at an example-level. This means that the retrieval cannot be based on semantic similarity which is why the model used for creating the retrieval index has been fine-tuned on a large pre-selected subset of the seen-task data. The entire setup being explored is an important and high-impact problem to study, and retrieval augmentation provides a compelling solution worth exploring as done in this paper.   
Also, the Bart0 vs. T0-3B result is quite interesting. 

Weaknesses:  
- The paper is missing a qualitative discussion on what sorts of examples are retrieved and what sorts of features might be in play for retrieving useful similar examples. Figure 3 doesn’t seem to be providing too many intuitions for this. This makes it a bit hard to understand where the improvements are coming from.
- The role of the reranker is unclear. Improvements over and above the simpler version of ReCross without the reranker seem small, mostly non-existent given the standard deviation, which is bizarre especially given how complex the whole distant supervision setup is. Would it be better to just remove this from the main paper and replace it with more analysis, data-related ablations and discussion on where the improvements are coming from? ReCross without the distant supervision is still interesting and perhaps a bit simpler. 
- The conclusion from the random retrieval baseline that says “This suggests that it is promising to study better retrieval methods to get consistent improvement in overall performance” seems to only hold true for winogrande..maybe it isn't the right conclusion? Random retrieval is a control experiment to remove confounds and understand how to interpret the improvement from ReCross. E.g. h-swag, anli_r3, squadv2 do not benefit from ReCross. 
- The paper, while comprehensive, can sometimes be verbose. Perhaps some revisions would help to pare it down. Figures 1 and 2, as is, may not be helping the reader to understand the approach. Is the example in Figure 1 an actual retrieval from the amazon polarity and gigawords tasks? These don’t seem like they should help, should they? Maybe cherry-picking a clearer example could help with this one? Figure 2 might benefit from being broken down into stages.   

EDIT: Updating the score from 4-->6 after the author response. 
questions: Please see the previous section which includes suggestions as well.
limitations: Missed the section discussing limitations, where is it included?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper considers the problem unsupervised cross-task generalization of multi-task LMs such as T0, where only a few unlabeled examples from the target task are available. It proposes to retrieve, from the multi-task training corpus, samples that are likely to help the target task using the unlabeled target task inputs, and fine-tune the LM on these retrieved samples. 
To retrieve helpful samples for a target task, they propose a retriever-ranker pipeline, where the more efficient retriever first retrieve, from the entire training set, an initial set of candidate samples, using the last-layer encoder representations from the upstream LM as embeddings for retrieval. It then trains a reranker using a meta-learning inspired approach, aiming to find training samples that are more likely to help an unseen target task.
They show that this retrieval-augmented approach achieves better cross-task generation than the original LM, as well as a naive retrieval approach based on semantic similarity (using SentenceBERT embeddings).
strengths_and_weaknesses: Strengths:

- The paper is clearly written. The problem setting and proposed approach is lucidly presented.

- Given that no target task labels are available in the "unsupervised" setting, the proposed method is sound, and produces non-trivial gains over the applying the vanilla upstream LM to unseen tasks.

Weaknesses:

- My main concern is the practicality or usefulness of this "unsupervised" setting. Given that only a few target task samples are required (the paper uses 16), the cost of obtaining the inputs vs. (inputs + labels) would probably not differ that much in the real world. It would at least be helpful to compare to the "supervised" cross-task generalization methods to show how much the gap is. And if this approach lags significantly behind a few-shot supervised one, it is arguably less expensive / time-consuming to simply label a few samples than to apply a complicated retriever-ranker pipeline.

EDIT after rebuttal: The added Appendix D.2 in the few-shot setting is welcome, and to some extent alleviates my concern here. Though it does not fully address this issue: Collecting labels is more expensive on certain tasks for sure, but this work only considers the tasks where the answer can be evaluated via exact match (e.g. multiple choice, classification, etc.). Arguably the cost for collecting 16 inputs vs. 16 (inputs + labels) does not differ that much. In addition, the concern over the availability of prompt templates for unseen tasks remains. Few-shot methods can transfer to new tasks with a few inputs + labels. I'm not convinced that this is inferior to the proposed paradigm of requiring a prompt template and a few inputs.
questions: Q1: Does this work assume the availability of prompt templates for the unseen target tasks? If I understand correctly, these templates are part of the "input" of the target task, which are assumed to exist? If this is correct, then it amplifies my main concern in the Weakness section. It is arguably more difficult / expensive to come up with these natural language prompts that work well for the upstream LM for each unseen task than to label a few samples.
limitations: N/A
review_title: null

Meta-review: metareview: This paper presents an approach called ReCross that improves zero-shot task performance by retrieving and fine-tuning on examples of similar supervised tasks. This method is shown to help multi-task finetuned models when evaluated zero-shot on novel tasks.

The interesting finding of the paper is that fine-tuning on relevant examples from different but possibly related tasks can help. This finding can help researchers in the areas of zero-shot learning and multitask models.

Otherwise, the method although conceptually simple, includes significant additional machinery, which likely makes it practically difficult to use as the reviewers point out. Similarly, the relative contribution of the re-ranking step seems small and the steps appears to add significant complexity. As one of the reviewers points out, the paper and the method may be clearer without that step.

The review process included a lengthy and productive discussion, which helped the paper clarify and improve on several points. As result two of the reviewers increased their scores. There is now consensus among the three reviewers that the paper should be accepted.


confidence: Certain
Decision: Accept