Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a method for selecting subsets of prompted multitask training data for fine-tuning  a language model (Data-Efficient FineTuning, or DEFT). Given a small set of unlabeled task examples, the paper uses nearest neighbors to retrieve a targeted task subset from a larger pool of prompted examples. Using this method to build a more data-efficient training set for fine-tuning, the paper reports improvements using a T5 model an equiv. 3B T0 model (outperforming in 11/12 tasks) while using <= 2% of the same P3 training data. Further improvements are reported for few-shot fine-tuning. 
strength_and_weaknesses: Strengths
- Optimizing fine-tuning for improved data efficiency (or more targeted task improvement) is an important research area.
- Nice connections to retrieval augmented language models with corresponding retrieval experiments.

Weaknesses
- The approach is extremely simple. This could also be a strength, but I wonder how much using only a single source (P3) of prompts is biasing results, since many templates used for creating that dataset are very similar across tasks. Expanding results across prompted datasets would be a good series of additional experiments. 
- Given the hypotheses around the the role of negative interference in hurting performance (and the resulting benefit of querying more similar examples for fine-tuning), the paper would be much stronger if there were direct experiments exploring this
- Some of the model naming conventions are confusing/inconsistent (DEFT-T5-base vs DEFT-base). It wasn't clear to me what DEFT-base was defined as (vs XL which was the 3B param variants).

clarity,_quality,_novelty_and_reproducibility: - Some of the model naming conventions could be clearer
- The method (nearest neighbor augmented retrieval) isn't novel, though providing a detailed empirical assessment of its performance is a good contribution
- Uses public data for reproducibility, though unclear if code will be shared (though again, the methods are straightforward)
summary_of_the_review: I think this is a reasonable empirical paper, though I have concerns about how well results would generalize outside of P3-style prompts (which tend to be shorter than other instruction tuning type datasets). This combined with the extreme simplicity of the method undercuts some of my enthusiasm for the acceptance, though I think the empirical work is generally solid.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper is interested in building language models that can generalize to new tasks when there is only unlabeled data for that task. The authors introduce a method called DEFT, where unlabeled instances for the task at hand are used to retrieve the most similar labeled prompted data in a pool of prompted instances of NLP tasks (such as the one used to train T0). The retrieved labeled prompted instances are then used to fine-tune an encoder-decoder language model (more specifically T5 LM), and the model is evaluated on the target task. Experiments show that the resulting DEFT model outperforms T0 at the same model size by a strong margin. When a few labeled data is available for the target task, the DEFT models provide a better initialization for few-shot fine-tuning than T0.

The authors use these data points to highlight that only a subset of the pool of prompted instances is required to achieve high performance on the target task and to highlight negative transfer between tasks when doing massively prompted fine-tuning (i.e. FLAN/T0).

strength_and_weaknesses: Strengths:
- The problem is well posed, the method is clearly explained while being relatively novel. 
- Numbers are strong and experiments well conducted and results are interesting.

Weaknesses:
- I have a few doubts about some of the stronger claims in the paper (in particular the negative transfer argument, see questions).
- Few-shot fine-tuning has a long history in NLP. I’d like to see a fairer comparison to stronger baselines. IA3 is particular in the sense that it only fine-tunes a fraction of the parameters. Using a parameter efficient fine-tuning baseline needs a better justification (especially since DEFT-Few seems to be fully fine-tuned).

clarity,_quality,_novelty_and_reproducibility: Questions and comments:
- How did you handle imbalances in the P3 dataset? P3 has large subsets that can be 100x bigger than other subsets. If these imbalances are not properly handled, DEFT could be overly sampling (random) or retrieving from the most represented subsets rather than the most relevant subsets (because the most relevant subsets are potentially small). In term, the “negative transfer” argument becomes less convincing.
While I agree, it’s hard to re-train T0-3B on a well balanced P3, I’d like to see the random sampling baseline with a balanced P3 if it’s not already the case.
- Introduction: “These findings suggest that prior work may have underestimated the cross-task generalization capabilities of large multitask models”. I think this is a generous claim or at least imprecise (if prior works mean T0/FLAN): while previous works have focussed on the true zero-shot case (where there is no knowledge of the target task, nor through its prompt or through any labeled or unlabeled instances), the setup you are interested in requires access to target task (unlabeled or labeled). It seems that a given DEFT model optimized for target task A would perform well on a target task A, but worse on a target task B, whereas T0/FLAN would perform relatively worse than a “specialized” DEFT but better on average.
- Could you showcase a few retrieved examples and their associated query?
- Missing reference: https://arxiv.org/abs/2005.00770

summary_of_the_review: Strong paper with well-executed experiments and a somehow novel story. Numbers are relatively strong.
Some minor doubts and questions about the experiments.
Recommend accept.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a simple data-efficient fine-tuning technique, where the few-shot data for a test task is augmented by retrieving similar examples from a multi-task training set (which does not contain the exact test task). The test task training set is augmented by taking the union of the training examples retrieved for each example of the test task. The authors operate in a setting where only unlabeled data for a test task is available to do retrieval and a setting where also labeled data is available. The method demonstrates gains on the test datasets used to evaluate T0, when compared to multi-task baselines that are either applied zero-shot to test tasks (e.g., T0) or are fine-tuned on the few-shot test task data (e.g., T-Few).

strength_and_weaknesses: Strengths:
- *The approach is simple and effective*: The main idea is simple and effective - perform a "task-attention" on a set of training tasks to augment the test task few-shot data. It is quite interesting to see that a naive example retrieval approach such as taking the union of the retrieved training examples can lead to significant gains (64.6 vs 60.4 on T0 test tasks for the few-shot setting).

- *The paper is clear and well-written*

Weaknesses:

- *Limited novelty w.r.t. related work*: One of the main weaknesses of this paper is the close similarity to ReCross (Li et al., 2022) to appear in NeurIPS 2022, and published on ArXiv last April. ReCross introduces the same idea: retrieving from a set of upstream tasks training data to benefit the downstream task data. The authors acknowledge this similarity in the related works, by stating that their approach differ from ReCross because ReCross "*make use of an additional re-ranker and multi-task trained model to retrieve useful instances.*". Per se, not using a modeling choice of a previously published paper is *not* a significant novelty: the novelty and contribution would come from showing that the modeling choice used in ReCross *is not necessary* or even *harmful* for achieving good performance in a different setting. These conclusions are currently missing from the paper. Given the good scores the paper reports on the T0 suite, I can imagine that the reranker might be only marginally useful, but this should be clearly stated and empirically supported. The community goes fast and probably the authors started their work before having noticed the relevant related work. Nevertheless, at the moment, this paper presents the idea as totally new. It would be more suitable, for example, to acknowledge the similarity to Li et. al, 2022 directly in the intro and maybe make an extra effort to offer a simplification/ablation of some components of the aforementioned approach. Some of the questions I can think of that would give a bit of novelty and require a small additional effort are:
1) Is the reranker used in ReCross useful?
2) What happens if you fine-tune a multi-task trained model (T0-3B in your case, similar to ReCross), rather than a "generalist" model (T5-XL-LM)?

- *Experimental setting can be stronger*: Differently from ReCross, the authors only test on the T0 evaluation suite and build heavily on top of IA3 (Liu et al, 2022). I do believe that another benchmark is necessary. It would be interesting for example to explore the proposed method with Natural Instructions V2 dataset. For example, this would give a better idea if the method works due to the nature of the prompts format in P3 (e.g . it retrieves examples with similar prompts) or due to other mechanisms.

- *Claims about task interference*: The authors claim that their result show that there is negative task interference. I would think that an important experiment to substantiate this claim is to report the results for 2) above, where the method is applied on top of a multi-task model trained on the multi-task training set.
clarity,_quality,_novelty_and_reproducibility: - It would be nice to have Figure 3 and Figure 4 for the few-shot setting, and compare the performance to the T0-3B+IA3 when fine-tuned on the original test task data vs T5-XL-LM fine-tuned on original test task data + 2x, 3x, 4x of augmented data.
- The applicability of the method outside of NLP tasks might be discussed in the conclusion, e.g. in vision or multi-modal settings.
summary_of_the_review: This paper proposes a simple and effective idea for improving zero-shot / few-shot task generalization. The reported results are convincing. Unfortunately, the overall contribution of this paper is diminished by the limited novelty with respect to ReCross (to appear at NeurIPS, 2022, and published on arXiv in April). The authors cite this paper in the related work, but I feel that the similarity is quite understated. It is questionable whether a paper published in April should be considered related work or not and whether this should impact my review. In the current context, I would suggest the authors to do a small extra effort to 1) highlight similarities with ReCross and 2) differentiate their contribution w.r.t. ReCross, for example, by reporting ReCross performance and ablating some of the ReCross components (e.g. the reranker and the fact that ReCross uses the multi-task model as base model). Right now, the paper appears to be written as if it was the first in proposing this retrieval idea. I didn't discover the similarity until I read the ReCross work in details. All in all, I find the idea exciting: the strong similarity to a previous work, despite this being quite recently published online, requires the paper to offer new perspectives that are not present just yet; experimental setting could be made stronger by adding an additional dataset (e.g. NIv2), to show more convincingly which aspects of the method are responsible for improved performance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new approach to multitask learning by retrieving relevant training examples from P3 and train the model on it. 
strength_and_weaknesses: ### Strength
1. This paper proposes a simple yet effective idea on how to use existing resources (i.e., P3) to facilitate unseen tasks.
2. The paper is well-written and easy to follow.
3. The evaluation is good - it considers both 0-shot and few-shot settings.

### Weaknesses
1. The idea is not completely novel. Similar approaches exist like kNN-LM. Also, the proposed approach is also related to Mixture-of-Experts.
2. It is unknown if the task retrieval mechanism in the paper is optimal. For example, what if we use BM25 for retrieval? Also is task embedding relevant here?
3. The proposed method has its drawback. It requires retrieval and training for each task. In other words, for 3 tasks there are 3 models trained instead of the open-box T0. In Table 1, DEFT outperforms T0 but the improvement is somewhat limited. 
clarity,_quality,_novelty_and_reproducibility: The paper is clear, but I doubt it can be easily reproduced by a third party given the scale of data involved. I recommend the authors release their code.
summary_of_the_review: This paper explores a simple yet effective way to facilitate multitask data but may lack novelty and practicality.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces a new method to improve zero/few-shot generalisation of large language models. Starting from a small set of unlabelled examples, the method retrieves similar labelled examples from a large collections of diverse tasks, which are then used to fine-tune the language model. The authors show that this technique leads to competitive results, compared to fine-tuning the model on the full set of examples, and suggest that there is a negative interference between tasks.

While the reviewers found the results encouraging, and liked the general direction of data curation, they also raised concerns which I tend to share. More specifically, reviewers are concerned by the baselines, which are mostly multi-task, while the proposed method trains a different model for each task. Second, the observed effect might be due to the unbalanced nature of the P3 dataset: when sampling uniformly over tasks (instead of uniformly over examples), the “rand” baseline improved a lot, from 45.9 to 50.4. This lead to the question of how well this baseline would do by increasing the number of examples (instead of limiting to the same number as the proposed method). Similarly, on Super-Natural Instructions, the method does not perform as well as the baseline. For these reasons, I believe that more empirical evidence is necessary to demonstrate the effectiveness of the method, and recommend to reject the paper.
justification_for_why_not_higher_score: I have concerns regarding the experimental setup of the paper, and the fact that some claims made in the paper might be due to the imbalance of the used dataset (P3), instead of negative interference between tasks. I believe that some additional experiments done during the discussion go in that direction, and thus, a more thorough empirical evaluation is needed before acceptance. 
justification_for_why_not_lower_score: None
Decision: Reject