Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new approach to find a subset of the pretraining corpus that supports BERT’s zero-shot predictions in a given task. This is done by iteratively finding pretraining examples whose gradient is the most similar to that of the downstream task examples. The technique is used to analyze the performance of BERT on IMDB and MNLI, finding that (i) BookCorpus is more relevant than Wikipedia despite the latter being larger, (ii) pretraining examples masking the downstream verbalizers are the most relevant, and (iii) the most relevant pretraining examples are not particularly similar to the downstream task examples on the surface, showing that the model is not relying on pure memorization.
strength_and_weaknesses: STRENGTHS
- The paper tackles an important problem (understanding how pretrained models are able to perform downstream tasks in a zero-shot fashion), and the approach followed is original and technically sound.
- The finding that the context of the supporting data evidence is not particularly similar to the task input data is interesting and not obvious.

WEAKNESSES
- While I really like the general idea of the paper, the analysis itself is rather shallow. The main findings are (i) BookCorpus is more relevant than Wikipedia despite the latter being larger, (ii) pretraining examples masking the downstream verbalizers are the most relevant, and (iii) the most relevant pretraining examples are not particularly similar to the downstream task examples. While (iii) is interesting, (ii) is not surprising at all, and more analysis is necessary to interpret (i). For instance, it could be that the verbalizers (e.g. good and bad) occur more frequently in BookCorpus, in which case (i) would be implied by (ii) and not surprising at all.
- I am not satisfied by the evaluation of the proposed approach in Section 4. ORCA outperforms all baselines in Table 1, but that is not surprising at all, as it is the only approach using labeled downstream data, and it works by picking examples from the pretraining corpus whose gradient is similar, therefore approximating gradient descent in downstream data. Some basic baselines are missing (e.g. only picking examples that mask the downstream verbalizers), and it would be useful to have a variant finetuning on the downstream task as an upper-bound to put the numbers in perspective. It would also be useful to finetune another pretrained model (e.g., RoBERTa) in the same set of examples (selected based on BERT): if these examples are truly supporting the task in question, I would expect them to be helpful for all models. In addition, the results in Table 2 are negative: even if the authors claim that they get an improvement on IMDB, this is likely not significant, as this applies to only one variant of the proposed method and even in that case the improvement is only 0.27 points while the standard deviation is 0.65. This is puzzling and deserves some more discussion: does this imply that all pretraining data is equally useful when using prompt-tuning?
- The manual analysis of the examples is dissapointing. We are only shown a single example for each task and verbalizer. As the authors say, it is not clear at all how the examples mined for MNLI can be useful to learn entailment. The authors say that the examples mined for IMDB do express sentiment, but I think that even that is questionnable (the second one does, but the first one is talking about the genetics of bacteria). I would have liked to see more examples in the appendix and a more systematic analysis of them, with the aim to understand how interpretable the examples are.
- The evaluation is limited to two tasks and a single model, using a very small fraction of the pretraining corpus. While I understand that the authors were constrained by compute, this cannot be use as a cheap excuse for everything. I would have liked to see more extensive experiments and, if truly not feasible due to compute constraints, the authors should at least report the compute required for each experiment to justify this (e.g. if each experiment takes a month on their hardware I would understand not having more, but if it takes a few hours this wouldn’t be a valid excuse).
clarity,_quality,_novelty_and_reproducibility: I think that the paper has enough novelty. The writing is ok although not particularly brilliant, and the mathematical formalization is understandable but could be made more clear. The paper does a good job in terms of reproducibility: I did not miss any critical experimental detail, and the authors promise that the code and data will be released.
summary_of_the_review: I think that this is a borderline paper, perhaps leaning a bit more on the negative said. It tackles an important problem from an original angle and is technically sound, but the analysis presented is rather shallow.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors want to explore a very interesting question why the prompting learning of large pretrained language models leads to strong performance in a variety of downstream tasks, especially in zero-shot setups? This paper proposed a novel method to identify the evidence of the model’s task-specific competence in prompt-based learning based on the general large pre-trained language model. It uses the gradient information related to the downstream task, IORCA can locate a small subset of pretraining data that is similar to the downstream tasks. This work is an very interesting exploration but not very novelty.
strength_and_weaknesses: Strengths:
This paper proposed a gradient-based method to identify the evidence of the model’s task-specific competence in prompt-based learning based on the general large pre-trained language model. It is interesting.


Weaknesses:
Novelty of the proposed method is somewhat limited.
clarity,_quality,_novelty_and_reproducibility: The Clarity and Quality is Good!
summary_of_the_review: Same with Summary Part.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for finding the subset of a large language model (LLM) training corpus that is most responsible for the 0-shot performance of the LLM on a given task. The method involves finding examples whose gradient is similar to the gradient of task-specific supervised data. This is done in an iterative process in which the original LLM is fine-tuned on the examples found during the previous step, in order to account for interactions among examples. Performance is evaluated by fine-tuning the LLM on the identified subset, then measuring task performance. 

On IMDB and MNLI tasks with BERT, the proposed method is found to outperform random-sampling and kNN baselines. The most striking finding in the resulting analysis is that most of the valuable examples come from the Books portion of BERT’s training corpus.

strength_and_weaknesses: Strengths:

1. The problem is important, as insights can inform selection of training data for future LLMs. The proposed technique thus has the potential to be useful in many different contexts.

2. The idea is novel. Although gradient-based similarity is widely used, this specific application is new, as far as I know.

3. The paper is very well written: the technique is described clearly and precisely, and the experiments should be easy to reproduce.

Weaknesses:

1. Although the technique is intuitive and easy to understand, it’s not clear to what extent it’s a good approximation of the true quantity of interest, namely performance if the LLM were retrained from scratch on the whole corpus with the identified subset removed. Influence functions obviously get at that more directly, and I think the paper should have included an influence-function baseline. This needn’t be an iterative procedure (which might be difficult to formulate), just the set of individual examples with the greatest influence.

2. The experiments are somewhat limited: only on BERT, and only on two tasks. They are also limited in exploring only tiny subsets. It’s interesting to know that fine-tuning on such a set can provide a good performance boost (at least in the case IMDB), but it would be better to have a sense of where this flattens out as the subset is expanded.

clarity,_quality,_novelty_and_reproducibility: See strengths and weaknesses.
summary_of_the_review: The paper is a bit thin on substance, both in terms of theoretical justification and breadth of experimental results. However, it attacks a very central problem, is clearly written and proposes a simple, potentially very useful technique, so I am leaning in favour of acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors try to locates a very small subset of pretraining data that directly supports the model’s predictions in a given task. The main idea is to use cosine similarity between gradients of the downstream samples and pre-training corpus.  They designed experiments based on 0.5% of the Wikipedia and BookCorpus over IMDB and MNLI.
strength_and_weaknesses: Strength:
1. The authors provide a novel metric to locate pretraining data that supports the model’s predictions in a given task.
2. Table 1 shows that using retrieved corpora can effectively improve the effectiveness of LMs in downstream tasks.

Weaknesses:
1. the problem is not well defined. A corpus that improves the effectiveness of a downstream task cannot necessarily be used to explain the predictions of the original language model. While many of the claims in this paper are about the interpretability of pretraining by locating training corpora, the experiments are mostly about performance improvement.
2. If the goal is to improve model effects by some continued pretraining, the related work and experiments in this paper lack comparison with state-of-the-art methods, e.g. dense retriever methods like REALM.
3. The method has some assumptions. However, they are not validated. For example, whether continuing training on the task data directly will improve the original model? Why the gradient of the sample to be found is similar to the gradient of the target task samples?
4. I think the method in this paper cannot be scaled to large LMs. In LMs, there are at least millions of gradients for each sample.
5. Some experiments are confusing. For example, I cannot understand why words like good/bad/yes/no happen to be masked out in Table 3. This is not consistent with the random mask strategy.
6. Table 3 is inconsistent with the intuition of this paper. According to the examples in Table 3, it looks like the model is looking for verbalizer words, instead of the training corpora that explain the model behavior in the downstream task.
7. I suggest that the authors use more downstream tasks for their experiments.
clarity,_quality,_novelty_and_reproducibility: Please refer to the Strength And Weaknesses.
summary_of_the_review: Overall, I think the problem is not clearly defined. This causes the inconsistencie in the method and experimental descriptions. In addition, there is an issue with the choice of baselines.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper attempts to interpret the performance of prompt-based model. The authors proposed to use the gradient of a downstream task to identify the samples that contribute the most to the task. Fine-tuning with theses samples will further improve the task performance, although the samples may not resemble the task.

All reviewers agree that the problem is interesting. However, concerns are raised about the formulation and experiments. We agreed that the paper may not meet the bar of ICLR.
justification_for_why_not_higher_score: While the problem is interesting, it is done in a superficial manner with concerns about the task formulation and experimentation.
Decision: Reject