ORCA: Interpreting Prompted Language Models via Locating Supporting Evidence in the Ocean of Pretraining Data
Authors: Xiaochuang Han
Yulia Tsvetkov
Venue: Submitted to ICLR 2023 (2023)
Keywords: interpretability
prompting language models
pretraining data as evidence

Abstract: Prompting large pretrained language models leads to strong performance in a variety of downstream tasks. However, it is still unclear from where the model learns task-specific knowledge, especially in zero-shot setups. In this work, we propose a novel method ORCA to identify evidence of the model's task-specific competence in prompt-based learning. Through an instance attribution approach to model interpretability, by iteratively using gradient information related to the downstream task, ORCA locates a very small subset of pretraining data that directly supports the model's predictions in a given task; we call this subset supporting data evidence. We show that supporting data evidence offers new insights about the prompted language models. For example, in the tasks of sentiment analysis and textual entailment, BERT shows a substantial reliance on BookCorpus---the smaller corpus of BERT's two pretraining corpora---as well as on pretraining examples that mask out synonyms to the task labels used in prompts.

Under review as a conference paper at ICLR 2023
ORCA: INTERPRETING PROMPTED LANGUAGE MODELS VIA LOCATING SUPPORTING EVIDENCE IN THE OCEAN OF PRETRAINING DATA
Anonymous authors Paper under double-bling brind review
ABSTRACT
Prompting large pretrained language models leads to strong performance a va- riety of downstream tasks. However, it is still unclear from where the model learns task-specific knowledge, especially in zero-shot setups. this work, we propose novel method ORCA to identify evidence of the model's tpsk-specific competence in prompt-based learning. Through an instance attribution approach model interpretability, by iteratively using gradient information related the downstream task, ORCA locates very small subset of pretraining data that di- rectly supports the model's predictions in given task; we call this subset sup- porting data evidence. We show that supporting data evidence offers new insights about the prompted language models. For example, in the tasks of sentiment anal- ysis and textual entailment, BERT shows substantial reliance on BookCorpus- the smaller corpus of BERT's two pretraining corpora- as well as on pretraining examples that mask out synonyms to the task labels used in prompts
INTRODUCTION
Large language models (LLMs) are trained on massive text corpora from the web, referred to the pretraining data (e.g., Devlin et al. 2019; Raffel et al. 2020). Due to their volume, pretraining data typically cannot be inspected manually and are prone to spelling/logie errors, domain mismatch w.r.t. target tasks, social biases, and other unexpected artifacts (Bender et al. (021). Yet, LLMSs pretrained with such noisy data attain surprisingly good performance on numerous downstream tasks, with little or task-specific tuning (Petroni al. 2019 Brown et al. 2020).
(e.g.,Devlin
There are several hypotheses explaining the power of pretrained LLMs. One hypothesis is that the pretraining data is huge and the model might be shallowly memorizing patterns in data (Bender et al. 202T; Carlini et al., 202T Razeghi et al., 2022). An alternative hypothesis is that LLMS might be learning to reason through observed patterns in the pretraining data in novel ways (MeCoy ct al. 2021). However, the evidence these conjectures, especially arbitrary downstream tasks, remains underexplored.
Such evidence is useful as it can help explain model decisions, surface problematic patterns in data model behavior, and shed new light on how to improve the model and data (Zhong et al., 2019 Han & Tsvetkov 2020, [021] Pruthi et al. (2022). Moreover, it can facilitate the trustworthiness of the models (Lipton) 2018 Jacovi et al. 2021).
In this work, we develop methodology to provide such evidence, Our hypothesis that among the enormous pretraining corpora, there subset of pretraining data that contributes to the model's behavior on downstream task more than the rest of the pretraining data. Therefore, our task is to locate a task-specific evidence set-a a very small amount pretraining data that particularly helps the model's performance on the task. We call it supporting data evidence (SDE). Such SDE can help interpret the model if we analyze its task-relevant patterns compared to the rest of the corpora.
A related line of interpretability research focuses on instance attribution (ohh & Liang 2017 Yeh
2018 Pruthi al. 2020 Han et al. 2020). where the goal to find which training examples al.
Coode and data will be released at ANON X8D upon publication.

---

Under review as a conference paper at ICLR 2023
most influential to the model's decision, focusing individual test examples. However, in this work we are interested in locating sets of pretraining data influencing the whole task (i.e., a full test set, rather than individual test instances). We seek such "global" evidence for the task because given the scale of the pretraining and task data, it could be inefficient or even infeasible to find and inspect the evidence for each of the task examples +
We first formulate the problem of finding SDE in pretraining data by upweighting the SDE set and measuring its impact on model performance (...1) In $2.2 we propose novel method ORCA that effectively identifies the SDE by iteratively using task-specific gradient information. On two classification tasks sentiment analysis and textual entailment-in a prompt-based setup ($3) we
show the effectiveness of the SDE discovered by ORCA compared to random data subsets and neirest-neighhor data an embedding space ($4)) Our analyses of the discovered SDE ($5) show
that our base language model BERT (Devlin et al. (019) has an interestingly high reliance on the smaller corpus of its two pretraining corpora (BookCorpus, (Zhu et al. 2015). Also the pretraining
examples in SDE typically mask out synonyms to the task verbalizers (i.e., words mapped to the task labels in the prompts, Schick & Schüizze 2021).
2 ORCA
We develop a method to explain the competence of large pretrained language models used zero- or few-shot prompt-based classification (Petroni et al. 2019 Brown et al. 2020 Without conven- tional finetuning, model decisions rely on knowledge learned from the pretraining data, and our goal identify what supporting data evidence (SDE) in pretraining data facilitates model's competence in specific downstream task.
2.1 PROBLEM FORMULATION
Assume GPT, LLM pretrained with a dataset DPT 3 cooteett Viiskod)- For example, for a masked language model Esentexi is a block of text with certain tokens masked, and ymasked are the masked tokens in their original forms, be reconstructed. GeT is trained to minimize loss L over the pretraining examples, OPT = arg ming L(DPT:))
.P
The LLM can be applied to many downstream tasks without finetuning, via prompting (Schick 2021 Liu et al. 2021)). Given a dataset in a downstream classification task Disk e the LLM applied by measuring pe (verbalizer(y
& Schütze
template(z**x))) The template supplies a prompt tailored to the task for the model, and the verbalizer maps the output of the language model to the task's label space (more details in $3.2).
(x lask lask
We interpret the model decisions by finding the SDE S C DPT w.r.t. the task data Dinkk The size of should be very small (e-g.. a few hundred) compared to the whole pretraining data, |S| < |DPT], to facilitate further manual or semi-automatic analyses. More importantly, S should "contribute" significantly to the performance of the model on the downstream task.
However, we first observe that defining this contribution is non-trivial problem. Prior work in instance attribution like influence functions (Koh & Liang 2017) adopts a "leave-one-out" perspec- tive (Cook 1977). In our case, this would mean removing S from retraining a new LLM from scratch, and testing it on D This is prohibitively expensive
We adopt an "upweighting" perspective. Instead of leave-one-out, we upweight certain pretraining examples (c.g., S) by training the model on these examples for an additional epoch. The resulting change the model should be small to prevent overfitting. Specifically, randomly batch the SDE S to mini-batches, thereby updating the model via a very small number of optimizer updates:
PTT new + PPT + updates (S))
Directly applying instance attribution methods to the task level has also been shown yield negative results (Kocijan & Bowman| 2020))
NNaeed after the marine mammal for nO PuRtiCular reAson.
'hile this work we focus on text classification, the framework is also adaptable to generation problems.
Moreover, the definition of influence functions and even leave-one-out can sometimes be arguable, espe-
cially in non-convex models (Basu al. 2021 K & Søgaard 202T).
2

---

Under review as a conference paper at ICLR 2023
For the simplicity of notation, we fold sequence of optimizer updates into updatesg.c that depends on a sequence of batched data S, the model parameters 8, the loss function L, and an optimization algorithm used during pretraining.
The quality of the data evidence is reflected in the performance difference in the downstream task be- tween the new model f and the original pretrained model GPT, measured by the metric associated with the task [
2.2 IDENTIFYING SUPPORTING DATA EVIDENCE
We now propose a novel method ORCA identifying the supporting data evidence S C. DPT The goal is to find a subset of the pretraining data S < [DPT| that is directly helpful to the downstream task when we continue pretraining the language model over it (as described in $2.1). The intuitions behind our method are simple: (1) We aim to find contributive examples in D that exert a similar change to the model parameters as Dunk would. (2) There could be multiple subsets of pretraining data that, in conjunction, are useful to the task. We thus select S in several iterations rather than at once.
We build our SDE S in m iterations, S1, the size of the subset at each iteration is 181 To find the first evidence subset Sj, we rely on the intuition that continuing training on the task data Diak directly is likely to improve the original model 0 on the task. We thus batch the task data and calculate a batch gradient VoLus (Diask; BPT) Descending along the gradient direction should improve 8PT, and we find a subset S1 of the pretraining data that exerts similar gradient of the model as Vask (Dusk; ;)!):
S fd DPT | cos( (ooLMML,, PPT) ToLama (PT)) 51}
We measure cosine similarity between the gradient for each example in DPT and the batch gradient for the task We then select for S1 the top-k examples in DPT with highest cosine; 51 simply the cosine score of the k-th ranked example. $3. specifies the size of selection along with other hyperparameters.
Now with the first data evidence subset S1, we continue pretraining an intermediate model APT:
gh + updateso.c(5)
The procedure find the rest of the data evidence subset S. with 2....... m is similar to the above but with one difference: these subsets should ideally be beneficial to the model in a way that not already fully captured by S..
We hypothesize that the intermediate model OPT captures information about St. Therefore, for later iterations we want to calculate a task batch gradient based on the previous intermediate model, V Crsk (Disk,, OPT .). The data evidence subset at each iteration should again exert similar gradi- ent:
S = (d E DPT cos ( (ILLM oT--)) V (iink, ,TT > 5.}
(4)
with 6; as a threshold for selecting |S,] elements like 8, /i - 1 is a design choice that can allow for a "lagged" model (i.e., computing the gradient of the LM loss w.r.t. the model several iterations before vs. the immediate previous intermediate model). This lagging aims to improve the stability of the method. For the experiments in this work, 8PT is by default GPT, for maximum lagging; OPT_ is 0PT, in cases denoted by NL (no lagging)
1-
At each iteration, having a total of i data evidence subsets, we continue pretraining an intermediate model OPT:
PT + PT + (indaatess_c()_=11
(5)
'A discussion over the limitations of our problem formulation can be found in A
"hhe task loss overa a single task example is Last (3 , = log pe (wwhaalizrrrrgg 'The LMM loss over single pretraining example LIM ) = log Po (y CCmpared to g²T, the no-lagging version uses 6.T,, which comes from continuing pretraining over small amount of examples and can have a higher variance. Adding this lagging also remotely shares intuition with RL methods addressing training stability (Mnih al. 2016).
(amplate ))-
3

---

Under review as a conference paper at ICLR 2023
It is worth noting that for every iteration, continue pretraining the original language model, and the data evidence subsets are unordered.
After the m-th iteration, complete building full supporting data evidence S = - The resulting model P is essentially the final upweighted model, i.e. 6 introduced in $2.1
Sy.
EXPERIMENTAL SETUP
3.1
BASELINE METHODS
ORCA is greedy algorithm and thus not guaranteed to find the global optimal SDE out of the candidates. To evaluate the efficacy of the identified evidence, we compare ORCA with the following baseline methods:
(ID))
Random sampling We simply sample at random |S| examples from DPT as the SDE.
Embedding nearest neighbors Enhancing language models using examples with nearest neigh- boring embeddings is common approach in domain adaptation of LMs and kNN-LMs (Gururangan
al. 2020 Khandelwal et al. (2000). Here we find nearest neighboring pretraining examples the task examples. We define a similarity score as below:
cos ( (ontext), Averbalizzrr (tmmlltteet
(6)
Mmusked is the last hidden representation the position of the masked pretraining token.
Averhalizer is the last hidden representation at the position of the task verbalizer token.
context the pretraining input to the model but containing the ground truth masked token.
template (G is the templated task input but supplying the ground truth verbalized label.
We the ground truth information here for fair comparison with method ORCA, where the calculation of gradients involves the ground truth information as well.
Practically, since [iikk] can be large and well over |S]. we first sample examples from D Then, for each of the l sampled task examples, we find the top-k nearest neighboring pretraining examples in DPT Finally, from the pool of the t k pretraining examples, we sample |S| of them the SDE. We additionally have a hyperparameter max-r controlling the maximum allowed data repetitions in the selected data evidence.
Iterative selection with embeddings We use the gradient information cos vos(VeLLM() (-M Vu (.) when collecting the SDE subsets ORCA iteratively. Here we test whether we can substitute the gradients with embeddings while keeping the selection iterative. Reusing the notations in the embed- ding nearest neighbors baseline, we use cos(Nrmnked(.))
[ Entox lvertuuizer (-)) for all the gradient
cosine operations in ORCA. Note that we use the average embeddings all task examples to replace the batch gradient over all task examples. Other design decisions of ORCA remain unchanged.
3.2 LANGUAGE MODEL AND DOWNSTREAM TASKS
BERT We use the BERT-large language model as PPT (Devlin et al. 2019 10
IMDB We primarily experiment with two text classification tasks, sentiment analysis and textual entailment "1] For sentiment analysis, we use the IMDB movie review datasct (Maas et al. 2011) Ip. The task data Dusk here is the IMDB test split containing 25,000 examples The template for the IMDB examples is "It was [MASKJ. <REVIEW>". The verbalizer maps the reconstruction of the [MASK] token to the label space, ""oood" - positice, "bad" -> negative).
We choose it primarily due to the limited computing resources we have BERT is small both in terms of the number of model parameters and the size the original pretraining data. ORCA is extendable to other language models as well.
Future work explore more tasks, but we select two typical ones here mainly due our computational resources (more details in {B).
12ehe use of test set is our deliberate choice in this work. In g we further discuss the purpose of and
show sanity check on alternative setup using the training set.
4

---

Under review as a conference paper at ICLR 2023
For the textual entailment task, the MNLI dataset (Williams al. 2018). The task data Dusk here is the MNLI matched validation split containing 9,815 examples (the test split is private). The template for the MNLI examples is "<PREMISE> [MASK], <HYPOTHE- SIS>". The verbalizer maps the reconstruction entailment, "no" contradiction, "mayhe" neutral.. We use the OpenPrompt library (Ding et al. 2022) to prompt the BERT model with the templates and verbalizers inherited from Gao et al. b
MNLI
the [MASK] token to the label space, {"yes"
Zero-shot transfer and prompt tuning When we formulate problem, are interested in the evidence in pretraining that directly impact the pretrained model's performance on the downstream task-a zero-shot transfer scenario. There is no notion of finetuning with the in-task training data. However, research in prompt tuning (e.g., [esterr et al.) (02T) folds the usage of in-task training data into the template for the task. A sequence of soft embeddings is added the beginning of the template and trained with the in-task training data. The language model parameters remain unchanged. Apart from our main experiments with the zero-shot transfer model, we also consider such prompt tuning scenarios, finding pretraining data evidence useful for the task when the template enhanced with some in-task training data [3
3.3 PRETRAINING DATA
Source BERT uses the English Wikipedia and BookCorpus (Zhu al., 2015) its pretraining data. During pretraining, 15% of the tokens are randomly masked out to be reconstructed. Though BERT's pretraining data is already small compared those of many other language models (e.g., Raffel et al.) 2020; Gao et al. 2021a), we unfortunately still do not have the resource to process the full dataset. In fact, this work we only randomly sample 0.5% of the full pretraining data.
Format During pretraining, BERT would reconstruct the masked 15% tokens in sequence in parallel (i.e., the reconstruction loss for each token independent). From the training perspective, this is efficient. However, this work aims to find the SDE. We particularly want know learning the reconstruction of which token could most impact the downstream task performance. Therefore, expand each pretraining data and treat each masked token as standalone example [4] More specifically in our setup, DPT э (aromtexss Frasked). x is a sequence of 512 tokens, and Vuniseed is a single masked token in the sequence. Together this makes | = 3,924,635 (with 52,640 unique context sequences). We choose most 2,000 instances from DT as the SDE S.
3.4 HYPERPARAMETERS
ORCA finds S in iterations. In this work we use m=20 iterations, with each iteration finding 100 examples from DPT (with a total |S]=2)00).
For the embedding kNN baseline, we sample (=1000 task examples and choose k=(10, 20, 50, 100} most similar pretraining data. Within the ( k candidate pool, we sample ,S 5220000 examples, with a max number of repetitions 11, 20, 2000) [5
During the continued pretraining for all methods, we use a batch size of 16, resulting in at most 125 optimizer updates from the original language model. The learning rate is set at one of BERT's default values 2c-5.
EVALUATION
We use different amounts of in-task training data for prompt tuning depending on the task performance. For IMDB, we use 100 examples per class, whereas for MNLI, we use 10,000 examples per class.
14 "Other masked tokens are still masked in input context, to he faithful to the original LMM objective.
In our method ORCA, though the examples within S; strictly non-overlapping. we don't enforce dis- tinetiveness across since we only work with 0.5% of the pretraining data. This maximum appear 20 times our method. Therefore, we include the r > 1 options for the embedding kNN baseline as well for a fairer comparison. In $ we further discuss the choice of f and k. Additional details and discussion of the results be found in $
an example could
5

---

Under review as a conference paper at ICLR 2023
Main results (accuracy) of ORCA and base- lines the zero-shot model. NL means the no- lagging variant. Numbers in regular fonts are aver- aged from 5 random seeds, while numbers in small fonts show a trajectory of performance with one seed [6
Table I:
We evaluate the supporting data evidence S, identified using ORCA and the base- lines, by quantifying the supportiveness of (as defined in $2.1). Note that this section does not focus whether not the discov- ered data evidence is plausible to humans. We will explore what humans can interpret from the actual data evidence in $
On zero-shot model
IMDB
MNLI
Null
73.50
43.70
Table shows our main results: the per- formance of our zero-shot language model pretrained additionally on S, as identified by different methods. We first notice per- formance gap between our original model on IMDB and MNLI, indicating the entail- ment task is intrinsically harder for mod- els that have only been trained pretrain- ing data. We observe a moderate perfor- mance improvement using the embedding nearest neighbors method (IS) 2000)- The best performance is achieved by our proposed method ORCA, especially in the task of IMDB by a large margin (even with [S] I 500), showing the effectiveness of our method.
Random
71.25 #2.56
42.56 40.43
Embedding kNN
76.55 #2.16
45.15 40.48
Iterative embeddings
75.11 1:4.9
43.74 10.68
ORCA (NL)
84.51 1:0.77
45,46 1..73
0< [S] x 500
79.81
44.85
500 < [S] S 1000
83.87
45.64
1000 < [S] & 1500
84,40
46.10
1500 < |S|| S 2000
85.17
46.49
ORCA 0 < < 500
84.33 +1.51
46.06
+0.35
81.60
45.99
500 < [S] < 1000
83,23
45.75
1000 < [S]] S 1500
84.42
46.40
1500 < [S| & 2000
85.15
46.26
Table 2] shows some additional results on the effect of the SDE on a prompt-tuned model. These results show that, compared to the zero-shot model, prompt-tuned model is more difficult to improve since the prompt may already be highly specialized towards the task, using the in-task training data. The additional signals in the pretraining data that useful to the task can be scarce. That said, the pretraining data S identified by ORCA still improves the model on IMDB.
5 ANALYSIS
While useful in showing the effectiveness of the SDE S, evaluations in do not provide us with tangible insights about the model it- self. In this section, we analyze some prop- erties of S, and see whether they reflect hu- mans' expectations for the model. We first show few qualitative examples of the evi- dence discovered by ORCA in Table 3
Table 2: Additional results (accuracy) of ORCA and the baselines on the prompt-tuned model. All the numbers are averaged from random seeds.
On prompt-tuned model
IMDB
MNLI
Null
87.83
70.19
Random
86.06 40.82
69.07 #0.50
Which source corpus does the support- ing data evidence come from? The pre- training data of BERT consist of the English Wikipedia and BookCorpus. We show the source corpus of examples S in Figure I
Embedding kNN
86.53 40.77
68.93 40.61
Iterative embeddings ORCA (NL) ORCA
87.80 1.0.0
68.45
1.036
87.65 11.10
68.79 20.28
88.10 40.06
68.61 40.44
We find that though the pretraining set consists of considerably more data from Wikipedia than from BookCorpus (76.5% vs. 23.5%), the SDE identified by ORCA has a dras- tically different source corpus distribution. In IMDB, 64. 1% and 92.6% of the examples in come from BookCorpus, using the default ORCA and its no-lagging variant respectively. The demotion of Wikipedia examples in the sentiment analysis task somewhat reasonable, since Wikipedia is meant to have neutral point of view (NPOV) (rov)]77 |n On the other hand, BookCorpus consists of novels that could involve strong emotions and sentiments.
"https://en..a wi httpps:/eenaiiiipeeeiaaoooo
/wik//wikiiia ped Neuttral point itt of

---

Under review as a conference paper at ICLR 2023
Table 3: Examples of the supporting data evidence (S) in pretraining data discovered by ORCA for IMDB and MNLI. The masked evidence for IMDB expresses sentiments, while less clear whether the example evidence for MNLI is related to entailment.
token (Vinskedd) in each example is underlined. The example
IMDB
we have to think that were awfully lucky as human beings have nice precise system. the sloppy system is probably good enough for bacteria. turns out much to geneticists surprise - that lowly bacteria store genes as whole units (weasel)
it was the only place she could afford. her meager earnings didnt provide much in the ways of clean, modern style along with the privacy she required. she felt better if she thought about how bad could be. year ago, shed lived with mother. anywhere was better than living with her
MNLI
... he then the cord that bound her hands and legs. you ok walk? he asked hoping the answer was yes, i think so. but im quite stiff, she said. he helped her up, stretch your legs little. theyll feel better ...
there way hide the shock on her face, and she knew he by his sigh. "do you think yourself less than me?" "no!" she absolutely didn't t but he nodded his head. "i see. you thought would think you were less than me." she ashamed. "I'm sorry.". ...
...
he shook his head, incredulous. in fact, he looked like he was considering throttling me. "you're re just not getting it. maybe that's my fault. maybe it's because don't tell you love you often enough. baby, you re the only 'good' thing that i've ever had
similar trend can be found in MNLI as well, with 99.0% and 92.9% of the examples in S coming from BookCorpus, ing the default and NI. variant of ORCA. We conjecture that the over-reliance on BookCorpus in MNLI could be due the lection of the colloquial verbal- izer words (e.g.. "yes", "maybe"), which can be scaree in Wikipedia. Also, the BookCorpus data could contain more everyday topics that match MNLI's genres (c.g.. fic- tion, letters, telephone speech). However, whether it reasonable for the model to rely on BookCorpus for textual entailment is arguable: Wikipedia should be more reliable source if we want the model to build more upon factual information.
Wikipedin
BookCorpus
1.00
0.75
0.50
0.00
All protraining data
IMDB SDE by Embedding ANN
IMDB SDE by ORCA (NL)
IMDB SDE by ORCA
MNLI SDE by Embedding KNN
MNLI SDE by ORCA (NL)
MNLI SDB by ORCA
Figure/ 1: Source corpus distribution of the supporting data ev- idence (SDE) in IMDB and MNLI.
What are the masked tokens in the supporting data evidence? Prompted language models a verbalizer to adapt to the downstream task. For example, outputting "good" for a templated IMDB input indicates positive sentiment, "yes" for MNLI indicates entailment, etc. For pretraining example that supports the task, are there any relations between its masked, fo-be-reconstruuted pre- training token f and the verbalizer words for the task (verbalizer( (ask ()) In Table 4 we show the 10 most frequent masked words (types) in S, for each method in IMDB and MNLI.
We observe that the verbalizer words, in their original forms, are always the most common masked token in S. For all of the methods in both tasks, over 50% of the masked tokens are exactly the verbalizer words. Though there is some noise in PT y (e.g., symbols that carry no rask-relevant meaning), most of the other masked tokens are synonyms to the verbalizer words in IMDB by observation. In MNLI, the other masked tokens may capture relations between clauses similar to the verbalizer words (e.g., then, to, probably). Overall, we find that PT in the discovered S is reasonable for the sentiment analysis and textual entailment task.

---

Under review as a conference paper at ICLR 2023
Table 4: Masked tokens (Fmaskea) in the supporting data evidence of IMDB and MNLI.
Task
Method
Most frequent Uimaseed in S
IMDB
Embedding KNN
bad, good, terrible, great, badly, excellent, worst, negative, better, disappointment, ... [1] distinct tokens in total, 94.8% verbalizer words) bad, good, worst, n, worse,", wrong, horrible, poisonous, ... [9/ distinct tokens in lotal, 90.0% verbalizer words]
ORCA (NI.)
ORCA
bad, good, " horrible, not, worse, ugly, hated, poor, terrible, [285 distinct tokens in total, 55.9% verbalizer words]
MNLI
Embedding kNN
no, yes, maybe, yeah, However, perhaps, n, ), No, [258 distinct tokens in total, 58.3% verbalizer words]
ORCA (NL.)
maybe, yes, no, n, that, - then, perhaps, the, ... [/76 distinct tokens in total, 69. 1% verbalizer words)
ORCA
maybe, yes, no, perhaps, to, probably, has, in, big, [125 distinct tokens in total, 59.4% verbalizer words]
Random
Embedding KNN ORCA
Is the context of the supporting data evidence sim- ilar to the task input data? We are interested in the relationship between the context of the selected pretraining data (1oomee))) and the input of the down- stream task (2**s). Are they exceptionally similar, indicating that the model may be memorizing shal- low patterns? Alternatively, are they considerably different, indicating that the model needs to transfer some learnt knowledge from pretraining to the task (either in a reasonable or spurious way)? Our ex- ploratory step uses an automatic metric between two distributions of texts, MAUVE (Pillutla et al.) (02T). to measure the similarity between our sets of 2 F'context and و takk As a method based on quantized language model embeddings, MAUVE similarity may capture text attributes such as topics and style
ORCA (NL)
MAUVE with task data
0.56
0.54
0.52
0.50
e-
c-50
c-100
c-200
5-512
Context window size
Figure 2: MAUVE similarity on IMDB, be-
tween the sets of 2.T in S and lisk
Apart from using all 512 tokens in the context of the data evidence (x we also truncate the context, keeping the surrounding e tokens of the masked token (Umasted) We control for the scope of the context by varying c. For liinkk, we randomly sample 2000 examples to match the size of S 19 Figure 2 shows the results IMDB. See $ for MNLI results.
We observe that the MAUVE scores between xe and task are all between 0.512 and 0.577. In contrast, the MAUVE score between the training set of the task and the test set (2:task is 0.998 and 0.997 for IMDB and MNLI respectively. This substantial difference in MAUVE scores may indicate disparity in topics and style between the context of the pretraining evidence and the task data. Additionally, the MAUVE score of our selected SDE is not higher than a random sample in most cases. This further shows that the signal in the evidence context useful for the task is subtle, in way that MAUVE cannot capture. This is in contrast to the conjecture that the model must have seen the exact inputs in the pretraining data and only performing a shallow memorization.
While not within the scope of this paper, future investigation can also extend the analysis of the sup- porting evidence with feature attribution methods (Pezeshkpour et al. 2022) or a human evaluation with domain experts of the task (e.g., what exact spans in the SDE contribute to their supportiveness). We further discuss the limitations of our method, computational resources, and future directions in A and B
"Gaamaticality can he another attribute as Pillutla (021) work with machine-generated texts. This less relevant in our case as our sets texts are naturally occurring. Here gi is without template, and .P has the masked token recovered. This is to give MAUVE most natural texts for evaluation.
8

---

Under review as a conference paper at ICLR 2023
6
RELATED WORK
LLMs have been showing competence in various downstream tasks in NLP with little to no task-
specific tuning. using prompts (Petroni al. 2019 Brown et al. 2020 Schick & Schütze 202T
Gao et al.) 2021b Lester et al.) 2021). We are especially interested in interpreting LLMs under
zero-shot setup, where the knowledge relevant to the downstream task must come from the noisy
pretraining data 20
One common interpretability method for NLP models feature attribution, where important tokens or spans in the inference-tine input are highlighted, indicating their contributions to the model's de- cision (Simonyan et al. 2014 Li et al., 2016 Ribeiro et al. 2016 Lundberg & Lee| 2017). However, information relevant to explaining the model's decisions (especially if abstract) is often not in the inference-time input (Han ct al. 2020 Wiegreffe & Marasovie, 2021] Pezeshkpour et al., (022). On language models, feature attribution has been used interpret and verify grammatical phenomena Yin & Neubig 2022).
Another type of interpretation that aligns more with our focus is instance attribution, where impor- tant training examples are highlighted for their influence on the model (Koh & Liang, 2017] Yeh al. 2018, Pruthi al., 2020; Han al., 2020 Guo et al. (02T). In this work, are instead interested in the influence of pretraining data and in finding SDE for the entire task rather than in- dividual test examples 21] There has also been prior work analyzing what amount of data is needed during pretraining to achieve models with certain capabilities (Zhang et al. 2021), but these works do not attribute model performance to specific pretraining data.
Our formulation may seem similar to prior work in task eesknennanning pretraining (Han & Eisenstein 2019 Gururangan et al.) 2020; Yao et al., 2021). However, such methods typically use a large amount of loosely relevant pretraining data along with the in-task training data, to improve perfor- mance. We instead aim to find an ofteerroo--mmnntuuee----
set of pretraining data, providing a clearer signal of their impact on the task for interpretability purposes.
Our proposed method to find the data evidence, ORCA, shares a similar intuition with prior work that reweighs training data [Wang et al. 2020), as both methods use the gradient information of the test data. However, their target model depends on an ordered sequence of data weights and model checkpoints. In contrast, we apply an unordered data evidence set to the original model, mimicking an upweighting in pretraining. In addition to the difference in methods, the purpose is different as well: theirs is performance-orieetedd while ours is interpretability-orrented
Other remotely related line of work in machine learning includes coreset construction (Coleman al. 2020 Mirzasoleiman et al. 2020 Huang et al. (202]) and dataset distillation (Wang et al.j 2018 Zhao et al.) (021).. Their focus is typically an empirical risk minimization problem on the training data, without notion of downstream tasks or task transfer. They aim to create a substitution set for the full training data for an efficiency purpose.
7 CONCLUSION
The source of competence of zero- and few-shot prompted language models on downstream tasks mysterious. The models should be gaining task-specific knowledge from the pretraining data, but what pretraining data leads to the capability of the models is an underexplored area of research. In this work, we formulate the problem of finding supporting data evidence in the pretraining data of L.MMs for downstream tasks. We propose ORCA to effectively identify such evidence with an iterative guide from task-specific gradient information. Deeper analyses into the evidence show that prompted BERT on sentiment analysis and textual entailment relies heavily the BookCorpus data, as well as on pretraining examples that mask out task verbalizers and their synonyms.
Interpreting the role of pretraining data in unprompted, finetuning setup intrinsically harder, but prior work (Chn et al. (2020) have made attempts.
21 A concurrent work by Akyürek et al. (2022) builds candidate set for fact-tracing in question answer- ing; the difference is the use of task-related training examples instead of pretraining data, and an information retrieval evaluation.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ekin Akyürek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, lan Tenney, Jacob Andreas, and Kelvin Guu. Tracing knowledge in language models back the training data. ArXiv, abs/2205. 1482, 2022.
Sanjeev Arora. Advanced algorithm design lecture 9: High dimensional geometry curse of dimen- sionality dimension reduction. 2013.
Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning are fragile. In ICLR, 2021.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Majoo and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? Proc. FAceT, 2021.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vore Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Rad- ford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, 0os/20055141663 2020.
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Xiaodong Song, Ülfar Erlingsson. Alina Oprea, and Colin Raffel. Extracting training data from large language models. In USENIX Security Sympo- sium, 2021.
Hongge Chen, Si Si, Yang Li, Ciprian Chelba, Sanjiv Kumar, Duane Boning, and Cho-Jui Hsieh. Multi-stage influence function. In Proc. NeurIPS, 2020.
Cody A. Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei A. Zaharia. Selection via proxy: Efficient data selection for deep learning. In Proc. ICLR, 2020.
R Dennis Cook. Detection of influential observation in linear regression. Technometrics, 19(1): 15-18, 1977.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL-HLT, 2019.
Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Haitao Zheng, and Maosong Sun. Openprompt: An open-source framework for prompt-learning. In Proc. ACL (System Demonstra- tions), 2022.
Leo Gao, Stella Rose Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. ArXiv, abs/2101.00022 2021a.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot leamners. In Proc. ACL, 2021b.
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In Proc. ICML, 2019.
Han Guo, Nazneen Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong, Fastif: Scalable influence functions for efficient model interpretation and debugging. Proc. EMNLP. 2021.
Suchin Gururangan, Ana Marasovié, Swabha Swayamdipta, Kyle Lo, Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In Proc. ACL, 2020.
Xiaochuang Han and Jacob Eisenstein. Unsupervised domain adaptation of contextualized embed- dings for sequence labeling. In Proc. EMNLP, 2019.
10

---

Under review as a conference paper at ICLR 2023
Xiaochuang Han and Yulia Tsvetkov. Fortifying toxic speech detectors against veiled toxicity. In Proc. EMNLP. 2020.
Xiaochuang Han and Yulia Tsvetkov. Influence tuning: Demoting spurious correlations via instance attribution and instance-driver updates. In Proc. Findings of EMNLP, 2021.
Xiaochuang Han, Byron C. Wallace, and Yulia Tsvetkov. Explaining black box predictions and unveiling data artifacts through influence functions. In Proc. ACL, 2020.
Horace He and Richard Zou, functorch: Jax-like composable function transforms for pytorch. https: ttttps://githie http://ggghhuub..oopttto.ooo wwt.yytocccffuntooooooo funct orch 2021.
Ari Holtzman, Peter West, Vered Schwartz, Yejin Choi, and Luke 7ettlemoyer. Surface form com- petition: Why the highest probability answer isn't 't always right. In Proc. EMNLP, 2021.
Jiawei Huang, Ru Huang, Wenjie Liu, Nikolaos M. Freris, and Huihua Ding. A novel sequential coreset method for gradient descent algorithms. In Proc. ICMI., 2021.
Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg. Formalizing trust in artificial intelli- gence: Prerequisites, causes and goals of human trust in ai. Proc. FAceT, 2021.
Karthikeyan K and Anders Søgaard. Revisiting methods for finding influential examples. ArXiv, abs/2111.04683. 2021.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In Proc. ICLR, 2020.
Vid Kocijan and Samuel R. Bowman. Influence functions do not seem to predict usefulness in nip transfer learning. httteps///aarr.aadd////////////0 -osee---ooootseettoo--e---e-- -ttto---seettt--eect--------e-e- ott.::/aa....p.ppoootoaeeeeeeeea
learning/ 2020.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In
Proc. ICML, 2017.
Brian Lester, Rami AI-Rfou, and Noah Constant, The power of scale for parameter-efffcient prompt tuning. In Proc. EMNLP, 2021.
Jiwei Li, Xinlei Chen, Eduard H. Hovy, and Dan Jurafsky. Visualizing and understanding neural models in NLP. Proc. NAACL-HLT, 2016.
Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of inter- pretability is both important and slippery. Queue, 16(3):31-57, 2018.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. ArXiv, abs/2107. 13586, 2021.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Proc. NeurlPS, 2017.
Andrew L. Maas, Raymond E. Daly, Peter Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proc. ACI, 2011.
R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven. ArXiv, abs/211 11/2211.0990 11.09509, 2021.
Baharan Mirzasoleiman, Jeff A. Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In Proc. ICMI, 2020.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In ICML, 2016.
11

---

Under review as a conference paper at ICLR 2023
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. Language models as knowledge bases? In Proc. EMNLP, 2019.
Pouya Pezeshkpour, Sarthak Jain, Sameer Singh, and Byron C. Wallace. Combining feature and instance attribution detect artifacts. Proc. Findings of ACL, 2022.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using diver- gence frontiers. In Proc. NeurIPS, 2021.
Danish Pruthi, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary Chase Lipton, Graham Neubig, and William W. Cohen. Evaluating explanations: How much explanations from the teacher aid students? TACL, 10:359-375,
Garima Pruthi, Frederick Liu, Mukund Sundararajan, and Satyen Kale. Estimating training data influence by tracking gradient descent. In Proc. NeurIPS, 2020.
Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh, Impact of pretraining term frequencies on few-shot reasoning. ArXiv, abs/2202.07206, 2022.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should trust you?": Explaining the predictions of any classifier. In Proc. KDD, 2016.
Timo Schick and Hinrich Schütze. Exploiting cloze- doze-questions for few-shot text classification and natural language inference. In Proc. EACI,, 2021.
Lloyd S Shapley. Notes on the n-person game-ii:
ii: The value an n-person game. 1951.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Proc. ICLR, 2014.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. In Proc. ACL, 2019.
Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation. ArXiv, abs/1811.10959, 2018.
Xinyi Wang. Hieu Pham, Paul Michel, Antonios Anastasopoulos, Jaime Carbonell, and Graham Neubig. Optimizing data usage via differentiable rewards. In Proc. ICML, 2020.
Sarah Wiegreffe and Ana Marasović. Teach me explain: A review of datasets for explainable natural language processing. In NeurIPS Datasets and Benchmarks, 2021.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proc. NAACL, 2018
Xingcheng Yao, Yanan Zheng, Xiaocong Yang, and Zhilin Yang. Nlp from scratch without large- scale pretraining: A simple and efficient framework. ArXiv, abs/2111.004130. 2021.
Chih-Kuan Yeh, Joon Sik Kim, Ian En-Hsu Yen, and Pradeep Ravikumar. Representer point selec- tion for explaining deep neural networks. In Proc. NeurlPS, 2018
Kayo Yin and Graham Neubig. Interpreting language models with contrastive explanations. ArXiv, abs/2202. 10419, 2022.
Yian Zhang, Alex Warstadt, Haau-Sing Li, and Samuel R. Bowman. When do you need billions of words of pretraining data? In Proc. ACI., 2021.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In ICLR, 2021.
12

---

Under review as a conference paper at ICLR 2023
Ruiqi Zhong, Steven Shao, and Kathleen McKeown. Fine- Fine-grained sentiment analysis with faithful attention. arXiv preprint wrX.v:990..00 2019.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proc. /CCV, 2015.
A LIMITATIONS OF THE PROBLEM FORMULATION
Implication of supporting data evidence One limitation of our setup is that the data outside our defined SDE S may have additional value on the model which the current formulation does not capture. For instance, there could be certain examples that exceptionally help the model capture the grammar of the language, making indirect contribution the task. However, they are rather unlikely to be picked as SDE: in the scope of this paper, we focus on finding the evidence that directly related to the downstream task. The indirect evidence is also interesting and may be valuable for future work to investigate further (e.g., via finding which examples contribute most to the contribution of SDE).
Alternatives to our problem formulation Another limitation of our problem formulation that the history of the model pretraining is ignored. it sufficient check whether the evidence data help the fully pretrained model? What if a data is truly related to the task but got overfitted during earlier stages of pretraining, so continuing pretraining on it would not change the model? We think this is possible, but only to some extent and with a limited risk, since the huge volume of pretraining data can make it difficult to overfit to all examples related to an arbitrary task in particular way. Nevertheless, to improve that case, a potentially better solution would be continuing pretraining each checkpoint the model across the whole pretraining procedure. This would share an intuition with some instance attribution methods based on model checkpoints (Pruthi et al.) 2020). However, lack the resources to perform such experiments in this work, so we defer that to future research
B ETHICS STATEMENT
One ethical consideration and limitation of our approach is the large computational cost, and conse- quently the environmental impact caused by our computationally-exxensss 2019). On machine with Nvidia A40 GPUs, the full 20 epochs of each ORCA experiment would take about 7 days in total. The long computing time is partly because we need to calculate per-sample gradients for 4M data points in each epoch, and also because there is no efficient way to calculate the per-sample gradients in PyTTrch at the time of our implementation However, our goal is to develop research prototype (which can be optimized in the future) that will enable open- ing up the black box of large language models. Insights into their pretraining data will potentially lead to positive impacts-rev removing problematic data sources, demoting spurious correlations, and alleviating other ethical issues caused by our current inability to interpret decisions of large language models.
method (Strubell et al.)
C CONTINUED DISCUSSION ON EXPERIMENTAL SETUP
The sample size in the embedding kNN baseline For the embedding kNN baseline, we sample t 1000 task examples from Dask and find = {10, 20, 50, 100) most similar pretraining data to each of the task example. This should already be large enough since even with the smallest k t A is well over |S|] (meaning that we need to downsample anyway). We did not use the entire Dusk because IMDB has 25K examples, and the calculation has a heavy requirement on the storage and memory. Nevertheless, to check whether the current t is adequate, we experiment with t = 8000 and 9800 for IMDB and MNLI respectively. The performance is 74.82 and 45.04, no better than the / = 1000 performance in the main evaluation.
With unconstrained resources, one can extend the checkpoint proposal and even measure Shapley values Shapley 1951) of the data, an equitable valuation method (Ghorbani & Zou 2019).
This may be possible with the latest release of functorch (He & Zou 2021
We plan to work on and
expect significant speedup.
13

---

Under review as a conference paper at ICLR 2023
Exact optimizer steps based on S For all of our experiments, have |S| 2000 and batch size of 16. We additionally held out 5% of S during optimization as a sanity check for the LM loss. Therefore, at most 119 optimizer steps were applied in all of the experiments.
D USING THE TEST SET AS D
The use of test set over training set is deliberate choice in this work reasons.
24 There are two main
We assume that the task's test data is the only instantiation of the task, since in- terpreting a model deployed in a zero-shot setup. We cannot assume the availability of a "training set" in such scenario.
More importantly, this work is about interpretability, not absolute model performance, We are interpreting why the model can achieve good test performance, not a good training performance. Similar to other interpretability research, no matter feature attribution or instance attribution (Ribeiro et al. 2016, Koh & Liang 2017). we must use the test data, since we are interpreting the model 's behavior on test examples exactly. This may also help us reveal the data artifacts inside the test set (Han al. 2020).
That being said, we do have sanity checks at the place we use test data in ORCA, that the effect of using the training set gradient would be very close. For all intermediate stages, the cosine similarity between the test set and training set gradients is 0.977 +0.03 for IMDB and 0.947+0.012 for MNLI, in a range of [-1,1]
E CONTINUED DISCUSSION ON MAIN EVALUATION
Importance of selecting the first epoch of ORCA without later iterations ((S) 100) would actually hurt the performance (57.39 and 37.43 for IMDB and MNLI respectively). This is due to the imbalance of the selected pretraining data, favoring only one label in the task. ORCA's iterative selection strategy in this case is essential, a difference from the instance attribution methods in previous interpretability research.
several iterations While not shown the main evaluation table,
Calibration While focus of this work, prompt-based language models can be improved with calibration techniques such as PMIpe (Holtzman et al. (021). We did not use such calibration in our work because our pilot study shows a rather even prior distribution among the labels applying PMInC on our model (LM, template, verbalizers) in IMDB yields less than 1% performance im- provement.
Hyperparameter search hyperparameter search our ORCA experiments (whereas we did search for the embedding kNN baseline). possible that some other sets of ORCA hyperparameters can lead to better perfor- mance numbers than those in the main evaluation table. We will release all of the code, experiment scripts, and data at ANNNYMIIEED upon publication.
Due the high computing cost mentioned in SB we did not perform
F ADDITIONAL ANALYSIS RESULTS
In Figure we show the MAUVE similarity analysis on MNLI, accompanying the MAUVE analysis on IMDB in the main paper.
Also note that all the investigated algorithms, ORCA and baselines, have the same Disk setup. These numbers mean the effect of using training and test gradicnts is extremely similar, noting that the dimension of the gradients is very large (340M). Given random vectors and this dimension.
Pr[ cos(a, b)] 0.9] 1
2/75x 10
Ameaa 2013
14

---

Under review as a conference paper at ICLR 2023
Random
Embedding KNN
ORCA (NL)
ORCA
0.58
MAUVE with task data
0,56
0.54
0,52
0.50
20
e=
c=10
200
c=512
Context window size of supporting data evidence
Figure 3: MAUVE similarity on MNLI, between the sets of PT context in S and y.ukk
15


Reviews:
[{"review_id": "e-39YsP1PG", "reviewer": "Reviewer_uxbN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for finding the subset of a large language model (LLM) training corpus that is most responsible for the 0-shot performance of the LLM on a given task. The method involves finding examples whose gradient is similar to the gradient of task-specific supervised data. This is done in an iterative process in which the original LLM is fine-tuned on the examples found during the previous step, in order to account for interactions among examples. Performance is evaluated by fine-tuning the LLM on the identified subset, then measuring task performance. \n\nOn IMDB and MNLI tasks with BERT, the proposed method is found to outperform random-sampling and kNN baselines. The most striking finding in the resulting analysis is that most of the valuable examples come from the Books portion of BERT’s training corpus.\n", "summary_of_the_review": "The paper is a bit thin on substance, both in terms of theoretical justification and breadth of experimental results. However, it attacks a very central problem, is clearly written and proposes a simple, potentially very useful technique, so I am leaning in favour of acceptance.\n", "strength_and_weaknesses": "Strengths:\n\n1. The problem is important, as insights can inform selection of training data for future LLMs. The proposed technique thus has the potential to be useful in many different contexts.\n\n2. The idea is novel. Although gradient-based similarity is widely used, this specific application is new, as far as I know.\n\n3. The paper is very well written: the technique is described clearly and precisely, and the experiments should be easy to reproduce.\n\nWeaknesses:\n\n1. Although the technique is intuitive and easy to understand, it’s not clear to what extent it’s a good approximation of the true quantity of interest, namely performance if the LLM were retrained from scratch on the whole corpus with the identified subset removed. Influence functions obviously get at that more directly, and I think the paper should have included an influence-function baseline. This needn’t be an iterative procedure (which might be difficult to formulate), just the set of individual examples with the greatest influence.\n\n2. The experiments are somewhat limited: only on BERT, and only on two tasks. They are also limited in exploring only tiny subsets. It’s interesting to know that fine-tuning on such a set can provide a good performance boost (at least in the case IMDB), but it would be better to have a sense of where this flattens out as the subset is expanded.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OIFGULYRyy", "reviewer": "Reviewer_NC89", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new approach to find a subset of the pretraining corpus that supports BERT’s zero-shot predictions in a given task. This is done by iteratively finding pretraining examples whose gradient is the most similar to that of the downstream task examples. The technique is used to analyze the performance of BERT on IMDB and MNLI, finding that (i) BookCorpus is more relevant than Wikipedia despite the latter being larger, (ii) pretraining examples masking the downstream verbalizers are the most relevant, and (iii) the most relevant pretraining examples are not particularly similar to the downstream task examples on the surface, showing that the model is not relying on pure memorization.", "summary_of_the_review": "I think that this is a borderline paper, perhaps leaning a bit more on the negative said. It tackles an important problem from an original angle and is technically sound, but the analysis presented is rather shallow.", "strength_and_weaknesses": "STRENGTHS\n- The paper tackles an important problem (understanding how pretrained models are able to perform downstream tasks in a zero-shot fashion), and the approach followed is original and technically sound.\n- The finding that the context of the supporting data evidence is not particularly similar to the task input data is interesting and not obvious.\n\nWEAKNESSES\n- While I really like the general idea of the paper, the analysis itself is rather shallow. The main findings are (i) BookCorpus is more relevant than Wikipedia despite the latter being larger, (ii) pretraining examples masking the downstream verbalizers are the most relevant, and (iii) the most relevant pretraining examples are not particularly similar to the downstream task examples. While (iii) is interesting, (ii) is not surprising at all, and more analysis is necessary to interpret (i). For instance, it could be that the verbalizers (e.g. good and bad) occur more frequently in BookCorpus, in which case (i) would be implied by (ii) and not surprising at all.\n- I am not satisfied by the evaluation of the proposed approach in Section 4. ORCA outperforms all baselines in Table 1, but that is not surprising at all, as it is the only approach using labeled downstream data, and it works by picking examples from the pretraining corpus whose gradient is similar, therefore approximating gradient descent in downstream data. Some basic baselines are missing (e.g. only picking examples that mask the downstream verbalizers), and it would be useful to have a variant finetuning on the downstream task as an upper-bound to put the numbers in perspective. It would also be useful to finetune another pretrained model (e.g., RoBERTa) in the same set of examples (selected based on BERT): if these examples are truly supporting the task in question, I would expect them to be helpful for all models. In addition, the results in Table 2 are negative: even if the authors claim that they get an improvement on IMDB, this is likely not significant, as this applies to only one variant of the proposed method and even in that case the improvement is only 0.27 points while the standard deviation is 0.65. This is puzzling and deserves some more discussion: does this imply that all pretraining data is equally useful when using prompt-tuning?\n- The manual analysis of the examples is dissapointing. We are only shown a single example for each task and verbalizer. As the authors say, it is not clear at all how the examples mined for MNLI can be useful to learn entailment. The authors say that the examples mined for IMDB do express sentiment, but I think that even that is questionnable (the second one does, but the first one is talking about the genetics of bacteria). I would have liked to see more examples in the appendix and a more systematic analysis of them, with the aim to understand how interpretable the examples are.\n- The evaluation is limited to two tasks and a single model, using a very small fraction of the pretraining corpus. While I understand that the authors were constrained by compute, this cannot be use as a cheap excuse for everything. I would have liked to see more extensive experiments and, if truly not feasible due to compute constraints, the authors should at least report the compute required for each experiment to justify this (e.g. if each experiment takes a month on their hardware I would understand not having more, but if it takes a few hours this wouldn’t be a valid excuse).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CrzOAdtHaE", "reviewer": "Reviewer_sFZJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors want to explore a very interesting question why the prompting learning of large pretrained language models leads to strong performance in a variety of downstream tasks, especially in zero-shot setups? This paper proposed a novel method to identify the evidence of the model’s task-specific competence in prompt-based learning based on the general large pre-trained language model. It uses the gradient information related to the downstream task, IORCA can locate a small subset of pretraining data that is similar to the downstream tasks. This work is an very interesting exploration but not very novelty.", "summary_of_the_review": "Same with Summary Part.", "strength_and_weaknesses": "Strengths:\nThis paper proposed a gradient-based method to identify the evidence of the model’s task-specific competence in prompt-based learning based on the general large pre-trained language model. It is interesting.\n\n\nWeaknesses:\nNovelty of the proposed method is somewhat limited.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6A6O_fUTQup", "reviewer": "Reviewer_st4M", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors try to locates a very small subset of pretraining data that directly supports the model’s predictions in a given task. The main idea is to use cosine similarity between gradients of the downstream samples and pre-training corpus.  They designed experiments based on 0.5% of the Wikipedia and BookCorpus over IMDB and MNLI.", "summary_of_the_review": "Overall, I think the problem is not clearly defined. This causes the inconsistencie in the method and experimental descriptions. In addition, there is an issue with the choice of baselines.", "strength_and_weaknesses": "Strength:\n1. The authors provide a novel metric to locate pretraining data that supports the model’s predictions in a given task.\n2. Table 1 shows that using retrieved corpora can effectively improve the effectiveness of LMs in downstream tasks.\n\nWeaknesses:\n1. the problem is not well defined. A corpus that improves the effectiveness of a downstream task cannot necessarily be used to explain the predictions of the original language model. While many of the claims in this paper are about the interpretability of pretraining by locating training corpora, the experiments are mostly about performance improvement.\n2. If the goal is to improve model effects by some continued pretraining, the related work and experiments in this paper lack comparison with state-of-the-art methods, e.g. dense retriever methods like REALM.\n3. The method has some assumptions. However, they are not validated. For example, whether continuing training on the task data directly will improve the original model? Why the gradient of the sample to be found is similar to the gradient of the target task samples?\n4. I think the method in this paper cannot be scaled to large LMs. In LMs, there are at least millions of gradients for each sample.\n5. Some experiments are confusing. For example, I cannot understand why words like good/bad/yes/no happen to be masked out in Table 3. This is not consistent with the random mask strategy.\n6. Table 3 is inconsistent with the intuition of this paper. According to the examples in Table 3, it looks like the model is looking for verbalizer words, instead of the training corpora that explain the model behavior in the downstream task.\n7. I suggest that the authors use more downstream tasks for their experiments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject