First is Better Than Last for Language Data Influence
Authors: Chih-Kuan Yeh
Ankur Taly
Mukund Sundararajan
Frederick Liu
Pradeep Kumar Ravikumar
Venue: NeurIPS 2022 Accept (2022)
Keywords: influence
interpretation
explanation

Abstract:     The ability to identify influential training examples enables us to debug training data and explain model behavior. Existing techniques to do so are based on the flow of training data influence through the model parameters. For large models in NLP applications, it is often computationally infeasible to study this flow through all model parameters, therefore techniques usually pick the last layer of weights. However, we observe that since the activation connected to the last layer of weights contains "shared logic", the data influenced calculated via the last layer weights prone to a "cancellation effect", where the data influence of different examples have large magnitude that contradicts each other. The cancellation effect lowers the discriminative power of the influence score, and deleting influential examples according to this measure often does not change the model's behavior by much. To mitigate this, we propose a technique called TracIn-WE that modifies a method called TracIn to operate on the word embedding layer instead of the last layer, where the cancellation effect is less severe. One potential concern is that influence based on the word embedding layer may not encode sufficient high level information.  However, we find that gradients (unlike embeddings) do not suffer from this, possibly because they chain through higher layers. We show that TracIn-WE significantly outperforms other data influence methods applied on the last layer significantly on the case deletion evaluation on three language classification tasks for different models. In addition, TracIn-WE can produce scores not just at the level of the overall training input, but also at the level of words within the training input, a further aid in debugging. 

First is Better Than Last for Language Data Influence
Chih-Kuan Yeh Google Inc. chihkuanyeh google.com
Ankur Taly Google Inc. ataly @ google.com
Mukund Sundararajan
Google Inc. mukunds @ google.com
Frederick Liu
Pradeep Ravikumar
Google Inc.
Carnegie Mellon University
Department of Machine Learning
pradeepr@cs.cmu.cd
Abstract
The ability to identify influential training examples enables us to debug training data and explain model behavior. Existing techniques to do so are based on the flow of training data influence through the model parameters (Koh & Liang 2017]
Yeh et al.) 2018) Pruthi et al. 2020). For large models in NLP applications, it often computationally infeasible to study this flow through all model parameters, therefore techniques usually pick the last layer of weights. However, we observe that since the activation connected to the last layer of weights contains "shared logic", the data influenced calculated via the last layer weights prone to a "cancel- lation effect", where the data influence of different examples have large magnitude that contradicts each other. The cancellation effect lowers the discriminative power of the influence score, and deleting influential examples according this measure often does not change the model's behavior by much. To mitigate this, we propose technique called TracIn- WE that modifies method called TracIn (Pruthi al., 2020) to operate on the word embedding layer instead of the last layer, where the cancellation effect less severe. One potential concern is that influence based on the word embedding layer may not encode sufficient high level information. How- ever, we find that gradients (unlike embeddings) do suffer from this, possibly because they chain through higher layers. We show that TracIn-We WE significantly outperforms other data influence methods applied on the last layer significantly the case deletion evaluation on three language classification tasks for different models. In addition, TracIn-WF WE can produce scores not just at the level the overall training input, but also at the level of words within the training input, further aid debugging.
Introduction
Training data influence methods study the influence of training examples model's weights (learned during the training process), and in turn on the predictions of other test examples. They enable us debug predictions by attributing them the training examples that most influence them, debug training data by identifying mislabeled examples, and fixing mispredictions via training data curation. While the idea of training data influence originally stems from the study linear regression (Cook& & Weisberg 1982), it has recently been developed for complex machine learning models like deep networks.
Prominent methods for quantifying training data influence for deep networks include influence functions (Koh & Liang] 2017). representer point selection (Yeh al. 2018). and TracIn (Pruthi lt al., 2020). While the details differ, all methods involves computing the gradients (w.nt. the loss) of the model parameters at the training and test examples. Thus, they all face common computational
part of work done in CMU
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

challenge of dealing with the large number of parameters in modern deep networks. In practice, this challenge is cireumvented by restricting the study of influence to only the parameters in the last layer of the network. While this choice may not be explicitly stated, it is often implicit in the implementations of larger neural networks. In this work, we revisit the choice of restricting influence computation to the last layer in the context of large-scale Natural Language Processing (NLP) models.
We first introduce the phenomenon of "cancellation effect" of training data influence, which happens when the sum of the influence magnitude among different training examples is much larger than the influence sum. This effect increases the influence magnitude of most training examples and reduces the discriminative power of data influence. We also observe that different weight parameters may have different level of cancellation effects, and the weight parameters of bias parameters and latter layers may have larger cancellation effects. To mitigate the "cancellation effect" and find scalable algorithm, we propose to operate data influence on weight parameters with the least cancellation effect - the first layer of weight parameter, which is also known as the word embedding layer.
While word embedding representations might have the issue of not capturing any high-level input semantics, we surprisingly find that the gradients of the embedding weights do not suffer from this. Since the gradient chain through the higher layers, it thus takes the high-level information captured in those layers into account. As result, the gradients of the embedding weights of a word depend on both the context and importance of the word in the input. We develop the idea of word embedding based influence in the context of TracIn due to its computational and resource efficiency over other methods. Our proposed method, TracIn-WE. can be expressed as the sum of word embedding gradient similarity over overlapping words between the training and test examples. Requiring overlapping words between the training and test sentences helps capture low-level similarity, while the word gradient similarity helps capture the high-level semantic similarity between the sentences. A key benefit of TracIn-WE WE is that it affords a natural word-level decomposition, which is not readily offered by existing methods. This helps understand which words in the training example drive its influence on the test example.
We evaluate TracIn-WE on several NLP classification tasks, including toxicity, AGnews, and MNLI language inference with transformer models fine-tuned on the task. We show that TracIn WE outperforms existing influence methods on the case deletion evaluation metric by 4 x A potential criticism of Tracln- FracIn-WEinn is its reliance on word overlap between the training and test examples, which would prevent it from estimating influence between examples that relate semantically but not syntactically. To address this, we show that the presence of common tokens in the input, such as "start" and "end" token (which are commonly found in modern NLP models), allows T to capture influence between semantically related examples without any overlapping words, and outperform last layer based influence methods on restricted set of training examples that barely overlaps with the test example
2 Preliminaries
Consider the standard supervised learning setting, with inputs x E X' outputs y E y, and training data
D = { , 31 (Z2, (3. }a)}. Suppose we train a predictor f with parameter θ by minimizing some given loss function l over the training data, so that @ arg ming Σn ((f(x:), y:). In the context of the trained model f, and the training data D, we are interested in the data importance of a training point to the testing point a'. which we generally denote as I(2,,")
2.1
Existing Methods
We first briefly introduce the commonly used training data influence methods: Influence func- tions (Koh & Liang, 2017), Representer Point selection (Yeh et al. 2018), and TracIn (Pruthi et al. 2020 We demonstrate that each method can be decomposed into a similarity term S(x, x²), which measures the similarity between training point and the test point x', and loss saliency terms L(x) and L(x'), that measures the saliency of the model outputs to the model loss. The decomposition largely derives from an application of chain rule the parameter gradients.
1(x, = Lxx) x (xx))
The decomposition yields the following interpretation. A training data x has larger influence on a test point z' if (a) the training point model outputs have high loss saliency, (b) the training point x and the test point x' are similar as construed by the model. In Section 3.3 we show that restricting
code is in https:/gittabb.om/cckuuuaaaaaaat WE.

---

the influence method to operate on the weights in the last layer of the model critically affects the similarity term, and turn the quality of influence. We now introduce the form of each method, and the corresponding similarity and loss saliency terms. Influence Functions:
Inf x²) Vol(z, H Voll'a
where He is the hessian En= Vz,((, () computed over the training examples. By an application of the chain rule, we can see that Inf( = (zz) (') with the similarity term S(x, , x'
0f(z,0) T be H H and the loss saliency terms L(x) = @
(= The work by Sui et al. 2021 is
de
very similar to extending the influence function to the last layer to satisfy the representer theorem. Representer Points:
1 al(x, +) Rep( = 2An of, (x, 9)
a(z, f) a(z'.0))
(1)
where a(x, θ) is the final activation layer for the data point æ, À is the strength of the E regularizer used to optimize O, and j is the targeted class to explain. The similarity term is S(z,z') = a(z, O)'a(z', (), and the loss saliency terms are L(x) 2An (6, L(x'==() =1. TracIn:
(((x,9)
TracIn (z, (²,,'' = Σ Ne Vo, 0c)
Ve, Ve. (1', O.),
(2)
where O₂ is the weight at checkpoint e, and f). is the learning rate at checkpoint e. the remainder of the work, in our notation, we suppress the sum over checkpoints of TracIn for notational simplicity, (This is not to undermine the importance of summing over past checkpoints, which is a crucial compo- nent in the working on TracIn.) For TracIn, the similarity term is S(z, 2z) Vef(a, Ve' O), while the loss terms are (xx) =
0((z.e) "f L(x') = 9((=,e))
2.2 Evaluation: Case Deletion
We now discuss our primary evaluation metric, called case deletion diagnostics (Cook & Weisberg 1982), which involves retraining the model after removing influential training examples and measuring the impact on the model. This evaluation metric helps validate the efficayy of any data influence method in detecting training examples to remove or modify for targeted fixing of misclassifications, which is the primary application we consider in this work. This evaluation metric was also noted as a key motivation for influence functions (Koh & Liang 2017). Given a test example x', when we remove training examples with positive influence on x' (proponents), we expect the prediction value for the ground-trutt class of x' to decrease. On the other hand, when we remove training examples with negative influence on z' (opponents), we expect the prediction value for the ground-truth class of x' to increase.
An alternative evaluation metric is based detecting mislabeled examples via self-influence (i.e. influence of a training sample on that same sample as a test point). We prefer the case deletion evaluation metric, as it more directly corresponds the concept of data influence. Similar evaluations that measure the change of predictions of the model after a group of points is removed is seen in previous works. Han et al. (2020) measures the test point prediction change after 10% training data with the most and least in influence are removed, and Koh et al. (2019) measures the correlation of the model loss change after group of trained data is removed and the sum of influences of samples in the group, where the group can be seen as manually defined clusters of data.
Deletion curve. Given a test example x' and influence measure I, we define the metrics DEL (a', k, I) and DEI. (x', k, I) as the impact on the prediction of x' (for its groundtruth class) upon removing top-k proponents and opponents of x' respectively:
DEL 1, = [[f (xl 0++)) fo(z',
DEL (z', I) fxfa(' 0_,+) f.(z',0))
where, O4.k (0_,a) are the model weights learned when top-k proponents (opponents) according to influence measure I are removed from the training set, and is the groundtruth class of x'. The expectation is over the number of retraining runs. We expect DEL., to have large negative, and DEL._ to have large positive values. To evaluate the deletion metric at different values of k, may plot DEL.,, (x', k I) and DEL.. (x', ks I) for different values of k. and report the area under the curve (AUC): AUC-DEL.
Σ km k= DEL (a', k, I). and AUC-DEL- Eenk, DELL ( k, I).
3

---

We note that the case deletion diagnostics is different to the leave-one-ous evaluation of Koh & Liang (0177) by two points. First, leave-one-out evaluation focuses on removing one point, which is more meaningful in the convex regime where the optimization is anitializationnivvriaa leave-k-out evaluation which is closer to actual applications, as one may need to alter more than one training data to fix prediction. Second, we consider the expected value of leave-k-out, to hedge the variance caused by specific model states, which was pointed out by [Sgaard et al. (202T) to be a major issue for leave-one-out evaluation (especially when the objective is no longer convex).
We consider the
3 Cancellation Effect of Data Influence
The goal of a data influence method is to distribute the test data loss (prediction) across training examples, which can be seen as an attribution problem where each training example is an agent. We observe cancellation across the data influence attributions to training examples, i.c., the sign of attributions across training examples disagree and cancels each other out. This leads to most training examples having a large attribution magnitude, which reduces the discriminatory power of attribution-bant based explanations.
Our next observation is that the cancellation effect varies across different weight parameters. In particular, when weight parameter is used by most of the training examples, the cancellation effect is especially severe. One such parameter is the bias, whose cancellation effect is illustrated by the following example:
Example 3.1. Consider an example where the imput x E Rdd is sparse, and x; has feature with value I and all other features with value O. The funciciionfunctionn function has the form f(x) = x w + b. It follows that a set of optimal parameters are w; = yi b O. We further assume that the parameter b is initialized to O and has never changed during the gradient descent progress. In this case, it is clear that the bias parameter b is irrelevant to the model (as removing it will not change the model at all). However, the inftuence to individual examples caused by the bias b is still non-zero This is because even that the sum of gradient for bias b is O, (s.t. Σ; aLif(wi).mk) 0), each individual term (T(((2)))))) (L(f(a:)))) is non-zero for most Tú Note that (L(f(2.):1)) contributes to the influence 8b to data x; directly, and thus the bias parameter will contribute to the inftuence of all the training 8I(((2)))))) is only non-zero for T. and thus the data. In the contrary, for each weight variable Wi. Bw,
(f(22)
Ub
weight variable w only contributes to the influence of one training data F... Thus, the bias would affect the influence for more training examples compared to the weights.
The above example illustrates that while the bias parameter is not important for the prediction model (removing the bias can still lead to the same optimal solution). the total gradient that flows through the bias still high. In fact, we find empirically that the total influence that flows through the bias is larger than that flowing through the weight, since each training example's 's gradient will affect the bias but the total contribution will be cancelled out, so the bias will remain 0. We also note that even for deep network models that do not have a sparse input, the neurons connected to the weight are often (due to ReLU types of activation functions). Thus, the gradient of weight parameters is often sparser compared to the gradient of bias parameters, and thus bias parameters would often have stronger cancellations, which we validate empirically.
3.1 Measuring the Cancellation Effect
In the above example, we defined strong cancellation effect when some weight parameters does not change lot during training (or has saturated in the training process), but the total strength of the gradient of the weight parameters summed over training data is large. For weights W, we first define two terms AW, and G(W)..
AW. | Well.
A( Wi)
G(W). =
Σ nell
||.
aw.
where AW" measures the norm of weight parameter change between checkpoint c and c c 1, and G(W)), measures the sum of weight gradient norm times learning rate summed over all training data. When AW, is small, this means that the weight W may have saturated at checkpoint e, and the weight may not actually affect the model output much (and thus the weight W is not important for this epoch of training). When G()), is large, this means that the sum of gradient norm with respect to W. is still large, and the influence norm caused by a((a..v))) will also be large.
JW.

---

To measure the cancellation effect, we define the cancellation ratio of a weight parameter W as:
C(W) Le G(W).
E.AW
When G(W), is large and AW. is small, this means that a non-important weight We greatly influenced the total influence norm, which may be only possible if the influence contributed from W. to different examples cancelled each other out. Applying this interpretation to the cancellation of bias parameters, the intuition that the bias parameters are not mainly responsible for the reduction of testing example loss change during the training process (since AW, is small). However, they dominate the total influence strength due to their dense nature (GW large). Parameters with high cancellation may not be ideal to the calculation of influences.
3.2 Removing Bias In TracIn Calculation to Reduce Cancellation Effect
To investigate whether removing weights with high cancellation effect really helps improve influence quality, we conducted an experiment on CNN text classification on Agnews dataset with 87% test accuracy. The model is defined as follows: first a token embedding with dimension 128, followed by two convolution layers with kernel size 5 and filter size 10, one convolution layers with kernel size 1 and filter size 10, a global max pooling layer, and a fully connected layer; all weights are randomly initialized. The first layer is the token embedding, the second layer is the convolution layer, and the last layer is a fully connected layer. The model has 21222 parameters in total (excluding the token embedding), in which 102 parameters are bias variables. We find C(bia)) to be 16789, and C( ((weight) to be 2555, which validates that the bias variables have a much stronger cancellation effect than the weight variables. A closer analysis shows that G(bias) is similar to G (weighht) (627206 and 559142), but A (bias) is much smaller than A (weight) (0.74 and 4.37.) Even though the bias parameters has a much smaller total change compared to the weight parameters, their impact on the gradient norm (and thus influence norm) is even higher than the weight parameters. This verifies the intuition in Example 3.T that the bias parameter has a stronger cancellation effect since the gradient to bias almost activated for all examples despite the actual bias change being small. To further verify that the TracIn score contributed by the bias may lower the overall discriminatory power, we compute AUC-DEL, and AUC-DEL_ for TracIn and -reigh- on AGnews with our CNN model. The AUC-DEL. for TracIn and TracIn- weight is -0.036 and - -0.065 respectively, and the AUC-DEL, for TracIn and TragIn-weight is 0.011 and 0.046. The result shows that by removing the TracIn score contributed by the bias (with only 102 parameters), the overall influence quality improves significantly. Thus, in all future experiments, we remove the bias in calculation of data influence if not stated otherwise.
3.3 Influence of Latter Layers May Suffer from Cancellation
As mentioned in Section for scalability reasons, most influence methods choose to operate only on the parameters of the last fully- colle-conneeted layer m We argue that this is not a great choice, as the influence scores that stems from the last fully-connected weight layer may suffer from cancellation effect, as different examples "share logics" in the activation representation of this layer, and have a higher gradient similarity for different examples. Farly layers, where examples have unique logic, may suffer less from the cancellation effect. We first measure the gradient similarity for different examples for each layer, which is .x,, COS-SIM[i((z.) (z.)//w, (l(z))/0u) where COS-SIM, is the cosine similarity. This measures the expected gradient cosine similarity between two examples. The expected gradient similarity for testing examples between different layers in the CNN classification are: first 0.035, second 0.075, third 0.21, last 0.23. This verifies that the latter layers in the neural network have more aligned gradients between examples, and thus share more logies between training examples. We report the cancellation ratio for each of the TracIn layer varaint in Table D where TracIn-first, Tracon-second, Tthcln- TracIn-last, TracIn-All refer TracIn scores based on weights of the first layer, second layer, third layer, last layer, and all layers (the bias is always omitted). As we suspected, early layers suffers less from cancellation, and latter layers suffers more from cancellation. To assess the impact on influence quality, we evaluate the AUC-DEL, and AUC-DEL_ score for TracIn calculated with different layers on the AGnews CNN model in Tab. D We observe that removing examples based on influence scores calculated using parameters of later layers (with more "shared logic"") leads worse deletion score compared removing examples based on influence scores calculated using parameters of earlier layers (with more "unique logic"). Interestingly, the performance of TracIn-firy even outperforms TracIn-all where all parameters are used. We hypothesize that since the TracIn score based on later layers contain too much cancellation,
'We note that our investigation of last layer cancellation is limited to the setting when the whole model is trained to produce single classification score, which may not hold in the setting where only the last layer is fine-tuned or tasks with generative output.
5

---

Table I: Cancellation Ratio and AUC-DEL table for various layers in CNN model in AGnews.
Dataset
Metric
TR- firsirst
TR-second
TR-third
TR-last
TR-all
AGnews
Cancellation L
1863
2019
3126
2966
2368
AUC-DEL+ L AUC-DEL-
-0.077
-0.075
0.012
-0.016
-0.065
0.045
0.022
0.006
-0.032
0.046
Table 2: Examples for word similarity for different examples containing word "not".
Example
Premise
Hypothesis
Label
S1
think he is very annoying.
do not like him.
Entailment
S2
think reading is very horing.
do like read.
Entailment
S3
think reuding is very buring.
do not hate burying myself in books,
Contradiction
S4
She not only sturted playing the piano before she could speak, but her dad taught her to compose music at the same time.
She started to playing music and mak-
Entailment
ing music from very long ago.
$5
think he is very annoying.
don't like him. She doesn't 't
Entailment
S6
She thinks reading is pretly boring She only started playing the piano before she could speak, but her dad taught her to compose music at the same time
read
Entailment Entailment
S7
She started to playing music and mak- ing music from quite long
is actually harmful include these weight parameters in the TracIn calculation. In the following, we develop data influence methods by only using the first layer of the model, which suffers the least from cancellation effect.
Word Embedding Based Influence
In the previous section, we argue that using the latter layers to calculate influence may lead to the cancellation effect, which over-estimates influence. Another option is calculate influence all weight parameters, but may be computational infeasible when larger models with several millions of parameters are used. To remedy this, we propose operating on the first layer of the model, which contains the less cancellation effect since early layers encodes "unique logit". The first layer for language classification models is usually the word embedding layer in the case of NLP models. However, there are two questions in using the first layer to calculate data influence: 1. the word (token) emhedding contains most of the weight parameters, and may be computational expensive 2. the word embedding layer may not capture influential examples through high-level information. In the rest of this section, we develop the idea of word embedding layer based training-date influence in the context of TracIn. We focus on TracIn due to challenges in applying the other methods to the word embedding layer: influence functions on the word embedding layer are computationally infeasible due to the large size (vocab size x embedding dimension) of the embedding layer, and representer is designed to only use the final layer. We show that our proposed influence score is scalable thanks to the sparse nature of word embedding gradients, and contains both low-level and high-level information since the gradient to the word embedding layer can capture both high-level and low-level information about the input sentence.
4.1 TracIn Word Embedding Layer We now apply TracIn on the word embedding weights, obtaining the following expression:
TracIn- TracIn-WE(z, WE(2, x') = 0l(x, r. θ) dE(x', θ)
(3)
d0wB
d0wE
Implementing the above form of TracIn- WE would be computationally infeasible as word embedding layers are typically very large (vocab size X embedding dimension). For instance, a BERT-bass model has 23M parameters in the word embedding layer. To circumvent this, we leverage the sparsity @f(z,0) which is a sparse vector where only embedding weights associated with words that occur in x have non-zero value. Thus, the dot product between two word embedding gradients has non-zero values only for words w that occur in both x x'. With this observation, we can rewrite TracIn-WFF as:
of word embedding gradients Ztew
)l(x)) T d0(x')
TracIn- (HaIn-WE(rr,d))
Σ
wExrwa' ae
de
where θ, are the weights of the word embedding for word w. We call the term 8F(#)" word gradient similarity between sentences x' over word
at(x') the 0e.. 80_

---

Table 3: Word Decomposition Examples for TracIn-WE
Sentence content
Label
Test Sentence
can always end my conversations so you would not get any answers because
Toxic
you are too lazy to remember anything
Test Sentence 2 T2
For me, the lazy days of summer is not over yet, and I advisc you to please kindly
Toxic
consider to end one's 's life, thank you
Train Sentence- S1
Oh yeah, if you're re too lazy to fix tags yourself, you' 'r supporting Al universal takeover in 2020, end it. kill now.
Non-Toxic
Word Importance
Total
TracIn-WE(S1, T1)
[S]: -0--0.2 -.228 [B]: -U.07, to: 0.115, lazy: -7.6, you: -0.3, end: c03: t00:-0.3
-9.2
(eESn T2)
[S]: -0.17, [E]: -0.23, to: 0.54, lazy: -0.25, you: 0.25, end: -3.12
-3.45
4.2
Interpreting Word Gradient Similarity
[ Fiives the impression that TracIn-WE merely considers a bag-of-words style similarity between the two sentences, and does not take the semantics of the sentences into account. This is surprisingly not true! Notice that for overlapping words, TracIn- WE considers the similarity between gradients of word embeddings. Since gradients are pack-propagated through all the intermediate layers in the model, they take into account the semantics encoded in the various layers. This is
Equation
aligned with the use of word gradient norm @f(=)
measure of importance of the word the prediction (xx) (Wallace al. 2019 Simonyan et al. 2013 Thus, word gradient similarity would be larger for words that are deemed important to the predictions of the training and test points.
Word gradient similarity is not solely driven by the importance of the word. Surprisingly, we find that word gradient similarity is also larger for overlapping words that appear in similar contexts in the training and test sentences. We illustrate this via example. Table 2 shows 4 synthetic premise- hypothesis pairs for the Multi-Genre Natural Language Inference (MNLI) task (Williams) al. 2018). An existing pretrained model (He al., 2020 predicts these examples correctly with softmax probability between 0.65 and 0.93. Notice that all examples contain the word 'not' once.
The word gradient importance || do for "not" is comparable in all 4 sentences. The value of word gradient similarity for 'not' is 0.34 for the pair S1-S2, and -0.12 for S1-S3, while it is -0.05 for S1-S4. This large difference stems from the context in which 'not' appears. The absolute similarity value is larger for S1-$2 and S1-S3, since 'not' appears in a negation context in these examples. (The word gradient similarity of $1-S3 is negative since they have different labels.) However, in $4, 'not' appears in the phrase "not only... ... but". which is not negation (or can be considered as double negation). Consequently, word gradient similarity for 'not' is small between SI and S4. In summary, we expect the absolute value of TracIn-WE score to be large for training and test sentences that have overlapping important words in similar (or strongly opposite) contexts. On the other hand, overlap of unimportant words like stop words would not affect the TracIn- WE score.
(f(
4.3 Word-Level Decomposition for TracIn-WE
An attractive property of WE is that decomposes into word-level contributions for both the testing point x' and the training point æ. As shown ( word w in contributes to TracIn-WE(r, x')
by the amount al(x)) T ae(=')
ae... de. [[ E æ²); similar word-level decomposition can be obtained for
x'. Such a decomposition helps identify which words in the training point (x) drive its influence towards the test point (x'). For instance, consider the example in Table. 3 which contains two test sentences (TI, T2) and training sentence SI. We decompose the score WEESI, TI) and into words contributions, and we see that the word "lazy" dominates TracIn- WE(S1, TI), and the word "end" dominates TracIn- WE(S1, T2). This example shows that different key words in training sentence may drive influence towards different test points. The feature- decomposition for influence introduces additional interpretability to why two examples are highly influenced. This is demonstrated in case study where we cluster difficult training examples based on a normalized TracIn-Wi WE score in Sec. A 4.4 An approximation for TracIn WE As we note in Sec. the space complexity of saving training and test point gradients scales with the number of words in the sentence. This may be intractable for tasks with very long sentences. We alleviate this by leveraging the fact that the word embedding gradient for word w is the sum of input word gradients from each position where w is present. Given this decomposition, we can approximate the word embedding gradients by saving only the top-k largest input word gradients for
7

---

E300
Figure I: Deletion Curve for removing opponents (top figure, larger better) and proponents (bottom figure, smaller better) on Toxicity (left), AGnews (mid), and MNLI (right).
each sentence. (An alternative is to save the input word gradients that are above certain threshold.) Formally, we define the approximation
0l(I, +)
al 9)
Σ
a
top-k
Ox'
iEwrtk
where x² is the word at position i, and tot the set of top-k input positions by gradient norm. We then propose
0E(x, O.))
af(z', Ow)
TracIn ThaIn-WEETTopkee (E-
E
80w
kop
d9w.
top-k.
(6)
Computational complexity Let L be the max length of each sentence, d be the word embedding dimension, and o be the average overlap between two sentences. If the training and test point gradients are precomputed and saved then the average computation complexity for calculating TracIn-WE for m training points and n testing points is ((mnod). This can be contrasted with the average computation complexity for influence functions on the word embedding layer, which dd where v the vocabulary size which is typically larger than 10 and typically less than 5. The approximation for TracIn-We- Me-n--W-Toopp Topk drops the computational complexity from O(mnod) O (mmord) where O₂ the average overlap between the sets of top-k words from the two sentences. It has the additional benefit of preventing unimportant words (ones with small gradient) from dominating the word similarity by multiple occurrences, as such words may get pruned. In all our experiments, we set & to 10 for consistency, and do not tune this hyper-parameter.
takes O(mnd
4.5 Influence without Word-Overlap
One potential criticism of TracIn WE is that it may not capture any influence when there are no overlapping words between v and x'. To address this, we note that modern NLP models often include a "start" and "end" token in all inputs. We posit that gradients of the embedding weights of these tokens take into account the semantics of the input (as represented in the higher layers), and enable TracIn-W WE to capture influence between examples that are semantically related but do not have any overlapping words. We illustrate this in S5-S7 in Tab, 2 via examples for the MNLI task. Sentence S5 has no overlapping words with S6 and S7. However, the word gradient similarity of "start" and "end" tokens for the pair S5-S6 1.15, while that for the pair S5-S7 is much lower -0.05. Indeed, sentence $5 is more similar to S6 than S7 due to the presence of similar word pairs (e.g., think and thinks, annoying and boring), and the same negation usage. We further validate that TracIn- --E can capture influence from examples without word overlap via controlled experiment in Sec. 5 5 Experiments
We evaluate the proposed influence methods on 3 different NLP classification datasets with BERT models. We choose a transformer- based model as it has shown great success on a series of down- stream tasks in NLP, and we choose BERT model as it is one of the most commonly used transformer model. For the smaller Toxicity and AGnews dataset, we operate on the Bert-Small model, as it already achieves good performance. For the larger MNLI dataset, we choose the Bert-Base model with 10M model parameters, which is decently large model which we believe could represent the effectiveness of our proposed method on large-scale language models. As discussed in Section 2.2 we use the case deletion evaluation and report the metrics on the deletion curve in Table for various methods and datasets. The standard deviation for all AUC values methods reported in Table 7
8

---

Table 4: AUC-DEL table for various methods in different datasets. Highest number is bold.
Dataset
Metric
Inf-Last
Rep
TR-last
TR-WE
TR to-
TR- TR-TFIDF
commono
Toxic
+UC-DEL+ 4
-0.008
-0.008
-0.013
-0.100
-0.099
-0.067
0.016
Bert
AUC-DEL- 1
0.014
0.021
0.023
0.149
0.151
0.063
0.014
AGnews
AUC-DEL+ 1
-0.018
-0016
-00021
-0.106
-0.174
-0.090
-0.017
Bert
AUC-DEL
0.033
0.028
0.028
0.130
0.131
0.072
0.023
MNLI
AUC-DEL+
0.006
-0.198
-0.004
Bert
AUC-DEL
0.026
0.169
0.005
Toxic
AUC-DEL+
0.011
-0.004
0.001
-0.030
-0.038
-0.001
-0.001
Roberta
AUC-DEL
0.023
0.012
0.003
0.033
0.030
0.006
0.010
Dataset
Metric
Inf-Last
Rep
TR-last
TR-WE
TR. TR-WE-topk
TR-WE-NOC
TR-common
Toxic
AUC-DEL 4
-0.009
-0.008
-0.006
-0.018
-0.016
0.003
-0.008
Nooverlap
AUC-DEL- f 0.008
0.007
0.010
0.026
0.026
0.001
0.015
Baselines One question to ask is whether the good performance of TracIn- is a result that captures the low-level word information well. To answer this question, we design a synthetic data influence score as the TF-IDF similarity Salton & Buckley (1988) multiplied by the loss gradient dot product for z and x'. TR-TFIDF can be understood by replacing the embedding similarity of TracIn-Last by the TF-IDF similarity, which captures low level similarity.
al(r, θ) T a0 θ)
TR-TFIDF(x, x/) - TF-Idd( x²)
(f(z. Q) (f
Toxicity. We first experiment on the toxicity comment classification dataset (Kaggle.com 2018). which contains sentences that are labeled toxic or non-toxic. We randomly choose 50, 000 training samples and 20, 000 validation samples. We then fine-tune BERT-small model on our training set, which leads to 96% accuracy. Out of the 20, 000 validation samples, we randomly choose 20 toxic and 20 non-toxic samples, for a total of 40 samples as our targeted test set. For each example x' in the test set, we remove top-k proponents and top-k opponents in the training set respectively, and retrain the model to obtain DEL+ (x', k, I) and DEL_ (x', kt I) for each influence method I. We vary k over {10, 20, ..... 1003. For each k. we retrain the model 10 times and take the average result, and then average over the 40 test points. We implement the methods Influence-last, Representer Points, TracIn-last, Tra TracIn-WE-Ton Wo-Topk, TracIn-TFIDF (introduced in Sec. G TracIn-common (which is a variant of TracIn only using the start token and end token to calculate gradient), and abbreviate TracIn with TR in the experiments. We see that our proposed TracIn-WE method, along with its variants TracIn-WE- Wo-k outperform other methods by a significant margin. As mentioned in Sec. 3.3 TF-IDF based method beats the existing data influence methods using last layer weights by a decisive margin as well, but is still much worse compared to TracIn-WEF Therefore, TracIn WE did not succeed by solely using low-level information. Also, we find that TracIn- WEE performs much better than TracIn-common, which uses the start and end tokens only. This shows that the keyword overlaps (such as lazy, end in Table 3 is crucial to the great performance of TracIn-WE.
Zhang et al. 2015). which contains a corpus of news with 4 different classes. We follow our setting in toxicity and choose 50, 000 training samples. 20, 000 validation samples, and fine-tune with the same BERT-small model that achieves 90% accuracy on this dataset. We randomly choose 100 samples with 25 from each class as our targeted test set. The AUC-DEL4 and AUC-DEL_ scores for k {10, 20, ..... 100} are reported in Table + Again, we see that the variants of TracIn significantly outperform other existing methods applied on the last layer. In both AGnews and Toxicity, removing 10 top-proponent or top- opponnenss for TracIn- WE has more impact on the test point compared to removing 100 top-proponents or top-opponents for TrasIn-last.
AGnews. We next experiment on the AG-news-subset (Gulli 2015
MNLI. Finally, we test on larger scale dataset, Multi-Genre Natural Language Inference (MultiNLI) Williams et al. (2018), which consists of 433k sentence pairs with textual entailment information, including entailment, neutral, and contradiction. In this experiment, we use the full training and validation set, and BERT-base which achieves 84% accuracy on matched-MNN valida- tion set. We choose 30 random samples with 10 from each class as our targeted test set. We only evaluate TracIn-WE-TToe run at large scale. We vary i E (20, 5000]. and the AUC-DEI+ and AUC-DEL._ scores for our test set are reported in Table g Unlike previous datasets, here TracIn-TFI TFIDF does not perform better
TracIn-last and TraCIn-TFIDF as those were the most efficient methods to

---

than TracIn-Last, which may be because input similarity for MNLI cannot be merely captured by overlapping words. For instance, single negation would completely change the label of the sentence. However, we again see TracIn- TracIn-WE-Topp significantly outperforms TracIn-Last and TracIn- TFFDD, demonstrating its efficacy over natural language understanding tasks as well. This again provides evidence that TracIn WE can capture both low-level information and high-level information. The deletion curve of Toxicity, AGnews, MNLI is in shown in Fig. 4.5 and Fig. DI
foxicity-Roberta To additionally test whether our experiment results apply to more modern models, we repeat our experiments on the toxicity dataset with a Roberta model Liu et al. (2019), while fixing other settings. We find that the TracIn- WEE and outperforms other results.
still significantly
No Word Overlap. To assess whether TracIn WE can do well in settings where the training and test examples do not have overlapping words, we construct a controlled experiment on the Toxicity dataset. We follow all experimental setting for Toxicity classification with the Bert model, but making two additional changes - (1) given test sentence z', we only consider the top-5000 training sentences (out of 50, 000) with the least word overlap for computing influence. We use TF-IDF similarity to rank the number of word overlaps so that stop word overlap will not be weighte,. (2) We also fix the token embedding during training (result when word-embedding is not fixed is in the appendix, where removing examples based any influence method does not change the prediction), as we find sentence with no word overlaps carry more influence when the token embedding is fixed. The AUC-DEL.4 and AUC-DEL__ scores reported in the lower section of Table 4 We find that TracIn-WE variants can outperform last-layer based influence methods even in this controlled setting, showing that TracIn-WE can retrieve influential examples even without non-trivial word overlaps. In Section 4.5 we claimed that this gain stems from the presence of common tokens ("start", "end", and other frequent words). To validate this, we compared with a controlled variant, TracIn- WE- NoCommon (TR-WE-NoC) where the common tokens are removed from TracIn TracIn-WE. As expected, this variant performed much worse on the AUC-DEL+ and claim. We also find that the result of TracIn-WFF is better than Tramln-common (which is TracIn-W WE with only "start" and "end"" tokens), which shows that the common tokens such as stop words and punctuation may also help finding influential examples without meaningful word overlaps.
thus confirming our
Related Work
In the field of explainable machine learning, works belongs training data importance (ooh &
Liang 2017 Yeh et al. 2018; Tia et al. [2019] Pruthi et al.
2020: Khanna et al. 2018)] Sui et al. (021))
Other forms of explanations include feature importance beature-based explanations, gradient- -base
explanations (Bachrens et al.
2010; Simonyan et al. 2013) Zeiler & Fergus 20014) Bach el al. 2015
Ancona et al.
2018:|Sundan (Sundararaaan
al. [201 7| (hhrikumar al. 2017 Ribeiro al. [016]] Lundberg
& Lee| 2017: Ych et al.
2019
Petsiuk et al. (2188) and perturbation-
explanations (Ribeiro et al. Yeh et al.] [2019] Petsiuk et al.) 2018). self-explaining models (Wang
2016
Lundberg & Lee
2017
Rudin 2015; Lee et al.
2019
Chen et al., [2019). counterfactuals to change the outcome of the model
(Wachter et al. [017]] Dhurandhar et al., [018]] Hendricks et al. 2018 van der Waa al. 2018 Goyal
et al. 2019), concepts of the model (Kim et al. 2018 Zhou et al. 2018). 3. For applications on applying
data importance methods on NLP tasks, there have been works identifying data artifacts (Han et al.
2020; Pezeshkpour et al.] (02T) and improving models (Han & Tsvetkov] 2020 (021) based on
existing data importance method using the influence function or TracIn this work, we discussed weight parameter selection to reduce cancellation effect for training data attribution. There has been works that discuss how to cope with cancellation in the context of feature attribution: [iu et al. (2020) discusses how regularization during training reduces cancellation of feature attribution, Kapishnikov ct al. (202T) discusses how to optimize IG paths to minimize cancellation of IG attribution, and Sundararajan et al. (2019) discusses improved visualizations to adjust for cancellation.
Conclusion
In this work, we revisit the common practice of computing training data influence using only last layer parameters. We show that last layer representations in language classification models can suffer from the cancellation effect, which in turn leads to inferior results on influence. We instead recommend computing influence on the word embedding parameters, and apply this idea to propose variant of TracIn called TracIn-Wi WE. We show that TracIn- WE significantly outperforms last versions of existing influence methods on three different language classification tasks for several models, and also affords a word-level decomposition of influence that aids interpretability.
10

---

References
Ancona, M., Ceolini, E., Öztireli, C., and Gross, M. A unified view of gradient- attribution methods for deep neural networks. International Conference Learning Representations, 2018.
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., and Samek, W. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10 (7):00300400, 2015.
Bachrens, D., Schroeter, T., Harmeling, S., Kawanabe, M., Hansen, K., and MAäller, K.-R. How to explain individual classification decisions. Journal of Machine Learning Research, 11(Jun): 1803-1831, 2010.
Barshan, E., Brunet, M.-E., and Dziugaite, G. K. Relatif: Identifying explanatory training samples via relative influence. In International Conference on Artificial Intelligence and Statistics, pp.
1899-1909. PMLR, 2020.
Chen, C., Li, O., Tao, D., Barnett, A., Rudin, C., and Su, J. K. This looks like that: deep learning for interpretable image recognition. In Advances in Neural Information Processing Systems, pp. 8928 8988-899,,98 2019.
Cook, D. and Weisberg, S. Residuals and influence in regression. New York: Chapman and Hall, 1982.
Dhurandhar, A., Chen, P.-Y., Luss, R., Tu, C.-C., Ting, P., Shanmugam, K., and Das, P. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. Advances in Neural Information Processing Systems, pp. 592-603. NeurlPS, 2018.
Goyal, Y., Wu, Z., Ernst, J., Batra, D., Parikh, D., and Lee, S. Counterfactual visual explanations. In International Conference on Machine Learning, pp. 2376-2384, ICML, 2019.
Gulli, A. Ag corpus of news articles. http://groups.di.mggg dinipii 2015.
unipi. it/ gulli/AG_corppp
Han, X. and Tsvetkov, Y. Fortifying toxic speech detectors against disguised toxicity. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp- 7732-7739, 2020.
Han, X. and Tsvetkov, Y. Influence tuning: Demoting spurious correlations via instance attribution and instance-driven updates. arXiv preprint 1rXiv:21/0.032// 2021.
Han, X., Wallace, B. C., and Tsvetkov, Y. Explaining black box predictions and unveiling data artifacts through influence functions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5553-5563, 2020.
Hanawa, K., Yokoi, S., Hara, S., and Inui, K. Evaluation of similarity -based explanations. In ICLR, 2021.
He, P., Liu, X., Gao, J., and Chen, W. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.0366 03654, 2020.
Hendricks, L. A., Hu, R., Darrell, T., and Akata, Grounding visual explanations. In ECCV. ECCV, 2018.
Jia, R., Dao, D., Wang, B., Hubis, F. A., Hynes, N., Gürel, N. M., Li. B., Zhang, C., Song, D., and Spanos, C. J. Towards efficient data valuation based on the shapley value. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 1167-1176. PMLR, 2019.
Kaggle.com. Toxic comment classification challenge: Identify and classify toxic online comments. 2018
Kapishnikov, A., Venugopalan, S., Avci, B., Wedin, B., Terry, M., and Bolukbasi, T. Guided integrated gradients: An adaptive path method for removing noise. In Proceedings of the IEEF/VVF Conference Computer Vision and Pattern Recognition, pp. 5050-5056 2021.

---

Khanna, R., Kim, B., Ghosh, J., and Koyejo, O. Interpreting black box predictions using fisher kernels. arviv preprint arXiv:1810.10111 pp. 3382-3390, 2018.
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F,, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International Conference on Machine Learning, pp. 2673-2688 ICMI,, 2018.
Koh, P. W. and Liang, P. Understanding black-box predictions via influence functions. In International Conference Machine Learning, pp. 1894. (CM,2011)
Koh, P. W. W., Ang, K.-S., Teo, H., and Liang, S. On the accuracy of influence functions for measuring group effects. Advances in neural information processing systems, 32, 2019.
Lee, G.-H., Jin, W., Alvarez-Melis, D., and Jaakkola, T. Functional transparency for structured data: game-theoretic approach. In International Conference on Machine Learning, pp. 3723-3733. PMLR, 2019.
Li B., Zhou, H., He, J., Wang, M., Yang. Y., and Li, L. On the sentence embeddings from pre-trained language models. arXiv preprint arXiv:201 1. .05864, 2020.
Liu, F., Najmi, A., and Sundararajan, M. The penalty imposed by ablated data augmentation. arXiv preprint arXiv:2006.047700 2020.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 10X.v:190..1199. 2019.
Lundberg, S. M. and Lee, S.-I. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pp. 4765-4774. NeurIPS, 2017.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Petsiuk, V., Das, A., and Saenko, K. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.07422 2018.
Peyré, G., Cuturi, M., et al. Computational optimal transport: With applications data science. Foundations and Trends@ in Machine Learning, (55-)):3556607) 2019.
Pezeshkpour, P., Jain, S., Singh, S., and Wallace, B. C. Combining feature and instance attribution to detect artifacts. arXiv preprint arXi::2107.003323 2021.
Pruthi, G., Liu, F., Kale, S., and Sundararajan, M. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33, 2020.
Ribeiro, M. T., Singh, S., and Guestrin, C. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 1135-1 144, ACM, 2016.
Salton, G. and Buckley, C. Term-weighting approaches in automatic text retrieval. Information processing & management, 24(5):513-523, 1988.
Shrikumar, A., Greenside, P., and Kundaje, A. Learning important features through propagating activation differences. International Conference on Machine Learning, 2017.
Simonyan, K., Vedaldi, A., and Zisserman, Deep inside convolutional networks: Visualising
image classification models and saliency maps. arXiv preprint arXiv: 13/2.6033 2013.
Søgaard, et al. Revisiting methods for finding influential examples. arXiv:2111.04683, 2021.
arXiv preprint
12

---

Sui, Y., Wu, G., and Sanner, S. Representer point selection via local jacobian expansion for post- hoc classifier explanation of deep neural networks and ensemble models. Advances in Neural Information Processing Systems, 34, 2021.
Sundararajan, M., Taly, A., and Yan. Q. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pp. 3319-333111 3328. PMLR, 2017.
Sundararajan, M., Xu, J., Taly, A., Sayres, R., and Najmi, A. Exploring principled visualizations for deep network attributions. In IUI Workshops, volume 4, 2019.
van der Waa, J., Robeer, M., van Diggelen, J., Brinkhuis, M., and Neerinex, M. Contrastive Explanations with Local Foil Trees. In 2018 Workshop on Human Interpretability in Machine Learning (WHI). WHI, 2018.
Wachter, S., Mittelstadt, B. D., and Russell, C. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. European Economics: Microeconomics & Industrial Organization eJournal, 2017.
Wallace, E., Tuyls, J., Wang, J., Subramanian, S., Gardner, M., and Singh, S. Allennlp interpret: A framework for explaining predictions of nlp models. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) System Demonstrations, pp. 7-12, 2019.
Wang. F. and Rudin, C. Falling rule lists. In Artificial Intelligence and Statistics, pp. 1013-1022, 2015.
Williams, A., Nangia, N., and Bowman, S. A broad-coverage challenge corpus for sentence derstanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Vol- ume 1 (Long Papers), pp. 1112-1122. Association for Computational Linguistics, 2018. URL http://aclwe org anthaaaghooooyaaaa -101
Yeh, C.-K., Kim, J., Yen, E.-H., and Ravikumar, K. Representer point selection for explaining deep neural networks. In Advances in Neural Information Processing Systems, pp. 929999911999111 9301.. NeurIPS, 2018.
Yeh, C.-K., Hsich, C.-Y., Suggala, A. S., Inouye, D. I., and Ravikumar, P. On the (in)fidelity and sensitivity of explanations. In NeurlPS, volume abs/1901. .00392,, pp. 10965-10976, 2019.
Zeiler, M. D. and Fergus, R. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833. Springer, 2014.
Zhang, X., Zhao, J., and LeCun, Y. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pp. 649-657, 2015.
Zhou, B., Sun, Y., Bau, D., and Torralba, A. Interpretable basis decomposition for visual explanation. In Proceedings of the European Conference Computer Vision (ECCV), pp. 119-134, ECCV, 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In Sec.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] In Sec.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results....
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
13

---

(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] In Sec. D
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] In Sec.
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent
obtained from people whose data you're
using/curating? [N/A] Only used public data.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "TIRBqCrcMfz", "reviewer": "Reviewer_R7Dm", "summary": "This paper thoroughly examines the effectiveness of different parameters serving as data influence measurement. It provides two important insights on the informativeness of parameters in lower layers and cancellation effect of parameters used by more training examples. Motivated by these insights, it develops a novel data influence method by replacing parameters of the last layer used in TracIn with embedding layer. Empirical results verify the insights and the effectiveness of the proposed method.", "questions": "* Is it essential to use *gradient-based* similarity as the similarity term $S$?\n* Some typos:\n  * line 36: parametersin -> parameters in\n  * line 194: \"share\" logic -> \"share logic\"\n  * line 215: multiple periods.\n  * Appendix Figure 4: or -> on\n", "limitations": "* The proposed method only works well when word embedding is fixed.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* This paper provides two valuable insights: \n  * First, the gradients of the embedding weights capture high-level information due to gradient chain.\n  * Second, parameters used by more training examples suffer more from 'cancellation effect' (i.e. gradients of multiple instances have large magnitude, but different signs).\n* The proposed method takes both low- and high-level similarity into consideration. \n* Good writing and clear presentation.\n\n**Weakness**\n* The terms of 'share logic' and 'unique logic' are vague. Could you provide explanations and evidence?\n* There should be an ablation study for using special tokens only on full dataset to demonstrate the effectiveness of word overlap.\n* Removing data points also change the training process (i.e. schedule of batches). How do you control related variables to be consistent for different methods? Does the random seed of data sampler influence the model performance significantly?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Pao4Fc3hlvQ", "reviewer": "Reviewer_iZ6x", "summary": "The paper proposes TracIn-WE that will produce data influence scores based on the level of the overall training input and at the level of words within the training input. The method mitigates an effect called “cancellation effect” resulting from the data influence extraction methods based the last layer weights. The word-embedding layer based proposed method (TracIn-WE) is built on the existing method called TracIn which is constructed on the last layer. ", "questions": "What is the interpretation of cancellation ratio? \n\n", "limitations": "They have discussed limitations.\n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\nThe method is based on the observation of existing data influence method’s cancellation across the data influence attributions (i.e., distributed test data loss) to training examples, i.e., the sign of attributions across training examples disagree and cancels each other out.  This kind of observation might be useful to develop improved methodologies for explainable NLP.\n\nWeakness\n\n1.\tExample 3.1 is not clear to me as the example is based on a sparse representation whereas the word embedding inputs are dense representations. \n\n2.\tThe word embeddings provide the input representation. As a result, the similarity of inputs (train and test) would better capture by the embedding space. However, the data’s influence on a prediction is related to the output layers (last layer) which is closely linked to loss function. Using embedding space for data influence extraction has some limitations in terms of deep neural network training (e.g., vanishing gradient problem). It is not clear how the proposed method solves those fundamental problem (e.g., learning and loss relevance) when calculating the data influence based on the embedding layer.  \n\n3.\tBias plays different rules than other weight. Removing Bias from the TracIn Calculation to Reduce Cancellation Effect needs further justification. What is the interpretation of cancellation ratio?  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5iHmGfCjcV", "reviewer": "Reviewer_LSEU", "summary": "This paper revisits the common practice of computing training data influence, especially methods for computing the influence only from the last layer and investigates improving the estimation of examples' influence.\nThe model size used in the NLP field is huge because we often incorporate a pre-trained language model as a base model.\nWe usually compute influence functions only from the final layers.\nThis paper also reveals that last layer representations in text classification models may suffer from the cancellation effect when computing the influence function on training samples.\nThis paper proposes a technique called TracIn-WE to mitigate the cancellation effect.\nTracIn-WE is a modification of the existing method, TracIn, to compute the influence of each sample on the word embedding layer instead of the last layer.\nThis paper conducts experiments on three text classification tasks and shows that TracIn-WE significantly outperforms other data influence methods applied on the last layer.\n", "questions": "* Have you ever tried to visualize the reducing cancellation effect for each layer?", "limitations": "This paper has limitation and potential negative societal impact sections in Appendix.\nThere are no additional concerns.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* This paper reveals the issues of existing influence computation methods by calculated from the last layer.\n* The empirical evaluation shows the better performance on the proposed method.\n* This paper tackles the important research theme involved in XAI.\n\n\nWeaknesses:\n* The method is actually a straight forward extension, so that the technical novelty is marginal.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept