Title: DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: in-context learning

data attribution

large language models
Abstract: In-context learning (ICL) allows transformer-based language models that are pre-trained on general text to quickly learn a specific task with a few "task demonstrations"  without updating their parameters, significantly boosting their flexibility and generality. ICL possesses many distinct characteristics from conventional machine learning, thereby requiring new approaches to interpret this learning paradigm. Taking the viewpoint of recent works showing that transformers learn in context by formulating an internal optimizer, we propose an influence function-based attribution technique, DETAIL, that addresses the specific characteristics of ICL. We empirically verify the effectiveness of our approach for demonstration attribution while being computationally efficient. Leveraging the results, we then show how DETAIL can help improve model performance in real-world scenarios through demonstration reordering and curation. Finally, we experimentally prove the wide applicability of DETAIL by showing our attribution scores obtained on white-box models are transferable to black-box models in improving model performance.

DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning

Abstract

In-context learning (ICL) allows transformer-based language models that are pretrained on general text to quickly learn a specific task with a few 'task demonstrations' without updating their parameters, significantly boosting their flexibility and generality. ICL possesses many distinct characteristics from conventional machine learning, thereby requiring new approaches to interpret this learning paradigm. Taking the viewpoint of recent works showing that transformers learn in context by formulating an internal optimizer, we propose an influence function-based attribution technique, DETAIL , that addresses the specific characteristics of ICL. We empirically verify the effectiveness of our approach for demonstration attribution while being computationally efficient. Leveraging the results, we then show how DETAIL can help improve model performance in real-world scenarios through demonstration reordering and curation. Finally, we experimentally prove the wide applicability of DETAIL by showing our attribution scores obtained on white-box models are transferable to black-box models in improving model performance.

1 Introduction

The rapid development of transformer-based language models [12, 14, 17, 71] has inspired a new in-context learning (ICL) paradigm [14], which allows a language model sufficiently pre-trained on general text to quickly adapt to specific tasks. This lightweight approach of customizing a general model for specific tasks is in contrast to fine-tuning [31, 68] that necessitates both access to the model parameters and resource-intensive step of tuning these parameters for model adaptation. In ICL, a few task demonstrations are included in the input text (i.e., prompt) together with a query to help the language model better understand how to answer the query. It has been shown that including task demonstrations in the prompt can enhance the capability of language models to apply common sense and logical reasoning [65, 72] and learn patterns from the supplied demonstrations [14], significantly enhancing the flexibility and generality of language models. In the ICL paradigm, each demonstration can be viewed as a 'training data point' for ICL. Analogous to how the performance of a conventional supervised machine learning (ML) model depends on the quality of training data, the performance of ICL depends on the quality of task demonstrations [41]. A research question naturally arises: How to attribute and interpret ICL demonstrations that are helpful or harmful for model prediction?

Though there are many prior works on interpreting and attributing model prediction for conventional ML models [25, 34, 52], these methods are not readily applicable to ICL due to its unique charac-

teristics. Firstly, many existing attribution techniques require either computing the gradients [58] or multiple queries to the model [19], both of which are slow and computationally expensive. In contrast, ICL is often applied in real-time to a large foundation model [12] that necessitates the attribution approaches for ICL to be fast and efficient. Secondly, ICL is known to be sensitive to ordering: The same set of demonstrations can result in significantly different model performance under different permutations [42, 44]. However, conventional methods do not explicitly consider the ordering of training examples. Thirdly, ICL demonstration is usually supplied as a sentence comprising a sequence of tokens, rendering conventional token-level attribution methods ineffective, as they do not capture contextual information of each ICL demonstration [6, 58]. Lastly, ICL does not update model parameters, rendering conventional techniques that analyze model parameter change [34] not applicable. Moreover, the absence of the need to update model parameters also allows a good attribution result for ICL to be transferable across different language models.

To address these challenges, we propose DETAIL , a novel technique that takes advantage of a classical attribution approach while tackling the unique characteristics of ICL. We adopt the perspective that transformers formulate an internal optimizer [3, 16, 61, 69] for ICL. Based on this internal optimizer, we design a method to understand the impact of each demonstration on the transformer's prediction. Notably, this approach allows us to leverage powerful existing analysis tools for transformer-based models in ICL, where otherwise the characteristics of ICL make applying these tools difficult.

Specifically, we describe an intuitive (re-)formulation of the influence function [34], a popular attribution method for conventional ML, on the internal optimizer and show that DETAIL addresses the challenges of computational cost, sensitivity to order, and attribution quality. Then, we empirically verify that our formulation can identify demonstrations helpful for model prediction and outlier demonstrations. Additionally, we apply our method to tasks with real-world implications including prompt reordering, noisy demonstration detection, and demonstration curation to show its effectiveness. We demonstrate that DETAIL achieves improved performance when applied to typical white-box large language models (LLMs). Furthermore, as many powerful LLMs are currently closed-source (thus black-box), we show that our DETAIL score obtained on a white-box LLM (e.g., Vicuna-7b [78]) exhibits transferable characteristics to the performance on a popular black-box model (ChatGPT).

2 Related Work

Understanding in-context learning. Prior works have attempted to understand the ICL capability of transformer-based language models [2, 3, 7, 10, 14, 20, 24, 37, 49, 51, 61, 69, 74]. [14] empirically demonstrated that language models can act as few-shot learners for unseen NLP tasks. [49] explained ICL by viewing attention heads as implementing simple algorithms for token sequence completion. [51, 69] studied the ICL capability of transformers by casting it as an implicit Bayesian inference. [3, 10, 24] further showed that transformers can learn (regularized) linear and discrete functions in context. [7] showed that transformers can adaptively implement appropriate algorithms for ICL. [37] provided a statistical generalization bound for ICL. [20, 61, 62, 74] mathematically showed that transformers with specific parameters can perform gradient descent on parameters of an internal optimizer given the demonstrations. [2, 74] further proved that the parameters (of the internal optimizer) can converge during forward passing. Inspired by the theoretical grounding, which is the focus of these works, we design our novel attribution technique for task demonstrations by adopting a similar view (i.e. transformers learn in context by implementing an optimization algorithm internally).

Data attribution. Past works have focused on explaining and attributing model performance to training data of conventional ML [8, 19, 25, 26, 34, 40, 52, 56, 70, 79]. The rise of LLMs has inspired research efforts on attribution w.r.t. prompts with a focus on task demonstrations [11, 43, 45, 54, 73], which is distinct from training data attribution since demonstrations are provided in context. Specifically, [43] used human annotators to evaluate the verifiability of attributing model answers to a prompt. [11, 73] relied on LLMs to evaluate attribution errors. These prior works are either computationally heavy (requiring additional queries of LLMs) or time-consuming (requiring human annotators). [54] proposed an interpretability toolkit for sequence generation models using gradientand perturbation-based methods. [45], a contemporary work, proposed to use a decoder module on the token embeddings for per-token attribution but requires costly training to learn the decoder weights. Moreover, these methods do not specifically target demonstration attribution. Some prior

techniques [19, 25] can be adapted for attributing ICL in LLMs but may be costly or ineffective. We empirically compare our method with those and the attribution methods consolidated in [54].

Attribution in LLMs. Past works have attempted to apply various methods including influence in attributing language models [26, 32, 35, 38, 39, 48, 63, 53, 67, 68, 75]. [48] considered a simplification of the influence function for task demonstration curation. [26, 35, 68] applied influence to pre-training and fine-tuning data of LLMs. [75] used influence to select demonstration inputs for annotation. [53] builds a classifier on the embeddings of demonstrations using a small LLM and computes influence w.r.t. the classifier for demonstration selection. In contrast, we demonstrate various use cases of our method including on-the-fly demonstration curation, reordering, and noisy demonstration detection. A contemporary work that shares technical similarity [53] focuses on demonstration selection whereas we focus on attribution and [53] is shown to be less effective than our method in Sec. 5.3. Additionally, compared to prior works leveraging influence to address specific problems, we apply influence function to provide a general attribution for demonstrations, with many applications that we empirically show.

3 Preliminaries

In-context learning (ICL). ICL is a learning paradigm that provides a few task demonstrations with formatted input and output for a pre-trained transformer (e.g., an LLM) to learn the mapping function from inputs to outputs in context (i.e., via the forward pass) [14]. Formally, a transformer model f takes in a prompt p ( S , x query ) comprising a formatted sequence of demonstrations S = ( z 1 , z 2 , . . . , z n ) where z i = { x i , y i } from a specific downstream task along with a query input x query (i.e., prompt) to predict its label as ˆ y query = f ( p ( S , x query )) . An visual for an example prompt is provided in App. C. We wish to attribute the model prediction ˆ y query to each z i ∈ set ( S ) .

ICL as implementing an internal optimizer. With the growing interest in the internal mechanism of transformers, previous works [3, 61, 62, 74] have theoretically shown that ICL can be treated as the transformer implementing an internal optimizer on the ICL demonstrations. Specifically, [61, 62, 74] formulated the objective of ICL optimizer with (regularized) mean-squared error on a linear weight applied to the token itself in linear self-attentions (LSAs) [74] or a transformation of tokens (i.e., kernelized mean-squared error) if an extra multi-layered perception is attached before the LSA [61, Proposition 2] in a recurrent transformer architecture. The transformer layers then function as performing gradient descent on the weight to minimize the objective [61, 74].

Influence function. Influence function [34] approximates the change of the loss of a test data point z test when up-weighting a training data point z i . Formally, the influence of z i on predicting z test is 1

where L ( z test , ˆ θ ) refers to the loss (function) on a test point z test of the model parameterized by ˆ θ and H ˆ θ := 1 /n ∑ n i =1 ∇ 2 θ L ( z i , ˆ θ ) is the Hessian. However, this definition cannot be directly applied to ICL since there is no model parameter change during ICL, unlike the learning settings in [8, 34]. We show how to adapt the formulation of Eq. (1) to ICL in our proposed method DETAIL next.

4 Influence Function on Internal Kernel Regression

Following the idea that transformers learn in context by implementing an internal kernelized leastsquare objective, we present our formulation of DETAIL by computing the influence function on a kernelized linear regression [28]. Specifically, we build the regression w.r.t. the following kernel

where m ( x ) ∈ R 1 × d refers to (the mapping of an ICL demonstration 2 to) an internal representation of x (e.g., hidden state of a transformer layer) with output dimension d . Let X := ( x 1 , x 2 , · · · , x n )

and Y := ( y 1 , y 2 , · · · , y n ) be the vectors of inputs and outputs in S respectively. The equivalent kernel regression can be written as ˆ Y := m ( X ) β where β ∈ R d × 1 is the weight vector over the kernelized feature space. In practice, the dimension d of m is usually much larger than the number of demonstrations, causing severe over-parameterization. Such over-parameterization renders the influence values fragile [9]. As such, we follow [9] and adopt an ℓ 2 regularization on β controlled by a hyper-parameter λ , which forms a kernelized ridge regression [47] with loss:

Taking the 2 nd derivative of Eq. (3), we obtain the hessian H β as

Adopting a matrix multiplication form for the summation in Eq. (4), we write the influence of training data on the model parameters β as follows,

where K := m ( X ) ⊤ m ( X ) ∈ R d × d is the Gram matrix and I reg ∈ R d refers to the influence of a particular demonstration ( x, y ) w.r.t. the kernel regression weights β . Then, combining Eqs. (1), (3) and (5), we can express the DETAIL score as the influence of a demonstration z on a query z test :

where β has a closed-form expression (shown in Alg. 1 in App. B). While inverting matrices in Eq. (6) requires O ( d 3 ) time, d is usually in the thousands: A typical LLM like Llama-2-7b [60] has an embedding size d = 4096 , allowing reasonable computation time of I (e.g., a few seconds). In practice, this computation is accelerated by the techniques already implemented in existing scientific computation libraries admitting sub-cubic complexity for matrix inversion.

Computing self-influence. One important application of the influence function in ML is identifying outliers via self-influence [8, 34]. The conventional definition of I trivially admits computing selfinfluence simply by replacing z test in Eq. (1) with z i . While the same approach applies to DETAIL , there are two shortcomings: (i) As the embedding is sensitive to the position, placing the same demonstration at the end of the prompt (as a query) or in the middle (as a demonstration) results in different embeddings, leading to unreasonable influence score. (ii) For each demonstration, it needs one forward pass of the model to compute the self-influence, which can be costly when the ICL dataset size is large. Instead, we implement self-influence for DETAIL by reusing the demonstration's embedding. This way, we keep the two sides of Eq. (1) consistent and only require one forward pass of the model to compute the self-influence for all demonstrations.

Further speedup via random matrix projection. While the current formulation in Eq. (6) is already computationally cheap, a relatively large embedding size (e.g. d = 4096 for Llama-2-7b) can become a bottleneck as inverting the matrix can be relatively slow. We apply an insight that for ICL, much of the information in the embedding m is redundant (we do not need a 4096 -dimensional β to fit 20 demonstrations). Hence, we project m to a much lower dimensional space via a random matrix projection while preserving the necessary information, following the Johnson-Lindenstrauss lemma [21, 33], precisely represented as a projection matrix P ∈ R d × d ′ with each entry i.i.d. from N (0 , 1 /d ′ ) . We provide a more detailed discussion in App. C. Empirically, we show that we can compress m to a much smaller dimension d ′ ≤ 1000 , resulting in a 10 × computation speedup on a typical 7B LLM on an NVIDIA L40 GPU (see Sec. 5.2).

A visualization of our proposed method is in Fig. 1 and a pseudo-code implementation is in App. B.

5 Empirical Evaluation

We evaluate the effectiveness of DETAIL (i.e., Eq. (6)) on two metrics: computational time (via logged system time required) and effectiveness in attribution (via performance metrics for tasks). We start by visualizing how the DETAIL scores, particularly I ( z test , z i ) (test influence, abbreviated as I test) and I ( z i , z i ) (self influence, abbreviated as I self) following [34, Sections 5.1 & 5.4], attribute demonstrations to a query first on a custom transformer and then on LLMs. Note that the hyperparameter λ varies under different scenarios and we discuss some heuristics for setting λ in App. C.

Tokens

x

1:1

m(x

1:1

)

Middle layer hidden state

Pick at target positions p

1:t

m

p[1]

m

test

β

m

p[i]

H

I

test

m

test

Test influence

H

β

β

m

p[i]

m(x

1:2

)

H

I

self

Self influence

β

β

m(y

1

)

m

p[i]

m(x

2:1

)

x

1:2

y

1

x

2:1

x

2:2

x

t+1:-1

m(x

2:2

)

z

1

z

2

x

test

Transformer

(partial)

m(x

t+1:-1

)

m

p[2]

m

p[t]

Random

projection

m

p[2]

m

p[t]

m

test

m

p[1]

I(2i,test) =VL(ztest,β)THVsL(2i,β)

L(x,y) = [m(x)3-y]²+入3Tβ

I(2i2i)=VL(β)THVβL(2,β)

H=

7

L(Ciyi)

i=1

Enforcing ICL behavior. We consider tasks where transformers learn from the demonstrations and form an internal optimizer. To evaluate the effectiveness of our method, we enforce the ICL behavior by mapping the labels of the demonstrations to a token that carries no semantic meaning, This way, pre-trained transformers cannot leverage memorized knowledge to produce answers but have to learn the correct answer from the supplied demonstrations. Specifically, we map all labels to one of { A,B,C,D,E } depending on the number of classes. More details are in each section.

5.1 Evaluation on a Custom Transformer

We use the MNIST dataset [22] to visualize how I test can be applied to attribute model prediction to each demonstration. We design a task where the transformer needs to learn a mapping of the digit image to a label letter in context. Specifically, for each image of 28 × 28 pixels, we flatten the pixels and concatenate them with a mapped label to form a 785 -dimensional token vector. For simplicity of illustration, we only use images of digits in { 0 , 1 } . For each ICL dataset, we assign to each distinct digit a letter in { A,B,C,D,E } randomly. We build a recurrent transformer based on the design in [61] with 10 recurrent layers each consisting of 15 attention heads. We pre-train the transformer with random label mappings from randomly selected digits so that the transformer cannot memorize the mapping but has to infer from the demonstrations. We use the hidden state after the 1 st layer as m to compute I test. A qualitative visualization of I test's attribution and a quantitative plot showing how the test prediction varies by removing demonstrations with the highest (lowest) I test are in Fig. 2. Left shows that removing tokens (represented by the image pixels and the corresponding label) with the largest I test makes the model make wrong predictions, whereas removing tokens with the lowest I test can retain the correct model predictions. Right shows that removing tokens with the lowest I test results in a slower decrease in prediction accuracy than removing the highest I test, demonstrating that tokens with the highest I test's are more helpful and vice versa .

5.2 Evaluation on Large Language Models

With the insight obtained in Sec. 5.1, we now apply DETAIL to full-fledged LLMs. We start with demonstration perturbation and noisy demonstration detection to demonstrate that DETAIL can be used to interpret the quality of a demonstration. Then, leveraging these results, we further show how we can apply the DETAIL scores to tasks more closely related to real-world scenarios.

A distinction between LLMs and the custom transformer used above is that demonstrations in LLMs are usually a sequence of tokens , whereas demonstrations in the custom transformer are single tokens representing the actual numerical values of the problems (see Fig. 2). This distinction makes it difficult to find an appropriate internal representation for each demonstration (i.e., m ). To overcome this challenge, we draw inspiration from prior works [64, 69] which suggest that information flows from input words to output labels in ICL. As an implementation detail, we take the embedding of the last token before the label token (hereafter referred to as the target position) in the middle layer where most of the information has flown to the target token positions. We include ablation studies on using different layers' embeddings in App. D.6 and using different target positions in App. D.7.

label: A

I

=-0.33

label: E

I

=-0.31

label: A

I

=-0.40

label: E

I

=-0.33

label: E

I

=-0.28

label: A

I

=-0.32

label: A

I

=-0.35

label: A

I

=-0.31

label: E

I

=-0.37

GT: E

pred: E

label: A

label: E

label: A

label: E

label: E

label: A

label: A

label: A

label: E

GT: E

pred: D

label: A

label: E

label: A

label: E

label: E

label: A

label: A

label: A

label: E

GT: E

pred: E

0

1

2

3

4

5

6

No. removed

0.55

0.60

0.65

0.70

remove high

remove low

remove random

Setup. We consider (for white-box models) mainly a Vicuna-7b v1.3 [78] and also a Llama-213b [60] on some tasks using greedy decoding to show that DETAIL works on models with different training data and of varying sizes. While our theoretical backing stands for transformer-based models, we experiment DETAIL on Mamba-2.8b [27], a state-space model architecture that has received increased attention recently in App. D.8. We primarily evaluate our method on AG News ( 4 classes) [77], SST-2 ( 2 classes) [57], Rotten Tomatoes ( 2 classes) [50], and Subj ( 2 classes) [18] datasets which all admit classification tasks. Due to space limits, some results are deferred to App. D.

Demonstration perturbation. We show that DETAIL can explain LLM predictions by showing how perturbation (i.e., corrupting the labels of some demonstrations to an incorrect class or removing some demonstrations) with the high/low I test affects the model's predictive power. We randomly pick 20 ICL datasets each comprising 20 demonstrations and 1 query from AG News and find the average and standard error of the accuracy of predicting the query after perturbation using Vicuna-7b and Llama-2-13b, shown in Fig. 3 (results for other datasets deferred to App. D.1). It can be observed that perturbing demonstrations with low I test results in a slower drop (or even improvement) in accuracy and vice versa , similar to the trend observed in Fig. 2, showcasing the applicability of DETAIL to LLMs. We additionally include the results using Falcon-7b [4] and Llama-2-7b [60] in App. D.1 where we perturb 10 demonstrations and observe a similar accuracy gap.

0

4

7

10

13

16

No. corrupted

0.1

0.2

0.3

0.4

0.5

vicuna-7b

corrupt high

corrupt low

corrupt random

0

4

7

10

13

16

No. corrupted

0.2

0.4

0.6

Llama-2-13b

corrupt high

corrupt low

corrupt random

0

4

7

10

13

16

No. removed

0.2

0.3

0.4

0.5

vicuna-7b

remove high

remove low

remove random

0

4

7

10

13

16

No. removed

0.3

0.4

0.5

0.6

0.7

Llama-2-13b

remove high

remove low

remove random

Noisy demonstration detection. We utilize the DETAIL score to detect noisy demonstrations with corrupted labels. The experiment setup largely follows [34, Section 5.4]. We randomly draw 100 ICL datasets each consisting of 20 demonstrations and 1 query. For each ICL dataset, we randomly corrupt the labels of 4 demonstrations (i.e., flipping the label to an incorrect class). The demonstrations are

then ranked in descending order of their I self. The fraction of noisy demonstrations detected is plotted in the first 3 figures of Fig. 4 (result for other datasets deferred to App. D.2). We compare our method with the leave-one-out (LOO) score [19] where the difference in cross-entropy loss of the model output is used as the utility. It can be observed that LOO performs close to random selection, whereas our method has a much higher identification rate w.r.t. the number of demonstrations checked. We also note that our method not only outperforms LOO in effectiveness but is also around 10 × faster than LOO since LOO requires multiple LLM inferences for each demonstration in the ICL dataset.

0

10

20

No. demos. checked

0.00

0.25

0.50

0.75

1.00

Fraction identified

vicuna-7b

DETAIL

LOO

random

0

10

20

No. demos. checked

0.00

0.25

0.50

0.75

1.00

Fraction identified

Llama-2-13b

DETAIL

LOO

random

AG N. SST-2

R.T.

0

2

4

Wall time (s)

3s

0.3s (10

×

)

DETAIL

LOO

4096 1000

100

10 5

1

Project dimension

0.2

0.4

0.6

0.8

0.6

0.7

Wall time

AUC

Dimension reduction via random projection. We analyze the impact of random projection on the effectiveness of DETAIL . Intuitively, dimension reduction trades off the effectiveness of DETAIL with computational efficiency, specifically the O ( d 3 ) cost for inverting K I . To understand the trade-off, we follow the same setup as the noisy demonstration detection experiment and compare the change in AUC ROC of detection and system wall time as the dimension d ′ of the projection matrix P decreases. The result for Subj is in the last figure of Fig. 4 (results for other datasets deferred to App. D.3), showing that wall time stays minimal ( ≈ 0 . 3 s) for project dimensions up to 1000 generally before it exponentially increases. Effectiveness measured in terms of AUC reaches optimal with d ′ ≥ 1000 . The results suggest a 'sweet spot' d ′ ≈ 1000 - for a low running time and high performance.

5.3 Applications of DETAIL

With the two experiments above verifying the effectiveness of DETAIL ( I self and I test) and the experiment on random projection which ensures computational efficiency, we demonstrate next how DETAIL , with I self for noisy demonstration detection and I test for demonstration perturbation, can be applied to real-world scenarios, achieving superior performance and speed.

ICL order optimization. One distinctive trait of ICL compared to conventional ML is that the order of demonstrations affects the model's predictive performance [42, 44]. We show that I self helps reorder the demonstrations with improved model predictive performance. We first show, using a Vicuna-7b model, that moving demonstrations with lower quality to the front (or back) of the prompt tends to improve the test accuracy of the model. To see this, we corrupt the label of a random demonstration and allocate this corrupted demonstration to different positions of the ICL dataset (each with 20 demonstrations with permutations drawn from a Sobol sequence [46] to capture the average performance better). A general trend with decreasing-then-increasing accuracy can be observed in Fig. 5: Allocating noisy demonstrations to the front (or the back) results in much higher test accuracy. Leveraging this insight, we utilize I self to reorder a random permutation of ICL demonstrations and show the reordered prompt improves the test accuracy. For each randomly ordered prompt, I self for each demonstration is computed (note that this computation only requires 1 pass of the LLM). Then, based on the trend observed in Fig. 5, for Subj and Rotten Tomatoes datasets, the demonstrations are reordered by placing the two demonstrations with the largest I self in front followed by the rest in ascending order. For SST-2, the demonstrations are reordered in descending order of I self. To simulate situations where demonstrations have varying quality, we additionally consider randomly perturbing 3 demonstrations (and 6 demonstrations in App. D.4) in each ICL dataset. We note a clear improvement in test accuracy of 1 . 4% ∼ 3 . 0% via reordering demonstrations only , as shown in Table 1. The improvement demonstrates that I self can identify demonstrations that are low-quality or inconsistent with other demonstrations in the ICL dataset.

0

4

8

12

16

19

Pos. of perturbed demo

0.4

0.5

0.6

0.7

Test accuracy

SST-2

ICL demonstration curation. In the demonstration perturbation experiment, we have verified that our I test can correctly attribute helpful demonstrations w.r.t. a query. A direct application is demonstration curation where a subset of most helpful demonstrations are selected to prompt the LLM while maintaining accuracy on a test dataset . 3 This application is useful, especially for saving the cost of querying LLMs. 4 For proprietary LLMs, reducing the prompt length can also significantly save inference time which scales quadratically in the prompt length. As a setup, we fix a randomly selected set of 120 demonstrations as the test set. In each trial, we randomly pick 20 demonstrations to form an ICL dataset and another 20 demonstrations as the validation set. The individual I test's on each validation demonstration are summed as the final score. Then, demonstrations with the lowest scores are removed (in position). We randomly corrupt 5 demonstrations in each ICL dataset to simulate prompts with varying qualities. The results are shown in Fig. 6 (results on other datasets deferred to App. D.5). A clear gap between the test accuracy after removing demonstrations with high/low I test can be observed for both Vicuna-7b and Llama-2-13 on both binary (Rotten Tomatoes) and 4 -way classification (AG News). Removing demonstrations with lower I test's maintains (or even improves) the test accuracy. Moreover, the gap for the 13B model is wider and more certain (shorter error bars), signaling better curation. We attribute this phenomenon to the better capability of the larger model to formulate an 'internal optimizer', which enhances the attributive power of I test .

0

4

7

10

13

16

No. removed

0.25

0.30

0.35

0.40

vicuna-7b

remove high

remove low

remove random

0

4

7

10

13

16

No. removed

0.3

0.4

0.5

Llama-2-13b

remove high

remove low

remove random

0

4

7

10

13

16

No. removed

0.45

0.50

0.55

vicuna-7b

0

4

7

10

13

16

No. removed

0.2

0.4

0.6

Llama-2-13b

5.4 Comparison with Other Attribution Methods

We compare our DETAIL score with other metrics proposed for demonstration attribution/selection or can be directly extended to attributing demonstrations. We analyze both the attributability via a demonstration curation experiment and the computational cost via recording the system wall time for performing the attribution. We select representative conventional approaches from different paradigms, including integrated gradients (IG) [58] and LIME [52] (Attention [6] in App. D.5). As these methods are originally designed for token-level attribution, we use the sum of the scores of all tokens in each demonstration as the attribution score. We also compare recent efforts on demonstration selection [15, 48, 53]. In natural language processing, a popular choice for attribution has been using BERT-based scores [23, 76] to match the similarity between texts. These methods enjoy the benefit

of fast inference since the scores are model-agnostic. However, the independence between the scores and the transformer being applied limits their interpretability and attribution accuracy. We compute the BERT score obtained on a popular sentence transformer model 5 and compare its performance against DETAIL . We select an ICL dataset of 20 demonstrations, compute the attribution scores on a validation set of 20 demonstrations, and record the accuracy after removing 10 demonstrations in place on 120 test queries. The results are tabulated in Table 2. For hyper-parameter, we choose M = 100 and k = 1 for [48], 5 iterations of LiSSA update [1] for [53], and M = 10 , K = 4 for datamodel [15]. We do not perform batch inference for a fair comparison of computational time as some approaches do not admit batch inference. We also use a projection of d ′ = 1000 to compute I test. It can be observed that DETAIL outperforms all other attribution methods in test accuracy. Our computation is efficient (with wall time of 4 ∼ 10 s), achieving over 5 × speedup compared to other methods except [53] and BERT score which achieve a comparable computational time to ours but a lower accuracy. Notably, IG and LIME perform close to random removal, which is likely because these methods are designed for token-level attribution and generalize poorly to demonstration-level attribution. Interestingly, we find that combing DETAIL and BERT score achieves state-of-the-art performance. We discuss the experiment setup in detail in App. D.5.

5.5 Transferability to Black-box Models

We evaluate the transferability of DETAIL on GPT-3.5, 6 a popular black-box model. We experiment with both the demonstration reordering and demonstration curation tasks where we compute the DETAIL scores on a Vicuna-7b model (white-box) and then test the performance on GPT. Our method produces promising results on both tasks as shown in Table 3 and Table 4 respectively. Notably, curating demonstrations with our method achieves a 17 . 9% average improvement in accuracy compared to random curating on the demonstration curation task (Table 4). We also note an over 2% improvement in accuracy for reordering task if we corrupt 3 demonstrations (Table 3). With no corrupted demonstration, reordering with our approach does not improve performance on GPT3.5, which we attribute to the stronger inference power of GPT-3.5, resulting in less variance w.r.t. demonstration orders, consistent with the findings in [44].

SST-2

No corrupted demo

Baseline (random)

0.708(7.79e-04)

0.799(7.52e-04)

0.901(6.01e-04)

Reorder (

DETAIL

)

0.711(7.51e-04)

0.792(9.01e-04)

0.909(4.84e-04)

Difference

↑

0.002(7.52e-04)

-0.007(6.49e-04)

0.008(6.14e-04)

Corrupt

3

demos

Baseline (random)

0.628(8.21e-04)

0.720(1.11e-03)

0.788(1.44e-03)

Reorder (

DETAIL

)

0.660(9.57e-04)

0.742(1.20e-03)

0.816(1.60e-03)

Difference

↑

0.032(8.61e-04)

0.022(8.92e-04)

0.028(1.10e-03)

6 Conclusion, Limitation, and Future Work

Wetackle the problem of attributing demonstrations in ICL for transformers. Based on the well-known influence function commonly used for attributing conventional ML, we propose DETAIL , an innovative adoption of the influence function to ICL through the lens of treating the transformer as implementing an internal kernelized ridge regression. Combined with a dimension reduction technique using random projection, DETAIL can be computed in real-time with an impressive performance on various real-world related tasks such as demonstration order optimization and demonstration curation. One limitation of our approach is the need to access the internal state of the transformer, which we mitigate by additionally showing that DETAIL scores are transferable to black-box models. As a first step toward attributing demonstrations w.r.t. a transformer's internal optimizer, we hope this work serves as a building block for future research to develop attribution techniques for more generalized prompting settings such as chain-of-thought [65].

Acknowledgments and Disclosure of Funding

This research is supported by the National Research Foundation Singapore and the Singapore Ministry of Digital Development and Innovation, National AI Group under the AI Visiting Professorship Programme (award number AIVP2024 -001 ). This research is supported by the National Research Foundation (NRF), Prime Minister's Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme. The Mens, Manus, and Machina (M3S) is an interdisciplinary research group (IRG) of the Singapore MIT Alliance for Research and Technology (SMART) centre.

References

Naman Agarwal, Brian Bullins, and Elad Hazan. Second-order stochastic optimization for machine learning in linear time. Journal of Machine Learning Research , 2017.

Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. In Advances in Neural Information Processing Systems , 2023.

Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models, 2023.

Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models, 2023.

Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang, Xiaodong Wang, William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. Pytorch 2: Faster machine learning through dynamic Python bytecode transformation and graph compilation. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems , 2024.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations , 2016.

Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. In Advances in Neural Information Processing Systems , 2023.

Elnaz Barshan, Marc-Etienne Brunet, and Gintare Karolina Dziugaite. Relatif: Identifying explanatory training samples via relative influence. In Proceedings of the International Conference on Artificial Intelligence and Statistics , 2020.

Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning are fragile. In Proceedings of the International Conference on Learning Representations , 2021.

Satwik Bhattamishra, Arkil Patel, Phil Blunsom, and Varun Kanade. Understanding in-context learning in transformers and llms by learning to learn discrete functions. In Proceedings of the International Conference on Learning Representations , 2024.

Bernd Bohnet, Vinh Q. Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, Tom Kwiatkowski, Ji Ma, Jianmo Ni, Lierni Sestorain Saralegui, Tal Schuster, William W. Cohen, Michael Collins, Dipanjan Das, Donald Metzler, Slav Petrov, and Kellie Webster. Attributed question answering: Evaluation and modeling for attributed large language models, 2022. URL https://arxiv.org/abs/2212.08037 .

Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models, 2022.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax .

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems , 2020.

Ting-Yun Chang and Robin Jia. Data curation alone can stabilize in-context learning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics , 2023.

Siyu Chen, Heejune Sheen, Tianhao Wang, and Zhuoran Yang. Training dynamics of multi-head softmax attention for in-context learning: Emergence, convergence, and optimality, 2024.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes,

Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.

Alexis Conneau and Douwe Kiela. Senteval: An evaluation toolkit for universal sentence representations. In Proceedings of the International Conference on Language Resources and Evaluation , 2018.

R. Dennis. Cook. Detection of influential observation in linear regression. Technometrics , 1977.

Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can GPT learn in-context? language models secretly perform gradient descent as meta-optimizers. In Findings of the Association for Computational Linguistics , 2023.

Sanjoy Dasgupta and Anupam Gupta. An elementary proof of a theorem of Johnson and Lindenstrauss. Random Structures & Algorithms , 2003.

Li Deng. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 2012.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , 2019.

Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? A case study of simple function classes, 2023.

Amirata Ghorbani and James Zou. Data Shapley: Equitable valuation of data for machine learning. In Proceedings of the International Conference on Machine Learning , 2019.

Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamil˙ e Lukoši¯ ut˙ e, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions, 2023.

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2023.

Jens Hainmueller and Chad Hazlett. Kernel regularized least squares: Reducing misspecification bias with a flexible and interpretable machine learning approach. Political analysis , 2014.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In Advances in Neural Information Processing Systems Deep Learning Workshop , 2015.

Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Conference on Empirical Methods in Natural Language Processing , 2023.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations , 2022.

Wenyang Hu, Yao Shu, Zongmin Yu, Zhaoxuan Wu, Xiangqiang Lin, Zhongxiang Dai, SeeKiong Ng, and Bryan Kian Hsiang Low. Localized zeroth-order prompt optimization. In Proceedings of the International Conference on Machine Learning , 2024.

William Johnson and Joram Lindenstrauss. Extensions of Lipschitz maps into a Hilbert space. Contemporary Mathematics , 1984.

Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the International Conference on Machine Learning , 2017.

Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. DataInf: Efficiently estimating data influence in LoRA-tuned LLMs and diffusion models. In Proceedings of the International Conference on Learning Representations , 2024.

Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing , 2021.

Yingcong Li, M. Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: generalization and stability in in-context learning. In Proceedings of the International Conference on Machine Learning , 2023.

Xiaoqiang Lin, Zhongxiang Dai, Arun Verma, See-Kiong Ng, Patrick Jaillet, and Bryan Kian Hsiang Low. Prompt optimization with human feedback. In ICML Workshop on Models of Human Feedback for AI Alignment , 2024.

Xiaoqiang Lin, Zhaoxuan Wu, Zhongxiang Dai, Wenyang Hu, Yao Shu, See-Kiong Ng, Patrick Jaillet, and Bryan Kian Hsiang Low. Use your instinct: Instruction optimization for llms using neural bandits coupled with transformers. In Proceedings of the International Conference on Machine Learning , 2024.

Xiaoqiang Lin, Xinyi Xu, Zhaoxuan Wu, See-Kiong Ng, and Bryan Kian Hsiang Low. Distributionally robust data valuation. In Proceedings of the International Conference on Machine Learning , 2024.

Hanxi Liu, Xiaokai Mao, Haocheng Xia, Jian Lou, and Jinfei Liu. Prompt valuation based on Shapley values, 2023.

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics , 2023.

Nelson F. Liu, Tianyi Zhang, and Percy Liang. Evaluating verifiability in generative search engines, 2023.

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the Annual Meeting of the Association for Computational Linguistics , 2022.

Gautam Machiraju, Alexander Derry, Arjun Desai, Neel Guha, Amir-Hossein Karimi, James Zou, Russ Altman, Christopher Ré, and Parag Mallick. Prospector heads: Generalized feature attribution for large models & data, 2024.

Rory Mitchell, Joshua Cooper, Eibe Frank, and Geoffrey Holmes. Sampling permutations for Shapley value estimation. Journal of Machine Learning Research , 2022.

Kevin P. Murphy. Machine Learning: A Probabilistic Perspective . The MIT Press, 2012.

Tai Nguyen and Eric Wong. In-context example selection with influences, 2023.

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022.

Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the Annual Meeting of the Association for Computational Linguistics , 2005.

Madhur Panwar, Kabir Ahuja, and Navin Goyal. In-context learning through the bayesian prism. In Proceedings of the International Conference on Learning Representations , 2024.

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should I trust you?": Explaining the predictions of any classifier. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2016.

Vinay M. S., Minh-Hao Van, and Xintao Wu. In-context learning demonstration selection via influence analysis, 2024.

Gabriele Sarti, Nils Feldhus, Ludwig Sickert, Oskar van der Wal, Malvina Nissim, and Arianna Bisazza. Inseq: An interpretability toolkit for sequence generation models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics , 2023.

Kuldeep Sharma, Nirmala Ramakrishnan, Alok Prakash, Lam Siew Kei, and Thambipillai Srikanthan. Evaluating the merits of ranking in structured network pruning. In 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS) , 2020.

Rachael Hwee Ling Sim, Jue Fan, Xiao Tian, Patrick Jaillet, and Bryan Kian Hsiang Low. Deletion-anticipative data selection with a limited budget. In Proceedings of the International Conference on Machine Learning , 2024.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the Conference on Empirical Methods in Natural Language Processing , 2013.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the International Conference on Machine Learning , 2017.

Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Advances in Neural Information Processing Systems , 2020.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.

Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In Proceedings of the International Conference on Machine Learning , 2023.

Johannes von Oswald, Eyvind Niklasson, Maximilian Schlegel, Seijin Kobayashi, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Blaise Agüera y Arcas, Max Vladymyrov, Razvan Pascanu, and João Sacramento. Uncovering mesa-optimization algorithms in transformers, 2023.

Jingtan Wang, Xiaoqiang Lin, Rui Qiao, Chuan-Sheng Foo, and Bryan Kian Hsiang Low. Helpful or harmful data? fine-tuning-free shapley attribution for explaining language model predictions. In Proceedings of the International Conference on Machine Learning , 2024.

Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing , 2023.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems , 2023.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-theart natural language processing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing , 2020.

Zhaoxuan Wu, Xiaoqiang Lin, Zhongxiang Dai, Wenyang Hu, Yao Shu, See-Kiong Ng, Patrick Jaillet, and Bryan Kian Hsiang Low. Prompt optimization with ease? efficient ordering-aware automated selection of exemplars. In Advances in Neural Information Processing Systems , 2024.

Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning, 2024.

Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of incontext learning as implicit Bayesian inference. In Proceedings of the International Conference on Learning Representations , 2022.

Xinyi Xu, Shuaiqi Wang, Chuan-Sheng Foo, Bryan Kian Hsiang Low, and Giulia Fanti. Data distribution valuation. In Advances in Neural Information Processing Systems , 2024.

Xinyi Xu, Zhaoxuan Wu, Rui Qiao, Arun Verma, Yao Shu, Jingtan Wang, Xinyuan Niu, Zhenfeng He, Jiangwei Chen, Zijian Zhou, Gregory Kang Ruey Lau, Hieu Dao, Lucas Agussurja, Rachael Hwee Ling Sim, Xiaoqiang Lin, Wenyang Hu, Zhongxiang Dai, Pang Wei Koh, and Bryan Kian Hsiang Low. Data-centric AI in the age of large language models. In Conference on Empirical Methods in Natural Language Processing , 2024.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems , 2023.

Xiang Yue, Boshi Wang, Kai Zhang, Ziru Chen, Yu Su, and Huan Sun. Automatic evaluation of attribution by large language models. arXiv preprint arXiv:2305.06311 , 2023.

Ruiqi Zhang, Spencer Frei, and Peter L. Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research , 2024.

Shaokun Zhang, Xiaobo Xia, Zhaoqing Wang, Ling-Hao Chen, Jiale Liu, Qingyun Wu, and Tongliang Liu. Ideal: Influence-driven selective annotations empower in-context learners in large language models. In Proceedings of the International Conference on Learning Representations , 2024.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In Proceedings of the International Conference on Learning Representations , 2020.

Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems , 2015.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems , 2023.

Zijian Zhou, Xinyi Xu, Rachael Hwee Ling Sim, Chuan Sheng Foo, and Kian Hsiang Low. Probably approximate shapley fairness with applications in machine learning. In AAAI Conference on Artificial Intelligence , 2023.

A Computational Resources

Hardware. All our experiments about 7B white-box models are conducted on a single L40 GPU. All experiments involving 13B white-box models are conducted on a single H100 GPU. Total budget for GPT-3.5 API calls to conduct the transferability experiments in Sec. 5.5 is estimated to be around US$500 (at the rate of US$0.5/1mln tokens for input and US$1.50/1mln tokens for output).

Software. All our experiments are conducted using Python3.10 on a Ubuntu 22.04.4 LTS distribution. We use Jax [13] for the experiments in Sec. 5.1 and use PyTorch 2.1.0 [5] for the experiments in Sec. 5.2 and Sec. 5.3. We adopt the implementations of the LLMs (transformer-based and SSM-based) provided in the Huggingface's 'transformers' [66] system throughout this work. The NLP datasets are also obtained from Huggingface's 'datasets' API [36]. The precise repository references and other dependencies can be found in the code provided in the supplemental materials.

B Algorithmic Implementation

We provide an implementation of computing our DETAIL score for LLM in Alg. 1. Line 6 shows the (optional) random projection where a random matrix P is multiplied by the embeddings. In line 8, β has a closed-form expression as the solution to a regularized ridge regression. In line 10 and line 14, we select the embeddings of the target positions. Then, depending on whether we use self-influence, we use different embeddings for computing ∇ β L as shown in lines 15-19. Line 20 computes the inverse hessian H -1 β = ( K I + λI ) -1 before finally calculating I i in line 21.

Algorithm 1 DETAIL

C Additional Discussion

Potential societal impact. We propose an attribution technique for improving the interpretability of in-context learning. We believe our research has potential positive societal impacts in improving the safety of LLMs via filtering out corrupted/harmful demonstrations as demonstrated by our

experiments as well as saving energy by curating the demonstration, hence reducing the cost of querying LLMs. We do not find any direct negative societal impact posed by our research contribution.

Setting λ . Generally, there is no golden rule for the most appropriate λ that regularizes the ridge parameters β . Intuitively, a larger λ likely works better when the dimension of β is large since the model tends to be over-parameterized (i.e., in a LLM). Therefore, we set a relatively large λ = 1 . 0 for LLMs and a relatively small λ = 0 . 01 for our custom transformer. When detecting noisy demonstrations, we may not want to regularize β too much because we wish to retain the information captured by the eigenvalues of the hessian H which can be eroded with a larger λ . As such, for the noisy demonstration detection task, we set a very small λ = 10 -9 to retain most of the information captured by H while ensuring that it is invertible.

Random projection matrix. We recall the Johnson-Lindenstrauss (JL) lemma [21, 33].

Theorem C.1 (Johnson-Lindenstrauss Lemma) . For any 0 < ϵ < 1 and any integer n , let d ′ be a positive integer such that

then for any set A of n points ∈ R d , there exists a mapping f : R d → R d ′ such that for all x i , x j ∈ A ,

A specific constructive proof is by setting A := 1 √ d ′ R where R i,j i.i.d. ∼ N (0 , 1) . 7 In our work, we treat each embedding m as x and the projected embedding mP as f ( x ) . The specific construction follows the abovementioned constructive proof defining

Empirically, our threshold d ′ = 1000 corresponds to ϵ ⪅ 0 . 164 , ensuring a good preservation of (Euclidean) distance between points.

ICL prompt example. We include a visualization of a prompt for ICL below. Each input-output pair consists of a task demonstration. The query is appended at the end of the prompt with only the input and the output header.

Example Prompt 1: Subj

Input: tsai may be ploughing the same furrow once too often .

Output: B

Input: equilibrium the movie , as opposed to the manifesto , is really , really stupid

.

Output: B

<More demonstrations...>

Input: a friendly vacation for four old friends - two couples from college - turns ugly . . . then

Output: A

Input: he meets god and is given all the powers of god .

Output:

Additional potential future directions. Apart from applying DETAIL to more generalized prompting settings, we think it is also an interesting direction to research whether DETAIL can provide meaningful attribution for pruned [55, 59] (or distilled [29, 30]) networks. This is especially useful

since, compared to very large models, pruned small networks can be deployed privately, admitting a straightforward application of DETAIL . Moreover, the size of the hidden states tends to be smaller, allowing for faster computation of DETAIL score.

D Additional Experiments

D.1 Additional Results for Demonstration Perturbation Task

We include the full results for the demonstration perturbation task using a Vicuna-7b v1.3 model in Fig. 7 and using a Llama-2-13b model in Fig. 8. A consistent trend can be observed across different datasets using both models.

0

4

7

10

13

16

No. corrupted

0.1

0.2

0.3

0.4

0.5

vicuna-7b

corrupt high

corrupt low

corrupt random

0

4

7

10

13

16

No. corrupted

0.2

0.4

0.6

vicuna-7b

0

4

7

10

13

16

No. corrupted

0.2

0.4

0.6

vicuna-7b

0

4

7

10

13

16

No. corrupted

0.4

0.6

0.8

vicuna-7b

0

4

7

10

13

16

No. removed

0.2

0.3

0.4

0.5

vicuna-7b

remove high

remove low

remove random

0

4

7

10

13

16

No. removed

0.4

0.5

0.6

0.7

vicuna-7b

0

4

7

10

13

16

No. removed

0.3

0.4

0.5

0.6

0.7

vicuna-7b

0

4

7

10

13

16

No. removed

0.5

0.6

0.7

0.8

vicuna-7b

0

4

7

10

13

16

No. removed

0.3

0.4

0.5

0.6

0.7

Llama-2-13b

remove high

remove low

remove random

(a) AG News Removal

0

4

7

10

13

16

No. removed

0.4

0.6

0.8

Llama-2-13b

(b) SST-2 Removal

0

4

7

10

13

16

No. removed

0.4

0.6

0.8

Llama-2-13b

moval

0

4

7

10

13

16

No. removed

0.4

0.5

0.6

0.7

Llama-2-13b

0

4

7

10

13

16

No. corrupted

0.2

0.4

0.6

Llama-2-13b

corrupt high

corrupt low

corrupt random

(e) AG News Corrupt

0

4

7

10

13

16

No. corrupted

0.2

0.4

0.6

0.8

Llama-2-13b

(f) SST-2 Corrupt

0

4

7

10

13

16

No. corrupted

0.2

0.4

0.6

0.8

Llama-2-13b

rupt

0

4

7

10

13

16

No. corrupted

0.4

0.6

Llama-2-13b

We compare (in addition to Vicuna-7b) a total of 3 LLMs: Llama-2-7b [60], Llama-2-13b [60] and Falcon-7b [4]. We reuse the same experimental setup as the demonstration label perturbation task

and compare the accuracy by removing and corrupting 10 among 20 ICL data with high/low DETAIL scores computed in different models. The results are tabulated in Table 5. A similar trend can be observed across these models where removing/corrupting demonstrations with high I test results in lower accuracy and vice versa . The results demonstrate that our method is robust against model pre-training/fine-tuning data as well as model size.

D.2 Additional Results for Noisy Demonstration Detection

We include the results on AG News, SST-2, Rotten Tomatoes, and Subj datasets using a Vicuna-7b model in Fig. 9. Similar trends as in the main text is observed. A counterpart experiment using Llama-2-13b is in Fig. 10, where a similar trend is observed.

0

10

20

No. demos. checked

0.00

0.25

0.50

0.75

1.00

Fraction identified

vicuna-7b

DETAIL

LOO

random

0

10

20

No. demos. checked

0.00

0.25

0.50

0.75

1.00

Fraction identified

vicuna-7b

DETAIL

LOO

random

0

10

20

No. demos. checked

0.00

0.25

0.50

0.75

1.00

Fraction identified

vicuna-7b

DETAIL

LOO

random

0

10

20

No. demos. checked

0.00

0.25

0.50

0.75

1.00

Fraction identified

vicuna-7b

DETAIL

LOO

random

0

10

20

No. demos. checked

0.00

0.25

0.50

0.75

1.00

Fraction identified

Llama-2-13b

DETAIL

LOO

random

0

10

20

No. demos. checked

0.00

0.25

0.50

0.75

1.00

Fraction identified

Llama-2-13b

DETAIL

LOO

random

0

10

20

No. demos. checked

0.00

0.25

0.50

0.75

1.00

Fraction identified

Llama-2-13b

DETAIL

LOO

random

0

10

20

No. demos. checked

0.00

0.25

0.50

0.75

1.00

Fraction identified

Llama-2-13b

DETAIL

LOO

random

AG N. SST-2

R.T.

0

1

2

3

4

Wall time (s)

3s

0.3s (10

×

)

DETAIL

LOO

(a)

Detecting noisy

label on AG News

(b)

Detecting label on SST-2

noisy

(c)

Detecting label

on

Tomatoes

D.3 Additional Results for Dimension Reduction

The experiments using Vicuna-7b on AG News, SST-2, Rotten Tomatoes, and Subj can be found in Fig. 11. It can be observed that the trend is consistent across different datasets.

4096 1000

100

10 5

1

Project dimension

0.4

0.6

0.8

0.55

0.60

Wall time

AUC

4096 1000

100

10 5

1

Project dimension

0.2

0.4

0.6

0.8

0.55

0.60

0.65

0.70

Wall time

AUC

4096 1000

100

10 5

1

Project dimension

0.2

0.4

0.6

0.8

0.55

0.60

0.65

0.70

Wall time

AUC

4096 1000

100

10 5

1

Project dimension

0.2

0.4

0.6

0.8

0.6

0.7

Wall time

AUC

D.4 Additional Results for Demonstration Reordering Task

We conduct additional experiments on the demonstration reordering task by perturbing 6 demonstrations in each ICL dataset of 20 demonstrations. The results are shown in Table 6. It can be observed that reordering with I self still achieves an improvement in test accuracy, demonstrating the robustness of our method.

D.5 Additional Results for Demonstration Curation Task

We include the full results for all datasets on both Vicuna-7b and Llama-2-13b in Fig. 12. It can be observed that the gap between removing demonstrations with high/low I test is wider with Llama2-13b. We believe this is because Llama-2-13b being a larger model possesses better capability of formulating the internal optimizer as compared to Vicuna-7b which is smaller.

noisy

Rotten

(d)

Detecting label on Subj

noisy

(e) Wall time compar- ison

0

4

7

10

13

16

No. removed

0.25

0.30

0.35

0.40

vicuna-7b

remove high

remove low

remove random

0

4

7

10

13

16

No. removed

0.45

0.50

0.55

0.60

vicuna-7b

0

4

7

10

13

16

No. removed

0.45

0.50

0.55

vicuna-7b

0

4

7

10

13

16

No. removed

0.55

0.60

0.65

vicuna-7b

0

4

7

10

13

16

No. removed

0.3

0.4

0.5

Llama-2-13b

remove high

remove low

remove random

0

4

7

10

13

16

No. removed

0.2

0.4

0.6

0.8

Llama-2-13b

0

4

7

10

13

16

No. removed

0.2

0.4

0.6

Llama-2-13b

0

4

7

10

13

16

No. removed

0.4

0.5

0.6

Llama-2-13b

We additionally provide the result for the demonstration curation task with Attention [6] and [53] using 10 iterations of LiSSA update for computing the hessian vector product [1] and compare it with the case with 5 iterations. The result is shown in Table 7. With 10 iterations, the wall time is much higher (over 20 × ) but accuracy is comparable to the case with 5 iterations. For Attention, the performance is comparable to random removal as shown in Table 2.

Given that BERT-score is light-weight, it is intuitive to think about whether it is possible to combine DETAIL and BERT score and reap the benefits of both worlds. We conduct a preliminary experiment in this direction. Specifically, we consider applying our formulation of DETAIL (Eq. (1)) using a hidden state modified by the BERT embedding. Specifically, we consider a weighted average of the BERT embedding and the transformer's hidden state as well as a direct concatenation of the two embeddings. The results are shown in Table 8. Surprisingly, we discover that using a weighted average of the transformer's hidden state and the BERT embedding improves the attribution accuracy.

D.6 Ablation of Different Transformer Layers for Computing DETAIL scores.

We experiment with the difference in the effectiveness DETAIL using the embeddings of different layers. We conduct experiments on demonstration removal, demonstration perturbation, and noisy label detection tasks. The results are shown in Fig. 13. It can be observed that obtaining the DETAIL scores from the later layers of the model consistently produces desirable results.

1

5

9

13 17 21 25 29

Layer number

0.4

0.6

0.8

remove high

remove low

remove random

(a)

Remove

demonstration

on

1

5

9

13 17 21 25 29

Layer number

0.2

0.4

0.6

0.8

corrupt high

corrupt low

corrupt random

(b)

Corrupt

demonstration

on

1

5

9

13 17 21 25

31

Layer number

0.55

0.60

0.65

0.70

0.75

Fraction identified

DETAIL

(c) Detecting noisy label on Subj

1

5

9

13 17 21 25 29

Layer number

0.2

0.4

0.6

0.8

remove high

remove low

remove random

(d)

Remove

demonstration

on

1

5

9

13 17 21 25 29

Layer number

0.2

0.4

0.6

0.8

corrupt high

corrupt low

corrupt random

(e) Corrupt demonstration on SST-

1

5

9

13 17 21 25

31

Layer number

0.55

0.60

0.65

0.70

0.75

Fraction identified

DETAIL

(f) Detecting noisy label on SST-2

SST-2

2

D.7 Ablation of Target Position for Computing DETAIL .

As a rule of thumb, for each demonstration, we generally want to take the embedding of its last few tokens because of the causal nature of inference and because information generally flows toward the end of the sequence [64]. We compare two possible choices of target position: the column position (immediately before the label) and the label position. We experiment on the demonstration removal task with these two choices of embeddings. The results are shown in Fig. 14. Using embeddings of both positions achieves decent task performance as reflected by the clear distinction in accuracy between removing demonstrations with high/low DETAIL scores, demonstrating that our method is robust against the choice of token embeddings. In our experiments, we adopt the column position to isolate information about the label from the embedding.

0

4

7

10

13

16

No. removed

0.4

0.5

0.6

0.7

0.8

vicuna-7b

remove high

remove low

remove random

(a) Subj using label posi-

tion

0

4

7

10

13

16

No. removed

0.5

0.6

0.7

0.8

vicuna-7b

(b) Subj using column po-

sition

0

4

7

10

13

16

No. removed

0.4

0.5

0.6

0.7

0.8

vicuna-7b

remove high

remove low

remove random

(c) SST-2 using label po-

sition

0

4

7

10

13

16

No. removed

0.4

0.5

0.6

0.7

vicuna-7b

(d) SST-2 using column

position

D.8 Experiment on State-Space Model Architecture

We consider experiments on a popular state-space model (SSM) architecture, a Mamba-2.8b model [27], which consists of 64 layers with d = 2560 . The tasks are described in the main text in detail. While we find that DETAIL can still successfully attribute Mamba on certain tasks and datasets, the performance is inconsistent. One hypothesis is that the largest currently available Mamba model (2.8B) is still significantly smaller than the 7B LLMs we conduct experiments on in the main text. A smaller model size reduces the inductive power of the model to formulate the 'internal optimizer', leading to less interpretive DETAIL scores. We would also like to note that DETAIL is not designed to work on SSMs.

Demonstration removal. The demonstration removal experiment follows the same setup as Sec. 5.2 but uses a Mamba-2.8b model instead, which is the largest model officially open-sourced. The results are shown in Fig. 15. Interestingly, removing demonstrations according to DETAIL scores still can influence predictive performance in the desirable manner where removing demonstrations with high I test leads to lower accuracy and vice versa .

0

4

7

10

13

16

No. removed

0.2

0.4

0.6

mamba

remove high

remove low

remove random

(a) AG News removal

0

4

7

10

13

16

No. removed

0.3

0.4

0.5

0.6

mamba

remove high

remove low

remove random

(b) SST-2 removal

0

4

7

10

13

16

No. removed

0.2

0.3

0.4

0.5

0.6

mamba

remove high

remove low

remove random

(c)

Rotten

Tomatoes re-

moval

0

4

7

10

13

16

No. removed

0.55

0.60

0.65

0.70

mamba

remove high

remove low

remove random

(d) Subj removal

Noisy demonstration detection. We further consider the noisy demonstration detection task as described in Sec. 5.2 on Mamba. Unfortunately, the performance is not consistent across datasets, as shown in Fig. 16: detecting demonstrations with high I self performs close to random selection on SST-2 and Rotten Tomatoes datasets, although the inference speedup is still significant. We leave the analysis of these failure cases to future work.

0

10

20

No. demos. checked

0.00

0.25

0.50

0.75

1.00

Fraction identified

mamba

DETAIL

LOO

random

(a)

Detecting

noisy

label on AG News

0

10

20

No. demos. checked

0.00

0.25

0.50

0.75

1.00

Fraction identified

mamba

DETAIL

LOO

random

(b)

Detecting

noisy

label on SST-2

0

10

20

No. demos. checked

0.00

0.25

0.50

0.75

1.00

Fraction identified

mamba

DETAIL

LOO

random

(c)

Detecting

noisy

label

on

Rotten

0

10

20

No. demos. checked

0.00

0.25

0.50

0.75

1.00

Fraction identified

mamba

DETAIL

LOO

random

(d)

Detecting

noisy

AG N. SST-2

R.T.

0

1

2

3

4

Wall time (s)

3s

0.3s (10

×

)

DETAIL

LOO

(e) Wall time compar-

Tomatoes

Figure 16: (a-d) Fraction of all noisy labels identified vs. the number of demonstrations ranked by our method (with projection down to 1000 dimension) and LOO checked respectively. (e) Wall time comparison across all datasets. λ = 10 -9 . All experiments are repeated with 10 independent trials. Lines and shades represent the mean and standard error respectively.

Demonstration curation. As DETAIL performs well using Mamba on the demonstration removal task, it is reasonable to hope that it works well on the demonstration curation task as well. As it turns out, DETAIL performs well on binary classification tasks as shown in Fig. 17 but performs poorly on AG News which is 4 -way classification. We hypothesize that this is due to Mamba's worse inductive power to formulate an internal algorithm successfully.

0

4

7

10

13

16

No. removed

0.26

0.27

0.28

0.29

mamba

remove high

remove low

remove random

0

4

7

10

13

16

No. removed

0.20

0.25

0.30

0.35

mamba

remove high

remove low

remove random

0

4

7

10

13

16

No. removed

0.25

0.30

0.35

0.40

mamba

remove high

remove low

remove random

0

4

7

10

13

16

No. removed

0.500

0.525

0.550

0.575

0.600

mamba

remove high

remove low

remove random

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We propose an attribution method for demonstrations via the viewpoint of treating the transformer as implementing an internal algorithm and demonstrate its effectiveness. Our abstract and introduction accurately reflect this claim with elaboration.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: A limitation of our work is discussed in Sec. 6.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We demonstrate the effectiveness of our method primarily by conducting experiments under different scenarios. There are theoretical insights that lead to our formulation, for which we have properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have included the specific details of our algorithmic implementation in App. B and an illustration in Fig. 1. Python code for reproducibility is also included in the supplemental materials. The datasets used in the experiments are open-source, commonly used, and well-specified.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Python code with the conda environment file is included in the supplemental materials with running instructions in the README file. Commands for reproducing experimental results are also included in a bash file. All datasets are freely downloadable with downloading code snippets written in the code.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Important experimental settings are specified in each subsection in Sec. 5.1, Sec. 5.2, and Sec. 5.3. Additional details are provided in App. D.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Average and standard errors are included in the experimental results either in shades (for figures) or in brackets (for tables).

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Computing hardware (in terms of GPU) is specified in App. A. Wall time for our approach on different tasks (and the corresponding GPU) is provided in Sec. 5.2 and Sec. 5.3.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the need for interpretability in in-context learning which our work addresses in the main text. Additional discussion is in App. C.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our contribution is mainly improving the interpretability of in-context learning. Data and models used are freely and readily available online.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The datasets used in our work are all properly cited.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DETAIL, a novel technique for attributing and interpreting in-context learning (ICL) demonstrations in transformer-based language models. The authors propose an adaptation of the influence function, typically used in conventional machine learning, to address the unique characteristics of ICL. DETAIL treats transformers as implementing an internal kernelized ridge regression, allowing for efficient and effective attribution of demonstrations. The method is evaluated on various tasks, including demonstration perturbation, noisy demonstration detection, and real-world applications such as demonstration reordering and curation. The authors demonstrate DETAIL's superiority over existing attribution methods in terms of both performance and computational efficiency.\n\nSTRENGTHS:\n1. This paper proposes a novel approach, i.e., DETAIL to address the specific challenges of ICL attribution by leveraging the internal optimizer perspective of transformers.\n2. The method incorporates random matrix projection to reduce dimensionality, resulting in significant speedups (up to 10x) while maintaining effectiveness.\n3. DETAIL is shown to be effective across multiple tasks, including noisy demonstration detection, demonstration reordering, and curation, demonstrating its broad applicability.\n4. The paper provides extensive experiments on both custom transformers and large language models (LLMs) like Vicuna-7b and Llama-2-13b, validating the method's effectiveness.\n5. The authors demonstrate that DETAIL scores computed on white-box models can transfer to black-box models like GPT-3.5, enhancing its practical applicability.\n\nWEAKNESSES:\n1. Section 5.1 presents an evaluation on a custom transformer using the MNIST dataset. While this provides an initial demonstration of DETAIL's capabilities, the paper doesn't clearly justify why this evaluation is necessary given the subsequent experiments on large language models. It's not immediately apparent how insights from this simplified setting transfer to more complex LLMs, potentially making this section feel disconnected from the main contributions of the paper.\n2. Figure 3 shows that even the Llama-2-13b model achieves only 60-70% accuracy on the AG News dataset without any perturbation. This is significantly lower than the typical performance range of 85-96% reported in the literature for this dataset. Could the authors give more insight about the performance of in context learning on the AG News dataset?\n\nQUESTIONS:\nsee the weakness section\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method to estimate the influence of ICL examples to a query. This influence estimation can better help ICL learning, for example, reorder ICL examples and curating ICL examples.\n\nSTRENGTHS:\n- The motivation is clear, and the paper overall is well-written.\n- The idea to estimate the influence of ICL examples is interesting, and can help better ICL learning for LMs.\n- Results in Table 2 seem to show that DETAIL performs better than existing influence functions (token-level) and also some recent work that estimates ICL influences.\n\nWEAKNESSES:\n- More comprehensive experiments: for applying DETAIL, the authors only showed a few tasks (all classification) and on one major public model (Vicuna-7b). To show the effectiveness of DETAIL, more comprehensive experiment results should be provided: e.g., Table 1 and 3 should use more tasks other than classification, and more models (can the authors add results on Llama-2-13b?).\n\n- DETAIL for ICL order optimization: based on the description in Section 5.3, it seems like the authors re-ordered the examples not just based on I_{self}, but also based on the trend plot in Figure 5. Does test accuracy w.r.t. perturbed position alway need to be computed first before deciding on the orders? This seems really expensive and also the choice of putting \"two demonstrations\" in the front seems very arbitrary. How would one use such a method for a random new task?\n\n- The need to use white-box models and poor transfer results: DETAIL requires white-box model access, and the transfer result in Table 3 is not significant on the more realistic setting (no corruption). Also, can the authors show the transfer results with more models, e.g., LLama-2 to GPT-3.5, and Vicuna-7b to GPT-4?\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nDiscussed in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces DETAIL, a novel influence-function based attribution technique to estimate the influence of each example in the demonstration sequence for the given target query for in-context learning (ICL). The authors empirically validate DETAIL’s effectiveness in stylized experiments and LLMs. Additionally, they demonstrate DETAIL’s applicability in real-world tasks like demonstration reordering and curation and the transferability of DETAIL’s attribution scores from white-box to black-box models.\n\nSTRENGTHS:\n1. Innovative Application of Influence Functions: The application of influence functions to interpret in-context learning (ICL) is both innovative and intriguing, offering a fresh perspective on model interpretability.\n2. Transferability to Black-Box Models: The method demonstrates promising results in transferring attribution scores from white-box models to black-box models, which significantly enhances its practical applicability.\n3. Promising Performance: The empirical performance of DETAIL is promising, showcasing its potential effectiveness in real-world scenarios\n\nWEAKNESSES:\n1. Strong Assumptions: The work is built on the assumption that transformers implement an internal optimizer. This assumption, while supported by some theoretical proof in stylized settings, may not universally hold. The proof further assumes a learn-to-learn setting for ICL, which differs from the definition provided in lines 88-105 in this paper. Additionally, Equation 3 assumes the loss function of ICL is a ridge regression function without robust theoretical guarantees.\n\n2. Lack of Analysis for Equation 6: Given the strong assumptions underlying the derived Equation 6, its reliability may be questionable. \n\n3. Sensitivity to Demonstration Ordering: Recent work has demonstrated that ICL is not sensitive to ordering with more powerful demonstration retrievers and advanced language models, contradicted  with the conclusion in  line 35.\n\n4. Limited Experimental Comparisons: The experimental validation of DETAIL in ICL demonstration curation could be strengthened. The authors should compare their method with other learning-free demonstration selection methods, such as those based on BERT embeddings.\n\n5. Computational Cost: The computational cost of DETAIL is relatively high for demonstration curation or selection. Existing methods that utilize BERT embeddings are more computationally efficient.\n\nQUESTIONS:\n1. How do you obtain $y_{test}$ for $I_{test}$?\n2. Have you explored initializing $m(x)$ and $y$ with BERT embeddings? This could be a potential area for improvement, leveraging pre-trained embeddings for better initialization.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper proposes a novel attribution method for demonstrations in ICL. The proposed method takes a perspective that the transformers learn in context by formulating an internal optimizer. The influence function is approximated as an internal kernelized ridge regression, where the representations are taken from the intermediate layers of the transformers and white-box LLMs. The paper also demonstrate that the attribution obtained from the white-box LLM exhibits transferable characteristics to black-box models. The paper also showcases a few promising applications of DETAIL.

STRENGTHS:
- By and large, the paper is written well. I especially appreciated the discussion on the relations to reinforcement learning, and how the potential functions differ from Lyapunov functions used in more classical settings.
- The idea of using the perspective that transformers formulating an internal optimize for attribution is novel and useful.
- The section on accelerating the computation with self-influence is also interesting, and highlights the authors attention to computational issues.
- The application of the DETAIL may open new research directions in LLMs for bettter instruction-tuning algorithms.
- The mathematical derivations appear to be correct.

WEAKNESSES:
- It would have been interesting to see how well this method would identify adversarial attacks for LM.
- Some additional commentary on how DETAIL can help with LLM training would have been very helpful.

QUESTIONS:
- Could you comment on whether DETAIL can be used to discover adversarial attacks of LLMs?
- Could you comment on how DETAIL can help with LLM training especially for instruction tuning?

LIMITATIONS:
Yes, the author addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-13