DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning
Authors: Zijian Zhou
Xiaoqiang Lin
Xinyi Xu
Alok Prakash
Daniela Rus
Bryan Kian Hsiang Low
Venue: NeurIPS 2024 poster (2024)
Keywords: in-context learning
data attribution
large language models
Primary area: interpretability_and_explainability
TL;DR: We propose DETAIL, an influence function-based technique to provide attribution for task demonstrations in in-context learning.

Abstract: In-context learning (ICL) allows transformer-based language models that are pre-trained on general text to quickly learn a specific task with a few "task demonstrations"  without updating their parameters, significantly boosting their flexibility and generality. ICL possesses many distinct characteristics from conventional machine learning, thereby requiring new approaches to interpret this learning paradigm. Taking the viewpoint of recent works showing that transformers learn in context by formulating an internal optimizer, we propose an influence function-based attribution technique, DETAIL, that addresses the specific characteristics of ICL. We empirically verify the effectiveness of our approach for demonstration attribution while being computationally efficient. Leveraging the results, we then show how DETAIL can help improve model performance in real-world scenarios through demonstration reordering and curation. Finally, we experimentally prove the wide applicability of DETAIL by showing our attribution scores obtained on white-box models are transferable to black-box models in improving model performance.

DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning
Zijian Zhou
Xiaoqiang Lin'
Xinyi Xu Alok Prakash
Daniela Rus Bryan Kian Hsiang Low!
Department of Computer Science, National University of Singapore, Singapore Institute for Infocomm Research, A A*STAR, Singapore
SSingapore-MIT Alliance for Research and Technology Centre, Singapore
*CSAIL, MIT, USA
{zhzij janzijiinann xiiiiinniiiiiiii wwi.aiiiiiingg. angii maangllin,xxxiyyii 1owklwwwkkeee Scomp. nus. edu. sg {zijian zhou, alok. prakash Gamaaa Csmart wwt.emt.mtt.eedieeee...mm mit.edu
rus@csail. mit. edu
Abstract
In-context learning (ICL) allows transformer- language models that are pre- trained general text to quickly learn specific task with few "task demonstra- tions" without updating their parameters, significantly boosting their flexibility and generality. ICL possesses many distinct characteristics from conventional machine learning, thereby requiring new approaches to interpret this learning paradigm. Taking the viewpoint of recent works showing that transformers learn in context by formulating an internal optimizer, we propose an influence function-based attri- bution technique, DETAIL, that addresses the specific characteristics of ICL. We empirically verify the effectiveness our approach for demonstration attribution while being computationally efficient. Leveraging the results, then show how DETAIL can help improve model performance in real-world scenarios through demonstration reordering and curation. Finally, experimentally prove the wide applicability of DETAIL by showing our attribution scores obtained on white-box models are transferable to black-box models in improving model performance.
1 Introduction
The rapid development of transformer-base language models [12, 14, 17, 71] has inspired new in-context learning (ICL) paradigm [14], which allows language model sufficiently pre-trained on general text to quickly adapt to specific tasks. This lightweight approach customizing a general model for specific tasks is in contrast to fine-tuning [31, 68] that necessitates both access to the model parameters and resource-intensing step of tuning these parameters for model adaptation. In ICI., a few task demonstrations are included in the input text (i.e., prompt) together with query to help the language model better understand how to answer the query. It has been shown that including task demonstrations in the prompt can enhance the capability of language models apply common sense and logical reasoning [65, 72] and learn patterns from the supplied demonstrations [14], significantly enhancing the flexibility and generality of language models. In the ICL paradigm, each demonstration can be viewed as a "training data point" for ICL. Analogous to how the performance of a conventional supervised machine learning (ML) model depends on the quality of training data, the performance of ICI. depends on the quality of task demonstrations [41]. A research question naturally arises: How to attribute and interpret ICL. demonstrations that are helpful or harmful for model prediction?
Though there are many prior works on interpreting and attributing model prediction for conventional ML models [25, 34, 52], these methods are not readily applicable ICL due its unique charac-
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

teristics, Firstly, many existing attribution techniques require either computing the gradients [58] or multiple queries to the model [19], both of which are slow and computationally expensive. In contrast, ICI. is often applied in real-fime to large foundation model [12] that necessitates the attribution approaches for ICL to be fast and efficient. Secondly, ICL is known to be sensitive to ordering: The same set of demonstrations can result in significantly different model performance under different permutations [42, 44). However, conventional methods do not explicitly consider the ordering of training examples. Thirdly, ICL demonstration is usually supplied
sentence compris- ing a sequence of tokens, rendering conventional token-level attribution methods ineffective, as they do not capture contextual information of each ICL demonstration [6. 58). Lastly, ICL does not update model parameters, rendering conventional techniques that analyze model parameter change [34] not applicable. Moreover, the absence of the need to update model parameters also allows good attribution result for ICI. to be transferable across different language models.
To address these challenges, we propose DETAIL, a novel technique that takes advantage of a classical attribution approach while tackling the unique characteristics of ICL. We adopt the perspective that transformers formulate an internal optimizer [3, 16, 61, 69] for ICL. Based on this internal optimizer, we design a method to understand the impact of each demonstration on the transformer's ' prediction. Notably, this approach allows us to leverage powerful existing analysis tools for transformer-basee models in ICL, where otherwise the characteristics of ICL make applying these tools difficult.
Specifically, we describe an intuitive (re- (eee)moorulation of the influence function [34], popular attri- bution method for conventional ML. on the internal optimizer and show that DETAIL addresses the challenges of computational cost, sensitivity to order, and attribution quality. Then, we empirically verify that our formulation can identify demonstrations helpful for model prediction and outlier demonstrations. Additionally, we apply our method to tasks with real-world implications including prompt reordering, noisy demonstration detection, and demonstration curation to show its effective- ness. We demonstrate that DETAIL achieves improved performance when applied to typical white-box large language models (LLMs). Furthermore, as many powerful LLMs are currently closed-source (thus black-box), we show that our DETAIL score obtained on a white-box LLM (e.g.. Vicuna-7b (78)) exhibits transferable characteristics to the performance on a popular black- -box model (ChatGPT).
2 Related Work
Understanding in-context learning. Prior works have attempted to understand the ICL capability
of transformer- based language models [2. 3. 7, 10, 14, 20, 24. 37, 49, 51, 61, 69, 74]. [14] empirically demonstrated that language models can act as few-shot learners for unseen NLP tasks. [49] explained ICL by viewing attention heads as implementing simple algorithms for token sequence completion. [51, 69] studied the ICL capability of transformers by casting it as an implicit Bayesian inference. [3, 10, 24] further showed that transformers can learn (regularized) linear and discrete functions in context. [7] showed that transformers can adaptively implement appropriate algorithms for ICL. [37] provided a statistical generalization bound for ICL. [20, 61, 62, 74] mathematically showed that transformers with specific parameters can perform gradient descent on parameters of an internal optimizer given the demonstrations. [2, 74] further proved that the parameters (of the internal optimizer) can converge during forward passing. Inspired by the theoretical grounding, which is the focus of these works, we design our novel attribution technique for task demonstrations by adopting a similar view (i.e. transformers learn in context by implementing an optimization algorithm internally).
Data attribution. Past works have focused on explaining and attributing model performance to training data of conventional ML [8, 19, 25, 26, 34, 40, 52, 56, 70, 79]. The rise of LLMs has inspired research efforts on attribution w.r.t. prompts with a focus on task demonstrations [11, 43, 45, 54, 73], which is distinct from training data attribution since demonstrations are provided in context. Specifically, [43] used human annotators to evaluate the verifiability of attributing model answers to a prompt. [11, 73] relied on LLMs to evaluate attribution errors. These prior works are either computationally heavy (requiring additional queries of LLMs) or time-consuming (requiring human annotators). [54] proposed an interpretability toolkit for sequence generation models using gradient- and perturbation-base methods. [45], contemporary work, proposed to use decoder module on the token embeddings for per-token attribution but requires costly training to learn the decoder weights. Moreover, these methods do not specifically target demonstration attribution. Some prior

---

techniques [19, 25] can be udapted for attributing ICL in LLMs but may be costly or ineffective. We empirically compare our method with those and the attribution methods consolidated in [54].
Attribution in LLMs. Past works have attempted to apply various methods including influence
in attributing language models [26, 32, 35, 38, 39, 48, 63, 53, 67, 68, 75]. [48] considered simplification of the influence function for task demonstration curation. [26. 35, 68] applied influence to pre-training and fine-tuning data of LLMs. [75] used influence to select demonstration inputs for annotation. [53] builds classifier on the embeddings of demonstrations using small LLM and computes influence w.r.t. the classifier for demonstration selection. In contrast, we demonstrate various use cases of our method including on-the-fly demonstration curation, reordering, and noisy demonstration detection. A contemporary work that shares technical similarity [53] focuses on demonstration selection whereas we focus on attribution and [53] is shown to be less effective than our method in See. 5.3, Additionally, compared to prior works leveraging influence to address specific problems, we apply influence function to provide a general attribution for demonstrations, with many applications that we empirically show.
Preliminaries
In-context learning (ICL). ICL is a learning paradigm that provides a few task demonstrations with formatted input and output for a pre-trained transformer (e.g., an LLM) to learn the mapping function from inputs outputs in context (i.e., via the forward pass) [14]. Formally, transformer model f takes in a prompt p(S, Iquery) comprising a formatted sequence of demonstrations (21, R2,,,2..... Zn) where 2 {24, from specific downstream task along with query input Ziquery (i.e., prompt) to predict its label as Jquery = J [p((S q)quer)). An visual for an example prompt is provided in App. We wish to attribute the model prediction Qquery to each 2, E set(S).
ICL as implementing an internal optimizer. With the growing interest in the internal mechanism of transformers, previous works [3, 61, 62, 74] have theoretically shown that ICL can be treated as the transformer implementing an internal optimizer on the ICL demonstrations. Specifically, [61, 62, 74] formulated the objective of ICL optimizer with (regularized) mean-squared error linear weight applied to the token itself in linear oeffaatentions (LSAs) [74] or a transformation of tokens (i.c., kernelized mean-squared error) if an extra multi-layered perception attached before the LSA [61, Proposition 2] in a recurrent transformer architecture. The transformer layers then function as performing gradient descent on the weight to minimize the objective [61, 74].
Influence function. Influence function [34] approximates the change of the loss of a test data point Zest when up- ppiieiitting training data point 24. Formally, the influence of 2 on predicting zest is!
I(2e, Zsess) V Brest 0) T Tne (2)) Vo 0) H - VoL( 0)
(1)
where (Zrest 8) refers the loss (function) on a test point zest of the model parameterized by i and Hg == 1/n = ViL(2,, 0) is the Hessian. However, this definition cannot be directly applicd to ICL since there is model parameter change during ICL, unlike the learning settings in [8, 34]. We show how to adapt the formulation of Eq. (1) to ICL in our proposed method DETAIL next.
Influence Function on Internal Kernel Regression
Following the idea that transformers learn in context by implementing internal kernelized least- square objective, we present our formulation of DETAIL by computing the influence function on a kernelized linear regression [28]. Specifically, build the regression w.r.t. the following kernel
k(æ, x²) = mxx) m(x')
where m(x) E xx refers to (the mapping of an ICL demonstration to) an internal representation of x (e.g., hidden state of transformer layer) with output dimension d. Let X (I1, T2, ..))
Following [8] and the experiment implementation in [34], we drop the negative sign our influence definition. The interpretation that higher values imply a more positive impact. For LLMs, each demonstration may consist of more than I token. We discuss how address this in Sec. 5.2.
3

---

and Y == (31. U..... ,B,) be the vectors of inputs and outputs in S respectively. The equivalent kernel regression can be written as Y == m(X) ) where / E g is the weight vector over the kernelized feature space. In practice, the dimension d of m is usually much larger than the number of demonstrations, causing severe over- veee-parameterization. Such wereperramttriization renders the influence values fragile [9]. As such, we follow [9] and adopt an & regularization on 8 controlled by A, which forms a kernelized ridge regnession [47] with loss:
(z,, v) [m(x)) y/² + ABT 3.
(3)
Taking the 2nd derivative of Eq. (3), we obtain the hessian Hg as
H3 == (1/m) E Laz, ui) (2/n) Ei (m(x;)"Tm(a) + AI)
(4)
Adopting a matrix multiplication form for the summation in Eq. (4), we write the influence of training data on the model parameters / as follows.
Ing (2)= == H3" VoL(x.y) y) n(K AI) (m(()) (m(2))8- y) AB]
(5)
where K := m(()) (X T xx) E Raxd is the Gram matrix and Zee E R2d refers to the influence of particular demonstration (z, y) w.r.t the kernel regression weights 3. Then, combining Eqs. (1), (3) and (5), we can express the DETAIL score as the influence of demonstration
query Steati
I( (Zress 2) = V (Iiees Yrea) 'Ireg(>) (2)
(6)
=n(m(iies)) (3 Ylea) AB + AT)- (m(()) y) + 1
where B has closed-form expression (shown in Alg. 1 in App. B). While inverting matrices in Eq. (6) requires (dd)) time, d is usually in the thousands: A typical LLM like Llama-2-7b [60] has an embedding size d 4096, allowing reasonable computation time of I (e.g., a few seconds). In practice, this computation is accelerated by the techniques already implemented in existing scientific computation libraries admitting sub-cubic complexity for matrix inversion.
Computing self-influence. One important application of the influence function in ML is identifying outliers via self ceff-influence [8, 34]. The conventional definition of I trivially admits computing self- influence simply by replacing Best in Eq. (1) with 2;. While the same approach applies to DETAIL, there are two shortcomings: (i) As the embedding is sensitive to the position, placing the same demonstration at the end of the prompt (as a query) or in the middle (as a demonstration) results in different embeddings, leading to unreasonable influence score. (ii) For each demonstration, it needs one forward pass of the model to compute the self-influence, which can be costly when the ICL dataset size is large. Instead, we implement self-influence for DETAIL by reusing the demonstration's embedding. This way, we keep the two sides of Eq. (1) consistent and only require one forward pass of the model to compute the self-influence for all demonstrations.
Further speedup via random matrix projection. While the current formulation in Eq. (6) is already computationally cheap, relatively large embedding size (e.g. d = 4096 for Llama-2-7b) can become a bottleneck as inverting the matrix can be relatively slow. We apply an insight that for ICL, much of the information in the embedding is redundant (we do not need 4096-dimensional 3 to fit 20 demonstrations). Hence, we project m to a much lower dimensional space via a random matrix projection while preserving the necessary information, following the Johnson-Lindeestrauus lemma [21, 331, precisely represented
projection matrix / E jadxd' with each entry i.i.d. from (O 1/d'). We provide a more detailed discussion in App. C. Empirically, we show that we can compress m a much smaller dimension d' < 1000, resulting in 10x computation speedup on typical 7B LLM on an NVIDIA L40 GPU (see Sec. 5.2).
A visualization of our proposed method is in Fig. and pseudo-code implementation is in App. B.
5 Empirical Evaluation
We evaluate the effectiveness of DETAIL (i.e., Eq. (6)) on two metrics: computational time (via logged system time required) and effectiveness in attribution (via performance metrics for tasks). We start by visualizing how the DETAIL scores, particularly I(zieu =i) (test influence, abbreviated as Ireu) and I(=;, =;) (self influence, abbreviated as Tueir) following [34, Sections 5.1 & 5.4), attribute demonstrations to a query first on a custom transformer and then on L.Mss. Note that the hyper- parameter À varies under different scenarios and we discuss some heuristics for setting À in App. C.

---

Zi Boar)
Vallirra Vallas
Laper
Randan projection
mlat
Test Influence
Transformar (partial)
I(
VaL(en 8)
H
Valiz 8)
Iu
Plan
Piee at target positions Fu.
Hg
Vi ai]
influence
Figure 1: Illustration of computing DETAIL score for transformer-base based ICL. Note that the same notation Mp[-] before and after the random projection since the projection is optional.
Enforcing ICL behavior. We consider tasks where transformers learn from the demonstrations and form an internal optimizer. To evaluate the effectiveness of our method, enforce the ICL behavior by mapping the labels of the demonstrations to token that carries no semantic meaning, This way, pre-trained transformers cannot leverage memorized knowledge produce answers but have to learn the correct answer from the supplied demonstrations. Specifically, we map all labels to one of (A, C, D, E} depending on the number of classes. More details are in each section.
5.1 Evaluation on a Custom Transformer
We the MNIST dataset [22] to visualize how Iea be applied to attribute model prediction to each demonstration. We design a task where the transformer needs to learn a mapping of the digit image a label letter in context. Specifically, for each image of 28 x 28 pixels, we flatten the pixels and concatenate them with a mapped label to form a 787-dimensional token vector. For simplicity of illustration, we only use images of digits in {0, 1]. For each ICL dataset, we assign to each distinct digit a letter in {A, B, C, D, E} randomly. We build a recurrent transformer based on the design in [61] with 10 recurrent layers each consisting of 15 attention heads. We pre-train the transformer with random label mappings from randomly selected digits so that the transformer cannot memorize the mapping but has infer from the demonstrations. We use the hidden state after the 1tt layer as m to compute Test A qualitative visualization of Tessss 's attribution and quantitative plot showing how the test prediction varies by removing demonstrations with the highest (lowest) Iest are in Fig. Left shows that removing tokens (represented by the image pixels and the corresponding label) with the largest Tee makes the model make wrong predictions, whereas removing tokens with the lowest Iee can retain the correct model predictions. Right shows that removing tokens with the lowest Zrest results in slower decrease in prediction accuracy than removing the highest Test demonstrating that tokens with the highest Tesst 's are more helpful and vice versa.
5.2 Evaluation on Large Language Models
With the insight obtained in Sec. 5.1, we now apply DETAIL to full-fledged LLMs. We start with demonstration perturbation and noisy demonstration detection to demonstrate that DETAIL can be used to interpret the quality of demonstration. Then, leveraging these results, we further show how we can apply the DETAIL scores to tasks more closely related to real-world scenarios.
A distinction between LLMs and the custom transformer used above is that demonstrations in LLMs are usually a sequence of tokens, whereas demonstrations in the custom transformer are single tokens representing the actual numerical values of the problems (see Fig. 2). This distinction makes it difficult find an appropriate internal representation for each demonstration (i.e., m). To overcome this challenge, we draw inspiration from prior works [64, 69] which suggest that information flows from input words to output labels ICL. As an implementation detail, we take the embedding of the last token before the label token (hereafter referred to the target position) in the middle layer where most of the information has flown to the target token positions. We include ablation studies on using different layers' embeddings in App. D.6 and using different target positions in App. D.7.
5

---

0-:0:33
1=-0.50
0.33
0.30.31
prod:
O
0.70
Accuracy
Jabel:
labsl:
label: label:
CT:
0.65
pred:
0.60
remove high
label.
label: E label. A. label: label. label. A label. label: A. labe).
GT:
remove low
0.55
powd:
remove random
b
0
1 2 3 4 5
6
label:
Tabell: E
label: label: E
labol:
label:
Tabel:
Tabel:
Tabel: E CT:EE
No. removed
Figure 2: (Left) Visualization of learning label mapping of MNIST digits in context. The left 9 images each row are demonstrations while the right-most one a query image. Below each image shows its mapped label ("A" to "E"). Above each ICI. image is its Teest w.r.t. the query image with high values highlighted in green and low values highlighted in red. Above the query image is the prediction (pred) made by the pre-trained transformer which is in green if consistent with the ground truth (GT) and red otherwise. Top row shows that using all demonstrations allows the transformer to learn the mapping in context "E=,red=E"". Middle shows removing 5 demonstrations with the highest Test results in most digit 0's removed, leading to a wrong prediction. Bottom shows removing 5 demonstrations with the lowest Ies results in 3 digit O's remaining for the transformer learn in context, leading to correct prediction. (Right) Average accuracy on 1013 ICL datasets repeated over 10 trials;
0.01; Lines and shades represent mean and standard error over 10 independent trials.
Setup. We consider (for white-box models) mainly a Vicuna-7b v1.3 [78] and also a Llama-2- 13b [60] on some tasks using greedy decoding to show that DETAIL works on models with different training data and of varying sizes. While our theoretical backing stands for transformer-basee models, we experiment DETAIL on Mamba-2.8h 8b [27]. state-space model architecture that has received increased attention recently in App. D.8. We primarily evaluate our method on AG News (4 classes) [77], SST-2 (2 classes) [57], Rotten Tomatoes (2 classes) [50], and Subj (2 classes) [18] datasets which all admit classification tasks. Due to space limits, some results are deferred to App. D.
Demonstration perturbation. We show that DETAIL can explain LLM predictions by showing how perturbation (i.e., corrupting the labels of some demonstrations to an incorrect class or removing some demonstrations) with the high/low Test affects the model's 's predictive power. We randomly pick 20 ICL datasets each comprising 20 demonstrations and 1 query from AG News and find the average and standard error of the accuracy of predicting the query after perturbation using Vicuna-77 7b and I. Llama-2-13b shown in Fig. 3 (results for other datasets deferred to App. D.I). It can be observed that perturbing demonstrations with low Iest results a slower drop (or even improvement) in accuracy and vice versa, similar to the trend observed in Fig. 2, showcasing the applicability of DETAIL to LLMs. We additionally include the results using Falcon-7b [4] and Llama-2-7b [60] in App. D.I where we perturb 10 demonsirations and observe a similar accuracy gap.
1Jam=2-13h
vicumm-7
1/mm::21
0.5
0.7
20.4
Ancursayy
Anmreey
0.4
0.6
0.5
0.4
corrupt high
coerupt high
n.3k
remove high remsove low
n.4
remove high remove low
0.2
corrupt low
cocrupt low
comrupt tau.dom
coerupt randoom.
0.2
remaove randoom
removve random
10 No. corrupted
16
10
13
16
10 13
10 13
16
No. corrupted
No. removed
No. removed
Figure 3: (1st and 2nd) Corrupting labels of demonstrations and (3rd and 4th) removing demonstra- tions with high/low DETAIL scores (Iust) on AG News. Perturbing demonstrations randomly result in an accuracy in the middle expected. All experiments are repeated 10 trials. X 1.0. Lines and shades represent the mean and standard error respectively.
Noisy demonstration detection. We utilize the DETAIL score to detect noisy demonstrations with comupted labels. The experiment setup largely follows [34, Section 5.4]. We randomly draw 100 ICL datasets each consisting 20 demonstrations and query. For each ICL. dataset, we randomly corrupt the labels of 4 demonstrations (i.c., flipping the label to incorrect class). The demonstrations

---

then ranked in descending order of their Iee.. The fraction of noisy demonstrations detected is plotted in the first 3 figures of Fig. (result for other datasets deferred to App. D.2). We compare our method with the leave-one-out (I.OO) score [19] where the difference in cross-entrop: loss of the model output used as the utility. It can be observed that L.O0 performs close to random selection, whereas our method has a much higher identification rate w.r.t. the number of demonstrations checked. We also note that our method not only outperforms LOO in effectiveness but is also around 10x faster than LOO since LOO requires multiple LLM inferences for each demonstration in the ICL dataset.
vicuna-7b
2amme2-1 8b
stentited
1.00
Traintonn stentiee
3A
Wa222 time (2)
0.7
DETAIL
0.50
DETAIL
0.50
DETAIL
LOO
0.4
0.25
LO0
0.25
1,00
Well time
0.00
rundum
0.00
random
0.2
AUC
1000
100
10
No. demos. checland
S5T-
Suhj
Project dimonsion
Figure 4: (Ist and 2nd) Fraction of noisy labels identified vs. number of demonstrations ranked by DETAIL (with d' 1000) and LOO checked Subj using Vicuna-7b and Llama-2-13 2-b respectively. (3rd) Wall time comparison between DETAIL and LOO on all datasets. (4th) wall time in seconds (left y-axis) and AUCROC (right y-axis) vs. projection dimension Subj using Vicuna-7b. All experiments are repeated 10 trials. = 10-9. -9, Lines and shades represent the mean and std. error.
Dimension reduction via random projection. We analyze the impact of random projection on the effectiveness of DETAIL. Intuitively, dimension reduction trades off the effectiveness of DETAIL with computational efficiency, specifically the O(d)) cost for inverting K7. To understand the trade-off, we follow the same setup as the noisy demonstration detection experiment and compare the change in AUC ROC of detection and system wall time the dimension d' of the projection matrix P decreases. The result for Subj is in the last figure of Fig. 4 (results for other datasets deferred to App. D.3), showing that wall time stays minimal (+= 0.3s) for project dimensions up to 1000 generally before exponentially increases. Effectiveness measured in terms of AUC reaches optimal with d' 2 1000. The results suggest "sweet spot" d' 1000 for low running time and high performance.
5.3 Applications of DETAIL
With the two experiments above verifying the effectiveness of DETAIL (I and Iest) and the experiment on random projection which ensures computational efficiency, we demonstrate next how DETAIL, with Iwer for noisy demonstration detection and I for demonstration perturbation, can be applied to real-world scenarios, achieving superior performance and speed.
ICL order optimization. One distinctive trait of ICL compared to conventional ML is that the order of demonstrations affects the model's predictive performance [42, 44]. We show that Iwstf helps reorder the demonstrations with improved model predictive performance. We first show, using a Vicuna-7b model, that moving demonstrations with lower quality to the front (or back) of the prompt tends to improve the test accuracy of the model. To see this, we corrupt the label of random demonstration and allocate this corrupted demonstration to different positions of the ICL. dataset (each with 20 demonstrations with permutations drawn from Sobol sequence [46] capture the average performance better). A general trend with decreasing-then -ee--thchnnniiinggggg accuracy can be observed in Fig. 5: Allocating noisy demonstrations to the front (or the back) results in much higher test accuracy. Leveraging this insight, we utilize Tear to reorder a random permutation of ICI. demonstrations and show the reordered prompt improves the test accuracy. For each randomly ordered prompt, I for each demonstration is computed (note that this computation only requires 1 pass of the LLM). Then, based on the trend observed in Fig. 5, for Subj and Rotten Tomatoes datasets, the demonstrations are reordered by placing the two demonstrations with the largest Ixelt in front followed by the rest in ascending order. For SST-2, the demonstrations are reordered in descending order of I To simulate situations where demonstrations have varying quality, we additionally consider randomly perturbing 3 demonstrations (and demonstrations in App. D.4) in each ICL dataset. We note a clear improvement in test accuracy of 1.4% ~ 3.0% via reordering demonstrations only, as shown in Table The improvement demonstrates that Iaeir can identify demonstrations that are low-quality or inconsistent with other demonstrations in the ICL dataset.
7

---

0.7
Table I: Predictive accuracy of demonstrations permuted randomly and based on Zeelf, respectively. The mean and standard error (in bracket) with 80 repeated trials is shown.
Ratten Tomatoes
Test accuracy
0.6
Subj
SST-2
0.5
Subj
SST-2
Rotten Tomatocs
0.4
No corrupted demo
12 16 19
Baseline (random)
0.722 (7.22e-03)
0.665 (5.24c-03)
0.660 (1.08c002)
Pos: of perturbed demo
Reorder (DETAIL)
0.743 (7.10c-03)
0.679 (5.42c-03)
0.684 (1-15)-02)
Difference 1
0.0206 (7.40c-03)
0,0139 (6.08e-03)
0.0244 (1.11c-02)
Figure 5: Test accuracy (mean and std. error) vs. position of the demonstration with the cor- rupted label over 50 trials.
Corrupt demos
Baseline (random)
0.655 (8.54c-03)
0.607 (7.61c-03)
0.553 (1.10-002
Reorder (DETAIL)
0.685 (9.39c-03)
0,630 (7.04e-03)
0,582 (1.42e-02)
Difference 1
0.0300 (9.10c-03) 0.0230 (7.22e-03)
0.0291 (1.06e-02)
ICL demonstration curation. In the demonstration perturbation experiment, we have verified that our Zest can correctly attribute helpful demonstrations w.r.t. query. A direct application is demonstration curation where a subset of most helpful demonstrations are selected to prompt the LLM while maintaining accuracy a test dataset." 3 This application is useful, especially for saving the cost of querying LLMs.* For proprietary LLMs, reducing the prompt length can also significantly save inference time which scales quadratically in the prompt length. As a setup, we fix randomly selected set of 120 demonstrations as the test set. In each trial, we randomly pick 20 demonstrations to form an ICL dataset and another 20 demonstrations as the validation set. The individual Tres's 's on each validation demonstration are summed as the final score. Then, demonstrations with the lowest scores are removed (in position). We randomly corrupt 5 demonstrations in each ICL dataset to simulate prompts with varying qualities. The results are shown in Fig. 6 (results on other datasets deferred App. D.5). A clear gap between the test accuracy after removing demonstrations with high/low Zee can be observed for both Vicuna-7b and Llama-2-13 on both binary (Rotten Tomatoes) and 4-way classification (AG News). Removing demonstrations with lower Twea maintains (or even improves) the test accuracy. Moreover, the gap for the 13B model is wider and more certain (shorter error bars), signaling better curation. We attribute this phenomenon the better capability of the larger model to formulate an "internal optimizer", which enhances the attributive power of Lestt-
Viruna-/b
Llama-2-133
Llama 2:13b
0.40
0.55
sccureop
0.35
Accuncy
Accuracy
0.50
0.0.
0.30
remsove high
remove high
0.4
0.45-
0.25
10
13
16
10
16
10
16
No. comoved
removed
remooved
No. ressoved
Figure 6: Test accuracy vs. number of demonstrations removed using Teest on (Ist and 2nd) AG news and (3rd and 4th) Rotten Tomatoes using Vicuna-7b and Llama-2- All experiments are repeated with 80 trials. Lines and bars represent the mean and standard error.
5.4 Comparison with Other Attribution Methods
We compare our DETAIL score with other metrics proposed for demonstration attributionn/ellcciion or can be directly extended to attributing demonstrations. We analyze both the attributability via a demonstration curation experiment and the computational cost via recording the system wall time for performing the attribution. We select representative conventional approaches from different paradigms, including integrated gradients (IG) [58] and LIME [52] (Attention [6] in App. D.5). As these methods are originally designed for leven-level attribution, we use the sum of the scores of all tokens each demonstration as the attribution score. We also compare recent efforts on demonstration selection [15, 48, 53]. In natural language processing, popular choice for attribution has been using BERT-based scores [23, 76] to match the similarity between texts. These methods enjoy the benefit
3 Note that a key difference between demonstration curation task and demonstration perturbation task that for curation, the test dataset is unknown when computing the DETAIL scores.
At the time of this writing, GPT-4 API costs $10/1mla tokens. See https: // /oppaai.eaaa.//////////... com/ com/pricing.
8

---

of fast inference since the scores are model-agnostic. However, the independence between the scores and the transformer being applied limits their interpretability and attribution accuracy. We compute the BERT score obtained on a popular sentence transformer model 5 and compare its performance against DETAIL, We select an ICI. dataset of 20 demonstrations, compute the attribution scores on a validation set of 20 demonstrations, and record the accuracy after removing 10 demonstrations in place on 120 test queries. The results are tabulated in Table 2. For nyper-parametes we choose M = 100 and k = 1 for [48], 5 iterations of LiSSA update [1] for [53], and M 10, K = 4 for datamodel [15]. We do not perform batch inference for a fair comparison of computational time as some approaches do not admit batch inference. We also use projection of d' 1000 to compute Iea. It can be observed that DETAIL outperforms all other attribution methods in test accuracy. Our computation is efficient (with wall time of 4 10s), achieving over 5x speedup compared to other methods except [53] and BERT score which achieve a comparable computational time to ours but a lower accuracy. Notably, IG and LIME perform close to random removal, which is likely because these methods are designed for token-level attribution and generalize poorly to demonstration-level attribution. Interestingly, we find that combing DETAIL and BERT score achieves state-of-the-ar performance. We discuss the experiment setup in detail in App. D.5.
Table 2: Test accuracy after curating the ICL dataset and the incurred wall time (in seconds on one L40 GPU). The mean and std. error (in bracket) is shown with 20 repeated trials.
Metric
DSTAIL 10000
IG
LIME
(48)
Datamodel [15)
BLKI Soore (16)
Random
Subg
Accuracy
(L747 (2.00-02)
0.688 (2
0.05 12.4e (2)
0,883
(2.75e 02)
0,556 (L.Me-2)
0.658 (26
0.671 (2-4e-10))
(2
time
5.22 (1.17e-1)
(1 (1.20e+01)
54.33,
9.37 04 196-01)
(3,424000
1.97 (3256-023)
SST-2
Accunacy
BU607 (2. (2)
0.458 (2 02)
04466 (1 (17
0,513
(1.58s 02)
0.493 (1.54--22
0u477 (2.5 (2.51e-02)
0.460 (2 36-
B.469 (2. (2.155)02)
Wall line
4.88 033
458 (7.9% (00)
337 (3))
121 (4.79e000)
10.6 (7.80-01)
713 (1.62400)
2.1 (2.78 (12)
N.A.
Rotten Tomalocs
Accuracy
0.555 (1 11..44e-00 92))
0.442 (2. (2.133-00)
0.43 (1 39. (12)
0.520 (2.17 02)
0.498 (1.7k))-76e))) 02)
0u484 (1 (1877-02)
0.435 (1 1.336- (2)
IL457 (2.19)
Wall tine
5.11 (1.86e 01)
525 (1.23e+01)
245 (3)
122 (4.65e+00)
9,74 (5.57e 01)
732 (2.00e+00)
2,95 (4.43- (23)
N.A.
AG News
Ansinayy
EL412 1L412 (1.
0 0.351 (1 (55-400)
0 1688// (1 7 Se-0t
0. 0.392 432-022)
0.361 (1 (33-02)
0 (1 3io40;
0 955
(1 Sko-tD)
0.979 (1 70-40))
Wall fime
10.4 (1 ()7o-01)
1208 (2. (ice01))
909 (1 (100+001)
81.3 (6.00-01))-0))
6.94 (4.79x-82)
997 (7. (7.552400)
1.55 (8.87-412)
N.A.
5.5 Transferability to Black-box Models
We evaluate the transferability of DETAIL on GPT-3.5, popular black-box model. We experiment with both the demonstration reordering and demonstration curation tasks where we compute the DETAIL scores on Vicuna-7b model (white-box) and then test the performance method produces promising results on both tasks as shown in Table 3 and Table 4 respectively. Notably, curating demonstrations with our method achieves a 17.9% average improvement in accuracy compared to random curating on the demonstration curation task (Table 4). We also note an over 2% improvement in accuracy for reordering task if we corrupt 3 demonstrations (Table 3). With no corrupted demonstration, reordering with our approach does not improve performance on GPT- 3.5, which we attribute to the stronger inference power of GPT-3.5, resulting in less variance w.r.t. demonstration orders, consistent with the findings in [44].
GPT. Our
Table 3: Accuracy (on GPT-3.5) of demonstrations Table 4: Accuracy (on GPT-3.5) on test (demos) permuted randomly and based on Teeti Mean dataset of size 20 after curating 10 demonstra- and std. error (in bracket) with 80 trials is shown.
tions from the ICL dataset. The mean and std. error (in bracket) of accuracy after removal is shown with 20 repeated trials.
Subj
SST-2
Rotion Tomatoos
corrupted demo
Baseline (random)
0.708/7.79e-00 0.711(7.5ee-04
8.799/7.52e004
0.901(6.01e-04)
Reorder (DETATI))
0.792/9.01-044
0.919X4 84c-
Dataset
DETAIL 1000)
Random
Difference 1
0.042)7.52e-84)
-0.007(6.49e 04)
0.00866.14e-04
Suhj
0.842 16e-122
0.660 (3.47e-02)
Corrupt 3 demos
Baseline (randoom)
0.68888 21e-44)
0.72XX 1e-033
0.788(1 (4e-13)))
SST-2
0,812 (1. 96e-))
0.618 05)51e-02)
Reorder (DSTAIL)
666001955 57c-01)
0.742(1 20x-03)
(.06611600-03)
Rotten Tomatoes
0.690 (4.66e-1)2)
0.420 (5. 14c-02)
Dilference 7
0.032(8.61e-04)
8.023/8.92e-044
(.0028(.1000033)
AG News
0.515 (3.08e-02)
0.447 02)73e-02
hat.uggiingacce.ccseetteeeeeee co/sentence- orananoooeeerr//eeaerrorrbbbbbbe
'We use g-t-3.--turbo- 1106. See https: // http::////a/mm.//eee/ee openai. aweenai.com/ooceeec httm///doo/ooooo/ptt--eooo hgtps///tttttteee turbo.

---

6 Conclusion, Limitation, and Future Work
We tackle the problem of attributing demonstrations in ICL for transformers. Based on the well-known influence function commonly used for attributing conventional ML, we propose DETAIL, an innovative adoption of the influence function to ICL through the lens of treating the transformer as implementing an internal kernelized ridge regression. Combined with a dimension reduction technique using random projection, DETAIL can be computed in real-time with an impressive performance on various real-world related tasks such as demonstration order optimization and demonstration curation. One limitation of our approach is the need to access the internal state of the transformer, which we mitigate by additionally showing that DETAIL scores are transferable to black-box models. As a first step toward attributing demonstrations w.r.t. a transformer's ' internal optimizer, we hope this work serves building block for future research to develop attribution techniques for more generalized prompting settings such as chain-of-thoughh [65].
Acknowledgments and Disclosure of Funding
This research is supported by the National Research Foundation Singapore and the Singapore Ministry of Digital Development and Innovation, National AI Group under the AI Visiting Professorship Programme (award number AIVP-2021-001). This research is supported by the National Research Foundation (NRF), Prime Minister's Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme. The Mens, Manus, and Machina (M3S) an interdisciplinary research group (IRG) of the Singapore MIT Alliance for Research and Technology (SMART) centre.
References
[1] Naman Agarwal, Brian Bullins, and Elad Hazan. Second-order stochastic optimization for machine learning in linear time. Journal of Machine Learning Research, 2017.
[2] Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. In Advances in Neural Information Processing Systems, 2023.
[3] Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models, 2023.
[4] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malar- tic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models, 2023.
[5] Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesen- sky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary Vetito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Fikan Wang, Xiaodong Wang, William Wen, Shunting Zhang, Xu Zhao Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. Pytorch 2: Faster machine learning through dynamic Python bytecode transformation and graph compi- lation. In ACM International Conference on Architectural Support P Programming Languages and Operating Systems, 2024.
[6] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations, 2016.
[7] Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. In Advances in Neural Information Processing Systems, 2023.
10

---

[8] Elnaz Barshan, Mare-Etienne Brunet, and Gintare Karolina Dziugaite, Relatif: Identifying ex- planatory training samples via relative influence. In Proceedings of the International Conference on Artificial Intelligence and Statistics, 2020.
[9] Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning are fragile. In Proceedings of the International Conference on Learning Representations, 2021.
[10] Satwik Bhattamishra, Arkil Patel, Phil Blunsom, and Varun Kanade. Understanding in-context learning in transformers and Ilms by learning to learn discrete functions. In Proceedings of the International Conference on Learning Representations, 2024.
[11] Bernd Bohnet, Vinh Q. Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, Tom Kwiatkowski, Ji Ma, Jianmo Ni, Lierni Sestorain Saralegui, Tal Schuster, William W. Cohen, Michael Collins, Dipanjan Das, Donald Metzler, Slav Petrov, and Kellie Webster. Attributed question answering: Evaluation and modeling for attributed large language models, 2022. URL https://arxiv. org/ abs 2212. 08037.
[12] Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang. Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri. Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovee, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas. Florian Tramèr, Rose E. Wang, William Wang. Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models, 2022.
[13] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Handerman-Milne and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://gitte cwm/ coon//ooole/jar.
[14] Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
[15] Ting- Yun Chang and Robin Jia. Data curation alone can stabilize in-context learning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2023.
[16] Siyu Chen, Heejune Sheen, Tianhao Wang, and Zhuoran Yang. Training dynamics of head softmax attention for in-context learning: Emergence, convergence, and optimality, 2024.
[17] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsyyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes,

---

Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury. Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin. Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Helllseri Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
[18] Alexis Conneau and Douwe Kiela. Senteval: An evaluation toolkit for universal sentence representations. In Proceedings of the International Conference on Language Resources and Evaluation, 2018.
[19] R. Dennis. Cook. Detection of influential observation in linear regression. Technometrics, 1977.
[20] Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can GPT learn in-context? language models secretly perform gradient descent as meta-optimiizess In Findings of the Association for Computational Linguistics, 2023.
[21] Sanjoy Dasgupta and Anupam Gupta. An elementary proof of a theorem of Johnson and Lindenstrauss. Random Structures & Algorithms, 2003.
[22] Li Deng. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 2012.
[23] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), 2019.
[24] Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? A case study of simple function classes, 2023.
[25] Amirata Ghorbani and James Zou. Data Shapley: Equitable valuation of data for machine learning. In Proceedings of the International Conference on Machine Learning, 2019.
[26] Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilè Lukošiūte, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions, 2023.
[27] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2023.
[28] Jens Hainmueller and Chad Hazlett, Kernel regularized least squares: Reducing misspecificatiion bias with a flexible and interpretable machine learning approach. Political analysis, 2014.
[29] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In Advances in Neural Information Processing Systems Deep Learning Workshop, 2015.
[30] Cheng-Yu Yu Hsich, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen- Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Conference on Empirical Methods in Natural Language Processing. 2023.
[31] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations, 2022.
[32] Wenyang Hu, Yao Shu, Zongmin Yu, Zhaoxuan Wu, Xiangqiang Lin, Zhongxiang Dai, See- Kiong Ng, and Bryan Kian Hsiang Low. Localized zeroth-order prompt optimization. In Proceedings of the International Conference on Machine Learning, 2024.
12

---

[33] William Johnson and Joram Lindenstrauss. Extensions of Lipschitz maps into Hilbert space. Contemporary Mathematics, 1984.
[34] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the International Conference on Machine Learning, 2017.
[35] Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datalnf: Efficiently estimating data influence in LoRA-tuned LLMs and diffusion models. In Proceedings of the International Conference on Learning Representations, 2024.
[36]
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: community library for natural language processing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2021.
[37] Yingcong Li, M. Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as al- gorithms: generalization and stability in in-context learning. In Proceedings of the International Conference on Machine Learning, 2023.
[38] Xiaoqiang Lin, Zhongxiang Dai, Arun Verma, Sce-Kiong Ng, Patrick Jaillet, and Bryan Kian Hsiang Low. Prompt optimization with human feedback. In ICML Workshop Models of Human Feedback for Al Alignment, 2024.
[39] Xiaoqiang Lin, Zhaoxuan Wu, Zhongxiang Dai, Wenyang Hu, Yao Shu, See-Kiong Ng. Patrick Jaillet, and Bryan Kian Hsiang Low. Use your instinct: Instruction optimization for I1ms using neural bandits coupled with transformers. In Proceedings of the International Conference on Machine Learning, 2024.
[40] Xiaoqiang Lin, Xinyi Xu, Zhaoxuan Wu, See-Kiong Ng, and Bryan Kian Hsiang Low. Distri- butionally robust data valuation. In Proceedings of the International Conference on Machine Learning, 2024.
[41] Hanxi Liu, Xiaokai Mao, Haocheng Xia, Jian Lou, and Jinfei Liu. Prompt valuation based on Shapley values, 2023.
[42] Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 2023.
[43] Nelson F. Liu, Tianyi Zhang, and Percy Liang. Evaluating verifiability in generative search engines, 2023.
[44] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2022.
[45] Gautam Machiraju, Alexander Derry, Arjun Desai, Neel Guha, Amir-Hossein Karimi, James Zou, Russ Altman, Christopher Ré, and Parag Mallick. Prospector heads: Generalized feature attribution for large models & data, 2024.
[46] Rory Mitchell, Joshua Cooper, Eibe Frank, and Geoffrey Holmes. Sampling permutations for Shapley value estimation. Journal of Machine Learning Research, 2022.
[47] Kevin P. Murphy. Machine Learning: A Probabilistic Perspective. The MIT Press, 2012.
[48] Tai Nguyen and Eric Wong. In-context example selection with influences, 2023.
13

---

[49] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Harfield-Dodd, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022.
[50] Bo Pang and Lillian Lee. Sceing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2005.
[51] Madhur Panwar, Kabir Ahuja, and Navin Goyal. In-context learning through the bayesian prism. In Proceedings of the International Conference on Learning Representations, 2024.
[52] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should I trust you?": Explaining the predictions of any classifier. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
[53] Vinay M. S., Minh-Hao Van, and Xintao Wu. In-context learning demonstration selection via influence analysis, 2024.
[54] Gabriele Sarti, Nils Feldhus, Ludwig Sickert, Oskar der Wal, Malvina Nissim, and Arianna Bisazza. Inseq: An interpretability toolkit for sequence generation models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2023.
[55] Kuldeep Sharma, Nirmala Ramakrishnan, Alok Prakash, Lam Siew Kei, and Thambipillai Srikanthan. Evaluating the merits of ranking in structured network pruning. In 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS), 2020.
[56) Rachael Hwee Ling Sim, Jue Fan, Xiao Tian, Patrick Jaillet, and Bryan Kian Hsiang Low. Deletion-anticipative data selection with a limited budget. In Proceedings of the International Conference on Machine Learning, 2024.
[57] Richard Socher, Alex Perelygin. Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the Conference Empirical Methods in Natural Language Processing, 2013.
[58] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the International Conference on Machine Learning, 2017.
[59] Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Advances in Neural Information Processing Systems, 2020.
[60] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marein Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarow, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[61] Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In Proceedings of the International Conference Machine Learning, 2023.
14

---

[62] Johannes von Oswald, Eyvind Niklasson, Maximilian Schlegel, Seijin Kobayashi, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Blaise Agüera y Arcas, Max Vladymy- rov, Razvan Pascanu, and João Sacramento. Uncovering mesa-optimization algorithms in transformers, 2023.
[63] Jingtan Wang, Xiaoqiang Lin, Rui Qiao, Chuan- Saann--hnng& Foo, and Bryan Kian Hsiang Low. Helpful or harmful data? fine-tuning-free free shapley attribution for explaining language model predictions. In Proceedings of the International Conference on Machine Learning, 2024.
[64] Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2023,
[65] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2023.
[66] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the art natural language processing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2020.
[67] Zhaoxuan Wu, Xiaoqiang Lin, Zhongxiang Dai, Wenyang Hu, Yao Shu, See-Kiong Ng, Patrick Jaillet, and Bryan Kian Hsiang Low. Prompt optimization with ease? efficient ordering-awe aware automated selection of exemplars. In Advances in Neural Information Processing Systems, 2024.
[68] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning, 2024.
[69] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in- context learning as implicit Bayesian inference. In Proceedings of the International Conference on Learning Representations, 2022.
[70] Xinyi Xu, Shuaiqi Wang, Chuan-Sheng Foo, Bryan Kian Hsiang Low, and Giulia Fanti Data distribution valuation. In Advances in Neural Information Processing Systems, 2024.
[71] Xinyi Xu, Zhaoxuan Wu, Rui Qiao, Arun Verma, Yao Shu, Jingtan Wang. Xinyuan Niu, Zhenfeng He, Jiangwei Chen, Zijian Zhou, Gregory Kang Ruey Lau, Hieu Dao, Lucas Agussurja, Rachael Hwee Ling Sim, Xiaoqiang Lin, Wenyang Hu, Zhongxiang Dai, Pang Wei Koh, and Bryan Kian Hsiang Low. Data-centric Al in the age of large language models. In Conference on Empirical Methods in Natural Language Processing, 2024.
[72] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, 2023.
[73] Xiang Yue, Boshi Wang, Kai Zhang, Ziru Chen, Yu Su, and Huan Sun. Automatic evaluation of attribution by large language models. arXiv preprint arXiv:2305.06311 2023.
[74] Ruiqi Zhang, Spencer Frei, and Peter L. Bartlett, Trained transformers learn linear models in-context. Journal of Machine Learning Research, 2024.
[75] Shaokun Zhang, Xiaobo Xia, Zhaoqing Wang, Ling-Hao Chen, Jiale Liu, Qingyun Wu, and Tongliang Liu. Ideal: Influence-driven selective annotations empower in-context learners in large language models. In Proceedings of the International Conference on Learning Representations. 2024.
[76] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In Proceedings of the International Conference on Learning Representations, 2020.
15

---

[77] Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, 2015.
[78] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging LLM-as auge with MT-benos and chatbot arena. Advances in Neural Information Processing Systems, 2023.
[79] Zijian Zhou, Xinyi Xu, Rachael Hwee Ling Sim, Chuan Sheng Foo, and Kian Hsiang Low. Prob- ably approximate shapley fairness with applications in machine learning. In AAAI Conference on Artificial Intelligence, 2023.
16

---

A Computational Resources
Hardware. All our experiments about 7B white-box models are conducted on single L440 GPU. All experiments involving 13B white-box models are conducted on a single H100 GPU. Total budget for GPT-3.5 API calls to conduct the transferability experiments in Sec. 5.5 is estimated to be around US$500 (at the rate of USSO.5/1mi min tokens for input and US$1.50/Imll tokens for output).
Software. All our experiments are conducted using Python3. 10 on a Ubuntu 22.04.4 LTS distribu- tion. We use Jax [13] for the experiments in Sec. 5.1 and use PyTorch 2.1.0 [5] for the experiments in Sec. 5.2 and Sec. 5.3. We adopt the implementations of the LLMs (transformer-
and (SM-based) provided in the Huggingface's "transformers" [66] system throughout this work. The NLP datasets are also obtained from Huggingface's ' "datasets" API [36]. The precise repository references and other dependencies can be found in the code provided in the supplemental materials.
B Algorithmic Implementation
We provide an implementation of computing our DETAIL score for LLM in Alg. 1. Line 6 shows the (optional) random projection where random matrix multiplied by the embeddings. has a closed-form expression as the solution to regularized ridge regression. In line 10 and line 14, we select the embeddings of the target positions. Then, depending on whether we use self-influence, we use different embeddings for computing V J. as shown in lines 15-19, Line 20 computes the inverse hessian H3* (Kz + X/) before finally calculating Z; in line 21.
line
Algorithm DETAIL
Input: model M, prompt tokens [[].]] label tokens V[1:e]. target positions P) total number of transformer layers L, transformer layer to compute DETAIL score regularization constant X,
projection matrix P ddxd' (default /)
2: I t for E
3: h1, h2,
hh x- M (Tpp:))
4: Pdemo - P[1:t-1]
Remove the last target which is the test query Remove the last label and convert one-hot Optional dimensionality reduction Kg (t((-))x(t1) for speed-up
5: Ydemo + one onn__ho((y1:0-1))
6: Mdemo tu[Pdemo)
7: Kg 3 + Medemo
d'
8: B [Kgg -II) "Mdemo]"
Yáemo
9: Prest + P(t-1:t)
10: Mass + h4 [Prea]
thest + one -on_hootyyee hot(H]t
12: KI +- m.demo Midemo
D.
Kr E TRdd' xd
for i (1,2,
1) do m: + hu hu[ppestu]]
14:
if self influence then
16:
V + m. (m,3 - Mdeemo [i]) + A
else
18:
L +- Miesi Meesa(M (Mest Vrea) + X3
end if
20:
Ireg - (Kr + A7) (m, (m,3 [1mmmmmmmB-.. ydeemo [i]) + X33
Eq.. (5) with the constant dropped
21:
Ij + I, + (V) I
& Eq. (6)
22:
end for
23: Return I
C Additional Discussion
Potential societal impact. We propose attribution technique for improving the interpretability of in-context learning. We believe our research has potential positive societal impacts in improving the safety of LLMs via filtering out corrupted/harfful demonstrations as demonstrated by our
17

---

experiments as well as saving energy by curating the demonstration, hence reducing the cost of querying LLMs. We do find any direct negative societal impact posed by our research contribution.
Setting X. Generally, there is no golden rule for the most appropriate À that regularizes the ridge parameters B. Intuitively, larger À likely works better when the dimension of [ is large since the model tends to be pere-paraettrizeed (i.e., in LLM). Therefore, we set relatively large 1.0 for LLMs and relatively small 0.01 for our custom transformer. When detecting noisy demonstrations, we may not want to regularize 3 too much because we wish to retain the information captured by the eigenvalues the hessian which can be eroded with larger As such, for the noisy demonstration detection task, we set very small À = 10-9 to retain most of the information captured by И while ensuring that is invertible.
Random projection matrix. We recall the Johnson-Linndenstruus (JL) lemma [3]. Theorem C.1 (Johnson-Lindenssraus Lemma), For any 0 < e < 1 and any integer n, let d' be positive integer such that
24
d' 2 362
log n1
2c3
then for any set n points E R2, there exists a mapping f Rd - Rdd such that for all 24, E A,
(1 - [|] =, < (f(x))) - s(x;)) s (1 + e e x,|1?
specific constructive proof by setting A R where Rs iid NO 1).? our work, we treat each embedding and the projected embedding mP as fxx). The specific construction follows the abovementioned constructive proof defining
V
1 R (0,
1 ह.
P =
Empirically, our threshold d' = 1000 corresponds to 2 0.164, ensuring good preservation of (Fuclidean) distance between points.
ICL prompt example. We include a visualization of prompt for ICL below. Each input-output pair consists of task demonstration. The query appended at the end of the prompt with only the input and the output header.
Example Prompt 1: Subj
Input: tsai may be ploughing the same furrow once too often Output: B
Input: equilibrium the movie as opposed to the manifesto is really. really stupid
Output: B
<More demonstrations..>
Input: friendly vacation for four old friends two couples from college turns ugly.. ... then
Output: A
Input:
he meets god and is given all the powers of god Output:
Additional potential future directions. Apart from applying DETAIL to more generalized prompt- ing settings, we think it is also an interesting direction research whether DETAIL can provide meaningful attribution for pruned [55, 59] (or distilled [29, 30]) networks. This is especially useful
Lecture notes.
18

---

since, compared to very large models, pruned small networks can be deployed privately, admitting straightforward application of DETAIL. Moreover, the size of the hidden states tends be smaller, allowing for faster computation of DETAIL score.
D
Additional Experiments
D.1 Additional Results for Demonstration Perturbation Task
We include the full results for the demonstration perturbation task using a Vicuna-7b v1.3 model in Fig. 7 and using Llama-2-15 model in Fig. 8. A consistent trend can be observed across different datasets using both models.
vicusa-76 7b
vicuDa- 7b
vicuna-75
0.5
20.4
Acmurey
0.6
Accurweyy
Accurery nerr
0.3
0.4-
0.4
0.68
high corrupt
0.2
corrupi run
0.1
?
10
13
16
10
corrigted
No. corrupted
No. corrapted
corrapted
0.5
0.7
Accurwoy
0.4
Accurscy
0.6
0.6
Accurasy
0.7
remove high
low
0.4
9.4
10
13
16
16
10
4
13
16
No. romoved
No. removed
No. remooved
No. pomovved
Figure 7: Corrupting and removing demonstration on datasets affects the model predictive power differently on AG News and SST-2, Rotten Tomatoes, and Subj from left to right using Vicuna-7b. Commpting/removin demonstrations with high DETAIL scores results in lower model accuracy and vice versa. Coruptiiggremoovig demonstrations randomly results in an accuracy in the middle as expected. All experiments are repeated with 10 independent trials. X = 1.0. Lines and shades represent the mean and standard error respectively.
11ama-2-13b
1lama-2-13b
Llama-2-13b
Lluma22113
Appiric
0.6
Accurecy
0.8
Annurury
0.6
0.69
0.6
0.4
remove high
0.5
removv
remove random
0.4
0.4
10
13
7
16
10
16
No. retooved
No. removed
No. rello9ed
No. pimooeddd
(a) AG News Removal
(b) SST-2 Removal
(c) Rotten Tomatoes Re-
(d) Subj Removal
moval
1lmmo-2-15h
Llmma-2-13 3b
(Jama-2-136
0.8
0.8
Aocurocyy
Apcuracy
0.6
Aocururg
Accurscy
0.6
0.4
corrupt high corrupt low
0.4
0.2
corrupt
10
13 16
7
10
16
10
13
4
7
10
13
16
No, corrupted
No. currugted
No. cocrupted
No.
corrupted
(c) AG News Corrupt
(f) SST-2 Corrupt
(g) Rotten Tomatoes Cor- rupt
(h) Subj Corrupt
Figure 8: Results of model prediction accuracy vs. number of demonstrations removed/corrupted on Llama-2-133 model. X = 1.0. Lines and shades represent the mean and standard error respectively.
We compare (in addition to Vicuna-7b) total of 3 L.MMs: Llama-2-7b [60], Llama-2-133 2-b3 [60] and Falcon-7b [4]. We reuse the same experimental setup the demonstration label perturbation task
19

---

and compare the accuracy by removing and corrupting 10 among 20 ICL data with high/low DETAIL scores computed different models. The results are tabulated in Table A similar trend can be observed across these models where removing/corupping demonstrations with high Tes results in lower accuracy and vice versa. The results demonstrate that our method is robust against model tre-traiinn/fine-tuiing data as well as model size.
Table 5: Performance on demonstration perturbation task across different models. The mean and stan- dard error (in bracket) of predictive accuracy after removal or corruption 10 out of 20 demonstrations of 20 randomly drawn ICL datasets is shown. All experiments are independently repeated 20 times.
remove high
remove low 1
remove random
corrupt high
corrupt low
corrupt random
Subj
Llama-2-7b
0,380 (0.034)
0.675 (0.025)
0.535 (0.035)
0,250 (0.036)
0.690 (0.031)
0.445 (0.019)
Llama-2-13b
0.570 (0.029)
0.700 (0.032)
0,575 (0.038)
0.380 (0.030)
0.635 (0.031)
0.500 (0.021)
Falcon-7h
0.450 (0.026)
0.690 (0.028)
0.580 (0.042)
0.280 (0.026)
0.630 (0.023)
0.445 (0.043)
SST-2
Llama-2-7b
0,480 (0.031)
0.670 (0.030)
0.540 (0.042)
0.145 (0.018)
0,445 (0.042)
0.275 (0.039)
Llama-2-130
0.655 (0.034)
0.830 (0.031)
0,740 (0.031)
0.220 (0.025)
0.410 (0.031)
0.345 (0.022)
Falcon-7h
0.560 (0.043)
0.775 (0.028)
0.680 (0.032)
0.225 (0.031)
0.545 (0.030)
0.340 (0.023)
Rotten toamtoes
Llama-2-77
0,435 (0.034)
0.670 (0.060)
0.540 (0.045)
0,120 (0.025)
0,420 (0.039)
0.235 (0.026)
Llama-2-138
0.635 (0.032)
0.795 (0.032)
0.690 (0.033)
0.205 (0.028)
0.420 (0.040)
0.315 (0.035)
Falcon-7b
0.475 (0.037)
0.780 (0.024)
0.620 (0.024)
0,225 (0.023)
0.590 (0.031)
0.445 (0.025)
AG News
Llama-2-7b
0.145 (0.018)
0.525 (0.036)
0.360 (0.026)
0,150 (0.016)
0.520 (0.030)
0.325 (0.037)
Llama-2-130
0,260 (0.018)
0.600 (0.041)
0.500 (0.032)
0.175 (0.026)
0.565 (0.049)
0.385 (0.03
Falcon-7b
0.155 (0.019)
0.460 (0.021)
0,335 (0.025)
0.085 (0.020)
0.465 (0.017)
0.265 (0.027)
D.2 Additional Results for Noisy Demonstration Detection
We include the results AG News, SST-2, Rotten Tomatoes, and Subj datasets using Vicuna-7b model in Fig. 9. Similar trends as in the main text is observed. A counterpart experiment using Llama-2- 2-b is in Fig. 10, where similar trend is observed.
viruna V
vicuma-
YScuma 7b
vicuna 7/b
Praction decotiiied
Draction udeaattie
Fraction Iecaltee
Practam decotiieed
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
DETAIL LOO
DETAIL
DETAIL
DETAIL
0.25
0.25
100
0.25
100
0.25
L00
mandom
random
mandom
0.00
0.00
0.00
0.00
No. demos. cbecked
checked
No. demos. checked
demos. diecked
Figure 9: (Left to right) Fraction of all noisy labels identified vs. the number of demonstrations ranked by our method (with projection down to 1000 dimension) and LOO checked respectively on AG News, SST-2, Rotten Tomatoes, and Subj datasets. X 10-9. All experiments are repeated with 10 independent trials. Lines and shades represent the mean and standard error respectively.
20

---

1.40
1.60
1.0
1.0
0.75
0.75
0.5
0.3a (10X)
DETALL
RETAB
BETAIL
BETAII
LOG
5.00
0.00
10
10
20
26
(a) Detecting noisy (b) Detecting noisy (c) Detecting noisy (d) Detecting noisy (e) Wall time compar- label
label AG News label on SST-2
Rotten label Subj
ison
Tomatoes
Figure 10: (a-d) Fraction of all noisy labels identified vs. the number of demonstrations ranked by our method (with projection down to 1000 dimension) and LOO checked respectively. (e) Wall time comparison across all datascts. À 10-9. All experiments are repeated with 10 independent trials using a Llama-2-133 model. Lines and shades represent the mean and standard error respectively.
D.3 Additional Results for Dimension Reduction
The experiments using Vicuna-7b on AG News, SST-2, Rotten Tomatoes, and Subj can be found in Fig. 11. It can be observed that the trend is consistent across different datasets.
0.8
0.70
0.60
0.65
0.83
Whll time
0.4
Wall
0.4
Wall time
0.60
0.4
Whll time
0.6
0.4
0.55
0.55
0.2
AUC.
1000
160
Project dimonsiio
Project dimension
diimnston
Project dimension
Figure 11: (Left to right) wall time in seconds (left y-axis) and AUCROC (right y-axis) vs. projection dimension d' on AG news, SST-2, Rotten Tomatoes, and Subj datasets. Experiments are repeated with 10 trials. Lines and shades represent the mean and standard error respectively.
D.4 Additional Results for Demonstration Reordering Task
We conduct additional experiments on the demonstration reordering task by perturbing 6 demonstra- tions in each ICL dataset of 20 demonstrations. The results are shown in Table 6. It can be observed that reordering with Ieei still achieves an improvement in test accuracy, demonstrating the robustness of our method.
Table 6: Predictive accuracy of demonstrations permuted randomly and based on Iaar respectively. The mean and standard error (in bracket) with 80 repeated trials shown.
Subj
SST-2
Rotten Tomatoes
Corrupt 6 demonstrations
Baseline (random)
0.588 (7.96e-03)
0.487 (9.45e-03)
0.398 (1.12e-02)
Reorder (DETAIL)
0.604 (7.39c-03)
0.520 (1.01c-02)
0.425 (1.37-02)
Difference
0.0164 (7.05c-03)
0.0323 (8.13e-03)
0.0267 (1.01e-02)
D.5 Additional Results for Demonstration Curation Task
We include the full results for all datasets on both Vicuna-7b and Llama-2-13b in Fig. 12. can be observed that the gap between removing demonstrations with high/low Ire is wider with Llama- 2-13b. We believe this because Llama-2- 13b being larger model possesses better capability of formulating the internal optimizer compared to Vicuna-7b which is smaller.
21

---

Vicuna 7b
Vicuna 7b
Vicnn 76
Vieuna-P 76
0.40
0.60
0.65
Accureoy
0.35
Accuracy
0.35
Ansuracyy
Ancuracy
0.60
0.30
retiove high
0.50
low
0.45
dom
0.25
12mm-2-13h
Llman-215b
Liama-3-18 3b
Clam 1.1m-2-1156
0.5
0.8
Accursci
0.6
Agcurweyy
0.6
Accuracy
0.6
0.4
remove high
9.4
0.3
ramov' low
random
0.2
0.4
10
retroved
Figure 12: (Left to right) test accuracy vs. number demonstrations removed using Test on AG news, SST-2, Rotten Tomatoes, and Subj datasets using (top) Vicuna-7b and (bottom) Llama-2-136 All experiments are repeated with 80 independent trials. Lines and bars represent the mean and standard emoor respectively.
We additionally provide the result for the demonstration curation task with Attention [6] and [53] using 10 iterations of LiSSA update for computing the hessian vector product [1] and compare with the with 5 iterations. The result is shown in Table 7. With 10 iterations, the wall time is much higher (over 20x) but accuracy is comparable to the case with 5 iterations. For Attention, the performance is comparable random removal as shown in Table 2.
Table 7: Test accuracy after curating the ICL dataset and the incurred wall time (in seconds on one L40 GPU). The mean and std. error (in bracket) is shown with 20 repeated trials.
Metric
Attention [6]
[53] (#5)
[53] (#10)
Subj (curate 10 demonstrations)
Accuracy 1
0.627 (2.01c-02)
0.556 (1.38c-02)
0.597 (3.23c-02)
Wall time
37.6 (1.09e+00)
9.37 (4. 19e-01)
245 (8.80e-01)
SST-2 (curate 10 demonstrations)
Accuracy 1
0.460 (2.60e-02)
0.493 (1.34e-02)
0.499 (1.64e-02)
Wall time
24.9 (7.26e-01)
10.6 (7.80e-01)
241 (9.63e-01)
Rotten Tomatoes (curate 10 demonstrations)
Accuracy 1
0.488 (2.05c-02)
0.498 (1.72c-02)
0.494 (1.74c-02)
Wall time +
36.8 (1.09c+00)
9.74 (5.57c-01)
240 (4.61c-01)
AG News (curate 10 demonstrations)
Accuracy 1
0.350 (1.35c-02)
0.416 02.10c-02)
0.346 (2.07c-02)
Wall time
138 (3.28e+00)
11.9 (5.78e-0))
441 (4.0e-01)
Given that BERT-score is light-weight it is intuitive to think about whether it is possible to combine DETAIL and BERT score and reap the benefits of both worlds. We conduct a preliminary experiment in this direction. Specifically, we consider applying our formulation of DETAIL (Eq. (1)) using hidden state modified by the BERT embedding. Specifically, we consider a weighted average of the BERT embedding and the transformer's hidden state as well as direct concatenation of the two embeddings. The results are shown in Table 8. Surprisingly, we discover that using a weighted average of the transformer's hidden state and the BERT embedding improves the attribution accuracy.
22

---

Table Test accuracy after curating the ICL dataset and the incurred wall time (in seconds on one L40 GPU). The mean and std. error (in bracket) is shown with 20 repeated trials.
Metric
BERT Embedding Only
Weighted Average (equal weight)
Concatenation
Subj (curate 10 demonstrations)
Accuracy
0.665 (2.36e-02)
0.758 (1.97e-02)
0.719 (2.75e-02)
SST-2 (curate 10 demonstrations) Accuracy
0.475 (1.31e-02)
0.579 (2.34e-02)
0.5% (1.96e-02)
Rotten Tomatoes (curate 10 demonstrations) Accuracy
(2.10022
0.575 (2.46e-02)
(2.75e-02
AG News (curate 10 demonstrations) Accuracy
0,408 (2.10)))2)
0.425 (1.76e-02)
0,412 (1.50e-02)
D.6 Ablation of Different Transformer Layers for Computing DETAIL scores.
We experiment with the difference in the effectiveness DETAIL using the embeddings of different layers. We conduct experiments on demonstration removal, demonstration perturbation, and noisy label detection tasks. The results are shown in Fig. 13. It can be observed that obtaining the DETAIL scores from the later layers of the model consistently produces desirable results.
0.8-
Fraction identifedd
0.8
Accuracy
Accuracy
0.70
0.6
remove high
corrupt high
0.65
0.4
corrupt
0.60
0.4
remove random
0.2
corrupt random
0.55
DETAIL
1 5 13 17 21 25 29 Layer number
1
9 13 17 21 25 29
5 9
13 17 21 25
31
Layer number
Layer number
(a)
Remove demonstration on Subj
(b)
demonstration on Subj
Corrupt
(c) Detecting noisy label on Subj
0.8
0.8
Fraction identified
0.75
Accuracy
Accuracy
0.6
0.4
0.65
high low
corrupt high
0.4
0.2
corrupt low
random
corrupt random
0.35
DETAIL
0.2
13 17 25 29
5
17 21 29
9
17 21 25 31
Layer number
Layer number
Layer number
(d) Remove demonstration SST-2
(c) Corrupt demonstration SST-
Detecting noisy label SST-2
Figure 13: Results of different task performance vs. the layer number in Vicuna-7b model which consists of 31 layers. Experiments are repeated with 10 trials. 1.0 for (a,b,d,e) and 1 for (c.f). Lines and shades represent the mean and standard error respectively.
D.7 Ablation of Target Position for Computing DETAIL.
As rule of thumb, for each demonstration, we generally want to take the embedding of its last few tokens because of the causal nature of inference and because information generally flows toward the end of the sequence [64]. We compare two possible choices of target position: the column position (immediately before the label) and the label position. We experiment on the demonstration removal task with these two choices of embeddings. The results are shown in Fig. 14. Using embeddings of both positions achieves decent task performance as reflected by the clear distinction in accuracy between removing demonstrations with high/low DETAIL scores, demonstrating that our method is robust against the choice of token embeddings. In our experiments, we adopt the column position to isolate information about the label from the embedding.
23

---

Vicuaa70 70
Vicuna 7D
vicuna-7B
vicuna-75
0.7
0.8
Acouzacy
0.8
0.7
0.7
0.6
Accuracy
0.7
0.6
high
9.6
high reanove candom
0.5
remoov
0.5
remove
0.3
remove random
0.4
0.4
0.4
13
16
10
16
No. removed
No. removed
No.
No. redroved
(a) Subj using label posi- tion
(b) Subj using column po- (c) SST-2 using label sition
(d) SST-2 using column position
sition
Figure 14: Results of model prediction accuracy vs. number of demonstrations removed using
different positions for taking embeddings. X 1.0. Lines and shades represent the mean and standard error respectively.
D.8 Experiment on State-Space Model Architecture
We consider experiments on a popular state-space model (SSM) architecture, a Mamba-2.8b model [27]. which consists of 64 layers with 2560. The tasks are described in the main text in detail. While we find that DETAIL can still successfully attribute Mamba on certain tasks and datasets, the performance is inconsistent. One hypothesis is that the largest currently available Mamba model (2.8B) is still significantly smaller than the 7B LLMs we conduct experiments on in the main text. A smaller model size reduces the inductive power of the model formulate the "internal optimizer", leading to less interpretive DETAIL scores. We would also like to note that DETAIL not designed to work on SSMs.
Demonstration removal. The demonstration removal experiment follows the same setup Sec. 5.2 but uses a Mamba-2.88 model instead, which is the largest model officially open-soureed. The results are shown in Fig. 15. Interestingly, removing demonstrations according to DETAIL scores still can influence predictive performance the desirable manner where removing demonstrations with high Tre leads to lower accuracy and vice versa.
0.6
mamba
0.6
mamba
0.70
mamba
0.4
Accuraty
0.5
0.5
Ancureec
00.65
remove high
9.4
remove high
remove high
0.60
remove high
remove low
remove Iow
remove Low
random
0.2
random
0.55
10 1's
10 removed
10
13
16
No. removed
(a) AG News removal
(b) SST-2 removal
(c) Rotten Tomatoes moval
(d) Subj removal
Figure 15: Results of model prediction accuracy number of demonstrations removed on Mamba. 1.0. Lines and shades represent the mean and standard error respectively.
Noisy demonstration detection. We further consider the noisy demonstration detection task as described Sec. 5.2 on Mamba. Unfortunately, the performance is not consistent across datasets, as shown in Fig. 16: detecting demonstrations with high Iuelf performs close to random selection on SST-2 and Rotten Tomatoes datasets, although the inference speedup is still significant. We leave the analysis of these failure cases to future work.
24

---

1.40
1.60
1.0
1.0
0.75
0.75
0.5
Da (1OX)
DESAIL
BETAB
BETAA
BETAII
LOG
8.00
0.00
10
10
20
26
(a) Detecting noisy (b) Detecting noisy (c) Detecting noisy (d) Detecting noisy (e) Wall time compar- label
label AG News label on SST-2
Rotten label Subj
ison
Tomatoes
Figure 16: (a-d) Fraction of all noisy labels identified vs. the number of demonstrations ranked by our method (with projection down to 1000 dimension) and LOO checked respectively. (e) Wall time comparison across all datasets. 10-9. All experiments are repeated with 10 independent trials. Lines and shades represent the mean and standard error respectively.
Demonstration curation. As DETAIL performs well using Mamba on the demonstration removal task, it is reasonable to hope that works well the demonstration curation task well. As turns out, DETAIL performs well on binary classification tasks as shown in Fig. 17 but performs poorly on AG News which is 4- 4-way classification. We hypothesize that this is due Mamba's worse inductive power to formulate an internal algorithm successfully.
manbb
mamba
mamba
0.600
mumbe
0.35
Accuraig
0.28
Accuracy
Accurace
Accurse
0.975
0.30
0.27
reestrve high
0.25
remonce high
remove high
0.550
high
remsree low
remove Juw
rumove low
0.26
dom
0.20
randon
0.25
Figure 17: (Left to right) test accuracy vs, number of demonstrations removed using Test on AG news, SST-2, Rotten Tomatoes, and Subj datasets using Mamba-2.8h All experiments are repeated with independent trials. Lines and bars represent the mean and standard error respectively.
25

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose an attribution method for demonstrations via the viewpoint of treating the transformer as implementing an internal algorithm and demonstrate its effectiveness. Our abstract and introduction accurately reflect this claim with elaboration.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: A limitation of our work is discussed in Sec. 6.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We demonstrate the effectiveness of our method primarily by conducting exper- iments under different scenarios. There are theoretical insights that lead to our formulation, for which we have properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We have included the specific details of our algorithmic implementation in App. B and an illustration in Fig. 1. Python code for reproducibility is also included the supplemental materials. The datasets used in the experiments open-source, commonly used, and well-specified.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Python code with the conda environment file is included in the supplemental materials with rumning instructions in the README file. Commands for reproducing experimental results are also included a bash file. All datasets are freely downloadable with downloading code snippets written in the code.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Important experimental settings are specified in each subsection in Sec. 5.1, Sec. 5.2, and See. 5.3. Additional details are provided in App. D.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
26

---

Justification: Average and standard errors are included in the experimental results either in shades (for figures) or in brackets (for tables).
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computing hardware (in terms of GPU) is specified in App. A. Wall time for approach different tasks (and the corresponding GPU) provided in Sec. 5.2 and Sec. 5.3.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / hwtprpp.cc/ppubiicccccoc EeEtei/teeeeeeeeeeeeeeeeee ceGuidelineess
Answer: [Yes]
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the need for interpretability in in-context learning which work addresses in the main text. Additional discussion is in App. C.
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our contribution is mainly improving the interpretability of in-context learning. Data and models used are freely and readily available online.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The datasets used in our work are all properly cited.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
27


Reviews:
[{"review_id": "Qf89uH0O0l", "reviewer": "Reviewer_ixwr", "summary": "The paper proposes a novel attribution method for demonstrations in ICL. The proposed method takes a perspective that the transformers learn in context by formulating an internal optimizer. The influence function is approximated as an internal kernelized ridge regression, where the representations are taken from the intermediate layers of the transformers and white-box LLMs. The paper also demonstrate that the attribution obtained from the white-box LLM exhibits transferable characteristics to black-box models. The paper also showcases a few promising applications of DETAIL.", "questions": "- Could you comment on whether DETAIL can be used to discover adversarial attacks of LLMs?\n- Could you comment on how DETAIL can help with LLM training especially for instruction tuning?", "limitations": "Yes, the author addressed the limitations.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- By and large, the paper is written well. I especially appreciated the discussion on the relations to reinforcement learning, and how the potential functions differ from Lyapunov functions used in more classical settings.\n- The idea of using the perspective that transformers formulating an internal optimize for attribution is novel and useful.\n- The section on accelerating the computation with self-influence is also interesting, and highlights the authors attention to computational issues.\n- The application of the DETAIL may open new research directions in LLMs for bettter instruction-tuning algorithms.\n- The mathematical derivations appear to be correct.", "weaknesses": "- It would have been interesting to see how well this method would identify adversarial attacks for LM.\n- Some additional commentary on how DETAIL can help with LLM training would have been very helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to the review of reviewer ixwr\n\nDear Reviewer ixwr,\n\nThank you for reviewing our paper and highlighting the novelty of our application of 'influence' for ICL which has the potential to inspire better instruction-tuning algorithms.\n\nWe would like to address your concerns in the following.\n\n### 1. Identifying adversarial attacks for LM\nWe thank you for suggesting this exciting potential use case. In this work, we have demonstrated how DETAIL can identify corrupted demonstrations, which is one form of adversarial attack where an attack is performed by perturbing the ICL demonstration label (Table 1 and Table 3). We believe it is an interesting area to research how DETAIL can be applied to other types of attacks. For example, DETAIL might be used to defend against attacks that attempt to steal private ICL demonstrations by filtering out or rephrasing queries that cause excessively high DETAIL scores.\n\n### 2. Additional comment on how DETAIL can help with LLM training\nWe thank you for the question. While we primarily focus on in-context learning settings in this work, the DETAIL score can also potentially be used for selecting demonstrations for instruction fine-tuning. One way to perform the selection is to compute the DETAIL score of each ICL demonstration on a validation set and then use the demonstrations with the highest DETAIL scores for fine-tuning.\n\nWe will incorporate the mentioned points in the future work section in our work.\n\nOnce again, we would like to thank you for the insightful suggestions. We hope our response has been helpful in addressing your questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AhZ4xxSmt2", "reviewer": "Reviewer_EuFx", "summary": "The paper introduces DETAIL, a novel influence-function based attribution technique to estimate the influence of each example in the demonstration sequence for the given target query for in-context learning (ICL). The authors empirically validate DETAIL’s effectiveness in stylized experiments and LLMs. Additionally, they demonstrate DETAIL’s applicability in real-world tasks like demonstration reordering and curation and the transferability of DETAIL’s attribution scores from white-box to black-box models.", "questions": "1. How do you obtain $y_{test}$ for $I_{test}$?\n2. Have you explored initializing $m(x)$ and $y$ with BERT embeddings? This could be a potential area for improvement, leveraging pre-trained embeddings for better initialization.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Innovative Application of Influence Functions: The application of influence functions to interpret in-context learning (ICL) is both innovative and intriguing, offering a fresh perspective on model interpretability.\n2. Transferability to Black-Box Models: The method demonstrates promising results in transferring attribution scores from white-box models to black-box models, which significantly enhances its practical applicability.\n3. Promising Performance: The empirical performance of DETAIL is promising, showcasing its potential effectiveness in real-world scenarios", "weaknesses": "1. Strong Assumptions: The work is built on the assumption that transformers implement an internal optimizer. This assumption, while supported by some theoretical proof in stylized settings, may not universally hold. The proof further assumes a learn-to-learn setting for ICL, which differs from the definition provided in lines 88-105 in this paper. Additionally, Equation 3 assumes the loss function of ICL is a ridge regression function without robust theoretical guarantees.\n\n2. Lack of Analysis for Equation 6: Given the strong assumptions underlying the derived Equation 6, its reliability may be questionable. \n\n3. Sensitivity to Demonstration Ordering: Recent work has demonstrated that ICL is not sensitive to ordering with more powerful demonstration retrievers and advanced language models, contradicted  with the conclusion in  line 35.\n\n4. Limited Experimental Comparisons: The experimental validation of DETAIL in ICL demonstration curation could be strengthened. The authors should compare their method with other learning-free demonstration selection methods, such as those based on BERT embeddings.\n\n5. Computational Cost: The computational cost of DETAIL is relatively high for demonstration curation or selection. Existing methods that utilize BERT embeddings are more computationally efficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to the review of reviewer EuFx\n\nDear reviewer EuFx,\n\nThank you for your review, compliment for the innovative application of the influence function in our work, and the acknowledgment of the real-world applications of DETAIL.\n\nWe would like to address your concerns as follows. Kindly note that all citation numbers follow the reference list of the main paper.\n\n### 1. Strong assumption\nWe wish to clarify that while the pioneering work [51] provided the theoretical proof under the learn-to-learn setting, follow-up works have theoretically shown that transformers can implement internal optimizers **even when they are pre-trained on random instances** [2, 52]. [20] further demonstrated the internal optimizer phenomenon w.r.t. GPT.\n\nWhile we acknowledge that the kernelized ridge regression does not have a formal theoretical guarantee, it should be noted that the formulation closely follows that mentioned in [51] with slight modification, as we apply it with the influence function (lines 128-131). Following the kernelized ridge regression, equation 6 is then rigorously derived by following the exact definition of the influence function.\n\n\n### 2. Sensitivity to demonstration ordering\nWe would appreciate it if you could kindly provide sources that have made this claim so that we could better address this concern.\n\nAssuming that it is the case where very large models are indeed insensitive to the ordering of demonstrations, we highlight that the claim in line 35 is based on previous works considering relatively smaller language models (typically less than 10b) [35, 37].\n\nSmaller language models are also widely used in real-life scenarios because of their cost-effectiveness (a 7b model can be run on a personal PC). In these applications, the models typically suffer from the problem of being sensitive to demonstration ordering.\n\nAdditionally, while it is understandable that very large models are more robust to re-ordering, recent work (e.g., Table 12 of Wu et. al., Arxiv 2405.16122) has shown that there can be up to 10% accuracy difference when large noise is injected by carefully optimizing the order of demonstrations on GPT-3.5.\n\n### 3. Curation experiment with BERT embedding\nThank you for suggesting the additional comparison baseline (i.e., curating demonstration with BERT-based methods). We have conducted an additional experiment under the setting of Table 2 using BERT score from a popular sentence-transformer model `sentence-transformers/msmarco-bert-base-dot-v5`. The results are tabulated below:\n\n\n| Metric | DETAIL ($d=1000$) | DETAIL ($d=50$) | BERT Score |\n| --- | --- | --- | --- |\n| Subj (Accuracy) | **0.747 (2.60e-02)** | 0.713 (2.43e-02) | 0.671 (2.43e-02) |\n| SST2 (Accuracy) | **0.607 (2.12e-02)** | 0.530 (4.09e-02) | 0.477 (2.54e-02) |\n| Rotten tomatoes (Accuracy) | 0.555 (1.94e-02) | **0.557 (2.32)** | 0.435 (1.33e-02) |\n| AG News (Accuracy) | **0.412 (1.35e-02)** | 0.387 (1.32e-02) | 0.355 (1.58e-02) |\n| |\n| Subj (Wall time) | 5.22 (1.17e-01) | 3.18 (4.63e-02) | **2.97 (3.05e-02)** |\n| SST2 (Wall time) | 4.88 (1.35e-01) | **2.65 (4.09e-02)** | 2.91 (2.78e-02) |\n| Rotten tomatoes (Wall time) | 5.11 (1.06e-01) | 2.97 (5.36e-02) | **2.95 (4.43e-02)** |\n| AG News (Wall time) | 10.4 (1.07e-01) | 6.17 (7.30e-02) | **3.15 (5.87e-02)** |\n\nWhile we acknowledge that BERT score is relatively faster than DETAIL ($d=1000$), the speed gain is mainly due to **BERT scores being model-agnostic** and thus likely comes at the cost of poorer performance compared to DETAIL (as reflected by the lower accuracy in the table above). We wish to highlight that one advantage of **DETAIL is that its attribution is dependent on the transformer used** which provides more interpretability and improved performance.\n\nMoreover, while BERT score is fast, we note that DETAIL is also very computationally efficient: DETAIL is much faster than most attribution methods shown in Table 2. DETAIL can be even faster by reducing $d$. As shown in the table, when $d=50$, DETAIL has a comparable running time as BERT score while still achieving higher accuracy.\n\n\n### Questions\n1. Obtaining $y_{\\text{test}}$: We wish to clarify that $y_{\\text{test}}$ is part of $z_{\\text{test}} = (x_{\\text{test}}, y_{\\text{test}})$ (line 103) which refers to the ground truth label of the query sample. $y_{\\text{test}}$ is provided as part of the test dataset.\n\n2. Incorporating BERT embedding in $m(x)$: Thank you for providing the insightful suggestion to incorporate BERT embedding in calculating $m(x)$. We believe it is an exciting direction for future research to consider a combination of BERT and transformer embeddings to improve performance. We have conducted a preliminary experiment with three different $m(x)$ computation tabulated below. For the transformer embedding, we use $d=768$, the same size as the BERT embedding. When using BERT embedding only, the performance is worse than DETAIL. Interestingly, using an equal-weighted average results in SOTA test accuracy on 3 out of 4 datasets.\n\n| Metric | Bert Embedding Only | Weighted Average (equal weight) | Concatenation |\n| --- | --- | --- | --- |\n| Subj (Accuracy) | 0.665 (2.36e-02) | **0.758 (1.97e-02)**  | 0.719 (2.75e-02) |\n| SST2 (Accuracy) | 0.475 (1.31e-02) | 0.579 (2.34e-02)  | 0.596 (1.96e-02) |\n| Rotten tomatoes (Accuracy) | 0.510 (1.85e-02) | **0.575 (2.46e-02)**  | 0.570 (2.75e-02) |\n| AG News (Accuracy) | 0.408 (2.10e-02) | **0.425 (1.76e-02)**  | 0.412 (1.50e-02) |\n| |\n| Subj (Wall time) | 5.96 (3.05e-01) | 8.05 (1.73e-01)  | 7.89 (1.33e-01) |\n| SST2 (Wall time) | 5.24 (1.63e-01) | 7.61 (1.69e-01)  | 7.51 (2.01e-01) |\n| Rotten tomatoes (Wall time) | 5.25 (2.93e-01) | 9.18 (1.78e-01)  | 9.11 (1.57e-01) |\n| AG News (Wall time) | 4.30 (8.07e-02) | 12.8 (2.00e-01)  | 11.2 (1.73e-01) |\n\n\nThank you again for providing a constructive review and offering insightful suggestions. Please let us know if there are any remaining questions. We are glad to offer further clarifications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "booIJoQxq7", "reviewer": "Reviewer_oPDt", "summary": "This paper proposes a new method to estimate the influence of ICL examples to a query. This influence estimation can better help ICL learning, for example, reorder ICL examples and curating ICL examples.", "questions": "See above.", "limitations": "Discussed in Section 6.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The motivation is clear, and the paper overall is well-written.\n- The idea to estimate the influence of ICL examples is interesting, and can help better ICL learning for LMs.\n- Results in Table 2 seem to show that DETAIL performs better than existing influence functions (token-level) and also some recent work that estimates ICL influences.", "weaknesses": "- More comprehensive experiments: for applying DETAIL, the authors only showed a few tasks (all classification) and on one major public model (Vicuna-7b). To show the effectiveness of DETAIL, more comprehensive experiment results should be provided: e.g., Table 1 and 3 should use more tasks other than classification, and more models (can the authors add results on Llama-2-13b?).\n\n- DETAIL for ICL order optimization: based on the description in Section 5.3, it seems like the authors re-ordered the examples not just based on I_{self}, but also based on the trend plot in Figure 5. Does test accuracy w.r.t. perturbed position alway need to be computed first before deciding on the orders? This seems really expensive and also the choice of putting \"two demonstrations\" in the front seems very arbitrary. How would one use such a method for a random new task?\n\n- The need to use white-box models and poor transfer results: DETAIL requires white-box model access, and the transfer result in Table 3 is not significant on the more realistic setting (no corruption). Also, can the authors show the transfer results with more models, e.g., LLama-2 to GPT-3.5, and Vicuna-7b to GPT-4?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to the review of reviewer oPDt\n\nDear reviewer oPDt,\n\nThank you for the review and for acknowledging the motivation and main idea of our work.\n\nWe would like to address your concerns in the following. Kindly note that all citation numbers follow the reference list from the main paper.\n### 1. Comprehensiveness of experiments\nWe wish to first highlight that our experiments serve mainly to demonstrate the interpretability of DETAIL's attribution and the applicability of DETAIL in various scenarios including demonstration curation and re-ordering. For models, in addition to Vicuna-7b, we have also considered Llama-2-7b, Llama-2-13b models in the paper, as well as an SSM architecture model, Mamba-2.8B, with some of the results in Appendix.\n\nWe acknowledge that while DETAIL itself can be useful in many tasks, our experiments only considers classification tasks currently. To extend our work to support other types of tasks, we may modify the loss function accordingly. For example, to support generation tasks where the ground truth is a sequence of desired outputs, we may modify the loss in Eq(3) to be an autoregressive cross-entropy loss instead of an MSE loss.\n\nWe thank you for suggesting including experiments for Table 1 and Table 3 using Llama-2-13b. Due to the character limit, we include the additional experimental results together with the transfer results in the PDF in the global rebuttal. We provide an analysis of the additional results in bulletpoint 4 below.\n\n### 2. Placement of demonstrations in the re-ordering task\nWe would like to clarify that figure 5 serves to verify how test accuracy changes with the position of the corrupted demonstration. Figure 5 shows that placing corrupted demonstrations at the two ends generally leads to better test accuracy for various datasets (e.g., subj, SST-2, and Rotten tomatoes). Hence, we use the DETAIL score as a proxy for the corruptedness of the demonstration and place a few demonstrations with the highest DETAIL score in front. We expect the same trend to hold on a random new task and we can use the same technique (placing a few demonstrations with the highest DETAIL score in front). The additional experiments in bulletpoint 4 are all conducted by placing 2 demonstrations with the highest score in front.\n\n### 3. White-box access and Transfer performance on re-ordering task\nWe acknowledge that DETAIL specifically considers the white-box setting to provide interpretable attribution. While methods under a black-box setting (e.g., [41, 46]) may offer wider applicability, they may also have limited interpretability and poorer performance due to the absence of access to the model's internal working (as shown in Table 2). We highlight that **interpretable attribution of ICL demonstrations is the main focus and contribution of this work**.\n\nWe acknowledge that transfer performance is less significant for the re-ordering task in Table 3 when no demonstration is corrupted. We think it is because GPT-3.5/GPT-4o can already achieve very high test accuracy with the baseline ordering ( baseline accuracy with no corrupted demo in Table 3 is on average 12.0 percentage points higher than that in Table 1). \n\nHowever, while reordering demonstrations may not improve performance when there are no corrupted demonstrations, it does not deteriorate performance either, nor does it require heavy extra computation (only 1 pass). As such, reordering offers a way to obtain **\"free\" potential performance gain without the costly curation or obtaining more demonstrations**.\n\nOn your comment that it is less \"realistic\" to have corrupted demonstrations in the ICL dataset, we argue that although a carefully curated ICL dataset contains few corrupted demonstrations, in various real-world scenarios, ICL demonstrations might not come from a well-prepared dataset but are generated in real-time. For example, LLM-powered search engines (e.g., Perplexity.AI) obtain ICL demonstrations in real-time from web-scrapped data based on the user query, which very often contains corrupted demonstrations. Recent works that consider self-generating ICL demonstrations from LLM (e.g., Chen et. al. \"SELF-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations\", EMNLP 2023) are also subjected to corrupted demonstrations due to LLM's hallucination. More importantly, as ICL demonstrations are typically generated in real-time in these scenarios, costly curation is not tractable and thus the proposed DETAIL-based re-ordering technique that can indeed be computed in real-time, can be highly effective.\n\n### 4. Analysis of additional results on re-ordering task\n\nWe kindly refer you to the PDF attached in the global author rebuttal for the additional experimental results.\n\nFor Llama-2-13b, performance improvement with no corrupted demonstration is less significant compared to Vicuna-7b, although with 3 corrupted demonstrations, we can still observe 1-2% improvement. We attribute this to the stronger baseline performance (baseline (no corrupted) average accuracy on Llama-2-13b is 16.4 percentage points higher than that on Vicuna-7b).\n\nFor transfer performance, we observe that the performance gain is not significant with no corrupted demonstrations. With 3 corrupted demonstrations, GPT-4o shows limited performance gain, which is probably because GPT-4o has stronger pretrained prior and can achieve very high test accuracy even with some bad demonstrations. However, for GPT-3.5, the performance improves by more than 3% in this scenario. On the other hand, when 6 demonstrations are corrupted, re-ordering shows a significant improvement of over 3% on GPT-4o as well. This demonstrates the benefit of re-ordering, even for large models, with noisy demonstration data.\n\n\n\nThank you for providing a constructive review and the questions. We hope our response has helped address your concerns. We are glad to provide further clarifications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r2HWWS9gUu", "reviewer": "Reviewer_smfw", "summary": "This paper introduces DETAIL, a novel technique for attributing and interpreting in-context learning (ICL) demonstrations in transformer-based language models. The authors propose an adaptation of the influence function, typically used in conventional machine learning, to address the unique characteristics of ICL. DETAIL treats transformers as implementing an internal kernelized ridge regression, allowing for efficient and effective attribution of demonstrations. The method is evaluated on various tasks, including demonstration perturbation, noisy demonstration detection, and real-world applications such as demonstration reordering and curation. The authors demonstrate DETAIL's superiority over existing attribution methods in terms of both performance and computational efficiency.", "questions": "see the weakness section", "limitations": "Yes, the authors adequately addressed the limitations", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a novel approach, i.e., DETAIL to address the specific challenges of ICL attribution by leveraging the internal optimizer perspective of transformers.\n2. The method incorporates random matrix projection to reduce dimensionality, resulting in significant speedups (up to 10x) while maintaining effectiveness.\n3. DETAIL is shown to be effective across multiple tasks, including noisy demonstration detection, demonstration reordering, and curation, demonstrating its broad applicability.\n4. The paper provides extensive experiments on both custom transformers and large language models (LLMs) like Vicuna-7b and Llama-2-13b, validating the method's effectiveness.\n5. The authors demonstrate that DETAIL scores computed on white-box models can transfer to black-box models like GPT-3.5, enhancing its practical applicability.", "weaknesses": "1. Section 5.1 presents an evaluation on a custom transformer using the MNIST dataset. While this provides an initial demonstration of DETAIL's capabilities, the paper doesn't clearly justify why this evaluation is necessary given the subsequent experiments on large language models. It's not immediately apparent how insights from this simplified setting transfer to more complex LLMs, potentially making this section feel disconnected from the main contributions of the paper.\n2. Figure 3 shows that even the Llama-2-13b model achieves only 60-70% accuracy on the AG News dataset without any perturbation. This is significantly lower than the typical performance range of 85-96% reported in the literature for this dataset. Could the authors give more insight about the performance of in context learning on the AG News dataset?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to the review of reviewer smfw\n\nDear reviewer smfw,\n\nWe would like to thank you for the review and for highlighting the novelty of our approach.\n\nWe would like to address your concern as follows.\n\n### 1. Connection between the MNIST experiment and the LLM experiment\nWe wish to clarify that we provide the MNIST experiment mainly as a visualization to show how DETAIL works as we believe it is more intuitive to understand visual similarity than semantic similarity of text. We also wish to use the MNIST experiment to highlight that our approach applies to ICL with transformers in general although we focus on transformer-based language models. We appreciate your feedback on the disconnection between the two parts. We will make the transition smoother by highlighting the above clarification in our revision.\n\n### 2. Accuracy on AG News\nWe believe two main factors are causing the gap in test accuracy. First, we use only $10$ demonstrations for ICL while AG News typically requires more than $30$ demonstrations to reach SOTA performance (e.g., Figure 3 (c) of [1]). Second, we modify the label names in our experiments to make sure it does not carry the semantic meaning to enforce ICL behavior (lines 171-176). The original label names (Worlds/Sports/Business/SciTech) make it easier for the language model to understand the task, leading to higher accuracy. The performance with altered label can also be observed in Figure 3 (c) of [1] (plotted line with legend $\\tau_{\\text{RA}}-ICL$) and is similar to the accuracy reported in our work.\n\nOnce again, we would like to thank you for taking the time to review. We hope that our response sufficiently addresses your concerns.\n\n\n[1] Jiaoda Li et. al., \"What Do Language Models Learn in Context? The Structured Task Hypothesis\", ACL 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for taking the time to provide constructive reviews and insightful suggestions. We have addressed the concerns in the respective rebuttal sections. As a general response, we would like to highlight that the main contribution of this work is to provide an interpretable attribution for ICL demonstrations on transformers that is also computationally efficient. Our experiments demonstrate DETAIL's effectiveness in attribution as well as in applications such as curation and re-ordering.

We have attached a PDF below including additional experiments on the re-ordering task. For additional experiments that consider BERT-related methods, please refer to our response to reviewer EuFx.

We are more than happy to clarify any additional questions regarding our rebuttal during the author-reviewer discussion period.

Meta-review: The paper uses influence function-based method for in-context learning. All reviewers agree that the submission can be accepted. The method is simple yet effective. The results are positive and can be transferred from white-box models to other black-box models.
Decision: Accept (poster)