Enhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspective
Authors: Xinhao Yao
Xiaolin Hu
Shenzhi Yang
Yong Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: Large language models
In-Context Learning
SVD
Theoretical generalization bounds
Primary area: natural_language_processing
TL;DR: We show SVD-based weight pruning boosts in-context learning in large language models, proved by theoretical analysis, and propose an effective, intuitive algorithm for downstream tasks.

Abstract: Pre-trained large language models (LLMs) based on Transformer have demonstrated striking in-context learning (ICL) abilities. With a few demonstration input-label pairs, they can predict the label for an unseen input without any parameter updates. In this paper, we show an exciting  phenomenon that SVD-based weight pruning can enhance ICL performance, and more surprising, pruning weights in deep layers often results in more stable performance improvements than in shallow layers. However, the underlying mechanism of those findings still remains an open question. To reveal those findings, we conduct an in-depth theoretical analysis by presenting the implicit gradient descent (GD) trajectories of ICL and giving the mutual information based generalization bounds of ICL via full implicit GD trajectories. This helps us reasonably explain the surprising experimental findings.  Besides, based on all our experimental and theoretical insights, we intuitively propose a simple, model-compression and derivative-free algorithm for downstream tasks in enhancing ICL inference. Experiments on benchmark datasets and open source LLMs display the method effectiveness.

Enhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspective
Xinhao Yao', Xiaolin Hu', Shenzhi Yang', Yong Liu!*
Renmin University of China, Beijing, China
(yaoxinhao021978, xiaolinhu, yangshenzhi2003, liuyonggsai Cruc. wew.eeruuuuuu.o.oeeeeeeoe
Abstract
Pre-trained large language models (LLMs) based on Transformer have demon- strated striking in-context learning (ICL) abilities. With a few demonstration input-label pairs, they can predict the label for an unseen input without any parame- ter updates. In this paper, we show an exciting phenomenon that SVD-based weight pruning can enhance ICL performance, and more surprising, pruning weights in deep layers often results more stable performance improvements than in shallow layers. However, the underlying mechanism of those findings still remains open question. To reveal those findings, we conduct an in-depth theoretical analysis by presenting the implicit gradient descent (GD) trajectories of ICL and giving the mutual information based generalization bounds ICL via full implicit GD trajectories. This helps reasonably explain the surprising experimental findings. Besides, based on all our experimental and theoretical insights, we intuitively pro- pose simple, tasks in enhancing ICL inference. Experiments on benchmark datasets and open source LLMs display the method effectiveness?.
and derivative- free algorithm for downstream
Introduction
Recently, large language models (LLMs) based on the Transformer architecture [43] have emerged striking in-context learning (ICL) capabilities: Given prompt containing demonstration sample and test data, the model can make prediction for the test data and achieve excellent performance without any parameter updates [7, 26, 34, 6]. This leads considerable works that aim to shed light on [54, 21, 59. 3, 10, 16, 2, 46, 35].
In this paper, we show our surprising findings in ICL inference by experimentally analyzing the effect of singular value decomposition (SVD)-based pruning the model performance differ- ent depth layers. As demonstrated in Figure 1 (i) SVD-based weight pruning can enhance ICL performance. Across all cases, it is evident that SVD-based weight pruning can generally enhance model performance at various depth layers, compared to the baseline performance without weight pruning (indicated by the dashed lines); (ii) Pruning weights in deep layers often results in more stable performance improvements than in shallow layers. Specially, deep layers weight matrices be drastically reduced without much degradation in model performance, and may even get large improvements on it, while the model performance collapses after sharp reduction at the shallow layers (see Section 2 for details). A similar case Sharma et al. [38] notes that large portion of singular values can be removed from linear layers in large language models without affecting or even improving reasoning performance. However, the underlying mechanism of this phenomenon still remains a mystery, and this paper seeks to explore the issue from the following two aspects.
*Corresponding author. 2The code is available at
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

SST 266PT: 68GATTN
SET-261 AMM22-
SST-25LL AMA3-7B6ATT
$.003
oi 0de 0.95
54.
65
31
621
23
05
KTESCPT) 8EG
ITESCPT S4TTN
K'ESLLANA2 785MLP
ATESLLAM42 7854T
2:311
2.05
Aac PEc. 2012
042
E44.
de old E
cb e5 de obi Kil
3.0
655
505
0.55
535
0.41
2.01
-40-
COPA6-OPT) 68SNLP
COPABOPT E66ATTN
2.501
6.39
COPAGLLAMAZ 786MLP
CORSSLLAMG2 78GATN
0.50
Dearts
R6680-1 195
Aaba0-1 C35
Figure 1: The effect of weight pruming across different layer types. The figure shows the phenomenon observed on the benchmark datasets (SST-2, RTE, COPA) and open source LLMs (GPT-J-6B and LLAMA2-7B). Each sub-figure corresponds only to the indicated type of dataset, model and module. Notice that this figure mainly focuses on exhibiting the impact of weight pruning to the first two and the last two layers of the model and different colors used to distinguish between these layers. The dashed line represents the pretrained model performance without SVD. We operate on the whole of MLP ATTN and specifically marked the points of highest performance. The amount of weight pruning is severe, for instance, the highest model performance sometimes occurs at clipping rate of 0.995. This is about 99.5% of the matrix's original rank. For the definitions of "deep" and "shallow", please refer to Appendix B.2.
Why this phenomenon? We conduct an in-depth theoretical analysis to explain the findings. To be more specific, we first analyse the ICL form with Transformer from the perspective of ICL as implicit gradient descent fine-tuning, and present the implicit gradient descent trajectories of ICL in Theorem 1 in Section 3.2. Afterwards, we use the information-then theeoretccc approaches [55, 32, 51] to give the generalization bounds of ICL via full implicit GD trajectories in Theorem 2 in Section 3.3, explaining (Q1) why SVD-based weight pruning can enhance ICL performance? and (Q2) why do deep and shallow layers exhibit different behaviors when their weight matrices are drastically reduced?
How to better use this phenomenon (Q3)? Based on all our experimental and theoretical insights, we intuitively propose a simple, derivative-free and model-compression algorithm in Section 3.4 for downstream tasks in enhancing ICL inference, providing developers and researchers with effective approach for handling downstream tasks. Experimental results for our algorithm on benchmark datasets [39, 58, 8, 12, 5, 11, 18] and open source LLMs [48, 42] verify that the method can visibly influence performance across different language understanding tasks in ICL inference.
Our primary objective is to establish general theoretical framework that uncovers the underlying mechanism behind the phenomenon that SVD-based weight pruning can enhance ICL performance. Based on our theoretical insights, one can design new ICL algorithms. Accordingly, we did not directly compare our approach with other pruning methods. The algorithm Section 3.4 presented solely to illustrate how theoretical analysis can guide experimental procedures effectively.

---

1.1 Related Works
Model compression. In recent years, there has been growing theoretical and experimental evidence that models can be significantly pruned with very little drop in accuracy. thereby significantly reducing their storage requirements. To name a few, Frankle and Carbin [15] indicate that neural networks can typically have over 90% of their weights eliminated with little no loss in performance. Analogous results have been demonstrated in both feed-forward and convolutional networks used for image classification [24, 49, 22, 37J. More specifically, Sharma al. [38] present that careful pruning done at specific layers of Transformer models can produce significant boosts in performance on some tasks. The discovery of this phenomenon heightens interest in the connection between generalization and pree-paramettrizationn [56, 57], it also spurs research into developing pruning strategies that facilitate efficient model inference [30]. Additionally, the works [19, 40] find that performance remains nearly unchanged until significant portion (up to half) of the layers are removed.
In-context learning and gradient descent. In order to better understand the ICL capabilities, considerabe works try to understand ICL capabilities from the perspective of gradient descent. Irie al. [21] and Dai et al. [10] explain ICL implicit fine-tuning by figuring out dual form of gradient descent for linear Transformer attention. However, the linear attention setting is less commonly used than Softmax attention in the LLMs and the details of gradient descent such as the choice of loss function and training data have not been clearly defined. Therefore, by using weight construction, von Oswald et al. [46] show the equivalence of linear self-attention mechanism and gradient descent linear regression task and Akyiurek et al. [3] prove that based on gradient descent and closed-for ridge regression, Transformers can learn linear models as learning algorithms. Further without using weight construction, Ren and Liu [35] connect Softmax attention with kernels and then give a novel interpretation that ICI. with Transformer really equivalent to a contrastive learning pattern.
coormationn-heoretii generalization bounds. The iiformatioonthhorelee generalization bounds have been developed to analyze the expected generalization error of a learning algorithm. Given that they are dependent of distribution and algorithm, they are ideal tools to study the generalization behaviour of models performed with specific algorithm. Russo and Zou [36] and Xu and Raginsky [55] first propose the Mutual information (MI) based bounds, which are then strengthened by additional techniques [4, 31, 20, 51]. Specifically, Negrea al. [31] derive MI-based bounds by developing PAC-Bayes-iide like bounding technique and Wang and Mao [51] develop generalization bounds for SGD via constructing an auxiliary iterative noisy process.
2 SVD-Based Weight Pruning can Enhance ICL
This section shows our surprising findings in ICL inference by performing motivating analysis of three benchmark datasets in conjunction with two open source LLMs, the details are as follows. We choose GPT-J (6B,2 ,22 layers) [48] and LLAMA2 (7B,32 layers) [42] as primary models for investigation, owing to their robust ICL performance and moderate model size, which align with our hardware resource. The attention (ATTN) layers are made up of key, query, value, out matrices both in GPT---6B and LLAAMA-7B. The mip (MLP) layers in GPT-J-6B consist of input, output matrices, while the MLP layers in LLAMA2-7B are made up of up. gate, down matrices. For datasets, SST-2 [39] for sentiment analysis, RTE [5] for textual entailment and COPA [18] for causal inference. Details regarding datasets and how they were used are shown in Appendix C. and C.3.
Specially, we use the optimal rank-r approximation mentioned later as SVD-based weight pruning method to show the effect of weight pruning across different layer and module types in Section 2.1, then we further analyze the effect of the ICL shot numbers on in Section 2.2.
The optimal rank-r approximation and SVD. Given matrix W E Rm and constant r S min (m, n), r E N. Eckart- Younrr Morsky theorem [14] provides an optimal solution Wr(r) = U. E,, V Urr, V,, are matrices containing the singular vectors corresponding to the largest
singular values [a][ Let = min E (0,1) 1) be the clipping rate.
2.1 Effect of Weight Pruning across Different Layer and Module Types
We plot the results of applying various amounts of clipping rate & to each module matrices in the Transformer architecture on the corresponding evaluation index for each dataset, depicted in Figure
3

---

1. These plots are grouped, such that each sub- sub-figure corresponds only to the indicated type of dataset, model and module. Notice that this investigation mainly focuses on assessing the impact of weight pruning to the first two and the last two layers of the model to further clarify the impact of layer depth on the model's performance and different colors are used to distinguish between these layers.
All sub-figures clearly show an interesting phenomenon about these models in ICL inference: SVD- based weight pruning can enhance ICL performance in both shallow and deep layers across different module types. More surprising, deep layers weight matrices can be drastically reduced without much degradation in model performance, and may even get large improvements on it. This suggests that pruning weights in deep layers can effectively reduce model complexity while maintaining or enhancing model performance. And the model performance collapses to 0.5 (the expectation of random guess in the binary task) after sharp reduction at the shallow layers by contrast, indicating a higher sensitivity of the model to changes in shallow layers.
Based on the surprising findings in ICL inference mentioned above, three questions are certain to
arise: (Q1) Why SVD-based weight pruning can enhance ICL performance? (Q2) Why do deep and shallow layers exhibit different behaviors when their weight matrices are drastically reduced? (Q3) How can better use the phenomena about ICL in downstream tasks? We will address the first two questions theoretically in Section 3.3 and give a heuristic algorithm in Section 3.4 to answer the last.
CPT 6B S...P6.10 chot
GPT) HBS.TTNGIS shot
LLAMAZ 78SMLP610 shot
11AM42-78 D-
3.3055
12107
0.1
615
016 0.0 0355
05 BD
GBT-)ERGME P56--hot
1LAMAO-7B6 98.7
0.8
030-
(V1)
Cipping Mat133)
Cipping F-la
GPT 686MLPP5 shot
(LAMA2-766M
(LAMA2 756ATN66 Sh0t
5.11
19971
18300
0304
02 ES0
2/00
03
Figure 2: The effect of different ICL shot numbers is not uniform. Here we show the effect of different ICL shot numbers on the phenomenon mentioned in Section 2.1 as studied on the SST-2 dataset. Each row represents the results of the same shot numbers in different layers and modules, and each column represents the results of the different shot numbers in different layers of the same module. We also specifically marked the points of highest performance.
2.2 Effect of Different ICL Shot Numbers
Given that ICL can achieve higher performance with more demonstrations [23, 1]. we further analyze the non-ignorable effect of different ICL shot numbers. To control for other influencing factors, we focus on SST-2 dataset [39] and retain the same test set for single random seed as Section 2.

---

As shown in Figure 2. we compare the settings of three different ICL shot numbers: 0, 4 and 10. Following this, we analyze how the phenomenon changes across different shot numbers.
Firstly, we note that without weight pruning, the performance of the model improves with an increase in the number of ICL shots. Which is consistent with prior works. Besides, for every shot numher, phenomenon consistent with what is described in Section 2.1 is observed: SVD-based weight pruning can enhance ICL performance, pruning deep layer weight is more stable than pruning shallow weight. Last but not the least, Figure 2 also demonstrates roughly that with decrease in the number of shots, the rate of performance collapse in the model slows down after a sharp reduction at the shallow layer. Intuitively, this is because LLMs exhibit a shift in focus in the ICL setting, which results in a reduced scope of the output space. The more shots there are, the more pronounced the shift becomes, and this also leads to a faster collapse. We also theoretically discuss it in Section 3.2 and 3.3 (Remark 6).
3 Theoretical Analysis Results
In this section, we first describe the core components of our study by reviewing some basic notations in Section 3.1, and present the implicit gradient descent trajectories of ICL in Section 3.2. A Afterwards, we give a mutual information based generalization bounds of ICL via full implicit GD trajectories in Section 3.3. Based on all our experimental and theoretical insights, we intuitively propose a derivative-free and effective method for downstream tasks in enhancing ICL inference in Section 3.4. Complete proofs can be found in the Appendix. For case of qualitative analysis, our theoretical analysis is mainly focuses linear attention setting. We discuss the standard Softmax attention setting in Appendix A.2 and forward (MLP) layers in Appendix A.3.
3.1 Preliminaries
We let H be the instance space and f be an unknown distribution H. specifying random variable h. In ICL setting, the model accepts a sequence of input H where H, [h1 he.... hy] represents the demonstration sample and hy+1 is the test data. In the information- -tteorrttt analysis framework, we let W Rd be the space of hypotheses related to the model, and Transformer performs an implicit stochastic learning algorithm A which takes the demonstration sample H. its input and outputs a hypothesis W E W according to some conditional distribution QWIH, Similar to previous works [55, 51], we give the definition of expected generalization error.
drawn i.i.d. from μ
Expected generalization error. Given a loss function E WXH + Rt, where E(w, h) measures the "unfitmess" or "crror" of any h H. with respect to hypothesis E W. We take E as continuous function and assume that is diffcrentiable almost everywhere with respect to w. The goal of learning is to find hypothesis that minimizes the population risk, and for any
W, the population risk is defined (w) & En [w, h)]. However, since μ via the sample H, only be partially observed, we instead turn to use the empirical risk, defined as LH, (w) 2 A Σn f(w, h.). Then the expected generalization error of A is defined as
error A I. - LH, (W)),
where the expectation is taken over (He, W) NN & QWIH.
In-context learning with Transformers By prompt design, most context semantic understanding tasks can be unified into classification tasks. Simplify the form of each token in H to h; = [x, y:l. where [x] V e de and [y,] E Rdduu are encoded input text and corresponding labels respectively. The test token has the form hN+1 = [xx+1, mask), where mask is the label needed to predict, and usually set as its initialization. Therefore, the form of attention with residual connection" as follows:
((WxH))WW WoH)
A H Wy HM Softmax
(1)
V Viscate
3 Please refer to Appendix B.1 for an explanation on how Eq.(1) and (2) can utilize same mask. In real-world scenarios, the residual connection module in Transformers is indispensable.
5

---

where Wv,Wk. Woe Rd + din) x (din are projection matrix, and the mask matrix M INXN 0
is included in the attention. Specifically, hN+1 can be formulated as
hx+1 = hN+1 + HvHM Softmax
(2)
Viscate
and for ease of qualitative analysis, (2.(2) can be approximated as relaxed linear attention mecha- nism by removing the Softmax operation and scale factor:
h = hw-1 + WyHM(WWar (ecHH)WWWWwer WahN+1-
(3)
3.2 The Implicit Gradient Descent Trajectories of ICL
To begin with, we present the implicit gradient descent of ICL, inspired by [21, 10, 35]. These works describe how ICL with attention can be connected to a meta-optimizer which produces implicit gradient. The following lemma demonstrates the result.
Lemma 1 (The Implicit Gradient Descent ICL in a Single Linear Attention Layer). Consider Transformer consists of a single linear layer attention with residual connection, parameterized by Wy. Wxx k. WQ as in (3)( Same to Section 3.1, let H = [H,, hy+1) be the input, where H. = [h h2. hy] represents the demonstration sample and b is the test data. And let bN+11 be the single layer output. Then, it holds that
N
AW ict = H,,,,W&HH, W
Σ
Wyh,&Wkh We:
h hN+1 hN+1 + AW Wiahh++1.
where W is regarded as the meta- meta-gradien gradient of ICL, which is used to generate the implicit gradient matrix AWis iet to act on the final feature output. See Appendix A. for a proof.
Remark 1. worth noting that rank( AWaa) :a) S rank(H, H.) S N by rank relations matrix multiplication, indicating that this is low-rank operation. This explains why the effect of different ICL shot numbers is not uniform in Section 2.2. Details of the discussion in standard Softmax attention setting can be found in Appendix A.2. Plus, a significant body of work has discovered that the ICL capabilities of models are not robust to the order of ICL sample. For instance, Lu et al. [28] observed that large language models (LLMs) are sensitive to the sequence of ICL examples, and Liu tend
et al. [27] reported that the context-bass based ICL performance exhibits a U-shaped curve-mode to perform better with information that appears at the beginning or the end of the input context. This aligns with observations from actual model training where the sequence of training samples affects outcomes especially when training with a batch size 1.
According to the above, we further analyze the implicit gradient descent trajectories of ICL and provide the following Theorem. We will denote A W iel by A W, when there is no ambiguity, where t represents t-th layer.
Theorem 1 (The Implicit Gradient Descent Trajectories of ICL). Consider Transformer of L linear attention blocks with residual connection. parameterized by [Wt [ W 14. [W
stack
Denote [xt]N+ N11 that
as the output of the t-th layer, [h]] as the initial input. Then for t [L], it holds
G. = -:=) AW W + EG,)
1=1
bly41 h +
G.bo -h241 + W.h N+1
J=1
where AW, EE Witt & Wkh W Wo O, W G, and G1
AW, 1- See Appendix 4.4 for a proof.
Remark 2. Note that the exclusion of Transformer weight (W I WW) wtl))
implies that
G. is only dependent W. t-1 and H,, this is consistent with gradient descent in terms of relevance.

---

3.3 Generalization Bounds of ICL via Full Implicit GD Trajectories
In this part, for simplicity of representation, we flatten the weight matrix into a vector form (vec(W))= Wi E R vec G,) = G. E I and conduct our analysis within the weight and implicit gradient space of hypotheses W as detailed in Section 3.1. Notably, Ahn et al. [2] observe that, with the optimal parameters, single layer linear Transformer implements single step of preconditioned gradient descent and the preconditioning matrix not only adapts to the distribution of input data but also to the variance caused by data inadequacy. Garg et al. [16] show empirically that the trained Transformer able to in-context learn the class of linear functions with respect to the prompt distribution, performing comparably to the optimal least squares estimator for any number of in-context examples considered, as exhibited in Figure 5a.
In addition, we also make a discussion on the noise of the implicit gradient in Section C.4, our analytical discussion indicates that the implicit gradients produced by Transformers in practical applications are noisy due to factors such as the extent of model pre-training and data characteristics (e.g., ICI. shot number). We first present the assumption used in this subsection.
Let G. & N E Vł; be the best implicit gradient of ICL that the model can produce, G, & ! E Ve, be the implicit gradient of ICL generated by the model in practical applications. N is the threshold and b is the the shot number in the actual input defined in Section C.4. And V2 - G. GG-G. G, is b(--1) (N Σ V - -G, is the implicit
the gradient noise caused by shot number, C, _ N-b gradient noise covariance. Similar to Wang and Mao [51]'s assumption in SGD, we approximate Vi up to its second moment.
Assumption 1. Assume the implicit gradient noise follows a Gaussian distribution, i.e., V₁ (0 Ct), then in ICL implicit gradient descent trajectories,
W = Wi-1
Wi-1 -/Gr + n/C,//2. No.
where No- ~ Io, is the standard Gaussian".
Remark 3. Regarding the validation of this assumption, empirical evidence from works [53, 25], suggests that SGD and E4)( can achieve the similar testing performance. Additionally, we refer readers to some recent works [21, 16, 3, 10, 2, 46, 35], where the authors empirically verify that in-context learning of Trunsformer can achieve the similar testing performance to SGD. Together suggesting that studying (4) is arguably sufficient to understand generalization properties of ICL.
Theorem indicates that the initial parameter Wo = 0, which is independent of all other random variables. And an L-layer Transformer does implicit GD of ICL stops after L updates, outputting Wr. as the implicit learned parameter. Our main results are mutual information based expected generalization error bounds of ICL., as presented in Theorem 2.
Theorem 2 (The Generalization Bounds of ICL via Full Implicit Gradient Descent Trajectories). Under the conditions of Theorem and Assumption I, assume the implicit gradient noise covariance Ct is positive-defime matrix, the loss Eww, h) is R-subGaussian for any W E Rd, then
1 1 1+ G, 2 + tr(C.)
R
L
Aw 2
error S
WW W.-.
d dog
Cr
d
where vec(G,) E Rd, AW, & C.N @ W' and G. - AW Wo E Gj) = AW W.-1). tr(.} denotes the trace of a matrix, II-I denotes the Frobenius norm of a matrix and TEA is the conditional expectation. Proof details in Appendix A.5.
Remark 4 (Deal with Q1). Theorem 2 indicates that one can control the generalization performance of ICL via controlling the implicit gradient norm along the entire ICL implicit GD trajectories. Specifically, modulating the norm of [G.]t [&W z) may enhance performance when utilizing ICL. Note that controlling implicit gradient norm can also control the magnitude of the trace of implicit
*Consider the continuous SDE: dW = LH, dW) + InC(W)) where C(W) is the gradient noise covariance W and 0 is Wiener process. We can view Eq. (4) as discretization of the SDE.
7

---

gradient noise covariance. This elucidates why weight pruming through SVD, even if it only alters single weight matrix, can confer advantages on the performance of Transformers in ICL inference. We will present example below demonstrating how weight pruning can affect the norm of G. AWr 6 thereby influencing the expected generalization error. Additional examples provided in Appendix A.6. This is also why, as illustrated in Figure I, the highest model performance sometimes occurs at clipping rate of 0.995. Furthermore, this could elucidate the utility of normalization in Transformers. However, weight pruning also impacts the expressive power, implying that increased pruning does not invariably lead to better outcomes. This clarifies why the highest model performance may also occur at clipping rates lower than 0.995.
Example 1 (Prune Wo. WK. Wv). Consider A -EEN - & W " W
when = k. And we primarily consider the changes in a upper bound written as UB(IAW&ll & EN [Wh
W/&ll Let r represents the remained rank and 8 represents the potential noise consisting of parts with small singular values, that is, Wy WW, + gv = Uy xY, (V + +ōv, the same operation is applied W and Wi as well. Then we have
| 112 = =I(Wt, + + -- 1
=| h 11 2(W
v. h (F (6rh;-)) + | 112 W h 11
| h 1 112 16v h
where W V. = UV and ov = U. (VK),, and U.r, U.: are orthometric (properties SVD), and further have
UB([AW
w h @ win l
2
W h 2 w b w
W h
w 11
- B -B([AAW()))) (**)
2.
where Eq. (*) is by
||vec(A) & vec(B)ll
ΣΣ
vBc)), max (vec(A)) @ vec(B))
and Eq. (#8) is by ||l|r of for any matrix A. B and |P|| 2 |P(T]]]
B((IAW&(()]1 the upper bound on lAW sll after using SVD.
Remark (Deal with Q2). It is notable that G. is highly correlated with the sequence (AW1,-.. AW.). More precisely, adjusting the weight matrix of the k-th layer, will invariably impact the norm of [AW AW.)ss further influence [Gr] When we calibrate the weight matrix of the k-th layer, the span of affected weight updates AW, encompasses L L- k + matrices, this indicates that the deeper the layer of the adjusted parameters is, the fewer the number of G, affected. It also suggests that tweaks to the deep layers yield a more steadfast influence on the global norm of [G.ll. thereby exerting a steadier infiuence over generalization performance. The enhanced stability observed in adjusting deeper layers, as demonstrated in our experimental findings in Section 2, can be theoretically explained based on the analysis presented above.
Remark 6 (How should Theorem 2 be interpreted?). Expected generalization error (Theorem 2) population risk (L,)) empirical risk (LH,).
). More specifically, on the one hand, Theorem 2 shows
For any vector and b. rank @ b) value. Combined with ||||| = v (P) and singular value is nonnegative, we can get ||a @ bliF VE, (a & b)- = max, (a.(a @ b)], therefore, the unique non-zero singular value will decrease after per- forming SVD and/or b.
so the matrix (a @ b) only has one non-zero singular
8

---

that clipping weights controls the F-norm of the implicit gradient ////f// which helps reduce the expected generalization error. On the other hand, we can evaluate the empirical risk (T,,) by assessing the model's performance on the validation set. If the generalization error is known, is possible to estimate the population risk (Ieg). Therefore, the most challenging aspect is addressing the generalization error. Furthermore, we will illustrate the interpretation of Theorem 2 from the perspectives of pruning methods and the ICL shot numbers. (i) Adjusting the F-norm of (I W W [ could enhance performance when utilizing ICL, suggesting that other weight-based pruning methods may also be effective. For example, magnitude-
pruning [52] directly controls the matrix F-norm. Certainly, there also some reyer-based pruning methods (e.g.. drop-layer method [19]). we discuss in detail in Appendix B.6. (ii) Reducing the ICL shot numbers to N' can indeed lead to more robust outcomes specifically in the "SVD weight pruning" operation, rather than in the model's overall performance. Experimentally, in Section 2.2, analyze the effect of different ICL shot numbers. As shown in Figure 2, with a decrease in the number of shots, the rate of performance collapse in the model slows down after a sharp reduction at the shallow layer. Theoretically, (W(N) - AW(N)= = (Ei=N'ta W @ Wgh:) W a, suggesting that the implicit gradient for N is expected to be more sensitive than that for N'.
3.4 Applications of Our Theoretical Understanding
In this portion, we deal with Q3. We further explore the relationship between model performance improvement and SVD, as illustrated by simple case study.
Case 1. Assuming that W has two unit eigenvectors X1 X2, based on the properties of eigenvectors:
Wx1 = X1 Wx2 = X2
and for vector b on the hyperplane, it can be decomposed into a linear combination of two eigenvectors:
12 _xx))))))) Wx b-11x++ (
w
where 4 are coefficients and X1, A2 are cigenvalues. Notice that i >> >2, when point b moves in the direction of X1, the value 1 changes but the solution set changes insignificantly. Conversely, if it moves in the direction of x2. the value of l2 changes and the solution set x changes dramatically, this is ill- ill-conditioned problem whose solution is highly biased under small perturbations.
Given that SVD can be considered to play a role in noise reduction in ICL inference, and that this noise can cause significant disturbances to model output. We thus introduce the matrix illccondtioonedness. which can be measured by the Matrix condition number defined as follows.
Matrix condition number Cond (W) = || W- Specifically, the condition number Cond( Zaus when p = 2, where mazz mii denote the maximum and minimum singular Jmin
values respectively. Generally, for matrices of the same order, the higher the condition number of matrix is, the greater its ill-conditionedness will be.
2.AITH caaditaa navber) LMMMU
Lax
XCB (01)
375-
22-
A0ik Gard
04-
18
11
28
74
Figure 3: 2-norm condition number of GPTJ-6B&LLAMA2-15 The condition numbers for MLP are significantly lower than those for ATTN. In deeper layers, condition numbers tend to be higher. These matrices ill-conditioned for they satisfy o Tmar >> min-
Now consider linear system Wx- b. where solution set x highly sensitive to the coefficients of W and b. In such cases, the system of equations is termed ill-conditioned. Matrix condition number is an option, any indicator that can guide the control of norms is potential option.

---

Then we analyze the matrix condition number of the models as shown in Figure 3. We observe that the condition number of deeper layers (especially the last layers) in the model is generally higher, indicating that the condition number may serve as reference for adjusting the model.
Based on all exciting insights, we find it intuitive to design ICL improvements based on them, especially in downstream tasks. We propose a method where we first select layers with the top-k largest condition numbers and then identify the layer with the largest number among these. We perform a greedy search for the optimal clipping rate 5 on the validation set and subsequently evaluate it on the test set. This procedure is reported in Algorithm
Appendix C.2.
Experimental setup. We conduct experiments for Algorithm 1 on widely adopted benchmark datasets, including SST-2 [39], AGNEWS [58], EMOC [8]. MRPC [12], RTE [5], CB [11]. COPA [18]. Please refer to Appendix C. and C.3 for more detailed dataset and prompt settings. And for models, the same models in Section 2.
Experimental results. As Figure shows, the results indicate that Algorithm can visibly influence performance across different language understanding tasks in ICL inference. Specially, the effective- ness of Algorithm enhancing performance varies not only by the model but also significantly by the task, indicating potential task-specific and model-specific threshold for the benefits derived from algorithmic enhancements. More notably, Algorithm is a derivative-free optimization method and compresses the model to some extent, providing developers and researchers with effective approach for handling downstream tasks. We also invite readers to refer to Appendices B.3, B and B.5 for discussions What would be the effect of pruning only single module?, Why optimal clipping rate & varies? and What would happen if we apply the same clipping rate to other datasets?
AccurscyiF3 (0
54
SST
AGNEWS
EMOC
MRPC
RTE
C8
COPA
ICL (GI1A
KLL.LAMMA--7)
ICL Algortthmm(LLLa... (LLAMA2-77)
Figure The Model Performance on Test set by different tasks. The results are obtained by comparing four scenarios: ICL (GPT-J-6B), ICL+Algorithml (GPT---6B), ICL (LLAMA2-7B) and ICL+Algorithml (LLAMA2-7B). ICL+AIgorithm demonstrates superior results over only ICL on different tasks. See Appendix C.5 for detailed numbers.
4 Conclusion and Limitation
In this paper, we show our surprising findings in ICL. inference: SVD-based weight pruning can enhance ICL performance both in shallow and deep layers across different module types, and pruning weights in deep layers often results in more stable performance improvements than in shallow layers. We conduct an in-depth theoretical analysis and explanation of these findings. Specifically, we first present the implicit gradient descent trajectories of ICL, afterwards, we give mutual information based generalization bounds of ICL via full implicit GD trajectories. Based on all our experimental and theoretical insights, we intuitively propose a derivative-fret and effective method for downstream tasks in enhancing ICL inference. However, further studies are required on (i) how to extend our generalization theory to more standard Transformer architecture, (ii) do the results about ICI. hold true for tasks beyond natural language processing and (iii) how to minimize the cost of searching the optimal clipping rate. Those will deepen understanding of the ICL capabilities.
10

---

Acknowledgments
This research was supported by National Natural Science Foundation of China 684.67772777 No.6207623), Beijing Natural Science Foundation (No.4222022) CCF-ALIMAMA TECH Kan- garoo SanndNo.CCFAALIM...
OF 2024008), and Huawei-Renmin University joint program on Information Retrieval. We also acknowledge the support provided by the fund for building worldclass universities (disciplines) of Renmin University of China and by the funds from Beijing Key Labo- ratory of Big Data Management and Analysis Methods, Gaoling School of Artificial Intelligence, Renmin University of China, from Engineering Research Center of Next-Generation Intelligent Search and Recommendation, Ministry of Education, from Intelligent Social Governance Interdisci- plinary Platform, Major Innovation & Planning Interdisciplinary Platform for the "DoubleFirst Class" Initiative, Renmin University of China, from Public Policy and Decision-making Research Lab of Renmin University of China, and from Public Computing Cloud, Renmin University of China.
References
[1] Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning. arXiv preprint arXiv:2404.1100 1018, 2024.
[2] Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn implement preconditioned gradient descent for in-context learning. In Advances in Neural Information Processing Systems (37th edition), pages 45614-45551 2023.
[3] Ekin Akyiurek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In International Conference on Learning Representations, 2023.
[4] Amir Asadi, Emmanuel Abbe, and Sergio Verdú, Chaining mutual information and tightening general- ization bounds. In Advances in Neural Information Processing Systems (32th edition). pages 7254,, 2018.
[5] Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In Text Analysis Conference, 2009.
[6] Sidncy Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler. Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An open-source autoregressive language model. In Proceedings of BigScience Episode #5 - Workshop on Challenges & Perspectives in Creating Large Language Models, pages 95-136, 2022.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (34th edition), pages 1877-1901, 2020.
[8] Ankush Chatterjee, Kedhar Nath Narahari, Meghana Joshi, and Puncet Agrawal. Scmeval-2019 task 3: Emocontext contextual emotion detection in text. In Proceedings of International Workshop on Semantic Evaluation (13th edition), pages 39-48, 2019.
[9] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins. Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Representations, 2021.
[10] Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhitang Sui, and Furu Wei. Why gpt leam in-context? language models secretly perform gradient descent as meta- meta-optimizers. Findings of the Association for Computational Linguistics, pages 4005-4019, 2023.
[11] Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection naturally occurring discourse. In Sinn und Bedeutung, volume 23, pages 107-124, 2019.
[12] William B. Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of International Workshop on Poraphrasing (3th edition), 2005.
[13] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang, Glm: General language model pretraining with autoregressive blank infilling, arXiv preprint 210:2103.10360 2022.
11

---

[14] Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika. 1 (3):211-218, 1936.
[15] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Training pruned neural networks. In International Conference on Learning Representations. 2018.
[16] Shivam Garg, Dimitris Tsipras, Perey Liang, and Gregory Valiant. What can transformers lear in-context? case study of simple function classes. In Advances in Neural Information Processing Systems (36th edition), pages 30583-30598, 2022.
[17] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forwart layers are key-valuc memories. In Proceedings of Conference Empirical Methods in Natural Language Processing (26th edition). pages 5484-5495, 2021.
[18] Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. SemEval-2001 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. *SEM 2012: The First Joint Conference on Lexical and Computational Semantics- - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 394-394 0984--99820 2012.
[19] Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17881 2024.
[20] Mahdi Haghifam, Jeffrey Negrea, Ashish Khisti, Daniel M Roy, and Gintare Karolina Dziugaite. Sharpened generalization bounds based conditional mutual information and an application to noisy. iterative algorithms. In Advonces Neural Information Processing Systems (34th edition), pages 9925-9935, 2020.
[21] Kazuki Irie. Róbert Csordás, and Jürgen Schmidhuber. The dual form of neural networks revisited: Connecting test time predictions training patterns via spotlights of attention. In International Conference on Machine Learning, pages 9639-9659 2022.
[22] Mikhail Khodak, Neil Tenenholtz, Lester Mackey, and Nicolò Fusi. Initialization and regularization of factorized neural layers. In International Conference on Learning Representations, 2021.
[23] Mukai Li, Shansan Gong, Jiangtao Feng, Yiheng Xu, Jun Zhang, Zhiyong Wu, and Lingpeng Kong. In-context learning with many demonstration examples. arXiv preprint arXiv:2302.0493) 2023.
[24] Yawei Li, Shuhang Gu, Lue Van Gool, and Radu Timofte. Learning filter basis for convolutional neural network compression. In Proceedings of the IEEE International Conference on Computer Vision, 2019.
[25] Zhiyuan Li, Sadhika Malladi, and Sanjeev Arora. On the validity of modeling sgd with stochastic differential equations (sdes). In Advances in Neural Information Processing Systems (34th edition), pages 12712-12725, 2021.
[26] Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. Jurassie-1: 1: Technical details and evaluation. Technical report, A121 Labs, 2021.
[27] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. In Transactions of the Association for Computational Linguistics (12th edition), pages 157-173, 2024.
[28] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming fewshot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8086-8098, 2022.
[29] J. Mercer. Functions of positive and negative type, and their connection with the theory of integral equations. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 209:415-446, 1909. URL http:// http://wwwjjjo www.wwtto.....oooo..wpw.wwwwwww. jator. . arg/stable, e/90443.
[30] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations, 2017.
[31] Jeffrey Negrea, Mahdi Haghifam, Gintare Karolina Dziugaite, Ashish Khisti, and Daniel M Roy. cintmatioo-theorette generalization bounds for sgid via data-dependent estimates. In Advances in Neural Information Processing Systems (33th edition), pages 1013-11023, 2019.
[32] Yury Polyanskiy and Yihong Wu, Lecture notes on information theory. Lecture Notes for 6.441 (MIT). ECE 563 (UIUC), STAT 364 (Yale), 2019.
12

---

[33] Zihan Qiu, Zeyu Huang. Youcheng Huang, and Jie Fu. Empirical study on updating key-value memories in transformer feed-forward layers. In International Conference on Learning Representations, 2024.
[34] Jack W Rac, Sehastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslamides, Sarah Henderson, Roman Ring. Susannah Young, al. Sealing language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112. 1446, 2022.
[35] Ruifeng Ren and Yong Liu. In-context learning with transformer really equivalent to a contrastive learning pattern. arXiv preprint arXiv:2310. 13..23200,3 2023.
[36] Daniel Russo and James Zou, Controlling bias in adaptive data analysis using information theory. In Artificial Intelligence and Statistics, pages 1232-1240, 2016.
[37] Steffen Schotthöfer, Emanuele Zangrando, Jonas Kusch, Gianluca Ceruti, and Francesco Tudisco. Low- rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations. In Advances in Neural Information Processing Systems (36th edition), pages 20051-20063, 2022.
[38] Pratyusha Sharma, Jordan T Ash, and Dipendra Misra. The truth is in there: Improving reasoning in lan- guage models with leyer-selective rank reduction. In International Conference on Learning Representations, 2024.
[39] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, 2013.
[40] Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Llm pruning and distillation in practice: The minitron approach. arXiv preprint arXiv:2408. 1 11796, 2024.
[41] Yuandong Tian, Yiping Wang, Zhenyu Zhang, Beidi Chen, and Simon Du. Joma: Demystifying mul- tilayer transformers via joint dynamics of mip and attention. In International Conference on Learning Representations, 2024.
[42] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Amn Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Ilama: Open and efficient foundation language models. arXiv preprint ar0iv: 1122..19777 13971, 2023.
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems (31th edition), 2017.
[44] Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in transformer language model. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63-76. 2019. URL https:// org/ 20t///1944888
[45] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self- attention: Specialized heads do the heavy lifting, the rest can be pruned. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797-5808, 2019. URL https: // //acl //aclanthologyy arg/P19-- 1580.
[46] Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers leam in-context by gradient descent. In International Conference on Machine Learning, pages 35151-35174, 2023.
[47] Alex Wang. Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpor language understanding systems. In Advances in Neural Information Processing Systems (33th edition), 2019.
[48] Ben Wang and Aran Komatsuzaki, Gpt-j-6b: A billion parameter autoregressive language model. https: // thub. com/ httm//inngofoll//eeett
ormer- jax, May 2021.
[49] Hongyi Wang, Saurabh Agarwal, and Dimitris Papailiopoulos. Pufferfish: Communication--efiiicnn models at no extra cost. In Proceedings of Machine Learning and Systems 3, 2021.
[50] Lean Wang, Lei Li. Damai Dai, Deli Chen, Hao Zhou, Fandong Meng. Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in co-context learning. In Empirical Methods in Natural Language Processing, 2023.
13

---

[51] Ziqiao Wang and Yongyi Mao. Two facets of sde under an informationn-heerrliee lens: Generalization of sgd via training trajectories and via terminal states. arXiv preprint 1rX1v:2211.10691, 2022.
[52] Wei Wen, Chunpeng Wu, Yandan Wang. Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems (30th edition), 2016.
[53] Jingfeng Wu, Wenging Hu, Haoyi Xiong, Jun Huan, Vladimir Braverman, and Zhanxing Zhu. On the noisy gradient descent that generalizes sgd. In International Conference Machine Learning, 2020.
[54] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning implicit bayesian inference. In International Conference Learning Representations, 2021.
[55] Aolin Xu and Maxim Raginsky. theereiiion- analysis of generalization capability of learning algorithms. In Advonces Neural Information Processing Systems (31th edition), pages 2524-2533, 2017.
[56] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017.
[57] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-135 2021.
[58]
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for classification. In Advances in Neural Information Processing Systems (29th edition), pages 649-657, 2015.
[59] Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization. arXiv preprint arXiv:2305.194D0
2023.
14

---

A Omitted Proofs and Additional Results
A.1 Proof of Lemma
Proof. Given that numerous studies [21, 10, 35] describe ICL with attention can be connected to a meta-optimizer which produces implicit gradient, show the dual form between linear attention and gradient descent. First, consider a very simple linear layer,
F(x) Wx,
where W Induutdii is the projection matrix. Given training inputs [x.] E Rddi with their labels fy E Rodout and the loss function L with learning rate gradient descent process produces the corresponding back-propagaticn signals [el] E Rdout, where e; -1 y! C and
y, Wx. During time, we
trained linear layer,
F(Xtest) = W Wxtest (W + AW)Xtest Xtest = WXirett +
Σ
e @ x;
Xtest:
(5)
where & denotes the outer product according to the chain rule differentiation. On the other hand, this process can be associated with linear attention.
LinearAttmi K, VK' -- v.
- = E(ve K.)q :=1
(6)
where [k.]ff [vvll E Rdin represent key and value vectors respectively, forming the key and
value matrix K, V Rdinx in the attention mechanism. Based on (6), rewrite (5) as:
F(xtest) = W Xtest +
Σ
e & X;
Xtest
WXtest + Linear Alln E, X, Xtest).
Then let's back to the ICL process and approximate standard attention Eq.(2) as relaxed linear attention mechanism by removing the Softmax operation and scale factor:
W'Whha Fia((N+a) z h hw+1 + WyHM(WW K " WohN+t
hN+1 WyH, HH, T
W.Wll WohNH1
hN+++ + WohN+1 WyH. (W k kH.) H WWhh++
(8)
hN+1 + WohN+1 + Σ Wrh, & Wgh, Wohn-1
hN+1 + WohN+1 Tinear ( (WyH, W WRH,, Wohn+1)
hh++1 Wohx+1 + AW, iahh+1
Where Wo = 0, and Wy H, is regarded as the meta-gradient of ICL, which is used to generate the implicit gradient matrix A W je act on the final feature output.
A.2 Softmax Attention and ICL Implicit GD
Inspired by Ren and Liu [35], by connecting Softmax Attention with Kernels, we can interpret ICL as gradient descent process in contrast learning pattern. For simplicity, in Section 3.1, scale factors are removed as follows:
Softmax ( (Wx "WWWWWWW.... WW H) H.,,A, A, cxp(( HT)" W)
(9)
A D A., diag(A,, In).
where exp(-) is element-wise. Following Choromanski et al. [9] and Ren and Liu [35], a Softmax kernel function KSottmax R.out x Iddout
- R.+ defined as:
| |2
||y
K Sofimax (x.)) = exp(x""x) xx = exp
K gauss (x, y) exp
(10)
2
15

---

where (xa exp Mercer's theorem [29], there exists some mapping function & satisfying K Softmax (x,y) = ((x)* (ly). Thus, rewrite (9) as:
(-1x is the Gaussian kernel with variance o² 1. According to
Aa((,) = exp( (W h,, / WWhh))
Softmux W k kh, Woh,) = (W,h, (Wohj))
(11)
then, connect with (2) have,
hN+1= hy-1 W.HM HM Softmax ((W (W&H) " "eel)"WohNaa WohN+1)
Vdccale
hhy-1 W
HM WyHM Softmax ((W K H W (ohN+1)
1
hh--1
Wy
W (W,),, o( ((Wkhr+))"'(W "(WWhw++)
D
(12)
= hy-1 + W W H..H&oWWWLL W W H, H
D'
-h--1* D
1
Σ (WwvH.
00 (l (W...) d(WohN41)
= hN-1 + AW (WohN+1).
where D' 1x (oroWWkt W W WohN+1) (WWkhr d (Wohx+1) represents constant for the normalized attention scores. can be considered that the mapping function Φ. or rather the effect of the Softmax function, is to project the original features into nigher-dimmesiionll space to capture more profound features. Subsequently, learning and meta-optimization are conducted under this new feature space. It is also noteworthy that rank(AW set) I rank(H,) < N.
A.3 Feed-Forward and ICL Implicit GD
The LLMs based on the Transformer architecture not only contain an Attention mechanism, but also include feed-forward (MLP) layers. It can be estimated that the feed-forward layers account for approximately two-thirds of the parameters a Transformer model [45, 44]. Thus, MLP layers are essential for LLMs and there has been considerable work hypothesizing and analyzing the model's ford-fo layers. For example, Geva et al. [17]. Qiu et al. [33] show that nred-forward layers in Transformer based language models operate key-valus memories, Tian et al. [41] propose Joint MLP/Attention (JoMA) dynamics by integrating out the self-attention layers Transformers, analyzes joint training of MLP and self-attention layers, and qualitatively explains dynamics of multi-layer Transformers. Based on the inspiration above, we take MLP into consideration and rewrite Eq. (3) as:
1iN41 hN41 + MLP(WW, HMWWHWWWWW/... (AMMWWH))) &H) W Woh++1)
-hN+1* + W Woutrelu(We W.n in WyHM WohN+1);
where Wv. v. W W (ddmm)) (dout din)), W in E R (dimm) x(diml)) W out ((dout+-din) din (dim2) are projection matrices and relu(-) is the activation function. And we can further relax the activation function for ease of qualitative analysis:
hx hN+1 + W out W.D in WyHM(W, x H)) T WahN+1
H -hN+1 W MLP W. HM(W, W Woh
where WMLP E didott+din) x(dim1) which can be seen as dimensional adaptation. And then do the similar derivation that we do in Appendix A.I, we can get AW" = W WMLP EN Wyh, @ Wxh.) We.
16

---

A.4 Proof of Theorem
Proof. Similar to A.1, Pay attention to the test token when = N + 1,
h h A WihNN
h N h + AW2h
-h + AWih W + +AW2(1 AW1)
h N++ h + AWah
-h + AW, + +AW,(1 AW, h
+ +WW (1 + AW, A + AW1) hx41
Thus, give the implicit GD) trajectories [G.]f of ICL., where W, W Gt.
Wo 0,
G₁ = AW1
G2 A WW21+ 4 Gi),
G3 = AW A (W+ (1wW2114440000 +AW))) G (13)
L-1
G, = Aw, (1 + Σ G.).
G.).
On the one hand, Eq.(13) shows G. AW + Wi-1), on the other hand, AW, & EE Wi wihh W which is only depend on (bo)f, Wwklf [Wilf. (W+ So the exclusion of Transformer weight ([ k]f: [Wolf, (W(l)) implies that G, is only dependent on W,-1 and H
And the prediction can be read from the corresponding position L-th layer output h as follows,
h (
ypred )
h& Weh (1 Wi)
mask )
XN+1
A.5
Proof of Theorem
The origin form of the mutual information based bound is predicated on a sample-specific MI, which quantifies the shared information between the output variable W and the input sample set H,. The following lemma shows the result:
Lemma 2. (Xu and Raginsky [55, Theorem 1.J). Assume the loss l(w, h) is R-subGaussian for any w E W. then
2R
error S
N (W:Hs).
where I(W; Hs) DxI(Qw.H, lQw @ QH:) is the mutual information and DEL denotes the KL divergence.
Unroll the terminal parameters' mutual information (W;Hs) to the full trajectories' mutual infor- mation will get:
Lemma 3. (Wang and Mao [51, Lemma 4.1). Let Assumption 1 hold, then 1(Wr; Hs) S
E I[-G, + C2/2 2, HslW), Let -G, C//2 N, & G.

---

Proof. Recall Assumption 1 (4), Wang and Mao [51] get.
(W);Hss = 1(W C/ NE):Hs) Hs)
< (WW,,v(-Gr,+CC, C. Hs)
(14)
1(W2-1;; + T(/ ((( C
(15)
= -IWW-1::H) + I 0
< I(-G, + C. N, HsHHs) I(W:: Hs)
t=
= Σ T(-G++ N,
where Eq. (14) is by the data processing inequality (e.g.. 7 - -x, Y) form a Markoy chain then [( + 2) (X,, Y: 2)), Eq. (15) is by the chain rule of the mutual information, and learning rate 1) is dropped since mutual information is scale-invariant. / (Wo; Hs) = 0 for Wo is independent of all other random variables in Theorem and Assumption 1.
Besides, we present the variational representation of mutual information:
Lemma 4. (Polyanskiy and Wu [32, Corollary 3.1.)). For two random variables X and Y, we have I(X;Y)= = inff (XxKK((Qvi ())), where the infimum is achieved at P = =Qy.
Lemma 5. (Wang and Mao 151, Lemma 5.1). At every time step t, let PÑ.j,,_, be any distribution satisfying DKL fl < co. we have ((Gr:H((W.-1) = Ew.- |inf inffs, HH DDK H,, W... _)]] where the infinuum is achieved when PN, W = Qu.iw.r. The KL divergence may be viewed as an estimate of the sensitivity of the full batch implicit gradient to specific demonstration sample H, H
From Lemma 5, every choice of PR.Iw, gives rise to upper bound of the MI of interest via IG 1, < Ew. B D (Qo, |Paaw..)]] Same to Wang and Mao [51], choose an isotropic Gaussian prior PN, w..; = V( where both 9t and or are only allowed to depend on W under Theorem 1, and optimize the KI. divergence in Lemma 5 over o for a fixed Ge- Additionally, Under Theorem where we define the implicit GD trajectories of ICL, assume Ct is positive-defintt matrix, for any E [L], we have,
(-G. + C//2 H. WWWW-1= w
S inf EH. Gr.0r
DKL (PG..cc!2NNWW 1,H, -H1, || MPP--
inf EH, 3r.0
1 log
(dot(o?Ta)) dell(C,)
) d
1
(G.-g.)11 3.)"" (Gi -G--u ge)
20 ( C)
tr
2
20
20
(16)
1 inf EH,
1 (IIG, -ill2 + tr (Ct)) + dlogo of d tr flog Ci)
(17)
2 ac of
where Eq. (16) is by the KL divergence between two Gaussian distributions: (xl(pl))) = log d
det(2,)) k (M₂ - ". (Mp Myy + tr(), and Eq. (17) is due the fact that G.G. tr(G,G?) and log (C, tr (log C.) when C, is positive-definite matrix.
Let (t) = EH, - 0el + tr [Ct)] and Az(t) = EH, [tr flog C.)] and fix ÿ. can rewrite
Eq. (17) as,
inf Ay(t) + d dlog of d - Az(t)
2 0,20 of
d log (t) 1
A2(t).
2
d
2
18

---

where the optimal o² V (A when we take the derivative of o,. Then combine Lemma 2 and finish the proof of the following lemma.
Lemma 3,
Lemma 6. Under the conditions of Theorem assume the implicit gradient noise covariance C₁ is a deefie matrix, the loss E(w, h) is R-subGaussian for any w E W E RR. For any E [L], let 9t be any constant vector for a given w 1, then
R
error S
Bw._ [dlog(Ai A2(t)]
|Lc 90
+ tr((
R
BHH
H W.-1
dlog
-tr(log
N
where A1(t) E.... |IG.
gell2 + tr(C.i] A2(() = Ew.,, [tr C. vec(G,) = G
ir(.) denotes the trace of a matrix and E& is the conditional expectation.
Further, we let ÿt = 0, reverse the process of flattening the weight matrix into a vector form, and by Eq. (19) obtain
vec AW.(1
-G,)) 9t
AW
Σ
Gj)
ur(C
J=1
t-
$ |A
1
Σ
G
tr((C)).
F
Then plug everything into Lemma 6, we conclude the proof.
A.6 Examples of Remark 4
First, we will present several expressions that will utilized later. For any matrix A and B, have,
|vcc(A) A) @ vec(B)]l
ΣΣ
vec(()) max lo (vec(A) @ vec(B)) (18)
|AB|| S || [B]F A + B S | + B
(19)
(20)
Same in Lemma 1, AW cet [EW Wyh, @ Wgh,) Wo. let represents the remained rank and å represents the potential noise consisting of parts with small singular values. Some SVD controlling the norm of A Wie examples
follows.
Example (Prune Wo). Suppose we decompose We by SVD, We We. + g9 (VQ)T + SQ. let z A EN Wrh, @ Wkh:)) for simplicity. (
JAW wel ||p Z (V @TTT + zdq
S lzZ Q- |ıáăQl r: where the inequality part takes advantage of the Eq. (20), so the upper bound on || llWiilll is decreasing when using SVD.
Example 3 (Prune Wy or Wk). Suppose we decompose Wv by SVD, Wy Wv. + +vv U. (VV) + ōy, and consider the upper bound defined in Example 1,
N
UB(AAW
Σ Weh @ Wghh |
N
Wy. & Wxhh W
(21)
19

---

where Eq. (21) is by Eq. (18), so the upper bound on is also decreasing. The same is true for decomposing W K.
Example 4 (Prune W MLPP). Suppose we decompose W Mlp by SVD, WMLP WMPP. ÓMPP- E. V MLPP, let 2 A EE Wyh, @ W Wxh,)) W Q for simplicity.
AW
-,, V 2 + SMLPZ
S ||WMLr, + l0MLLZZlp:
where recall Appendix A.3, AW". idl
WMIP E Wyh, @ Wich;) Wo. And the inequality
part takes advantage of the Eq. (20), so the upper bound on A AW.. iol SVD.
is decreasing when using
B
More Discussions
B.1 Explanation of the Mask Matrix
Our notation here follows the related work [2], which explains: Note that the prompt is asymmetric since the label for XN+1 is excluded from the input. To reflect this asymmetric structure, the mask matrix M is included in the attention. More specifically, if you pay attention to the (N 1)-th item, M is supposed to represent causal mask (For H E IR (gddut+din) x(N+1), HM (hi,..., hN, hN+ ) (hi,... , 0) H,). Besides, this mask method is used in GLM training [13]: Part A tokens can attend to each other, but cannot attend to any tokens in B. Part B tokens can attend to Part A and antecedents in B, but cannot attend to any subsequent tokens in B. So is reasonable for E1. and (2) use the same mask.
B.2 Definitions of Deep and Shallow
Indeed, there is no universally accepted definition for the terms "deep" and "shallow" their interpretation can be subjective and dependent on the reference frame (e.g., model size). Intuitively, in this paper, definition is similar to that of work [50]: "shallow" layers refer those closer the input, while "deep" layers are closer to the output. In Figure and Figure 2 (Section 2), "shallow" typically denotes the first few layers, and "deep" denotes the last few layers of the network.
B.3 The Effect of Pruning Only Single Module
In our theory, the example (pruning only W or Wy) is provided in Appendix A.6, demonstrating how weight pruning can affect the norm of AA W, W.]'// Th Thus, it may confer advantages on the performance of Transformers in ICL inference. However, in Theorem 2 (Remark 5), it also suggests that modifications the shallow layers have less steady impact. Additionally, please review the supplementary experimental results provided below. (We choose key projection matrix W K and select the 3-layer (large matrix condition number Figure of GPT-J-6B, others are the same to Section 1..).
Task Name
Optimal g*
Test Accuracy/F1 Improve
SST-2
0.0
0.7828 0.7828
RTE
0.5
0.5413 0.5413
CОРA
0.995
0.53 0.54
20

---

B.4 Why Optimal Clipping Rate Varies?
As we mentioned in Section 3.3 and Section C.4, the implicit gradients produced by Transformers in practical applications noisy due factors such the extent of model pre-training and data characteristics (e.g., ICL shot number/task difficulty). Therefore, [W ! in Theorem 1 have varies noise. That is why optimal varies.
Gradient quality derived from context (i.c., G.).
Theorem (Remark 2), G, only dependent on W,- and H., this is consistent with gradient descent in terms of relevance (Conventionally. gradients in training are only related to the current parameters and the training samples).
In real-world training scenarios, SGD computes gradient by selecting small batch of samples per iteration. This approach approximates the true gradient, inherently introducing noise. Similarly, in In-Context Learning, a small subset of samples (context examples) is used to generate implicit gradient (i.e., G.), which also results in the introduction of noise.
B.5 Apply the Same Clipping Rate to Other Datasets
As we mentioned Appendix B.4, A A (G) in Theorem exhibit varying levels of noise, causing the optimal clipping rate to vary among different tasks, it is dependent the specific task and data. So we apply the clipping rate (0.95) as used SST-2 to other datasets, the model performance can either improve deteriorate. Additionally, It is possible to conduct a certain number of experiments to find a range of optimal clipping rate that broadly applicable.
B.6 What Would Happen the Layer Was Dropped Entirely?
Dropping the layer could be the best option specifically for optimizing generalization error. Here's detailed analysis:
(1) our theoretical framework, we model each layer of the Transformer do single iteration of implicit gradient descent (ICI.) in Theorem 1. This scientific analysis references [46, 3, 10].
(2) In Theorem 2, L-layer Transformer:
Expected generalization error = population risk (L,)) empirical risk (LH,)
Expected generalization error
R
Ei
|G..l (only show the main part). If
you drop the entire layer, it will change from E Σi= Therefore, dropping the layer may in fact be the best option for generalization error.
(3) However, according the traditional statistical learning viewpoint, performance be bounded by the sum of optimization error and generalization error (Please see Remark for "How should Theorem 2 be interpreted?").
Thus, during the pruning process, there is trade-off between optimization and generalization. Therefore, as pruning increases, the model's 's performance tends to first improve and then decline (the drop-layer method [19] does not harm but also does not improve), as demonstrated the experiments shown in Figure 1.
In conclusion, dropping the entire layer be potential method (best option for generalization error) in our theoretical framework, but it may not necessarily be the best option for model performance.
C Extension to Experiments
Computational resources. use single NVIDIA GeForce RTX 3090 GPU and most tests run take 1-5 hours depending dataset size and context length.
21

---

C.1 Prompts
Table 1: The prompts of the datasets used in our experiments. Here regard the types of tasks: classification, multiple-choice (cls.), (meh.). < represents input from the dataset. For (mch.) tasks, we put in different candidates in the prompt and calculate the average log- ligllikeihood of each candidate, and select the candidate with the highest score.
Dataset
Type
Prompt
SST-2
(cls.)
text: <text> sentiment: <label>
AGNEWS
(cls.)
text: <text> classification: <label>
EMOC
(cls.)
text: <text> sentiment: <label>
MRPC
sentence1: <sentence 1> sentence2: <sentence2> label : <label> (cls.)
RTE
(cls.)
<premise> Does this mean that <hypothesis> is true? select Yes or No? <label>
CB
(cls.)
Suppose <premise> Can infer that <hypothesis>? Yes, No,
Maybe? <label>
CОРА
(mch.)
<pemee>>quustion><chhcee>
C.2
Algorithml
Algorithm Search the Optimal Clipping Rate for Downstream Tasks
Require:
Pretrained model M,, dataset D. predefined clipping rate candidate C, predefined module O E [ATTN, MLP]
Split D into ICL demonstration sample set H.s validation set V, and test set T Compute condition numbers for all layers in M
Select layers L with top-k largest condition numbers given
Select the largest layer number | from C Initialize g* =0 0
Initialize score" = 0 for each & in C do
M = 1,,,(
score = E'valaae H.,) score score* then g = score score end if end for M = -Clip(M.1.5'.O)
lest score Evaluate (M" H.s) Output lest score on T
The details. Firstly, the details of the algorithm can be reviewed in the code provided. Specifically, the set of clipping rate candidates is predefined. In our study, the clipping rate candidates are set as shown in Figures and [0. 0.1, 0.5, 0.75, 0.9, 0.95, 0.99, 0.995]. Besides, we analyze the impact of different hyperparameters through comparative experiments (details in Section 2 and Section 3.4). as detailed below. (1) Clipping rate W: search for the optimal & in the predefined clipping rate candidates. (2) Predefined module O: The module containing the target pruning weights, which can be chosen from [k: fk_proj. prorj, v_proj, out cuf_proj.. fc_in, wc__ootttttt all, mlp, attn). (3) Selected layer 1: The layer containing the target pruning weights. For examlpe: In Section 2, we mainly focus on comparing the impact of weight pruning the first two and the last two layers of the model. (4) ICL shot number The demonstration number in ICL, we analyze the effect of different ICL shot numbers Section 2.2.
22

---

C.3 Dataset Details
For experiments, we consider numerous classification datasets, including: SST-2 [39], AGNEWS [58], EMOC [8], MRPC [12], RTE [5], CB [11]. We also include a multiple-choice dataset COPA [18].
SST-2. SST-2 focuses on sentiment analysis, specifically the task of determining the sentiment of movie reviews. The dataset consists of sentences labeled as having a positive or negative sentiment. These sentences are divided into a training set, a validation set, and test set. The training set includes about 67,349 sentences, whereas the validation set contains around 872 sentences. The test set comprises approximately 1,821 sentences without labels. We random sample (10)) data from the train set as demonstration sample, 8,000 data from the train set as validation set to search the optimal clipping rate, the rest 59,000+ data as test set.
The AGNews dataset is a collection designed for text classification tasks, specifically for news categorization. It consists of news articles gathered from the AG's corpus of news on the web, which is categorized into four main topics: World, Sports, Business, and Science/Technone terms of structure, the AGNews dataset comprises approximately 120,000 training samples and 7,600 test samples. We random sample N88) data from the train set as demonstration sample, 8000 data from the train set as validation set to search the optimal clipping rate, the 7,600 test samples as test set.
AGNews.
In
EmoC. The EmoC dataset focuses on contextual emotion detection in text. It consists of short text conversations extracted from three-turn English Tweets. The dataset is annotated with four emotion labels: happiness, sadness, anger, and others, we relabel the dataset with happiness and others. The training set includes 30, 160 samples, while the test set comprises 5,509 samples. We random sample N(10) data from the train set as demonstration sample, 5,000 data from the train set as validation set search the optimal clipping rate, the 5,509 test samples test set.
MRPC. The MRPC dataset focuses on paraphrase detection text. It consists of sentence pairs automatically extracted from online news sources. The dataset is annotated with binary labels indicating whether the sentences are paraphrases of each other. The training set includes 3,668 pairs, the validation set includes 408 pairs, while the test set comprises about 1,725 pairs. We random sample N(0) data from the train set as demonstration sample, the 408 pairs from validation set to search the optimal clipping rate, the 1,725 test samples test set.
RTE. Recognizing Textual Entailment (RTE) task, which involves assessing the relationship between a pair of sentences premise and hypothesis. The objective of the task is to determine whether the hypothesis can be logically inferred from the premise. Specifically, the model must evaluate whether the relationship between the two sentences is one of "entailment," where the content of the hypothesis directly or indirectly supported by the premise, or "non-entailment:; " which includes both contradiction and neutrality. In RTE from SuperGLUE [47], the training set includes about 2,490 items, whereas the validation set contains around 277 items. We random sample NV01)) data from the train set demonstration sample, the remain items from train set to search the optimal clipping rate, the 277 validation samples as test set.
CB. The CommitmentBank (CB) dataset is part of the SuperGLUE benchmark and focuses on textual entailment with an emphasis on pragmatic inference. The task involves determining whether a hypothesis can be logically inferred from given text, which in this dataset, typically comprises premise followed by a hypothesis. The training set includes about 250 items, whereas the validation set contains around 56 items. We random sample N(15) data from the train set as demonstration sample, the remain 235 items from train set to search the optimal clipping rate, the 56 validation samples as test set.
The Choice of Plausible Alternatives (COPA) dataset is specifically designed evaluate causal reasoning abilities in natural language processing models. This dataset presents a task where models must determine causal relationships within simple scenarios. Each question in COPA consists of a premise and two possible choices, one of which is the correct cause or effect of the premise. COPA's 's dataset is relatively straightforward and consists of 500 questions split evenly into training (400) and validation (100) sets. We random sample N(10) data from the train set as demonstration sample, the 200 items from train set to search the optimal clipping rate, the 100 validation samples as test set.
COРА.
23

---

C.4 Noise Discussion of the Implicit Gradient
LR65500000 Train Steps
LR&Different Train Steps Comparison
1.2
Transformer Least Squares
train steps: 25000
train steps: 50000
1.0
1.0
train steps: 75000
train steps: 100000
0.8
0.8
squared error
squared error
UM
0.6
0.4
0.4
0.2
0.2
0.0
0
20
30
10
20
30
40
in-context examples
#in-context examples
(a) Trained Transformer performs comparably to (b) The effect of different train steps on Trained the optimal least squares estimator
Transformer
Figure 5: Evaluating the trained Transformer on in-context learning linear functions. (a) Garg
al. [16] consider the class of linear functions F {f]((z) w x, w E RR), in d dimensions where d = 20. They sample Zi...., Zk, Equery, and u: independently from the isotropic Gaussian distribution (N,
They then compute each 3/ = w x, and construct the prompt as P = (I11 U1, 32, 32....... Fk. yk F. This figure plots the normalized squared error of the Transformer ((M(P) "T= )²query2// /d), the errors are normalized so that the trivial zero estimator achieves an error of (dashed line). Besides, when the number of in-context examples reaches the problem dimension (here 20), least squares achieves error while the Transformer achieves an error of 0.02. (b) We follow the same setting of [16] in (a) to compare the Trained Transformer with different train steps.
As mentioned in Garg et al. [16]'s work, the trained model is able to learn unseen linear functions from in-context examples with performance comparable the optimal least squares estimator, this can be seen in Figure 5a. More importantly, the number of in-context examples (shot number) plays a significant role. In this case, the error approaches zero only when the shot number is greater than or equal to d. This indicates that the implicit gradients of ICL are influenced by the shot number, and there exists threshold N, when the actual shot number b is below this threshold, the implicit gradients are noisy. On the other hand, following the same setting of [16] in figure 5a, we compare the performance of the Trained Transformer at different train steps as shown in figure 5b. This indicates that the performance of the model varies with the number of train steps, which also means that the implicit gradients of ICL generated by the model are influenced by the extent of its pre-training.
In conclusion, actual implicit gradient descent involves noise, which primarily stems from the shot number and the degree of model pre-training.
C.5 The Detailed Results of Algorithm 1
Table 2: The results of Algorithm 1 on SST-2
Model Name
Layer Number
Module Name
Optimal g*
Test Ace Improve
GPT-J-6B
26
MLP
0.95
0.7527 0.8437(t)
GPT-J-6B
27
ATTN
0.995
0.7527 - 0.7642(1)
LLAMA2-7B
30
MLP
0.99
0.9228 0.9257(1)
LLAMA2-7B
30
ATTN
0.95
0.9228 0.9287(1)
24

---

Table The results of Algorithm on AGNEWS
Model Name
Layer Number
Module Name
Optimal
Test Acc Improve
GPT-J-6B
26
MLP
0.1
0.76434 0.76947
GPT-J-6B
27
ATTN
0.95
0.76434 0.77026
LLAMA2-7B
30
MLP
0.995
0.77026 0.84881
LLAMA2-7B
30
ATTN
0.1
0.77026 0.77039
Table 4: The results of Algorithm on EmoC
Model Name
Layer Number
Module Name
Optimal
Test Acc Improve
GPT-J-6B
26
MLP
0.1
0.69032 0.74278
GPT-J-6B
27
ATTN
0.5
0.69032 - 0.69758
LLAMA2-7B
30
MLP
0.1
0.77110 0.79106(1
LLAMA2-7B
30
ATTN
0.1
0.77110 0.76910(L)
Table 5: The results of Algorithm on MRPC
Model Name
Layer Number
Module Name
Optimal 5"
Test Ace Improve
GPT-J-6B
26
MLP
0.995
0.66492 0.66492
GPT-J-6B
27
ATTN
0.995
0.66492 0.66492
LLAMA2-7B
30
MLP
0.5
0.66666 0.67536
ILAMA2-7B
30
ATTN
0.995
0.66666 0.66608(4)
Table 6: The results of Algorithm 1 on RTE
Model Name
Layer Number
Module Name
Optimal 5"
Test Acc Improve
GPT-J-6B
26
MLP
0.99
0,56884 0.57971 (t)
GPT-J-6B
27
ATTN
0.56884 0.56884
LLAMA2-7B
30
MLP
0.75
0,56159 0.57608
LLAMA2-7B
30
ATTN
0.9
0.56159 0.57971(1)
Table 7: The results of Algorithm 1 on CB
Model Name
Layer Number
Module Name
Optimal g*
Test Ace Improve
GPT-J-6B
26
MLP
0.95
0.58181 0.67272
GPT-J-6B
ATTN
0.58181 0.58181
LLAMA2-7B
30
MLP
0.95
0.83636 0.85454(1)
LLAMA2-7B
30
ATTN
0.99
0.83636 0.87272(1)
Table 8: The results of Algorithm 1 on COPA
Model Name
Layer Number
Module Name
Optimal ş*
Test F1 Improve
GPT-J-6B
26
MLP
0.95
0.58 0.59(1)
GPT-J-6B
27
ATTN
0.99
0.58 0.58( -)
LLAMA2-7B
30
MILP
0.5
0.57 0.62
LLAMA2-7B
30
ATTN
0.99
0.57 - 0.56(4)
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper discusses enhancements in-context learning (ICL) performance for pre-trained large language models (LLMs), particularly focusing the effects of SVD- based weight pruning. Key contributions made include: What is the phenomenon? Why this phenomenon? How to better use this phenomenon?
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section 4, we discuss the limitations of our work and identify them as valuable future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
26

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The full set of assumptions and complete proofs can be found in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: First, our code is available the anonymous url https: httttt/tt////oo////t /aap://////ommmuus.... 4open. science Cingnnnnnnnnnnnnnnnnnnnnnnnnnnnn www.sssWrrrnnnnsseee.
(camera ready https://githe Second, details regarding datasets and how they were used are shown in Appendix C.I and C.3.
com/ chen Mhhen12SCrlss /0tte/00//////a/annnnnnnCCCC wtw.eiinnnnno_SSSiniiii...
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
27

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: For details on data and model download, please refer to the code. our code is available at the anonymous url https://anonymous 4open
Wiience/r/Enaaccinge Surrr chen1 2atmn133rrrlssa LLL SieaaaaiiiiIII__SViiiiiii
(3335553333/mmrr ready https://gi com/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
sommiiiiiiiicc for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. httppblcc/gguueess.....ppppp//ee http:/uwdee//eeeeeeeeee /wt////oooimssiieeeee
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: First, details regarding datasets and how they were used shown in Appendix C. and C.3. Second, how hyperparameters were chosen also detailed in paper and code.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The primary focus of our paper is on the theoretical analysis of the underlying mechanism of phenomenon we discovered. While we have conducted experiments on various publicly available datasets and different models support our theoretical findings, we did not include discussions error bars. Our experiments across different datasets and
28

---

models reflect to some extent the correspondence between theory and practice. And due to limitations computational resources, we did not report error bars, confidence intervals, or conduct statistical significance tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eτror rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section 3.4, we provide sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics attps://neurips: hctppubicc/thhheeetitiiiii
Answer: [Yes]
Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics in every respect, and we make sure to preserve anonymity.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
29

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work mainly focuses theoretically analysing the underlying mechanism of phenomenon that found. For positive societal impacts, understanding the underlying mechanism can contribute to use ICL. For negative socictal impacts, the understanding of the mechanism not be used in harmful ways.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
30

---

Justification:
We cite the original papers that produced the code packages or datasets. For example, Section C.4 we cite the original owner of Figure 5a.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswitheeeee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
31

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
32


Reviews:
[{"review_id": "rro3C7Vk3F", "reviewer": "Reviewer_itEx", "summary": "This paper discusses the phenomenon: SVD-based weight pruning can increase the in-context learning abilities of transformer based LLMs. In this paper, the authors conduct theorectical analysis by presenting the implicit gradient descent trajectories of ICL and providing the generation bounds visa full implicit gradient descent trajectories. This paper also provide a simple yet effective algorithm to clip the LLM by SVD to enhance ICL inference.", "questions": "1. Matrix condition number is an option for the indicator. But could there be more options, such as compute the decreasing rate of eigenvalues? Because when p=2, conditional number only leverages two values among all the eigenvalues.\n2. Could authors provide further more clarification why optimal $\\xi$ varies, and is there a way to explain this phenomenon under current theoretical framework provided in this paper?", "limitations": "No", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "First, this paper has a clear writing and is easy to follow. \n\nIt provides a detailed theoretical analysis on why SVD based weight pruning will improve ICL performance by leveraging the implicit gradient descent trajectories. It also provides the generalization bounds of ICL, in Theorem 2, it can be inferred that the noise level and the norm of of gradient contribute to the error bound. It provides the theoretical insight of SVD based method.\n\nThe authors provides a simple algorithm to leverage the discovered phenomenon to improve ICL performance of LLM in a gradient-freee way. The ratio between $\\sigma_{max} $ and $\\sigma_{min}$ is a good choice of heuristic conditional number.", "weaknesses": "1. More details of algorithms is not shared. e.g. the range / number of clipping rate candidates set. \n2. In experiments result of C.5, the optimal $\\xi$ varies a lot across different tasks and different modules. However, this phenomenon is not touched in the theoretical part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and valuable comments. Below we address specific questions.\n\n> **W1:** More details of algorithms is not shared. e.g. the range / number of clipping rate candidates set.\n\n> **A:** Firstly, the details of the algorithm can be reviewed in the **code** provided. Specifically, the set of clipping rate candidates is predefined. In our study, the clipping rate candidates are set as shown in **Figures 1 and 2**: [0, 0.1, 0.5, 0.75, 0.9, 0.95, 0.99, 0.995]. \n> Besides, we analyze the impact of different hyperparameters through comparative experiments (details in **Section 2** and **Section 3.4**), as detailed below.\n>\n> - Clipping rate $\\xi$：We search for the optimal $\\xi$ in the predefined clipping rate candidates.\n>\n> - Predefined module $\\mathcal{O}$ : The module containing the target pruning weights, which can be chosen from \n>   ['k_proj', 'q_proj', 'v_proj', 'out_proj', 'fc_in', 'fc_out', 'all', 'mlp', 'attn']\n>\n> - Selected layer $l$ : The layer containing the target pruning weights. For examlpe: In **Section 2**, we mainly focuse on comparing the impact of weight pruning to the **first two and the last two layers** of the model.\n>\n> - ICL shot number : The demonstration number in ICL, we analyze the effect of different ICL shot numbers in **Section 2.2**.\n\n---\n\n> **W2:** In experiments result of C.5, the optimal $\\xi$ varies a lot across different tasks and different modules. However, this phenomenon is not touched in the theoretical part.\n\n> **A:** That's a great question! The impact of different tasks and various module prunings on performance is indeed significant. \n>\n> However, our aim here is to provide a general analytical framework, so we have not delineated the impact of different layers/modules on performance in detail. \n>\n> In our theory, the simplification of the MLP and the handling method for the ATTN layer （mentioned in the main text） ：\n>\n> -  Feed-forward (MLP) layer：In **Appendix B.3**, it can be seen as a dimensional adaptation to correspond to residual connection. Then we can get the Implicit Gradient Descent with MLP:  $\\Delta W_{icl}^{''}=W_{MLP}\\left( \\sum_{i=1}^N W_V{h_i} \\otimes W_K{h_i} \\right) W_Q$\n>\n> - Attention (ATTN) layer:\n>\n>   (i) In main text **Section 3.2**, our theoretical analysis is mainly focuses on linear attention setting. \n>   \n>   $\\hat{h}\\_{N+1} = h\\_{N+1}+ \\Delta W\\_{icl} h\\_{N+1}, \\Delta W\\_{icl}=\\left( \\sum\\_{i=1}^N W_V{h_i} \\otimes W\\_K{h\\_i} \\right) W\\_Q$\n>\n>   (ii) In Section **Appendix B.2**,  it can be considered that the mapping function $\\phi$, or rather the effect of the **Softmax** function, is to project the original features into a higher-dimensional space to capture more profound features. Subsequently, learning and meta-optimization are conducted under this **new feature space**. \n>\n>   $\\hat{h}\\_{N+1}=h\\_{N+1} + \\Delta W_{icl}^{'}\\phi(W\\_Q h\\_{N+1}), \\Delta W_{icl}^{'}=\\frac{1}{D^{'}} \\left[\\sum_{i=1}^N (W_VH_s)_i \\otimes \\phi(W_KH_s)_i \\right]$\n>\n> Then if one consider more than one layer,  a more detailed analysis can be provided within the current framework to depict the interaction relationship between the input feature and these different  modules.\n---\n> **Q1:** Matrix condition number is an option for the indicator. But could there be more options, such as compute the decreasing rate of eigenvalues? Because when p=2, conditional number only leverages two values among all the eigenvalues.\n\n> **A:**  Good question! Let's first review our theory in **Theorem 2**: Modulating the norm of $[G\\_t]_1^L$ or $[\\Delta W\\_{t}]_1^L$  could enhance performance when utilizing ICL. \n>\n> Our primary objective in this work is to provide a general analytical framework. The simple algorithm is employed to demonstrate the effectiveness of theoretical analysis in guiding experimental procedures. So, Any indicator that can guide the control of norms is a potential option. For instance: \n>\n> (i) Consider the behavior of the first few and last few singular values.\n>\n> (ii) Assessing the intrinsic rank of the matrix can be valuable, as it offers another dimension of understanding beyond just the condition number. [1] [2] \n>\n> [1] Armen Aghajanyan et al. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. URL https://arxiv.org/abs/2012.13255. \n>\n> [2] Edward Hu et al. Lora: Low-rank adaptation of large language models. ICLR 2022.\n\n---\n\n> **Q2:** Could authors provide further more clarification why optimal $\\xi$ varies, and is there a way to explain this phenomenon under current theoretical framework provided in this paper?\n\n> **A:** (i) As we mentioned in **Section 3.3 and Section C.4**,  the implicit gradients produced by Transformers in practical applications are noisy due to factors such as the extent of model pre-training and data characteristics (eg. ICL shot number/task difficulty). Therefore,  $[\\Delta W_t]_1^L$/$[G_t]_t^L$ in **Theorem 1** have varies noise. That is why optimal $\\xi$ varies. \n>\n> (ii) Besides,  expected generalization error (**Theorem 2**) = population risk - empirical risk ($L_{\\mu}$-$L_{H_s}$).  we search the optimal $\\xi$ on the validation set and subsequently evaluate it on the test set (detailed in **Section 3.4**).\n>\n> On the one hand, in **Theorem 2**, clipping weight controls the F-norm of the implicit gradient ( $[\\Delta W\\_t]\\_{1}^{L}/[G\\_t]\\_{1}^{L}$), which reduces the expected generalization error, on the other hand, we can evaluate the empirical risk ($L_{H_s}$) by observing the model's performance on the validation set and estimate the population risk ($L_{\\mu}$). (Note that different types of tasks affect $L_{H_s}$ differently, and different modules have varying impacts on the expected generalization error. See **A** to **W2**.)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EkIUZaxEim", "reviewer": "Reviewer_Zh6a", "summary": "This paper demonstrates that (1) SVD-based weight pruning can sometimes achieve better in-context learning performance, and (2) pruning weights in deeper layers often results in more stable outcomes compared to shallow layers. The authors explain their findings through theoretical analysis and propose an intuitive matrix condition number-based weight pruning algorithm to achieve both stable and improved performance.", "questions": "[1] How should Theorem 2 be interpreted? It seems only provide a weak upper bound for in-context learning stableness. Can this also be applied to empirical risk, such as $L_{H_S}$ and $L_\\mu$?\n\n [2] Theorem 2 gives the upper bound for expected generalization error. If we fix N in the constructed transformer and reduce the number of in-context examples to $N'$ in the input sequence, then we can find that while the factor $R^2/N$ remains unchanged in theorem 2 , $\\Delta W_t$ will change from $(\\sum_{i=1}^N)…$ to $(\\sum_{i=1}^{N'})…, where $ ($N' < N$). Based on the analysis across different layers, could this mean that fewer context examples are more robust for SVD weight pruning?\n\n[3] Note that in fig-3, large matrix condition numbers can exist in some modules of shallow layers, such as the attention key (K) in GPT-J-6B . What would be the effect of pruning only a single module in a shallow layer (e.g., the key projection matrix) rather than pruning the entire attention module (including Q, K, and V)?\n\n [4] In C.5, it's noted that the optimal clipping rate is sometimes very small and varies across datasets. What would happen if we apply the same clipping rate (e.g., 0.95) as used in SST-2 to other datasets?", "limitations": "See Weaknesses and Questions. Besides, there seem many mistakes on pages 7 and 8, for example, the equation between line 228 and 229, and the equation for about F-norm in lines 230,231,232. These inaccuracies cast doubt on the overall reliability of the paper's findings. If there are any misunderstandings on my part, please point them out, and I will reconsider my evaluation of this work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This work conducts an in-depth analysis to explain the \"stability\" of transformer weight pruning across different layers. The framework is interesting and validated through experiments. Moreover, the theoretical analysis can be applied to design new algorithms like algorithm 1 in this paper .", "weaknesses": "Despite adopting various simplifications (such as using a linear attention transformer without MLP and layer normalization, treating each in-context example as a single vector, implementing attention masks for query tokens, and using meta-gradients for in-context learning) in their theoretical analysis, the results are still limited. They only explain why SVD-based weight pruning can achieve \"stable\" performance, leaving the more intriguing question of why transformers can achieve \"better\" performance with pruning unclear. Additionally, even with detailed hyperparameter tuning, the effectiveness of Algorithm 1 remains uncertain. Further details are provided in the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, and for your thoughtful comments and questions. They will certainly help improve the revised paper. Please see our response below.\n\n> **A to Weaknesses:** Thanks to Reviewer Zh6a for raising the issue, which gives us the opportunity to clarify this matter. \n>\n> (1) Our goal is to provide insightful conclusions under simplified conditions. However, we also discuss the standard Softmax attention setting in **Appendix B.2** and feed-forward (MLP) layers in **Appendix B.3**.\n>\n> (2) The goal of this paper is to study ICL from a generalization perspective. According to the traditional statistical learning viewpoint, performance can be defined by the sum of optimization error and generalization error. This paper reveals that SVD decomposition is beneficial for generalization error, but does not analyze its impact on optimization. Given that theoretical analysis of ICL is rapidly developing, the impact of weight pruning on optimization has not yet been fully explored. We will consider this aspect in our future work.\n\n> **A to Q1** (How should Theorem 2 be interpreted? Can this also be applied to empirical risk ?):\n>\n> Expected generalization error (**Theorem 2**) = population risk ($L\\_{\\mu}$) - empirical risk ($L\\_{H\\_s}$). \n>\n>  More specifically, on the one hand, in **Theorem 2**, clipping weight controls the F-norm of the implicit gradient  $([\\Delta W\\_t]\\_1^L/[G_t]\\_1^L)$), which reduces the expected generalization error, on the other hand, we can evaluate the empirical risk ($L\\_{H\\_s}$) by observing the model's performance on the validation set. If the generalization error is determined, it is at least possible to estimate the population risk ($L_{\\mu}$). Thus, the most challenging aspect is addressing the generalization error. For the direct theoretical analysis of empirical risk, we will consider this aspect in our future work.\n\n> **A to Q2** (Could this mean that fewer context examples are more robust for SVD weight pruning?):  \n>\n>That's a pretty worthy question! Yes, fewer context examples can indeed lead to more robust results in SVD weight pruning. \n>\n> (1) Experimentally, in **Section 2.2**, we analyze the effect of different ICL shot numbers. As shown in Figure 2, with a decrease in the number of shots, the rate of performance collapse in the model slows down after a sharp reduction at the shallow layer.\n>\n> (2) Theoretically, $\\Delta {W}(N)-\\Delta {W}(N^{'}) = \\left( \\sum\\_{i=N^{'}+1}^N{W}\\_V{{h}\\_i} \\otimes {W}\\_K{{h}\\_i} \\right) {W}\\_Q,$ so it is supposed that the implicit gradient is more sensitive for SVD weight pruning.\n>\n> (3) The robustness mainly concerns the operation \"SVD weight pruning\", rather than the model's performance.\n\n> **A to Q3** (What would be the effect of pruning only a single module?):\n>\n> Good question! In our theory, the example (pruning only K or V) is provided in **Appendix B.6**, demonstrating how weight pruning can affect the norm of  $[\\Delta W_t]_1^L$/$[G_t]_t^L$. Thus, it may confer advantages on the performance of Transformers in ICL inference. However, in **Theorem 2** (Remark 5), it also suggests that modifications to the shallow layers have a less steady impact.\n>\n> Additionally, please review the supplementary experimental results provided below. (We choose key projection matrix and select the  3-layer (large matrix condition number) of GPT-J-6B, others are the same to **Section 2.1**).\n>\n> | Task | Optimal $\\xi$ | Accuracy/F1 Improve      |\n> | --------- | --------------------- | ------------------------ |\n> | SST-2     | 0.0                  | 0.7828 - 0. 7828 |\n> | RTE       | 0.5                   | 0.5413 - 0.5413   |\n> | COPA      | 0.995                 | 0.53 - 0.54 |\n\n> **A to Q4** (What would happen if we apply the same clipping rate to other datasets?):\n>\n> As we mentioned in **Section 3.3 and Section C.4**,  the implicit gradients produced by Transformers in practical applications are noisy due to factors such as the extent of model pre-training and data characteristics (eg. ICL shot number/task difficulty). Therefore,  $[\\Delta W_t]_1^L$/$[G_t]_t^L$ in **Theorem 1** exhibit varying levels of noise, causing the optimal clipping rate to vary among different tasks, as it is dependent on the specific task and data. So if we apply the clipping rate (0.95) as used in SST-2 to other datasets, the model performance can either improve or deteriorate. Additionally, It is possible to conduct a certain number of experiments to find a range of optimal clipping rate that is broadly applicable.\n\n> **A to Limitations:**   \n>\n> - For the equation between line 228 and 229.\n>\n> $||W^k\\_{V\\_r}h\\_i^{k−1}+\\delta\\_Vh\\_i^{k−1}||\\_2^2=||W^k\\_{V\\_r}h\\_i^{k−1}||\\_2^2+2(W^k\\_{V\\_r}h\\_i^{k−1})^T(\\delta\\_Vh\\_i^{k−1})+||\\delta\\_Vh\\_i^{k−1}||\\_2^2=||W^k\\_{V\\_r}h\\_i^{k−1}||\\_2^2+||\\delta\\_Vh\\_i^{k−1}||\\_2^2$. \n>\n> Note that  $W^k_{V_r}=U_{:r}\\Sigma_{:r}V_{:r}^T$ and $\\delta_V=U_{r:}\\Sigma_{r:}V_{r:}^T$, and $U_{:r},U_{r:}$ are **orthometric** (properties of SVD). \n>\n> Therefore, $(W^k_{V_r}h_i^{k−1})^T(\\delta_Vh_i^{k−1})=(h_i^{k−1})^TV_{:r}\\Sigma_{:r}(U_{:r}^TU_{r:})\\Sigma_{r:}V_{r:}^Th_i^{k−1}=0$.\n>\n> - For the equation for about F-norm in lines 230,231,232. Thank you for pointing out the typo (231). We appreciate your attention to detail and will have made the necessary corrections. However, this minor typo does not affect the final conclusion. A detailed analysis is as follows:\n>\n>   For any vector $\\mathbf{a}$ and $\\mathbf{b}$, $\\text{rank}(\\mathbf{a}\\otimes\\mathbf{b})=1$, so the matrix ($\\mathbf{a}\\otimes\\mathbf{b}$) only has **one** non-zero singular value.\n>\n>   Combined with $||A||\\_F=\\sqrt{\\sum\\_{i}\\sigma\\_i^2(A)}$ and singular value is nonnegative, we can get \n>\n>   $||\\mathbf{a}\\otimes\\mathbf{b}||\\_F = \\sqrt{\\sum\\_{i}\\sigma\\_i^2(\\mathbf{a}\\otimes\\mathbf{b})}=\\max\\_i[\\sigma\\_i(\\mathbf{a}\\otimes\\mathbf{b})]$.\n> Therefore, the unique non-zero singular value will decrease after performing SVD on $\\mathbf{a}$ and $\\mathbf{b}$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2qNcqtotph", "reviewer": "Reviewer_Nxqr", "summary": "This paper investigates the effect of singular value decomposition (SVD)-based weight pruning on the in-context learning (ICL) performance of large language models. \n\nThe Authors show that SVD-based pruning can enhance ICL performance, with deeper layers showing more stable improvements. \nThey provide theoretical analysis to explain these findings, presenting implicit gradient descent trajectories for ICL and deriving generalization bounds. \n\nBased on their insights, they propose a simple algorithm for enhancing ICL inference in downstream tasks.", "questions": "I suggest the Authors to address 1) the concerns regarding a better framing of the work in the current literature. In particular, there is a large body of evidence that is growing on the resilience of LLMs to pruning (see for example \"The Unreasonable Ineffectiveness of the Deeper Layers\", Gromov et al, https://arxiv.org/pdf/2403.17887, and references therein), and 2) the quality of writings by proofreading it together with a native speaker.", "limitations": "Limitations are discussed in the final section (4).", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The Authors provide a theoretical analysis to explain their empirical findings, including the derivation of implicit gradient descent trajectories and generalization bounds for ICL.\n\n- Furthermore, they propose a simple, derivative-free algorithm for enhancing ICL performance in downstream tasks, demonstrating the practical value of their theoretical insights.", "weaknesses": "- The theoretical analysis primarily focuses on linear attention, which may not fully capture the complexities of standard Softmax attention used in most transformer models\n\n- The proposed algorithm is derivative-free, but the search for optimal clipping rates may still be computationally expensive for very large models or datasets\n\n- There is a substantial lack of comparison with other pruning methods: the study focuses on SVD-based pruning but doesn't compare it with other pruning techniques, which could provide context for the method's effectiveness\n\n- Poor language, frequent typos, and grammatical errors are significant issues in this paper. This does not help readability, and would likely be a barrier to publication in its current form.\n\n- An essential part of the paper, which is the discussion of related works is not part of the main text. Furthermore, this discussion is prone to criticism. For example, quoting the seminal paper by Frankle and Carbin as an example of low-rank properties of neural networks is clearly misleading. I think that this discussion should be an essential part of the main text, and should also be substantially revised in order to avoid conceptual confusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks so much for your time and insightful comments. Please find our point-by-point response below.\n\n> **W1:**  The theoretical analysis primarily focuses on linear attention, which may not fully capture the complexities of standard Softmax attention used in most transformer models\n\n> **A:**\tFirstly, to facilitate qualitative analysis, our main text primarily focuses on the theoretical aspects of linear attention.  This is our first step on this issue, and our goal is to provide insightful conclusions under simplified conditions.\n>\n> Secondly,  a considerable portion of the works also consider from the linear attention to explore the ICL [1,2].\n>\n> However, we also discuss the standard Softmax attention setting in **Appendix B.2**. Specifically, it can be considered that the mapping function $\\phi$, or rather the effect of the **Softmax** function, is to project the original features into a higher-dimensional space to capture more profound features. Subsequently, learning and meta-optimization are conducted under this **new feature space**. \n>\n> $\\hat{h}\\_{N+1}=h\\_{N+1} + \\Delta W\\_{icl}^{'}\\phi(W\\_Q h\\_{N+1}),\\ \\Delta W\\_{icl}^{'}=\\frac{1}{D^{'}} \\left[\\sum\\_{i=1}^N (W\\_VH\\_s)\\_i \\otimes \\phi(W\\_KH\\_s)\\_i \\right]$ \n>\n> [1] Ekin Akyiurek et al. What learning algorithm is in-context learning? investigations with linear models. ICLR 2023.\n>\n> [2] Johannes von Oswald et al. Transformers learn in-context by gradient descent. ICML 2023.\n\n> **W2:**  The proposed algorithm is derivative-free, but the search for optimal clipping rates may still be computationally expensive for very large models or datasets\n\n> **A:**  (1) Our primary objective in this work is to provide a general theoretical framework that reveals the underlying mechanism behind the phenomenon that SVD-based weight pruning can enhance ICL performance. Based on this theory, a simple algorithm is provided to demonstrate the effectiveness of theoretical analysis in guiding experimental procedures.\n>\n> In **Theorem 2**, the **key insight** is that managing the expected generalization error by controlling the norm of $\\Delta W_t/G_t$, indicating that pruning methods are not unique. Therefore, some computationally efficient pruning methods could be considered: (a) Random SVD, (b) Magnitude-based pruning [1], and so on.\n>\n> [1] Wen, W et al. Learning structured sparsity in deep neural networks. NeurIPS 2016. \n\n> **W3:**  There is a substantial lack of comparison with other pruning methods: the study focuses on SVD-based pruning but doesn't compare it with other pruning techniques, which could provide context for the method's effectiveness\n\n> **A:** Thanks for your suggestion. \n> Similar to the answer to the above **W2**, we primarily focus on theoretical analysis. And the simple algorithm is just used to validate the effectiveness of the theory, so we have not provided comparative methods. There is a possibility that better pruning methods exist.\n>\n> For instance, if we use magnitude-based pruning on matrix $A$, the $||A||_F$ will be decrease.\n\n> **W4:**  Poor language, frequent typos, and grammatical errors are significant issues in this paper. This does not help readability, and would likely be a barrier to publication in its current form.\n\n> **A:** Thank you for your feedback. We sincerely apologize for the language issues, typos, and grammatical errors present in the paper. Rest assured, we are fully committed to addressing these concerns and will do everything possible to correct the issues before resubmission. Your patience and understanding are greatly appreciated, and we are dedicated to improving the paper to meet the required standards.\n\n> **W5:**  An essential part of the paper, which is the discussion of related works is not part of the main text. Furthermore, this discussion is prone to criticism. For example, quoting the seminal paper by Frankle and Carbin as an example of low-rank properties of neural networks is clearly misleading. I think that this discussion should be an essential part of the main text, and should also be substantially revised in order to avoid conceptual confusions.\n\n> **A:**  (1) Due to page limitations, we previously omitted the detailed discussion of related works in the main text, but we will include it in future versions. (2) We acknowledge your viewpoint and will make substantial revisions accordingly. Our findings show that pruning does not necessarily worsen performance (Note that performance depends not only on generalization but also on optimization). In some datasets, it may even improve results. This aligns with previous work, which we will discuss in detail in the Related Works section, such as [1]. In future research, we will study the effects of pruning different layers and the performance variations across different tasks.\n>\n> [1] Pratyusha Sharma et al. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. ICLR 2024.\n\n> **Q1:**  I suggest the Authors to address 1) the concerns regarding a better framing of the work in the current literature. In particular, there is a large body of evidence that is growing on the resilience of LLMs to pruning (see for example \"The Unreasonable Ineffectiveness of the Deeper Layers\", Gromov et al, https://arxiv.org/pdf/2403.17887, and references therein), and 2) the quality of writings by proofreading it together with a native speaker\n\n> **A:** Thanks. We will definitely revise the literature review (Gromov et al,and references therein) & our writings.\n>\n> However,  please let us demonstrate the advantages of our theoretical analysis (pruning weight) compared to the example (remove layer) you provided.\n> If we just remove the entire layer, it will result in two main issues: (a) It will become challenging to adjust the weights for another downstream task. (b) The implicit gradient update will be reduced by one step (**Theorem 1**), perhaps leading to a decline in model performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KFwFKguftY", "reviewer": "Reviewer_CqwV", "summary": "The authors provide a theoretical perspective on the stability of in context learning via implicit gradient descent trajectories. Ultimately, the analysis suggests that high condition numbers of the weight matrices belonging to layers with a high index can be pruned in order to achieve a model which performs better on ICL tasks.", "questions": "- I don't believe the clipping process was adequately explained. Once the SVD operation is done, do you clip starting from the largest singular value? or starting from the smallest?", "limitations": "The limitations have been discussed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- In context learning is important, and something which has not been studied as deeply as other topics of ML due to the recent rise of transformers and ICL in general.\n- The method intuitively makes sense and is something which can be conditionally tuned after training based on specific tasks if a validation set is available.", "weaknesses": "- It would be good to define deep and shallow, as these are subjective terms depending on the reference frame.\n- Figure 1 cpation says: \"We operate on the whole of MLP or ATTN.\" What does this mean?\n- If as figure 1 states, you can clip 99.5% of the original weights, what happens if you just drop that layer entirely? Recent work has shown that the deeper layers can be completely dropped without much effect. [1]\n  - I cannot see much benefit gained from pruning part of the weights with SVD when it seems that the in nearly all cases, the benefit can be had by dropping the layer entirely.\n  \n- Is the mask on L138 supposed to represent a causal mask? If so, I do not think the notation is correct, as the Identity matrix would only have $N$ binary values which is much less than is needed for a causal mask.\n- How can equation 1 and 2 use the same mask?\n- Example 1 appears to be incorrect:\n  - There is no parentheses around $W_{V_r}^k + \\delta_V h_i^{k-1}$ in the first line.\n  - The triangle inequality seems to say that line 2 $\\geq$ line 1\n  - Given the above, I do not see what conclusion can be drawn from this equation.\n  - Have I missed something here?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your response on finding our work thorough and informative. Below we address specific questions.\n\n> **W1:** It would be good to define deep and shallow, as these are subjective terms depending on the reference frame.\n\n> **A:** Indeed, there is no universally accepted definition for the terms \"deep\" and \"shallow\"  as their interpretation can be subjective and dependent on the reference frame (eg. model size). Intuitively,  in this paper, our definition is similar to that of Work [1]: “shallow”  layers refer to those closer to the input, while “deep”  layers are closer to the output. In Figure 1 and Figure 2 (**Section 2**), \"shallow\" typically denotes the first few layers, and \"deep\" denotes the last few layers of the network.\n>\n>\n> [1] Lean Wang et al. Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning. EMNLP 2023.\n\n> **W2:** Figure 1 cpation says: \"We operate on the whole of MLP or ATTN.\" What does this mean?\n\n> **A:** It refers to pruning all the weight matrices in these modules. The striking model architecture Transformer block consists of an attention (ATTN) layer and a feed-forward (MLP) layer. As mentioned in **Section 2**, the attention (ATTN) layers consist of key, query, value, and output matrices in both GPT-J-6B and LLAMA2-7B. The MLP layers in GPT-J-6B include input and output matrices, while in LLAMA2-7B, they are composed of up, gate, and down matrices.\n>\n\n> **W3:** If as figure 1 states, you can clip 99.5% of the original weights, what happens if you just drop that layer entirely? Recent work has shown that the deeper layers can be completely dropped without much effect. [1] \n>\n> - I cannot see much benefit gained from pruning part of the weights with SVD when it seems that the in nearly all cases, the benefit can be had by dropping the layer entirely.\n\n> **A:** Thank you for your good question. To provide a thorough and accurate answer, could you please share the reference for [1] that you mentioned? Understanding the specifics of this recent work would help us address your inquiry more effectively.\n>\n> Regarding your question, I have two key points to consider:\n>\n> (i) Even though we can \"clip 99.5% of the original weights,\" the remaining weights are still significant and play a crucial role. This is because the remaining 0.5% corresponds to the larger singular values after performing SVD, which are vital for maintaining the model's performance.\n>\n> (ii) As stated in **Theorem 1**, from the perspective of ICL (In-Context Learning) and gradient descent, each attention layer corresponds to an implicit gradient descent step (**L** layers with Implicit Gradient Descent Trajectories $[\\Delta W_t]_1^L$/$[G_t]_t^L$). \n>\n> Therefore, if we just drop the entire layer, it will result in two main issues: (a) It will become challenging to adjust the weights for another downstream task. (b) The implicit gradient update will be reduced by one step, generally leading to a decline in model performance.\n>\n> Thank you again for your understanding.\n\n> **W4:** Is the mask on L138 supposed to represent a causal mask? If so, I do not think the notation is correct, as the Identity matrix would only have $N$ binary values which is much less than is needed for a causal mask.\n>\n> **W5:** How can equation 1 and 2 use the same mask?\n\n> **A:** Our notation here follows the related work [1], which explains: Note that the prompt is asymmetric since the label for $x_{N+1}$ is excluded from the input. To reflect this asymmetric structure, the mask matrix $M$ is included in the attention.\n>\n> More specifically, if you pay attention to the $(N+1)$-th item, L138 is supposed to represent a causal mask.\n>\n>  (For $H\\in \\mathbb{R}^{(dout+din)\\times(N+1)}$, $HM=(h_1,..,h_N,h_{N+1})M=(h_1,..,h_N,0)=H_s$)\n>\n> Besides, this mask method is used in GLM training [2]: Part A tokens can attend to each other, but cannot attend to any\ntokens in B. Part B tokens can attend to Part A and antecedents in B, but cannot attend to any subsequent tokens in B. So it is reasonable for equation 1 and 2 use the same mask.\n>\n> [1] wangjun Ahn et al. Transformers learn to implement preconditioned gradient descent for in-context learning. NeurIPS 2023.\n>\n> [2] Zhengxiao Du et al. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. ACL 2022.\n\n> **W6:** Example 1 appears to be incorrect:\n>\n> - There is no parentheses around $W^k_{V_r}+ \\delta_Vh_i^{k−1}$ in the first line.\n> - The triangle inequality seems to say that line 2 ≥ line 1\n\n> **A:** Thank you for pointing out the typo (parentheses). We appreciate your attention to detail and will have made the necessary corrections. However, this minor typo does not affect the final conclusion. A detailed analysis is as follows:\n>\n>  In triangle inequality, it is ture that $||\\mathbf{a}||_2+||\\mathbf{b}||_2\\geq ||\\mathbf{a}+\\mathbf{b}||_2$ for any vector $\\mathbf{a},\\mathbf{b}$ (same dimension). \n>\n>  **But in this Example, we did not use the triangle inequality**:\n>\n> $||W^k\\_{V\\_r}h\\_i^{k−1}+\\delta\\_Vh\\_i^{k−1}||\\_2^2=||W^k\\_{V\\_r}h\\_i^{k−1}||\\_2^2+2(W^k\\_{V\\_r}h\\_i^{k−1})^T(\\delta\\_Vh\\_i^{k−1})+||\\delta\\_Vh\\_i^{k−1}||\\_2^2=||W^k\\_{V\\_r}h\\_i^{k−1}||\\_2^2+||\\delta\\_Vh\\_i^{k−1}||\\_2^2$. \n>\n> Note that $W^k_{V_r}=U_{:r}\\Sigma_{:r}V_{:r}^T$ and $\\delta_V=U_{r:}\\Sigma_{r:}V_{r:}^T$, and $U_{:r},U_{r:}$ are **orthometric** (properties of SVD). \n>\n> Therefore, $(W^k_{V_r}h_i^{k−1})^T(\\delta_Vh_i^{k−1})=(h_i^{k−1})^TV_{:r}\\Sigma_{:r}(U_{:r}^TU_{r:})\\Sigma_{r:}V_{r:}^Th_i^{k−1}=0$, line 2 = line 1.\n\n> **Q1:** I don't believe the clipping process was adequately explained. Once the SVD operation is done, do you clip starting from the largest singular value? or starting from the smallest?\n\n> **A:** As mentioned in **Section 2**, \"The optimal rank-r approximation and SVD,\" we retain the components corresponding to the largest r singular values. Therefore, the clipping process begins with the smallest singular values.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful to all reviewers for their detailed and constructive feedback! We are encouraged to see that reviewers find:

> - **Reviewer itEx**: It provides a detailed theoretical analysis on why SVD based weight pruning will improve ICL performance...... It provides the theoretical insight of SVD based method. The authors provide a simple algorithm to **leverage the discovered phenomenon** to improve ICL performance of LLM in a gradient-freee way.
>
> - **Reviewer Zh6a**: This work conducts an in-depth analysis to explain the "stability" of transformer weight pruning across different layers. The framework is interesting and validated through experiments. Moreover, the **theoretical analysis can be applied to design new algorithms** like algorithm 1 in this paper.
>
> - **Reviewer Nxqr**: The Authors provide a theoretical analysis to explain their empirical findings..... Furthermore, they propose a simple, derivative-free algorithm for enhancing ICL performance in downstream tasks, **demonstrating the practical value of their theoretical insights**.
>
> - **Reviewer CqwV**: The method intuitively makes sense and is something which can be **conditionally tuned after training** based on specific tasks if a validation set is available.

We have addressed all the questions raised by the reviewers through detailed clarifications, providing separate responses to each reviewer. Additionally, we would like to address some common concerns in a consolidated global response.

> **(1) The main contribution of this work.**
>
> Our primary objective is to provide a general **theoretical framework** that reveals the underlying mechanism behind the phenomenon that SVD-based weight pruning can enhance ICL performance.  Based on our theoretical insights, we can design new algorithms to enhance ICL performance. Consequently, we did not compare our approach with other pruning methods. Algorithm 1 is presented solely to illustrate how theoretical analysis can guide experimental procedures effectively.
>
> Specifically, in **Theorem 2**: Modulating the F-norm of $[G\_t]\_1^L$ or $[\Delta W\_{t}]\_1^L$​  could enhance performance when utilizing ICL.  So, there may be other potential **weight-based** pruning methods. For instance: magnitude-based pruning [1], which can directly control the $||A||\_F=\sqrt{\sum\_i\sum\_ja\_{ij}^2}$.
>
> Of course, there are also some **layer-based** pruning methods, as mentioned in (**Reviewer Nxqr Q1**) and (**Reviewer CqwV W3**). We would like to clarify the drawbacks of the drop-layer method under our theoretical framework：
>
> As stated in **Theorem 1**,  L layers with implicit gradient descent trajectories $[\Delta W_t]_1^L$/$[G_t]_t^L$. Consequently, dropping an entire layer leads to two main issues: 
>
> (a) Adjusting the weights for other downstream tasks becomes more challenging. 
>
> (b) The implicit gradient update is reduced by one step, which perhaps results in a decline in model performance.
>
> [1] Wen, W et al. Learning structured sparsity in deep neural networks. NeurIPS 2016. 

> **(2) The explanation for simplifications.**
>
> To facilitate qualitative analysis, our main text primarily focuses on the theoretical aspects of linear attention, a considerable portion of the works also consider from the linear attention to explore the ICL [2,3].  And our goal is to provide insightful conclusions under simplified conditions.
>
> However, we also discuss the standard Softmax attention setting and MLP in **Appendix B.2** and **Appendix B.3**:
>
> Specifically, it can be considered that the mapping function $\phi$, or rather the effect of the **Softmax** function, is to project the original features into a higher-dimensional space to capture more profound features. Subsequently, learning and meta-optimization are conducted under this **new feature space**. 
>
> $\hat{h}\_{N+1}=h\_{N+1} + \Delta W\_{icl}^{'}\phi(W\_Q h\_{N+1}),\ \Delta W\_{icl}^{'}=\frac{1}{D^{'}} \left[\sum\_{i=1}^N (W\_VH\_s)\_i \otimes \phi(W\_KH\_s)\_i \right]$ 
>
> For the MLP layer, it can be seen as a dimensional adaptation to correspond to the residual connection. Then, we can get the implicit gradient descent with MLP:  $\Delta W_{icl}^{''}=W_{MLP}\left( \sum_{i=1}^{N} W_V{h_i} \otimes W_K{h_i} \right) W_Q$
>
> [2] Ekin Akyiurek et al. What learning algorithm is in-context learning? investigations with linear models. ICLR 2023.
>
> [3] Johannes von Oswald et al. Transformers learn in-context by gradient descent. ICML 2023.

> **(3)  Why optimal clipping rate $\xi$ varies? How should Theorem 2 be interpreted?**
>
> (a) As we mentioned in **Section 3.3 and Section C.4** (which follows [4]),  the implicit gradients produced by Transformers in practical applications are noisy due to factors such as the extent of model pre-training and data characteristics (eg. ICL shot number/task difficulty). Therefore,  $[\Delta W_t]_1^L$/$[G_t]_t^L$ in **Theorem 1** have varies noise. **That is why** optimal $\xi$ varies. 
>
> (b) Expected generalization error (**Theorem 2**) = population risk ($L\_{\mu}$) - empirical risk ($L\_{H_s}$). 
>
> More specifically, on the one hand, **Theorem 2** shows that clipping weights controls the F-norm of the implicit gradient $([\Delta W\_t]_1^L/[G\_t]\_1^L)$ which helps reduce the expected generalization error. On the other hand, we can evaluate the empirical risk $(L\_{H\_s})$ by assessing the model's performance on the validation set. If the generalization error is known, it is possible to estimate the population risk ($L\_{\mu}$). Therefore, the most challenging aspect is addressing the generalization error.
>
> [4] Shivam Garg et al. What can transformers learn in-context? a case study of simple function classes. NeurIPS 2022.

Thank you once again to all the reviewers for your patience and invaluable comments. We hope our responses have clarified your initial concerns and questions. We are happy to provide further clarifications if necessary.

Meta-review: The reviewers generally acknowledge the importance of research on in-context learning (ICL) and largely agree that the proposed method is intuitive and simple and that the manuscript is insightful. It was initially brought up, that the theoretical analysis was limited in that it only explains why stable performance is achievable, but not why better performance can be achieved with SVD-based weight pruning. After clarification in the discussion phase, reviewers appeared to be convinced that the presented analysis was sufficiently detailed and provides an explanation of the empirical results. The authors’ response to the question by reviewer CqwV, on whether dropping a layer entirely may sometimes be the best option compared to pruning a layer’s weights, seems reasonable.
Overall, all reviewers agree that the work meets the standards of a NeurIPS publication and I recommend acceptance.
Decision: Accept (poster)