Title: Enhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspective
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large language models

In-Context Learning

SVD

Theoretical generalization bounds
Abstract: Pre-trained large language models (LLMs) based on Transformer have demonstrated striking in-context learning (ICL) abilities. With a few demonstration input-label pairs, they can predict the label for an unseen input without any parameter updates. In this paper, we show an exciting  phenomenon that SVD-based weight pruning can enhance ICL performance, and more surprising, pruning weights in deep layers often results in more stable performance improvements than in shallow layers. However, the underlying mechanism of those findings still remains an open question. To reveal those findings, we conduct an in-depth theoretical analysis by presenting the implicit gradient descent (GD) trajectories of ICL and giving the mutual information based generalization bounds of ICL via full implicit GD trajectories. This helps us reasonably explain the surprising experimental findings.  Besides, based on all our experimental and theoretical insights, we intuitively propose a simple, model-compression and derivative-free algorithm for downstream tasks in enhancing ICL inference. Experiments on benchmark datasets and open source LLMs display the method effectiveness.

Enhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspective

Abstract

Pre-trained large language models (LLMs) based on Transformer have demonstrated striking in-context learning (ICL) abilities. With a few demonstration input-label pairs, they can predict the label for an unseen input without any parameter updates. In this paper, we show an exciting phenomenon that SVD-based weight pruning can enhance ICL performance, and more surprising, pruning weights in deep layers often results in more stable performance improvements than in shallow layers. However, the underlying mechanism of those findings still remains an open question. To reveal those findings, we conduct an in-depth theoretical analysis by presenting the implicit gradient descent (GD) trajectories of ICL and giving the mutual information based generalization bounds of ICL via full implicit GD trajectories. This helps us reasonably explain the surprising experimental findings. Besides, based on all our experimental and theoretical insights, we intuitively propose a simple, model-compression and derivative-free algorithm for downstream tasks in enhancing ICL inference. Experiments on benchmark datasets and open source LLMs display the method effectiveness 2 .

1 Introduction

Recently, large language models (LLMs) based on the Transformer architecture [43] have emerged striking in-context learning (ICL) capabilities: Given a prompt containing demonstration sample and a test data, the model can make a prediction for the test data and achieve excellent performance without any parameter updates [7, 26, 34, 6]. This leads considerable works that aim to shed light on it [54, 21, 59, 3, 10, 16, 2, 46, 35] .

In this paper, we show our surprising findings in ICL inference by experimentally analyzing the effect of singular value decomposition (SVD)-based pruning on the model performance at different depth layers. As demonstrated in Figure 1: (i) SVD-based weight pruning can enhance ICL performance. Across all cases, it is evident that SVD-based weight pruning can generally enhance model performance at various depth layers, compared to the baseline performance without weight pruning (indicated by the dashed lines); (ii) Pruning weights in deep layers often results in more stable performance improvements than in shallow layers. Specially, deep layers weight matrices can be drastically reduced without much degradation in model performance, and may even get large improvements on it, while the model performance collapses after a sharp reduction at the shallow layers (see Section 2 for details). A similar case in Sharma et al. [38] notes that a large portion of singular values can be removed from linear layers in large language models without affecting or even improving reasoning performance. However, the underlying mechanism of this phenomenon still remains a mystery, and this paper seeks to explore the issue from the following two aspects.

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.4

0.5

0.6

0.7

0.8

0.9

Accuracy/F1 (0-1)

0.8364

0.8606

0.8598

0.8293

SST-2&GPT-J-6B&MLP

layer0

layer1

layer26

layer27

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.40

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

Accuracy/F1 (0-1)

0.7829

0.8076

0.7921

SST-2&GPT-J-6B&ATTN

layer0

layer1

layer26

layer27

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Accuracy/F1 (0-1)

0.9120 0.9185

0.9164

0.9243

SST-2&LLAMA2-7B&MLP

layer0

layer1

layer30

layer31

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Accuracy/F1 (0-1)

0.9120

0.9187

0.9134

SST-2&LLAMA2-7B&ATTN

layer0

layer1

layer30

layer31

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.40

0.42

0.44

0.46

0.48

0.50

0.52

0.54

0.56

0.58

Accuracy/F1 (0-1)

0.5539

0.5489

0.5664

0.5489

RTE&GPT-J-6B&MLP

layer0

layer1

layer26

layer27

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.46

0.48

0.50

0.52

0.54

0.56

0.58

Accuracy/F1 (0-1)

0.5614

0.5489

0.5413

RTE&GPT-J-6B&ATTN

layer0

layer1

layer26

layer27

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.40

0.45

0.50

0.55

0.60

Accuracy/F1 (0-1)

0.5614 0.5639

0.5990

0.5639

RTE&LLAMA2-7B&MLP

layer0

layer1

layer30

layer31

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.40

0.45

0.50

0.55

0.60

Accuracy/F1 (0-1)

0.6115

0.5714

0.5940

0.5639

RTE&LLAMA2-7B&ATTN

layer0

layer1

layer30

layer31

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.46

0.48

0.50

0.52

0.54

0.56

0.58

0.60

0.62

Accuracy/F1 (0-1)

0.5500

0.5450

0.5700

0.6100

COPA&GPT-J-6B&MLP

layer0

layer1

layer26

layer27

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.48

0.49

0.50

0.51

0.52

0.53

0.54

0.55

0.56

Accuracy/F1 (0-1)

0.5450

0.5350

0.5350

0.5500

COPA&GPT-J-6B&ATTN

layer0

layer1

layer26

layer27

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.40

0.45

0.50

0.55

0.60

0.65

0.70

Accuracy/F1 (0-1)

0.6800

0.6200

0.6450

0.6200

COPA&LLAMA2-7B&MLP

layer0

layer1

layer30

layer31

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.50

0.52

0.54

0.56

0.58

0.60

0.62

0.64

Accuracy/F1 (0-1)

0.6400

0.6350

0.6350

0.6250

COPA&LLAMA2-7B&ATTN

layer0

layer1

layer30

layer31

no SVD

Why this phenomenon? We conduct an in-depth theoretical analysis to explain the findings. To be more specific, we first analyse the ICL form with Transformer from the perspective of ICL as implicit gradient descent fine-tuning, and we present the implicit gradient descent trajectories of ICL in Theorem 1 in Section 3.2. Afterwards, we use the information-theoretic approaches [55, 32, 51] to give the generalization bounds of ICL via full implicit GD trajectories in Theorem 2 in Section 3.3, explaining ( Q1 ) why SVD-based weight pruning can enhance ICL performance? and ( Q2 ) why do deep and shallow layers exhibit different behaviors when their weight matrices are drastically reduced?

How to better use this phenomenon (Q3)? Based on all our experimental and theoretical insights, we intuitively propose a simple, derivative-free and model-compression algorithm in Section 3.4 for downstream tasks in enhancing ICL inference, providing developers and researchers with an effective approach for handling downstream tasks. Experimental results for our algorithm on benchmark datasets [39, 58, 8, 12, 5, 11, 18] and open source LLMs [48, 42] verify that the method can visibly influence performance across different language understanding tasks in ICL inference.

Our primary objective is to establish a general theoretical framework that uncovers the underlying mechanism behind the phenomenon that SVD-based weight pruning can enhance ICL performance. Based on our theoretical insights, one can design new ICL algorithms. Accordingly, we did not directly compare our approach with other pruning methods. The algorithm in Section 3.4 is presented solely to illustrate how theoretical analysis can guide experimental procedures effectively.

1.1 Related Works

Model compression. In recent years, there has been growing theoretical and experimental evidence that models can be significantly pruned with very little drop in accuracy, thereby significantly reducing their storage requirements. To name a few, Frankle and Carbin [15] indicate that neural networks can typically have over 90% of their weights eliminated with little to no loss in performance. Analogous results have been demonstrated in both feed-forward and convolutional networks used for image classification [24, 49, 22, 37]. More specifically, Sharma et al. [38] present that careful pruning done at specific layers of Transformer models can produce significant boosts in performance on some tasks. The discovery of this phenomenon heightens interest in the connection between generalization and over-parameterization [56, 57], it also spurs research into developing pruning strategies that facilitate efficient model inference [30]. Additionally, the works [19, 40] find that performance remains nearly unchanged until a significant portion (up to half) of the layers are removed.

In-context learning and gradient descent. In order to better understand the ICL capabilities, considerable works try to understand ICL capabilities from the perspective of gradient descent. Irie et al. [21] and Dai et al. [10] explain ICL as implicit fine-tuning by figuring out a dual form of gradient descent for linear Transformer attention. However, the linear attention setting is less commonly used than Softmax attention in the LLMs and the details of gradient descent such as the choice of loss function and training data have not been clearly defined. Therefore, by using weight construction, von Oswald et al. [46] show the equivalence of linear self-attention mechanism and gradient descent on a linear regression task and Akyiurek et al. [3] prove that based on gradient descent and closed-form ridge regression, Transformers can learn linear models as learning algorithms. Further without using weight construction, Ren and Liu [35] connect Softmax attention with kernels and then give a novel interpretation that ICL with Transformer is really equivalent to a contrastive learning pattern.

Information-theoretic generalization bounds. The information-theoretic generalization bounds have been developed to analyze the expected generalization error of a learning algorithm. Given that they are dependent of distribution and algorithm, they are ideal tools to study the generalization behaviour of models performed with a specific algorithm. Russo and Zou [36] and Xu and Raginsky [55] first propose the Mutual information (MI) based bounds, which are then strengthened by additional techniques [4, 31, 20, 51]. Specifically, Negrea et al. [31] derive MI-based bounds by developing a PAC-Bayes-like bounding technique and Wang and Mao [51] develop generalization bounds for SGD via constructing an auxiliary iterative noisy process.

2 SVD-Based Weight Pruning can Enhance ICL

This section shows our surprising findings in ICL inference by performing a motivating analysis of three benchmark datasets in conjunction with two open source LLMs, the details are as follows. We choose GPT-J (6B,28 layers) [48] and LLAMA2 (7B,32 layers) [42] as our primary models for investigation, owing to their robust ICL performance and moderate model size, which align with our hardware resource. The attention (ATTN) layers are made up of key, query, value, out matrices both in GPT-J-6B and LLAMA2-7B. The mlp (MLP) layers in GPT-J-6B consist of input, output matrices, while the MLP layers in LLAMA2-7B are made up of up, gate, down matrices. For datasets, we use SST-2 [39] for sentiment analysis, RTE [5] for textual entailment and COPA [18] for causal inference. Details regarding datasets and how they were used are shown in Appendix C.1 and C.3.

Specially, we use the optimal rankr approximation mentioned later as SVD-based weight pruning method to show the effect of weight pruning across different layer and module types in Section 2.1, then we further analyze the effect of the ICL shot numbers on it in Section 2.2.

The optimal rankr approximation and SVD. Given a matrix W ∈ R m × n and a constant r ≤ min( m,n ) , r ∈ N . Eckart-Young-Mirsky theorem [14] provides an optimal solution W ∗ ( r ) = U : r Σ : r V T : r , U : r , V : r are matrices containing the singular vectors corresponding to the largest r singular values [ σ ] r 1 . Let ξ = 1 -r min( m,n ) ∈ (0 , 1) be the clipping rate.

2.1 Effect of Weight Pruning across Different Layer and Module Types

We plot the results of applying various amounts of clipping rate ξ to each module matrices in the Transformer architecture on the corresponding evaluation index for each dataset, as depicted in Figure

1. These plots are grouped, such that each sub-figure corresponds only to the indicated type of dataset, model and module. Notice that this investigation mainly focuses on assessing the impact of weight pruning to the first two and the last two layers of the model to further clarify the impact of layer depth on the model's performance and different colors are used to distinguish between these layers.

All sub-figures clearly show an interesting phenomenon about these models in ICL inference: SVDbased weight pruning can enhance ICL performance in both shallow and deep layers across different module types. More surprising, deep layers weight matrices can be drastically reduced without much degradation in model performance, and may even get large improvements on it. This suggests that pruning weights in deep layers can effectively reduce model complexity while maintaining or enhancing model performance. And the model performance collapses to 0.5 (the expectation of a random guess in the binary task) after a sharp reduction at the shallow layers by contrast, indicating a higher sensitivity of the model to changes in shallow layers.

Based on the surprising findings in ICL inference mentioned above, three questions are certain to arise: ( Q1 ) Why SVD-based weight pruning can enhance ICL performance? ( Q2 ) Why do deep and shallow layers exhibit different behaviors when their weight matrices are drastically reduced? ( Q3 ) How can we better use the phenomena about ICL in downstream tasks? We will address the first two questions theoretically in Section 3.3 and give a heuristic algorithm in Section 3.4 to answer the last.

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.4

0.5

0.6

0.7

0.8

0.9

Accuracy/F1 (0-1)

0.8364

0.8606

0.8598

0.8293

GPT-J-6B&MLP&10-shot

layer0

layer1

layer26

layer27

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.40

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

Accuracy/F1 (0-1)

0.7829

0.8076

0.7921

GPT-J-6B&ATTN&10-shot

layer0

layer1

layer26

layer27

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Accuracy/F1 (0-1)

0.9120 0.9185

0.9164

0.9243

LLAMA2-7B&MLP&10-shot

layer0

layer1

layer30

layer31

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Accuracy/F1 (0-1)

0.9120

0.9187

0.9134

LLAMA2-7B&ATTN&10-shot

layer0

layer1

layer30

layer31

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.40

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80

Accuracy/F1 (0-1)

0.7229

0.7017

0.7280

0.7531

GPT-J-6B&MLP&4-shot

layer0

layer1

layer26

layer27

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.4

0.5

0.6

0.7

0.8

0.9

Accuracy/F1 (0-1)

0.7236

0.8530

0.7047

0.7195

GPT-J-6B&ATTN&4-shot

layer0

layer1

layer26

layer27

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Accuracy/F1 (0-1)

0.8908

0.9059

LLAMA2-7B&MLP&4-shot

layer0

layer1

layer30

layer31

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Accuracy/F1 (0-1)

0.8908

0.9134

0.9048

0.8965

LLAMA2-7B&ATTN&4-shot

layer0

layer1

layer30

layer31

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.48

0.50

0.52

0.54

0.56

0.58

0.60

Accuracy/F1 (0-1)

0.5152

0.5465

0.5288

0.5885

GPT-J-6B&MLP&0-shot

layer0

layer1

layer26

layer27

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.490

0.495

0.500

0.505

0.510

0.515

0.520

Accuracy/F1 (0-1)

0.5032

0.5022

0.5074

0.5025

GPT-J-6B&ATTN&0-shot

layer0

layer1

layer26

layer27

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.4

0.5

0.6

0.7

0.8

0.9

Accuracy/F1 (0-1)

0.8220

0.7136

0.7184

0.7482

LLAMA2-7B&MLP&0-shot

layer0

layer1

layer30

layer31

no SVD

0

0.1

0.5

0.75

0.9

0.95

0.99

0.995

Clipping Rate(0-1)

0.40

0.45

0.50

0.55

0.60

0.65

0.70

Accuracy/F1 (0-1)

0.70140.7026 0.7065

LLAMA2-7B&ATTN&0-shot

layer0

layer1

layer30

layer31

no SVD

2.2 Effect of Different ICL Shot Numbers

Given that ICL can achieve higher performance with more demonstrations [23, 1], we further analyze the non-ignorable effect of different ICL shot numbers. To control for other influencing factors, we focus on SST-2 dataset [39] and retain the same test set for a single random seed as Section 2.1.

As shown in Figure 2, we compare the settings of three different ICL shot numbers: 0, 4 and 10. Following this, we analyze how the phenomenon changes across different shot numbers.

Firstly, we note that without weight pruning, the performance of the model improves with an increase in the number of ICL shots. Which is consistent with prior works. Besides, for every shot number, a phenomenon consistent with what is described in Section 2.1 is observed: SVD-based weight pruning can enhance ICL performance, pruning deep layer weight is more stable than pruning shallow weight. Last but not the least, Figure 2 also demonstrates roughly that with a decrease in the number of shots, the rate of performance collapse in the model slows down after a sharp reduction at the shallow layer. Intuitively, this is because LLMs exhibit a shift in focus in the ICL setting, which results in a reduced scope of the output space. The more shots there are, the more pronounced the shift becomes, and this also leads to a faster collapse. We also theoretically discuss it in Section 3.2 and 3.3 ( Remark 6 ).

3 Theoretical Analysis Results

In this section, we first describe the core components of our study by reviewing some basic notations in Section 3.1, and present the implicit gradient descent trajectories of ICL in Section 3.2. Afterwards, we give a mutual information based generalization bounds of ICL via full implicit GD trajectories in Section 3.3. Based on all our experimental and theoretical insights, we intuitively propose a derivative-free and effective method for downstream tasks in enhancing ICL inference in Section 3.4. Complete proofs can be found in the Appendix. For ease of qualitative analysis, our theoretical analysis is mainly focuses on linear attention setting. We discuss the standard Softmax attention setting in Appendix A.2 and feed-forward (MLP) layers in Appendix A.3.

3.1 Preliminaries

We let H be the instance space and µ be an unknown distribution on H , specifying random variable h . In ICL setting, the model accepts a sequence of input H = [ H s , h N +1 ] drawn i.i.d. from µ , where H s = [ h 1 , h 2 , ..., h N ] represents the demonstration sample and h N +1 is the test data. In the information-theoretic analysis framework, we let W ∈ R d be the space of hypotheses related to the model, and Transformer performs an implicit stochastic learning algorithm A which takes the demonstration sample H s as its input and outputs a hypothesis W ∈ W according to some conditional distribution Q W | H s . Similar to previous works [55, 51], we give the definition of expected generalization error.

Expected generalization error. Given a loss function ℓ : W×H→ R + , where ℓ ( w, h ) measures the "unfitness" or "error" of any h ∈ H with respect to a hypothesis w ∈ W . We take ℓ as a continuous function and assume that ℓ is differentiable almost everywhere with respect to w . The goal of learning is to find a hypothesis w that minimizes the population risk, and for any w ∈ W , the population risk is defined as L µ ( w ) ≜ E h ∼ µ [ ℓ ( w, h )] . However, since µ via the sample H s can only be partially observed, we instead turn to use the empirical risk, defined as L H s ( w ) ≜ 1 N ∑ N i =1 ℓ ( w, h i ) . Then the expected generalization error of A is defined as

where the expectation is taken over ( H s , W ) ∼ µ N ⊗ Q W | H s .

In-context learning with Transformers 3 . By prompt design, most context semantic understanding tasks can be unified into classification tasks. Simplify the form of each token in H to h i = [ x i , y i ] , where [ x i ] N 1 ∈ R din and [ y i ] N 1 ∈ R dout are encoded input text and corresponding labels respectively. The test token has the form h N +1 = [ x N +1 , mask ] , where mask is the label needed to predict, and usually set 0 as its initialization. Therefore, the form of attention with residual connection 4 is as follows:

where W V , W K , W Q ∈ R ( dout + din ) × ( dout + din ) are projection matrix, and the mask matrix M = ( I N × N 0 0 0 ) is included in the attention. Specifically, ˆ h N +1 can be formulated as

and for ease of qualitative analysis, Eq.(2) can be approximated as a relaxed linear attention mechanism by removing the Softmax operation and scale factor:

3.2 The Implicit Gradient Descent Trajectories of ICL

To begin with, we present the implicit gradient descent of ICL, inspired by [21, 10, 35]. These works describe how ICL with attention can be connected to a meta-optimizer which produces implicit gradient. The following lemma demonstrates the result.

Lemma 1 (The Implicit Gradient Descent of ICL in a Single Linear Attention Layer) . Consider a Transformer consists of a single linear layer attention with residual connection, parameterized by W V , W K , W Q as in Eq.(3). Same to Section 3.1, let H = [ H s , h N +1 ] be the input, where H s = [ h 1 , h 2 , ..., h N ] represents the demonstration sample and h N +1 is the test data. And let ˆ h N +1 be the single layer output. Then, it holds that

where W V H s is regarded as the meta-gradient of ICL, which is used to generate the implicit gradient matrix ∆ W icl to act on the final feature output. See Appendix A.1 for a proof.

Remark 1. It is worth noting that rank (∆ W icl ) ≤ rank ( H s ) ≤ N by rank relations in matrix multiplication, indicating that this is a low-rank operation. This explains why the effect of different ICL shot numbers is not uniform in Section 2.2. Details of the discussion in standard Softmax attention setting can be found in Appendix A.2. Plus, a significant body of work has discovered that the ICL capabilities of models are not robust to the order of ICL sample. For instance, Lu et al. [28] observed that large language models (LLMs) are sensitive to the sequence of ICL examples, and Liu et al. [27] reported that the context-based ICL performance exhibits a U-shaped curve-models tend to perform better with information that appears at the beginning or at the end of the input context. This aligns with observations from actual model training where the sequence of training samples affects outcomes especially when training with a batch size of 1 .

According to the above, we further analyze the implicit gradient descent trajectories of ICL and provide the following Theorem. We will denote ∆ W t icl by ∆ W t when there is no ambiguity, where t represents t -th layer.

Theorem 1 (The Implicit Gradient Descent Trajectories of ICL) . Consider a Transformer as a stack of L linear attention blocks with residual connection, parameterized by [ W t V ] L 1 , [ W t K ] L 1 , [ W t Q ] L 1 . Denote [ h t i ] N +1 1 as the output of the t -th layer, [ h 0 i ] N +1 1 as the initial input. Then for t ∈ [ L ] , it holds that

where ∆ W t ≜ ( ∑ N i =1 W t V h t -1 i ⊗ W t K h t -1 i ) W t Q , W 0 = 0 , W t = W t -1 + G t and G 1 = ∆ W 1 . See Appendix A.4 for a proof.

Remark 2. Note that the exclusion of Transformer weight ( [ W t K ] L 1 , [ W t Q ] L 1 , [ W t V ] L 1 ) implies that G t is only dependent on W t -1 and H s , this is consistent with gradient descent in terms of relevance.

3.3 Generalization Bounds of ICL via Full Implicit GD Trajectories

In this part, for simplicity of representation, we flatten the weight matrix into a vector form ( vec ( W t ) = W t ∈ R d , vec ( G t ) = G t ∈ R d ) and conduct our analysis within the weight and implicit gradient space of hypotheses W as detailed in Section 3.1. Notably, Ahn et al. [2] observe that, with the optimal parameters, a single layer linear Transformer implements a single step of preconditioned gradient descent and the preconditioning matrix not only adapts to the distribution of input data but also to the variance caused by data inadequacy. Garg et al. [16] show empirically that the trained Transformer is able to in-context learn the class of linear functions with respect to the prompt distribution, performing comparably to the optimal least squares estimator for any number of in-context examples considered, as exhibited in Figure 5a.

In addition, we also make a discussion on the noise of the implicit gradient in Section C.4, our analytical discussion indicates that the implicit gradients produced by Transformers in practical applications are noisy due to factors such as the extent of model pre-training and data characteristics (e.g., ICL shot number). We first present the assumption used in this subsection.

Let G t ≜ 1 N ∑ N i =1 ∇ ℓ i be the best implicit gradient of ICL that the model can produce, ˜ G t ≜ 1 b ∑ b i =1 ∇ ℓ i be the implicit gradient of ICL generated by the model in practical applications. N is the threshold and b is the the shot number in the actual input defined in Section C.4. And V t ≜ G t -˜ G t is the gradient noise caused by shot number, C t ≜ N -b b ( N -1) ( 1 N ∑ N i =1 ∇ ℓ i ∇ ℓ T i -G t G T t ) is the implicit gradient noise covariance. Similar to Wang and Mao [51]'s assumption in SGD, we approximate V t up to its second moment.

Assumption 1. Assume the implicit gradient noise V t follows a Gaussian distribution, i.e., V t ∼ N (0 , C t ) , then in ICL implicit gradient descent trajectories,

where N t ∼ N (0 , I d ) is the standard Gaussian 5 .

Remark 3. Regarding the validation of this assumption, empirical evidence from works [53, 25], suggests that SGD and Eq.(4) can achieve the similar testing performance. Additionally, we refer readers to some recent works [21, 16, 3, 10, 2, 46, 35], where the authors empirically verify that in-context learning of Transformer can achieve the similar testing performance to SGD. Together suggesting that studying Eq.(4) is arguably sufficient to understand generalization properties of ICL.

Theorem 1 indicates that the initial parameter W 0 = 0 , which is independent of all other random variables. And an L -layer Transformer does implicit GD of ICL stops after L updates, outputting W L as the implicit learned parameter. Our main results are mutual information based expected generalization error bounds of ICL, as presented in Theorem 2 .

Theorem 2 (The Generalization Bounds of ICL via Full Implicit Gradient Descent Trajectories) . Under the conditions of Theorem 1 and Assumption 1 , assume the implicit gradient noise covariance C t is a positive-define matrix, the loss ℓ ( w, h ) is R-subGaussian for any w ∈ W ∈ R d , then

where vec ( G t ) ∈ R d , ∆ W t ≜ ( ∑ N i =1 W t V h t -1 i ⊗ W t K h t -1 i ) W t Q and G t = ∆ W t (1 + W 0 + ∑ t -1 j =1 G j ) = ∆ W t (1 + W t -1 ) . tr {·} denotes the trace of a matrix, ∥·∥ F denotes the Frobenius norm of a matrix and E X Y is the conditional expectation. Proof details in Appendix A.5.

Remark 4 (Deal with Q1 ) . Theorem 2 indicates that one can control the generalization performance of ICL via controlling the implicit gradient norm along the entire ICL implicit GD trajectories. Specifically, modulating the norm of [ G t ] L 1 or [∆ W t ] L 1 may enhance performance when utilizing ICL. Note that controlling implicit gradient norm can also control the magnitude of the trace of implicit

gradient noise covariance. This elucidates why weight pruning through SVD, even if it only alters a single weight matrix, can confer advantages on the performance of Transformers in ICL inference. We will present an example below demonstrating how weight pruning can affect the norm of G t or ∆ W t , thereby influencing the expected generalization error. Additional examples provided in Appendix A.6. This is also why, as illustrated in Figure 1, the highest model performance sometimes occurs at a clipping rate of 0.995. Furthermore, this could elucidate the utility of normalization in Transformers. However, weight pruning also impacts the expressive power, implying that increased pruning does not invariably lead to better outcomes. This clarifies why the highest model performance may also occur at clipping rates lower than 0.995.

Example 1 (Prune W Q , W K , W V ) . Consider ∆ W k = ( ∑ N i =1 W k V h k -1 i ⊗ W k K h k -1 i ) W k Q when t = k . And we primarily consider the changes in a upper bound written as UB ( ∥ ∆ W k ∥ 2 F ) ≜ ∑ N i =1 ∥ ∥ W k V h k -1 i ⊗ W k K h k -1 i ∥ ∥ 2 F ∥ ∥ W k Q ∥ ∥ 2 F . Let r represents the remained rank and δ represents the potential noise consisting of parts with small singular values, that is, W k V = W k V r + δ V = U V : r Σ V : r ( V V : r ) T + δ V , the same operation is applied to W k Q and W k K as well. Then we have

where W k V r = U V : r Σ V : r ( V V : r ) T and δ V = U V r : Σ V r : ( V V r : ) T , and U : r , U r : are orthometric (properties of SVD), and further have

where Eq. (*) is by 6

and Eq. (**) is by ∥ P ∥ F = √∑ i σ 2 i ( P ) for any matrix A , B and ∥ P ∥ F ≥ ∥ P ( r ) ∥ F . UB ( ∥ ∆ W k ( r ) ∥ 2 F ) is the upper bound on ∥ ∆ W k ∥ 2 F after using SVD.

Remark 5 (Deal with Q2 ) . It is notable that G t is highly correlated with the sequence (∆ W 1 , ..., ∆ W t ) . More precisely, adjusting the weight matrix of the k -th layer, will invariably impact the norm of [∆ W t ] L t>k , further influence [ G t ] L t>k . When we calibrate the weight matrix of the k -th layer, the span of affected weight updates ∆ W t encompasses L -k + 1 matrices, this indicates that the deeper the layer of the adjusted parameters is, the fewer the number of G t affected. It also suggests that tweaks to the deep layers yield a more steadfast influence on the global norm of [ G t ] L 1 , thereby exerting a steadier influence over generalization performance. The enhanced stability observed in adjusting deeper layers, as demonstrated in our experimental findings in Section 2, can be theoretically explained based on the analysis presented above.

Remark 6 (How should Theorem 2 be interpreted?) . Expected generalization error ( Theorem 2 ) = population risk ( L µ ) - empirical risk ( L H s ). More specifically, on the one hand, Theorem 2 shows

that clipping weights controls the F-norm of the implicit gradient ([∆ W t ] L 1 / [ G t ] L 1 ) , which helps reduce the expected generalization error. On the other hand, we can evaluate the empirical risk ( L H s ) by assessing the model's performance on the validation set. If the generalization error is known, it is possible to estimate the population risk ( L µ ). Therefore, the most challenging aspect is addressing the generalization error. Furthermore, we will illustrate the interpretation of Theorem 2 from the perspectives of pruning methods and the ICL shot numbers. (i) Adjusting the F-norm of ([∆ W t ] L 1 / [ G t ] L 1 ) could enhance performance when utilizing ICL, suggesting that other weight-based pruning methods may also be effective. For example, magnitude-based pruning [52] directly controls the matrix F-norm. Certainly, there are also some layer-based pruning methods (e.g., drop-layer method [19]), we discuss in detail in Appendix B.6. (ii) Reducing the ICL shot numbers from N to N ′ can indeed lead to more robust outcomes specifically in the 'SVD weight pruning' operation, rather than in the model's overall performance. Experimentally, in Section 2.2, we analyze the effect of different ICL shot numbers. As shown in Figure 2, with a decrease in the number of shots, the rate of performance collapse in the model slows down after a sharp reduction at the shallow layer. Theoretically, ∆ W ( N ) -∆ W ( N ′ ) = ( ∑ N i = N ′ +1 W V h i ⊗ W K h i ) W Q , suggesting that the implicit gradient for N is expected to be more sensitive than that for N ′ .

3.4 Applications of Our Theoretical Understanding

In this portion, we deal with Q3 . We further explore the relationship between model performance improvement and SVD, as illustrated by a simple case study.

Case 1. Assuming that W has two unit eigenvectors x 1 , x 2 , based on the properties of eigenvectors:

and for a vector b on the hyperplane, it can be decomposed into a linear combination of two eigenvectors:

where l 1 , l 2 are coefficients and λ 1 , λ 2 are eigenvalues. Notice that if λ 1 ≫ λ 2 , when point b moves in the direction of x 1 , the value of l 1 changes but the solution set x changes insignificantly. Conversely, if it moves in the direction of x 2 , the value of l 2 changes and the solution set x changes dramatically, this is an ill-conditioned 7 problem whose solution is highly biased under small perturbations.

Given that SVD can be considered to play a role in noise reduction in ICL inference, and that this noise can cause significant disturbances to model output. We thus introduce the matrix ill-conditionedness, which can be measured by the Matrix condition number defined as follows.

Matrix condition number 8 . Cond ( W ) = || W || p || W -1 || p . Specifically, the condition number Cond ( W ) = σ max σ min when p = 2 , where σ max , σ min denote the maximum and minimum singular values respectively. Generally, for matrices of the same order, the higher the condition number of a matrix is, the greater its ill-conditionedness will be.

0

3

6

9

12

15

18

21

24

27

Layer

100

150

200

250

300

350

400

450

500

Cond(W)

2-norm condition number(GPTJ-6B)

mlp.fc_in

mlp.fc_out

0

3

6

9

12

15

18

21

24

27

Layer

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Cond(W)

1e6

2-norm condition number(GPTJ-6B)

attn.out_proj

attn.k_proj

attn.q_proj

attn.v_proj

0

4

8

12

16

20

24

28

Layer

0

25

50

75

100

125

150

175

200

Cond(W)

2-norm condition number(LLMMA2-7B)

mlp.up_proj

mlp.gate_proj

mlp.down_proj

0

4

8

12

16

20

24

28

Layer

0.0

0.2

0.4

0.6

0.8

1.0

Cond(W)

1e7 2-norm condition number(LLMMA2-7B)

attn.out_proj

attn.k_proj

attn.q_proj

attn.v_proj

Then we analyze the matrix condition number of the models as shown in Figure 3. We observe that the condition number of deeper layers (especially the last layers) in the model is generally higher, indicating that the condition number may serve as a reference for adjusting the model.

Based on all our exciting insights, we find it intuitive to design ICL improvements based on them, especially in downstream tasks. We propose a method where we first select layers with the top-k largest condition numbers and then identify the layer with the largest number among these. We perform a greedy search for the optimal clipping rate ξ ∗ on the validation set and subsequently evaluate it on the test set. This procedure is reported in Algorithm 1 in Appendix C.2.

Experimental setup. We conduct experiments for Algorithm 1 on widely adopted benchmark datasets, including SST-2 [39], AGNEWS [58], EMOC [8], MRPC [12], RTE [5], CB [11], COPA [18]. Please refer to Appendix C.1 and C.3 for more detailed dataset and prompt settings. And for models, we use the same models as in Section 2.

Experimental results. As Figure 4 shows, the results indicate that Algorithm 1 can visibly influence performance across different language understanding tasks in ICL inference. Specially, the effectiveness of Algorithm 1 in enhancing performance varies not only by the model but also significantly by the task, indicating a potential task-specific and model-specific threshold for the benefits derived from algorithmic enhancements. More notably, Algorithm 1 is a derivative-free optimization method and compresses the model to some extent, providing developers and researchers with an effective approach for handling downstream tasks. We also invite readers to refer to Appendices B.3, B.4 and B.5 for discussions on What would be the effect of pruning only a single module? , Why optimal clipping rate ξ varies? and What would happen if we apply the same clipping rate to other datasets ?

SST-2

AGNEWS

EMOC

MRPC

RTE

CB

COPA

0.0

0.2

0.4

0.6

0.8

Accuracy/F1 (0-1)

ICL(GPT-J-6B)

ICL(LLAMA2-7B)

ICL+Algorithm1(LLAMA2-7B)

4 Conclusion and Limitation

In this paper, we show our surprising findings in ICL inference: SVD-based weight pruning can enhance ICL performance both in shallow and deep layers across different module types, and pruning weights in deep layers often results in more stable performance improvements than in shallow layers. We conduct an in-depth theoretical analysis and explanation of these findings. Specifically, we first present the implicit gradient descent trajectories of ICL, afterwards, we give a mutual information based generalization bounds of ICL via full implicit GD trajectories. Based on all our experimental and theoretical insights, we intuitively propose a derivative-free and effective method for downstream tasks in enhancing ICL inference. However, further studies are required on (i) how to extend our generalization theory to a more standard Transformer architecture, (ii) do the results about ICL hold true for tasks beyond natural language processing and (iii) how to minimize the cost of searching the optimal clipping rate. Those will deepen our understanding of the ICL capabilities.

Acknowledgments

This research was supported by National Natural Science Foundation of China (No.62476277, No.6207623), Beijing Natural Science Foundation (No.4222029), CCF-ALIMAMA TECH Kangaroo Fund(No.CCF-ALIMAMA OF 2024008), and Huawei-Renmin University joint program on Information Retrieval. We also acknowledge the support provided by the fund for building worldclass universities (disciplines) of Renmin University of China and by the funds from Beijing Key Laboratory of Big Data Management and Analysis Methods, Gaoling School of Artificial Intelligence, Renmin University of China, from Engineering Research Center of Next-Generation Intelligent Search and Recommendation, Ministry of Education, from Intelligent Social Governance Interdisciplinary Platform, Major Innovation & Planning Interdisciplinary Platform for the 'DoubleFirst Class' Initiative, Renmin University of China, from Public Policy and Decision-making Research Lab of Renmin University of China, and from Public Computing Cloud, Renmin University of China.

References

Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning. arXiv preprint arXiv:2404.11018 , 2024.

Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. In Advances in Neural Information Processing Systems (37th edition) , pages 45614-45650, 2023.

Ekin Akyiurek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In International Conference on Learning Representations , 2023.

Amir Asadi, Emmanuel Abbe, and Sergio Verdú. Chaining mutual information and tightening generalization bounds. In Advances in Neural Information Processing Systems (32th edition) , pages 7245-7254, 2018.

Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In Text Analysis Conference , 2009.

Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An open-source autoregressive language model. In Proceedings of BigScience Episode #5 - Workshop on Challenges & Perspectives in Creating Large Language Models , pages 95-136, 2022.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (34th edition) , pages 1877-1901, 2020.

Ankush Chatterjee, Kedhar Nath Narahari, Meghana Joshi, and Puneet Agrawal. Semeval-2019 task 3: Emocontext contextual emotion detection in text. In Proceedings of International Workshop on Semantic Evaluation (13th edition) , pages 39-48, 2019.

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Representations , 2021.

Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models secretly perform gradient descent as meta-optimizers. In Findings of the Association for Computational Linguistics , pages 4005-4019, 2023.

Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. In Sinn und Bedeutung , volume 23, pages 107-124, 2019.

William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of International Workshop on Paraphrasing (3th edition) , 2005.

Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. arXiv preprint arXiv:2103.10360 , 2022.

Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika , 1 (3):211-218, 1936.

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Training pruned neural networks. In International Conference on Learning Representations , 2018.

Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. In Advances in Neural Information Processing Systems (36th edition) , pages 30583-30598, 2022.

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of Conference on Empirical Methods in Natural Language Processing (26th edition) , pages 5484-5495, 2021.

Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012) , pages 394-398, 2012.

Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17887 , 2024.

Mahdi Haghifam, Jeffrey Negrea, Ashish Khisti, Daniel M Roy, and Gintare Karolina Dziugaite. Sharpened generalization bounds based on conditional mutual information and an application to noisy, iterative algorithms. In Advances in Neural Information Processing Systems (34th edition) , pages 9925-9935, 2020.

Kazuki Irie, Róbert Csordás, and Jürgen Schmidhuber. The dual form of neural networks revisited: Connecting test time predictions to training patterns via spotlights of attention. In International Conference on Machine Learning , pages 9639-9659, 2022.

Mikhail Khodak, Neil Tenenholtz, Lester Mackey, and Nicolò Fusi. Initialization and regularization of factorized neural layers. In International Conference on Learning Representations , 2021.

Mukai Li, Shansan Gong, Jiangtao Feng, Yiheng Xu, Jun Zhang, Zhiyong Wu, and Lingpeng Kong. In-context learning with many demonstration examples. arXiv preprint arXiv:2302.04931 , 2023.

Yawei Li, Shuhang Gu, Luc Van Gool, and Radu Timofte. Learning filter basis for convolutional neural network compression. In Proceedings of the IEEE International Conference on Computer Vision , 2019.

Zhiyuan Li, Sadhika Malladi, and Sanjeev Arora. On the validity of modeling sgd with stochastic differential equations (sdes). In Advances in Neural Information Processing Systems (34th edition) , pages 12712-12725, 2021.

Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. Jurassic-1: Technical details and evaluation. Technical report, AI21 Labs, 2021.

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. In Transactions of the Association for Computational Linguistics (12th edition) , pages 157-173, 2024.

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming fewshot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 8086-8098, 2022.

J. Mercer. Functions of positive and negative type, and their connection with the theory of integral equations. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character , 209:415-446, 1909. URL http://www.jstor.org/stable/91043 .

Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations , 2017.

Jeffrey Negrea, Mahdi Haghifam, Gintare Karolina Dziugaite, Ashish Khisti, and Daniel M Roy. Information-theoretic generalization bounds for sgld via data-dependent estimates. In Advances in Neural Information Processing Systems (33th edition) , pages 11013-11023, 2019.

Yury Polyanskiy and Yihong Wu. Lecture notes on information theory. Lecture Notes for 6.441 (MIT), ECE 563 (UIUC), STAT 364 (Yale), 2019.

Zihan Qiu, Zeyu Huang, Youcheng Huang, and Jie Fu. Empirical study on updating key-value memories in transformer feed-forward layers. In International Conference on Learning Representations , 2024.

Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 , 2022.

Ruifeng Ren and Yong Liu. In-context learning with transformer is really equivalent to a contrastive learning pattern. arXiv preprint arXiv:2310.13220 , 2023.

Daniel Russo and James Zou. Controlling bias in adaptive data analysis using information theory. In Artificial Intelligence and Statistics , pages 1232-1240, 2016.

Steffen Schotthöfer, Emanuele Zangrando, Jonas Kusch, Gianluca Ceruti, and Francesco Tudisco. Lowrank lottery tickets: finding efficient low-rank neural networks via matrix differential equations. In Advances in Neural Information Processing Systems (36th edition) , pages 20051-20063, 2022.

Pratyusha Sharma, Jordan T Ash, and Dipendra Misra. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. In International Conference on Learning Representations , 2024.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pages 1631-1642, 2013.

Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Llm pruning and distillation in practice: The minitron approach. arXiv preprint arXiv:2408.11796 , 2024.

Yuandong Tian, Yiping Wang, Zhenyu Zhang, Beidi Chen, and Simon Du. Joma: Demystifying multilayer transformers via joint dynamics of mlp and attention. In International Conference on Learning Representations , 2024.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems (31th edition) , 2017.

Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in a transformer language model. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pages 63-76, 2019. URL https://aclanthology.org/W19-4808 .

Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head selfattention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 5797-5808, 2019. URL https: //aclanthology.org/P19-1580 .

Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning , pages 35151-35174, 2023.

Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems (33th edition) , 2019.

Ben Wang and Aran Komatsuzaki. Gpt-j-6b: A 6 billion parameter autoregressive language model. https://github.com/kingoflolz/mesh-transformer-jax , May 2021.

Hongyi Wang, Saurabh Agarwal, and Dimitris Papailiopoulos. Pufferfish: Communication-efficient models at no extra cost. In Proceedings of Machine Learning and Systems 3 , 2021.

Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In Empirical Methods in Natural Language Processing , 2023.

Ziqiao Wang and Yongyi Mao. Two facets of sde under an information-theoretic lens: Generalization of sgd via training trajectories and via terminal states. arXiv preprint arXiv:2211.10691 , 2022.

Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems (30th edition) , 2016.

Jingfeng Wu, Wenqing Hu, Haoyi Xiong, Jun Huan, Vladimir Braverman, and Zhanxing Zhu. On the noisy gradient descent that generalizes as sgd. In International Conference on Machine Learning , 2020.

Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations , 2021.

Aolin Xu and Maxim Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. In Advances in Neural Information Processing Systems (31th edition) , pages 2524-2533, 2017.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations , 2017.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021.

Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems (29th edition) , pages 649-657, 2015.

Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization. arXiv preprint arXiv:2305.19420 , 2023.

A Omitted Proofs and Additional Results

A.1 Proof of Lemma 1

Proof. Given that numerous studies [21, 10, 35] describe ICL with attention can be connected to a meta-optimizer which produces implicit gradient, we show the dual form between linear attention and gradient descent. First, consider a very simple linear layer,

where W ∈ R dout × din is the projection matrix. Given training inputs [ x i ] N i =1 ∈ R din with their labels [ y i ] N i =1 ∈ R dout and the loss function L with learning rate η , gradient descent process produces the corresponding back-propagation signals [ e i ] N i =1 ∈ R dout , where e i = -η ( ∇ y ′ i L ) and y ′ i = Wx i . During test time, we can use a trained linear layer,

where ⊗ denotes the outer product according to the chain rule of differentiation. On the other hand, this process can be associated with linear attention,

where [ k i ] N 1 , [ v i ] N 1 ∈ R din represent the key and value vectors respectively, forming the key and value matrix K , V ∈ R din × N in the attention mechanism. Based on Eq.(6), rewrite Eq.(5) as:

Then let's go back to the ICL process and approximate standard attention Eq.(2) as a relaxed linear attention mechanism by removing the Softmax operation and scale factor:

Where W 0 = 0 , and W V H s is regarded as the meta-gradient of ICL, which is used to generate the implicit gradient matrix ∆ W icl to act on the final feature output.

A.2 Softmax Attention and ICL Implicit GD

Inspired by Ren and Liu [35], by connecting Softmax Attention with Kernels, we can interpret ICL as a gradient descent process in contrast learning pattern. For simplicity, as in Section 3.1, scale factors are removed as follows:

where exp( · ) is element-wise. Following Choromanski et al. [9] and Ren and Liu [35], a Softmax kernel function K Softmax : R dout × R dout → R + is defined as:

where K gauss ( x , y ) = exp ( -∥ x -y ∥ 2 2 ) is the Gaussian kernel with a variance σ 2 = 1 . According to Mercer's theorem [29], there exists some mapping function ϕ satisfying K Softmax ( x , y ) = ϕ ( x ) T ϕ ( y ) . Thus, rewrite Eq.(9) as:

then, connect with Eq.(2) we have,

where D ′ = 1 N ϕ ( W K H s ) T ϕ ( W Q h N +1 ) + ϕ ( W K h N +1 ) T ϕ ( W Q h N +1 ) represents a constant for the normalized attention scores. It can be considered that the mapping function ϕ , or rather the effect of the Softmax function, is to project the original features into a higher-dimensional space to capture more profound features. Subsequently, learning and meta-optimization are conducted under this new feature space. It is also noteworthy that rank (∆ W ′ icl ) ≤ rank ( H s ) ≤ N .

A.3 Feed-Forward and ICL Implicit GD

The LLMs based on the Transformer architecture not only contain an Attention mechanism, but also include feed-forward (MLP) layers. It can be estimated that the feed-forward layers account for approximately two-thirds of the parameters in a Transformer model [45, 44]. Thus, MLP layers are essential for LLMs and there has been considerable work hypothesizing and analyzing the model's feed-forward layers. For example, Geva et al. [17], Qiu et al. [33] show that feed-forward layers in Transformer based language models operate as key-value memories, Tian et al. [41] propose Joint MLP/Attention (JoMA) dynamics by integrating out the self-attention layers in Transformers, analyzes joint training of MLP and self-attention layers, and qualitatively explains dynamics of multi-layer Transformers. Based on the inspiration above, we take MLP into consideration and rewrite Eq. (3) as:

where W V , W K , W Q ∈ R ( dim 1) × ( dout + din ) , W in ∈ R ( dim 2) × ( dim 1) , W out ∈ R ( dout + din ) × ( dim 2) are projection matrices and relu ( · ) is the activation function. And we can further relax the activation function for ease of qualitative analysis:

where W MLP ∈ R ( dout + din ) × ( dim 1) , which can be seen as a dimensional adaptation. And then do the similar derivation that we do in Appendix A.1, we can get ∆ W ′′ icl = W MLP ( ∑ N i =1 W V h i ⊗ W K h i ) W Q .

A.4 Proof of Theorem 1

Proof. Similar to A.1, Pay attention to the test token when i = N +1 ,

Thus, we give the implicit GD trajectories [ G t ] L 1 of ICL, where W t = W t -1 + G t ,

On the one hand, Eq.(13) shows G t = ∆ W t (1 + W t -1 ) , on the other hand, ∆ W t ≜ ( ∑ N i =1 W t V h t -1 i ⊗ W t K h t -1 i ) W t Q , which is only depend on ( [ h 0 i ] N 1 , [ W t K ] L 1 , [ W t Q ] L 1 , [ W t V ] L 1 ). So the exclusion of Transformer weight ( [ W t K ] L 1 , [ W t Q ] L 1 , [ W t V ] L 1 ) implies that G t is only dependent on W t -1 and H s .

And the prediction can be read from the corresponding position in L -th layer output h L N +1 as follows,

A.5 Proof of Theorem 2

The origin form of the mutual information based bound is predicated on a sample-specific MI, which quantifies the shared information between the output variable W and the input sample set H s . The following lemma shows the result:

Lemma 2. (Xu and Raginsky [55, Theorem 1.]). Assume the loss ℓ ( w, h ) is R-subGaussian for any w ∈ W , then

where I ( W ; H S ) = D KL ( Q W, H S ∥ Q W ⊗ Q H S ) is the mutual information and D KL denotes the KL divergence.

Unroll the terminal parameters' mutual information I ( W ; H S ) to the full trajectories' mutual information will get:

Lemma 3. (Wang and Mao [51, Lemma 4.]). Let Assumption 1 hold, then I ( W L ; H S ) ≤ ∑ L t =1 I ( -G t + C 1 / 2 t N t ; H S | W t -1 ) . Let -G t + C 1 / 2 t N t ≜ G t .

Proof. Recall Assumption 1 Eq.(4), Wang and Mao [51] get,

where Eq. (14) is by the data processing inequality (e.g., Z -( X,Y ) -( X + Y ) form a Markov chain then I ( X + Y, Z ) ≤ I ( X,Y ; Z ) ), Eq. (15) is by the chain rule of the mutual information, and learning rate η is dropped since mutual information is scale-invariant. I ( W 0 ; H S ) = 0 for W 0 is independent of all other random variables in Theorem 1 and Assumption 1 .

Besides, we present the variational representation of mutual information:

Lemma 4. (Polyanskiy and Wu [32, Corollary 3.1.]). For two random variables X and Y , we have I ( X ; Y ) = inf P E X [ D KL ( Q Y | X || P )] , where the infimum is achieved at P = Q Y .

Lemma 5. (Wang and Mao [51, Lemma 5.]). At every time step t , let P ˜ N t | W t -1 be any distribution satisfying D KL ( P G t | W t -1 ∥ P ˜ N t | W t -1 ) < ∞ , we have I ( G t ; H s | W t -1 ) = E W t -1 [ inf P ˜ Nt | Wt -1 E W t -1 H S [ D KL ( Q G t | H s ,W t -1 ∥ P ˜ N t | W t -1 ) ]] , where the infimum is achieved when P ˜ N t | W t -1 = Q G t | W t -1 . The KL divergence may be viewed as an estimate of the sensitivity of the full batch implicit gradient to a specific demonstration sample H s = H s .

From Lemma 5 , every choice of P ˜ N t | W t -1 gives rise to an upper bound of the MI of interest via I ( G t ; H s | W t -1 ) ≤ E W t -1 [ E W t -1 H s [ D KL ( Q G t | H s ,W t -1 ∥ P ˜ N t | W t -1 ) ]] . Same to Wang and Mao [51], choose an isotropic Gaussian prior P ˜ N t | W t -1 = N ( ˜ g t , σ 2 t I d ) , where both ˜ g t and σ t are only allowed to depend on W t -1 under Theorem 1 , and optimize the KL divergence in Lemma 5 over σ t for a fixed ˜ g t . Additionally, Under Theorem 1 where we define the implicit GD trajectories of ICL, assume C t is a positive-definite matrix, for any t ∈ [ L ] , we have,

where Eq. (16) is by the KL divergence between two Gaussian distributions: D KL ( p ∥ q ) = 1 2 [ log det(Σ q ) det(Σ p ) -k +( µ p -µ q ) T Σ -1 q ( µ p -µ q ) + tr(Σ -1 q Σ p ) ] , and Eq. (17) is due to the fact that G T t G t = tr( G t G T t ) and log det( C t ) = tr(log C t ) when C t is a positive-definite matrix. Let A 1 ( t ) = E H s [ ∥ G t -˜ g t ∥ 2 2 + tr { C t } ] and A 2 ( t ) = E H s [tr { log C t } ] and fix ˜ g t can rewrite Eq. (17) as,

where the optimal σ ∗ = √ A 1 ( t ) d when we take the derivative of σ t . Then combine Lemma 2 and Lemma 3 , we can finish the proof of the following lemma.

Lemma 6. Under the conditions of Theorem 1 , assume the implicit gradient noise covariance C t is a positive-define matrix, the loss ℓ ( w, h ) is R-subGaussian for any w ∈ W ∈ R d . For any t ∈ [ L ] , let ˜ g t be any constant vector for a given w t -1 , then

where A 1 ( t ) = E H s W t -1 [ ∥ G t -˜ g t ∥ 2 2 +tr { C t } ] , A 2 ( t ) = E H s W t -1 [tr { log C t } ] , vec ( G t ) = G t , tr {·} denotes the trace of a matrix and E X Y is the conditional expectation.

Further, if we let ˜ g t = 0 , reverse the process of flattening the weight matrix into a vector form, and by Eq. (19) we obtain

Then plug everything into Lemma 6 , we conclude the proof.

A.6 Examples of Remark 4

First, we will present several expressions that will be utilized later. For any matrix A and B , we have,

Same in Lemma 1 , ∆ W icl = ( ∑ N i =1 W V h i ⊗ W K h i ) W Q , let r represents the remained rank and δ represents the potential noise consisting of parts with small singular values. Some SVD controlling the norm of ∆ W icl examples are as follows.

Example 2 (Prune W Q ) . Suppose we decompose W Q by SVD, W Q = W Q r + δ Q = U Q : r Σ Q : r ( V Q : r ) T + δ Q , let z ≜ ( ∑ N i =1 W V h i ⊗ W K h i ) for simplicity.

where the inequality part takes advantage of the Eq. (20) , so the upper bound on ∥ ∆ W icl ∥ F is decreasing when using SVD.

Example 3 (Prune W V or W K ) . Suppose we decompose W V by SVD, W V = W V r + δ V = U V : r Σ V : r ( V V : r ) T + δ V , and consider the upper bound defined in Example 1 ,

where Eq. (21) is by Eq. (18), so the upper bound on ∥ ∆ W icl ∥ F is also decreasing. The same is true for decomposing W K .

Example 4 (Prune W MLP ) . Suppose we decompose W MLP by SVD, W MLP = W MLP r + δ MLP = U : r Σ : r V T : r + δ MLP , let z ≜ ( ∑ N i =1 W V h i ⊗ W K h i ) W Q for simplicity.

where recall Appendix A.3, ∆ W ′′ icl = W MLP ( ∑ N i =1 W V h i ⊗ W K h i ) W Q . And the inequality part takes advantage of the Eq. (20) , so the upper bound on ∥ ∥ ∥ ∆ W ′′ icl ∥ ∥ ∥ F is decreasing when using SVD.

B More Discussions

B.1 Explanation of the Mask Matrix

Our notation here follows the related work [2], which explains: Note that the prompt is asymmetric since the label for x N +1 is excluded from the input. To reflect this asymmetric structure, the mask matrix M is included in the attention. More specifically, if you pay attention to the ( N + 1) -th item, M is supposed to represent a causal mask (For H ∈ R ( dout + din ) × ( N +1) , HM = ( h 1 , ..., h N , h N +1 ) M = ( h 1 , ..., h N , 0) = H s ). Besides, this mask method is used in GLM training [13]: Part A tokens can attend to each other, but cannot attend to any tokens in B. Part B tokens can attend to Part A and antecedents in B, but cannot attend to any subsequent tokens in B. So it is reasonable for Eq.(1) and Eq.(2) use the same mask.

B.2 Definitions of Deep and Shallow

Indeed, there is no universally accepted definition for the terms 'deep' and 'shallow' as their interpretation can be subjective and dependent on the reference frame (e.g., model size). Intuitively, in this paper, our definition is similar to that of work [50]: 'shallow' layers refer to those closer to the input, while 'deep' layers are closer to the output. In Figure 1 and Figure 2 (Section 2), "shallow" typically denotes the first few layers, and "deep" denotes the last few layers of the network.

B.3 The Effect of Pruning Only a Single Module

In our theory, the example (pruning only W K or W V ) is provided in Appendix A.6, demonstrating how weight pruning can affect the norm of [∆ W t ] L 1 / [ G t ] L t . Thus, it may confer advantages on the performance of Transformers in ICL inference. However, in Theorem 2 ( Remark 5 ), it also suggests that modifications to the shallow layers have a less steady impact. Additionally, please review the supplementary experimental results provided below. (We choose key projection matrix W K and select the 3-layer (large matrix condition number in Figure 3) of GPT-J-6B, others are the same to Section 2.1).

B.4 Why Optimal Clipping Rate Varies?

As we mentioned in Section 3.3 and Section C.4, the implicit gradients produced by Transformers in practical applications are noisy due to factors such as the extent of model pre-training and data characteristics (e.g., ICL shot number/task difficulty). Therefore, [∆ W t ] L 1 / [ G t ] L t in Theorem 1 have varies noise. That is why optimal ξ varies.

Gradient quality derived from context (i.e., G t ).

In Theorem 1 (Remark 2), G t is only dependent on W t -1 and H s , this is consistent with gradient descent in terms of relevance (Conventionally, gradients in training are only related to the current parameters and the training samples).

In real-world training scenarios, SGD computes gradient by selecting a small batch of samples per iteration. This approach approximates the true gradient, inherently introducing noise. Similarly, in In-Context Learning, a small subset of samples (context examples) is used to generate implicit gradient (i.e., G t ), which also results in the introduction of noise.

B.5 Apply the Same Clipping Rate to Other Datasets

As we mentioned in Appendix B.4, [∆ W t ] L 1 / [ G t ] L t in Theorem 1 exhibit varying levels of noise, causing the optimal clipping rate to vary among different tasks, as it is dependent on the specific task and data. So if we apply the clipping rate (0.95) as used in SST-2 to other datasets, the model performance can either improve or deteriorate. Additionally, It is possible to conduct a certain number of experiments to find a range of optimal clipping rate that is broadly applicable.

B.6 What Would Happen if the Layer Was Dropped Entirely?

Dropping the layer could be the best option specifically for optimizing generalization error. Here's a detailed analysis:

(1) In our theoretical framework, we model each layer of the Transformer do a single iteration of implicit gradient descent (ICL) in Theorem 1 . This scientific analysis references [46, 3, 10]. (2) In Theorem 2 , L-layer Transformer:

Expected generalization error = population risk ( L µ ) - empirical risk ( L H s )

Expected generalization error = √ R 2 N ∑ L t =1 d log( ∥ ∆ W t ∥ 2 F ∥ G t ∥ 2 F ) (only show the main part). If you drop the entire layer, it will change from ∑ L t =1 to ∑ L -1 t =1 . Therefore, dropping the layer may in fact be the best option for generalization error.

(3) However, according to the traditional statistical learning viewpoint, performance can be bounded by the sum of optimization error and generalization error (Please see Remark 6 for 'How should Theorem 2 be interpreted?').

Thus, during the pruning process, there is a trade-off between optimization and generalization. Therefore, as pruning increases, the model's performance tends to first improve and then decline (the drop-layer method [19] does not harm but also does not improve), as demonstrated in the experiments shown in Figure 1.

In conclusion, dropping the entire layer can be a potential method (best option for generalization error) in our theoretical framework, but it may not necessarily be the best option for model performance.

C Extension to Experiments

Computational resources. We use a single NVIDIA GeForce RTX 3090 GPU and most tests run take 1-5 hours depending on dataset size and context length.

C.1 Prompts

C.2 Algorithm1

Algorithm 1

The details. Firstly, the details of the algorithm can be reviewed in the code provided. Specifically, the set of clipping rate candidates is predefined. In our study, the clipping rate candidates are set as shown in Figures 1 and 2: [0, 0.1, 0.5, 0.75, 0.9, 0.95, 0.99, 0.995]. Besides, we analyze the impact of different hyperparameters through comparative experiments (details in Section 2 and Section 3.4), as detailed below. (1) Clipping rate ξ :We search for the optimal ξ in the predefined clipping rate candidates. (2) Predefined module O : The module containing the target pruning weights, which can be chosen from [ k _ proj, q _ proj, v _ proj, out _ proj, fc _ in, fc _ out, all, mlp, attn ] . (3) Selected layer l : The layer containing the target pruning weights. For examlpe: In Section 2, we mainly focus on comparing the impact of weight pruning to the first two and the last two layers of the model. (4) ICL shot number : The demonstration number in ICL, we analyze the effect of different ICL shot numbers in Section 2.2.

C.3 Dataset Details

For experiments, we consider numerous classification datasets, including: SST-2 [39], AGNEWS [58], EMOC [8], MRPC [12], RTE [5], CB [11]. We also include a multiple-choice dataset COPA [18].

SST-2. SST-2 focuses on sentiment analysis, specifically the task of determining the sentiment of movie reviews. The dataset consists of sentences labeled as having a positive or negative sentiment. These sentences are divided into a training set, a validation set, and a test set. The training set includes about 67,349 sentences, whereas the validation set contains around 872 sentences. The test set comprises approximately 1,821 sentences without labels. We random sample N (10) data from the train set as demonstration sample, 8,000 data from the train set as validation set to search the optimal clipping rate, the rest 59,000+ data as test set.

AGNews. The AGNews dataset is a collection designed for text classification tasks, specifically for news categorization. It consists of news articles gathered from the AG's corpus of news on the web, which is categorized into four main topics: World, Sports, Business, and Science/Technology. In terms of structure, the AGNews dataset comprises approximately 120,000 training samples and 7,600 test samples. We random sample N (8) data from the train set as demonstration sample, 8000 data from the train set as validation set to search the optimal clipping rate, the 7,600 test samples as test set.

EmoC. The EmoC dataset focuses on contextual emotion detection in text. It consists of short text conversations extracted from three-turn English Tweets. The dataset is annotated with four emotion labels: happiness, sadness, anger, and others, we relabel the dataset with happiness and others. The training set includes 30,160 samples, while the test set comprises 5,509 samples. We random sample N (10) data from the train set as demonstration sample, 5,000 data from the train set as validation set to search the optimal clipping rate, the 5,509 test samples as test set.

MRPC. The MRPC dataset focuses on paraphrase detection in text. It consists of sentence pairs automatically extracted from online news sources. The dataset is annotated with binary labels indicating whether the sentences are paraphrases of each other. The training set includes 3,668 pairs, the validation set includes 408 pairs, while the test set comprises about 1,725 pairs. We random sample N (10) data from the train set as demonstration sample, the 408 pairs from validation set to search the optimal clipping rate, the 1,725 test samples as test set.

RTE. Recognizing Textual Entailment (RTE) task, which involves assessing the relationship between a pair of sentences-a premise and a hypothesis. The objective of the task is to determine whether the hypothesis can be logically inferred from the premise. Specifically, the model must evaluate whether the relationship between the two sentences is one of "entailment," where the content of the hypothesis is directly or indirectly supported by the premise, or "non-entailment," which includes both contradiction and neutrality. In RTE from SuperGLUE [47], the training set includes about 2,490 items, whereas the validation set contains around 277 items. We random sample N (10) data from the train set as demonstration sample, the remain items from train set to search the optimal clipping rate, the 277 validation samples as test set.

CB. The CommitmentBank (CB) dataset is part of the SuperGLUE benchmark and focuses on textual entailment with an emphasis on pragmatic inference. The task involves determining whether a hypothesis can be logically inferred from a given text, which in this dataset, typically comprises a premise followed by a hypothesis. The training set includes about 250 items, whereas the validation set contains around 56 items. We random sample N (15) data from the train set as demonstration sample, the remain 235 items from train set to search the optimal clipping rate, the 56 validation samples as test set.

COPA. The Choice of Plausible Alternatives (COPA) dataset is specifically designed to evaluate causal reasoning abilities in natural language processing models. This dataset presents a task where models must determine causal relationships within simple scenarios. Each question in COPA consists of a premise and two possible choices, one of which is the correct cause or effect of the premise. COPA's dataset is relatively straightforward and consists of 500 questions split evenly into training (400) and validation (100) sets. We random sample N (10) data from the train set as demonstration sample, the 200 items from train set to search the optimal clipping rate, the 100 validation samples as test set.

C.4 Noise Discussion of the Implicit Gradient

0

10

20

30

40

in-context examples

0.0

0.2

0.4

0.6

0.8

1.0

1.2

squared error

LR&500,000 Train Steps

Transformer

0

10

20

30

40

#in-context examples

0.0

0.2

0.4

0.6

0.8

1.0

1.2

squared error

LR&Different Train Steps Comparison

train steps: 25000

train steps: 50000

train steps: 75000

train steps: 100000

zero estimator

Figure 5: Evaluating the trained Transformer on in-context learning linear functions. (a) Garg et al. [16] consider the class of linear functions F = { f | f ( x ) = w T x, w ∈ R d } , in d dimensions where d = 20 . They sample x 1 , . . . , x k , x query , and w independently from the isotropic Gaussian distribution N (0 , I d ) . They then compute each y i = w T x i and construct the prompt as P = ( x 1 , y 1 , x 2 , y 2 , . . . , x k , y k , x query ) . This figure plots the normalized squared error of the Transformer (( M ( P ) -w T x query ) 2 /d ) , the errors are normalized so that the trivial zero estimator achieves an error of 1 (dashed line). Besides, when the number of in-context examples reaches the problem dimension d (here 20), least squares achieves 0 error while the Transformer achieves an error of 0.02. (b) We follow the same setting of [16] in (a) to compare the Trained Transformer with different train steps.

As mentioned in Garg et al. [16]'s work, the trained model is able to learn unseen linear functions from in-context examples with performance comparable to the optimal least squares estimator, this can be seen in Figure 5a. More importantly, the number of in-context examples (shot number) plays a significant role. In this case, the error approaches zero only when the shot number is greater than or equal to d . This indicates that the implicit gradients of ICL are influenced by the shot number, and there exists a threshold N , when the actual shot number b is below this threshold, the implicit gradients are noisy. On the other hand, following the same setting of [16] in figure 5a, we compare the performance of the Trained Transformer at different train steps as shown in figure 5b. This indicates that the performance of the model varies with the number of train steps, which also means that the implicit gradients of ICL generated by the model are influenced by the extent of its pre-training.

In conclusion, actual implicit gradient descent involves noise, which primarily stems from the shot number and the degree of model pre-training.

C.5 The Detailed Results of Algorithm 1

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The paper discusses enhancements in in-context learning (ICL) performance for pre-trained large language models (LLMs), particularly focusing on the effects of SVDbased weight pruning. Key contributions made include: What is the phenomenon? Why this phenomenon? How to better use this phenomenon?

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Section 4, we discuss the limitations of our work and identify them as valuable future work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The full set of assumptions and complete proofs can be found in the Appendix. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: First, our code is available at the anonymous url https://anonymous.4open. science/r/EnhancingICL_SVDPruning-5383/ (camera ready https://github. com/chen123CtrlS/EnhancingICL_SVDPruning ). Second, details regarding datasets and how they were used are shown in Appendix C.1 and C.3.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: For details on data and model download, please refer to the code. our code is available at the anonymous url https://anonymous.4open. science/r/EnhancingICL_SVDPruning-5383/ (camera ready https://github.com/ chen123CtrlS/EnhancingICL_SVDPruning ).

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: First, details regarding datasets and how they were used are shown in Appendix C.1 and C.3. Second, how hyperparameters were chosen is also detailed in paper and code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The primary focus of our paper is on the theoretical analysis of the underlying mechanism of a phenomenon we discovered. While we have conducted experiments on various publicly available datasets and different models to support our theoretical findings, we did not include discussions on error bars. Our experiments across different datasets and

models reflect to some extent the correspondence between theory and practice. And due to limitations in computational resources, we did not report error bars, confidence intervals, or conduct statistical significance tests.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In Section 3.4, we provide sufficient information on the computer resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics in every respect, and we make sure to preserve anonymity.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our work mainly focuses theoretically analysing the underlying mechanism of a phenomenon that we found. For positive societal impacts, understanding the underlying mechanism can contribute to use ICL. For negative societal impacts, the understanding of the mechanism can not be used in harmful ways.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original papers that produced the code packages or datasets. For example, in Section C.4 we cite the original owner of Figure 5a.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors provide a theoretical perspective on the stability of in context learning via implicit gradient descent trajectories. Ultimately, the analysis suggests that high condition numbers of the weight matrices belonging to layers with a high index can be pruned in order to achieve a model which performs better on ICL tasks.\n\nSTRENGTHS:\n- In context learning is important, and something which has not been studied as deeply as other topics of ML due to the recent rise of transformers and ICL in general.\n- The method intuitively makes sense and is something which can be conditionally tuned after training based on specific tasks if a validation set is available.\n\nWEAKNESSES:\n- It would be good to define deep and shallow, as these are subjective terms depending on the reference frame.\n- Figure 1 cpation says: \"We operate on the whole of MLP or ATTN.\" What does this mean?\n- If as figure 1 states, you can clip 99.5% of the original weights, what happens if you just drop that layer entirely? Recent work has shown that the deeper layers can be completely dropped without much effect. [1]\n  - I cannot see much benefit gained from pruning part of the weights with SVD when it seems that the in nearly all cases, the benefit can be had by dropping the layer entirely.\n  \n- Is the mask on L138 supposed to represent a causal mask? If so, I do not think the notation is correct, as the Identity matrix would only have $N$ binary values which is much less than is needed for a causal mask.\n- How can equation 1 and 2 use the same mask?\n- Example 1 appears to be incorrect:\n  - There is no parentheses around $W_{V_r}^k + \\delta_V h_i^{k-1}$ in the first line.\n  - The triangle inequality seems to say that line 2 $\\geq$ line 1\n  - Given the above, I do not see what conclusion can be drawn from this equation.\n  - Have I missed something here?\n\nQUESTIONS:\n- I don't believe the clipping process was adequately explained. Once the SVD operation is done, do you clip starting from the largest singular value? or starting from the smallest?\n\nLIMITATIONS:\nThe limitations have been discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the effect of singular value decomposition (SVD)-based weight pruning on the in-context learning (ICL) performance of large language models. \n\nThe Authors show that SVD-based pruning can enhance ICL performance, with deeper layers showing more stable improvements. \nThey provide theoretical analysis to explain these findings, presenting implicit gradient descent trajectories for ICL and deriving generalization bounds. \n\nBased on their insights, they propose a simple algorithm for enhancing ICL inference in downstream tasks.\n\nSTRENGTHS:\n- The Authors provide a theoretical analysis to explain their empirical findings, including the derivation of implicit gradient descent trajectories and generalization bounds for ICL.\n\n- Furthermore, they propose a simple, derivative-free algorithm for enhancing ICL performance in downstream tasks, demonstrating the practical value of their theoretical insights.\n\nWEAKNESSES:\n- The theoretical analysis primarily focuses on linear attention, which may not fully capture the complexities of standard Softmax attention used in most transformer models\n\n- The proposed algorithm is derivative-free, but the search for optimal clipping rates may still be computationally expensive for very large models or datasets\n\n- There is a substantial lack of comparison with other pruning methods: the study focuses on SVD-based pruning but doesn't compare it with other pruning techniques, which could provide context for the method's effectiveness\n\n- Poor language, frequent typos, and grammatical errors are significant issues in this paper. This does not help readability, and would likely be a barrier to publication in its current form.\n\n- An essential part of the paper, which is the discussion of related works is not part of the main text. Furthermore, this discussion is prone to criticism. For example, quoting the seminal paper by Frankle and Carbin as an example of low-rank properties of neural networks is clearly misleading. I think that this discussion should be an essential part of the main text, and should also be substantially revised in order to avoid conceptual confusions.\n\nQUESTIONS:\nI suggest the Authors to address 1) the concerns regarding a better framing of the work in the current literature. In particular, there is a large body of evidence that is growing on the resilience of LLMs to pruning (see for example \"The Unreasonable Ineffectiveness of the Deeper Layers\", Gromov et al, https://arxiv.org/pdf/2403.17887, and references therein), and 2) the quality of writings by proofreading it together with a native speaker.\n\nLIMITATIONS:\nLimitations are discussed in the final section (4).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper demonstrates that (1) SVD-based weight pruning can sometimes achieve better in-context learning performance, and (2) pruning weights in deeper layers often results in more stable outcomes compared to shallow layers. The authors explain their findings through theoretical analysis and propose an intuitive matrix condition number-based weight pruning algorithm to achieve both stable and improved performance.\n\nSTRENGTHS:\nThis work conducts an in-depth analysis to explain the \"stability\" of transformer weight pruning across different layers. The framework is interesting and validated through experiments. Moreover, the theoretical analysis can be applied to design new algorithms like algorithm 1 in this paper .\n\nWEAKNESSES:\nDespite adopting various simplifications (such as using a linear attention transformer without MLP and layer normalization, treating each in-context example as a single vector, implementing attention masks for query tokens, and using meta-gradients for in-context learning) in their theoretical analysis, the results are still limited. They only explain why SVD-based weight pruning can achieve \"stable\" performance, leaving the more intriguing question of why transformers can achieve \"better\" performance with pruning unclear. Additionally, even with detailed hyperparameter tuning, the effectiveness of Algorithm 1 remains uncertain. Further details are provided in the questions section.\n\nQUESTIONS:\n[1] How should Theorem 2 be interpreted? It seems only provide a weak upper bound for in-context learning stableness. Can this also be applied to empirical risk, such as $L_{H_S}$ and $L_\\mu$?\n\n [2] Theorem 2 gives the upper bound for expected generalization error. If we fix N in the constructed transformer and reduce the number of in-context examples to $N'$ in the input sequence, then we can find that while the factor $R^2/N$ remains unchanged in theorem 2 , $\\Delta W_t$ will change from $(\\sum_{i=1}^N)…$ to $(\\sum_{i=1}^{N'})…, where $ ($N' < N$). Based on the analysis across different layers, could this mean that fewer context examples are more robust for SVD weight pruning?\n\n[3] Note that in fig-3, large matrix condition numbers can exist in some modules of shallow layers, such as the attention key (K) in GPT-J-6B . What would be the effect of pruning only a single module in a shallow layer (e.g., the key projection matrix) rather than pruning the entire attention module (including Q, K, and V)?\n\n [4] In C.5, it's noted that the optimal clipping rate is sometimes very small and varies across datasets. What would happen if we apply the same clipping rate (e.g., 0.95) as used in SST-2 to other datasets?\n\nLIMITATIONS:\nSee Weaknesses and Questions. Besides, there seem many mistakes on pages 7 and 8, for example, the equation between line 228 and 229, and the equation for about F-norm in lines 230,231,232. These inaccuracies cast doubt on the overall reliability of the paper's findings. If there are any misunderstandings on my part, please point them out, and I will reconsider my evaluation of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses the phenomenon: SVD-based weight pruning can increase the in-context learning abilities of transformer based LLMs. In this paper, the authors conduct theorectical analysis by presenting the implicit gradient descent trajectories of ICL and providing the generation bounds visa full implicit gradient descent trajectories. This paper also provide a simple yet effective algorithm to clip the LLM by SVD to enhance ICL inference.\n\nSTRENGTHS:\nFirst, this paper has a clear writing and is easy to follow. \n\nIt provides a detailed theoretical analysis on why SVD based weight pruning will improve ICL performance by leveraging the implicit gradient descent trajectories. It also provides the generalization bounds of ICL, in Theorem 2, it can be inferred that the noise level and the norm of of gradient contribute to the error bound. It provides the theoretical insight of SVD based method.\n\nThe authors provides a simple algorithm to leverage the discovered phenomenon to improve ICL performance of LLM in a gradient-freee way. The ratio between $\\sigma_{max} $ and $\\sigma_{min}$ is a good choice of heuristic conditional number.\n\nWEAKNESSES:\n1. More details of algorithms is not shared. e.g. the range / number of clipping rate candidates set. \n2. In experiments result of C.5, the optimal $\\xi$ varies a lot across different tasks and different modules. However, this phenomenon is not touched in the theoretical part.\n\nQUESTIONS:\n1. Matrix condition number is an option for the indicator. But could there be more options, such as compute the decreasing rate of eigenvalues? Because when p=2, conditional number only leverages two values among all the eigenvalues.\n2. Could authors provide further more clarification why optimal $\\xi$ varies, and is there a way to explain this phenomenon under current theoretical framework provided in this paper?\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}