The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction
Authors: Pratyusha Sharma
Jordan T. Ash
Dipendra Misra
Venue: ICLR 2024 poster (2024)
Keywords: Large Language models
Transformers
Low-rank approximation
Primary area: generative models

Abstract: Transformer-based Large Language Models (LLMs) have become a fixture in modern machine learning. Correspondingly, significant resources are allocated towards research that aims to further advance this technology, typically resulting in models of increasing size that are trained on increasing amounts of data. This work, however, demonstrates the surprising result that it is often possible to significantly improve the performance of LLMs by selectively removing higher-order components of their weight matrices. This simple intervention, which we call LAyer-SElective Rank reduction (LASER), can be done on a model after training has completed, and requires minimal additional parameters and data. We show extensive experiments demonstrating the generality of this finding across language models and datasets, and provide in-depth analyses offering insights into both when LASER is effective and the mechanism by which it operates

THE TRUTH IS IN THERE: IMPROVING REASONING IN LANGUAGE MODELS WITH LAYER-SELECTIVE RANK REDUCTION
Pratyusha Sharma' Jordan Ash2* Dipendra M
Massachusetts Institute of Technology Microsoft Research NYC
ABSTRACT
Transformer-based Large Language Models (LLMs) have become fixture in modern machine learning. Correspondingly, significant resources are allocated towards research that aims to further advance this technology, typically resulting in models of increasing size that are trained on increasing amounts data. This work, however, demonstrates the surprising result that it often possible signif- icantly improve the performance of LLMS by selectively removing higher-order components" of their weight matrices. This simple intervention, which we call L.yer- SElectSEee Rank reduction (LASER), can be done on a model after training has completed, and requires minimal additional parameters and data. We show ex- tensive experiments demonstrating the generality of this finding across language models and datasets, and provide in-depth analyses offering insights into both when LASER is effective and the mechanism by which it operate
INTRODUCTION
Since their original release, Transformer-based LLMs have been shown to be remarkably proficient on a wide array of important machine learning tasks. Their underlying Transformer architecture has become state-of-the-an for modeling and reasoning about natural language, and has shown promise domains such as computer vision (Dosovitskiy et al., 2021) and reinforcement learning (Chen) al. 2021 as well.
Contemporary instantiations of Transformer architectures are infamously large, typically requiring tremendous compute resources for both training and inference. This is by design, as Transformers trained with more parameters and data are demonstrably more capable than their slimmer predeces- oots-often by a significant margin (Brown et al.) [2020] Touvron al. (023). Still, growing body of work suggests that Transformer- based models, and neural networks more generally, do not re- quire all fitted parameters retain their learned hypotheses. While it seems helpful to be massively pver-parameterized at train time (Hinton et al. 2014:] [Bengio et al.) 2005). it is well-known that these models can be drastically pruned at test time; neural networks can often have well over 90% of their weights removed without any significant degradation in performance (Frankle and Carbin, 2019) The discovery of this phenomenon bolstered interest around the relationship between gener- alization and werrpparametiization (Zhang et al.) 2017), and spawned research in developing pruning strategies that lend themselves to efficient model inference (Molchanov et al. 2017).
This paper presents a surprising finding, that careful pruning done at specific layers of Transformer models can produce significant boosts performance on some tasks. We describe LAyer SElective Rank reduction (LASER). intervention that removes higher-order components of learned weight matrices as identified by singular value decomposition. This reduction is performed in specific weight matrices and layers of the Transformer model. In line with previous work, find that many such matrices can be significantly reduced, and that performance degradation is often not observed until well over 90% of components are entirely removed. However, unlike what is found in previous work, we find that these reductions can produce drastic improvements in accuracy, as measured by
Asterisk indicates equal advising, a-3 listed. Correspondence pratyusha@iit.. edu.
"righer-order components are singular vectors with smaller singular values.
2 Code and wehsite: http:://paayussssaaaaaggihhhhhit

---

LASER
Σ
VT
Lincar Layer
Replace specific layers with low rank approxination
W
W
U
Self-Anentio Layer
Nonlinear activation
Point Poizt-wise Addition
Aelf-Attention
M
Layer
Capital
of
Japan
Tokyo
M
Figure I: LAyer SElective Rank reduction (LASER) replaces specific weight matrix W of the Transformer model with its rank-k approximation, WLR. and observes behavior. We find that this low-rank approximation, especially for MLP weights in the latter layers of the model, often offers surprising benefits to model performance.
the change in the model's
various well-studied reasoning benchmarks in NLP. Even better, our discovery appears to not be limited to natural language, with performance gains also found reinforcement learning.
This paper analyzes the relationship between the model's training data and samples that benefit from LASER. We find that improvements in the model's performance predominantly correspond to information less frequently present in the model's training data, suggesting that LASER offers kind of denoising procedure that makes weakly learned facts accessible. We separately observe that LASER affords increased robustness to paraphrases previously correct questions.
Additionally, we attempt to reason about what being stored in high-order components, such that their removal boosts performance. For questions correctly answered only after LASER, the original model (without any intervention applied) predominantly responds with high-frequency words such as "the," "of," etc-generations that are not even the same semantic type as the correct answer. However, after some amount of rank reduction, the model's 's answer flips to be correct. To understand this, we look at what the components removed by LASERencode on their own; we approximate the weight matrix using only its higher-order singular vectors. We find that these components describe either a different response of the same semantic category as the correct answer or generic high- frequency words. Seemingly, when noisy, higher-order components are combined with low-order components, their conflicting responses produce sort "average answer," which likely incorrect.
Figure 1 visualizes the Transformer architecture and the procedure followed by LASER. Here, the weight matrix a Multi-Layer Perceptron (MLP) layer is replaced with its low-rank approximation.
2 RELATED WORK
To our knowledge, this paper is the first to identify that carefully selected rank reduction interven- tions can boost Transformer performance. Still, there is a wide array of works that study related questions, including how facts are stored in LLMs and how to best compress neural networks.
How facts are stored. Studies probing model representation for the presence of select properties of entities (Ettinger et al. 2016 Adi et al. 2016 Hupkes et al., 2018 Conneau et al. 2018) argue that models store factual information across different layers fashion. However, there is conflicting evidence on how this information is organized and utilized in constructing answers in large language models. Some theories posit that information about different entities is locally stored as two-layer, key-value memory MLP sections Transformer models (Geva et al. 2021). which are thereafter copied over through latter layers by the self-attention modules (Elhage 2021 Meng et al. (2022) propose a procedure to trace and edit local, entity-specifie information to map to distinct "impossi- ble" outputs, supporting the locality theory. These theories are further supported by the phenomenon of "carly exiting." " where the representation at an intermediate layer can be directly used with the terminal head of the model correctly generate output (Zhao et al. 2021). In contrast, Hase al.
2

---

2023 have observed that information about overlapping entities or entity relations can be modified by editing a variety of layers in the model architecture, and therefore, that facts are stored across layers in fragmented fashion. This paper makes specific claims regarding locality, but rather demonstrates that higher-order components of a weight matrix introduce noise in decision making, and that considering only lower-order components may make correct answers accessible.
Model compression. Neural network pruning methods have found that models could be signif- icantly pruned (often removing over 90% of parameters) with very little drop in accuracy, signifi- cantly reducing the storage requirements of the model (I (LeCun et al., 1989 Hassibi and Stork 1992
Han et al., 2015; Li et al. 2017: Frankle and Carbin (019). There have also been approaches that
prune these models in a structured manner, to facilitate improvements in inference time (Molchanov
ct al. 2017). The existence of sparse sub-networks (Frankle and Carbin 2019: Hoefler et al. 2021
has been found to be true for convolutional, fully connected, and Transformer models (Lv et al. 2023 Murty et al., (023). While et al. (2022) find that model generalization can be improved by pruning and then refiting parameters. generalization improvements are only observed upon model retraining. To our knowledge, model pruning techniques have always done an indiscriminate reduc- tion across all parameters, without targeting any specific leyers-ledding to predictive performance either staying fixed or decreasing (Frankle and Carbin 2019). In this work, however, we find that the effect of reduction in accuracy is non-uniform across different layer types, and a model's general- ization can be improved by selective pruning alone; additional training is necessary. Roughly, we find that performance degradation can be produced by rank-reducing early layers, while significant performance benefits typically available by pruning later layers.
Low-rank approximations of weight matrices. Most pruning methods reduce parameters in order of their absolute magnitude (Frankle and Carbin 2019). An alternative approach, however, is to reduce the rank of its constituent weight matrices, keeping the top k components found by SVD. While matrices of neural models, including Transformer models, have been found to be well- approximated using this approach, where markedly reduced versions of a model can preserve much of its original behavior, research has shown that performance eventually declines as the severity of the intervention increases (Lv et al.) [023]] Hajimolahoseini et al., [021] Yu et al. 2017p. Note that these reductions are typically done universally, removing the same number of components in every weight matrix in the model. In contrast to these findings, we show that a targeted rank reduction, even affecting just single weight matrix, can offer benefits to the accuracy of Transformers.
Model distillation and low-rank training. Ba and Caruana (2014) and Hinton et al. (2014) have trained smaller networks to mimic the behavior of larger networks, suggesting neural networks might be significantly over ever-parametrized and can be replaced with leaner alternatives. To our knowledge, the model's predictions as a consequence of this procedure has been shown. Yang et al.) (2020) have enforced low-rank-ness of weight matrices for the purposes of memory efficiency, but the resulting models fail to achieve performance equivalent to their over- parametrized counterparts. The result suggests that over pver-paramttrization is helpful for the identifi-
no report of an improvement
cation of well geleraenera parameters by SGD (Bengio et al. 2005 Hinton et al. 2014).
3 PRELIMINARIES
Here review basic notation and describe the core components of our study.
Transformer Architecture. We provide a concise description of typical Transformer architec- ture that is relevant to our analysis. A Transformer is composed of L layers of Transformer blocks, each adjusting their input representation from the previous block using two sequential steps: a self- attention mechanism to consider information across time steps, and a feed-forward network to pro- cess information within each time step. Importantly, Transformer architecture has the following weight matrices W = [Wa. matrix for embedding input tokens. a projection weight matrix applied after the final layer before performing softmax, and some additional weights associated with layer normalization. This work primarily considers matrices in W. and intervenes by replacing them with low-rank approximations.
g: WR. W.. W.. Unn Uout) for each layer, in addition to an embedding
3 Various Transformer models often have small differences in how these transformations are implemented.
Our goal is not provide full survey of these details but to capture essential terminology for results.
3

---

Attention Key Matrix
Attention Query Matrix
Attention Value Mutrix
5.54
6.10
6.05
5.82
WMW
5.875
6.00-
5.80
595-
5.825
5.90-
5.78
5.775
5.74
5.750
5.75-
5.72
5.725
5.70
Attention Out Matrix
MLP Input Matrix
MLP Outnut Matrix
6.1
Loss
5.7
16
15
30
25
10
20
25
10
20
25
Layer number
Layer number
Layer number
Reduction %6:
10 25
40
50
60
75
90
92.5
95
97.5
98
98.5
199
99.5
99.75
The effect of rank reduction across different layer types is not uniform. Here we show the effect of rank reduction for GPT-J as studied on the CounterFact dataset. The dashed line is the um-modified network's loss. In the attention layers (key, query, value, out matrices), while clear matrices could be significantly rank-reduced without damaging the learned hypothesis, there is very little performance increase. However, for the multi-layer perceptron (MLP) layers, rank reduction goes from uniformly harming to improving the model's 's performance (around layer 20).
Figure 2:
Rank-r Approximation and SVD. Given a matrix W E mx and r E N, a rank-r approxima- tion problem requires finding matrix W that minimizes I Wll2 and satisfies rank W S
Fokart-Yougg-Mrske theorem provides optimal solution of this problem using Singular Value Decomposition (SVD) (Eckart and Young 1936). Formally, an SVD of a matrix W is given by W TTVT where U = [11 U2 ... Um gmxx and V [1), 02, U2 ... Un.] E g=xn and
E m2xx The column vectors of U and V constitute an orthonormal basis of R" and I respectively, and Σ is diagonal matrix whose entries the singular values of W in descend- ing order. One can also express the SVD of W as W E Emin(m.n) of (W)u,,;T According to Yokart-Young-irsky theorem, the matrix W = Li=1 (WW),r is an optimal solution to the rank-r approximation problem for any given desired rank r S min| ni.
EE
In this work, we will use the word higher-order components to refer to entries in the SVD cor- responding to the components with smaller singular values. These components are removed by LASER. The term lower-order components is used to refer to singular vectors corresponding to large singular values. These components are kept in low-rank approximation the matrix.
4 LAYER SELECTIVE RANK REDUCTION (LASER)
In this section, we formally describe the LASER method. A single-step LASER intervention is defined by three quantities, (r, E, p). consisting of parameter type т, layer number L, and rank reduction p respectively. These values together describe which matrix will replaced by its low-rank ap- proximation and how severe the approximation will be. The parameter type categorizes the matrix type in which we will intervene. We focus on the matrices W = {W., We, W., Wo, Unn Vout] which consist of the matrices in the MLP and attention layers. The layer number describes the layer at which we intervene (the first layer is indexed from 0). E.g.. the Llama-2 has 32 layers and so l {0. 2, ..1)). Finally, p E 10, describes what fraction of the maximum rank should be preserved upon doing its low-rank approximation. For example, let t = Unn E dddd then the maximum rank of this matrix is d. We replace with rank [p- d --approximation.
31).
4

---

Figure shows example of LASER. In this figure, we have Unn and l = L indicating that update the weight matrix in the first layer of MLP in the Transformer block of the thh layer. The other parameter (not shown in the figure) controls the k in the rank-k approximation.
LASER throttles the flow of certain information in the network, which surprisingly can produce sig- nificant performance benefits. These interventions can also be easily composed- we can apply set of interventions {(T), L, pt) in any order. The LASER approach to simply search over inter- ventions of this type, and exercise the modification that offers the greatest benefit. There are many other ways in which one can combine these interventions, however, and we defer this to future work.
5 EXPERIMENTS
This section studies the consequences of LASER throughout various layers of the Transformer ar- chitecture. We first perform a motivating analysis of the CounterFact (Meng et al. 2022) question- answering dataset in conjunction with a pretrained GPT-J model (Wang and Komatsuzaki (02T), and investigate the performance of the model and its variability as we search over potential interven- tions. We then examine the effect of LASER across different models, datasets and modalities.
GPT-J, CounterFact and PILE. We use the GPT-J model with 27 layers and 6B parameters pretrained on the PILE dataset. The first part of our analysis focuses on GPT-J, largely because its training data is publicly available. We evaluate the model's behavior on the CounterFact dataset, which consists of samples organized as (subject, relation, answer) tuples and three paraphrased prompts for each question. For example, (Danielle Darrieux, mother tongue, French).
5.1 THOROUGH ANALYSIS WITH GPT-J ON THE COUNTERFACT DATASET
Figure 2 shows the result of upplying various amounts of rank reduction to each matrix in the Trans- former architecture on the classification loss for this dataset. These plots are grouped, such that cach sub-figure corresponds only to the indicated type of weight matrices. Note that cach Transformer layer consists of a small, two-layer MLP. The constituent input and output matrices are shown sep- arately. Different colors indicate different percentages of removed components.
The attention plots in this figure exemplify what is already known about these models: weight matrices can be drastically reduced without much degradation in model performance. The more interesting result, however, is in the MLP layers. Here, not only can matrices be rank-reduced without degrading classification performance, but large performance improvements are possible by reducing later layers of the model. This trend is most stark in the input matrix of the MLP. While there are gains with LASER in the attention layers too, the benefits are typically smaller. In the section that follows, we demonstrate the effectiveness of LASER across a wide array of datasets and Transformer models. Because a thorough search can be computationally intensive, and consistent improvements seem concentrated to reducing the MLP layers, all results that follow this section consider a reduced search over only these layers unless stated otherwise.
Improved accuracy and robustness to paraphrases. The CounterFact dataset is used to test the model's factual knowledge of data from Wikipedia. Since GPT-J is trained on PILE, whose contents include Wikidata, different facts in CounterFact are part of the model's training data, albeit in different quantities. As all answers are single token in this setting, we compute top-k accuracy based on whether the correct answer is in the top-k predicted tokens. As seen in Figure 2 2|and Table 1 we find that the model's 's top-1 accuracy on facts in CounterFact increases from 13.3% to 24.1% when reductions are done on a single layer. It is important to note that these improvements are result of rank-reduction alone, and do not involve any further training or fine-tuning of the pre-trained GPT-J model. Furthermore, the improvements that come with rank-reduction are systematic. The set of datapoints that the model gets correct only grows with increasing amounts of reduction as opposed to a random movement of datapoints into and out of the set or correct items; if a model gets an answer right with a certain amount of rank reduction (r), the model continues to get the answer correct for larger rank reductions (y where y > Φ). We evaluate the model's robustness to paraphrases by computing the percentage of datapoints where the model gets all paraphrases of given question correct. For datapoints that the model already gets correct, the model's 's robustness to paraphrases also improves with LASER by roughly 24.8 percentage points.
5

---

Answers corrected OOrignallly correct
The mother longue of
Danielle Yvonne Marie Antoinette Darrieux French actress of stage, television and Bil, as well dancer.
(Danielle
0.175
Darrieux; [French)
0.4-
0.150
The capital of [Indaa] is
Clemens, best known by his name Mark
Sanuel Langhorne
Accuracy
0.3
0.125
(New Delhi)
Twain, was un American writer, humorist, publisher, and lecturer.
0.075
With LASER
Danielle Darrieux,
Original model
0.050
French actress whose carcer spanned eight decades, dies
0.025
1000
2000
3000
4000
5000
<50
50-1000 1000
Train
N = Frequency of D in 9 Train (b)
N (c)
a
Figure 3: Which datapoints benefit from LASER? We analyze how frequently in the training data "corrected" facts occur. GPT-J an ideal test bed for such analysis since its training data (Drrain). the PILE dataset, is publicly available. (a) For GPT-J evaluated on CounterFact (DQA) we retrieve all the datapoints that correspond to each sample in DQ (b) A plot depicting the cumulative top-10 accuracy of the model on all datapoints that occur in the training data less than or equal to the frequency indicated on the x-axis. Here show accuracy with and without LASER. (c) The largest boost in performance occurs for low-frequency samples. This bar chart displays the amount of boost offered by LASER for data binned by the frequency with which corresponding facts occur in raiin- Maximal improvements in accuracy are from datapoints that have less-freed eess-frequent occurrences in training data.
Dreein that contain a mention of both the entity of interest and the answer
Effeet on language modeling and fluency. While the model's factuality improves, does the reduc- tion affect the model's performance other metrics? To understand this, we evaluate the model's perplexity, i.c. its original training objective, on its training data. For layers corresponding to the MLP input matrices, the perplexity of the model increases from 4.8 to 5.0, showing that the language modeling objective is indeed slightly effected. For the MLP output layers, the perplexity of GPT-J on PILE increases from 4.8 4.9 with LASER. It may be possible to fix this small degradation by calibrating the temperature of the model.
Composing reductions across layers. We find that even further improvements in the model's performance can made by perform- ing different amounts of rank reduction across several layers. This done by greedily search- ing over (r, é, p) starting from the largest & and smallest To speed things up, here we do this search only over MLP layers, as this is where the largest improvements are typically found. Consistent with other experiments, the search done on a validation set, and re- sults are reported on the test set. On Coun- terFact, the 0-1 accuracy of the base GPT-J model is 13.1%. After doing the best single- step LASER the model's accuracy improved to 24.0%. Performing LASER across different lay- ers improved the top-10 accuracy to 29.2%, 5.2% absolute improvement in accuracy over single layer. The re-
32
28
Accuracy
24
20
Val Set: Multi Step
Test Set:
Multi Step
16
Single Step
Test Set: Single Step
12
2
6 Number of Laser Interventions
10
14
16
Figure 4: Composing LASER operations across multiple layers further enhances model perfor- mance. Here we show how accuracy improves using a simple compositional strategy for both validation data, which was used to identify each (T, é, p), and held-out test data.
performing LASER sults of the combinatorial search across differ- ent & and p values be seen in Figure 4

---

(8)
Paul Citroen a native speaker of
Correct
Dutch
Correct
the
Beenge Wrong
Dutch
Average answer
the
W
the
Fronch
W
Wrong answer
Average
French
the
Average unswer
French
Nofligle
Isp
Bottom
Top
Bottom
Saced Akhtar Mirza is originally from The original ianguage of Hansar Ballad is Kalahhra follows religion of Emmamucile Devos's profession Walter Zenga professional Soccer
Mumboi
Pakistan
netn city of Wellington
Sydney
of
Russian
Portuguese
Kharkty twin
Worsan
The
Ruddhiam
The native language Irane Massa Dutch
Acter
Teacher
The headguarter of Mory Music is located in Abba Eban war employed in Yihha Harari speaks
Berlin
Photographer
Jerusatem
The
Mike Holmgren plays in the pusition of Quarterback
Gouikeeper
(e)
Hebrew
To
Figure 5: (a) [Left] LASER approximates learned matrices by its lower-order components. We find that for samples where the model's 's predictions improve after LASER, if we instead use the entire ma- trix (including higher-order components), the model often predicts only "generic" words. (a) [Right] To understand what these higher-order components encode, approximate the learned weight ma- trix with the higher-order components instead. We find that these higher-order components some- times encode the correct semantic type of the answer but the incorrect response. (b) Analytically, computing the semantic similarity (cosine distance between the true answer and answers generated by the bottom k% of the singular vectors) shows that, average, the answer computed by higher- order components is more similar to the real answer. (c) Shows examples from the dataset and corresponding answers computed by the top fraction and bottom fraction of components.
5.1.1 WHICH FACTS THE DATASET ARE RECOVERED BY RANK REDUCTION?
To understand this phenomenon, we look at the questions correctly answered after LASER and the effect of how often the information associated with the question appears in the training data. For every datapoint in CounterFact, we retrieve all the examples in PILE that contain a mention of both the entity and the answer. We then compute how often information associated with each evaluation question appears in the training data. We find that the facts recovered on rank reduction are most likely to be infrequently present in the data (Figure 3). Here, "Originally correct" describes samples that are correctly classified even without any intervention. "Answer-corrected" refers to questions the model gets correct only after intervening with LASER.
5.1.2 WHAT ARE HIGHER-ORDER COMPONENTS STORING?
We saw above how retaining the lower-order components improves model performance on the task of open-ended question answering. We find that for the task of question answering, improvements often on questions whose answers supported by less frequently occurring data in the train- ing set. While it is clear that climinating the higher-order components "denoises" the model and helps recover "hidden," less- less-frequent information, it is not clear what the higher-order components are representing such that that their removal improves performance. This section studies this ques- tion using the CounterFact dataset and GPT-J.
To understand what higher-order components are representing, we approximate the final weight matrix using its higher-order components (as opposed to approximating using lower-order components as done by LASER) as shown in FFiiue 5(a). Following this, we analyze how the model's behavior changes on data points that GPT-J originally gets incorrect but are flipped being correct upon performing LASER.
First, we note that when the original, unmodified model does not answer these questions correctly, often responds with common words, such as "a," "the," "of," and other highly frequent tokens. After performing LASER, where we retain only the top-k components, the model's answers to these ques- tions flip from generic words the correct entity. For the same datapoints, when approximate the model by instead retaining the higher-order components, we find that the model either predicts incorrect entities that of the same semantic type as the correct answer high-frequency tokens

---

Dataset
Model Name
Roberta
GPT-J
LLama2
LASER
LASER
LASER
Acc
17.3
19.3
13.1
24.0
35.6
37.6
CounterFact
Loss
5.78
5.43
5.78
5.05
3.61
3.49
Acc
6.1
6.7
19.6
19.5
16.5
17.2
HotPotQA
Loss
10.99
10.53
3.40
3.39
3.15
2.97
Acc
50.0
52.3
50.2
56.2
59.3
64.5
FEVER
Loss
2.5
1.76
1.24
1.27
1.02
0.91
Acc
87.5
93.7
70.9
97.5
75.5
88.4
Bios Gender
Loss
0.87
1.13
3,86
4.20
3,48
2.93
Acc
64.3
72.5
75.6
82.1
85.0
86.7
Bios Profession
Loss
4.91
6.44
4.64
4.91
4.19
4.05
Acc
56.2
56.2
54.9
55.6
50.5
56.2
TruthfulQA
Loss
1.60
1.42
1.02
1.01
0.95
1.04
Acc
37.1
41.8
37.1
38.3
44.8
63.4
BigBench--Eiisemin Reasoning
Loss
9.39
6.80
0.74
0.62
0.78
0.73
Acc
28.0
30.7
51.8
65.9
59.5
62.0
BigBench-WikidattO0
Loss
9.07
7.69
3.52
2.86
2.40
2.31
Table I: The effect of LASER intervention on eight natural language understanding datasets. We find the best LASER intervention for each model and task using accuracy/O-1 on a validation set and report its performance on a held-out test set. We notice that in some cases, while the model's accuracy improves, its loss slightly degrades.
such as "a," "the," and "of," as shown in Figure 5)c). However, as we systematically include lower-order components, the model's output changes to predicting frequent tokens. To investigate this systematic degradation, we measure the average cosine similarity of the "true" answer with respect to the predicted answer when the matrix is approximated with different amounts of higher- order components, as shown in Figure 5tb). The average cosine similarity between the predicted answer worsens, demonstrating this effect.
We hypothesize that these matrices often encode multiple conflicting responses, and that when all components are used they clash to produce generic token. Removing the higher-order compo- nents, which anecdotally appear to often capture incorrect responses of the correct type, resolves this internal conflict and allows the model respond accurately.
5.2 How GENERALLY DOES THIS HOLD?
We study the generality of our findings by applying three different LLMs to variety of natural language understanding tasks.
Natural Language Understanding Tasks. We evaluate model performance before and after
LASER on seven datasets, including CounterFact (Meng et al. 2022). HotPotQA (Yang et al. 2018),
FEVER (Thorne al.) 2018). Bias in Bios (De-Arteaga et al. 2019) [Gender and Profession] Truth-
fulQA (Lin et al. 2021). BigBench-Epiitemic Reasoning (Bowman et al. 2015) and BigBench- WikidataQA. These datasets evaluate different aspects of language understanding problems. Coun- terFact, Fever, and Bigbench-We Wiki data test a model's world knowledge and factuality. Bias in Bios benchmarks model bias by predicting the gender and profession of a person given a short biography. We define Bios Gender as the gender prediction problem in Bias in Bios, and Bios Profession as the profession prediction problem. HotPotQA provides more challenging open-ended question an- swering task with long answers containing many tokens. The Epistemic Reasoning dataset from Big Bench Hard (BBH) tests a model's logic and reading comprehension. Finally, TruthfulQA tests an LLM's truthfulness. We use 20% of the dataset as validation set and select the best LASER hyperpa- rameters (r, L, p) using this validation set. We report results on the remaining 80% the dataset with the chosen hyperparameter. The models used for the task of question answering include, Roberta (Liu et al.) 2020), GPT-J (6B) (Wang and Komatsuzaki (021). and LLAMA2 (7B) (Touvron et al, 2023 Details regarding datasets and how they were used can be found in Appendix A
8

---

Evaluation metrics. For each of these tasks, evaluate the model's performance using
1. Generation accuracy. We generate sequence of N tokens using the LLM and then report one the answer text in the generated text and zero otherwise,
2. Classification accuracy. If the answer lies a small set of potential values, like a standard classification problem, we consider response correct if it puts more probability mass on the correct answer than on any of the other candidates.
3. Loss. We report the log-loss on held-out data. For datasets with small set of possible labels we report the (acc) classification accuracy, and for others we use generation accuracy.
We test the generality of this result by evaluating a collection of language models on different benchmarks. As seen in Table I] we find that even severe reductions result in deterioration in the model's accuracy and can lead to improvements in their performance. The amount of reduction required differs from model model.
5.3 NON-TEXT DOMAINS
To understand if this phenomenon effective outside of question answering in the textual domain, evaluate the effect of rank reduction reinforcement learning agent.
Policy learning. For Policy learning, evaluate the effect of LASER on a decision Transformer model trained on the game of Sokoban and evaluated the same game. This is a challenging planning problem where the agent has move and push several blocks to holes. The task is completed when all blocks are on top of holes. The imput the decision Transformer is the visual state of the environment at a given state, and the output is the low-level action. We find that for de- cision Transformer trained on Sokoban, models solved 3% more tasks with LASER (Table 2p. Details of the experiment can be found Appendix B
Model Name
Acc.
Return
Transformer with LASER
50.67
0.575
53
0.965
Table 2: Effect on LASER on a 6-layer De- cision Transformer agent. The base model is trained and evaluated in a challenging 10 10 Sokoban domain.
Although the improvements are much smaller, they are consistent despite the severity with which reductions are performed. This can because the phenomenon either text-specific or requires a large enough Transformer model.
CONCLUSION AND DISCUSSION
This paper describes LASER, a phenomenon where performing a low-rank approximation specific layer types specific layers of the transformer block can improve the performance of LLMs on the task of question answering. We find this to be true across five different datasets and three different language model models. Furthermore, the resulting LASER reductions are reduced at times to 99% of their original rank, which is much lower than their effective rank (C.I). However, despite extreme reductions, the performance of the model on tasks continues to improve. We also observe performance gains for decision Transformer in an embodied domain. We find that the largest improvements in the model accuracy correspond information that less common in the training data and that LASER jointly makes the model more robust to paraphrases of the questions. We further found that the higher-order components of some of these matrices encode either high-frequency words alternate answers of the same semantic type the correct answer. These noisy, higher-order components can overpower the stable lower-order components and result in the model answering questions incorrectly. In these cases, performing LASER acts denoising technique and reduces the internal conflicts in potential responses.
extreme. The matrices
Despite this analysis, the success of LASER requires further study. Learning (i) why higher-order components in weight matrices accumulate noisy answers in the course training, (ii) the effect of model architecture and other structural choices on the occurence of this phenomenon and (iii) why this is specifically true for later layers in the MLP is important to not only for our understanding of the success of LASER, but for understanding the behavior of large language models more generally.

---

ACKNOWLEDGEMENTS
This work done when PS was intern at Microsoft Research, New York City, with DM and JTA. The authors would like to thank Minyoung Huh, Shikhar Murty, Han Guo, Cyril Zhang. David Bau, Jacob Andreas, Antonio Torralba, and John Langford for helpful discussions and feedback.
REFERENCES
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Picter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 00015558400000 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3971, 2023.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. NIPS, 2014.
Yoshua Bengio, Nicolas Roux, Pascal Vincent, Olivier Delalleau, and Patrice Marcotte. Convex neural networks. In Y. Weiss, B. Schölkopf, and J. Platt, editors, Advances in Neural Information Processing Systems, volume 18. MIT Press, 2005.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Rep- resentations, 2017.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Repre- sentations, 2017.
Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. Probing for semantic evidence of composi- tion by means of simple classification tasks. In Proceedings of the 1st Workshop on Evaluating Vector-Spuce Representations for NLP, pages 134-139, Berlin, Germany, August 2016. Associa- tion for Computational Linguistics. doi: 10. 18653/v1/W/ 11////161222244
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. ICLR, abs/1608 abs/1608.04207,, 04207, 2016.
Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. J. Artif. Intell. Res., 61(1)9007-926, January 2018.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single $& vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2126-211 2136, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/ 1/P51//1/P111- 1198.
10

---

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer Key- Value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484-5495, Online and Punta Cana, Dominican Republic, Novem- ber 2021. Association for Computational Linguistics.
layers
N. Elhage. A mathematical framework for transformer circuits. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. https: //transformer- 2onuiissuub/2222/faaeewwrrknnnn htmll, 2021.
Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36, 2022.
Sumu Zhao, Damian Pascual, Gino Brunner, and Roger Wattenhofer. Of Non- Mon-Linearity and Com- mutativity in BERT. In International Joint Conference on Neural Networks (IJCNN), Virtual-only, July 2021.
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. In Thirty-seventhh Conference Neural Information Processing Systems, 2023.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann 1989.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain sur- geon. In S. Hanson, J. Cowan, and C. Giles, editors, Advances in Neural Information Processing Systems, volume 5. Morgan- Kaufmann, 1992.
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural network. In NIPS, 2015.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. International Conference on Learning Representations, 2017.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: pruning and growth for efficient inference and training in neural networks. J. Mach. Learn. Res., 22())11882211005 January 2021.
Xiuqing Lv, Peng Zhang, Sunzhu Li, Guobing Gan, and Yueheng Sun. LightFormer: Light-weight transformer using SVD-based weight transfer and parameter sharing. In Findings of the Asso- ciation for Computational Linguistics: ACL 2023, Toronto, Canada, July 2023, Association for Computational Linguistics.
Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D Manning. Characterizing intrinsic compositionality in transformers with tree projections. In The Eleventh International Conference on Learning Representations, 2023.
Tian Jin, Michael Carbin, Daniel M. Roy, Jonathan Frankle, and Gintare Karolina Dziugaite. Prun- ing's s effect on generalization through the lens of training and regularization. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Informa- tion Processing Systems, 2022.
Habib Hajimolahoseini, Mehdi Rezagholizadeh, Vahid Partovinia, Marzieh Tahaei, Omar Mohamed Awad, and Yang Liu. Compressing pre- pre-trrieed language models using progressive low rank de- composition. Advances in Neural Information Processing Systems, 2021.
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In 2017 IEEE Conference Computer Vision and Pattern Recog- nition (CVPR), pages 67-76, July 2017.
Jimmy Ba and Rich Caruana. Do deep nets really need be deep? In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Process- ing Systems, volume 27. Curran Associates, Inc., 2014.
11

---

Huanrui Yang, Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li, and Yiran Chen. Learning low-rand deep neural networks via singular vector orthogonality regularization and sin- gular value sparsification, 2020.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- Irika, (3):211-218, 1936.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model, May 2021.
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing, 2018.
James Thome, Andreas Vlachos. Christos Christodoulopoulls, and Arpit Mittal. FEVER: a large- scale dataset for fact extraction and VERification. In NAACL-HLT, 2018.
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, jan 2019.
doi: 10.1 10.1145/2885500328877777
Stephanie C. Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Annual Meeting of the Association for Computational Linguistics, 2021.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning, A large anno- tated corpus for learning natural language inference. Lluís Mårquez, Chris Dallison-Burchs and Jian Su, editors, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, 642, Lisbon, Portugal, September 2015. Association for Computational Linguisties. doi: 10. 10.18653/1/DDD5-10755
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Ro(bert]a: A robustly optimized {bert) pre- training approach, 2020.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language mod- els. Transactions of the Association for Computational Linguistics, 9, 2021.
Max-Philipp B. Schrader. gym-sokoban, 2018.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European Signal Processing Conference, pages 606-610, 2007.
12

---

APPENDIX
A DATASET DETAILS
CounterFact. The CounterFact dataset is derived from the PARAREL dataset (Elazar et al. 2021) and contains knowledge tuples of the kind t" (s, o²), where is the subject, is the relation and o is the object. These tuples are constructed using entities listed in Wikidata. The datapoints accompanied by handwritten prompt templates for each category. The CounterFact dataset also contains suggested edits to the true facts represented in the dataset. For this study, the set of coun- terfactual edits are not used.
PILE. The PILE dataset is an approximately 1TB language modeling dataset that was used to pre-train GPT-J. It contains text from 22 smaller datasets, including Wikipedia, OpenWebTextt and StackExchange, name few. The PILE dataset was used to study the effect of LASER the behavior of the model on the original training data distribution. For the study on quantifying the occurrences of entities in the training data, the training data split of PILE was used. However, the measure of change in perplexity of the model after LASER was measured on the validation split of the dataset.
HotpotQA. We use the HotPotQA dataset available on HuggingFace. An example question is "What are the names of the current members of American heavy metal band who wrote the music for Hurt Locker The Musical?" ?" and the answer "Hetfield and Ulrich, longtime lead guitarist Kirk Hammett, and bassist Robert Trujillo". We use the validation split of this dataset for identifying LASER parameters, containing 7,100 datapoints, and the train split, containing 90k datapoints, for evaluation.
The FEVER (Fact Extraction and Verification) dataset consists of claims and label char- acterizing each as either true or false. The goal is to predict the label given the claim. It consists of 185,445 such claims and is constructed using data from Wikipedia. An example claim is "Roman Atwood is a content creator." and the label is "true."
FEVER.
Bias in Bios. Bias in Bios dataset that consists of descriptions of people, and the task is to correctly identify the gender and occupation of the person. An example datapoint is "Nancy Lee graduated from Lehigh University, with honours 1998. Nancy has years of experience in weight loss surgery, patient support, education, and diabetes." the gender label is "Female" and the profession is "Nurse."
BegBench-Epistemii Reasoning: The epistemic reasoning dataset tests a model's ability make entailment decisions from a pair of sentences. Specifically, epistemic reasoning assesses given model's understanding of "theory of mind," which involves the capacity to analyze the cognitive states of various agents. A sample consists of a text containing premise and hypothesis and label which is either "entailment" or "non-entailment." " An example datapoint is "Premise: Emma knows that James thinks that there is milk in the fridge. Hypothesis: James thinks that there is milk in the fridge." The label here "entailment."
TruthfulQA: TruthfulQA evaluates the extent to which model replicates human falsehoods. The benchmark contains 817 questions that span 38 categories, including health, law, finance and pol- itics. The dataset emphasizes that solely increasing the scale of models may not be as effective in enhancing truthfulness the web. An example datapoint is "What happens to you if you eat watermelon seeds? ['Nothing happens,' 'You eat watermelon seeds,' 'The watermelon seeds pass through your digestive system," 'You will not digest the watermelon seeds,' "The watermelon seeds will be excreted," 'You grow wa- termelons in your stomach," 'You get sick,' "You have bad dreams," 'You die,' 'You get indigestion," 'You fall unconscious." 'You digest the watermelon seeds']".
fine-tum tieetunnng with training objectives that go beyond imitating text from
QA Wiki Data: The task tests a model's ability to answer open domain, cloze-style question answering randomly sampled Wikipedia triplets. This evaluates world knowledge large
13

---

collection of facts and information from knowledge graph extracted from Wikipedia data. An example datapoint from this dataset is "Gabon shares border with Cameroon."
Figure 6: An example of Sokoban task. The game requires agent to move the orange boxes to their desired locations (red squares) complex warehouse-like environment without getting trapping themselves. Playing successfully requires the agent reason long time horizons effectively.
B DETAILS OF THE DECISION TRANSFORMER DOMAIN
Sokoban is a warehouse- keeping transportation game that requires long-horizon reasoning and planning over multiple time steps. The task is to move all boxes to their target locations without getting trapped. We use the Gym Sokoban environment [Schrader (2018), and train a 5-layer decision transformer model using 106 optimal episodes of the game. In our setting, the maximum return of the game is set to 10.
Sokoban Details. We show an image of the Sokoban task Figure 6
C EXTENDED ANALYSIS
C. ARE WEIGHT MATRICES ALREADY LOW-RANK?
As seen in Figure 7 we find that LASER approximated matrices with their low-rank approximations much beyond their effective rank computed by (Roy and Vetterli 2007). To study this, we com- puted the effective rank of the MLP matrices for which LASER helps for GPT-J model using the method described by Roy and Vetterli (2007). The plot shows that although matrices of the later layer have lower effective rank than the earlier layers, the computed effective rank is significantly larger than the reduction % until which LASER helps.
C.2 HOw MUCH REDUCTION IS TOO MUCI??
We that for many of the matrices in cases where reduction helps, with increasing amounts of rank-reduction, the model first monotonically improves before it starts to worsen, as seen infFigure g The point up which it improves varies depending the layer type and layer number. However, the monotonic improvement and worsening are observed consistently.
What is the effect of removing the layer completely? We find that, removing the layer completely can be better than retaining its matrix with its full rank, however it is observed to be worse than the model with the low-rank approximation of the matrix.
14

---

3800
Layer Type
3600
mip.fc in.weight
mip.fc_oute heppfc_out.weijpe out.htight
Elfectiive Rank
3400
attn.k_proj, weight
attnn.__proj. prr.proo.wweggg weight
attn.out_prr _oout_prowwwwoo weight
3200
attn.v proj. weight
3000
2600
10
20
25
Layer Number
Figure Effective rank of the matrices computed as described by Roy and Vetterli (2007)
C.3 DOES LASER SELECT THE SAME LAYER FOR DIFFERENT TASKS?
We find that the maximum improvements on different tasks come from LASER on different layers of the model. Figure 9 shows that for GPT-J different tasks, the best-performing models across tasks have reduced matrices in different layers.
C.4 MEASURING PERPLEXITY ON PILE.
To measure the effect of the interventions language modelling, we compute the perplexity of the reduced model on the evaluation set of PILE. The perplexity of the fixed-length GPT-J model is evaluated using sliding window strategy over the sequence of tokens with stride of 512 tokens. While there is an improvement in the task at hand, model's perplexity worsens slightly after applying LASER. We do not yet fully understand what the worsening in perplexity of the model corresponds to and leave this for future study.
C.5 FINAL LASER SEARCH RESULTS
Table 3) shows the final search results of LASER for models and datasets from Table I These values obtained by reporting the optimal LASER parameters that maximize the validation accuracy. The results show that the optimal improvements in the models typically come from later layers in denotes that the intervention is done on the first layer. For reference, recall that Llama2 has 32 layers, GPT-J has 28 layers, and Roberta has layers. The magnitudes of reduction are also quite large, with the rank at times being reduced 1% of the original matrix's rank.
the transformer model, typically from reducing the MLP Input matrix. Note that
D ALTERNATE PRUNING METHODS
Instead of approximating weight matrices with their rank-k approximations, tried Absolute Weight Pruming (Frankle and Carbin 2019). Here, we zero out the bottom x% of the weights of the matrix by their absolute magnitude. The results for GPT-J on CounterFact be in Fig- ure 1O this case too, we find that the accuracy of the model increases with pruning later layers of the ML.. We leave further study of this phenomenon for future work.
15

---

0.52
Top-10 10 accuracy
0.50
0.48
0.46
20
40
60
80
100
% Reduction
Figure 8: While the performance of the models continues to improve with large amounts of reduc- tion, after point starts to worsen. The plot shows the top- 10 accuracy of GPT-J on CounterFact. dip in performance is observed at 99.95% reduction.
Dataset
Model
Roberta [T, pl
GPT-J [7, E, pl
Llama2 7B [T,
CounterFact
[Uin: 8, 0.8
[Uins 27, 0.01]
[Unn 28, 0.05]
HotPotQA
[Uout 2, 0.4]
[Uin, 27, 0.1]
[Uin, 27, 0.2]
FEVER
[Uen; 0.4]
[Uins 24, 0.01]
[Uin, 30, 0.2]
Bios Gender
[Un, 0.9
[Uin, 14,0 0.01]
[Umn, 24, 0.01]
Bios Prof.
[Un 3, 0.9]
[Umn 18, 0.01]
[Vout 30, 0.4]
BigBench- -eittch--ppittmiccccc Reasoning
[Uout, 1, 0.4]
[Unn 26, 0.01]
[Vout: 28, 0.01]
TruthfulQA
Unn 0, 0.01]
[Umn: 7.0.8]
[Uin, 30, 0.05]
BigBench- BigBench-Wikidatee WikidataQA
[Un., 0.0.4
[Umn 27, 0.01)
[Um, 27, 0.01]
Table 3: Final search results of LASER: In op-perrooning models, significant benefits from rank reduction are typically observed in later layers. The amount of reduction is severe, for example, in GPT-J on CounterFact, the rank of the MI.P matrix reduced from 4096 rank This is about 99% of the matrix's original rank.
16

---

MLP Output Matrix
530
MLP Input Matrix
51-
1413
140-
140-
335
Lyy
CounterFact
FEVER
HotPot
Bias Bios
Figure 9: For GPT-J across different datasets, the largest benefit of LASER comes from reductions on different layer numbers. Even though the largest benefits are typically from the MLP layers in the later layers of the model, the layer number differs for different dataset-model pairs.
Attention Key Matrix
Attention Query Matirx
Attention Value Matirx
6.1
5.90
6.0-
585
5.9
5.75
5.70
5.7
Attention Out Matrix
MLP Input Matrix
MLP Output Mutrix
7.50
6.1
7.25
7,00
6.75
7,0-
5.9
5.8
6.0-
5,50
5.5
10
15
20
25
5
10
15
20
25
5
10
15
20
25
Luyer number
Layer number
Layer number
Reduction %6:
10
25
40
150
66
75
90
92.5
995
97.5
198
18.55
99
99.5
99.75
Figure 10: Similar to LASER, we perform layer-selective absolute weight pruning, where fraction of the weights with smaller absolute magnitudes are set zero. For GPT-J on CounterFact, we find similar improvement in model performance with this intervention we do with LASER. A thorough analysis of how absolute weight pruning might help improve model performance on different natural language understanding tasks and its connection to LASER is left for future work.
E IMPLEMENTATION DETAILS
E. DATASET PROCESSING DETAILS
We process each dataset described in Table sepaarately. In each case, we use 20% of the processed dataset the validation which to select the best LASER hyperparameters (7, p, f). This
17

---

validation set can be different from the validation set of the original unprocessed dataset. We use accuracy on the validation set for selecting the best hyperparameters for each LLM and given dataset. Table 4 summarizes the size of these filtered dataset. We describe the dataset-specific processing below:
CounterFact. We use the original dataset which consists of roughly 20,000 examples and 3 para- phrase for each example. This gives us 65,757 examples for the entire dataset. The set of possible labels in this QA task is open-ended. For Roberta and GPT-J LLMs, the labels always exactly one token, while for Llama2 the labels can be multiple tokens long.
Hotpot. We combine the included validation and training sets of Hotpot increase the dataset size. We then filter out all examples where the answers are more than 15 tokens long according to the Llama2 tokenizer. We convert the original question the prompt "iquestion" The answer is" " (if the question ends with ? or ) or "iquestion?" ? The answer is" where the prompt variable iquestiong is replaced by the original question. This gives dataset of size 14,618. The set of possible labels in this QA task is open-ended and are multi-token long for all three LLMs that we consider.
Fever. We merge the dev and test split of the original Fever dataset. We then filter out samples where with duplicate claims (imputs) but different labels (outputs). This results in a dataset of 13,086 samples, including 6,510 from the original dev set. Here there are only two possible labels: true and false. We convert each question into the prompt "Consider the following claim: iquestiong. Is this claim true or false. The claim is". The prompt variable iquestiong replaced by the original question.
Bios Gender. We use only the dev split of the original Bias in Bios dataset. This gives us a dataset of size 39,642. The only possible labels in this QA task are two: male and female. We convert each input bio into the prompt "Consider the following tex:: ;biog. Is the person this text male or female? ? The person is" The prompt variable jbiog is replaced by the original bio.
Bios Profession. We use only the dev split of the original Bias in Bios dataset. The goal here is to predict the profession for given bio. We only keep datapoints which contains profession with a few tokens, namely, journalist, poet, composer, model, teacher, architect, painter, and professor. This gives us a dataset of size 19,223. The aforementioned professions compose the list of possible labels. We convert each input bio into the prompt "Consider the following text: jbioq- What the profession of the person in this text? The profession of this person is". The prompt variable jbiog is replaced by the original bio.
BigBench Epistemic Reasoning. We merge the validation and train split of the Big Bench epis- temic reasoning dataset. This gives us dataset size 2000. The set possible labels here are: entailment and non-entailment which are multi-token long for all LLMs. We do not process the text.
Truthful QA. We use the validation split of the Truthful QA dataset. The truthful QA dataset consists of multiple choice questions. We convert the dataset into separately checking the correctness of each answer independent of other answers. Specifically, a sample with 4 multiple choice answers gets converted into 4 separate samples, each with a true or false answer. We convert each question and answer pair into the prompt "jquestiinn jansweri- this statement true or false. This statement convert it into "iquestion; janswery Is
is" if the answer does not end with period (.), otherwise, this statement true or false. This statement is" The prompt variables iquestion; and janswerj are replaced by the original question and answer respectively. The processed dataset consists of 5,882 samples.
BigBench Wikidata QA. We merge the validation and train split of the Big Bench Wikidata QA dataset. We filter out examples where the number target labels are more than This gives us a dataset of size 20,321. This QA task has open-ended set of labels.
18

---

Dataset Name
Dataset Size
CounterFact
65757
HotpotQA
14618
FEVER
13086
Bios Gender
39642
Bios Profession
19223
TruthfulQA
5882
BiggBench-Fppitemnee Reasoning
2000
BesBench-Wikidatee
20321
Table 4: Size of the filtered dataset used for evaluating LASER. We use 20% of the dataset for selecting LASER hyperparameters (T, p) and the evaluate the best model the remaining.
E.2 DETAILS FOR COMPUTING ACCURACY AND LoG Loss
The procedure used to compute accuracy and log loss varies across the different datasets. Typically, for QA datasets with open-ended labels, we generate the predicted answer by doing greedy sampling using the LLM, i.e., with temperature answer is in the generated text. We lower case and strip whitespaces before comparing text. We call this the generation accuracy. In contrast, for datasets with small of possible label choice, predict the label with the highest probability under the LLM and report the prediction as correct if and only the predicted label is the correct label. We call this the classification accuracy.
O. We report the prediction correct if and only if the
As Roberta is a masked language model, we do generation by creating prompt with jmask; tokens, and predicting these masked tokens. When generating the response, we use a fixed number of masked tokens which may not correspond to the number of tokens in the answer. However, when computing the log-loss of the answer, we add as many masked tokens the number of tokens in the answer, and compute the log probabilities of the answer tokens under the model corresponding to these masked tokens.
The procedure for computing log-loss of the gold answer given context is the same across all dataset. We describe the dataset specific details for computing accuracies below.
CounterFact. We use generation accuracy to evaluate success, For GPT-J and Roberta, we gen- crate single token all labels are single token long, whereas for Llama we generate up 10 tokens.
HotPotQA. We use generation accuracy to evaluate success. For GPT-J and Llama2, generate up to 15 tokens. For Roberta we only use 5 tokens since Roberta struggles to fill-in more than a few masked tokens, this is understandable tokens (typically 15% tokens), and using these models to predict a large number of masked tokens then suffers from distributional shift.
Roberta is trained by masking out small number of
Fever: We use classification accuracy to measure success. predict the label from fulse) that has the highest probability under the model.
Bios Gender. We classification accuracy to measure success. We predict the label from {male, female) that has the highest probability under the model.
Bios Profession. We classification accuracy to measure success. We predict the label from the list of possible professions that has the highest probability under the model.
BigBench Epistemic Reasoning. We use classification accuracy to measure success. We predict the label from (entailment, non-entailment) that has the highest probability under the model.
TruthfulQA. We use classification accuracy to measure success. We predict the label from {true, false) that has the highest probability under the model.
19

---

LASER hyperparameter
Search Space
MLP weight matrices Uin and Vout layers the model (0.9, 0.8, 0.6, 0.2, 0.1, 0.05, 0.013
Table 5: LASER hyperparameters
BigBench WikidataQA. As the of labels are open-ended, we compute the accuracy using generation similar to CounterFact. For GPT-J and Llama2 we generate up to 10 tokens, whereas for Roberta we generate tokens.
E.3
CODE
We PyTorrh for all experiments. We use the HuggingFace implementation for all three large language models. We use Llama2 7GB weights provided by Meta. We use the SVD implementation available in PyTorch for experiments. The code can be found http://gttbb..ommpaaaysshhrraae
E.4 COMPUTE DETAILS
We ran each experiment on cluster with V100 and A2600 GPUs. Each experiment took about 1-3hrs to finish. For all settings, we search over hyperparameters listed in Table 5) For the GPT- J+CounterFact setting, depending on the experiment and plots, we run a much more fine-grained search over each hyperparameter.
20


Reviews:
[{"review_id": "Iwb6NFvMnW", "reviewer": "Reviewer_F88a", "summary": "LLMs are usually considered “the larger the better”, but this paper presents a surprising result: it is often possible to improve the performance of LLMs by simply removing higher-order components of their constituent weight matrices in the MLP layers. This paper presents this rank reduction method, LASER, that removes the components in the {Q,K,V,O} matrices that have smaller singular values (i.e., those higher-order components).\n\nThis paper finds that the effects of reduction is not uniform across layers. The performance degradation can be found by reducing early layers, while significant performance benefits are available, often by pruning the later layers. This effect is the most obvious in the MLP output, and is also observable in the k, q, v matrices.\n\nThis paper further dives into studying what types of facts are recovered by rank reduction, and finds that the facts recovered on rank reduction are most likely those infrequently present in the data. \n\nWhy are the higher-ordered components noisy? And what are the higher-ordered components computing? This paper approximates the final weight matrix using its higher-ordered components, and analyze how the model’s behavior changes on the datapoints that GPT-J’s lower-order components lead to incorrect outputs. They find that the model predicts incorrect entities of the same semantic type as the correct answer. As more lower-ordered components are included, the output changes to predicting common word tokens.\n\nWith additional experiments (on text domains including QA and non-text-domains including learning policies, images), this paper studies the generalizability of the findings.", "questions": "Seems like the MLP layers are the key components in storing the noise vs storing the “useful inductive biases”. I wonder if some structural choices (e.g., different position embedding methods, different activation functions, number of heads, etc.) can affect Transformer’s low-rank vs high-rank component behavior as well.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- This paper is well-written and easy to read.\n- The experiments are designed thoughtfully, and nicely supports the hypothesis.\n- The findings are important for both the understanding and the developments of better models in the future.", "weaknesses": "I do not see obvious weaknesses in this paper. There is a typo: Table 3 needs a horizontal line at the bottom.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iy4UYPTHMj", "reviewer": "Reviewer_4cdi", "summary": "This work discussed a traditional idea, i.e. the low-rank approximation using SVD, for language model compression. Its major observation is that using a low-rank approximation on the MLP layer of transformers can even improve the downstream performance.\nThis observation is verified across different tasks and different transformer models.", "questions": "I find that the dimension of GPTJ is 4096, which should be $d$ in your notation. So in Fig2, what is the rank of Reduction 99.5%/99.75% and others with .5%? (4096*0.0025=10.24, not an integer?)\n\nThe used CounterFact is very similar to the table-to-text generation task (but for a qa task), which is not a frequently used dataset to test even the QA and factual/world-knowledge reasoning performance of LLM. Any reason for choosing the dataset?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "-The use of low-rank approximation should be an effective and general way to allow the models to obtain more robust generalization abilities, while being more computationally efficient in the inference.\n-The authors tried very hard to demonstrate the major observation by showing the results across different transformer models, which are actually not even for language modeling tasks.", "weaknesses": "Albeit the strengths above, I would like to say the major weakness of this work is that it draws its conclusion not very rigorously:\n-Current LLMs are often evaluated from multiple aspects including their reasoning abilities such as commonsense reasoning, world knowledge, reading comprehension etc, as well as their language generation abilities such as multilingual ability etc. And each aspect contains well-known benchmark datasets for the evaluation, such as MNLU, AGIEval and BBH. However, this work uses none of them. Therefore, I am not convinced that this robust performance can be achieved across all the above-mentioned aspects.\n-The authors do not provide the final search results of rank reduction, i.e. the layers selected for compressing and the reduced rank, in the final performance in Table1, 2 & 3. It is very important to provide these results to show that the selected model is indeed in a reduced rank.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aiMmZHkCtG", "reviewer": "Reviewer_Zz8q", "summary": "This paper introduces a layer-selective rank-reduction method called LASER. The authors demonstrate that the performance in open-ended question answering is improved when rank-reduction is applied to the specific weight matrix. Moreover, they confirm consistent performance enhancements in tasks on non-text domains such as policy learning and image classification. Additionally, through analysis, it has been observed that high-order components contain factually incorrect knowledge which degrades question answering performance.", "questions": "- In Figure 3(c), what is the number of originally correct and answer-corrected datapoints?\n- It appears that there is a significant improvement in overall performance with GPT-J compared to LLama2 or Roberta. What could be the underlying cause of the result?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The authors conduct extensive experiments with respect to layer number, parameter type, and rate reduction to identify setups that lead to performance improvement.\n- The authors provide interesting observations such as a correlation between rank reduction and question answering accuracy.\n- The authors demonstrate that the proposed method can also be applied to various domains such as image classification.", "weaknesses": "- Analysis on other text domain tasks such as reading comprehension could provide further insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)