Streamlining Language Models via Semantic Basis Analysis
Authors: Yang Li
Daniel Agyei Asante
Changsheng Zhao
Ernie Chang
Yangyang Shi
Vikas Chandra
Venue: Accepted by TMLR ()

Abstract: As the size of language models increases, they deliver substantial performance improvements across a variety of applications. However, this growth also leads to greater computational demands, making deployment on resource-constrained devices—such as personal computers and mobile or wearable devices—more challenging, and significantly raising inference costs on cloud servers. To address these challenges, we introduce Basel, a method to streamline language models by leveraging the semantic structure of their weight matrices. Specifically, Basel treats each weight matrix as a linear combination of bases, selectively retaining those that are associated with essential semantics for the target application, pruning redundant ones, and introducing new bases that enhance task performance. Experimental results demonstrate that Basel achieves significant model size reduction compared to baseline techniques, while maintaining comparable or even superior accuracy across diverse applications.

Published in Transactions on Machine Learning Research (12/2025)
Streamlining Language Models via Semantic Basis Analysis
Yang Li
yangli Giastale. edu
Towa State University and Meta
Daniel Agyei Asante" lowa State University
Lasante edu
Changsheng Zhao
cszhao@meta.co com
Meta
Ernie Chang
erniecyc@meta.com
Meta
Yangvang Shi
yyshi@mela.ccom com
Meta
Vikas Chandra
vchandra@meta.cm com
Meta
Reviewed on OpenReview: https: // openreview. net/ forum? id= qq 7NNAXvW
Abstract
As the size language models increases, they deliver substantial performance improvements across a variely of applications. However, this growth also leads greater computational demands, making deployment on resource-constrained devices such as personal computers and mobile wearable devices-mon challenging, and significantly raising inference costs on cloud servers. To address these challenges, introduce Basel, method to streamline language models by leveraging the semantic structure of their weight matrices. Specifi- cally, Basel treats each weight matrix as linear combination of bases, selectively retaining those that are associated with essential semantics for the target application, pruning redun- dant ones, and introducing new bases that enhance lask performance. Experimental results demonstrate that Basel achieves significant model size reduction compared to baseline tech- niques, while maintaining comparable even superior accuracy across diverse applications.
1 Introduction
have signifi- cantly enhanced the performance of various applications in natural language processing, computer vision, and beyond. However, their large model sizes pose bottleneck for many practical uses. The substantial computing resources required for LLM inference make it challenging to deploy them devices with limited capabilities, such as personal computers and mobile/ /werrbbe devices (T.i et. al. 2024b 2025). Moreover, even on hardware platforms with ample computing power, deploying LLMs consumes a significant amount of energy, raising concerns about sustainability. Therefore, it essential to reduce the size LLMs after pretraining to ease their computational demands and lower energy consumption.
Large language models (LLMs) (Touvron et al., 2023a OpenAl et al. 2024 Google 2023
Our approach exploits the relationship between pretrained models and specific target applications. Large language models are typically pretrained on vast datasots encompassing a wide range of tasks, many of which
"Equal contribution.
i "Correpponding author. Address: 2434 Osborn Dr, Ames, TA 50011, United States. Email: yangli @lastate.edu.

---

Published in Transactions on Machine Learning Research (12/2025)
Importance
Singular
Pretrained Model
Basel:
Target Application
Values
Select Important Bases for Target Applications
u
V2
V
Bases
u
U, v
Bases
Figure 1: Basel: Identify and select the important bases for target applications during compression.
share common characteristics. This shared pretraining fosters synergies that enhance the performance of LLMs. However, the diversity among these tasks also introduces redundancy into the models. As demon- strated by our results in Section B LLMS contain redundant components that are unnecessary for specific target applications. By removing these redundant parts and retaining only the relevant oncs, we can re- duce the model's size while preserving its performance on the target application. Since many scenarios only require support for specific type of application (Chang et al. 2024), this approach effectively lowers the computing resource requirements and reduces inference costs.
However, how do we identify the beneficial and redundant components of LLMs for a specific application? this work, we address this problem through the lens of matrix factorization. Singular value decomposition (SVD) (ollu Van Loan 1996) factorizes a weight matrix W into the product of three matrices U S, and V, ie., W USV E s,u,v,", where s; are (positive) singular values, and u, and Vi are column vectors of U and V with unit norms. Our results show that these column vectors u, and V; may carry specific meanings. For instance, in the LLama 2-7B model (Touvron et al. 2023b), when factorizing the weight matrix W W of an attention head that is likely useful for code generation tasks, de-embedding the resulting column vectors u, and v, reveals tokens like _in, <0x0A>, _and, ,o, and _or. These column vectors are evidently useful for code generation but may be less relevant for tasks such as mathematical reasoning.
Inspired by this observation, we propose Basel, low-rank decomposition approach effectively compress LLMs for target applications. Figure I illustrates the key idea of Basel. We view cach weight matrix in LLMs as linear combination of bases u.v? with singular values 8; their weights. These bases are valuable representations stored in the pretrained model, learned from large pretraining datasets. For a target application, some bases are advantageous while many others are not. To select the bases beneficial for the target application, we propose retraining the singular values (i.c., the weights of the bases) while keeping the bases fixed, using the training set of the larget application. After retraining, we prune the bases associated with small singular values, as they are less important for the target application, and retain only those with large singular values, which are most critical for the target application. This approach allows us to climinate the redundant parts of the original LLMs and retain only the components essential for the target application. To handle the data distribution differences between the pretraining dataset and the target application, we also augment the model with new bases learned from the training set of the target application during the pruning process. This enables us to learn the new bases necessary for the target application that are absent in the pretrained model.
We evaluate Basel across multiple settings. First, for mathematical reasoning and code generation, we com- press Llama 2-7B and Llama 2-13B with Basel and mcasure pass@1 accuracy GSM8K (Cobbe et al. 2021 and MATH (Hendrycks et al. 2021 as well as on HumanEval (Chen et al. 2021a and MBPP (Austin ct al. 2021 J. Compared to the low-rank compression basclines SVD and FWSVD (Hsu et al.) 2022) Basel achieves up to 2.7x additional model size reduction while maintaining comparable accuracy on these mod- els and benchmarks. Second, for language modeling, we compress Llama-7B and evaluate perplexity on WikiText-2 (Merity al. 20166. In this case, Basel yields up 4x greater size reduction than SVD, FWSVD (Hsu et al. (2022). and SVD-LLM (Wang et al. 2025).. while also improving performance. Third, examine quantization by comparing Basel with the quantization baseline QLoRA (Dettmers et al. 2023
2

---

Published in Transactions on Machine Learning Research (12/2025)
and we assess pruning by comparing Basel against FLAP An et al. 2024 and Wanda (un et al. 2024 under both quantized and unquantized settings. Finally, analyze Basel's 's system-level benefits, showing improvements in token throughput and reductions in memory footprint.
This paper makes the following critical contributions:
We analyze the relationship between pretrained models and target applications, highlighting the opportunity and underlying ralionale for using low-rank decomposition lo compress large language models while maintaining performance on target applications.
We propose Basel, low-rank decomposition approach to compress pretrained large language models for target applications. Basel identifies the beneficial and redundant components of large language models by relearning the importance (i.e., singular values) of bases using the training set. of the target application, and then selects bases based on their importance.
We evaluate Basel across multiple tasks and models, demonstrating its superior performance in deep compression.
A preprint of this work is available on arXiv Li et al. 2024a and the source code of the work is publicly available al htttp:://githu. wttp:///ooeetttatttttiiirrereee AI S-sst------ttttt---orp/aassss
2 Related Work
Singular value decomposition (SVD) (Golub & Van Loan 1996) has been applied to reduce the size of machine
learning models. Prior research [ue et al. 2013 Jaderberg et al. 2014 Denton et al.) 2014 Zhang et al.
Acharya et al. 2019: Noach & Goldberg 2020 has developed various SVD algorithms to compress different components of models, such as DNNs, CNNs, and embedding layers for a range of applications inchuding natural language processing, speech, and vision. The primary distinction between our work and these prior studies is that they do not relearn the importance of bases using the training data of target applications. Instead, they typically prune bases according the singular values from the original or finetuned models. FWSVD (Hsu et al. 2022) evaluates the importance of individual weights rather than bases during SVD. Considering the importance of bascs may improve performance. As shown in Section our approach surpasses FWSVD in deep compression performance.
2015 Povey et al.) 2018 Chen et al. 2018
A recent study (Sharma et al. 2024) applied SVD to large language models. Its focus is on determining the optimal rank for each layer, while our emphasis is on basis selection. The two methods are orthogonal but complementary and can be combined. Chen et. al. (2021b); Yu & Wul (2023 Yuan et al. (2023 Wang
et al. 20255; Chowdhury et al. (2025) propose reconstructing bases by minimizing the discrepancy between activations before and after compression. These methods are also orthogonal to ours: they concentrate on basis reconstruction, whereas we focus on basis selection. In principle, their techniques may be integrated with ours to achieve stronger compression performance.
Beyond pruning bases, compression can also be achieved by pruning at other granularities, such as weight pruning (An al. 2024 Sun al. 2024) and layer pruning (Hu et al. 2025).
3 Basel
In this section, we describe our proposed compression method, Basel.
For linear layer y Wx+ b, Singular Value Decomposition (SVD) factorizes its weight matrix W paxm as the product of three matrices U, S, and V:
W USV
(1)
3

---

Published in Transactions on Machine Learning Research (12/2025)
where U= = [ui.... u S diag ($1.-.. s,), and V [v1, vyl. The values {s; E &R,i=1 1.... r) are
positive singular values The vectors (n, E IR"; i 1, .r and {v; E RR" i 1,.... .r) are orthonormal,
i.e, L norms Therefore, we can factorize matrix W as the following series:
||ull22 1, l|villa = 1, Euclidean inner products (u,, uz) = (vi, v2) = 0, for i j.
W Σ s...,
(2)
Let matrix W, = u.vi then
Frobenius norm WillF = Vtr (W. w.) (v.u..vvv)) vur(viV)) vir(vTT.) =
Frobenius inner product (W,, W tr (W. w,) = =r (v.u.u.v) 0, i
Therefore, (..v i = 1. r can be seen as a group of orthonormal bases in a subspace of g="x and {st, 1.... ,r) are their weights, making the weight matrix W a lincar combination of these bases.
This group of bases be viewed series of filters that manipulate the input signal produce the output signal y:
y = - Σ s.u.
St vx) u. +
In other words, for each basis (i.e., filter) u.v. the similarity between the input signal x and the unit direction vector V, is measured by their inner product. This inner product is then multiplied by the (positive) singular value to determine the weight for the unit direction vector u, The output signal y is the weighted sum of Figure illustrates this interpretation of the role of bases from the perspective of signal processing.
$1
U1 V
X
Sr
u V
Figure 2: An interpretation of the role of bases from the perspective of signal processing.
Large language models pretrained diverse datasets learn bases that capture wide range of semantic concepts. To examine this, factorize the W W/y matrix from the attention layers of both the original Llama 2-7B model and its math-finetuned counterpart. Focusing bases associated with large singular values, we extract the corresponding vectors and interpret their semantics via a de-embedding process. Specifically, we project each u vector into the vocabulary space by passing it through the output layer (i.e., the Im lm_ mo_hca)), yiclding a logit vector over the vocabulary. We then identify the token with the highest logit as the de-embedding result. Table summarizes our findings. First, some bases are associated with semantic content, such as concepts related Python, locations, non-English characters, and math symbols. Sccond, even within the same layer, different bases may correspond to different meanings. For example, in the matth-inettued model at layer 22, altention head 6, basis 1 corresponds to non- non-English characters, while basis 3 corresponds to math symbols. Revealing the full semantic structure of these bases requires more in-depth analysis, such as the approaches explored in Oikarinen et al. 2025), which we leave for future work.
We drop sero singular values and the corresponding columns matrices and V.
4

---

Published in Transactions on Machine Learning Research (12/2025)
Table 1: Meaning of bases in the vanilla and math-finetuned Llama 2-7B.
Domain
Basis
Top ten most probable tokens corresponding to the basis
Vanilla model, Layer 17,
Python
_in
<0x0A>
and _o for
Head 6, Basis I
Vanilla model, Layer 17,
Massachusetts
_Boston Harvard _Connecticut ILL Bulg BBllarr
Illinois
Chicago
DDan
Location
Head 25, Basis 1 Math-finetunee model,
Non-Fngglihh
1 Yā AE
A2
Layer 22, Head 6, Basis I
Math-finetuner model,
Math
{ H rr ={ L' _" H
-4{ _0;
Layer 22, Head 6, Basis 3
These findings suggest that some bases in the model are useful for some tasks, but may be irrelevant for others. When these irrelevant bases are used as filters in non-target applications, two scenarios can occur: the filter may not be activated (due to a small inner product v:)), or worse, the filter is activated, introducing harmful information into the output and degrading performance. This indicates that pruning such bases could reduce model size with minimal performance loss, and in some cases, even enhance performance for the target application.
In our approach, Basel, we determine the importance of the bases from the pretrained model by retraining their singular values on the training set of the target application. The weight matrix W in Basel is represented as:
W Σ S. + Juy 1=1
(3)
In the first term, u.v. represents the original bases in the pretrained model. To assess their importance, we initialize their weights ä; with their original singular values and then retrain these weights (while keeping the bases fixed) on the training set of the target application. The aim is that, after retraining, the bases important for the target application will have larger singular values, whereas those that are useless or detrimental will have zero or very small singular values. This allows us to identify and prune the less useful bases. Relcarning the importance of bases for the target. application distinguishes our approach from previous methods. Prior approaches either use the singular values in the original model (Xue et al., 2013 Jaderberg et al. 2014
Denton et al.] 2014: Zhang et al., 2015 Povey et al. 2018 Chen et al.) 2018 Acharya et al.) 2019 Noach & Goldberg, 2020: Sharma et al. 2024) or assess the importance of weight parameters, other than the importance of the bases, to prune them (Hsu et al. 2022). They do not relearn the importance of the bases useful for the target application. From a signal processing perspective, this first term allows us to adjust the weight for each filter, catering to the needs of the target application.
In the second term, the vectors ü, and Vj are learnable and serve two main purposes. First, due distribu- tional differences between the pretraining data and the target application, some bases essential for the latter may be missing. These vectors are used to learn such missing bases. Second, although each pruned basis may have little effect individually, their collective removal can lead to a noticeable performance drop. The additional vectors help mitigate this loss. The mumber of learnable vectors, denoted F, is called the additional dimension. From a signal processing perspective, the second term introduces new filters that enhance the model's performance on the target task. While this term adds new parameters, they do not fully contribute to model size growth. This is because SVD is eventually applied to the sum of the first and second terms, reducing the rank if dependencies exist between the newly learned and original, kept bases.
Algorithm presents our proposed approach. Basel operates on pretrained or fine-tuned model as in- put. Similar to Hsu et al. (2022;; Sharma et al. (2024), Basel iteratively prunes the bases of all linear layers- excluding the embedding layer including the Wo, WK, Wy. and Wo matrices in the attention layer the linear layers in the feedforward layers, and the output layer. The same compression process is
In our implementation, matrices Wy and Wo are pruned independently. Exploring alternative strategies, such as jointly pruning Wy and Wo, left for future work.
5

---

Published in Transactions on Machine Learning Research (12/2025)
Algorithm 1: Basel Algorithm
Input: Pretrained Finetuming Model M
Output: Compressed Model M'
Data: Hyperparameters including KeepRatio, PrumingTimes KeepingEpoch, PruningEpoch,
Bosd"inneTuuinnEEppor F
Iuerang round (NuntterationsPertppcd PruningEpoch Pruning'Timess
Peenruniioee/Puuingg KeepRatio (IARRaatioO//uiiiii (1/PruningTTims))
Convert the weight matrix of each linear layer in M M-excluding the embedding layer-into the form specified by
equation 3
for i == to Keehhnggpoch do Tune the learnable parameters in equation 13 including s., ūy V,, i 1, ..... , ,, end
for i == to ProningEpoch do Tune the learnable parameters; if IterationID is a multiple of IterationsPerPruinng then for each linear layer do
10
11
Prune bases with smaller singular values & such that after pruning, the sum of the singular values of the remaining bases is
of the sum before pruning;
end
13
end
14 end
16
for each layer do
16
Compute the low rank matrix based on equation
SU','', V)= SVD(W;;;
17
18
Use two linear layers to substitute for the original layer;
The first layer's weight matrix is S'V'
19
20
The second layer's weight matrix is U'
end
22
for == to PostFine TaniFinneTnnngggaae Epoch do
FineTune the new model M';
24 end
applied across these components. After each pruning step, the learnable parameters 8y ûy and v, are fine- tuned to compensate for performance degradation. Ultimately, a new weight matrix W with smaller rank r' is learned. We perform a standard SVD on it, representing it as the product of matrices U', S', and V We then replace the original layer with two new layers: S'V' becomes the weight matrix of the first new layer, and U' becomes the weight matrix of the second new layer. This reduces the number of parameters from nm to (n + m)r'. The new model is subsequently further finetuned enhance its performance on the target application.
Overhead Analysis. Basel updates only the singular values and the newly introduced bases, while keeping the original bases frozen. Unlike full fine-tuning, Basel involves far fewer learnable parameters, substantially reducing training overhead. Table 2 compares Bascl and full finc-tuning on Llama-2-7B in terms of GPU hours and memory consumption, evaluated on NVIDIA L40S GPUs (batch size = 32, max sequence length = 512). Under this setting, Basel fits on a single L40S GPU, whereas full fine-tuning requires at least three L40S GPUs due to its higher memory demand. Overall, Basel consumes only about 46% of the GPU hours and 30% of the GPU memory used by full fine-tuning.
Table 2: GPU hours and GPU memory consumption of Basel versus full fine-tuning on Llama 2-7B using NVIDIA L40S GPUs (batch size 32, max sequence length 512).
GPU Hours per Batch
Total GPU Memory (GB)
Bascl Full fine-tuning
2.87 10-3
40.7
6.19 10-3
136.2

---

Published in Transactions on Machine Learning Research (12/2025)
4 Experiments
4.1 Evaluation Methodology
We evaluate the performance of low-rank compression algorithms on three tasks: mathematical reasoning, code generation, and language modeling. For the mathematical reasoning task, we utilize two evaluation datasets: GSMSK (Cobbe al. 2021 and Hendrycks' MATH (Hendrycks al. 2021]. The GSM8K dataset comprises verbally described mathematical questions, containing 1,319 samples used for evaluation. The Hendrycks' MATH dataset covers more complex topics such linear algebra and geometry, consisting of 5, 5,000 question-answer pairs used for evaluation. Due to its complexity, the Hendrycks' MATH datasct neces- sitates more sophisticated computations and reasoning, resulting in lower accuracy compared to GSM8K. For the code generation task, we use two evaluation datasets: MBPP (Austin et al. 2021) and HumanEval (Chen et al. 2021a). Both datasets evaluate the models' ability to generate Python code. MBPP comprises 500 code generation questions, while HumanEval inchudes 164 questions. For the language modeling task, we evaluate on WikiText-22 (Merity et al. 20161, dataset consisting of Wikipedia articles.
We evaluate Basel against low-rank compression methods SVD, FWSVD, and SVD-LLM proaches from other compression paradigms. SVD, a widely used technique in prior work on model com- pression (Sharma et al. 2024 Acharya et al.]
well as ap-
Noach & Goldberg. 2020 Xue et al. 2013 Jaderberg et al. 2014 Denton et al. 2014 Zhang et al. 2015 Povey et al. 2018), is applied in our experiments to
2019
compress models that have already been fine-tuned on the larget lask. FWSVD (Hsu al., 2022) extends
SVD by incorporating gradient- based profiling: it estimates weight importance using gradients computed on the fine-tuning dataset and leverages this information during decomposition. SVD-LLM (Wang et al. 2025
reconstructs weight matrices in low-rank form by minimizing the discrepancy between activations belore and after decomposition. Beyond low-rank approaches, we also compare Basel with quantization and pruming methods. QLoRA (Dettmers et al. 2023) is a quantization- training LoRA adapters on the fine-tuning dataset. FLAP (An et al. 2024) and Wanda (Sun et al., 2024 are pruning-basee based methods that prune weights based on weight magnitudes, logether with either the magnitude (Wanda) or the variance (FLAP) of input activations profiled on the target task.
technique that improves performance by
Our implementation of Basel is configured with the following key hyperparameters: KeepRatio varies from 70% to 5%, PruningTimes 100, KeepingEpoch = 1, PruningEpoch = 2 (for math reasoning and code gencration) and (for language modeling), PostFine TuningEohoh = 3, and F = 32 (sce Algorithm for further details).
4.2 Evaluating Low-Rank Compression Methods and Pruning Methods for Mathematical Reasoning
Figures (a) and (b) depict the performance of various low-rank compression algorithms on the Llama 2-7B model (Touurron et al. 2023b for the mathematical reasoning task. We evaluate the models' accuracy
60
SVD
WANDA
20
SVD
WANDA
FWSVD
Basel
FWSVD
Basel
Accuracy (%)
FLAP
Accuracy (%)
15
FLAP
40
10:
20
0
2
7 12
1
2
4 6
8 99999 10 12 16
Compression Ratio
Compression Ratio
(a) GSM8K
(b) MATH
Figure 3: Pass@1 accuracy and model size of Llama 2-7B compressed with various low-rank algorithms and
pruning algorithms the mathematical reasoning task. Fxact values are listed in Tables 4 and in the appendix.

---

Published in Transactions on Machine Learning Research (12/2025)
SVD FWSVD FLAP
WANDA Basel
SVD FWSVD
WANDA
20.
60
Basel
Accuracy (%)
Accuracy (9)
15-
FLAP
40
10-
20
5-
01
200
2
8
12 14
16 18 20
2
6
10 12
16
18
20
Compression Ratio
Compression Ratio
(a) GSMSk
(b) MATH
Figure 4: Pass@1 1 accuracy and model size of Llama 2-13B compressed with varions low-rank algorithms and
pruning algorithms on the math reasoning task. Exact values are listed in Table 6 and in the appendix.
SVD
SVD
Accuracy (%)
20
FWSVD
Accuracy (9)
20
FLAP
FWSVD
FLAP
15-
WANDA
WANDA
10
Basel
10
Basel
5
0
0
2 13 14 15 Compression Ratio
6 7
10 11
3
10 12 13 15 16 Compression Ratio
(a) HumanEval
(b) MBPP
Figure 5: Pass@1 accuracy and model size of Llama 2-7B compressed with various low-rank algorithms and pruning algorithms the code generation lask. Exact values are listed in Tables and in the appendix.
(Pass@1) at different compression ratios (original model size vs. compressed model size). For low compression ratios (below 6), all low-rank compression methods achieve similar accuracy. However, our Basel method significantly outperforms SVD and FWSVD at higher compression ratios. For instance, at 7x compression ratio, Basel achieves around 46% and 12% accuracy on GSM8K and MATH datascts, respectively, while FWSVD drops below 2% accuracy on both datasets and SVD reaches only 27% and 5% accuracy on GSM8K and MATH, respectively. We also find that Basel, at a compression ratio of 16, achieves better accuracy on both GSM8K and MATH compared to FWSVD and SVD at a compression ratio of 6. This suggests that Basel reduces the model size by up 2.7 times more than SVD and FWSVD while maintaining similar accuracy. This highlights the effectiveness of Basel for deep compression, especially when aiming for aggressive model size reduction.
Similar trends emerge for the larger Llama 2-13B model in Figures (a) and (b). Once again, Basel significantly putperfsss SVD and FWSVD at compression ratios exceeding 6. At a 7x compression ratio, Basel achieves 47% and 11% accuracy on GSMSK and MATH datasets, respectively, demonstrating its advantage. This is in stark contrast to SVD's performance (23% and 5% accuracy on GSM8K and MATH) and FWSVD's performance (around 2% on both datasets). These results solidify Basel's effectiveness for deep compression across different model sizes.
Figures 3 and also compare Basel with the pruning algorithms FLAP and Wanda on GSMSK and MATH.
The results show that Basel achieves substantially greater model size reduction while preserving comparable or superior performance.
4.3 Evaluating Low-Rank Compression Methods and Pruning Methods for Code Generation
Similar results extend to code generation tasks (Figures and 6). For both Llama 2-7B and Llama 2-13B models, all low-rank compression methods perform comparably at. lower compression ratios (below 4). How-

---

Published in Transactions on Machine Learning Research (12/2025)
30
SVD
SVD
FWSVD
FWSVD
Accuracy (%)
20-
Accuracy (9)
FLAP
20
FLAP
WANDA
WANDA
Basel
Basel
10
0
20
Compression Ratio
Compression Ratio
(a) Eval
(b) MBPP
Figure 6: Pass@1 1 accuracy and model size of Llama 2-13B compressed with varions low-rank algorithms and pruning algorithms on the code generation task. Fxact values are listed in Tables TO and in the appendix.
10c
10a
Perplexity
10
SVD
FWSVD
10a
SVD-LLM
Basel
10a
10
2
10 11
Compression Ratio
Figure 7: Perplexity and model size of Llama-7B compressed with various low-rank algorithms on WikiText-2. Lower perplexity indicates better language modcling performance. Results for SVD, FWSVD, and SVD- LLM are taken from [Wang al. 2025). Exact values are listed in Table 12 in the appendix.
ever, Basel exhibits clear superiority higher compression ratios. On Llama 2-7B a 6x compression ratio, Basel achieves 12% and 9% accuracy on HumanEval and MBPP datasets, respectively, significantly outper- forming SVD (5% and 2%) and FWSVD (6% and 2%). Similar trends hold for Llama 2-13B. Moreover, Basel consistently outperforms pruning baselines FLAP and Wanda, particularly under aggressive compression. These findings further solidify Basel's effectiveness for deep compression across diverse tasks and model sizes.
4.4 Evaluating Low-Rank Compression Methods for Language Modeling
For the language modeling task, we compress Llama-7B using Basel and baseline methods SVD, FWSVD, and SVD-LLM, and evaluate the resulting models in terms of perplexity. Perplexity serves measure of language modcling quality, with lower values indicating better performance. As shown in Figure 7 Basel consistently yields lower perplexity than SVD, FWSVD, and SVD-LLM the same compression ratio. Conversely, for a given perplexity, Basel produces a substantially smaller model. For instance, Basel achicves a perplexity of 10.45 at a 10x compression ratio, whereas SVD-LLM reaches a perplexity of 15.00 at only 2.5x. This demonstrates that Basel attains up to four times greater compression than SVD-LLM (and even more than SVD and FWSVD) while delivering superior performance.
4.5 Combining Low-Rank Compression and Quantization for Mathematical Reasoning
Quantization reduces the precision of weight parameters to shrink model size and is widely used technique for model compression. When using 8-bit quantization or higher, the performance degradation is typically minimal. However, more aggressive quantization (e.g., 4-bit) often leads to a significant drop in accuracy. We apply both 8-bit and 4-bit quantization to Llama 2-7B and Llama 2-13B models on mathematical reasoning.
For Llama 2-7B, the 8-bit quantized model achieves Pass@1 accuracy of 66.0% on GSMSK and 20.3% on MATH, closely matching the unquantized model's performance (66.4% and 20.6%, respectively, with bf16).

---

Published in Transactions on Machine Learning Research (12/2025)
20
Accuracy (%6
Accuracy (%)
40
8bit FLAP
FLAP
8bit WANDA
10
8bit WANDA
4bit Quant
4bit Quant
4bit QLoRA
QLORA
8bit Basel
0
8bit Basel
3
Model Size (GB)
Model Size (GB)
(a) GSM8K
(b) MATH
Figure 8: Pass@1 accuracy and model size of Llama 2-7B compressed using quantization alone, as well as quantization combined with low-rank methods or pruning, on the mathematical reasoning task. Exact values are provided in Table 13 in the appendix.
20
Accuracy (%)
60
Accuracy (%)
8bit FLAP
8bit FLAP
40
* 8bit WANDA
10
8bit WANDA
20
4bit Quant
Quant
4bit QLoRA
4bit QLoRA
8bit Basel
0
8bit Basel
14
12
10
8
4
2
0
14
12
10
8
6
4
2
Model Size (GB)
Model Size (GB)
(a) GSM8K
(b) MATH
Figure 9: Pass@l accuracy and model size of Llama 2-13B compressed using quantization alone, well quantization combined with low-rank methods pruning, the mathematical reasoning task. Exact values are provided in Table 14 in the appendix.
A similar trend holds for Llama 2-13B, where 8-bit quantization results only a negligible accuracy drop. In contrast, 4-bit quantization causes substantial degradation. On Llama 2-7B, accuracy drops from 66.0% 39.2% on GSM8K and from 20.3% 8.8% on MATH. Llama 2-13B exhibits similarly significant losses. These results suggest that relying solely quantization especially at lower bit-widths can be detrimental to performance.
To address this, we explore combining quantization with low-rank compression using our Basel method. Figures g and 9 illustrate that combining Basel with 8-bit quantization not only achieves a smaller model size but also significantly outperforms 4-bit quantization in accuracy. On compressing Llama 2-7B, Basel + 8-bit achieves 59.6% on GSM8K and 18. on MATH with 3.28 GB model size, compared to 4-bit quantization's 39.2% and 8.8% with a larger 3.37 GB model. On compressing Llama 2-13B, Basel + 8-bit yields 61.9% GSM8K and 17.5% MATH with 6.13 GB size, while 4-bit quantization achieves only 52.8% and 12.5% with 6.51 GB.
We further compare Basel + 8-bit quantization against three additional approaches: (1) QLoRA with 4- bit quantization, (2) combining 8-bit quantization with FLAP, and (3) combining 8-bit quantization with Wanda. As shown in Figures g and 9 Basel + 8-bit consistently outperforms both in terms accuracy and size. For instance, on compressing Llama 2-7B, improves accuracy by 5.5% on both GSM8K and MATH compared to QLoRA, while reducing the model size by an additional 0.13 GB.
4.6 Combining Low-Rank Compression and Quantization for Code Generation
We also compare Basel + 8-bit quantization to three alternative met hoth on the code generation task: (1) standard 4-bit quantization, (2) QLoRA with 4-bit quantization, (3) FLAP combined with 8- b-tit quantization, and (4) Wanda combined with 8-bit quantization. Figures 10 and 11 present the results for compressing Llama 2-7B and Llama 2-13B. On both models, Basel 8-bit quantization consistently outperforms 4-bit quantization while achieving smaller model sizes. For example, Llama 2-7B, it improves accuracy by
10

---

Published in Transactions on Machine Learning Research (12/2025)
Accuracy (%)
20
Accuracy (%)
8bit FLAP
8bit FLAP
8bit WANDA
8bit WANDA
10
4bit Quant
10
4bit Quant
4bit QLoRA
4bit QLORA
8bit Basel
0
8bit Basel
4 3
Model Size (GB)
Model Size (GB)
(a) HumanEval
(b) MBPP
Figure 10: Pass@1 accuracy and model size of Llama 2-7B compressed using quantization alone, as well as quantization combined with low-rank methods or pruning, on the code generation task. Exact values are provided in Table 15 in the appendix.
Accuracy (%)
Accuracy (%)
8bit FLAP
20
8bit FLAP
8bit WANDA
Sbit WANDA
10
4bit Quant
10
4bit Quant
4bit QLORA
4bit QLoRA
8bit Basel
0
8bit Basel
14
12
2
12
10
8
6
4
Model Size (GB)
Model Size (GB)
(a) HumanEval
(b) MBPP
Figure 11: Passãl accuracy and model size of Llama 2-13B compressed using quantization alone, as well quantization combined with low-rank methods pruning, the code generation task. Exact values are provided in Table 16 in the appendix.
SVD TP
FWSVD TP
BASELT TP
SVD Mem
FWSVD Mem
BASEL Mem
Throughput (Tokens/s)
20000
Memory (MiB)
15000
500
10000
400
5000
300
5
Model Size (x10* Parameters)
Figure 12: Throughput and memory consumption of compressed models.
12.8% on HumanEval and 6.4% on MBPP, with model size similar to that of the 4-bit. version. This further highlights the benefit of combining low-rank compression with quantization.
Basel + 8-bit quantization also significantly outperforms both QLoRA (4-bit), FLAP + 8-bit quantization, and Wanda + 8-hit quantization on Llama 2-7B and Ilama 2-13B. For instance, on compressing Llama 2-7B, it achieves 6.7% higher accuracy on HumanEvvl and 1.0% higher on MBPP compared to QLoRA, while also reducing the model size by an additional 0.17 GB.
4.7 Inference
Figure 12 presents the inference throughput and memory consumption of models compressed from Llama 2-7B single A100 GPU, using GSM8K as the evaluation set. The results show that low-rank
11

---

Published in Transactions on Machine Learning Research (12/2025)
Accuracy (%)
60
Addt'l Dim
Accuracy (%)
20
Addt'l Dim 32
Addt'l Dim
Addt'l Dim
10
20
0
0
9
13
15
17
3
11
13
15
17
Compression Ratio
Compression Ratio
(a) GSM8k
(b) MATH
Figure 13:
Ablation study: Impact of varying the additional dimension in Basel on the compression of
Llama 2-7B for the mathematical reasoning task. Exact values listed in Table 19 in the appendix.
Accuracy (%6
60
100 times times
Accuracy (%)
20
100 times
2 times
40
10
20
0
5
11 Compression Ratio
9
13
15
17
3
7
11
13
15
17
Compression Ratio
(a) GSM8k
(b) MATH
Figure 14: Ablation study: Impact of varying the pruning times in Basel on the compression of Llama 2-7B for the mathenatical reasoning task. Exact values are listed in Table 20 in the appendix.
pression methods, including SVD, FWSVD, and Basel, lead to reduced memory consumption and improved throughput as the model size decreases. Throughput and memory usage are primarily dependent on model size, with no significant differences between the methods at equivalent sizes. However, since our proposed Basel method achieves a greater reduction model size while maintaining similar accuracy to SVD and FWSVD, it improves throughput by up to 19% and reduces memory consumption by up 37%.
4.8 Ablation Study
We conduct a series of ablation studies to better understand the design of Basel.
Impact of hyperparameters. We first evaluate the effects of two key hyperparameters: the additional dimension (F in Equation 33) and the number of pruning iterations. The additional dimension is intro- duced help recover information lost during pruning, particularly under high compression. As shown in Figure 13 setting = 32 improves accuracy when compressing Llama 2-7B for mathematical reasoning, especially beyond a 7x compression ratio. However, this accounts for only small fraction of Basel's 's overall performance gain relative to baselines (Figures 3 and 13 Similarly, applying pruming gradually over many iterations allows the model to better adapt to parameter reduction, which is especially beneficial under ex- treme compression. Figure 14 illustrates this effect: pruning 100 times consistently yields higher accuracy than pruning only twice when the compression ratio exceeds 4.
Freezing basis vectors. Basel freezes the basis vectors and updates only the singular values (and any additional bases) during compression. To evaluate this design choice, compare Basel with variant that also updates the basis vectors. As shown in Figure 15 freezing the bases results in better accuracy on Llama 2-7B for mathematical reasoning. This strategy also reduces compression time by 33%, since backpropagation involves fewer trainable parameters.
Inclusion of L1 rogularization. Finally, we examine the effect of adding an L1 penalty on the learnable singular values s, The goal is to encourage sparsity in š;, thereby enabling additional compression. We experiment with two regularization weights, À 0.02 and X 0.1. Figure 16 reports the results. With
12

---

Published in Transactions on Machine Learning Research (12/2025)
Accuracy (%)
60
Accuracy (%)
20
40
10
Basel (Proposed)
Basel (Proposed)
Basel (Free Basis)
Basel (Free Basis)
0
3
Model Size (billion)
Model Size (billion)
(a) GSM8K
(b) MATH
Figure 15: Ablation study: Pass01 accuracy and size of Llama 2-7B compressed by Basel (proposed) and
Basel (free basis) on mathematical reasoning. Exact values are listed in Table 21 in the appendix.
Basel A=0.1)
Basel (L1, A=0.1)
60
Basel (L1, x=0.02)
Basel (L1, x=0.02 02)
Accuracy (%)
Accuracy (%)
50
Basel (Proposed)
15
Basel (Proposed)
40
10-
30
20
2
3
4
5
6
7
8
9
10
12
13 14 15 16
1
2
4
5
12 13 14 15 16 Compression Ratio
6
8
1011
Compression Ratio
(a) GSM8K
(b) MATH
Figure 16: Ablation study: Pass&1 accuracy and model size of Llama-2-7B compressed with Basel, with and without L1 regularization, on the mathematical reasoning task. Exact values are reported in Table 22 in the appendix.
= 0.02, performance is similar to standard Basel; however, a larger weight (A = 0.1) degrades performance under deep compression.
4.9 Compression Training from Scratch
In many real-world LLM deployment scenarios, compute and hardware constraints impose target model size tailored to a specific application, which may not match any existing pretrained models. For example, Llama-2 offers 7B, 13B, and 70B variants, but no 4B option. To obtain a model of the required size for the target application, one can either (1) train a new model from scratch or (2) compress a nearby larger model. The first approach is often impractical due high computational costs training Llama reportedly requires up to one million A100 GPU hours Touvron et al. (2023a) and limited access to pretraining data. In contrast, model compression far more efficient; low-rank methods such Basel typically require less than 0.01% of the pretraining compute.
To assess the effectiveness of compression, we compare finc-tuned Llama 3.2-1B model with compressed version of Llama 3.2-3B reduced to 1B parameters using Basel. As shown in Table 3 the compressed model achieves similar performance on mathematical reasoning. This indicates that, in this setting, compression can
Table 3: @ accuracy and model size of Llama 3.2-3B
compression via Basel for mathematical reasoning.
3B
1B
Basee-coompresed--33
Model Size (billion)
3.21
1.24
1.24
GSMSK Acc (%)
72.5
54.4
55.3
MATH Acc (%)
26.1
17.6
16.7
13

---

Published in Transactions on Machine Learning Research (12/2025)
serve as viable alternative to training from scratch offering substantial savings in compute and eliminating the need for pretraining data. These results demonstrate the potential of Basel to support efficient. model scaling under deployment constraints.
scratch-offering
5 Conclusion
The significant size of large language models leads to high inference costs and demands substantial comput- ing resources. To mitigate these issues, we focus on compressing large language models to meet the specific requirements of target applications. Our approach involves examining these models through the lens of ma- trix factorization. By viewing the weight matrix large language models as a linear combination of a group of bases, we have identified that pretrained models often contain many redundant bases that are less useful for target applications. To address this, we propose Basel, compression algorithm that evaluates the im- portance of cach base for target applications and prunes those that are less significant. Experimental results remonstrrte that Basel significantly outperforms baseline low-rank compression algorithms in achieving deep compression. Basel greatly reduces the inference cost of large language models, making them more accessible and practical for a wider range of applications. This advancement has the potential to democratize the use of large language models, facilitating their adoption and integration across diverse fields and industries.
Acknowledgments
This research is partially supported by Meta and by a faculty startup grant from Iowa State University. Com- putational resources are provided by Meta and HPC@ISU, The resources from HPC@ISU include equipment funded by the U.S. National Science Foundation under MRI Grant Nos. 1726447 and 2018594.
References
Anish Acharya, Rahul Goel, Angeliki Metallinou, and Inderjit Dhillon. Online Embedding Compression for Text Classification Using Low Rank Matrix Factorization. In AAAI, 2019.
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-based Adaptive Structured Pruning for Large Language Models. In AAAT, 2024.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Fllen Jiang, Carric Cai, Michael Terry, Quoc Le, and Charles Sutton. Program Synthesis with Large Language Models. Xixi preprinl arXio:2108.07720 2021.
Ernie Chang, Pin-Jie Lin, Yang Li, Changsheng Zhao, Daeil Kim, Rastislav Rabatin, Zechun Liu, Yangyang Shi, and Vikas Chandra. Target-Aware Language Modeling via Granular Data Sampling. In EMNLP, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating Large Language Models Trained on Code. arXiv preprial arXiv:2107.03374 2021a.
Patrick Chen, Si Si, Yang Li, Ciprian Chelba, and Cho-Jui Hsieh. GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking. In NeurIPS, 2018.
Patrick Chen, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. DRONE: Data-Aware Low-Rank Com- pression for Large NLP Models. In NeurIPS, 2021b.
Md Mokarram Chowdhury, Daniel Agyei Asante, Ernie Chang, and Yang Li. Importance-Aware Activation Space Reconstruction. arXie preprint NXii:2507.03828 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems. arxiv preprint ar Xiv:22110..11166 2021.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Emily L. Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting Linear Structure within Convolutional Networks for Efficient. Evaluation. NearIPS, 2014.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. In NewrlPS, 2023.
Gene H. Golub and Charles F. Van Loan. Matriz Computations. Johns Hopkins University Press, Baltimore and London, 1996.
Gemini Team Google. Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312. 11805, 2023.
ar preprint
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving with the MATH Dataset. In NewrIPS, 2021.
Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language Model Compression with Weighted Low-Rank Factorization. In ICLR, 2022.
Lanxiang Hu, Tajana Rosing, and Hao Zhang. TrimLLM: Progressive Layer Dropping for Domain-Speeite LLMs. In ACI, 2025.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding Up Convolutional Neural Networks with Low Rank Expansions. arXiv preprint arXic:1405.3800 2014.
Yang Li, Daniel Agvei Asante, Changsheng Zhao, Ernie Chang, Yangyang Shi, and Vikas Chandra. Basis Selection: Low-Rank Decomposition of Pretrained Large Language Models for Target Applications. arziv preprint arXie:2405. 15877, 2024a.
Yang Li, Liangzhen Lai, Yuan Shangguan, Forrest N. landola, Ernie Chang, Yangyang Shi, and Vikas Chan- dra. Folding Attention: Memory and Power Optimization for On-Device Transformer-based Streaming Speech Recognition. In ICASSP, 2024b.
Yang Li, Yuan Shangguan, Yuhao Wang, Liangzhen Lai, Ernie Chang, Changsheng Zhao, Yangyang Shi, and Vikas Chandra. Breaking Down Power Barriers On-Device Streaming ASR: Insights and Solutions. In NAACL (Industry Trock), 2025.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer Sentinel Mixture Models. arXiv preprint ar ar 1609. 160.007443 2016.
Matan Ben Noach and Yoav Goldberg. Compressing pre-trained language models by matrix decomposition. In AACL/IJCNLP, 2020.
Tuomas Oikarinen, Ge Yan, and Tsui- Weng. Evaluating Neuron Explanations: Unified Framework with Sanity Chocks. In ICML, 2025.
OpenAl et al. GPT-4 Technical Report. ar preprint #xXXx:2303.08777, 2024.
Daniel Povey, Gaofeng Cheng, Yiming Wang, Ke Li, Hainan Xu, Mahsa Yarmohammadi, and Sanjeev Khu- danpur. Semi- Orthhgonllll Low-Rank Matrix Factorization for Deep Neural Networks. In INTERSPEECH, 2018.
Pratyusha Sharma, Jordan T. Ash, and Dipendra Misra. The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction. In ICLR, 2024.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A Simple and Effective Pruning Approach for Large Language Models. In ICLR, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and Efficient Foundation Language Models. Xix preprint or Xiv:2330 13971, 2023a.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open Foundation and Finetuned Chat Models. arXiv preprint 0rXiu2237.099880 2023b.
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD-LLM: Truncation-aware Singular Value Decom- position for Large Language Model Compression. In ICLR, 2025.
Jian Xue, Jinyu Li, and Yifan Gong, Restructuring Deep Neural Network Acoustic Models with Singular Value Decomposition. INTERSPEECH, 2013.
Hao Yu and Jianxin Wu. Compressing Transformers: Features Are Low-Rank, But Weights Are Not! In AAAI, 2023.
Zhihang Yuan, Yuzhang Shang, Yue Song, Dawei Yang, Qiang Wu, Yan Yan, and Guangyu Sun. ASVD: Activation-aware Singular Value Decompasition for Compressing Large Language Models. arXiv preprinl arXiv:2312.08 05821, 2023.
Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating Very Deep Convolutional Networks for Classification and Detection. JEEE Transactions Pattern Analysis and Machine Intelligence, 38 (10):1943 1955, 2015.
Appendix
Table 4: Pass@l accuracy and model size of Llama 2-7B compressed with various low-rank algorithms on the mathematical reasoning task.
Model Size (billion)
6.74
5.02
3.18
1.73
1.11
0.56
SVD
GSM8K Acc. (%)
66.4
63.0
61.0
53.9
32.9
11.9
MATH Acc (%)
20.6
18.3
17.4
13.7
5.3
2.8
Model Size (billion)
6.74
4.79
2.95
1.54
0.96
0.47
FWSVD
GSM8K Acc (%)
66.4
62.7
62.5
56.5
1.5
1.9
MATH Acc (%)
20.6
19.2
17.6
14.2
1.8
1.5
Model Size (billion)
6.74
5.14
3.28
1.83
1.21
0.67
0.43
Basel
GSM8K Acc (%)
66.4
63.8
61.6
56.2
50.2
40.2
34.2
MATH Acc (%)
20.6
19.7
17.6
15.5
13.4
10.0
7.8
Table 5: Pass@1 accuracy and model size of Llama. 2-7B compressed with various pruning algorithms on the mathematical reasoning task.
Model Size (billion)
6.74
5.39
4.72
4.04
3.37
2.70
1.35
FLAP
GSM8K Acc (%)
66.4
55.0
40.7
24.8
9.10
0
0
MATH Acc (%)
20.6
14.5
8.4
4.2
1.5
0
0
Model Size (billion)
6.74
5.39
4.72
4.04
3.37
2.70
1.35
Wanda
GSMSK Acc (%)
66.4
61.0
44.6
19.0
0.5
0.3
0
MATH Acc (%)
20.6
16.7
12.1
3.5
0.1
0.1
16

---

Published in Transactions on Machine Learning Research (12/2025)
Table 6: Pass@1 accuracy and model size of Llama 2-13B compressed with various low-rank algorit hiss on the mathematical reasoning task.
Model Size (billion)
13.02
9.70
6.10
3.27
2.07
1.01
SVD
GSM8K Acc (%)
72.7
69.5
63.5
50.0
26.9
6.7
MATH Acc (%)
22.2
20.8
17.8
10.8
5.2
2.2
Model Size (billion)
13.02
9.24
5.67
2.93
1.79
0.83
FWSVD
GSM8K Acc (%)
72.7
67.9
63.9
51.9
2.4
3.9
MATH Acc (%)
22.2
20.3
18.0
12.4
1.2
1.9
Model Size (billion)
13.02
9.75
6.13
3.32
2.14
1.12
0.68
Basel
GSM8K Ace (%)
72.7
67.9
64.4
55.0
50.3
41.8
37.9
MATH Acc (%)
22.2
20.9
18.7
15.5
13.1
10.4
8.0
Table 7: Pass@1 accuracy and model size of Llama 2-13B compressed with various pruning algorithms on the mathematical reasoning task.
Model Size (billion)
13.02
10.41
9.11
7.81
6.51
5.21
3.90
FLAP
GSMSK Acc (%)
72.7
60.7
51.2
4.6
0.2
0
MATH Acc (%)
22.2
14.5
10.2
4.5
0.4
0
Model Size (billion)
13.02
10.41
9.11
7.81
6.51
5.21
3.90
Wanda
GSM8K Acc (%)
72.7
66.0
50.3
0.1
0
0
MATH Acc (%)
22.2
17.9
12.0
0.4
0.1
0
Table 8: Pass@1 accuracy and model size of Llama 2-7B compressed with various low-rank algorithms the code generation task.
Model Size (billion)
6.74
5.02
3.18
1.73
1.11
0.56
SVD
HunanEval Acc (%)
23.8
20.7
20.1
9.1
4.9
3.7
MBPP Ace (%)
27.4
21.8
18.6
9.6
2.0
0.4
Model Size (billion)
6.74
4.84
3.01
1.58
0.99
0.49
FWSVD
HunanEval Acc (%)
23.8
22.0
20.1
11.6
4.9
0
MBPP Ace (%)
27.4
24.4
17.4
10.4
0.6
Model Size (billion)
6.74
5.14
3.28
1.83
1.21
0.67
0.43
Basel
HunanEval Ace (%)
23.8
22.0
20.7
14.6
12.8
7.9
7.3
MBPP Ace (%)
27.4
26.6
18.6
12.2
8.8
7.4
4.0
Table 9: Pass@1 I accuracy and model size of Llama 2-7B compressed with various pruning algorithms on the code generation task.
Model Size (billion)
6.74
5.40
4.72
4.04
3.37
2.70
1.35
FLAP
HumanEval Acc (%)
23.8
17.7
9.1
4.3
1.2
0.6
MBPP Acc (%)
27.4
22.0
14.4
2.8
1.4
0.2
Model Size (billion)
6.74
5.40
4.72
4.04
3.37
2.70
1.35
Wanda
HumanEval Acc (%)
23.8
18.9
15.9
3.7
0
0
MBPP Acc (%)
27.4
24.4
19.4
7.2
0
17

---

Published in Transactions on Machine Learning Research (12/2025)
Table 10: Pass@1 accuracy and model size of Llama 2-13B compressed with various low-rank algorithms the code generalion task.
Model Size (billion)
13.02
9.70
6.10
3.27
2.07
1.01
SVD
HumanEval Acc (%)
27.4
18.9
18.3
3.0
3.7
0.6
MBPP Acc (%)
30.0
25.4
18.2
10.6
1.4
0.6
Model Size (billion)
13.02
9.31
5.73
2.97
1.83
0.85
FWSVD
HumanEval Ace (%)
27.4
26.2
20.1
8.5
3.7
0
MBPP Acc (%)
30.0
27.2
21.6
12.2
0.8
0
Model Size (billion)
13.02
9.75
6.13
3.32
2.14
1.12
0.68
Basel
HumanEval Acc (%)
27.4
26.2
22.0
15.2
12.8
7.9
7.3
MBPP Acc (%)
30.0
27.8
20.6
13.6
10.8
6.4
3.6
Table 11: Pass@1 accuracy and model size of Llama 2-13B compressed with various pruning algorithms the code generation task.
Model Size (billion)
13.02
10.41
9.11
7.81
6.51
5.21
3.90
FLAP
HumanEval Acc (%)
27.4
17.1
15.9
5.5
2.4
0.6
0
MBPP Acc (%)
30.0
20.8
16.0
5.6
2.8
0.6
0
Model Size (billion)
13.02
10.41
9.11
7.81
6.51
5.21
3.90
Wanda
HumanEval Acc (%)
27.4
16.5
12.2
0
0
MBPP Ace (%)
30.0
21.8
13.8
0
0
Table 12:
Perplexity and model size of Llama-7B compressed with various low-rank algorithms on WikiTexi-
2. The results of SVD, FWSVD, and SVD-LLM are cited from WWag 2025).
Model Size (billion)
6.74
5.10
3.88
2.68
1.29
SVD
Perplexity
5.68
20061
52489
105474
687291
Model Size (billion)
6.74
5.10
3.88
2.68
1.29
FWSVD
Perplexity
5.68
1727
18156
32194
96872
Model Size (billion)
6.74
5.10
3.88
2.68
1.29
SVD-LLM
Perplexity
5.68
7.73
9.27
15.00
31.79
Model Size (billion)
6.74
5.15
4.08
3.21
1.83
1.22
0.67
Basel
Perplexity
5.68
6.11
6.68
6.99
7.93
9.21
10.45
18

---

Published in Transactions on Machine Learning Research (12/2025)
Table 13: P I accuracy and model size of Llama 2-7B compressed and 8-bit-quantized by various algo-
rithms for mathematical reasoning.
Model Size (GB)
6.74
5.39
4.72
4.04
3.37
2.70
1.35
8-bit FLAP
GSM8K Ace (%)
66.0
53.4
39.1
22.7
7.4
0
MATH Acc (%)
20.3
13.5
7.2
4.4
1.0
0
Model Size (GB)
6.74
5.39
4.72
4.04
3.37
2.70
1.35
8-bit Wanda
GSM8K Acc (%)
66.0
59.7
44.4
15.8
0.3
0.2
0
MATH Ace (%)
20.3
16.3
11.3
2.7
0.1
0
0
Model Size (GB)
6.74
5.14
3.28
1.83
1.21
0.67
0.43
8-bit Basel
GSM8K Acc (%)
66.0
62.1
59.6
54.1
49.7
41.1
35.3
MATH Acc (%)
20.3
18.2
18.1
15.7
13.1
10.1
7.5
Table 14: Pass@1 accuracy and model size of Llama 2-13B compressod and 8-bit-quantized by various algorithms for mathematical reasoning.
Model Size (GB)
13.02
10.41
9.11
7.81
6.51
5.21
3.90
8-bit FLAP
GSM8K Acc (%)
72.7
59.5
53.9
6.1
0.2
0
0
MATH Acc (%)
21.8
13.9
9.4
4.8
0.1
0
Model Size (GB)
13.02
10.41
9.11
7.81
6.51
5.21
3.90
8-bit Wanda
GSM8K Acc (%)
72.7
66.4
50.5
0.2
0
0
MATH Acc (%)
21.8
17.0
10.7
0.5
0.1
Model Size (GB)
13.02
9.75
6.13
3.32
2.14
1.12
0.68
8-bit Bascl
GSMSK Ace (%)
72.7
67.4
61.9
53.4
51.3
40.7
35.9
MATH Acc (%)
21.8
19.3
17.5
14.2
12.5
9.6
7.7
Table 15: Pass@1 accuracy and model size of Llama 2-7B rithins for code generation.
compressed and 8-bit-quantized by various algo-
Model Size (GB)
6.74
5.39
4.72
4.04
3.37
2.70
1.35
8-bit FLAP
HumanEval Acc (%)
23.8
16.5
9.1
3.7
1.2
MBPP Acc (%)
26.2
20.8
12.4
3.4
1.2
0.2
Model Size (GB)
6.74
5.39
4.72
4.04
3.37
2.70
1.35
8-bit Wanda
HumanEval Ace (%)
23.8
16.5
13.4
3.7
0
MBPP Acc (%)
26.2
22.2
18.2
4.6
0
Model Size (GB)
6.74
5.14
3.28
1.21
0.67
0.43
8-bit Basel
HumanEval Acc (%)
23.8
20.1
19.5
11.6
9.1
7.3
MBPP Ace (%)
26.2
22.4
17.2
8.2
4.6
3.2
19

---

Published in Transactions on Machine Learning Research (12/2025)
Table 16: Pass@1 accuracy and model size of Llama 2-13B compressed and 8-bit-quantized by various algorithms for code generation.
Model Size (GB)
13.02
10.41
9.11
7.81
6.51
5.21
3.90
8-bit FLAP
HumanEval Acc (%)
25.0
17.7
7.9
4.9
3.2
0.6
0
MBPP Acc (%)
29.0
18.8
12.0
6.8
3.2
0.4
0
Model Size (GB)
13.02
10.41
9.11
7.81
6.51
5.21
3.90
8-bit. Wanda
HumanEval Acc (%)
25.0
16.5
12.2
0
0
MRPP Ace (%)
29.0
20.6
14.2
0
Model Size (GB)
13.02
9.75
6.13
3.32
2.14
1.12
0.68
8-bit Basel
HumanEval Ace (%)
25.0
25.0
15.9
12.2
11.0
9.1
6.1
MBPP Acc (%)
29.0
26.4
18.4
10.4
9.2
5.4
3.4
Table 17: Pass@l accuracy and model size of Llama 2-7B and -13B 4-bit-quantied by QLoRA for mathe-
matical reasoning and code generation.
Model
Task
Model size (GB)
Accuracy (%)
GSMSK MATH
3.45
54.1
3.45
12.6
4-bit QLORA 7B
HumanEval
3.45
12.8
MBPP
3.45
16.2
GSM8K
6.63
58.8
MATH
6.63
13.2
4-bit QLoRA 13B
HumanEval
6.63
13.4
MBPP
6.63
18.6
Table 18: Pass@1 accuracy and model size of 4-bit-quantized Llama 2-7B and Llama 2-13B for mathematical reasoning and code generation.
Model
Task
Model size (GB)
Accuracy (%)
GSMSK MATH
3.37
39.2
3.37
8.8
4- 4-bit-quantized 7B
HumanEval
3.37
6.7
MBPP
3.37
10.8
GSM8K
6.51
52.8
MATH
6.51
12.5
4bit-qqantized 13B
Human Eval
6.51
8.5
MBPP
6.51
14.2
20

---

Published in Transactions on Machine Learning Research (12/2025)
Table 19: Ablation study: Impact of varying the additional dimension in Basel the compression of
Llama 2-7B for the mathematical reasoning task.
Model Size (billion)
6.74
5.14
3.28
1.83
1.21
0.67
0.43
Addt'l Dim 32
GSM8K Acc (%)
66.4
63.8
61.6
56.2
50.2
40.2
34.2
MATH Acc (%)
20.6
19.7
17.6
15.5
13.4
10.0
7.8
Model Size (billion)
6.74
5.06
3.20
1.75
1.13
0.59
Addt'l Dim 0
GSM8K Acc (%)
66.4
64.4
61.6
55.4
48.3
34.9
MATH Ace (%)
20.6
19.3
17.4
15.1
12.2
7.1
Table 20: Ablation study: Impact of varying the pruning times in Basel on the compression of Llama 2-7B for the mathematical reasoning task.
Model Size (billion)
6.74
5.14
3.28
1.83
1.21
0.67
0.43
100 times
GSM8K Acc (%)
66.4
63.8
61.6
56.2
50.2
40.2
34.2
MATH Acc (%)
20.6
19.7
17.6
15.5
13.4
10.0
7.8
Model Size (billion)
6.74
5.11
3.27
1.82
1.20
0.65
2 times
GSM8K Acc (%)
66.4
62.9
59.1
54.4
47.2
32.1
MATH Acc (%)
20.6
18.6
18.1
14.5
11.0
6.2
Table 21: Ablation study: Pass@1 accuracy and model size of Llama 2-7B Basel (proposed) and Basel (free basis) the mathernttcal reasoning task.
Model Size (billion)
6.74
5.14
3.28
1.83
1.21
0.67
0.43
Basel (Proposed)
GSM8K Acc (%)
66.4
63.8
61.6
56.2
50.2
40.2
34.2
MATH Acc (%)
20.6
19.7
17.6
15.5
13.4
10.0
7.8
Model Size (billion)
6.74
5.14
4.14
2.51
1.21
0.67
0.43
Basel (Free Basis)
GSM8K Acc (%)
66.4
61.6
59.8
56.7
44.7
29.0
16.8
MATH Acc (%)
20.6
17.9
17.1
14.9
10.3
6.3
2.6
Table 22: Ablation study: Impact of incorporating L1 regularization in Basel on the compression of Llama 2- 7B for the mathematical reasoning task.
Model Size (billion)
6.74
5.14
3.28
1.83
1.21
0.67
0.43
Basel (Proposed)
GSM8K Ace (%)
66.4
63.8
61.6
56.2
50.2
40.2
34.2
MATH Acc (%)
20.6
19.7
17.6
15.5
13.4
10.0
7.8
Model Size (billion)
6.74
5.14
3.28
1.83
1.21
0.67
0.43
Basel (L1 norm, 0.02)
GSM8K Acc (%)
66.4
61.8
61.2
55.2
49.4
39.3
33.7
MATH Acc (%)
20.6
18.8
17.8
15.3
12.5
8.9
8.1
Model Size (billion)
6.74
5.13
3.28
1.83
1.21
0.67
0.43
Basel (L1 norm, 0.1)
GSM8K Acc. (%)
66.4
63.2
61.9
54.1
50.0
40.9
21.2
MATH Acc (%)
20.6
19.5
17.5
15.0
12.4
8.8
3.8
21


Reviews:
[{"review_id": "Jw0D8ybB1N", "reviewer": "Reviewer_iAwZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Basel, a method to prune LLMs by reinterpreting weight matrices via SVD. Compared to the vanilla SVD pruning method, Basel relearns basis on target task. The approach is evaluated on LLaMA-2 (7B, 13B) across math (GSM8K, MATH) and code (HumanEval, MBPP), and the results show less accuracy drop off at high compression ratios compared to other baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The paper compares with 2 SVD pruning variants - SVD and FWSVD, at high compression. Claims of accuracy gains and model size reduction are well supported by tables/figures. The qualitative analysis of bases adds credibility. \n- Section 4.8 makes a compelling argument with evidence.", "audience_yn": "Yes", "audience_explanation": "The method is novel enough to differentiate from its variants whilst being interpretable.", "requested_changes": "- Since Basel requires an additional retraining phase (re-learning singular values) absent in SVD and FWSVD, can you quantify this computational overhead? I would assume it is much less than the full dense model but it is still nice to know.\n\n- Broaden the experiment if possible. The current evaluation covers math and code, which are both specialized, knowledge-intensive tasks, but it would be nice to include a demonstration on a more conventional NLP dataset such as wikitext2. Even a brief result in the appendix would help illustrate that the method isn’t limited to the two domains chosen. If new experiments are not feasible, consider adding a sentence in the experiments section or conclusion noting that “Basel can in principle be applied to other domains; we chose math and code as challenging test cases, but expect similar benefits in NLP tasks, etc.” I consider this a nice-to-have because the paper is already strong; it’s not a deal-breaker, but would add value if doable.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ba8PD1qons", "reviewer": "Reviewer_TuPk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Basel, a new method for compressing LLMs for domain specific applications. The main idea is a truncated-SVD-like compression. The authors argue that the singular vectors often carry semantic meaning, and therefore in specific domains, certain singular vectors can be safely omitted. A repeated two-step procedure is therefore presented, where singular values are first re-learned, conditioned on the original orthogonal bases, and then a truncated SVD is performed. Basel is compared to a naive truncated SVD, QLoRA, and a 4-bit quantization, and shows generally superior performance for a given compression ratio. \n\n## Strengths\n\n- I'm generally positive about the core idea, which I find simple and intuitive (a gradient-based reweighted SVD, followed by truncation, iteratively executed). A further intuitive connection is built via the \"semantic bases\" experiment, which illustrates that orthonormal bases may indeed group to fairly specific concepts.\n\n- Basel seems to work well in the experiments presented. For small compression ratios on the experiments presented, it is competitive with FWSVD, while maintaining performance for much more aggressive compression ratios.\n\n## Weaknesses\n\n- The claims regarding semantic bases are much overclaimed, in my opinion. There is anecdotal evidence presented, and I think claims like \"these bases may sometimes carry semantic meaning\" would be okay, but claims such as \"our analysis reveals that the bases of these weight matrices encode distinct semantic components, some of which are redundant for specific target applications\" are quite a bit stronger than the evidence presented.\n\n- The baselines presented are fairly limited and weak. There are now many different approaches to fine tuning + compression/quantization/pruning/etc., and several more recent methods claim much better performance at high compression ratios.\n\n- While illustrative, it is somewhat limiting to consider exclusively math and coding datasets, especially with small models that perform quite poorly on these benchmarks in the first place.\n\n- I couldn't find any code available, which I think would greatly improve both the reproducibility of this study, and the usability of the method. I highly encourage the authors to include reproducible code.", "claims_and_evidence_yn": "No", "claims_explanation": "As discussed above, there are a few concrete points where I think the paper lacks evidence:\n\n- I do not think that the strong and general claims regarding the interpretability of the weight matrices are substantiated. The example presented is motivating, but anecdotal. I suspect that showing the semantic meaning more generally is quite difficult, and would suggest a weakening of claims, treating this as a motivating factor, rather than a central claim.\n\n- Claims with respect to state-of-the-art seem lacking, as some reasonable baselines are not included. For example, SVD-LLM [1] and ASVD+ [2] are both recent, and demonstrate improved performance at high compression ratios.", "audience_yn": "Yes", "audience_explanation": "Yes, pruning of LLMs is clearly of interest to the TMLR audience.", "requested_changes": "- [Critical] As discussed above, the inclusion of several more recent and more competitive baselines, such as [1], [2].\n- [Critical] As discussed above, either further evidence, or a weakening of claims, surrounding the interpretation of \"semantic bases.\"\n- [Critical] Discussion of hyperparameters used for the experiments. For example, what value of ${\\tilde r}$ is used? Same for IterationsPerPruning, KeepingEpoch, PruningEpoch, etc.\n- As discussed above, experiments in a wider range of fine-tuning settings, other than just math and code generation, potentially with larger models.\n- Inclusion of reproducible code.", "broader_impact_concerns": "N/A", "additional_comments": "Some minor comments:\n\n- *Page 1, first paragraph (Large language models [...])*: This paragraph should have some citations regarding its claims.\n\n- *Page 1, \"as demonstrated by our interpretation results in Section 3 shows\"*: this sentence is quite hard to read, and I would recommend revising.\n\n- *Page 1, last paragraph, \"our interpretation results\"*: this phrase is once again difficult to read.\n\n- *Page 3, first paragraph, \"the bases hold significant physical meanings\"*: as detailed above, I do not believe this claim to be substantiated. However, even if substantiated, I think \"semantic\" was meant here, rather than \"physical.\"\n\n- *Page 3, above Equation (2)*: instead of writing $u_i \\perp u_j$ etc., I would recommend something to the effect \"$\\langle u_i, u_j \\rangle = \\langle v_i, v_j \\rangle = 0$ for $i \\neq j$\", as the inner product notation is used directly after, and $\\perp$ is never properly introduced.\n\n- *Page 3, equations*: there are many potential matrix norms and inner products. The Frobenius norm and inner product are used -- this should be noted somehow.\n\n- *Algorithm 1*: Wherever equation (3) is referenced, \"equation equation 3\" is written. This occurs on lines 3, 5, and 16. \n\n- *Algorithm 1*: \"Tune the learnable parameters\" is a bit vague -- I understand that this means ${\\tilde s}_i$, ${\\tilde u}_j$ and ${\\tilde v}_j$ for $i = 1, \\dots, r$ and $j = 1, \\dots, {\\tilde r}$, but I recommend writing this in the algorithm block for clarity.\n\n## References\n\n[1] Wang, X., Zheng, Y., Wan, Z., & Zhang, M. (2024). SVD-LLM: Truncation-aware singular value decomposition for large language model compression. ICLR 2025.\n\n[2] Yuan, Z., Shang, Y., Song, Y., Wu, Q., Yan, Y., & Sun, G. (2023). ASVD: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821.", "extra_scores": {}, "extra_text": {}}, {"review_id": "se9meZFbD2", "reviewer": "Reviewer_mkFe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method for reducing the size of pretrained language models when adapting them for downstream applications, by using a SVD decomposition of their internal weights. The method involves fixing the orthogonal SVD basis vectors of the original model, but relearning the singular values, and combining this with an auxiliary low-rank additive component. The model is then iteratively pruned and fine-tuned on the downstream task, dropping basis vectors from the original model that have small singular values after each fine-tuning stage.\n\nThe authors demonstrate that their method is more effective than directly doing SVD-based pruning on a fine-tuned model, even when SVD is done using a task-aware weighting (FWSVD), and enables models to retain more of their performance even at substantial compression ratios. They also show that it can be combined with quantization, and that in this case it outperforms some non-SVD based methods involving LoRAs or pruning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "[Edit: Changed from \"no\" to \"yes\" after revision. Original review below]\n\nThere are two aspects of the paper as currently written that I think are not fully supported by evidence (although I believe both can be addressed with fairly minor revisions).\n\n### 1. Claims about \"bases encode highly specific semantic content\"\nThe paper claims that \"Our analysis reveals that the bases of these weight matrices encode distinct semantic components, some of which are redundant for specific target applications\" and states that \"bases encode highly specific semantic content\". However, the evidence for this claim is quite weak: the authors present a table of five basis vectors, along with the ten tokens whose unembedding dot-product with these are the highest, and then categorize them based on what those ten tokens look like.\n\nI think this claim is too strong for two reasons:\n1. Knowing that a basis vector has high cosine similarity with some tokens related to a semantic concept (e.g. \"iOS\" and \"Xcode\") does **not** imply that this basis vector actually \"encodes\" this concept. Figuring out the semantic meaning of directions in a model is a complex research problem in its own right, and this kind of claim is very susceptible to confirmation bias. (Not exactly the same setting, but see [this paper](https://arxiv.org/abs/2506.05774) for context on how difficult it can be to justify this kind of claim in practice.)\n2. It is not at all clear from the evidence provided that the basis vectors are actually \"highly specific\". For one, there are only five basis vectors presented, and it is not clear to me how they were chosen. (Were they cherry-picked for being the most specific-looking?) Also, I would guess that many random directions in activation space would have similar-looking top-ten tokens, because the tokens themselves are clustered in embedding space. Determining whether the basis vectors are _actually_ highly specific would require doing some sort of fair comparison between the basis directions and some other set of directions (e.g. random directions in the output space of the matrix), and possibly also evaluating the specificity based on something beyond the top-ten token list.\n\nI don't think this is a major issue with the paper, because this part of the paper seems mostly to be used as motivation for the technique, which works well. But I think this claim should either be weakened substantially or supported with more evidence.\n\n### 2. Comparisons to non-low-rank compression methods\nThe paper states that the method is more effective than \"state-of-the-art techniques\", and in section 4.1 it mentions comparing to SVD, FWSVD, QLoRA, and FLAP. However, the main experiments in the paper seem to only compare against SVD and FWSVD; the comparisons to QLoRA and FLAP are only done in combination with quantization.\n\nIt seems like quantization might affect the results of this comparison (especially given that FLAP was proposed in the absence of quantization), so it seems important to also compare to non-SVD-based methods in the unquantized setting. (I also think the paper would be improved by comparing to more than one pruning baseline, although this is not strictly required for the \"supported by accurate, convincing and clear evidence\" criterion.)", "audience_yn": "Yes", "audience_explanation": "The method appears to be quite effective and is straightforward to implement, is novel to the best of my knowledge, and focuses on a relevant and timely application area. I think some of TMLR's audience would be quite interested in learning about this method and the results presented in the paper.", "requested_changes": "### Qualify claims about the bases \"encoding\" certain concepts and being \"highly specific\" (or add more supporting evidence)\n\nThe authors should weaken/qualify the claims in the paper about the bases found by SVD. In particular, I think saying that the bases \"encode\" semantic concepts, or that they are \"highly specific\", are not justified from the evidence provided. I would suggest using weaker and more correlation-based terminology, e.g. that the bases _are associated with_ semantic concepts, or that they _seem to correspond to_ particular semantic concepts. Similarly I think the authors should avoid saying that \"the non-English character basis\" is \"irrelevant\" to math problems; it is possible that this basis direction is being used by the model for something else as well, and that the top-ten tokens are not representative of the function of this direction on a math dataset.\n\n(Alternatively, if the authors prefer to keep these claims, they should add substantially more empirical evidence to justify them.)\n\n### Compare to a pruning baseline (perhaps FLAP) in the main unquantized experiments (figures 3,4,5,6), or weaken claims about outperforming SOTA compression\nCurrently, the bulk of the experiments do not compare against non-SVD-based techniques, which are only used in the quantization setting. To support the claims that the method outperforms SOTA techniques, and ensure that quantization is not interfering with the results, I think the method should be compared against some non-SVD-based techniques in the unquantized setting.\n\n(Optional: The paper would also be stronger if it compared to more non-SVD baselines than just QLoRA and FLAP. For instance, [TrimLLM](https://aclanthology.org/2025.acl-long.33.pdf) is a recent technique that drops entire layers from the model, based on somewhat similar motivation to this paper.)", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "I have a few other questions for the authors about their method:\n\n1. The analysis in the first part of Section 3 studies the factorization of the $W^h_O W^h_V$ matrix of the attention layers (which I assume means the product of these). However, the description on page 5 says that Basel prunes \"the $W\\_Q$, $W\\_K$, $W\\_V$, and $W\\_O$ matrices in the attention layers\". Does this mean that $W\\_O$ and $W\\_V$ are pruned independently, with independent SVD decompositions, even though the method was motivated by the SVD of the product of these?\n    - If so, it might be worth adding a brief footnote about this. I would also be curious if you have tried decomposing them jointly, and how well that performs if so.\n2. I am curious if you have considered regularizing the re-learned $\\tilde{s}_i$ weights using something like L1 regularization to encourage them to become sparse, as an alternative to (or in addition to) the iterative pruning process? This might guide the optimization process toward selecting lower-rank subsets of bases on its own.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)