Choose Your Model Size: Any Compression of Large Language Models Without Re-Computation
Authors: Martin Genzel
Patrick Putzky
Pengfei Zhao
Sebastian Schulze
Mattes Mollenhauer
Robert Seidel
Stefan Dietzel
Thomas Wollmann
Venue: Accepted by TMLR ()

Abstract: The adoption of Foundation Models in resource-constrained environments remains challenging due to their large size and inference costs. A promising way to overcome these limitations is post-training compression, which aims to balance reduced model size against performance degradation. This work presents Any Compression via Iterative Pruning (ACIP), a novel algorithmic approach to determine a compression-performance trade-off from a single stochastic gradient descent run. To achieve parameter efficiency, we use an SVD-reparametrization of linear layers and iteratively prune their singular values with a sparsity-inducing penalty. Importantly, the pruning order of the parameters is used to derive a global score map that allows compressing a model to any target size without re-computation. We evaluate ACIP on a large selection of open-weight LLMs and downstream tasks, demonstrating state-of-the-art results compared to existing factorization-based compression methods. We also show that ACIP seamlessly complements common quantization-based compression techniques.

Published in Transactions on Machine Learning Research (11/2025)
Choose Your Model Size: Any Compression of Large Language Models Without Re-Computation
Martin Genzel'
Merantiz Momentum, Berlin, Germany
Patrick Putzky"
Merantia Momentum, Berlin, Germany
Pengfei Zhao".t
Understandable Machine Intelligence Lab
Leibniz Institute for Agriculture and Bioeconomy, Potsdam, Germany
Sebastian Schulze
Meranliz Momentum, Berlin, Germany
Mattes Mollenhauer
Merantic Momentum, Berlin, Germany
Robert Seidel!
Stefan Dietzel
Merantie Momentum, Berlin, Germany
Thomas Wollmann
Merantiz Momentum, Berlin, Germany
Equal Contribution
Work done while at Merantix Momentum
Reviewed on OpenReview: https: openreview. net/ pormffrmm? id= 16hdYf8tsg
Abstract
The adoption of Foundation Models in resource-connttaained environments remains challenging due to their large size and inference costs. A promising way to overcome these limitations is post-training compression, which aims to balance reduced model size against performance degradation. This work presents Any Compression via Iterative Pruning (ACIP), novel algo- rithmic approach to determine a compresion--eeformance trade-off from a single stochastic gradient descent run. To achieve parameter efficiency, we use an SVD-reparamettizzaton of lincar layers and itcratively prune their singular values with a sparsity-inducing penalty. Importantly, the pruning order of the parameters is used to derive a global score map that allows compressing model to any target size without re-computation. We evaluate ACIP a large selection of weigh LLMs and downstream tasks, demonstrating state-of-the-art results compared to existing factorization- based compression methods. We also show that ACIP seamlessly complements common quantization-based compression techniques.
1 Introduction
Post-training compression of Foundation Models, especially Large Language Models (LLMs), promises access to powerful tools where resources limited, e.g., in automotive systems, mobile deployments, shop

---

Published in Transactions on Machine Learning Research (11/2025)
floors (Gholami al., 2022). Typical reasons for resource scarcity include constrained access to hardware, monetary limitations, high inference speed requirements, and environmental concerns (Hohman et al. 2024
The original promise of model compression was to eliminate redundant parameters, resulting in almost lossless methods (Han et al., 2016). While working well for models trained on smaller datasets, this hypothesis does not hold up anymore in the era of LLMs and scaling laws (Allen-Zhu & Lij 2024). For modern "densely trained" models, compression is almost always lossy, leading to a fundamental trade-off between model size and downstream performance. While characterizing this trade-off supports practitioners in deployment decisions (Boggust et al, 2025), the scientific literature typically focuses benchmarks at preset compression levels (Zhu et al.) 2024). This gap between research and practice implics that, for model users, the process is often perceived as a "black box", requiring siguificant expertise and trial-and-error to identify an acceptable setup. We argue for the opposite approach, one that empowers users to seamlessly customize a compression algorithm for their specific use cases.
Any Compression
Base model
Choose size
Compute
Compress
With this motivation in mind, we advocate for methods that permit Any Compression of pre-trained models. Here, 'Any' signifies an algorithm's ability to scale given base model to an arbitrary target size, guided by the user's specific needs and limitations, rather than the algorithm dictaling possible sizes. To facilitate decision-making, such an algorithm must efficiently reveal the compression- performance trade-off without extensive re-computation. In existing post-training compression approaches, we iden- tify two practical challenges to achieving Any Compres- sion.
Chcap
Expensive
(a)
Conventional Model Compression
Base model
Compute
Choase size
Compress
Problem 1 (Presct compression rates). The size reduc- tion of large Foundation Models is a prominent. research area, with existing methods largely falling into three cat- egories: knowledge distillation (training smaller student model), quanlization (reducing numerical precision), and pormmeter pruming (removing redundant weights) for a comprehensive survey, we refer to Zhu et al. (0024). While effective, these approaches often impose constraints that conflict with the goal of achieving Any Compression as they are restricted to preset (discrete) compression fac- tors. Indeed, knowledge distillation is limited to a single compression rate defined by the fixed size of the student model (Hinton et al. 2015). Quantization is limited by fixed bit-length reductions (typically from 16-bit to 4-bit or 8-bit representations) (Gholami al. 2022 Dettmers
Expensive
Cheap
(b) Any Compression
Figure 1: Compared to conventional compression algorithms (a), an Any Compression algorithm (b) swaps the computational calibration step and the decision step, so that models of different target sizes can be materialized without re-computation.
et al. 2023). Similarly, (unstructured) parameter pruning often relies on rigid sparsity patterns (e.g., nam sparsity) to ensure efficient memory allocation and hardware acceleration, thereby restricting the possible model sizes (Choquette et al., ). In practice, preset compression rates are especially problematic when deploying under specific, non-discrete constraint, e.g., a fixed memory budget, forcing users into a costly "guess-and-check" cycle.
Solution (Structured pruning by weight factorization). Weight factorization is a technique where (linear) model weights are decomposed into several sub-matrices (Eckart & Young. 1936). This approach enables fine-grained model compression and high parameter efficiency by pruning the inner dimensions of the resulting matrioes Yuuan et al., 2024 Wang et al., 2024b. A key advantage is compatibility with standard hardware, since it only requires basic matrix-vector multiplications.
Problem 2 (Different compression rates require re-computation). Most existing post-training compression techniques inherently inefficient. for exploring the trade-off between model size and performance. Conven-
2

---

Published in Transactions on Machine Learning Research (11/2025)
Expensive
Chrap
Base model
V
Linear layers
20% %
Iterationn
Score map
30%
90 %
M(p)
V
Low rank adapter
Model Reparametrization
Scoring by Iterative Pruning
Any Compression
Learnabl model parameters
Frozen model parameters
A visual overview of ACIP. The linear layers of the base model are reparametrized in terms of their singular value decomposition UMEVT, with (binary) singular value mask M = M(p) and low-rank adapter A. An objective function is optimized via gradient descent. over the mask parameters and adapters where sparsity is induced on p by an increasing l l1-penalty. This leads to pruned entries in the mask M(p). The optimization path p gives rise to score map that determines the global importance of the singular values across the full model. Potential compression errors are compensated by A. Based on the parameter scores, the base model can be flexibly compressed to any target size by masking the entries of Σ. The learned adapters A are used correction for any compression level.
Figure 2:
tionally, user selects a target compression rate and other hyperparameters, initiating costly computational step that take minutes to hours (Frantar et al.) 2023 Sun et al. 2024 Yuan et al. 2024). Fach new compression rate requires repeating this entire process, making comprehensive evaluation of the trade-off landscape impractical (see Figure [[a]..
Solution (Any Compression through amortization). We argue that reversed workflow is preferable: a single, upfront compulational investment that enables the subsequent materialization of a model al any compression rate in almost real-time (see Figure [ig)re[(b)... The initial effort, which could be handled by the model supplier, would empower users to efficiently select an optimal model instance for their needs at a negligible cost. The challenge, therefore, is design algorithms that are compatible with this "compute once, compress dynamically" approach.
Contributions and Overview
In this work, we introduce Any Compression via Iterative Pruning (ACIP) which is specifically developed to address the above problems. To the best of our knowledge, ACIP is the first algorithm that enables large-scale model compression to any size in real-time without requiring re-computation or re-calibration.
To overcome Problem 1 ACIP follows low-rank factorization strategy, based on pruning singular values of large linear layers. This partieular form of structured parameter pruning allows for fine-grained compression levels and is parameter-efficient at the same time, only singular values altered. Compared to quantization and conventional (unstructured) pruning, an efficient implementation docs not come with any restrictions to the underlying hardware, since it is purely based on standard matrix-vecior multiplications.
Pronounced like sip' of coffee.
3

---

Published in Transactions on Machine Learning Research (11/2025)
ACIP solves Problem 2 by explicitly decoupling aoptimizattonn-aaeedd pruning stage from the actual compression stage. The former can be viewed as dala-dependent calibration step, which estimates the global importance of all target parameters (the singular values of the base model layers) through an iterative pruming scheme. The resulting score map is then used to implement a simple compression step that enables the instantiation of models of any desired size without, further computational costs. The detailed methodology and technicalities of ACIP are presented in Section (see Figure 2 for visual overview).
We empirically demonstrate the effectiveness of ACIP a range of recent LLMs in Section 3 accompanied
by a series of additional experiments in the supplementary material (Appendices C and D In particular, we verify that our approach outperforms other Aetorizationn-asee methods on multiple benchmarks (Section 3.2 and seamlessly complements quantization-besed compression techniques (Section 3.4). In the spirit of scaling laws (Kaplan et al., 2020), ACIP provides consistent and robust size-performance trade-offs, which allows model users to predict downstream capabilities from few data points.
Finally, we put our results in the broader context of related literature in Section and conclude in Section 5 by outlining limitations well as promising avenues for future research that build on our contributions.
2 Method
Figure provides a schematic overview of Any Compression via Iterative Pruming (ACIP). In initial stage, ACIP builds reparametrization of large linear network layers by a singular value decomposition (SVD), which enables model compression through rank reduction (see Section 2.2. for details). Unlike existing SVD-based methods (Idelbayev & Perpiiaa- 2020 Yuan et al. 2024 Wang et al. 2024h, Any Compression is achieved by decoupling the pruning and compression stages (cf. Figure [(b].. More specifically, we construct a score map that establishes a global importance ranking of singular values across all linear layers within the network. The score map is derived by running a (stochastic) gradient descent on a sparsity-inducing objective, using the pruning order of the singular values as a proxy for feature importance (see Section (.2.2). In an independent step, the score map can then be used to compress the base model any desired size without re-computation (see Section 2.2.3
2.1 Preliminaries
This section provides several preliminaries to better understand the algorithmic details of ACIP, which are presented Section 2.2
2.1.1 Score-Based Parameter Pruning
As motivated above, the overarching goal of this work is to allow Any Compression of pre-trained model by decoupling the computational stage from the compression stage. To this end, we use score- score-based parameter pruning, a framework that has been successfully applied to model compression since the 1980s (LeCun et al.) 1989 Hassibi et al. 1993). In score-based pruning, a score map p is created that assigns an importance score 0; to each larget parameter 0,. Naturally, this approach gives rise to a (global) ranking of parameters that allows for model compression at any desired rate.
There many ways to design uscful score maps. For example, they can be derived based on curvature at local optimum (LeCun et al.
1989: Hassibi et al. 1993) by using hand-designed local features (Sun) et al. 2024 Frantar & Alistarh 2023). ACIP takes novel, data-driven approach to score maps that does not
require any handcrafting or foature engineering (see Scction 2.2.2
2.1.2 Low-Rank Compression of Linear Layers
Linear layers are the molecular building blocks of modern machine learning models, typically accounting for more than 90% of model parameters in transformers (Vaswaní et al., 2017), which makes them natural larget for compression. Accordingly, we consider linear layers of the form
y = Wx + b
(1)
4

---

Published in Transactions on Machine Learning Research (11/2025)
where W is an matrix, b is bias term, and x and y are layer imputs and outputs, respectively. In this work, we specifically aim for a low-rank compression based on matrix factorization such that
y = PQx + b,
where P and Q are matrices of sizes m x k and &o x n, respectively, and & < min(m. n2). The layer parametrization in (2) may be interpreted as (lossy) compression of the parametrization in ( leading to a smaller memory footprint whenever k(m + n) < mn. Note that matrix factorization can be used to compress the parameters of any linear layer, including dense layers as well as efficiently parametrized layers such as convolutional layers (Idelbayev
Perppinn 2020
Singular value decomposition. To determine suitable low-rank factors P and Q in (2) we follow recent
work on (large) model compression (Idelbayev & Parpeira 2020: Yuan et al. 2024 Wang et al. 2024) and leverage singular value decomposition (SVD). SVD factorizes X n weight matrix W, of rank r < min(m, n) at layer as
W, = -VUU.
(3)
where Uj and Vi are m X and r matrices of (orthonormal) singular vectors, and Eg is r X diagonal matrix containing the singular values s(") > 0. Note that we consider the compact SVD here, which ignores the null-space vectors of the full SVD. We may reduce the inner dimension r of the SVD to h <r with
W Ú,E.
(4)
by defining U, and V to only contain & singular vectors associated with selected subset of singular values. Mcasuring the approximation error in terms of (Frobenius) matrix norm, selecting the h largest singular values leads to an optimal approximation of Wr under rank constraints (Mirsky 1960p. It has been previously argued, however, that this approach does not yield satisfactory results in deep learning as is does not take into account the underlying (training) data and downstream task (Hsu al.) 2022). Different approaches
have been presented to address this problem (Hsu et al. 2022 Yuan et al. 2024 Wang et al., 2024 Chen
et al.
2021
2.1.3 L R--guularizzaton
Consider a generic loss functional L(X; 0) evaluated on some model that is specified by a parameter vector 0 and data X. During model training, sparsity in 0 is typically encouraged by solving the penalized optimization problem
min ((x:0) + 111 >>0, 0,
55)
which is known as & fi-regularization, or least absolute shrinkage and selection operator (LASSO) in case of (generalized) linear models (Tibshirani] 1996: Hastie et al. 2015). Feature selection plays a key role in this optimization problem. Indeed, increasing À in 5 lcads to sparser solutions (()). This effectively gives risc to feature importance ranking for the solution of (5) through the so-called regularization path [Tibshirani 1996 Efron et al. 2004 Mairal & Yu, (2022) an idea that inspired the approach of ACIP. Here, 'feature importance" is understood as the relative contribution to the (task- and data-specific) loss L(X; 0), i.e., feature is considered more important if its removal causes larger increase in the loss. In particular, different choices of L may lead to different solution haths and feature rankings. Furthermore, previous results indicate that optimization paths of iterative schemes can be related to regularization paths (Suggala et al., 2018). Intuitively, this suggests that general optimization paths of iterative schemes for E {--ojeectvvs reveal information about feature importance in the context of sparse models as well.
In ACIP, we will successively incrcase X to introduce a higher degree of model compression in terms of parameter sparsity in a controlled manner (see also Remark 2.1 below).
2.2 Any Compression via Iterative Pruning (ACIP)
Algorithmiiaally ACIP consists of the following three key steps, which are detailed in the sections below. For a schematic visualization of ACIP, we refer to Figure 2
5

---

Published in Transactions on Machine Learning Research (11/2025)
Step 1. (Model Reparametrization) Apply SVD to the weights of all (dense) linear layers according
to G Introduce low-rank adapters A and singular value masks as tunable parameters.
Step 2. (Scoring via Iterative Pruning) Choose surrogate loss C and calibration data set X. Perform iterative pruning of singular value masks p and simultaneous tuning of low-rank adapters A by applying f--regularized gradient- based optimization as in Obtain global parameter score map P by using Algorithm
Step 3. (Any Compression) Choose any desired compression rate. Use the score map p and low-rank adapters A to materialize the compressed model in real-time.
2.2.1 Step 1. Model Reparametrization
We start by reparametrizing all linear layers of network using SVD described in 3 and assign
Wi + U + AA
(6)
where I denotes the layer index, M, is a diagonal matrix with binary entries m (0)
{0,1} masking the singular values 8) in En, and A, is a low-rank adapter (LoRA) (Hu et al. 2022). In all subsequent steps of ACIP, we freeze Eg, U1, and Vi.
We find that adding a low-rank adapter helps to compensate for potential errors that are introduced by pruning in Step 2 (Section 2.2.2 We initialize M, as the identity malrix and A as zero weights. In this way, the reparametrized model remains identical to the original model up to numerical precision.
We assign the binary masks m (7) such that s()) = ml ( represents the pruned retained singular values, respectively. Thus, m, (5) decouples the magnitude of a singular value and the pruning decisions based on its
importance.
The above paramotrization leads to a parfmetor-efficien compression scheme. Indeed, given an m x n matrix W, the number of non-zero singular values bounded by r = min(m, n), which means the number of tunable mask parameters scales linearly in the feature dimensions.
Mask parametrization.
We parametrize the binary masks through a thresholding operation of the form
0, for P < 0 1, for P > 0"
m
(7)
where Pi (i) are scalar learnable parameters. As this operation is not differentiable, we use the ttraight-through estimator for backpropagation (Bengio et al. 2013 Yin et al. 2018).
2.2.2 Step 2. Scoring via Iterative Pruning
We now aim to build global score map over all singular values the reparametrized layers, which guide model compression subsequently in Step (Section 2.2.3). Leveraging the sparsity-inducing property of
(0) L d-regularization (see Section 2.1.3 we progressively shrink the mask parameters p to zero and derive
a score map based on the pruning ordor. The two key algorithmic components this "iterative scoring" stralegy are presented next.
Iterative pruning. The optimization problem solved by ACIP takes the form
min
O, L(::0, A) + ||pl, A.A
(8)
where L denotes a suitable calibration loss for the model, p (Po. P2} the set all mask parameters, A ( A is the set of all low-rank adapters, and 0 is the of all remaining model parameters that
FFoloowing common practice, ignore the embedding layer and classification head (decoder-only) transformers.
6

---

Published in Transactions on Machine Learning Research (11/2025)
params: parameters vectorized; acores: pruning scores, vectorized; initialized to Nal
0.5
def apdate_scores (scores, params): previously pruned parameters <=
score_mast
[not score_mase maak] parans [not score_mask]
scores [score_mask] 1.
0
10
return scores
0
200
400
600
800
1000
1200
Train Steps
Algorithm 1: The procedure for updating the score map after each optimization step. For unpruned mask parameters p, the score is their current mag- nitude, which estimates the future pruning order. Once a parameter pruned, its score becomes a neg- ative integer value that tracks the pruning history by decrementing at each step. This dual mechanism establishes global importance ranking used to com- press the model to any target size in Step 3.
Figure 3: Progressive shrinkage of exemplary mask parameters p in Attn-V layer 1 30 of LLaMA- 7B based on [8] Each plotted line corresponds to the evolution of a parameter value over training time. The starting points of shrinkage are predic- tive of the pruning order, typical phenomenon l--regularization. In ACIP, this pruning order de- termines the score of associated singular values (cf. Algorithm I
are frozen during optimization. We perform gradient-based based optimization until a preset maximum compression ratio Ystop is reached (see Appendix D.4 for further discussion). Optionally, we perform post-tuning for a fixed number of steps by freezing the masks p and continuing the optimization the low-rank adapters A. Remark 2.1 (Scaling of A). If is chosen too small, the maximum compression ratio Tstop might never be reached. If À is too large, training might become unstable and the score map ambignous. Therefore, use simple linear scheduler that scales À by fixed factor >1 every j optimization steps, that pruning becomes increasingly aggressive over time.
Remark 2.2 (Role the Calibration Loss). As pointed out in Section 2.1.3 a. feature importance ranking as produced by optimizing (S) depends on the specific loss L(X; 0, p. A). Given a model to be compressed, this leaves two important design decisions the user of ACIP, namely the choice of loss function and calibration data. In the case of L.MSs, it is natural to the same loss function in pre-training, Le., the negative log- likellhoooo loss for next-token prediction. The calibration data should be representative of the task for which the compressed model(s) should perform well. This choice is therefore very use-case dependent and could range from very specific fine-tuning tasks to general-purpose prediction. In this work, we focus on the latter.
From iterative pruning to score map. The optimization process of ( used construct our score map. Based on the discussion of Section 2.1.3 we hypothesize that there close relationship between (i) vanish and their importance for the model the least important parameters are pruned first and so on. When conducting our experiments, we observed a shrinkage behavior
the order in which the parameters pi
that supports this hypothesis; see Figure 3 for specific example.
Algorithm describes how the score map is updated after each optimization step represent feature importances. In plain words, the score map is built based the pruning order. A negative number in the map indicates how many steps ago parameter was pruned. For all parameters that have not been pruned, the score is set to the value of the corresponding parameter. We refer to Appendix D.9 for visual examples of ACIP score maps.
The approach of Algorithm ] future pruning based on the parameter magnitudes. Note that absolute values of the score are irrelevant for parameter ranking.
ensures that (i) the score map stores the pruning history, and (ii) estimates

---

Published in Transactions on Machine Learning Research (11/2025)
2.2.3 Step 3. Any Compression
From Step 2, we only relain the score map and the low-rank adapters A. In particular, the pruned masks m are discarded, as they are irrelevant for compression at this stage (cf. Figure 2 As motivated above, the score map allows us to globally rank all singular values based on their score. This Ieads to fully independent compression stage where we can flexibly create a model of any reduced size: we prune as many singular values 5,") (and the corresponding singular vectors) according to their scores p so that a given compression rate r is achieved. Note that there is a monotonic but non-linear relationship between the total number of pruned singular values l and compression rate r (see Section 2.1.2 Given a target rate r; we find & via a binary search (in almost real-time). As this compression procedure operates directly on the reparametrized model from Step 1, is reversible and therefore indeed allows for Any Compression; for a slightly refined version, see Appendix D.3
Finally, to matcrialize model at fixed target rate, all pruned singular values and vectors are discarded, that the initial SVD-reparametrizal rank k 2 mn..
turns into an actual low-rank factorization. For layers with determined m4n' i... where a factorization would not save any parameters, we avoid an inefficient storage usage by simply recovering the (dense) weight matrix from its SVD components.
Remark 2.3 (Merging Low-Rank Adapters). Above, we have treated the truncated weight matrices Wi and the corresponding low-rank adapters A as distinct components. It is possible to merge the two decompositions into single one to improve model throughput. To this end, let A A, be the factorization of the low-rank adapter. Then, we can express the reparametrization at layer I in 6 by matrix concatenations of the form
W = U,E, A
/
V B
After specifying a compression rate, the corresponding weight matrix can be expressed
W = ش A
(10)
B
M2
.M
where Ü, Σ, V7 are the truncated singular components, see Note that Wi is never explicitly materialized as dense matrix in memory (to ensure actual memory savings) and its forward pass is implemented as two consccutive vector-matrix multiplications with Mj and M2.
2.3 Computational Considerations
In this section, we discuss the computational costs and memory overhead of ACIP and compare them to other common compression paradigms. As detailed Section 2.2 the overall process of ACIP can be divided into three distinct stages, each with different computational characteristics. A summary of the empirical runtime and memory costs for LLaMA-7B base model is provided in Table A5
The most resource-intensive stage of ACIP is scoring via iterative pruning (Step 2). It involves backpropagation through the entire model to update the singular value masks and the low-rank adapters. As such, these updates are parameter- -effiiceete,,,i,ntt,..... but the overall memory consumption is still proportional to the full model size, which can be demanding. In contrast, backpropagation-frre, layer-wise methods such as ASVD (Yuan) ct al., 2024) and SVD-LLM (Wang et al.] (2024) are notably more memory-officient, as they only need process and store the data for a single layer at any given time. However, is crucial to emphasize that the iterative pruning stage is compatible with Any Compression and represents one-time, upfront computational investment. This calibration can be performed once by model provider, amortizing the cost across all subsequent uses. Another noteworthy aspect of Step 2 is that the SVD-reparametrizatio initially leads to an increased model size compared to the original model (cf. Table A5 since the singular vector matrices U and both need to be stored in memory.
Once the score map is generated, the compression stage (Step 3) is exceptionally efficient. As shown in Table A5 compressing the model to any target size by discarding singular vectors based their global scores
8

---

Published in Transactions on Machine Learning Research (11/2025)
LLaMA-7B
0.63
LLaMA-7B
LLaMA-138
LLaMA-13B
Perplexity Dog scale]
L1aMA-2-78
0.6
11AMA-2-78
LLMMA-2-13B
0.53
LLaMA-2-139
LLMMA-3.1-88
Accuracy
LLaMA-3.1-8B
0.5
Misaml-7R--v1.3
Mistral-7--003
Qwen2,5-38
Qwen2
Qwen2.5-7B
Qwea2.5-1
10
Qwen2,5-148
0.4
QwenZ 5-14B
0.35
1B
10B
10
108
Number of Parameters [log scale]
Number of Parameters [log scale]
tradc-offs gen- erated by ACIP C4. Each curve was obtained by the Any Compression stage (Step 3 in Section 2.2.3). i.e., no additional computation was required except for a perplexity evaluation. Square marks denote the base model performance.
Figure 4: Compression-perfomnee
Figure 5: Compessioo--poffrmmnne trade-off curves generated by ACIP, using average accuracy all LM-Eval tasks as metric.
occurs in near real-time. This allows practitioners to dynamically select the optimal cmmpresion-peeformann trade-off for their specific application without incurring any significant computational delay.
3 Experiments
Experimental setup. To demonstrate effectiveness across architectural differences in LLMs, we evaluate ACIP on a selection of popular open-weight. models: L.AMA-7B/133B (Touvron et al. 2023a), LLaMA-2-
7B/13B (Touvron et al. (023b), LLaMA-3.1-81 (Grattafiori et al. 2024), Qwen2.5-7B 14B (Qwen al. 2024h, and M (Jiang et al., 2023). We use a subset of C4 training data (Raffel al. 2019) for the pruning stage. Regarding evaluation tasks, we follow Wang al. (2024) and report perplexity
validation held-outs of C4 (Raffel et al. 2019) and WikiText-2 (Merity et al. 2017 7). and we consider seven
zero-shot tasks from EleutherAl LM Evaluation Harness (LM-Eval) (Gao et al. 2023). More implementation
details about ACIP and choices of hyperparameters can be found in Appendix
3.1 Analyzing Compression-Performee
Trade-Oft Offs
trade-offs powered by ACIP. Figures 4 and 5 demonstrate smooth and consistent curve shapes for all considered models; analogous results for WikiText-2 and individual zero-shot LM-Fval tasks can be found in Figure A8 We note that monotonic relationship between size and performance is not self-evident, e.g., see Figure A13 in Appendix D.6 for a trivial approach that uses magnitude- based pruning. More insights on the runtime and memory consumption of ACIP as well as inference speed of compressed models are reported in Table A3 in Appendix D.I
We first study compression-performee
A remarkable observation is that the oldest models, LLAMA-7B/13BB perform best perplexity-wise, while newer, more capable models like Qwen2.5-7B/ dominate on LM-Eval as expected, especially on the lower compression levels. This apparent contradiction is likely caused by deviation of the pre-training data distributions from C4 in the case of more recent models.
A second noteworthy outcome of Figures 1 and 5 are the gaps between LLMs of different base model sizes in the same family. Indeed, ACIP cannot match the performance of base models of smaller size, e.g., compare the compressed Qwen2.5-7B with the original Qwen2.5-3B. This not surprising because the corresponding smaller-size base models were obtained by pre-training or knowledge distillation (Hinton et al. 2015 Busbridge et al. 2025 which are orders of magnitudes more expensive than ACIP
9

---

Published in Transactions on Machine Learning Research (11/2025)
Table Any Compression under SVD reparameterization. 'Zero-shot evaluation of LLaMA-7B. Comparison with basclines SVD magnitude pruning (SVD-Magn.), ASVD Yuan et al. 2024). SVD-LLM (Wang al.,
2024]. and Dobi-SVD (without remapping) QQinsi et al. 2025 t: larger is better; L: smaller is better; best
rosults for each task and size ratio are marked in bold. The scores for ASVD and SVD-LLM are taken from
Wang et al. (0024) and the scores for Dobi-SVD from Qinsi et al. 2025
C4
WikiText-2
Openb.
ARC e
WinoG.
HellaS.
ARC c PIQA
MathQA
LM Eval
Size
Method
4
1
Avg. 1
100%
Original
7.34
5.68
0.28
0.67
0.67
0.56
0.38
0.78
0.27
0.52
SVD-Magu.
9819.35
11312.47
0.15
0.27
0.49
0.26
0.20
0.20
0.30
ASVD
15.93
11.14
0.25
0.53
0.64
0.41
0.27
0.68
0.24
0.43
80%
SVD-LLMM
13.84
7.94
0.22
0.58
0.63
0.43
0.29
0.69
0.24
0.44
Dobi-SVD
10.01
8.54
0.26
0.59
0.66
0.44
0.31
0.70
0.23
0.46
ACIP (ours)
10.92
8,83
0.28
0.66
0.63
0.49
0.32
0.74
0.23
0.48
SVD-Magu.
15186.30
12575.68
0.15
0.26
0.51
0.26
0.22
0.52
0.20
0.30
ASVI)
41.00
51.00
0.18
0.43
0.53
0.37
0.25
0.65
0.21
0.38
70%
SVD-LLM
25.11
9.56
0.20
0.48
0.59
0.37
0.26
0.22
0.40
ACIP (ours)
12.22
10.35
0.28
0.64
0.62
0.47
0.31
0.73
0.23
0.47
SVI>-Magn.
14414.87
20987.82
0.115
0.25
0.50
0.25
0.22
0.52
0.19
0.30
ASVD
1109.00
1407.00
0.13
0.28
0.48
0.26
0.22
0.55
0.19
0.30
60%
SVD-LLM
49.83
13.11
0.19
0.42
0.58
0.33
0.25
0.60
0.21
0.37
Dobi-SVD
23.54
13.54
0..22
0.41
0.58
0.34
0.27
0.61
0.23
0.38
ACIP (ours)
13.91
12.46
0.25
0.61
0.59
0.44
0.30
0.71
0.24
0.45
SVD-Magu.
62899.32
109019.52
0.15
0.26
0.50
0.26
0.23
0.18
0.30
ASVD
27925.00
15358.00
0.12
0.26
0.51
0.26
0.22
0.52
0.19
0.30
50%
SVD-LLM
118.57
23.97
0.16
0.33
0.54
0.29
0.23
0.56
0.21
0.33
ACIP (ours)
16.47
16.16
0.21
0.57
0.57
0.40
0.27
0.68
0.22
0.42
SVD-Magn.
29804.73
29364.55
0.14
0.27
0.51
0.26
0.22
0.53
0.20
0.30
ASVD
43036.00
57057.00 .00
0.12
0.26
0.49
0.26
0.21
0.51
0.18
0.29
40%
SVD-LLM
246.89
42.30
0.14
0.28
0.50
0.27
0.22
0.55
0.21
0.31
Dobi-SVD
190.62
46.18
0.15
0.31
0.52
0.28
0.20
0.54
0.32
ACIP (ours)
21.05
23.99
0.19
0.49
0.55
0.35
0.24
0.64
0.21
0.38
3.2
Comparison to Existing Works
We now compare ACIP to recent works focusing SVD-based structured pruning, namely ASVD Yuan et al. ). SVD-LLM (Wang al. 2024), and Dobi-SVD (without remapping) (Qinsi et al. 2025 The former two approaches are backpropagation-frees and perform (activation-awaree) layer-wise updates instead, while Dobi-SVD proposes a differentiable truncation mechnism for singular values. Moreover, we evaluated a simple SVD magnitude pruning approach (SVD-Magn.), where we set the score map equal to the singular values of the weight matrices. This technique allows for Any Compression analogously ACIP and therefore serves as another natural bascline; see Appendix D.6 for further study.
Table I shows that ACIP consistently outperforms all baseline methods with growing gap for higher compression levels. Note that SVD-LLM and Dobi-SVD were calibrated on WikiText-2 instead of C4, which might explain slightly better results on the former datasct for 70% and 80% size. We think that these results underpin the benefits of an end-to-end scheme: (i) a simultaneous correction, e.g., by LoRA, can drastin irallic improve performance, and (ii) robust pruning patterns can found without leveraging any specific features of the SVD factorization. Morcover, we note that re-computations are required generate each row of Table for ASVD, SVD-LLM, and Dobi-SVD, whereas ACIP only needs single run. Analogous results for ACIP applied to all other models can be found in Table A2
10

---

Published in Transactions on Machine Learning Research (11/2025)
ACIP
W4A16 - ACIP
24 22
ACIP FT
W4A16 - ACIP W16A16
Perplexity [log seale]
ACIP W4A16
Perplexity J. [log scale]
ACIP
ACIP W4A16 FT
18-
16
14-
10
10
10
14
0.5
0.6
0.7
0.3
0.9
Effective Model Size [GB, log scale]
Relative Model Size [%)
trade-off curves for LLaMA-7B C4 showing the impact of fine- tuning and quantization after compression with ACIP. The horizontal axis measures size in terms of required (weight) memory to visualize the gains of quantization more clearly.
Figure 6: Compression-
Figure 7: Comprssioo--prformmncc trade-off curves for LLaMA-7B C4, showing that quantization before ACIP leads to similar results as without.
3.3 Improving Performance Through Fine- Taning
While the main goal of this work to produce a full family of accurate, compressod models from a few optimization steps, their performance can be certainly improved through continued fine-tuning. Figure G highlights the gains of fine-tuning LLaMA-7B, other models. We observe that fine-tuning leads to performance offset that is almost constant across all compression levels, which underlines the predictive capacity of ACIP. Note that we even observe a jump at zero compression because inserting the low-rank adaplers learned by ACIP leads to a slight initial performance drop (see Appendix D.3 for a potential improvement).
Table A2 for more detailed numerical results all
An optional fine-tuning step is not exclusive to ACIP but can be applied many other compression approaches as well. Table A3 provides a comparison with ASVD (Yuan et al. 2024) and SVD-LLM (Wang et al.] 2024 (ef. Section 3.2 when fine-tuned with LoRA. While ACIP still performs best in this respect, argue that post-compression fine-tuning should be still seen an independent (and much more costly) algorithmic step for two reasons. (i) Its outcome strongly depends on the specific training protocol and data, making fair and direct comparison challenging; (ii) requires us to fix a compression level, which breaks the crucial Any Compression feature of ACIP. Therefore, promoting a costly fine-tuning step after compression is not the primary concern of our work.
3.4
Combining ACIP with Quantization
In the field of low-cost compression for LLMs, quantization is still considered as the gold standard (ohhman (et al.) 2024 Zhu et al. 2024), so that a practitioner might not be willing to exchange its gains for the benefits of ACIP. Fortunately, ACIP only tunes a tiny fraction of weights with high precision, so that all remaining modules are suitable for quantization. In our experiments, we quantize all paramcterized and unparametrizod linear layers to 4-bit in fp4-formal (Dettmers al. 2023) using the bi tsandbytsss- Packkge (W4A16), except for the embedding layer and final classification head. We study the gains of quantization for ACIP in the following two ways.
Compress first, then quantize. We first apply ACIP as usual, compress the model to a given target size, and then quantize all linear layers. Figure 6 confirms that this approach works fairly well, only producing a slight performance drop compared non-quantized versions; see Table A4 for a full evaluation on all other metries. We also observe that an optional fine-tuning step as in Section 3.3 can almost fully compensate for the errors introduced by quantization after compression. This finding is well in line with the effectiveness of
11

---

Published in Transactions on Machine Learning Research (11/2025)
the popular QLoRA approach (Dettmers et al. 2023). Moreover, Figure 0 reveals drastic improvement through quantization in terms of required memory. Here, the ACIP-trade-off allows practitioners to study and apply more fine-grained compression on top of quantization.
Quantize first, then compress and transfer. Compared to layer-win wise methods like ASVD and SVD- LLM, ACIP has a higher demand in GPU memory due to backpropagation. A quantization of all frozen weight matrices can be an effective remedy in this respect. For the experiment shown in Figure 7 we have applied quantization before ACIP, which leads to very similar pepppresiioo-prrrrmmanee in the non-quantized case. Going one step further, this quantized version of ACIP back to full precision: We load the basc model in bf16, apply layer-wise SVD-parametrizaline
trade-ofls transfer the score maps and low-rank adapters from
insert the low-rank adapters learned by quantized ACIP, and use the corresponding score map to obtain a compressed model (W16A16). The resulting trade-off curve in Figure confirms that this simple strategy works fairly well, especially for lower compression levels.
3.5 Further Experiments and Ablations
Several additional experiments are presented in Appendix D analyzing the impact of several key components and design aspects of ACIP. Starting with an analysis of algorithmic efficiency and latency (Appendix D.1)
we study the impact of the low-rank adapters (Appendix D.2 D.2. compression rule (Appendix D.3) stopping
criterion (Appendices D.4 and D.5 score map design (Appendix D.6p, post-tuning (Appendix D.7 and
specific types of linear layers (Appendix D.S). Finally, show examples of score maps (Appendix D.9 and
prompt completions by compressed models (Appendix D.10
4 Related Work
In this section, we pick up on our broader discussion on the field of model compression from Section 1 and put our work in context with several directly related branches of research.
Structured pruning & low-rank factorization. Conceptually, ACIP falls under the umbrella of structured parameter pruning, specifically, low-rank matrix dccomposition. The rationale behind this compression approach is to approximate large weight matrices by products of low-rank factors reduce the total parameter count, and at the same time, to preserve critical information. After initial efforts into this direction for smaller language models (Edalati et al. 2022 Tahaei et al. 2022 techniques for LLMs primarily built on (weighted) SVD of linear layers (Ben Noach & Goldberg 2020 Hsu et al. 2022).
However, a key challenge of SVD-based pruning is that simply truncating singular values based on magnitude alone is insufficient and makes additional fine-tuning on downstream tasks necessary. Follow-up work recognized that the poor approximations are cansed by LLM weights being high-rank and instead turned decomposing network features which are sparse (Kaushal et al. 2023 Yu & Wu 2023). Similarly, recent. studies (Sharma et al. 2023 Yuan et al. 2024 Jaiswal al. 2024 have shown rank reduction differently affect layers in a network and proposed heuristics for non-unilorm pruning. Going even further, ASVD Yuan et al.)
approximations, proposing a training-free compression method that takes the (calibration) data distribution into account. Building on this, SVD-LLM
2024] pointed out the importance of activation
2024 recently derived an analytical layer-wise correction leading to superior compression results. While ACIP promotes an activation-aware solution as well, relies on gradient-based optimization, which
/Wang et al.
avoids any SVD-specific feature engineering and allows for simultaneous errors corrections.
Another relevant line of work on structured pruning aims to jointly remove groups of parameters entire network components, e.g., weight. matrix columns/rows, network layers, attention heads (Frantar & Alistarh 2023 Ma et al. 2023 Xia et al.) 2024 Ashkboos et al. 2024 Kim et al. 2024). At high compression rates, however, such "coarse" approaches often remove critical substructures, causing a significant performance drop that is only recoverable through additional fine-tuning.
Score maps & Any Compression. A common feature of the aforementioned structured pruning ap- proaches is that. they first. truncate parameters to preset target size and then compute an error correction.
12

---

Published in Transactions on Machine Learning Research (11/2025)
This design choice means that exploring the full compresion--erffrmance tradc-off requires repeated, costly computations for each desired compression ratio (ef. Figure [aa In contrast, ACIP overcomes this limitation by using score maps to determine global parameter importance. As such, score maps have been used as a tool for model compression since the 1980s (LeCun et al., 1989; Hassibi et al. 1993.. p. However, in the cra of LLMs, deriving a score for each parameter poses significant challenges in terms of scalability. Addressing this concern, ACIP enables scalable Any Compression by (i) using weight factorization to significantly reduce the score map size (e-g., ~900k parameters for 7B model), and (ii) decoupling the scoring and compression stages (Step 2 and 3 in Section 2.2 respectively).
Any-size pre-training. Beyond post-training compression, an alternative paradigm for obtaining models of varying sizes is to incorporate this flexibility into the (pre-)training process itself. For example, (Cai el al. (2020) train a single, large "Once-for-all" network from which specialized net-nks of different sizes can be extracted without retraining. More recently, the MatFormer (Devvrit et al.) 2024) has demonstrated how to build a family of "any-size" models directly through a nested pre-training methodology, based on Matryoshka Representation Learning (Kusupati et al.) 2022). The recently published Gemma-3n model uses this approach in production to make LLMs ready for mobile and edge devices (Gonzalez & Shivanna, 2025). While these methods also produce a trade-off between model size and performance, they require a significantly higher upfront computational budget associated with complex pre-training from scratch. ACIP provides a lightweight, post-training alternative that offers similar flexibility for any existing pre-trained model.
Rate-distortion theory. Finally, our work can be viewed through the lens of rate-distortion theory, which investigates the analytical trade-off between achievable data compression rates and the error (distortion) introduced by lossy compression (Cover & Thomas 2006). While some recent work (Gao et al. 2019 Isik
et al. 2022) investigates rale-distortion theory of machine learning models for simple architectures under rather specific assumptions, the cnermatiionthhhoetee limits of neural network compression are generally unknown in practically relevant settings. In this context, the family of compressed models generated by ACIP conveniently provides an empirical (upper) bound on the distortion-rate function of a large-scale model from a single optimization run.
5 Conclusion
In this work, we have introduced Any Compression via Iterative Pruning (ACIP), simple end-to-end algorithm to determine the compression-perfommant trade-off of pre-trained models. The underlying score map ranking allows us to materialize models of any compression rate in real-time. We have demonstrated empirically that the downstream performance of the resulting models is superior to existing, layer- wise factorization approaches. The flexibility and efficiency of ACIP make it a practical tool for deploying large- scale models in resource-constrainee settings, especially in combination with other compression techniques such as quantization.
Discussion. Our main results in Figures 4 and resemble the well-known phenomenon of scaling laws
(Kaplan et al.] 2020 Holfmann et al.
2022
Recently, it has been shown that any-size models can be achieved
through pre- pre-training Devvrit et al.
2024
Gonzalez & Shivanna 2025 (see also Section ! exhibiting similar
trade-offs as ACIP. Establishing a rigorous connection between these two fields of research could be a fruitful avenue of future work.
In a similar vein, we observe that more recent models tend to be less compressible (c.g., compare the slopes of LLaMA-13B and Qwen2.5-14B in Figure 5 We hypothesize that this relates newer models carrying denser information per weight, since they were trained on much larger datasets (Allen-Zhr & Li 2024). Also, the distribution of the calibration dataset (C4 in our case) might play an important role in this context.
A notable lechnical limitation of our work is that we have only focused on models that are tunable on single (NVIDIA H100) GPU in bfi6-precision. Hence, the scaling behavior of ACIP for larger LLMs (30B+) remains to be explored. We also emphasize that ACIP could be transferred to other modalitics, architectures, and tasks without any notable modifications. Finally, a more detailed study of inference speed (beyond the results of Appendix D.I could provide useful insights into the interplay of low-rank models and their efficiency.
13

---

Published in Transactions on Machine Learning Research (11/2025)
Broader Impact Statement
The primary broader impact of our work lies in increasing the accessibility and practicality of large Foundation Models. By empowering practitioners to elfortlessly navigate the costly recomputation, ACIP helps democratize the deployment of advanced A1. This could unlock novel applications in resourco-scarce domains from on-device mobile assistants intelligent systems in manufac- luring and automolive sectors and support researchers with limited computational budgets. Ultimately, our approach contributes to more sustainable and inclusive AT ecosystem by enabling the efficient use of pre-trained models, reducing both computational and environmental overhead.
trade-off without
In parallel with these benefits, it is crucial to acknowledge the ethical implications and responsibilities that come with democratizing powerful technology. The same accessibility that fosters innovation could also lower the barrier for malicious applications, such the efficient generation of spam or real-time misinformation on edge devices. This underscores the critical need for the AI/ML community to proactively develop robust safety mechanisms and ethical guidelines that are effective for models of all sizes. By doing so, ensure the positive impacts of accessibility are not undermined by potential misuse.
Software and Data
Code is available under http:///s/ggggaiieoc////aaaaaaaa https://huggingface http:/ggnnnnaccc.coolooeeeeeeete http:///w.....oooooo.o.///ooo//a
ACIP-Models are available under
Acknowledgments
The authors thank Brennan Wilkerson and Ziyad Sheebaelhamd for helpful discussions. We thank John Arnold and Jannis Klinkenberg for their technical cluster support.
We kindly acknowledge funding by the German Federal Ministry of Education and Research (BMBF) within the project "More-with-Less: Effiziente Sprachmodelle für KMUs" (grant 011523013A). Computational resources were provided by the German Al Service Center WestAl and used to conduct the numerical experiments of this work.
References
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws, 2024. arXiv:2404.05405 [cs].
Saleh Ashkboos, Maximilian I.. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress Large Language Models by Deleting Rows and Columns, 2024. arXiv:2401.1500 [cs].
Matan Ben Noach and Yoav Goldberg, Compressing Pre-trained Language Models by Matrix Decomposition. In Kam-Fai Wong, Kevin Knight, and Hua Wu (eds.), Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 884-889. Association for Computational Linguistics, 2020.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation, 2013. arXiv:1308.3432 [cs].
Angie Boggust, Venkatesh Sivaraman, Yannick Assogba, Donghao Ren, Dominik Moritz, and Fred Hohman. Compress and Compare: Interactively Evaluating Efficiency and Behavior Across ML Model Compression Experiments. IEFE Transactions on Visualization and Computer Graphies, 31 (1):8009--8) 2025.
Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. Distillation scaling laws, 2025. arXiv:2502.08606 [cs].
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once for all: Train one network and specialize it for efficient deployment. In The Eighth International Conference on Learning Representations, 2020.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Patrick Chen, Hsiang-Fi Fu Yu, Inderjit Dhillon, and Cho-Jui Hsich. DRONE: Data-aware Low-rank Compression for Large NL.P Models. In Advances in Neural Information Processing Systems, volume 34, pp- 19321-29333 Curran Associates, Inc., 2021.
Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. NVIDIA A100 Tensor Core GPU: Performance and Innovation. IEEE Micro, 41(2):29-35 March 2021. ISSN 1937-4143. doi:
(0.100//M..200 .33619944.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley-Interscience, Hoboken, N.J, 2nd edition, 2006.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Newral Information Processing Systems, volume 36, pp. 10088-10115. Curran Associates, Inc., 2023.
Devvrit, Sneha Kudugunta, Aditya Kusupati, Tim Dettmers, Kaifeng Chen, Inderjit Dhillon, Yulia Tsvetkov, Hannaneh Hajishirzi, Sham Kakade, Ali Farhadi, and Prateek Jain. Matformer: Nested transformer for clastic inference. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neurol Information Processing Systems, volume 37, pp. 140535-140564. Curran Associates, Inc., 2024.
Carl Eckart and Gale Young. The Approximation of One Matrix by Another of Lower Rank. Psychometrika, 1(3):211 218, September 1936. ISSN 0033-3123, 1860-09S0. doi:
Ali Edalati, Marzich Tahaci, Ahmad Rashid, Vahid Nia, James Clark, and Mchdi Rezagholizadeh. Kronecker Decomposition for GPT Compression. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 219 226. Association for Computational Linguistics, 2022.
Bradley Efron, Trevor Hastie, lain Johnstone, and Robert Tibshirani. Least angle regression. The Anmals of Statistics, 32(2). 2004.
Elias Frantar and Dan Alistarh. SparseGPT:: Massive language models can be accurately pruned in one-shot. In Proceedings of the 40th International Conference on Machine Learning, pp. 10323-10337. PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefter, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Fleventh International Conference Leurning Representations, 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Fric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 2023. versio:: v0.4.0.
Weihao Gao, Yu-Han Liu, Chong Wang, and Sewoong Oh. Rate Distortion For Model Compression: From Theory To Practice. In Proceedings of the 96th International Conference on Machine Learning, pp- 2102-2111. PMLR, 2019.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. A Survey of Quantization ion Methods for Efficient. Neural Network Inference. and Hall/CRC, 2022. ISBN 978-1-00-316281-0. Num Pages: 36.
Compuler Vision. Chapman
Lucas Gonzalez and Rakesh Shivanna. Announcing Gemma 3n preview: powerful, efficient, mobilo-first AI, 2025. URL http::/ddevellpperrggoollloogo/ htt.g//llllg.cco/////attttttnnnn -arnggg--gmmmaa0 3n/ Google Blog. Accessod /2/22
15

---

Published in Transactions on Machine Learning Research (11/2025)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia- Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Fhab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, et al. The Llama 3 Herd of Models, 2024, arXiv:2407. [cs].
Song Han, Huizi Mao, and William J. Dally. Deep Compression: Compressing Deep Neural Networks with Pruming, Trained Quantization and Huffman Coding, February 2016. arXiv:1510.0014: [cs].
B. Hassibi, D.G. Stork, and G.J. Wolff. Optimal Brain Surgeon and general network pruning. In IEEE International Conference on Neural Networks, pp. 293 299, 1993.
Trevor Hastie, Robert Tibshirani, and Martin Wainwright. Statistical Learning with Sparsity: The Lasso and Generalizations. Chapman and Hall, 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network, 2015.
arXiv:1503.cm 1503.0233 [stat].
Jordan Hoffmann, Sebastian Borgeaud, A. Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, K. Simonyan, Erich Elsen, Jack W. Rae, O. Vinyals, and L. Sifre. Training Compute-Optimal Large Language Models, 2022. arXiv:2203.155: [cs].
Fred Hohman, Mary Beth Kery, Donghao Ren, and Dominik Moritz. Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pp. 1 18. Association for Computing Machinery, 2024.
Yen- Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization, 2022. axxiv:2207.00111 [cs].
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Adaptation of Large Language Models. In ICLR, 2022.
Weizhu Chen. LoRA:
Yerlan Idelbayev and Miguel A. Parreira-Perptiánn Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8046-8056 2020.
Berivan Tsik, Tsachy Weissman, and Albert No. An Information-Theortii Justification for Model Pruning. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, pp. 3821-3846.
PMLR, 2022.
Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang. From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients, 2024.
12Xiv:2407.112399 [cs).
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Maric-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B, 2023. arXiv:2310.0688 [cs).
J. Kaplan, Sam McCandlish, T. Henighan, Tom B. Brown, B. Chess, R. Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling Laws for Neural Language Models, 2020. arXiv:2001.08361 [cs].
16

---

Published in Transactions on Machine Learning Research (11/2025)
Ayush Kaushal, Tejas Vaidhya, and Irina Rish. LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compression, 2023. arXiv:2309.1400 [cs]
Bo-Kyeong Kim, Geonmin Kim, Tac-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Ki Kyu Song. Shortened LLaMA: A simple depth pruning for large language models. In ICLR 2024 Workshop Mathematical and Empérical Understanding of Foundation Models, 2024.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), ICLR, 2015.
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramamujan, William Howard-Snyderr Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. Matryoshka representation learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 90233-30243 Curran Associates, Inc.,
2022.
Yann LeCun, John Denker, and Sara Solla. Optimal Brain Damage. In Advances in Newral Information Processing Systems, volume 2. Morgan-Kantn Kaufmann, 1989.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner, On the structural pruning of large language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 21702 21720. Curran Associates, Inc., 2023.
Julien Mairal and Bin Yu. Complexity analysis of the lasso regularization path. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pp. 1835-1842. Omnipress, 2012.
Stephen Mcrity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
L. Mirsky. Symmetric Gauge Functions and Unitarily Invariant Norms. The Quarterly Journal of Mathematics, 11(1):50-59, 1960.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library, 2019. arX1v:1912.01703 [cs).
Wang Qinsi, Jinghan Ke, Masayoshi Tomizuka, Kurt Keutzer, and Chenfeng Xu. Dobi-SVD: Differentiable SVD for LLM compression and some new perspectives. In The Thirteenth International Conference on Learning Representations, 2025.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mci Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 Technical Report, 2024. arxiv:2412.15115 [es].
Colin RAffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michaol Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Tae Text Transformer. Journal of Machine Learning Research, 2019.
Sebastian Raschka. New LLM pre-training and post-training paradigms, 2024. Blog post. Accessed 1/31/2025.
Pratyusha Sharma, Jordan T. Ash, and Dipendra Misra. The Truth is There: Improving Reasoning in Language Models with Tayer-Selective Rank Reduction. In ICLR, 2023.
17

---

Published in Transactions on Machine Learning Research (11/2025)
Arun Suggala, Adarsh Prasad, and Pradcep K Ravikumar. Connecting Optimization and Regularization Paths. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A Simple and Effective Pruning Approach for Large Language Models, 2024. arXiv:2306.111 11669.
Marzieh Tahaei, Flla Charlaix, Vahid Nia, Ali Ghodsi, and Mehdi Rezagholizadeh. KroneckerBRRII Significatt Compression of Pre-trained Language Models Through Kronecker Decomposition and Knowledge Distillation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Seattle, United States, 2022. Association for
Långuislies: Human Language Technologies, pp. Computational Linguistics.
Robert Tibshirani. Regression Shrinkage and Selection Via the Lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267-255 1996.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, M. Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models, 2023a. 1rXiv:2302.13971 [cs].
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumyy Batra, Prajiwal Bhargava, Shruti Bhosale, D. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, A. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. Korenev, Punit Singh Koura, M. Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, M. Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023b. arXiv:2307.09286 (cs).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD-LLM: Truncation-aware Singular Value Decompo- sition for Large Language Model Compression, 2024. ar arXiv:2403.07378 [cs].
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning, 2024. arXiv:2310.06694 [es).
Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets. In International Conference on Learning Representations, 2018.
Hao Yu and Jianxin Wu. Compressing Transformers: Features Are Low-Rank, but Weights Are Not! Proceedings of the AAAI Conference on Artificial Intelligence, 37(9):11007 11015, 2023. Number: 9.
Mengxia Yu, De Wang, Qi Shan, Colorado Reed, and Alvin Wan. The Super Weight in Large Language Models, 2024. arXiv:2411.07191 [cs).
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models, 2024. arXiv:2312.05821 [cs].
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A Survey on Model Compression for Large Language Models. Transactions of the Association for Computational Linguistics, 22:1556-5777 2024.
18

---

Published in Transactions on Machine Learning Research (11/2025)
Contents of Appendix
A Additional Remarks
20
B Implementation Details
21
C Supplementary Results for Section 3.1 Section 3.4
22
D Further Experiments and Ablations
26
D.1 Efficiency Analysis
26
D.2 Impact of Low-Rank Adapters
27
D.3 Impact the Compression Rule
27
D.4 Impact of the Stopping Criterion
27
D.5 Impact of the Score Map Forecasting Pruning Patterns
28
D.6
Impact of the Score Map A Trivial One Does Not Work
28
D.7 Impact of Post- Tunnng
29
D.8 Impact of Individual Layers Example I.I.a I.LaMA2 MA.-M22113
D.9 Examples of Score Maps Generated by ACIP
30
D.10
Examples of Generated Text by Compressed Models
30
19

---

Published in Transactions on Machine Learning Research (11/2025)
A Additional Remarks
In this section, we discuss few additional aspects (in Q&A format) about our method and experimental design that were not (fully) addressed in the main part for the sake of brevity.
Q1. Why did you not direclly compare your resulls quanlization and full-weighh (unstructured) pruning?
A1. We argue that these are fundamentally different compression approaches. Full weight manipulations, in principle, have the potential to lead to more powerful compressions because they have more degrees of freedom (analogously to full-weight fine-tuning vs. PEFT). Therefore, they should not be seen as competing methods but complementary ones. We admit that practitioners probably would not favor ACIP over well-established and widely supported quantization techniques. However, the adapter-style nature of ACIP makes it suitable for combination. This can, for example, allow ACIP to be further improved through quantization of the singular vector matrices demonstrated in Section 3.4
Q2.
Why did you not compare with model distillation or combine ACIP with it?
A2. While model distillation can lead to outstanding compression results, o.g., see Bushridge et al. 2025 Raschka 2024). this approach requires significantly more resources than ACIP, typically orders of magnitudes more. A direct comparison is therefore not meaningful from our point of view, as it should at least be based on approximately the same computational budget.
Q3. Why do you propose luckproppagation-baaad algorithm instead of layer-wise weight updates?
A3. Let us first summarize several benefits of our end-to-end optimization approach from the main paper: (i) it is conceptually simple and requires no feature engineering, (ii) an error correction can be injected with almost extra costs, (iii) it allows us to perform efficient and accurate Any Compression.
Apart from that, and the best of our knowledge, existing compression algorithms that use layer-wise updates like ASVD (Yuan et al., 2024), SVD-LLL (Wang et al. 2024), or WeLore (Jaiswal al.) 2024) require separate fine-tuning step to achieve competitive downstream performance at stronger compression ratios. Therefore, the lower costs of layer-wise compression are actually dominated by a more expensive bockpropagation-blee boneb step. It remains open if similar results can be obtained by a fully tuning-free algorithm.
Q4. Why do you matriz factorization, and SVD in particular?
A4. Commilting to a backpropagation--basll algorithun (see Q3) means that we have to deal with increased memory requirements. As such, matrix factorization is not helpful in that respect because the number of parameters might even increase initially (for instance, an SVD-parametrization basically doubles the size of quadratic weight matrix). On the other hand, tuning and pruning only the bottleneck layer (i.e., the singular value masks in case of ACIP) has the potential for drastic size reductions and is highly parameter-efficient. For example, the number of tunable mask parameters for LLaMA-7B with ACIP <1M.
With this in mind, SVD as a specific matrix factorization is an obvious candidate due to its beneficial mathematical and numerical properties, in particular, optimal low-rank matrix approximation and stable matrix operations due to orthogonality.
20

---

Published in Transactions on Machine Learning Research (11/2025)
B Implementation Details
In this section, we report more technical details and hyperparameters used for our experiments.
Dataset and models. Following previous work on LLM compression, we use C4 (Raffel et al. 2019) for training as it is a good proxy of a general-purposs dataset. In the context of ACIP, it should be primarily seen as calibration dataset that allows us to propagate meaningful activations through a pre-trained model while performing structured pruning. Overfitting to the distribution of C4 implicitly mitigated, since we only tune very few parameters (masks and low-rank adapters) compared to the total model size. As loss function in {
use the standard negative log- ligliike loss for next-token prediction.
All considered (evaluation) datasets and pre-trained models are imported with the HuggingFace transformers- library in boat 16 1------isiion Our experiments were implemented with PyTorch (Paszke et al. 2019 and the Lightning package.
ACIP-specifics. As mentioned in Remark 2.1 we apply a linear scheduler that increases the regularization parameter λ dynamically over the pruning process. This ensures that the pruning becomes more and more nggressive time and the stopping criterion will be reached at some point. Across all experiments, we use = initial value and increase it by factor of 1.01 every 4 steps (this amounts to doubling of À at about every 280 steps).
As pointed out in Section 2.2.2 we choose a target compression rate as a stopping criterion for ACIP, In most experiments, a rate of sstop = 0.4 is reasonable (i.c., only 40% or the original parameters remain), and we refer to Appendix D.4 for further discussion and analysis. After the slopping crilerion is reached, we lune the low-rank adapter for 1k more steps while the masks are frozen (see Section 2.2.2
The mask parameters in 77 are rescaled by fixed factor of 0.02 to ensure better alignment with the numerical range of the remaining nerw weights. The low-rank adaplers are created with 32, a 16, and dropout 0.05. For LLaMA-7B, the number of tunable parameters amounts to <IM mask parameters and approximately SOM low-rank adapter parameters.
For sample data from C4, we use 1024 tokens per sample and a batch size of We use Adam (Kingma Ba 2015) as optimizer without weight decay and a learning rate of 5e-5.
Runtime analysis. ACIP requires significantly fewer steps than fine-tuning. Depending when the stopping criterion is reached, it typically takes 1.5k 2.5k steps, including 1k post-tuning steps of the low-rank adapters. For LLaMA-7B, for example, this amounts to wall clock runtime of 30 minutes, including the initial SVD computations for the base model parametrization. All runs were performed on single NVIDIA H100 GPUs. See also Appendix D.1 for more detailed efficiency analysis.
Fine-tuning. In all post-compression fine-tunings (see Section 3.3 we simply contimue training ACIP's low-rank adapters (the optimizer states are reset). We train for 25k steps on C4 with batch size of and a learning rate of 2e-4.
21

---

Published in Transactions on Machine Learning Research (11/2025)
C
Supplementary Results for Section 3.1 Section 3.4
Figure A8 complements the trade-off curves in Figures 4 and 5 by all other considered evaluation metries reports these results in terms of numbers, including all line-tuning results for all analyzes the effect of fine-tuning of ACIP compared to existing SVD-based provides more detailed evaluation results on fine-tuning a quantized and compressod LLaMA-7B model (see Section 3.4).
(see Section 3.1 Table A2
models (see Sention 3.3 Table A3
compression methods. Table A4
22

---

Published in Transactions on Machine Learning Research (11/2025)
100
LLaMA-7B
LLaMA-78
0.55
[1.MMA138
LTaMA-13R
Perplexity [log seale]
LLaMA-2-78
0.5
LLaMA-2-78
11.MMA2--138
0:45
L1aMA-2-13R
LLAMA-3.1-4B
Accuracy
LLaMA-3.1-88
Mistral- 7B.+03
Mistral-B--v.33
0.35
Owea22.533
Q.ea2.5388
10
Qwen2.5-71
0.3
Qwen25-70
Qwen2.5--4B
0:25
Qwea255114
18
10B
18
108
Number of Parameters Dlog scale]
Number of Parumeters (log scale]
(a) 2
(b) ARC Challenge
0.85
LLaMA-7B
0.65
LLaMA-7B
0.8
LLLMA-138
0.5
LTaMA-13R
LlaMA-2-78
LLaMA-2-78
0.7
L1-MA-2-138
-1aMM-2-13R
Accurccy
I.aMA-3 1-8B
Accurncy
LLaMA-3.14B
0.65
Mistral 7B v0.
0.45
Mistral 7B.
0.6
0.55
Qwen2.5-77 7-5B
Qwen2.5
0.4
Qwea2.538
Qwen2.5-0
Qwen2.5 2:4D
0.35
Qwea25:14B
0.45
0.3
18
10B
18
108
Number of Parameters (log scale]
Number of Parumeters (log scall]
(c) ARC Easy
(d) HellaSwag
LL-MA-7B
0.35
LLAMA-7B
0.5
LLaMA-138
LLLMM-118
LLaMA-2-7B
LLaMA-2-7B
0.45
LL-MA-2-138
LLaMA-2-138
Accurayy
0.4
LLaMA-3.14B
Accuracy 7
LLaMA-3.14BB
0.35
Mistral 7B.3
0.25
Mistral 7B 7B-
He- Qwen2.5 5 39
.e- Qwen2.5 3B
0.3
Qwen2 5-38
0.2
Qwe2 2-78
0.25
Qwen2.5 14B
Qwen2: 14B
0.2
1B
10B
108
Number of Parameters Dlog scale]
Number of Parameters [log scale]
(e) MathQA
(f) OpenBookQA
LLAMA-7B
LLAMA-TB
0.8
LL-MA-138
LLaMA-138
LL.MA-2.7B
0.7
LLAMA-2.77
LLaMA-2-138
LLLMMA--133
Accursey
LLaMA
Accuracy
0.65
LLaMA
0.7
Mistral vO.3
Mistral vo.3
Qwen2.5 5 3D
0.5
Qwen2.5 3D
0.65
Qwen2.5-1400
5:38
0.55
114
Qwan2 5-3B
0.6
0.5
1B
10B
18
100
Number of Parameters (log scale]
Number Parameters [log scale]
(g) PIQA
(h) Winogrande
Figure A8: Compresioon-eeformance trade-off curves generated by ACIP on WikiText-2 and individual LM-Eval tasks, complementing the results of Figures 4 and 5
23

---

Published in Transactions on Machine Learning Research (11/2025)
Table A2: Evaluation results for ACIP on all considered LLMs Scores on C4 and WikiText-2 are measured in perplexity (smaller is better), and the LM-Fvvll zero-shot tasks are measured in accuracy (higher is better). "The results of LLaMA2-133 were achieved by ignoring all up-projection layers in ACIP (see Appendix D.S
LM Exall
C4 4
W
ARC c
ARC
e
HellaS. t
MatbQA f
Opeab. t
PIQA
WinnG. f
Avg. 1
Type Sizer
ACIP
FT
ACIP
FT
ACIP
FT
ACIP
FT
ACIP
FT
ACIP
FT
ACIP
FT
ACIP
FT
ACIP
FT
ACIP
FT
Model
40%
21.05
15.66
23:99
17:33
0.24
0.24
0.49
0.53
0.35
0.35
0.21
0.21
0.19
0.20
0.64
0.67
0.55
0.57
0.38
0.40
50%
16.47
12.89
16.16
11.88
0.27
0.27
0.57
0.59
0.40
0.43
0.22
0.23
0.21
0.23
0.68
0.71
0.57
0.60
0.42
0.44
60%
13.91
11.14
12.46
9.63
0.30
0.32
0.61
0.64
0.44
0.47
0.24
0.23
0.25
0.26
0.71
0.73
0.50
0.63
0.45
0.47
LLaMA-7B
70%
12:22
9.84
10.35
8:27
0.34
0.61
0.47
0.50
0.23
0.24
01:28
0.73
0.75
0.62
0.66
0.49
80%
10.92
8.81
8.83
7.19
0.32
0.38
0.66
0.71
0.49
0.53
0.23
0.24
0.28
0.31
0.74
0.77
0.63
0.68
0.48
0.52
90%
9.75
7.69
7.56
6.12
0.33
0.39
0.67
0.73
0,50
0.56
0.25
0.25
0.27
0.33
0.76
0.78
0.63
0.69
0.19
0.53
100%
9.52
7.32
7:20
5.75
0.40 0.33
0.69
0.75
0.51
0.37
0.25
0.26
0.26
0.34
0.76
0.79
0.63
0.70
0.49 0.54
Orig.
7.31
5.68
0.42
0.75
0.57
0.27
0.31
0.79
0.70
0.55
40%
16.64
13.38
17.66
13.42
0.28
0.28
0.57
0.59
0.41
0.43
0.22
0.23
0.23
0.24
0.69
0.70
0.60
0.62
0.43
0.44
50%
13.06
11.12
12.42
10.49
0.32 0.34
0.63
0.63
0.47
0.48
0.23
0.23
0.26
0.28
0.72
0.73
0.02
0.64
0.47
0.47
60%
11:31
9.76
9.79
8.300
0.35
0.33
0.67
0.50
0.24
0.24
0.28
0.30
0.74
0.76
0.65
0.67
0.50
LLMMA-13B
70%
10.10
8.72
8.17
7.06
0.38
0.38
0.70
0.69
0.53
0.54
0.24
0.26
0.31
0:31
0.76
0.77
0.67
0.70
0.51
0.52
80%
9.06
7.91
6,91
6,21
01.1
0.41
0.74
0.74
0.55
0.57
0.26
0.27
0.32
0.34
0.77
0.78
0.68
0.69
0.53
0.54
90%
8.01
7.06
5.98
5.10
0.42
0.41
0.75
0.76
0.57
0.39
0.29
0.29
0.32
0.33
0.79
0.79
0.70
0.72
0.55
0.56
100%
7.86
6.79
5.83
5.15
0.46 0:47
0.75
0.57
0.29
0..30
0.79
0.79
0.70
0.73
0.57
Orig.
6,77
5.09
0.77
0,60
0.30
0.33
0.79
0.73
0.57
40%
21.62
16.74
29.20
18.00
0.21
0.22
0.46
0.50
0.31
0.37
0.20
0.21
0.18
0.19
0.62
0.66
0.51
0.55
0,36
0.39
60%
18.36
13.32
19.64
12.25
0.25
0.27
0.53
0.57
0.38
0.42
0.22
0.22
0.24
0.23
0.67
0.00
0.53
0.56
0.40
0.42
60%
15.27
11.15
11.47
9.51
0.28
0.31
0.39
0.43
0.46
0.23
0.21
0.25
0.69
0.72
0.57
0.62
0.11
0.46
LLaMA-2-7B
70%
12.96
9.73
10.47
7.74
0.33
0.35
0.62
0.68
0.46
0.50
0.25
0.24
0.27
0.26
0.73
0.74
0.60
0.64
0.47
0.49
80%
11.31
8.63
8.46
6.54
0.33
0.37
0.66
0.70
0.49
0.53
0.25
0.26
0.28
0.31
0.74
0.77
0.63
0.66
0.48
0.51
90%
9.48
7.43
6.69
5.45
0.34
0.43
0.69
0.75
0.51
0.56
0.26
0.28
0.28
0.33
0.76
0.78
0.63
0.69
0.54
100%
19.54
7.06
6,54
5,13
0.31
0.43
0.70
0.76
0.51
0.37
0.26
0,28
0.27
0.33
0.76
0.78
0.61
0.69
0.50
0.55
Orig.
7.04
5.11
0..4
0.76
0.57
0.28
0.31
0.78
0.60
0.55
4075
27.55
8-1.28
11.22
1:45.79
0.23
0.23
0.43
0.46
0.33
0.313
0.21
0.21
0,18
0.18
0.62
0.63
0.52
0.52
0.36
0.36
50%
17.30
12.76
17.89
133.12
0.32
0.58
0.41
0.44
0.21
0.233
0.227
0.69
0.70
0.56
0.58
0.45
60%
13:29
10.05
11.11
8.13
0.32
0.36
0.65
0.68
0.47
0.50
0.22
0.23
0.29
0.31
0.72
0.74
0.00
0.63
0.47
0.19
LL-MA-2-13B
70%
11.01
8.61
8.40
6.71
0.35
0.11
0.70
0.74
0.51
0.54
0.223
0.25
0.30
0.33
0,74
0.77
0.62
0.66
0.19
0.53
80%
9.54
7.68
6.80
5.06
0.37
0.44
0.72
0.76
0.54
0.37
0.25
0.28
0.30
0.34
0.77
0.78
0.64
0.71
0.51
0.55
90%
8.26
6.86
5.70
4.87
0.40
0.47
0.74
0.78
0.55
0.60
0.26
0.31
0.31
0.34
0.78
0.79
0.66
0.72
0.53
0.57
100%
7.87
6.56
5.42
4.61
0.41 0.47
0.75
0.79
0.56
0.60
0.28
0.32
0.31
0.35
0.78
0.79
0.60
0.71
0.54
0.58
Orig.
6.52
4.57
0.48
0.79
0.60
0.32
0.35
0.79
0.72
0.58
50%
43.32
26.52
61.77
29.52
0.23
0.27
0.51
0.33
0.37
0.22
0.23
0.17
0.19
0.64
0.68
0.53
0.54
0.41
60%
$1.55
21.00
36.69
19.26
0:29
0.29
0.6D
0.61
0.37
0.42
0.24
0.24
0.22
0.23
0.69
0.72
0.55
0.56
0.42
0.44
70%
24.90
17.08
23.06
13.55
0.33
0.32
0.64
0.66
0.41
0.47
0.26
0.27
0.23
0.27
0.71
0.73
0.58
0.62
0.45
0.48
LLMMA-3.1-B
80%
20.78
1/1.21
15.60
10.12
0.38
0.40
0.69
0,72
0.46
0.51
0.28
0.30
0.28
0.29
0.74
0.77
0.61
0.66
0.49
0.52
90%
16.25
11.28
9.80
7:24
0.41
0.48
0.74
0.52
0.57
0.33
0.35
0.27
0.31
0.77
0.79
0.67
0.71
0.53
0.57
100%
11.57
9.41
8.01
5.95
0.51 0.40
0.75
0.82
0.53
0.60
0.36
0.10
0.27
0.34
0.78
0.80
0.66
0.74
0.54 0.60
Orig.
9.31
5.86
0.51
0.82
0.60
0.39
0.33
0.80
0.74
0.60
10%
28.09
19:21
44.29
23.24
0.24
0.26
0.48
0.53
0.35
0.35
0.21
0.21
0.18
0.19
0.06
0.67
0.55
0.67
0.38
0.40
50%
21.44
14.86
28.60
16.53
0.28
0.28
0.57
0.50
0.40
0.43
0.21
0.23
0.22
0.21
0.69
0.70
0.58
0.60
0.42
0.43
60%
16.80
12.49
21.19
12.29
0.32
0.32
0.63
0.66
0.45
0.48
0.24
0.24
0.20
0.23
0.72
0.73
0.60
0.62
0.45
0.47
Mistral-7B-v0.3
70%
13.75
10.95
13:25
9.69
0.35
0.34
0.67
0.68
0.49
0.52
0.27
0.28
0.21
0.26
0.74
0.76
0.63
0.63
0.48
0.49
80%
11.80
9.81
8.70
7.19
0.39
0.70
0.73
0.52
0.35
0.29
0.29
0.76
0.65
0.69
0.50)
0.53
90%
10.42
8.84
6.51
5.85
0.40
0.43
0.72
0.75
0.54
0.59
0.31
0.333
0.25
0.31
0.78
0.79
0.68
0.70
0.53
0.56
100%
9.85
8.31
6.01
5.31
0.40
0.15
0.73
0.77
0.55
0.60
0.33
0.34
0.26
0.33
0.78
0.79
0.68
0.72
0.53
0.57
Orig.
8.05
4.96
0.49
0.79
0.61
0.33
0..34
0.80
0.73
0.59
40%
71.23
36.86
91.51
39.44
0.20
0.22
0.45
0.61
0.29
0.31
0.21
0.22
0.15
0.16
0.00
0.64
0.50
0.52
0.34
0.37
50%
37.17
29.43
62.12
26.92
0:22
0.25
0.49
0.57
0.32
0.31
0.22
0.21
0.18
0.19
0.63
0.67
0.52
0.53
0.37
0.39
60%
43.30
23.30
38.26
18.38
0.26
0.29
0.57
0.63
0.35
0.39
0.23
0.23
0.21
0.24
0.67
0.69
0.54
0.56
0.40
0.43
Qwen2.5-3B
70%
34.24
19.04
25.81
13.50
0.34
0.62
0.39
0.43
0.25
0.26
0.24
0.70
0.72
0.57
0.58
0.44
0.47
80%
25.50
16.16
17.02
10.68
0.34
0.36
0.68
0.73
0.43
0.47
0.28
0.31
0.26
0.24
0.72
0.74
0.58
0.57
0.47
0.49
90%
20.10
133.82
11.99
8.57
0.37
0.42
0.73
0.77
0.46
0.52
01.33
0.36
0.27
0.33
0.74
0.78
0.60
0.67
0.50
0.55
100%
18.73
12.81
10.63
7.70
0.47 0.36
0.72
0.78
0.46
0.54
0.33
0.41
0.24
0.31
0.74
0.78
0.69 0.68
0.60
0.49
0.57
Orig.
12.90
7.61
0.77
0.35
0.337
0.300
0.78
40%
46.43
29.26
49.04
27.24
0.23
0.24
0.52
0.58
0.31
0.34
0.22
0.21
0.19
0.20
0.64
0.67
0.53
0.53
0.38
0.40
50%
31.90
23.26
29.
29.96
19.72
0.27
0.30
0.59
0.61
0.35
0.39
0.22
0.23
0.23
0.25
0.67
0.70
0.51
0.57
0.11
0.11
60%
27.84
18.73
21.98
13.73
0.31
0.33
0.63
0.39
0.45
0.25
0.26
0.25
0.28
0.70
0.73
0.55
0.00
0.44
0.48
Qwen2.5-7B
70%
22.97
15.96
15.72
10.71
0.35
0.42
0.68
0.74
0.44
0.19
0.28
0.30
0.29
0.73
0.76
0.57
0.63
0.48
0.52
80%
19.68
14.02
12.07
8.80
0.40
0.46
0.73
0.78
0.48
0.53
0.33
0.36
0.30
0.33
0.75
0.78
0.50
0.67
0.51
0.56
90%
17.09
12.59
9.82
7:63
0.44
0.48
0.75
0.80
0.52
0.56
0.39
0.41
0.31
0.32
0.77
0.79
0.64
0.70
0.54
0.58
100%
15.34
11.43
8..8
6.60
0.43
0.50
0.75
0.53
0.59
0..43
0.46
0.77
0.79
0.66
0.72
0.55
0.60
Orig.
11.47
6.55
0.48
0.81
0.60
0.43
0.34
0.79
0.73
0.60
40%
36.51
25.58
33.788
22.22
0.26
0.29
0.55
0.61
0.36
0.35
0.22
0.233
0.244
0.26
0.67
0.68
0.54
0.57
0.41
0.43
50%
26.27
19.53
20.15
11..7
0.32
0.33
0.65
0.68
0.13
0.41
0.25
0.25
0.26
0.27
0.70
0.71
0.57
0.59
0.15
0.17
60%
21.29
13.63
14.87
10.48
0.40
0.70
0.73
0.49
0.28
0..32
0.28
0.74
0.76
U.62
0.66
0.50
0.53
Qwen2.5-14t
70%
17.98
13.99
11.25
8.89
0.42
0.43
0.73
0.76
0.53
0.54
0.33
0.36
0.31
0.32
0.77
0.78
0.64
0.68
0.53
0.55
80%
15.23
11.97
8.73
7.11
0.41
0.48
0.77
0.78
0.57
0.58
0.39
0.14
0.31
0.36
0.78
0.79
0.68
0.73
0.57
0.60
90%
13.05
10.77
0.86
6.04
0.48
0.52
0.80
0.82
0.59
0.60
0.49
0.40
0.32
0.35
0.79
0.81
0.70
0.74
0.00
0.62
100%
12.37
9.98
6.23
6.11
0.47
0.53
0.79
0.82
0.58
0.62
0.51
0.53
0.32
0.35
0.79
0.81
0.73
0.77
0.00
0.63
Orig.
9.90
5.05
0.56
0.82
0.63
0.53
0.35
0.81
0.75
0.64
24

---

Published in Transactions on Machine Learning Research (11/2025)
Table A3: Evaluation of LLaMA-7B on WikiText-2 (perplexity, smaller is better) under different com- pression ratios, with and without post-training fine-tuning. We compare ACIP with the existing SVD-based compression methods ASVD (Yuan et al., 2024 and SVD-LLM (Wang et al. 2024), see also Scction 3.2 The scores for ASVD and SVD-LLM are taken from Wang et al. [2024 Table 4). Note that ACIP was on C4, while ASVD and SVD-LLM tne-tuned on WikiText-2 directly.
Compression Ratio
40%
50%
60%
70%
80%
Method
ASVD
57057.00 00
15358.00
1407.00 .00
51.00
11.14
ASVD LORA FT
44.81
21.83
14.86
10.16
8.37
SVD-LLM
12.30
23.97
13.11
9.56
7.94
SVD-LLM + LORA FT
17.93
13.26
10.65
9.14
7.78
ACIP
24.00
16.17
12.46
10.34
8.83
ACIP +
17.33
11.88
9.63
8.27
7.19
Table A4: More detailed evaluation results for our quantization experiments in Section 3.4 of numbers.
reported in terms
RIf. model size [GB]
C4
WikiTexxt22 ARC ARC e HellaS. MathQA Openb. PIQA WinoG. LM Fval
Size
Ablation
t
Avg. f
ACIP
5.47
21.05
24.00
0.24
0.49
0.35
0.21
0.19
0.65
0.55
0.38
ACIP FT
5.47
15.66
17.33
0.24
0.53
0.38
0.21
0.20
0.67
0.57
0.40
ACIP W4A16
1.89
27.12
35.40
0.22
0.46
0.33
0.20
0.18
0.62
0.53
0.36
ACIP W4A16 FT
1.89
16.67
18.90
0.23
0.52
0.37
0.22
0.20
0.67
0.57
0.40
ACIP
6.70
16.47
16.17
0.28
0.58
0.40
0.22
0.21
0.68
0.57
0.42
ACIP FT
6.70
12.89
11.88
0.27
0.59
0.43
0.23
0.23
0.71
0.60
0.44
50%
ACIP W4A16
2.21
19.
19.96
0.26
0.54
0.37
0.21
0.20
0.67
0.55
0.40
ACTP W4A16
2.21
13.85
13.33
0.25
0.58
0.41
0.23
0.23
0.70
0.59
0.43
ACTP
7 7,88
13.91
12.46
0.30
0.61
0.43
0.23
0.25
0.71
0.60
0.45
ACIP FT
7.88
11.14
9.63
0.32
0.64
0.47
0.23
0.26
0.73
0.63
0.47
60%
ACIP W4A16
2.51
15.84
14.64
0.29
0.58
0.42
0.22
0.22
0.69
0.57
0.43
ACIP W4A16
2.51
11.77
10.31
0.29
0.64
0.45
0.22
0.26
0.72
0.61
0.46
ACIP
12.22
10.34
0.31
0.64
0.47
0,23
0.27
0.73
0.62
0.47
ACIP FT
9.84
8.27
0.34
0.69
0.50
0.24
0.28
0.75
0.66
0.49
70%
ACIP W4A16
13.45
11.80
0.29
0.63
0.45
0.23
0.24
0.72
0.45
ACIP W4A16
10.38
8.74
0.67
0.48
0.23
0.28
0.75
0.48
ACIP
8.83
0.67
0.49
0.23
0.28
0.74
0.48
ACIP
10.30
8.81
7.19
0.71
0.53
0.52
80%
W4A16
9.87
0.65
0.47
0.23
0.28
0.74
0.47
ACIP W4A16
9.26
7.60
0.52
0.24
0.30
ACIP
11.50
9.75
7.56
0.34
0.65
0.50
0.25
0.27
0.75
0.49
ACIP FT
11.50
6.12
0.73
0.56
0.25
0.33
0.78
0.53
90%
ACIP W4A16
10.25
7.90
0.32
0.66
0.50
0.24
0.27
0.75
0.48
ACIP W4A16
3.44
7.97
6.39
0.73
0.56
0.25
0.35
0.78
0.53
ACIP
12.70
9.52
7.20
0.33
0.69
0.51
0.25
0.26
0.76
0.49
ACIP FT
12.70
7.32
5.75
0.40
0.75
0.57
0.26
0.34
0.79
0.70
0.54
100%
ACIP W4A16
3.75
9.75
7.37
0.34
0.69
0.51
0.25
0.27
0.76
0.63
0.49
ACIP W4A16 FT
3.75
7.52
5.94
0.40
0.75
0.56
0.27
0.34
0.78
0.69
0.54
25

---

Published in Transactions on Machine Learning Research (11/2025)
D Further Experiments and Ablations
In this section, present several supplementary experiments analyzing the impact of some key algorithmic components and design choices of ACIP in more detail. Note that the most detailed analyses and ablations are carried out with LLaMA-7B as it was most exlensively studied in previous research on structured weight pruning.
D.1 Efficiency Analysis
Table A5 reports several statistics the efficiency of the ACIP algorithm and inference speed of compressed models. While these preliminary results do not immediately indicate gains in inference speed, we expect that further optimization like merging the low-rank adapters can compensate for the matrix-factorizatti overhead (one additional matrix-vector multiplication) and outperform the base model. Moreover, we note that compared to performance-size trade-offs, which are our main concern, analyzing inference speed-upe requires very careful consideration about the hardware in use (accelerator model, parallel processing units, etc.) and measurement. setup (sequence length, batch size, etc.).
Table A5: Efficiency analysis ACIP for LLaMA-7B. The first three rows report the runtime and memory statistics of ACIP's key steps (see Section 2.2 and Figure both lerms of numbers and their qualitative asymptotics. Here, the model sizes are measured (uncompressed) checkpoint sizes. "Runtime pruning" refers to the process of pruning the mask parameters to desired compression ratio (revertible), whereas "Runtime compress" refers to the process of discarding pruned singular vectors and possibly umparametrizing linear layers, so that the model gets actually compressed (see Step 3 in Section 2.2.3 The statistics of inference speed were obtained by generating new text of sequence length 64 and batch size 64. To measure FLOPs, we use the fvcore package and an input sequence of length 512.
Stage
Metric
LLaMA-7B
Runtime [min]
4.95
ACIP Step 1 (Model Reparametrization) O(#Layers x SVD of Layer)
Size parametrized model [GB]
19.71
Size base model [GB]
12.70
Runtime [min]
23.12
ACIP Step 2 (Scoring by Iterative Pruning)
Reserved GPU memory peak [GB]
62.45
O4#tteps of Masks & LoRA Updates)
Steps /
1.68
ACIP Step 3 (Any Compression) O#HLyers Layer Input Dimension)
Runtime pruning [s]
0.49
Runtime compress [s]
0.18
Size model [GB]
5.47
Reserved GPU memorr peak [CB]
25.68
Inference at 40% Size
Latency [s]
2.57
Tokens s
1594.99
GigaFT.OPs
1335.85
Size model [GB]
9.10
Reserved GPU memory peak [GB]
29.43
Inference at 70% Size
Latency [s]
2.17
Tokens /
1658.63
GigaFLOPs
2265.03
Size model [GB]
12.70
Reserved GPU memory peak [GB]
32.79
Inference at 100% Size (Original)
Latency [s]
1.67
Tokens /s
2447.75 75
GigaFLOPPs
3188,63
26

---

Published in Transactions on Machine Learning Research (11/2025)
100
ACIP
ACIP without LoRA
Perplexity [log scale]
Relative Model Size [%)]
Figure A9: Compression- performance trade-off curves for LLaMA-7B C4 with and without using LoRA-adaptes for correction ion in ACIP.
Sempreesion-peffrmancce
D.2 Impact of Low-Rank Adapters
The primary purpose the low-rank adapters used in ACIP to correct compression errors on-the-fly during the optimization. A surprising finding of our work is that the final adapters are "universal" in the sense that they can be used across all seen compression levels. While we expect that other PEFT-style approaches would lead to similar findings, it is natural to ask how ACIP would perform without any correction, i.c., just the mask parameters are tuned according to ( This ablation study is shown in Figure A9 While performing significantly worse than with LoRA, we observe that the perplexity does not blow up and the results are even slightly better than SVD-LLM (see Table D This stable behavior of ACIP is closely related to our paramcterization of the mask in 7 which ensures that the forward pass corresponds to the actual outputs
of the pruned model with binary masks. On the her hand, the straight-through estimator still enables backpropagation.
D.3 Impact of the Compression Rule
The Any Compression stage of ACIP (Step 3 in Section 2.2.3) relies on a simple algorithmic rule: Prune as many singular values according to the learned score map P as needed for a desired compression rate r. Here, the tuned low-rank adapters A used across all compression levels, even for 1.0, i.e., the size of the (uncompressed) base model. While the usage of low-rank adapters is helpful for simultaneous error correction along with iterative pruning (Step 2 in Section 2.2.2 it can lead to a slight drop in performance for lower compression levels. This became already apparent in the ablation of Figure A9 above, where LoRA was fully omitted.
Figure A10 shows that this performance gap can reduced by a refinement of Step in ACIP: a linear layer is not compressible for given rate (i.e., low-rank factorization would not save any parameters), reset the corresponding mask parameters Pa to 1.0 and disable the low-rank adapter A In this way, the layer is fully reset, and for 1.0, exactly recover the base model. Note that this adapted rule is fully reversible and therefore still allows for Any Compression.
D.4 Impact of the Stopping Criterion
In most experiments, we have used rstop 0.4 maximum reasonable compression ratio, i.e., the pruning of masks is stopped if the size of the model is only 40% of the original one (measured in number of parameters of all target weight matrices). We have observed that at this point, the model performance has typically dropped so much that even a fine-tuned model would be of limited practical use.
Nevertheloss, interesting to explore the sensitivity of compression-performee stopping ratios. The comparison shown in Figure provides several insights in this respect: (i) "Forecasting" compressed models beyond the stopping ratio does not work very well, especially when stopping very
curves against different
27

---

Published in Transactions on Machine Learning Research (11/2025)
22
ACIP without layer (defauit)
20-
ACIP with layer reset
Perplexity [log scale]
18
12
Relative Model Size [%)]
Figure A10: Compression-peformmncc trade-off curves for LLaMA-7B C4 with and without resetting
linear layers if they are incompressible for given larget compression rate r.
Stop ACIP at
100k
Stop ACIP 03
Parlexity [log scale]
Stop ACIP 0.
10k
Stop ACIP 0.6
1
Stop ACIP at 0.8
2
Stop ACIP at 0.95
100
0.2
0.4
0.6
0.8
Relative Model Size [96]
Figure A11: Compression-perfrmnaee trade-off curves for LLaMA-7B on C4, using different stopping compression ratios Tstop for ACIP
early (> 0.8). (ii) The predictive capacity of ACIP remains valid for even stronger stopping compression ratios than 0.4. However, finding the largest reasonable stopping ratio is highly model-dependent.. For less compressible models like LLaMA-3.1-8B, it could make sense to stop even earlier than 0.4 (cf. Figure general, we hypothesize that older models are more compressible than new ones, as the latter "carry" more information per weight due significantly more training data [Allen-Zhi Zhu & Li 2024
D.5 Impact of the Score Map Forecasting Pruning Patterns
Here, we pick up the observation from Appendix D.4 that forecasting the performance of compressed models beyond the stop ratio leads to inaccurate predictions, i.c., the model is compressed more strongly than it has been done by ACIP itself. However, it turns out that the score map itself exhibits a certain forecasting capability. To this end, run ACIP as usual until stop ratio is reached, say Tstop = 0.4, but we stop updating the score map earlier in the optimization process. A few compression compression-peffrmmance performaace curves with this modification are reported in Figure A12, We observe very similar curve shapes even the score map is frozon after only tiny fraction mask parameters was pruned. This underpins our intuition from Section 2.2.2 that the pruning path of each parameter is fully determined at very early stage of ACIP.
D.6 Impact of the Score Map A Trivial One Does Not Work
There are certainly alternative ways to design useful score maps. For example, simply accumulating the gradients of all mask parameters entrywise over ACIP-run works equally well the strategy proposed in
28

---

Published in Transactions on Machine Learning Research (11/2025)
Stop map 0.4
Stop map 0.6
Perplexity [log scale)
26
Stop map 0.8
24
Stop map 0.95
20
Stop map
18
16
10
Relative Model Size [%)]
Figure A12: peaeression-eefformanne trade-off curves for LLaMA-7B on C4, stopping updates of the score
map before the actual stopping criterion of ACIP
Singular value score map
Singular value score map - Finetuning
Parlexity [log scale]
10k
1000
100
0.4
0.6
0.8
Relative Model Size (%)
Figure A13: Compression-perfornnace trade-off curves for LLaMA-7B on C4, using a trivial score map based on the initial singular values of the base model.
Section 2.2.2 It is therefore valid to ask whether one could even design score maps without any optimization. We demonstrate that perhaps the most obvious approach, namely setting the score map equal to the singular values of the weight matrices, does not work very well. Figure A13 shows that this training-free approach does not produce any reasonable compressed models and decent performance cannot be easily recovered with LonA-finetuning. This simple experiment confirms that designing useful score maps not a trivial endcavour and requires a carefully craftd algorithmic approach.
D.7 Impact of Post Tuning
Our main experiments performed with 1K post-tuning steps in ACIP (see the description in Section 2.2.2 and Appendix Bp. Figure A14 shows analogous prrpfrsaen trade-off curves for fewer no post- tuning steps. We observe that post-tuning can indeed notably increase performance for higher compression ratios.
D.8 Impact of Individual Layers Example of LLaMA2-13B
As pointed out in the caption of Table A2 the linear layers targeled by ACIP were slightly modified for LLaMA2-13, namely all up projection layers were ignored. Figure [A15 shows what would happen they are compressod as well. While the performance predictions for 0.6 look decent, the perplexity explodes for stronger compression; note that even additional fine-tuning does not recover reasonable performance this situation. We hypothesize that ACIP has pruned one more singular values of the up projection layers
29

---

Published in Transactions on Machine Learning Research (11/2025)
ACIP with post steps
ACIP with 100 post-tune steps
Perplexity [log scale]
ACIP with 500 post-tume steps
ACIP with 1000 post-tune (default)
20
18
14
Relative Model Size [%)
Figure A14: Comprestion-performance trade-off curves for LLaMA-7B on C4 with different numbers of post-luning steps in ACIP
without up-projections with up-projections
Perplexity [log scale]
100
Relative Model Size [96]
Figure A15: Compression-perfornnnce trade-off curves for LLaMA2-13B C4, (not) ignoring the up projection layers in ACIP.
that are crucial for model's integrity. This finding might be related to the recent work by Yu et al. (2024) on pruning so-called super weights. In any case, ACIP is capable of revealing this undesirable behavior as demonstrated in Figure A15
D.9 Examples of Score Maps Generated by ACIP
Figure A16 and Figure A17 show two typical score maps generated by ACIP for LLaMA-7B and Qwen2.5-7B. respectively. characteristic feature is that attention layers can be pruned more aggressively than the MLP layers. Similarly, we observe non-uniform pruning patlerns for layers of the same lype across all transformer layers. This confirms the findings of (Yuan al., 2024 Jaiswal al. 2024) and demonstrates that non-uniform structured compression can be achieved without any feature engineering.
D.10
Examples of Generated Text by Compressed Models
Table A6 shows examples of generated text by compressed versions of LLaMA-7B.
30

---

Published in Transactions on Machine Learning Research (11/2025)
Mask Parnmeter Index
0.2
Mask Parameter Index
0.2
004
00.4
0.6
000
0.8
0.8
20
30
201
30
Layer
Layer
(a)
Down Projections
(b) Up Projections
512
512
Mask Parameter Index
1024
-02
Mask Parmmctor Index
1024
002
1536
1536
-04
2048
2048
2560
2560
3584
3584
20
30
201
30
Layer
Layer
(c) Gate Projections
(d) Attention-O
$12
$12
Mask Parameter Index
1024
Mask Parnmeter Index
1004
1536
1536
2048
2048
2560
0.6
2560
0.6
3072
-08
3072
-088
3584
3584
Layer
Layer
(e)
Attention-
ff Attention-Q Q
512
Mask Parmmeteer Index
1024
-02
0.4
2048
2560
0.6
3072
0.8
3584
20
30
Layer
(g) Attention-K
Figure A16: Example score maps generated by ACIP for LLaMA-7B. The negative values (cf. Algorithm normalized to -1 for the purpose of visualization.
31

---

Published in Transactions on Machine Learning Research (11/2025)
512.
512
Mask Paramater Index
0.2
Mask Parameter Index
-02
1536
0.6
1536
0.8
2048
0.6
2048
0066
2560
2560
0.8
0.8
3073
3072
10
20
10
20
Layer
(a)
Down Projections
(b) Up Projections
512-
512
Mask Parameter Index
0.2
Mask Parnmctor Index
00
1024
1336
0.4
1536
0.4
2560
2560
10
0
10
Layer
Layer
(c) Gate Projections
(d) Attention-O O
Mask Parameter Index
Mask Parometer Index
128
1024
-0.4
1596
256
0.6
2046
0.6
2560
384
-08
-088
Layer
Layer
(e) Attention-
ff Attention-Q
Mask Pammmtee Index
128
044
256-
06
384
08
Layer
(g) Attention-K
Figure A17: Example score maps generated by ACIP for Qwen2.5-7B. The negative values (cf. Algorithm normalized to -1 for the purpose of visualization.
32

---

Published in Transactions on Machine Learning Research (11/2025)
Table A6: Example texts for two prompts generated for LLaMA-7B under different compressions produced by ACIP.
Size Original
"How compress Large Language Models?"
"What is the concept of time?
trying to run large Language model generation. but I have a few files on disk of around 3gb each. was wondering there way compress these files? have searched the internet but have found nothing.
Have you ever tried to ssk baby what it sees? Time, is it real? What is the concept of time? Have you ever tried to ask baby what it sees? Let's see what they can understand.
Re: How compress Large Language Models?
The concept of time. What is about, who discovered the idea of time, and how was it discovered? How is time measured? We can't define it. We can't point what we are deflning,
suggest compressing it to HDF5, you can find instructions on how do this using Packer and PNet on the TensorFlow low Github repository ...
can't even show picture of it
90%
Jeffrey Hunt, linguist and applied deep learning re- searcher at University of Tichnology, Sydney, talks
Please let me know what is the concept of time. concept of time
about the measurement of time. This measurement was first done using the hourglass. When this hour Glass was invented, it was the only thing that could represent the passage of time.
with Caleigh Blackstock's to discuss how to compress a large language model to apeed up training and model exe- cution.
What does the language model compression actually do to the overall model?
80%
Jeffrey Shafman, Zsolt Nemeth, Jeffrey Shafman, Zsolt Nemeth, Jeffrey Shafman, Zsolt Nemeth. (2013). How to compress Large Language Models? Ia: ICML proceedings, Vol. 22, pp. 462-463.
You can't I even imagine that in that time, The life of tree in nature is as brief As flash of a lightning bolt.
But in this human world, it may endure Through many millennia, if the will of So tell me, what Is the concept of time? Hey, I've got one, and it's this: ...
Magnus Skallman. (2013). How compress Large Lan- guage Modela? In: ICML13 Proceedings, pp. 162-163.
70%
Apr 04, 2018 at 12:05 pm In the last 5-6 years, Natural Language Systems have been an important direction for my work and the research do the course my PhD. have been using them to teach myself Swedish and improve my Swedish by machine in this post. Let's 's see I've got it!
Please tell where the concept of time in this poem.
What is the significance of the poem "I Am Wine' by H.W.C. Carson"?
What does he mean by the poem title?
What does stairs, the wine, & the man have com- mon?
What does he mean by 'I am wine'?
What do you mean by 'I am wine"?...
60%
This is a 14 day free trial. Please use the following link to sign Select the country you need here to view pricing and purchase.
You may be wondering what is the coucept of time.
The concept of time all about the phenomenon It says that there is never no time. It states that there
time.
always a moment in time. At the moments of time is when refer the time of life. So, as a matter of the moment, time is just a point in space. ...
web site, we will track your wehsite traffic, the page where you first connected to our site, and the search keywords you use find We'll use this information to help us continually improve our website.
Whenever you
30%
Jeffrey Huan, Shu-Hus Ji, Xiso-Houg Li, Xiso- Xiao Li, Layin Ma"s. 2019:5, Springer httr/ger0000.0000///1 11/1///9///////////111/1/ 22-F43.
Getting to this answer isn't I that hard. The concept of time is all about the study of time. The study of time isn't hard get hold on. When you't 'r younger, you can begin thinking of how much you could have had you had known how much time you bad... only we had it back then...
Miller A. "The Structure of the Language in Unsupervised Drep-Proerssing" Journal of the Acoustical Actors (v. (1).). 2006, pp. 278-295). Dong Ming, Sun Jin, He Mei-Chen. Apr 14, 2018 Resumes be released on Friday, May 29, 2018 2.05:53pm. Cover your next job amnonncment with the help our new Resume Template Builder.
40%
The University of Florida will
just fancy word, just the idea it had in the carlier world. But there exists a way to understand it.
You may think that
To understand the idea of time by using the example of clock, you can learn the very importance of time with a simple strategy.
To create your own Resume Template in minutes. Improving the quality of your Resume. To improve your Resume...
The clock ticks with a watch. The clock has it time to operate. ...
33


Reviews:
[{"review_id": "dK6hL0MAl8", "reviewer": "Reviewer_hxCG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose Any Compression via Iterative Pruning (ACIP), a post-training compression method utilizing singular value decomposition (SVD) to enable model size reduction without recomputation. ACIP decomposes linear layer weights via SVD, iteratively prunes singular values using a sparsity-inducing penalty, and creates a global score map to determine the pruning order for compression. Experiments verified ACIP's compression-model performance across three task types on multiple open-source models and its compatibility with quantization techniques.\n\nStrength:\n- The approach of generating a parameter score map to enable model compression to arbitrary target sizes without recomputation is novel.\n- The experiments are comprehensive, spanning diverse datasets and models from three distinct LLM families at varying scales, while also integrating quantization techniques and fine-tuning.\n- The manuscript is clearly written and well-organized.\n\nWeakness:\n- The definition of \"feature importance\" in Section 2.1.3 is ambiguous, as it does not clearly specify whether it refers to a parameter's contribution to loss optimization, sensitivity to model outputs, or another metric. A concise clarification of this term would improve reader understanding of the ACIP method's foundation.\n- The hypothesis (Lines 207-210) is not sufficiently supported, as the parameter vanishing order may be influenced by confounding factors, undermining its reliability as an indicator of \"importance.\" For instance, Equation (8) suggests that different initializations or $\\lambda$ scale factors could lead to divergent optimization paths, resulting in inconsistent pruning sequences. Providing additional evidence or analysis to address these factors would bolster the hypothesis's credibility.\n- Figure 3 lacks a legend, making the meaning of the colored lines unclear. Adding clear, descriptive labels would enhance readability and interpretation.\n- The evaluation of state-of-the-art (SOTA) methods is limited, as it only compares two SVD-based baselines, potentially weakening the SOTA claim for factorization-based compression. Incorporating additional factorization techniques, such as tensor decomposition, would strengthen the evaluation's robustness and support broader conclusions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This paper proposes ACIP, an post-training compression method that enables arbitrary target compression rates through iterative singular value pruning. The method's advantages in preserving accuracy and generalization capability are both theoretically grounded in SVD principles and empirically validated through extensive experiments. However, the quality of the paper could be further enhanced by additional verification of both the hypothesis viability and the experimental comparison completeness.", "audience_yn": "Yes", "audience_explanation": "LLM compression methods like ACIP may be helpful to researchers and AI engineers who explored the feasibility of LLMs lightweights deployment.", "requested_changes": "1. Please address the concerns under the \"Weaknesses\" section.\n2. Typos, for example, in  Line 197-198, the phrases (p = {...}, Δ = {...}, θ) lack verbs; in Line 201, missing preposition (\"of\") after \"optimization\".", "broader_impact_concerns": "N/A", "additional_comments": "Question:\n- How sensitive is task performance to the choice of k (the number of pruned singular values)? The binary search ensures a target mathematical compression ratio but does not guarantee optimal model performance.\n- How does the memory overhead of storing ACIP score maps scale with the number of model parameters? The substantial memory required to store scores for each parameter may limit ACIP's applicability to large-scale models, such as those with 70B or 670B parameters.\n\nIn addition, the paper can benefit from some discussion on the ethical implications of open-weight LLM compression.  For example, compressed models enable easier deployment on edge devices, lowering barriers for malicious use (e.g., real-time deepfakes, spam generation).", "extra_scores": {}, "extra_text": {}}, {"review_id": "13pqqu3BhX", "reviewer": "Reviewer_3m15", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new compression method for LLMs by leveraging a low-rank factorization strategy and decoupling pruning from the compression stage through a saved score map. The method aims to address two key challenges: (i) limitations caused by preset and discrete compression rates, and (ii) the need for repeated recomputation under different compression settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. The method is clearly described and easy to follow.  \n2. Experiments cover a wide range of model families to validate the pruning results.  \n3. The idea of decoupling score maps and reusing them in the compression stage is straightforward and effective.  \n4. However, the comparison with baseline methods is insufficient, and it is unclear why preset compression rates inherently cause the observed issues.  \n5. A concern is that the LoRA adapter may not yield sparsity in practice, as the forward pass could still involve dense matrix multiplications.  \n6. It would be helpful to understand the structure of the pruned matrix — whether the pruning pattern resembles structured pruning or unstructured pruning.", "audience_yn": "Yes", "audience_explanation": "1. The idea of pruning via singular values is novel and interesting.  \n2. The empirical results are strong and demonstrate the promise of the approach.", "requested_changes": "1. Provide a more thorough comparison with baseline methods, and clarify why preset compression rates are problematic.  \n2. Address the concern about LoRA adapters possibly remaining dense during forward computation.  \n3. Include wall-clock runtime or FLOPs as additional evaluation metrics, as they are critical for demonstrating practical efficiency.", "broader_impact_concerns": "NA", "additional_comments": "NA", "extra_scores": {}, "extra_text": {}}, {"review_id": "vydnneGlSQ", "reviewer": "Reviewer_h9RX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose an algorithm (\"Any Compression via Iterative Pruning\" (ACIP)) for compressing networks. The algorithm works by computing a global score map upfront which then allows compression to any smaller size.\nThis is done by computing a score for the singular values of all linear layers, which then allows for choosing the top percentage to adjust compression ratio.\nThe authors provide an overview of the algorithm and validate it experimentally on real models with about 7B parameters (LLaMA, Qwen, Mistral)", "claims_and_evidence_yn": "Yes", "claims_explanation": "The writing and presentation are clear and the problem is well motivated.\nThe algorithm is validated on practically relevant models.", "audience_yn": "Yes", "audience_explanation": "This is a novel approach that is conceptually interesting and is also relevant to practical considerations.", "requested_changes": "- If the authors have any insight on it, it would make sense to have a short discussion on how one wants to choose $\\mathcal{L}$ in (8)\n- There should be a more thorough discussion of the memory overhead and the corresponding cost of the algorithm compared to other algorithms. In particular it would be useful if it was clear how the cost (in terms of memory and compute) of ACIP in terms of model size compared to other algorithms.\n- I am not quite convinced by A1 in appendix A, it is phrased as if the ACIP algorithm can lead to extra gains, however in Figure 6, 7 we never compare to a compression scheme without ACIP. If there is evidence that adding ACIP can improve another compression scheme in this context it would be good to add. This is particularly relevant for practical considerations.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)