Adaptive Nonlinear Compression for Large Foundation Models
Authors: Liang Xu
Shufan Shen
Qingming Huang
Yao Zhu
Xiangyang Ji
Shuhui Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Model Compression
Low-Rank Factorization
Large Language Models
Vision Model
Primary area: optimization

Abstract: Despite achieving superior performance, large foundation models (LFMs) have substantial memory requirements, leading to a growing demand for model compression methods.
While low-rank approximation presents a promising hardware-friendly solution, existing linear methods suffer significant information losses due to rank truncation. Nonlinear kernels can enhance expressiveness by operating in higher-dimensional spaces, yet most kernels introduce prohibitive overhead and struggle to support adaptive rank allocation across heterogeneous matrices.
In this paper, we propose a compression method called Nonlinear Low-Rank Approximation with Adaptive Budget Allocation (NLA). 
Instead of relying on linear products, we employ piecewise-linear kernels with a forward-pass optimization operator to approximate weight matrices, enhancing the recovery of high-rank weight matrices from low-rank matrices.
Moreover, considering the heterogeneous representation abilities and dynamic sensitivities of different weight matrices, we adaptively allocate the compression ratio of each weight matrix during the re-training process by cubic sparsity scheduling.
Through evaluations on large language models and vision models across various datasets, NLA demonstrates superior performance while achieving a higher compression ratio compared to existing methods.

Published as a conference paper at ICLR 2026
ADAPTIVE NONLINEAR COMPRESSION FOR LARGE FOUNDATION MODELS
Liang Xu 12, Shufan Shen Qingming Huang Yao Zhu", Xiangyang Ji Shuhui Wang!
1 State Key Lab. of AI Safety, Institute of Computing Technology. Chinese Academy of Sciences
2 University of Chinese Academy of Sciences Peng Cheng Laboratory
Zhejiang University Tsinghua University
(xuliang23) shenshufan22z, wangshuhuil aecictta..... ac. cn gmhuang@ucas.ast ac. en
ee.zhuy0z edu.cn
xyjie peiiittsnnhuaaa edu.
ABSTRACT
Despite achieving superior performance, large foundation models (LFMs) have sub- stantial memory requirements, leading growing demand for model compression methods. While low-rank approximation presents a promising hardware-frice -riearrr solution, existing linear methods suffer significant information losses due to rank truncation. Nonlinear kernels can enhance expressiveness by operating in higher- dimensional spaces, yet most kernels introduce prohibitive overhead and struggle to support adaptive rank allocation across heterogeneous matrices. In this paper, we propose a compression method called Nonlinear Low-Rank Approximation with Adaptive Budget Allocation (NLA). Instead of relying on linear products, we employ piecewise-linear kernels with a forward-pass optimization operator to approximate weight matrices, enhancing the recovery of high-rank weight matrices from low-rank matrices. Moreover, considering the heterogeneous representation abilities and dynamic sensitivities of different weight matrices, adaptively al- locate the compression ratio of each weight matrix during the re-training process by cubic sparsity scheduling. Through evaluations large language models and vision models across various datasets, NLA demonstrates superior performance while achieving higher compression ratio compared to existing methods. Our codes will be released in lttpt::: https://gitthnn //tth/b.ttt/tt//// com/ Saamm/Liaggg//NA
INTRODUCTION
Large Foundation Models (LFMs) (Tou-
Lineur Approsimations
vron et al.) 2023a by OpenAl 2023)
a-
have demonstrated remarkable perfor- mance across various tasks. Despite their impressive capabilities, LFMs possess an excessive number of pre-trained parame- ters that require high memory usage for inference. This substantial memory us- age severely limits the deployment of such scenarios. To this end, researchers have proposed var- ious methods to reduce the parameter vol- ume LFMs, which can be categorized into distillation-based and compression- based strategies. The distillation-baase strategy (Magister al.) (023) trains smaller model to emulate the behavior of the original model. thereby redueing re-
Lowerdinessioall spare
W
Non-linear Low-rank Approximations
k(
(D)
(a)
models in essource-constriingg
Hizka-dinnan dinka-linnnsonaal
feature
Figure Existing low-rank compression methods (top) rely linear approximations, resulting in significant information loss. In contrast, NLA approximates the weight matrix non-linearly (bottom), achieving less in- formation loss with the same number of parameters.
source requirements by only deploying this small model. However, retraining such small model
"Corresponding author.

---

Published as a conference paper at ICLR 2026
from scratch entails substantial data requirements and high computational costs. In contrast, the compression- srppeessioo-bbeeddee strategy directly reduces the scale of the original model in terms of parameter size or memory by pruning weights (Xia et al., 2023) quantizing (Park et al. 2022) them to lower precision. Despite reducing the resource requirements with minimal retraining costs, these methods impose higher demands specialized hardware for model deployment, such GPUs that support
unstructured sparse inference (Han et al., 2016) and low-bit precision computations (Jouppi et al. 2017).
Recently, researchers have proposed low-rank approximation methods (Wang et al. 2024 Yuan ct al. 2023] |aha et al.) 2023) that compress the model by representing the large pre-trained weight matnx using small low-rank matrices. As these low-rank matrices remain compact while preserving the precision of the original matrix, this line of methods avoids the dependence on specialized hardware. Existing methods typically rely singular value decomposition (SVD) to perform low- rank approximation. First, they decompose the pre-trained weight matrix into linear combination of singular vectors weighted by their corresponding singular values. Second, model size is reduced by truncating these singular vectors associated with smaller singular values. Finally, the model performance is restored through lightweight re- traininng of these resulting truncated matrices.
However, the truncation of singular vectors inevitably discards many singular directions. As the rank decreases, more informative components removed from the approximated matrix, leading to severe information loss and performance degradation under high compression ratios. An effective way to mitigate this issue is replace the linear inner product with nonlinear kernel functions. By implicitly mapping the low-rank matrices into a higher-dimensional Hilbert space, kernel functions allow the approximation to operate in a richer feature space and thus enhance the expressiveness of low-rank matrices. However, directly applying nonlinear functions poses critical challenges: most common kernels require reconstructing the full weight matrix during forward propagation, which significantly increases the computational cost and memory consumption. Moreover, dynamic rank allocation significantly improves performance after compression, as different matrices exhibit heterogeneous sensitivity. Yet, kernel-based formulations make it difficult to directly adopt such adaptive rank adjustment strategies during the re- ce-training process.
In this paper, propose Nonlinear Low-rank approximation with Adaptive budget allocation (NLA), a low-rank approximation method that compresses LFMs non-linearly with an adaptive compression ratio allocation mechanism. To make nonlinear low-rank approximation practical, we design -inecars--inerrrrre piecewise linear forward propagation operator that directly operates on the low-rank factors through self-term and interaction-term decomposition, thus avoiding explicit reconstruction of the full weight matrix. This design preserves the expressive power of nonlinear kernels while significantly reducing their computational and memory overhead. Considering the heterogeneous representation abilities and the dynamic sensitivities of different parameters during retraining, we propose an end-to-end method to adaptively adjust the compression ratio of each weight matrix. By calculating the importance scores of the parameters, we adjust the number of active pieces in the low-rank matrices, and the overall compression ratio is determined by the number of pieces preserved. Across different layers, the allocation is guided by their importance scores, with more sensitive matrices being assigned larger number of pieces. The budget allocation is controlled by a cubic sparsity scheduling (Sanh) et al.) 2020) strategy, which starts conservatively to preserve model capacity and becomes more aggressive to achieve higher compression ratios.
In experiments, we validate the effectiveness of NLA on both pre-trained Large Language Mod- els (LLMs) [Touvron et al., 2023a) and vision models (Liu et al. (02T). On LLMs, NLA consistently outperforms of methods (Hsu et al. 2022 Yuan al. 2023 Wang al. 2024). For example, on LLaMA-7B (Touvron et al, (023a). NLA achieves a perplexity of 15.21 on Wiki under 50% compression ratio, compared with 23.97 for SVD-LLM (Wang et al. 2024 After fine- tuning, NLA further reduces perplexity to 9.68 (13.26 for SVD-LLM). On common sense reasoning benchmarks, NLA achieves a relative improvement of about 12% over the SOTA method (0.37 vs. 0.33). On Swin-Base (Liu et al. 202I), 1, NLA achieves higher accuracy (82.7% vs. 82.4%) while compressing the model significantly greater extent (54.2% vs. 29.2%) compared the current SOTA method. These results demonstrate that NLA achieves higher compression ratios with nearly lossless accuracy, offering a flexible and effective solution for large-scale model compression.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Model Compression. With the rapid growth in model parameter size, variety of compression methods have emerged to address the challenges of high memory demands for LFMs (Zhu et al.
2024). These methods can be broadly categorized into four categories: quantization (Park et al.
2022 Yao et al. 2022 Dettmers et al., 2022), pruning (Frantar & Alistarh, 2023] Ma et al. 2023).
knowledge distillation (Hang et al. [022]] Jiang al.) 2023), and low-rank approximation (uan
et al. 2023, Saha et al. 2023). Despite demonstrating effective performance in model compression,
existing methods have several limitations. Distillation methods typically require large amount of data for training, while pruning and quantization methods often impose high hardware requirements. Additionally, these methods face significant performance degradation when applied in extreme compression settings. Among existing studies, low-rank approximation methods stand out for their low dependence on specialized hardware and their ability to combine with other compression methods. Our proposed method also belongs to this category.
Low-rank Approximation. Low-rank approximation, method that employs small low-rank matrices trainable parameters in place of large weight matrices, is widely used to reduce memory usage during fine-tuning (Hu et al. 2021] Valipour al.] 2022] Shen et al., 2024). Considering its advantage in memory efficiency, the low-rank approximation has been employed for model compression (Wen et al. 2017 Khodak et al. 2021 Li et al. 2023) to reduce memory usage during inference. PELA (Guo et al. 2024) combines low-rank compression and fine-tuning by applying feature distillation and weight perturbation in compressed models. ASVD (Yuan et al. (023) scales the weight matrix using a matrix that represents the influence of the input channels. In FWSVD (Hsu et al. 2022), Fisher information is introduced during weighted SVD to measure the importance of parameters model predictions, ensuring that the compressed model achieves accuracy closer the original model. These methods directly truncate the matrix based on the magnitude of the singular values, which can lead performance degradation. To address this, SVD-LLM (Wang et al.) 2024), which establishes direct mapping between singular values and compression loss, exhibits less performance degradation and is better at recovering performance through fine-tuning. However, these methods remain limited to linear approaches for low-rank compression. These linear approaches are accompanied by large and irreducible information loss between high-rank weight matrices and the low-rank matrices during compression. Beyond standalone quantization or low-rank approaches, several recent works explore hybrid compression strategies that jointly leverage both techniques. One such method is CALDERA (Saha et al. 2024), compression algorithm for LLMs that exploits the inherent low-rank structure by approximating the weight matrix W via a low-rank, low-precision decomposition. Different from these linear approaches, our proposed NLA leverages non-linear methods to achieve less information loss with larger compression ratios.
Nonlinear Matrix Representation Methods. Nonlinear methods have been widely applied in the field of machine learning. KPCA(Kernel Principal Component Analysis, (Schölkopf et al. (1998) uses kernel functions map the data into sigh-dimessional feature space to capture the nonlinear structures in the data. Dynamics-inspired Neuromorphic Learning (DyN) (Pei & Wang 2023) and its subsequent work (Pei et 2024) a new paradigm that uses neural-weight-free architecture consisting of finite sub-models instead of the weight-based model structure inspired by the theory of Hamilton dynamics. DyN interprets FC layers as integrals of the neuronal path, demonstrating comparable or even better performance than weight-based neural models with fewer parameters. Similar ideas have been applied various domains, such as fine-tuning (Shen et al. 2024) and graph neural networks (Sun al. 2024). Inspired by these methods, we treat the piecewise- linear model as a low-rank approximator for pre-trained weight matrices, achieving a high compression ratio and low-perfor www-pefforaace degradation simultancously. By designing an optimized forward propagation algorithm and introducing adaptive strategies, we facilitate a more comprehensive application of the nonlinear method model compression.
3 METHOD
3.1 PRELIMINARIES
Low-Rank Approximation. The low-rank approximation methods YYuan et al. 2023 Wang et al.) 2024) have emerged as promising technique for model compression, which approximates the
3

---

Published as a conference paper at ICLR 2026
Calibration Data
W R
Importance Scoring
Misk
W' (A,
Ranking & Top BC) Selection
Updati
Á
Rd
4(PL)
(MV-Rull Roll
Malced Panaza
(CA.)
Page
Cabic Budget Scholuling
Kerne-based Nonlinear Approxination
Adaptive Compression
Figure 2: Overview our proposed NLA framework. (Left) Kernel-based Nonlinear Approximation: approximate a pre-trained weight matrix W with two low-rank factors A, B and shared coefficients through piecewise-ling linear kernel function kpw- (Right) Adaptive Compression: compute the important scores of the triplets, rank the scores under cubic budget scheduling strategy, and mask the least important triplets while updating the others.
pre-trained weight matrix W E mR with the product of smaller low-rank matrices. Specifically, they first apply singular value decomposition (SVD) to decompose the weight matrix, then compress W to W, by retaining only the top singular values and their corresponding singular vectors.
UEVT, W, = U,E,V'
(I)
where diag (olag(o1..... (o..... miin(m..)) with singular values 01 > 02 ... min(m.n)- Despite being easy to implement, SVD-based linear truncation inevitably leads significant information loss, as more informative components are discarded with increasing compression ratios.
Kernel Function. A kernel function k(::)) .) implicitly maps data from input space X' to high- dimensional Hilbert feature space H via a mapping function & + H. The inner product H can be computed efficiently in the original space X' by:
4(x, x (<(x), o(x')) )xx
This capability of kernel methods, operating in high- or even anfinit-ddimnniionn Reproducing Kernel Hilbert Spaces (RKHS), allows them to represent highly nonlinear functions that go beyond simple linear hypothesis classes (Hofmann et al., (208),, thereby providing powerful tool of modeling complex relationships in high-dimensional spaces by leveraging low-dimensional vectors.
3.2 KERNEL-BASED NONLINEAR APPROXIMATION
To capture richer interactions beyond linear low-rank structure, we choose piecewise-linear kernel inspired by Pei & Wang (023),, which introduces nonlinearity while remaining computational lightweight. Given a weight matrix W IRm mm we approximate with a matrix W' represented by two smaller matrices A muxhxr and B E TRaxhxr with piecewise linear kernel kpas
W = kpw (A BJ.,)
Σ M ||A Bya,-12
where W denotes the (i, j)-t element of W', To minimize the difference between the approximation matrix W and the target weight matrix W, we use the gradient descent optimization method to adjust the distribution of -imeenensional vectors. The optimization objective is
W arg min EMSE(W' W).
A.B
Through the optimization, we can compress a pre- weight matrix containing m x n parameters into low-rank matrices containing (m + n) r h + h parameters in non-linear manner. The introduction nonlinearity largely reduces the information loss during compression.
4

---

Published as a conference paper at ICLR 2026
However, nonlinearity is not free lunch. During the forward propagation, introducing nonlinearity necessitates merging the low- rank matrices into a large-weight matrix be- fore processing input features. Unlike the linear approach, one cannot directly multi- ply the features by the low-rank matrices, which leads to significant memory consump- tion during training and inference. To this end, we propose a specifically designed for- ward algorithm [ This algorithm enables forward propagation for nonlinearly com- pressed low-rank models without the need to merge weight matrices, significantly reduc- ing memory usage during training and infer- ence. Algorithm kernel approximation in a factorized manner by decomposing the kernel into three com-
Algorithm 1 Forward with Nonlinear Kernels
Require: Input features X Rbxm input neurons Qin Rmmxxx,, output neurons Qun E Rexbxr,
shared coefficients
R²
Compute input & output self-term:
InS - Σ k== Qmli i, k) @
OutS + Ek=1 Qi k)
Aggregate input & output contribution:
Xin +- -- IIS'
Xour t 1,
x OutS
Compute interaction term:
Xin&out (Quu (Q -0 ()) 0
Xin + - XinRou!! [
1 computes the nonlinear
Final output:
Y + Xin + Xout - Xin&cott
ponents: the imput self-term, the output self-term and the interaction term. Specifically, the piecewise linear kernel is expanded into its additive parts, and each component computed using only the nonlinear factors and the shared coefficients without ever constructing the full weight matrix. The detailed mathematical proofs and algorithmic resource consumption are provided in Appendix A
3.3 ADAPTIVE COMPRESSION
Through the above method, can compress the weight matrix into low- rank matrices. However, compressing all weight matrices to the same rank not the optimal strategy. This is because compressing different matri- ces has varying impacts on model per- formance, and the difficulty of com- pression also differs between matrices. As shown in Fig. 3 the final mean loss and the rate of loss change dur- ing approximation vary significantly between different layers and matrices, such the Q and FC matrices. This
Q
FC Matrix
Lewn-27 22
Menn
Stepss
Figure 3: The loss value during nonlinear approximation of different weight matrices. Experiments are conducted with the matrix and FC matrix of different layers in OPT-1.3B. highlights that compression difficulty is not uniform across all matrices.
To achieve higher compression rate while preventing significant performance degradation, design an adaptive compression method for the nonlinear low-rank approximation. Inspired by (Zhang al. 2023), adaptively adjust the compression ratio for each weight matrix based on its importance score.
For weight matrix W in MLP or Attention layers, our nonlinear formulation decomposes into three low-rank components due to Section 3.2 A E mRxx B E Raaxxx and shared coefficient vector μ E h This naturally defines segment-level triplets
7 (A: (a.,,., E [m, /4, Bi = Csa.. [1, , h1
(5)
This structure provides a convenient granularity for computing importance scores and adaptively allocate budgets of each matrices.
To adaptively compress the matrix, we assign an importance score sw.r to each triplet based on its contribution to the model. This score is weighted sum of the importance s(.) of the shared coefficient hi and the average importance of its associated low-rank matrices Ai and B,
sw. = s(w)) m X
s(A..s)
s (Br.ss)
(6)
n X
==
5

---

Published as a conference paper at ICLR 2026
To make the importance scores more robust, an individual parameter w is defined as the absolute product of its value and the gradient: I(w) = |w- -. To account for the dynamic nature of the training process, compute smoothed importance score and a corresponding uncertainty measure U over time step:
cenitivittybbsse approach. The importance of
(w) =3, -3, (w) + (1 - 81) (w)
t (w) -33 + - 32) ( (w))
Here, 31 and 82 control the exponential smoothing strength. The final importance score s(w) for any parameter is then defined as the product of the smoothed importance and its uncertainty:
((x) T(w) (w)
(8)
During training, we dynamically adjust the overall compression budget to preserve performance. This crucial because aggressively compressing the model too early destabilize the training process. We use cubic sparsity scheduling method (Sanh et al. 2020) to gradually reduce the number of active parameters. The budget b(l) at given step / is defined as:
bn
t<t,
b(1)
by (bo b,) (1-1-111 T-
t, < < <
by
T T <<<
Here, by is the initial budget and by is the final budget. The warm-up and cool-down steps, t; and ty, can ensure a smooth transition.
By combining the importance scores with the dynamic budget, mask the less important shared coefficients μ4 and thus effectively adjust the compression ratio for each matrix. The mask is applicd during the update step:
14 V
Swi is the top of all scores in S else
i
(10)
0
where the set of importance scores for shared coefficients, and b is the current model budget. This process allows us to allocate the parameter budget while minimizing the impact on performance degradation for model compression.
EXPERIMENTS
Baselines. For LLM, we compare our method against four baselines, including SVD and state-of of-thh- art linear low-rank approximation methods FWSVD (Hsu et al.) 2022), ASVD (Yuan al. 2023). and SVD-LLM (Wang et al. 2024). For vision models, we compared our method with Tiny Jiao
et al. 2020), MaskAlign (Xue et al. 2023), and PELA (Guo et al. 2024).
Datasets and base models. To demonstrate the effectiveness our method, we evaluate the per- formance of NLA and the basclines in several models and datasets. For LLM, select the models LLaMA (Touvron et al., 2023a) and OPT (Zhang et al., 2022). The models are estimated in ten differ- datasets, including language modeling datasets (Wikitext2 (Merity al. 2017). PTB (Mareus) ct al. 1993), C4 (Raffel et al. 20200) and common sense reasoning datasets (Openbook()) QA (Mi- haylov al. 2018), ARC-e and ARC-c (Clark al. 2018). WinoGrande (Sakaguchi al. 202 HellaSwag (Zellers et al. 2019) PIQA Biikk et al. 2020), and MathQA (Amini et al. 2019) )) For
vision models, we test on wwin-Transforme (Liu et al.) [202T) with the ImageNetik (Deng al.)
2009) dataset. All of our experiments are conducted on 2 NVIDIA A100 GPUs. The hyperparameter
details of the experiments are provided Appendix C
4.1 MAIN EXPERIMENTAL RESULTS
Experiments LLMs. To thoroughly compare the performance of NLA with the baselines,
perform three sets of experiments with LLaMA-7b (Touvron et al., 2023a): (i) language modeling tasks, (ii) common sense reasoning tasks, and (iii) the performance of compressed models after

---

Published as a conference paper at ICLR 2026
Table The perplexity of LLaMA-7b compressed by NLA and baselines under different compression ratios on Wikitext2 (Merity et al. 2017). PTB (Mareus et al. 1993) and C4 (Raffel al. 2020
Table 2: Perplexity of LLaMA-7b com- pressed by NLA and SVD-LLM (Wang et al. 2024) under different compression ratios on Wikitex12 (Merity et al. 2017) before and after fine-tuning.
METHOD
RATIO
WikiText-2
PTB
C4
Original
0%
5.68
8.35
7.34
METHOD
RATIO
WikiText-2
40%
52489
59977
47774
Original
0%
5.68
SVD
50%
131715
87227
79815
Performance Before Fine-tuning
105474
79905
106974
40%
18156
20990
12847
SVD-LLM
50%
23.97
FWSVD
50%
24391
28321
23104
NLA (ours)
50%
15.21
60%
32194
43931
29292
SVD-LLM
70% 70%
185
1407
3290
1109
NLA (ours)
39.91
ASVD
15358
27925
Performance After Fine-tuning
57057
45218
43036
50%
23.97
150.58
118,57
SVD-LLM
50%
13.26
SVD-LLM
60%
42.30
321.27
246.89
NLA (ours)
50%
9.68
50%
15.21
84.65
70.32
SVD-LLM 70%
28.45
NLA (Ours)
70%
39.91
320.35
240.56
NLA (ours)
70%
17.77
Table 3:
Performances of I-LaMA7bb compressed by NI.A and baselines under different compression ratio common reasoning datasets measured accuracy. Compression ratio is calculated as the ratio of the number of reduced parameters to the number of parameters in the original model.
METHOD
RATIO
Openb.
ARC_e
WinoG.
HellaS.
ARC.c
PIQA
MathQA
Average
Original
0%
0.28
0.67
0.67
0.56
0.38
0.78
0.27
0.52
40%
0.15
0.26
0,51
0.26
0.21
0.54
0.22
0,30
SVD
50%
0.16
0.26
0.50
0.26
0.23
0.52
0.19
0.30
60%
0.16
0.26
0.50
0.26
0.22
0.52
0.21
0.30
40%
0.16
0.26
0.51
0,26
0.22
0.53
0.21
0.30
FWSVD
50%
0.12
0.26
0.50
0,26
0,23
0.53
0.20
0,30
60%
0.15
0.26
0.49
0,26
0,22
0.53
0.18
0.30
40%
0.13
0.28
0.48
0.26
0.22
0.55
0.19
0.30
ASVD
50%
0.26
0.51
0.26
0.22
0.52
0.19
0.30
60%
0.12
0.26
0.49
0.26
0.21
0.51
0.18
0..99
50%
0.16
0.33
0.54
0.29
0.23
0.56
0.21
0.33
SVD-LLM
60%
0.14
0.28
0.50
0.27
0.22
0.55
0.21
0.31
50%
0.19
0.43
0.58
0.33
0.26
0.60
0.21
0.37
NLA (ours)
70%
0.16
0.33
0.50
0.28
0.23
0.57
0.22
0.33
fine-tuning. The compression ratio is defined as the fraction of parameters removed relative to the total number of original parameters.
We first evaluate the perplexity of LLaMA-BB compressed by NLA and baseline models on three different datasets, Wikitex12, PTB, and C4. The results in Table | show that NLA consistently achieves lower perplexity than the baselines at the same compression ratio. For instance, at a 50% compression ratio, NLA reduces perplexity to 15.21 on Wikitext2, which is significantly better than SVD-LLMM (37 and far below all other baselines. More importantly, even under more aggressive compression (e-g.. 70%). NLA still maintains competitive performance: its perplexity (39.91) substantially lower than SVD-LLM at the same ratio (185) and even outperforms SVD-LLM at 60% compression (42.30). Similar trends are also observed on PTB and C4, These results indicate that NLA not only achieves superior performance comparable compression ratios but also retains robustness when pushed to higher compression.
To further evaluate the effectiveness of NLA, test the performance of LLaMA-7b compressed by NLA and baseline methods on seven common sense reasoning datasets. The results are summarized in Table 3 As shown, NLA achieves the best overall performance 50% compression ratio,
7

---

Published as a conference paper at ICLR 2026
with average accuracy of 0.37, surpassing the best-perforning baseline (SVD-LLM 50%, 0.33) by 0.04. Even at a higher compression ratio of 70%, NLA maintains competitive accuracy (0.33 average), comparable SVD-LLM 60% while reducing demonstrate that NIAA not only achieves superior performance under moderate compression but also sustains competitive accuracy under aggressive compression, highlighting its robustness and effectiveness for commonsense reasoning tasks.
parameters. These results
To estimate the performance NLA with fine-tuning, fine-tune LLaMA-7B compressed by NLA, ASVD, and SVD-LLM on Wikitext-2. As shown in Table 2 before and after fine-tuning. Specifically, on the WikiText-2 dataset, the perplexity of NLA is more than 70% lower than that of SVD-LLM with the same compression ratio before pre- pre-training, and 37% lower after pre-training. Furthermore, fine-tuning the NI.A-compressed model results in 55% reduction in perplexity, indicating that the nonlinear low-rank approximation not only compresses the model effectively but also preserves its capacity to improve performance through fine-tuning.
NLA consistently surpasses SVD-LLM
In summary, both pre- and post-fine-tuning results consistently verify that NLA delivers a more efficient and effective compression strategy than existing SVD-based methods. It achieves better performance at the same compression ratio, and even under higher compression ratios, NLA remains competitive with, or superior to, the best baselines. This shows the effectiveness and robustness of NLA for compressing large-scale language models.
Table 4: Image classification performance of the compressed model on ImageNet- I
METHOD
Params (M)
Ratio
GFL.Pss
Throughput
Acc (%)
Swin-Base
87.8
0%
33.7
293.73
83.5
TeryBert
58.6
33.3%
20.6
78.8
MaskAlign
58.6
33.3%
20.6
79.1
PELA
62.2
29.2%
21.3
295.96
82.4
NLA (ours)
40.2
54.2%
21.9
231.68
82.7
Experiments vision models. We apply our method to the Swin-Transformes After compressing the Swin-Baie Base model with compression ratio of approximately 54%, we fine-tune it on the ImageNet- Ik dataset and test on the corresponding test set. The results are reported in Table 4 As shown, NLA achieves higher accuracy than other baselines after fine-tuning, while reducing the parameter count more substantially. In particular, compared to PELA, NLA reduces the number of parameters by 35.3% while delivering better top-1 accuracy. Moreover, NLA retains performance close to the original model (82.7% vs. 83.5%) despite a reduction of more than half of the parameters.
The experiments above validate the advantages of NLA over existing low-rank approximation techniques in compressing both vision and lan- guage models. The results emphasize the flex- ibility of NLA, which adapts effectively var- ious types of models, making it a robust choice for wide range of real-world applications. Furthermore, NLA's ability to retain and even enhance performance after compression and fine-tuning positions it as promising solution for model compression tasks, particularly for large-scale models.
Table 5: Perplexity NLA and SVD-based methods Wikitext-2, we only convert the lay- ers of LLaMA-7B without further training
METHOD
RATIO
WIKITEXT-2
SVD
60%
105474
FWSVD
60%
32194
ASVD
60%
$7057
NLA (LOW-RANK ONLY)
70%
16774
Table 6: Mean loss during the fitting of weight matrices of OPT-1.3B using SVD and nonlinear low-rank approximation.
4.2 ABLATION STUDY
LAYER
NAME
SVD Loss
NLA Loss
Ability of linear vs. nonlinear low-rank ap- proximation. We conduct an experiment to verify the differences in the ability of linear low- rank approximation and nonlinear low-rank ap- proximation to fit the weight matrices mod- els. We first compare SVD-based methods and NLA LLaMA-7B without further training. As
0,0118
0.0053
0,0119
0.0052
FC1
0.0177
0.0106
5
0.0250
0.0074
8

---

Published as a conference paper at ICLR 2026
Table 7: Memory usage during training and inference of Table 8: Ablation of adaptive budget different models with without the adoption of Algorithm
allocation with ImageNet- IK and OPT on WikiTex12
on
in NLA. The batch size used for testing is 1.
MODEL
Params (M)
With Alg.
Inference (MB)
Training (MB)
MODEL
Adaptive
Acc (8)
Perplexity
Swin-Tiny
9.3
/
X
105.76
190.01
1053.47
155.47
Swin-Tiny
x
77.63
77.40
Swin Base
40.2
/
X
430.58
273.64
3683.84
456.21
Swin-Base
82,55
82.73
OPT-125m
73.5
<
x 3440,70
374.10
8857.18
1562.14
OPT-125M
31.52
31.34
OPT-1.3B
461.7
X
18053.07
2204.22
12623.75
>-8920
OPT-L.B
20.13
19.77
Table 9: Ablation on kernel functions. We report aver- age inference runtime (s) and peak GPU memory (GB) measured on OPT-1.3B and LLaMA-7B
Table 10: Perplexity of LLaMA-7B com-
pressed by GPTQ combined with SVD- LLM and NLA.
OPT-1338
LLaMA-7B
Method
Memory (GB)
Perplexity
Kernel
Time (s)
Mem (GB)
Time (8)
Mem (GB)
4-bit
3.9
6.21
0.45
2.7
3-bit
2.8
16.28
Sigmoid
0.37
2,4
1.32
6,3
SVD-LLM 4-bit
2.1
13.29
Cosine
0.31
2.3
1.26
Pieco-wisc Lincar (ours)
0.25
2.8
0.82
NLA + 4-bit
1.9
12.18
shown in Table 5 NLA achieves perplexity of 16,774 at 70% compression ratio, which is over 70% lower than the best SVD-based result (32, 194 with FWSVD at 60%). To further evaluate the performance of SVD and NLA in fitting weight matrices, model that compares the mean loss for each layer using both SVD and NLA. As shown in Table 6 for different weight matrices in various layers, approximating with NLA gains much less mean loss than with SVD. These results demonstrate that nonlinear approximation (NLA) consistently achieves lower mean losses compared to SVD. highlighting its superior ability better fit the model's weight matrices.
conduct experiments
the OPT- 1.3B
we adopt the forward algorithm in Algo- rithm to reduce memory usage during model execution. To verify its effectiveness, we measure the memory footprint of the compressed models on both vision and language tasks, considering inference (without gradients) and training (with backpropagation) under a batch size of 1. The test results are shown in Table 7 For OPT-1.3B, the proposed algorithm reduces memory consumption to 12.2% of the baseline during training and 15, 15.4% during inference. Similar reductions are observed for other models. Overall, these results demonstrate that the proposed memory-saving algorithm substantially decreases memory usage in both training and inference across different architectures.
Memory-saving algorithm. As discussed in Section 3.2
Ablation on Kernel Functions. To assess the effect of different kernel choices in our nonlinear low-rank approximation, we compare the proposed piecewise- linear kernel with alternative options, including Gaussian, Sigmoid, and Cosine kernels.
the inference costs in terms of runtime (s) and peak GPU memory (GB). We observe that all kernels consume similar amount of GPU memory during inference (2.3-2.8 GB & 6.2-6.7 GB), while the Picce-wise Linear kernel achieves the fastest inference speed both models. These results suggest that although different ker- nels provide comparable memory footprints, their computational efficiency varies significantly. The piecewise-linens kernel consistently yields the most favorable balance between accuracy. efficiency, and memory usage, which justifies our choice to adopt it as the default kernel in NLA.
Ablation of adaptive budget allocation. We compare NLA with and without the adaptive budget allocation module on both vision and language models in Table g For vision tasks, we evaluate Swin-Tiny and Swin-Base; incorporating adaptive budget allocation yields higher top-1 accuracy. For language tasks, test OPT-125M and OPT-1.3B WikiText2, where adaptive allocation consistently lowers perplexity. Across both model families, the addition of adaptive budget allocation brings measurable performance improvements, confirming its effectiveness.
Computational Complexity and Throughput. We compare the computational complexity and throughput of nonlinear approximation methods and linear methods the vision model and language model. As shown in Table 4 with comparable computations and throughput, our NLA achieves superior performance with fewer parameters Swin-Base Base, demonstrating that the integration of

---

Published as a conference paper at ICLR 2026
non-linear techniques can achieve more favorable trade-off between performance and resource requirements. For language models, NLA introduces a slightly longer inference time but reduces GPU memory usage by 35%, while achieving superior performance (17.77 17.93) compared to linear SVD baselines, as shown in Table T3 This significant reduction in memory makes NI.A particularly suitable for deploying large models on memory-constrainge devices.
Quantization and Low-Rank Combination. To demonstrate the compatibility of NLA with other compression methods, tested the perplexity of LLaMA-7B on Wikitext-2 when compressed by GPTQ (Frantar et al. 2022). combining with two low-rank approximation methods. As shown in Table [0] NLA combined with 4-bit quantization attains the lowest memory footprint (1.9 GB) while delivering the best perplexity (12.18). outperforming both GPTQ-3bit and SVD-LLM combined with GPTQ-4bit. This result shows the strong compatibility between NI.A and quantization methods.
CONCLUSION
In this paper, we present NLA, a nonlinear low-rank approximation method with adaptive budget allocation. By replacing linear products with sicewwse-linear kernel and dynamically allocating compression budgets across layers, NLA achieves higher compression ratios while preserving most of the original ability within the pre-trained model. For memory-efficient inference with nonlinear methods, we design a special forward algorithm that enables inference and training without re- computing full weight matrices. Across multiple baselines on vision and language models. NLA demonstrates more favorable compression-pert -atrressiio-efffmmnnnee performance trade-offs than linear low-rank methods, especially under high compression ratios, and ablations confirm the benefits of adaptive allocation and the choice of kernel. In future work, we aim to extend NLA to multi-mode tasks and investigate strategies to further reduce its computational and temporal costs.
ETHICS STATEMENT
We assert that the methodology and applications of this study do not involve human subjects, generate harmful insights, or engage in applications that could lead to discrimination, bias, social harm. All datasets used in this research are publicly available benchmark datasets widely used in academic studies. We commit adhering all applicable data privacy and intellectual property laws and regulations. We confirm that we have read and understood the ICLR Code of Ethics and pledge to comply with its guidelines in all aspects of our research, paper submission, and subsequent interactions.
ACKNOWLEDGEMENT
This work was supported in part by the National Natural Science Foundation of China under grant number 62236008, in part by the Natural Science Foundation of Beijing under grant number L,51 1082, and in part by Shandong Provincial Natural Science Foundation under project 7R20257D01.
REFERENCES
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Kenzel-Kedziorsii, Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation -baed formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 2357-2367, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: (0.18653/w1/N19) 1245.
Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY. USA, February 7-12, 2020, pp. 1432-7433 AAAI Press, 2020. URL https://eaat .aceaai.rrg//oo www.oog//////oooooooo/oooooo...o ndex php/ AAAT
www 6239
10

---

Published as a conference paper at ICLR 2026
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv preprint. abs/ abs/1803.05457, 2018. hRRLhhtt:://aaat org abs 8803.8003... 1803.05457 05457
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Putlern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248-255 IEEE URL http::::/doo..og////////
Computer Society, 2009, doi: 10. ww..109/CVPR.200.5500000
1109/ CVP.C 2009. 5206848
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 Q): 8-bit matrix multiplication for transformers at scale, Advances in Neural Information Processing Systems, 35: 30318-30332, 2022.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323 (0323-10337. PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training paanerattvepe--trinnne https://arxiv ott/oogg///a/ooooooo http://b///22200
quantization for generative pre-traimed transformers. ArXiv preprint, abs/2210.17323,, 2022. URL
Yangyang Guo, Guangzhi Wang, and Mohan Kankanhalli. Pela: Learning parameter-efficient models with low-rank approximation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15699-15709, 2024.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William J Dally. Fie: Efficient inference engine on compressed deep neural network. ACM SIGARCH Computer Architecture News, 44(3):243-255 2016.
Thomas Hofmann, Bernhard Schölkopf, and Alexander Smola. Kernel methods machine learning. 2008.
Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. In International Conference on Learning Representations, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ArXiv preprint, 014/21060096855 2021. URL [ttEp::///axxiii/ /tA.pp///aaaii.oorr///////.....
abtp///s..22.e.e/ 09685
Yukun Huang, Yanda Chen, Zhou Yu, and Kathleen McKeown. In-context learning distillation: Trans- ferring few-shot learning ability of pre-trained language models. ArXiv preprint, abs/2212.10670, 2022. URL BtLhhtpss://
10670
Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. Lion: Adversarial distillation of proprictary large language models. ArXiv preprint, abs/2305.12870, 2023. URL https://
arxiv.org/ abs 2205
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for natural language understanding. In Findings of the As-
sociation for Computational Linguistics: EMNLP 2020, pp. 4163-4174, Online, 2020. As- sociation for Computational Linguistics. doi: 12.18653/v1/20200 htt.:///2220iiiinneeeeeeee findinggs-emll..3777 URL findings am.lp.
https: httpp::/aclamtbooooyyy.oo aclantholog..org//
Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre lue Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke. Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt
11

---

Published as a conference paper at ICLR 2026
Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang. Eric Wilcox, and Doe Hyun Yoon. In-datacenter performance analysis of a tensor processing unit. ArXiv preprint, abs/1704.04760, 2017. URL
ht.tps://arxvv.or wttp wwwpg////////// 04760
Mikhail Khodak, Neil Tenenholtz, Lester Mackey, and Nicolo Fusi. Initialization and regularization of factorized neural layers. ArXiv preprint, abs/2105.01029, 2021. URL htt.:httt://. https:///axii
org/ ht5//bb//2000 01029
Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He. Weizhu Chen, and Tuo Zhao, Losparse: Structured compression of large language models based on low-rank and sparse approxi- mation. In International Conference on Machine Learning, pp. 20336-20350. PMLR, 2023.
Chi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. Modegpt: Modular dccomposition for large language model compression. ArXiv preprint, abs/2408.0966 096622,,9999 2024. URL htt.::ttts://rttt www.siivvvrrgoooo org/ abs 2408 09632
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702- 2023.
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Short Papers), pp. 1773-1781, 2023.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):31-330, 1993. URL
TlEtps:/aaclatteen
org/ J93-2003
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https //openreview. net htt////ooummmmm////eeeee 172uxxe
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 1-291,, Brussels, Belgium. 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/D
URL https
//aclantho wwwlaanthooooy..ooom ogy.ooy..ooooooo......... wrw.y//-g//11111 1260
OpenAl. GPT-4 Technical Report. ArXiv preprint, abs/2303.08774, 2023. URL httLhtss:///xii
08774
Gunho Park, Baeseong Park, Minsub Kim, Sungiae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models. ArXiv org/ abs 2206. 09557
preprint, abs/2206.099 09557,, 2022. htttst/tt:
Zhengqi Pei and Shuhui Wang. Dynamics-inspired neuromorphic visual representation learning. In International Conference Machine Learning, pp. 27521-27541. PMLR, 2023.
Zhengqi Pei, Anran Zhang, Shuhui Wang, Xiangyang Ji, and Qingming Huang. Data-free neural rep-
resentation compression with riemannian neural dynamics. In Forty-firsi International Conference on Machine Learning, 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 167,, 2020.
12

---

Published as a conference paper at ICLR 2026
Rajarshi Saha, Varun Srivastava, and Mert Pilanci. Matrix compression via randomized low rank and low precision factorization. Advances in Neural Information Processing Systems, 36, 2023.
Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea Goldsmith, and Mert Pilanci. Compressing large language models using low rank and low precision decomposition. Advances in Neural Information Processing Systems, 37:88981-89011 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106,
2021.
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning, Advances in neural information processing systems, 1--0078--2089) 2020.
Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Nonlinear component analysis as kernel eigenvalue problem. Neural computation, (0(5):1299-1311 1998.
Shufan Shen, Junshu Sun, Xiangyang Ji, Qingming Huang, and Shuhui Wang. Expanding sparse tuning for low memory usage. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
Junshu Sun, Chenxue Yang, Xiangyang Ji, Qingming Huang, and Shuhui Wang. Towards dynamic message passing graphs. In The ehirtt-eighth Annual Conference Neural Information Processing Systems, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Erie Hambro, Faisal Azhar, al. Llama: open and efficient foundation language models. arxiv. ArXiv preprint, abs/2302.13971, 2023a. URL
htttss:/aarxis http://axxxxv..aaaaa.aaaa//a
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, hlx/2337.099888 2023b. URL https: //arxiv
org/ btgg/abs/ae 2307. 09288
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free free low-rank adaptation. ArXiv preprint, 1i./2210.00558. 2022. URL
rxxi.... org/ abs 221 0. 07558)
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-llm: Truncation-awan singular value decomposition for large language model compression. ArXiv preprint, abs/2403.07378, 2024.
URL htt.:/ttt///axiiiii org/ abs /203. 07378
Wei Wen, Cong Xu, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Coordinating filters for faster deep neural networks. IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 658-666. IEEE Computer Society, 2017. doi: 10. 10.1009/CC2. URL httph:ttttttttttt
1ww1/11I1.......... 2017. 78
Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu, Yong Li, Wei Lin, and Shuaiwen Leon Song. Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity. Proceedings of the VLDB Endowment, 17(2):211-224, 2023.
Hongwei Xue, Peng Gao, Hongyang Li, Yu Qiao, Hao Sun, Houqiang Li, and Jiebo Luo. Stare at what you see: Masked image modeling without reconstruction. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22732-22741 2023.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training guantization for large-scale transformers. 2022.
Advances in Neural Information Processing Systems, 35:27
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation- aware singular value decomposition for compressing large language models. ArXiv preprint, abs/2312.05821, 2023. URL https:
org /bs /2322. 220..//22222221111111 05821
13

---

Published as a conference paper at ICLR 2026
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, 2019. Association for Computational wwwphhhlllg..og///gg11oo/
Linguistics. doi: 10.18653/v1//19- 1472. URL htt.:httt///attt
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao, Adaptive budget allocation for parameter-efficient fine-tuning, In International Conference on Learning Representations. Openreview, 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. ArXiv preprint, abs/2205.0100 2022. URL httts:://axii: org abs 2205 01068
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12:1556-1577, 2024.
14

---

Published as a conference paper at ICLR 2026
A FORWARD PROPAGATION WITH NONLINEAR LOW-RANK APPROXIMATION
A.1 FORWARD CODE
der forward x):
in_shape x.shape
assert in_shape[-1] == in__iiturreee
x x. x.reshape (-1, in.
In_Qa_Square, 0ut_Qe_
Ielf.1n_ Qut_
In_ _u_uur_ ta_Qa_8quara sun((1) shared_ unaqueeze
LeIn__e_Squaee - -I Se_uare pezaute pee.euuaeeeeeetteeee... 0)
9_in_0s_Square S_In_ _a_arar__u__.___ sun (-1) unsqueeze(11
Out_qe_Square Qut_Qe__quare sun (-1) *eelf sharede .uequeeeeeeeeeeeeeee (1)
x_aun x. aum
(-m()1) (-1) (-1)) (1, self aun_ s_0ut_Qa_Bquar x_aum Qut_qe__Buare
8 B_Inner_Pal Product I_Iouer_Produet 8 S_inner_Produs Product self. a S_Inner_ s_Icaer_Prodie
self In_Qs. peraute (0, 2, x.T
S_Inmer_ seelucttteeff..heeeeee. unsqueeze unsqueeze (-1)
8 S_Inner_ Iroucct s I.Inee_Product..ee
pernute (1, 0)
out sut_e list (in_shape)
out out_shape [-1]
S_In_Rs_Squar S_Qut_Qs_ Qe_qure x_out x_out. r_ (tut return x_out
Figure 4: The Python code for the forward propagation corresponding to Algorithm]
A.2 FORMULA DERIVATION
For A E |mxxXxx B RaxHxr representing the in and out neurons in Section 3.2 and representing the shared coefficients, we have
R
wl Σ Uh aahh bynll
(11)
where ash byh E RR, For an input x [.1.... Fml E IR",, we can calculate the output = [y....., ynl by
H Mh |laih byall
My
w/ 24
1=1
1=1
h=1
m
F2
ph (ash
byn) (amh byn)
k=1
(12)
H
p4
[lann
112
||by
2ath byh
1=1
k=1
14
aash
1h [byhl 11
14ath byh
k=1
h=1
1=1
h=1
Thus, the multiplication activations and matrices can be swapped with the piccewise lincar operation, allowing for the separate calculation of A², B2 and A B, thereby saving memory.
A.3
COMPLEXITY ANALYSIS
The FLOP count of Algorithm T can be calculated as follows:
The FLOP count for SumXX. O(H in) O(H r- in) O(2 b- H in)
15

---

Published as a conference paper at ICLR 2026
The FLOP count for SumXout 20(H out) + O(2 b. H out).
For inkoot the FLOP count is 0 H rr- b- (in + out)).
So the total FLOP count of Algorithm i
FLOPs =20(H in) + 20 H -n.inn) 20 out)
(13)
+ +0 H out)) + 20 -r (in + out))
"in" and "out" represent the input and output dimensions, respectively, and "b" denotes the batch size. And the total memory of Algorithm [
is:
2 H r (in + oul) 4 4+b. H 4+ b. out 4 b. H out
(14)
The computational results show that the computation and memory usage increase linearly with the number of parameters, so the increase model parameters does lead to explosive growth in computation and memory usage.
B KERNEL INTERPRETATION OF THE PIECEWISE-LINEAR FUNCTION
The piecewise-linge -inear function can be expressed as:
(A, = Σ ||Ar, Bi
(15)
1=1
For any coefficients or with E, 0ri we have
s
Σ ajaj |0,-,,11" Σ a Σ a,1lx 2z11
Σ a
aj
(1<;
| 112
2x; -xzz)
0j Σ as|l=:11?? + -{_
(16)
Σ
Eggless1 - - ag Σ
-2
Σ
a;T;
<0 0
So that each individual term ||a - 2 is a conditionally negative definite function, and thus its negation -l[a b is valid conditionally positive definite(CPD)) kernel. The weighted sum of such kernels
K(A, B) = - (A, B)
Σ ||Ar, - -B,,
(17)
1=1
also a CPD kernel. So the piecewise-linear function can be used as kernel function.
Convergence guarantee. NLAA is effectively performing low-rank approximation Hilbert space where standard convergence guarantees of kernel methods apply, and where higher-order interactions can be represented linearly (Schölkopf et al. 1998). Moreover, the expressiveness of piecewise- linear structures is supported by the Dynamical Universal Approximation Theorem (Pei & Wang 2023). which states that piecewise-lined systems can approximate arbitrary sequential functions.
C EXPERIMENTAL DETAILS
We applied compression operations to the weight matrices of all linear layers, including those in the attention modules (Q, K. V, and Out). For LLaMA-7B. the hyperparameter is set 20, by is the final budget determined by the target compression ratio, and the initial budget bn = 1.2by. The hyperparameter H is obtained by dividing the initial budget bn by the number of weight matrices.
16

---

Published as a conference paper at ICLR 2026
It can be observed from Appendix A.3 that under the same compression ratio (i.e., fixed H r), larger H leads to higher computational complexity and memory usage. However, H is too small, the number of segments in the piecewise linear function becomes insufficient, which may degrade model performance. Therefore, we chose a moderate setting for H and r to balance efficiency and performance.
In the retraining step, we use the Adam optimizer. We select 512 texts from the Wikitext-2 dataset as training data, with batch size of 8, a learning rate of 5e-6, and train for 4 epochs.
For Swin- Transformer in Table 4, we used the standard fine-tuning recipe: AdamW optimizer, learning rate le-4, weight decay 0.05, batch size 64, 50 epochs. All low-rank baselines and NI.A were retrained using exactly these hyperparameters with only the compression methods changed.
D EXPERIMENTS
D.1 ABLATION STUDY OF THE BUDGET ALLOCATION STRATEGY
Table 11: Comparison of Cubic and Linear Budget Allocation Strategies
MODEL
STRATEGY
PERPLEXITY
CUBIC LINEAR
19.77
1.3B
43.24
CUBIC
31.34
OPT- 125M
LINEAR
63.04
We compare the cubic budget allocation strategy in Equation 9 with a linear strategy on OPT-1.31 and OPT- 125m. The experimental results shown in Table The experimental results indicate that the cubic setting leads to less performance degradation compared to the linear setting.
D.2
APPLICABILITY OF NLA ACROSS DIFFERENT MODEL. SCALES
Table 12: Comparison of NLA and Baseline Methods on Models of Different Scales
METHOD
BASE MODEL
PARAMS.
PERPLEXITY (U)Acc. (f)
PELA
SWIN-90M
62.2M
82.4
NLA
SWIN-90M
40.2M
82.7
MODEGPT
OPT-125M
75M
38.37
NLA
OPT-125M
63M
31.34
MODEGPT
OPT-1.3B
0.78B
21.92
NLA
OPT-1.3B
0,65B
19.77
SVD-LLM
LLAMA-7B
2.9B
17.93
NLA
LLAMA-7B
2.1B
17.76
To verify the scalability NLA, we conduct experiments on models different scales, including vision models (Swin-Traae Transformffrr ImageNet-IK) and language models (OPT and LLaMA Wikitext2). As shown in Table [2] NLA consistently achieves a lower parameter count while maintaining even improving performance. On WikiText2, NLA demonstrates clear advantages over MoDeGPT (Lin et al.) 2024) and SVD-LLM (Wang et al. 2024) across different model sizes: reduces parameters
aggressively and achieves lower perplexity both OPT and LLaMA-7B. These results confimm that NLA provides a scalable and robust compression framework applicable to both vision and language models of various sizes.
D.3 LINEAR VS. NONLINEAR APPROXIMATION LOSS
Table serves supplementary result to Table d providing more fine-graines comparison between linear (SVD) and nonlinear (NLA) low-rank approximation. For different weight matrices in various layers, approximating with NLA gains much less mean loss than with SVD.
17

---

Published as a conference paper at ICLR 2026
Table 13: Mean loss during the fitting of different weight matrices using SVD and nonlinear low-rank approximation.
LAYER
NAME
SVD Loss
NLA Loss
Q
0.0118
0.0053
0.0119
0.0052
0
0.0056
0,0026
0
0.0047
0.0021
FC1
0.0177
0.0106
5
0.0250
0.0074
8
0.0290
0.0114
D.4 INFERENCE MEMORY UNDER DIFFERENT COMPRESSION RATIOS.
Table 14: Memory usage during inference under different compression ratios. The batch size is set to
MODEL
RATIO
INFERENCE MEMORY (MB)
0%
210.19
60%
100.89
SWIN-SMALL
73.57
80%
48.34
366.69
60%
161.26
SWIN-BASE
70%
117.34
80%
77.03
0%
555.83
50%
350.33
OPT-125M
55%
301.03
60%
269.34
5175.34
60%
2456.31
OPT-1.3B
70%
1870.06
80%
1296.57
We evaluate the GPU memory consumption during inference various compression ratios, as summarized in Table 14 Across all tested models, memory usage decreases substantially as the compression ratio increases. The effect is especially pronounced for larger models: for OPT- 1.3B, inference memory drops from 117..3 MB at 0% compression to only 1296.57 MB at 80% compression. Similar trends are observed for vision models such as Swin-Small and Swin-Ban Base. These results demonstrate that compression technique significantly reduce inference memory footprint while preserving model functionality, which is particularly critical for deploying large-scale models in resource-constrainee environments.
D.5 TIME AND MEMORY COST FOR LANGUAGE MODEL
Table 15: Inference Time and Memory Usage of LLaMA-7B compressed by SVD and NLA
METHOD
PARAMS
PERPLEXITY
RATIO
AVE. InFERENcE TIME (S)
GPU MEMORY (GB)
ORIGIN
7B
5.68
0%
26.27
SVD
3.0B
42.30
60%
0.409
13.89
NLA
2.1B
39.91
70%
0.593
8.97
18

---

Published as a conference paper at ICLR 2026
E REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of the results presented in this paper. All novel algorithms, including our Kerne-bbaeed Nonlinear Approximation method and the Adaptive Com- pression strategy, are described in detail in Section 3 Appendix A and Appendix B We will release our implementation as part of the supplementary material. Details regarding the experimental setup, hyperparameter configuratioss, and all utilized benchmark datasets are comprehensively documented in Section 4 and Appendix C
F LLM USAGE
Large Language Models (LLMs) were used solely as general-purpor purpose assist tools refine the language and improve the overall clarity of the writing. Specifically, an LLM was utilized for minor tasks such as grammar correction and enhancing sentence structure.
19


Reviews:
[{"review_id": "rodp93gXj0", "reviewer": "Reviewer_G7Qc", "summary": "This work proposes non-linear low-rank approximation (NLA), a compression method for LLMs and vision transformers that reduces memory footprint. Traditional low-rank compression uses linear SVD truncation, which incurs large information loss at high compression ratios. On the other hand, NLA applies a piecewise-linear kernel to approximate weight matrices in a nonlinear feature space, enabling richer representations from low-rank factors. Furthermore, to avoid reconstructing the full dense matrix from the low-rank factors during inference (which would be impractical), the authors design a custom forward operator that computes outputs directly from low-rank factors, reducing compute and memory overhead.\n\nTo handle heterogeneous layer importance, NLA further introduces adaptive budget allocation -- it dynamically assigns different compression ratios to different layers based on sensitivity (gradient x weight) score and a cubic sparsity schedule. Experiments on Llama-7B, OPT, and Swin-Transformer show that NLA outperforms state-of-the-art linear methods (e.g., SVD-LLM, ASVD, FWSVD), achieving lower perplexity and higher accuracy at similar or higher compression ratios. NLA can also be combined with quantization, enabling more aggressive memory savings with minimal accuracy loss.", "questions": "1. What is the function $s(\\cdot)$ in eq. (5)?\n\n2. In general, Sec. 3.3 should be explained in more details -- seems like a big jump from prior text.\n\n3. Why have more recent models like Llama 3/4, Wen, etc. not considered? Compressing these newer models are more challenging and results compared to prior works will be more interesting.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper moves beyond conventional SVD-based linear rank truncation by using piecewise-linear kernels, and demonstrates that nonlinear structure retains more information under high compression. The custom forward operations that enables forward pass without reconstructing the entire matrix is also interesting -- this allows doing memory- efficient inference despite the non-linear mapping. \n\nThe authors also address the varying sensitivities of the different layers -- allowing for non-uniform rank-budget allocation -- which is a key practical issue in compression. Since the proposed methods is compatible with other compression methods like quantization, this suggest potential stacking to improve the compression ratios.\n\nCompared to prior works, it consistently outperforms across LLM and vision benchmarks.", "weaknesses": "I have some concerns, and would appreciate it if they are addressed:\n\n1. My primary concern with the paper is that the presentation can be improved. For example, does nonlinear low-rank attention a more general case of linear SVD based low-rank factorization? Under what conditions does the former boil down to the latter? The Alg. 1 pseudocode should also be properly explained in text.\n\n2. Moreover, the nonlinear formulation in eq. (3) is intuitively motivated and empirically effective, but the paper lacks deeper theoretical justification or analytical guarantees on approximation error or convergence, especially compared to the well-studied linear methods like SVD.\n\n3. The computational overhead of doing the forward pass is discussed, but the experimental section where latency numbers are mentioned in Table 9 requires more elaboration. How re the latency number obtained -- these increased compute cost and latency are not rigorously quantified for large-batch or real-time inference settings. Do they involve customized kernel implementations (CUDA, Triton, etc.) for the forward path?\n\n4. Combination of quantization with low-rank is only vaguely discussed. It would be appreciated if the authors discussed relevant works in this context such as https://proceedings.neurips.cc/paper_files/paper/2024/hash/a20e8451ffb07ad25282c21945ad4f19-Abstract-Conference.html", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aAyrD0r6oz", "reviewer": "Reviewer_ddCq", "summary": "This paper proposes NLA, a nonlinear low-rank compression method with adaptive budget allocation for large foundation models. It introduces a piecewise-linear kernel operator to enhance expressiveness and avoid reconstructing full matrices, and uses an importance score to allocate compression capacity efficiently. NLA outperforms existing linear methods (like SVD, SVD-LLM, etc) across LLaMA, OPT, and Swin-B at high compression ratios.", "questions": "- What is the overhead (time, resources) for NLA?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Exploring nonlinear approximation for low-rank decomposition is interesting.\n\n- Experiments demonstrate consistent improvements across other baselines.\n\n- Ablation studies is throughout, including some interesting experiments like End-to-End memory/throughput comparison, capability with quantization, etc.", "weaknesses": "- In Line 188, it's better to explain the reason for selecting the nonlinear kernel format.\n\n- The results in Table 3 looks strange to me. For example, PiQA is a binary classification task, the accuracy around 50% is already random guess, the improvements is not meaningful. Same as other tasks in Line 328. If the experiments is evaluated on the generation format rather than logits comparison, then better select instruction version of the model to ensure the answer follows the prompt format.\n\n- Table 4 shows that NLA reduces throughput, making the method less compelling in practice. In general, memory-efficient techniques can either enable inference that would otherwise result in OOM, or improve throughput in memory-bound scenarios. However, from these two perspectives, it's difficult to directly observe the benefits of NLA. It would be better to demonstrate specific scenarios that NLA can help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "59dEDKPa6x", "reviewer": "Reviewer_nNEe", "summary": "NLA compresses neural networks using nonlinear kernel functions instead of linear matrix products, achieving better quality than SVD methods. However, it's much slower at inference than SVD methods, lacks fine-tuning results on key benchmarks, and needs broader baseline comparisons and theoretical justification.", "questions": "1. Table 3 shows common sense reasoning results WITHOUT fine-tuning, where all methods perform poorly (0.30-0.37 average). However, Table 2 shows dramatic improvements with fine-tuning for perplexity. Can you provide fine-tuning results for all seven common sense reasoning tasks in Table 3? Specifically, does NLA maintain its 4-point advantage (0.37 vs 0.33) over SVD-LLM after both methods are fine-tuned? This is critical for fair comparison since fine-tuning is standard practice.\n\n2. The paper fixes r=20 and derives h from the compression ratio (Appendix C). Can you justify:  Why r=20 specifically? Did you try r=10, 15, 25, 30?\n\n3. Table 1 shows results at different compression ratios for different methods (ASVD at 40/50/60%, NLA at 50/70%). Can you provide a complete comparison table: Method 40% 50% 60% 70%?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Novel application of nonlinear kernels to compression\n2. Consistent exp improvements\n3. Clever efficient algorithm", "weaknesses": "1. Incomplete Evaluation: Table 3 lacks fine-tuning results for common sense reasoning, making it impossible to fairly assess NLA vs. SVD methods post-fine-tuning. This is important given that fine-tuning is standard practice.\n2. Computational Cost: The paper shows better final quality but shower than SVD methods, and likely orders-of-magnitude longer compression time. \n3. Limited Theoretical Foundation: No convergence guarantees, approximation bounds, or principled hyper-parameter selection. Purely empirical. When and why does this work?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FBg4EmUDSC", "reviewer": "Reviewer_5mVN", "summary": "This paper proposes NLA (Nonlinear Low-Rank Approximation with Adaptive Budget Allocation), a new model compression framework for large foundation models (LFMs) such as LLaMA-7B and Swin-Transformer. NLA replaces conventional linear low-rank approximations (e.g., SVD-based) with a piecewise-linear kernel that captures nonlinear relationships, allowing higher expressiveness and reduced information loss under high compression ratios. To manage varying layer sensitivities, it introduces an adaptive compression mechanism that allocates compression budgets dynamically via cubic sparsity scheduling based on parameter importance scores. Additionally, a custom forward algorithm enables nonlinear low-rank inference without reconstructing full weight matrices, reducing memory usage. Experiments on language and vision benchmarks show that NLA achieves substantially lower perplexity and comparable or higher accuracy at significantly higher compression ratios than prior methods (e.g., SVD-LLM, FWSVD, ASVD).", "questions": "1. Were the NLA-compressed models retrained under the same training protocols and hyperparameters as the baseline methods (e.g., SVD-LLM, ASVD, FWSVD)?\n\n2. Could you specify the retraining setup (optimizer, learning rate, epochs, batch size, etc.) used for vision models?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "## Novelty: \n\nThe paper introduces Nonlinear Low-Rank Approximation (NLA) — a fundamentally new compression paradigm that integrates nonlinear kernel-based approximation with adaptive compression ratio allocation. Unlike traditional SVD-based methods that rely purely on linear decomposition, NLA leverages a piecewise-linear kernel to approximate weight matrices in a higher-dimensional space, effectively recovering information lost during rank truncation.\n\n## Differences from Prior Work:\n\n- Prior low-rank compression methods (e.g., SVD-LLM, ASVD, FWSVD) perform linear approximations, inevitably losing important information at high compression ratios.\n- Kernel-based nonlinear approaches have existed but are computationally expensive due to explicit matrix reconstruction. NLA overcomes this via a custom forward operator that performs nonlinear propagation without reconstructing full weight matrices.\n- Previous methods use uniform compression across layers, while NLA introduces adaptive budget allocation using importance scoring and cubic sparsity scheduling, enabling per-layer dynamic compression control.", "weaknesses": "### Fairness of Experimental Comparison:\n\nIt is unclear whether the proposed NLA and all baseline methods were retrained under the same training protocol. Appendix C provides insufficient training details, particularly for the vision model experiments. For example, there is no description of retraining settings for Swin-Transformer, nor any indication of whether the counterparts in Table 4 were trained under identical configurations. Providing consistent and transparent training settings is essential for a fair comparison.\n\n### Generality and Model Coverage:\n\nThe baselines used in this paper are relatively outdated (LLaMA-7B, OPT, Swin Transformer). To demonstrate the generality of NLA, it would be valuable to include experiments on more recent models such as Qwen2.5/Qwen3 or DeepSeek. For vision, evaluating on modern architectures like Vision Transformer (ViT) more widely used on multimodal VLMs would better reflect current practice.\nAdditionally, scalability across different model sizes should be examined. For instance, if compressing Qwen3-8B by 50% leads to performance degradation such that it underperforms the smaller Qwen3-4B, the practical value of compression becomes questionable. A similar analysis should be provided for vision models (e.g., comparing ViT-B-NLA vs. ViT-S).\n\n### Throughput and Memory Efficiency Claims:\nThe paper should report throughput and memory consumption alongside perplexity and accuracy, ideally comparing them directly against SVD-based baselines (FWSVD, ASVD, SVD-LLM). While Table 4 presents FLOPs and throughput, the results are inconsistent — NLA achieves higher compression (54.2%) but lower throughput than the original Swin-Transformer. This inconsistency suggests that FLOPs alone cannot justify claims of “comparable throughput.” As shown in Table 4, even with lower FLOPs, NLA exhibits worse throughput than PELA, contradicting the statement at line 462 (“with comparable computation and throughput”). These efficiency claims should be tempered or supported with real GPU inference timing and memory profiling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y2yUFUQhyQ", "reviewer": "Reviewer_Lzw5", "summary": "The paper proposes a novel compression method for model weights with non-linear low-rank parametrization and adaptive budget allocation. Comparing to other methods, the authors claim improved perplexity / accuracy at equal or higher compression rates, which implies lower memory usage, at the cost of a modest inference-time overhead. The experimental validation includes LLM pretraining and reasoning benchmarks, ImageNet-1K classification for vision transformer, and comparisons against other low-rank compression methods.", "questions": "1. Please report the performance (perplexity / accuracy) vs. inference time for NLA and other methods. Can NLA outperform other methods at the same (or smaller) inference time and memory budget? \n2. How does NLA impact training times?\n3. How exactly was TinyBert applied to the Swin Transformer baseline?\n4. Is the strategy of fixing r=20 and varying h optimal? Please justify.\n5. How does NLA compare with other compression-based methods like SparceGPT and GPTQ?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed method demonstrates significant performance retention at compression rates over 50%, for example, LLaMA-7B with NLA at 70% compression outperforms SVD-LLM baseline at 60% on Wikitext-2 perplexity. The piecewise linear kernel allows for an efficient implementation thanks to reformulation through a series of matrix multiplications.", "weaknesses": "1. My main concern is an unexplored computational complexity introduced by the method. Naturally, using more expressive non-linear layer structure would allow for a better model performance compared to linear methods, but it does so at a significant computational cost. In Table 15 the inference time is x1.5 higher for NLA compared to SVD, even at a more aggressive compression ratio. The higher inference cost makes the method less likely to be adopted in real deployment. \nTraining efficiency is also not discussed; it's unclear how much longer retraining took compared to other baselines.\n2. It is not explained how TinyBert was applied for compression of the Swin transformer model. TinyBert is a language model, and if the authors have used the Transformer distillation method proposed in the said paper (Jiao et al., 2020), they should be clearer with their wordings.\n3. Different compression rates were used to compare methods in Tables 1, 3, 4, 14, 15, which doesn’t allow for the fair comparison. For example, in Table 3 the compression rates of 40, 50 and 60% were reported for the first 3 methods, then 50 and 60% percent for SVD-LLM and 50 and 70% for NLA.\nI understand the authors’ aim to highlight how even with more aggressive compression the proposed method stayed competitive, but all the numbers should be reported nonetheless. A standardized approach to comparisons could be: a) to compare performance at the same compression rates, b) to compare compression rates respective to the same performance level(s), c) to create a plot with (compression rate, performance) axes and compare the trade-off curves for different methods. \n4. The reported experiments lack consistency: the authors report the main results only for the LLaMA-7B model, then in Section 4.2 they use the OPT family models for several ablation studies. \n5. Algorithm 1 lacks analysis with regard to its hyperparameters. While the authors propose the adaptive search algorithm for the optimal values of h, they fix the hyperparameter r = 20. As performance depends on both of them, the dependency on r is not explored.\n6. The proposed adaptive budget allocation delivers small absolute gains. It's not the main source of improvement, but it adds computational complexity in training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)