UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMs
Authors: Hung-Yueh Chiang
Chi-Chih Chang
Yu-Chen Lu
Chien-Yu Lin
Kai-Chiang Wu
Mohamed S. Abdelfattah
Diana Marculescu
Venue: ICLR 2026 Poster (2026)
Keywords: large language model
model quantization
model compression
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: UniQL is designed as a general framework that integrates quantization and pruning for Transformers, State Space Models (SSMs), and hybrid models to cater to diverse edge applications.

Abstract: Deploying large language models (LLMs) on mobile platforms faces significant challenges due to the limited memory and shared computational resources of the device. Resource availability may be an issue as it is directly impacted by on the current device workload, adding to the uncertainty of model deployment. We introduce UniQL, a unified post-training quantization and low-rank compression framework, with on-device configurable pruning rates for edge LLMs. UniQL is a general framework that integrates quantization and low-rank compression for Transformers, State Space Models (SSMs), and hybrid models to cater to diverse edge applications. In our proposed joint framework, we introduce an efficient structured weight-sorting that speeds up the computation by 20×, quantization-aware singular value decomposition (SVD) decompositions to minimize the quantization errors, state-aware weight sorting for SSMs, and a fused rotary embedding (RoPE) kernel for the pruned models. Our framework performs weight-sorting, fine-tuning, and quantization in the cloud in a one-shot fashion, while enabling on-device configurable pruning rates up to 35%. Our experiments show that quantized and pruned models offer a memory reduction of 4×–5.7× and a token throughput improvement of 2.7×–3.4×, maintaining accuracy within 5% of the original models at 15% pruning rates across Transformers (Llama3 and Qwen2.5), SSMs (Mamba2), and hybrid models (Nemotron-H and Bamba-v2). The code and quantized models will be released at: https://github.com/enyac-group/UniQL.

Published as a conference paper at ICLR 2026
UNIQL: UNIFIED QUANTIZATION AND LOW-RANK COMPRESSION FOR ADAPTIVE EDGE LLMS
LlungYYueh Chiang Chi-Chih Chang 2, Yu-Chen Lu , Chien-Yu Lin 4,
Kai- Kai-Chiang Wu 3 Mohamed S. Abdelfattah 2, Diana Marculescu
Chandra Family Department of Electrical and Computer Engineering,
The University of Texas at Austin
2 Department of Electrical and Computer Engineering, Cornell University
3 Department of Computer Science National Yang Ming Chiao Tung University
The Paul G. Allen School Computer Science and Engineering, University Washington
ABSTRACT
Deploying large language models (LLMs) on mobile platforms faces significant challenges due to the limited memory and shared computational resources of the device. Resource availability may be an issue as it is directly impacted by on the current device workload, adding the uncertainty of model deployment. We introduce UniQL. a unified post-training quantization and low-rank com- pression framework, with on-device configurable pruning rates for edge LLMs. UniQL is general framework that integrates quantization and low-rank com- pression for Transformers, State Space Models (SSMs), and hybrid models to cater to diverse edge applications. In our proposed joint framework, we in- troduce an efficient structured weight-sorting that speeds up the computation by 20x, quantization-aaware singular value decomposition (SVD) decompositions minimize the quantization errors, state-aware weight sorting for SSMs, and fused rotary embedding (RoPE) kernel for the pruned models. Our frame- work performs weight-sorting. fine-tuning, and quantization in the cloud a one-pass fashion, while enabling on-device configurable pruning rates up 35%. Our experiments show that quantized and pruned models offer a mem- ory reduction of 4x-5.7x and token throughput improvement of 2.7x-3.4x, maintaining accuracy within 5% of the original models at 15% pruning rates across Transformers (Llama3 and Qwen2.5), SSMs (Mamba2), and hybrid models (Nemotron-H and Bamba-v2). The code and quantized models will be released at: https://githun com/ htrpe/eeyacccccee gropp/unirro
INTRODUCTION
Numerous emerging applications, such as question answering on VR/AR glasses, are powered by large language models (LLMs). Yet, models with parameters on the order of billions (e.g.. 10B)
restrict the platforms and applications that can utilize them. Extensive research investigates quanti- zation (Xiao et al. 2023 [in et al.) 2024a (b) and compression (Qinsi et al.. 2025] Wang et al. [0225b] 2 Lin et al. 2025) for LLMs to lower memory and computing needs for deployment. However, the limited and shared resources (e.g., the unified memory architecture) on edge devices still pose huge challenges for model deployment. Since resources (e.g., memory) are dynamically managed by the operating system, the availability of the resources highly depends on the system workload. As a result, the pompesspees pre-quantized language models with fixe model sizes may not a device under high workload scenarios.
Re- Re-compressing re-quantizing the model to fit it into available memory is not practical due to the high computational costs, ie., several hours on cloud GPUs (Lin et al. 2025) Frantar et al.)
Corresponding authors: (hungyueh.chl chi.ccciaa chiang, di dianammuutexas.com

---

Published as a conference paper at ICLR 2026
-C-
Uniqq. (W4A16)
Nemotron- H
One-pass compression
On-device pruning
TAD-HQO (W4AL6)
Quantize and deploy
R
Low
Prune
75
Ulama-311- 3.1-86
Avg. Acc. (%)
Prune less
70
Adaptiive size
Mambaa--88
Memory utilization
High
Prune more
Prune
1668
60
more
IPIb (tst.)
MB
2G8
55
TTLT (sec)
128
(a) The UniQL framework
(b) Accuray vs. Latency
Figure 1: (Proposed framework overview.) UniQL supports Transformers, SSMs, and hybrid models, enabling one-shot compression using single server-class GPU. The on-device pruning of the quantized model is feasible and configurable based on the current device workload. We present actual latency on Nano 8G in relation accuracy for different pruning rates across three distinct models on the right. Circle sizes correspond to model sizes.
2023). A solution to address this issue is storing several model replicas at different compression rates. Nonetheless, producing pre-compressed replicas of different sizes is both time- and storage- consuming. Alternatively, employing elastic training (Cai al. 2024 2025) to a pre-trained model enables the derivation of various sizes from the model. Yet, this approach requires availability of GPU resources and training on curated datasets support flexible deployment for one specific type of model, e.g., Llama-3.1-81 limiting the applicability.
Our proposed work addresses this issue under the post-training setting when access to server-class GPUs and curated datasets is limited. As illustrated in Figure ! our framework supports quantiza- tion and structured praning, performing efficiently on one server GPU. Our objective is support and design compression algorithms for various model architectures, including Transformers, State Space Models (SSMs), and hybrid models. Our pipeline is shown in Figure We group the weights within the block, gather channel corrections from a calibration dataset, and apply weight-sorting algorithms. Our multi-layer perceptron (MLP) weights are decomposed without any gradient infor- mation or expensive full matrix pseudo-inverse, yielding speedup of 20x compared to prior art (Lin et al. 2025). For W. and Wo in self-attention layers, we develop a quantization-awe aware singu- lar value decomposition (SVD) of weights to minimize quantization errors. For SSMs and hybrids models, we find that SSM blocks are particularly sensitive to state matrices, and propose a state- aware weight-sorting strategy mitigate this. We then apply a masked fine-tuning to the sorted model. In each fine-tuning step, global pruning rate P: is chosen randomly, masking the least ranked channels in the layers. The refined model then quantized in low bit-width and deployed on the edge platform. The entire process is performed once in the cloud. For the deployed model, we prune the models according to a specified global pruning rate, e.g. Pas = 35%. on the edge device. Our contributions are summarized as follows:
Our study explores broad spectrum of models, such as Transformers, SSMs, and hybrid, and algorithms for these blocks.
introduces efficient pruning and quantization--rieete
To the best of our knowledge, UniQL is the first post-training framework that systematically combines quantization and structured pruning for LLMs in a one-shot fashion.
We develop an integrated kernel to support the pruned RoPE, conducting comprehensive profil-
ing to demonstrate 2.7 x x334x latency speedups for adaptive pruning on edge devices.
2 RELATED WORK
Transformer compression. Prior work has aimed to reduce the size of Transformer- based LLMs for efficient deployment by utilizing low bit-width data types (Xiao et al. 2023 Lin et al. 2024b a
Zhao et al.) 2224 ( Liu et al. 2025 |225]] Ashkboos et al.) 2024b). minimizing storage needs and optimizing
hardware for how-hit computations. Unstructured (Frantar & Alistarh 2023 Sun et al. 2024 and
2

---

Published as a conference paper at ICLR 2026
Pruiing
The
P--P
Pinu-fran
Si Sg Sin Wa
MLPP
Wiyms
Memory utiization
more
Prune
prantzztioonawaar S 100m/90,00
Pis
(96 (16)
(WG) [HG)
(as R
Stare
WWS.
Wiors
Mamba
[Wa]] [N2
Lilsyors
(a) Decompose and sort the channels
(b) Masked fine-tune (c) Fusc and quantize (d) Deploy and adaptively prune
(The UniQL pipeline.) We devise pseudo-inverse-free quantization-aware and state- aware matrix decomposition methods for the grouped weights to obtain sorted weights (a). During fine-tuning, we sample global pruning rates, and masked out the weight channels (b). The refined patches are fused into the weights, followed by model quantization for deployment (c). Based on the system utilization, we perform on-device adaptive pruning of the quantized model (d).
Figure 2:
semi-structured pruning (e.g.. N:M sparsity) (Li et al., 2023) for reducing model size by removing specific parameters while minimizing accuracy loss. Nonetheless, deploying such methods requires specialized hardware (Taka et al. 2025 Xia et al. 2023). Structured pruning (Wang et al.] 2025a Lin al. 2025 Ma et al. 2003| Ashkboos et al. (024a) removes whole elements (e.g., channels and heads), enabling faster inference on standard hardware but sacrificing performance. Some studies focus on one-shot compression (Genzel al.) 2025 Wang al. 2025c), flexible bit-width quan- tization (Park et al. 2024), and quantization with semi-structured sparsity (Mozaffari et al.) 2025). Our framework is systematically designed for quantization and on-device structured pruning.
SSM compression. State Space Models are memory-cfficien alternatives Transformers, Recent studies (Xu et al. 2025 Chiang et al. 2025b a) introduce low-bit quantization techniques for SSMs. Structured (Taghibakhshi et al. 2025 Muñoz et al. 2025) and unstructured pruning (Tuo & Wang. 2025 (hhhab et al. (2025) strategies have been developed for SSMs. For example, TTahibakhshi et al. (2025) eliminate the SSM heads and restore performance through knowledge distillation training. Muñoz et al. (2025) explore block-wise (e.g., Mamba and Transformer blocks) and module wise (e.g.. SSM and self-attention modules) structured pruning methods. Some work explores the token pruning (Zhan et al. (024) dimension reduction (Chi et al. (024)) for vision SSMs. Our focus is on analyzing a broader structured pruning and quantization framework for Transformers, SSMs, and hybrids, distinguishing it from previously mentioned approaches.
Elastic training for LLMs. Elastic training aims to enable a pre-trained LLM to dynamically adapt to varying deployment constraints such memory, compute, and latency budgets. Flex- tron (Cai et al. (024) and LLaMaFlex (Cai et al. 2025) introduce many-in-one architectures through pruning and weight sharing, allowing adaptive inference under dynamic resource constraints. Jet- Nemotron (Gu et al. 2025) further leverages post-training neural architecture search to generate compact LLM variants. These methods require GPU resources and training on curated datasets for flexible deployment tailored to a particular model type and size, e.g., L.ama-3. 1-8B, thereby restrict- ing their general applicability. In contrast, our work focuses on post-training on a single server GPU for the most common model architectures and supports on-device adaptation.
3 PROPOSED FRAMEWORK: UNIQL
3.1 NOTATIONS
Let T represent the sequence length. Da. Dud, Da, and Dine denote the hidden, head, state, and intermediate dimensions used in Transformer and Mamba blocks. D' is the post-pruning dimension.
H., Hev. and Hm are the number of attention heads, key-value heads, and SSM heads, respectively. G, is the number of SSM groups X E T and W E I Din Dant. are the activations and weights. C = X E R P.. Dim is the correlation matrix of X. The matrix S E R is associated with group of weights for sorting their columns and rows. We denote the element-wise multiplication, broadcasted outer product, and activation function as O, 3 and respectiell. U, E, and V denote the SVD decomposition, with eigenvalues d; on E's diagonal.
3

---

Published as a conference paper at ICLR 2026
MHSA
5 W
Mamba
MILP Pinw
State- aooord
(W WG- Mil
Quandariin--nee Sware
55] W6)
(Wa)
CC 030
P Norm
[Wa-ee)
(Mc
Sm W
MHSA
SauCCox U
fused RaPE p()
SSM
int (ps
D'
Conv (+)-
Ad
Ad
D'
D2
DA
W Wa.
W SAx
S.p
CTU,UE
W,5, W
MLP
(b) MHSA
(c) Mamba
Figure 3: (Joint weight decomposition.) We visualize the group of sorted weights in MLP (a), MHSA (b), and Mamba (c) blocks. The group of weights for joint decomposition is shown in the same background color, e.g., W, and W, in the pink background, and other groups are dis- tinguished by different colors. We devise different types of joint compression algorithms that are efficient and quantization-awan to support on-icvice pruning.
3.2 STRUCTURED WEIGHT SORTING
Our objective is to enable adaptive on-device pruning by sorting the weights according to their im- portance scores, allowing device to prune the least significant columns. Inspired by recent studies (Lin et al. 2025 Koike-Akino al.) (225)) we group the weights and conduct joint decomposition, as shown in Figure 3 We co-design the pruning algorithm alongside quantization and fused kernels for Transformers, SSMs, and hybrids.
Multi-layer perceptron (MLP). The MLP includes up W. projections Wa E
R R x Dint and down
Di. with an optional gate projection W E RDxx The formulation defined as [MI.((X) XW.)Wa To derive S. to sort the weight matrices in the MLP layer, we collect the intermediate activation Xnn = a (XW,) , XW. a from the calibration
(XW,, g) @
Table 1: (Pseudo-inverse.) Pseudo-inverse latency for FP64 matrices on A6000 (in minutes).
and calculate the channel corre- lation = X Xiat E R Dam Din, We average the correlation matrix over the calibration set, and compute the ridge leverage scores (McCurdy 2018)) defined by diag (C(C + A/)- We set in our experiments. We the scores and create a column sorting matrix Sm E R Pimt Dist that reorders the output columns for W,, and W as W S.S and W Sm and the input rows of the W as S,, Wa. d. as shown in Figure (a).
Matrix Size
Lat. (min.)
[1024, 1024]
0.02
ridge lambda
[4096, 4096]
0.57
[8192, 8192]
4.24
[14336, 14336]
20,58
Our approach does not rely on time-consuming pseudo-inverse to sort the MLP weight matrices. Although the pseudo-inverse (i.e., Moore-Penrose inverse (Penrose 1955)) provides a theoretical bound in pruning errors (Lin et al., 2025), it exhibits three major drawbacks: (1) Pseudo-inverse has complexity of (n)) for a n-size squared matrix. This is particularly time-consuming when computing the pseudo-inverse of correlation matrices in MIP layers because Dint is a large number in most LLM designs, e.g., Llama-3-8B D = 14336. (2) Pseudo-invers inverse computation requires high-precision FP64 to maintain numerical stability (Lin et al.) 2025), which demands substan- tial memory usage for full-precision weights. (3) Matrix inverse breaks the equivalence of pruned (Wiy, requiring recomputation for different pruning rates. Here, W DDD, W' is a submatrix of W. where W DxD' with D D'. Wi represents the inverse matrix. We show the latency of pseudo-inverse computation for FP64 matrices on A6000 in Table
weights, resulting in (W')f
Multi-head self-attention (MHSA). For simplicity, we set the attention heads H, and the key- value heads H&. equivalent. The formulation of 7th head within the MHSA is provided fol-
4

---

Published as a conference paper at ICLR 2026
SVD-LLME W= (UVE) (VEV)
rrot
X
Sin/Cos
kernel
Gather amb.
(XL d))
Pi Quantization friendly
P2
Indices
7x
Quantization untrondly
syy
MoDeGPT: W
valuea
hal
(X,; 8)
Fused Pyforch
gather
P20
head
Ith head
Quantization unfilendly
D2ma/2
Quentization-aa
(Ours): W
(UE)V
Polto
X
Quantization friendly
group
(a) Fused RoPE kemne
(b) SVD decomposition
Figure 4: (The fused kernel and SVD decomposition.) In the left illustration, gathering and slicing rotary positional embeddings by the index vector for Q and K are fused in one kernel to reduce memory access. The embeddings for the pruned head dimension Día are gathered from the index array Say in the fused kernel. On the right, we combine the diagonal matrix E with U as the group shares a quantization scaling factor to reduce the quantization errors.
lows: fMHSA(X, 1) = Softmax (p (XW:) P (XW1) 1) (xw; W:), where p(-) denotes RoPE SSu et al. 2021). The weights in MHSA are divided into two groups: { (W", qr W and (W; W For the W and W we obtain activations X = (XW;) and X (XW;), and compute
the channel correlation C² x x and C = x X where C₁ and C₁ E Dhax The
1/2 Il. and averaged over the calibration sample. Since the embedding positions are broken by our structured sorting, we have to gather the corresponding indices for the rotary positional embeddings, i.e., sin and cos. Since RoPE is expressed as RoPE(X;; 0) = cos @ X + sin @ R(X), where R(X) rotates by splitting X into two components along the last axis: X [x1 such that R(X) 1-x2, x1]. we apply symmetric sorting Ssym 81 + 82, where { (y $1, $2) E Pha/2 and = [2]. As such, we
sorting scores s E Pha are calculated as s = ||c,1/21 || ||c
construct the final sorting matrix Syx E IR Daa x Dnd with [8aym: Ssym] that sorts the output columns
as W Sg and W.S Symmetric sorting is hardware-efficien since we only need to store and load half of the index vector into our fused RoPE kernel, illustrated in Figure 4 (a).
2025a For the W and W or we perform an activation-scaled SVD decomposition WWang et al.
Yuan et al. 2023). With the input correlation matrix C = X X, we follow Lin et al. 2025 to per- form joint decomposition by consecutive SVD operations: C W: SVD W W U. V w = U, U,,.VD(E, VT, W: = UU UEVT. The SVD decomposition ranks the eigen- vectors by eigenvalues. To reduce quantization errors, we fuse the diagonal matrix & into W unlike prior art (Wang et al.] [2025 Lin et al. 2025). We present the final sorted weights with quantization- aware SVD decomposition as W C U.UE and W VT. as shown in Figure (b). Low-bit quantization (i.e.. 3- or b-t) is sensitive to the distribution within the quantization group. We de- construct the weight matrix W- UEV by merging the long-tailed eigenvalues Σ with U, such that W = (, where each column of is scaled by its eigenvalue Ø; Thus, o; acts as the scaling factor for the group without distorting the distributions, as depicted in Figure
We show the details of the proposed algorithms in Algorithm 2 and 3 This joint weight decomposi- tion also supports Grouped-Query Attention (GQA) (Ainslie et al. 2023). as shown in Figure (b) and Algorithm and in Appendix A
Mamba.
The Mamba block encompasses five primary weight matrices. For simplicity, we decom-
pose the entire computation and express the SSM's 7th head and 9²d group as /Mambal (X, i, i,g) =
Norm (o(XW!) @ SSM(AA, @(XW),, A((XW)) B), ((WW)) w) W where {Wi, 2y W E RP x Dua, and (Wg. B. W E RD,DD, and the output weight W E R Pi, Norm and d(-) denote normalization and 1D causal convolution layer fused with activation, respectively. The
5

---

Published as a conference paper at ICLR 2026
SSM(.) function performs the linear recurrence computations h4 A,A + A, A, Cehy for the 1th head and g'h group at each time step t with a parameterized step size A (Dao & Gu
2
2024.. AA and AB" (o(WW)) B) are the matrix forms of A, and A, H. is the matrix form of the SSM state hy. To perform the joint weight decomposition, we first break the computation of
Mamba block into two sub-formulations: (1) SSM input mask M: fM((X,) g) C* (*
((W))) C) (A)(XW))) and (2) SSM state H: fx (X, 9: i, ho) = (ha( + A with an initial state ho- We denote X (XW",, the output of the causal convolution activation.
We first focus on sorting the weights {W B> W that compute the SSM input mask M. For the SSM group g with H Hm SSM heads in the group, we obtain the activations B Φ (X and C = & () where BD and C E RT''D, Unlike self-attention, B is discretized using the input-dependent variable A E Rx by a broadcasted outer product (AB)² A 60 B E HH xTxD, As result, we compute the channel correlation AC₁, = (AB) (AB)* and CE C9 CC, where ACg E IR. H x xD. and Cg E R.D.D. The sorting scores & E R.D. are calculated as 8 Er=oll ( ) @ |Icg 1/2 and averaged over the calibration samples. We use s to construct the sorting matrix SBC E D.D.D, D. that sorts the output columns of W and W as SB and BCS as shown in Figure (c).
H
We propose a state-aware method to compress the other group of weights {W 21 W x1 W by collecting the correlations from the SSM states, Hi E R(TxDD)xDDh. such that CR = Hi Hi, Cg E PRaaax and averaging over the calibration samples. The ridge leverage score diag (CH(CH + +))- is computed as the MLP layer, where we set ridge lambda À = 1. We use the scores and design a column sorting matrix S. E R Phax x Dad that organizes the output columns for W and W as Wis, and W.S.. The input rows for W are sorted accordingly by S. W Figure (c) illustrates these sorted weights.
3.3 MASKED LORA FINE-TUNING
We conduct a LoRA- Hu et al. previous work (Wang et al., 2025b a) that fine-tunes the pruned model, we fine-tune the un-pruned sorted model in one shot, as shown in Figure
2022) recovery fine-tuning (FT) on the sorted model. Unlike
2 We derive the layer-wise pruning rates rı using
Block Influence (BI) scores (Lin et al. 2025 Men al. 2024) for all pre-determined global pruning rates P [PIG: P20....,, such as Pis
The BI score is defined as s =
y: 1 where the 2y and y/1 represent the input and output of the 7th layer, respectively. During fine-tuning, we randomly draw a pruning rate Pt ~ P at time step to mask out the pruned channels. We follow the prior work (Wang et al., 2025b) to perform instruction tuning on the Alpaca dataset TTooi et al. (023) for five epochs. The entire fine-tuning process is conducted on a single cloud GPU. Our sorted model provides configurable pruning rates on the device after one-shot masked fine-tuning. We note that our fine-tuning inherently supports downstream tasks for any application,
[xx|I
such as summarization and question-answeringg datasets.
3.4 QUANTIZATION AND ON-DEVICE ADAPTIVE PRUNING
We quantize the fine-tuned full model to minimize the on-device storage needs. We employ group-
wise uniform symmetric quantization, the most commonly supported method by hardware, to convert floating-poin values into --bit discrete form. The quantization function for a group of
weight W(t.)) in column i is defined as W(i.g) = Clamp W ( -2 N 1211,1211111
-1) 1). where
8 = Max W (..g) (ax))W(.)))))/ (2/(22111)))) - 1) is the scaling factor (i.e., quantization step). For the quantization- aware SVD decomposition, we fuse the eigenvalues of the ith column to the group of weight factor such that o, W(ig) We set N = 4 and group size 128 and adapt GPTQ (Frantar et al. 2023) to quantize our models. We fuse Hadamard matrices into the weights and apply 4-bit quantization to the embedding and output layers. The parameters of normalization layers are fused to the weights before applying quantization. After deploying the quantized model, we prune the channels on the device by reducing the intermediate dimension Dint in the MLP layer, head dimension Dna in the MHSA layer, and both the state dimension D, and head dimension Dnd in the Mamba layer. We

---

Published as a conference paper at ICLR 2026
Table 2: (Compared with structured pruning.) We compare all models in FP16. UniQL en- ables all compression rates in single pass. The symbols o, t. and $ denote Transformers, Mamba- Transformer hybrids, and Mamba models, respectively.
Llama-2
Llama-3.1
Qwen-2.5 Bamba-v2 Nemotron-H Mamba2
Prun. % Prun. Method
+FT
7B
8B
7B
9B
8B1
8B
0%
68.8%
74.0%
72.4%
74.6%
76.0%
70.6%
MoDeGPT
66.2%
72.4%
52.1%
SVD-LLM
56.3%
56.7%
62.6%
15%
UniQL (Ours)
66.7 %
70.5%
69.1
70.9% %
68.9%
65.6%
SVD-LLM
64.7%
64.5%
69.5%
UniQL (Ours)
67.2%
71.9% %
70.0%
72.9 %
73.0%
66.4%
MoDeGPT
63.4%
64.9%
40.8%
SVD-LLM
50.8%
45.8%
53.2%
25%
UniQL (Ours)
63.7% %
67.0%
62.1 %
66.4% %
60.6%
59.8%
SVD-LLM
62.4%
59.5%
66.8%
UniQL (Ours)
64.9 %
69.6%
65.8%
69.7%
67.3%
62.7 2
Table 3: (Compared with PTQ.) We benchmark all weight-only PTQ methods without fine-tuning and pruning. represents the FP16 embeddings and output layers as per the official implementation. denotes the GPTQ (Frantar al. 2023 implemented all models an additional baseline.
Llama-2
Llama-3.1
Qwen-2.5
Bamba-v2 Nemotron-H Mamba2
PTQ Method
W-bit
7B
8B
7B
9B*
8B1
8B
FP16
16
68.8%
74.0%
72.4%
74.6%
76.0%
70.6%
TRT-AWQ
4"
68.1%
71.9%
70.3%
TAQ-HQQ
4"
68.4%
72.4%
72.1 %
UniQT. (Ours)
68.2% 4"
72.9%
72.0%
74.8%
74.9%
69.3%
GPTQ
4
67.9%
71.3%
70.0%
73.6%
74.9%
68.1%
UniQL. (Ours)
67.8%
72.3%
71.0%
73.8%
74.8%
69.3%
keep the hidden state dimension Da. the same across all pruned models. For INT4 weights, we unpack them online, prune channels, and repackage them into INT32 for the 4-bit kernel.
EXPERIMENTAL RESULTS
4.1
SETUPS
Models and setups. We experiment with Transformers Llama-2-7B (Touvron et al. 2023). Llama-
3.1-8B Meta 2024), Qwen-2.5-7B (Hui et al.) 2024). hybrid models Nemotron H-8B (B Blakeman cet al. 2025). Bamba-9B-v2 IBM 2025), and the SSM model Mamba-2-8B (Dao & Gu 2024). o, t. and $ denote Transformers, hybrid and SSMs, respectively. FT, PTQ, and W-bit stand for fine-tune, post-training quantization, and the bit-width of weights. Prun. and R.size represent the pruning rate in percentage (%) and the reduction of model size (x), respectively.
Structured pruning baselines. We compare UniQL. to cutting-edge model compression methods, MoDeGPT (Lin al., (2025) and SVD-LLM (Wang et al. 2025b). As MoDeGPT not publicly available, we duplicate their method based on the paper and achieve similar accuracy to what was reported. We adapt SVD-LLM official implementation to experiment with Llama-2-7B. Llama-3.1- 8B, and Qwen-2.5-7B for compression and quantize models. For MoDeGPT and SVD-LLM, we adhere to the hyper-parameters outlined the papers.
Post-training quantization baselines. We adopt AWQ (Lin et al. 2024a) in TensorRT-MO' (TRT-AWQ) (NVIDIA) 2024 2023) and HQQ (Badri & Shaji 2023) in TorchAO" (torchao) 2024) (TAO-HQQ) as our quantization baselines, both W4A16 libraries ready for PTQ. We evaluate UniQL in terms of model size, average accuracy downstream tasks, and latency A6000 and Nano 8G against TRT-AWQ and TAO-HQQ. our experiments, we quantize the embedding and output (i.e,, 1m head) layers to 4 bits, cutting memory usage in contrast to TRT-AWQ" and TAO-
7

---

Published as a conference paper at ICLR 2026
Table (One-pass adaptive pruning.) We evaluate UniQL in one run across pruning rates. R.size stands for reduction of model size (x). UniQL enables all pruning rates in single pass. The symbols o, f. and & denote Transformers, Mamba- Tranb---ranfommer hybrids, and Mamba models, respectively. represents the FP16 embeddings and output layers as per the official implementation. We apply
GPTQ (Frantar et al. 2023) on MoDeGPT (Lin et al.
2025) denoted as "
One
W
Prun.
R.size
Llama-2
Llama-3.1
Qwen-2.5
Bambu-v2 Nemotron-H Mamba2
Method
pass
+FT
bit
%
(x)
7B
8B
7B
9B1
8B1
SB1
FP16
16]
0%
0x
68.8%
74.0%
72.4%
74.6%
76.0%
70.6%
15%
4.7x
63.7%
64.2%
52.2%
MoDeGPT
25%
4.7x
60.3%
59.3%
48.4%
x
4"
15%
4.7x
63.2%
60.6%
66.8%
SVD-LLM
X
4"
25%
4.7x
59.1%
54,2%
64.6%
0%
4x
67.6%
73.6%
72.4%
15.1%
73.3%
69.3%
UniQL (Ours)
15%
4.7x
65.6%
71.4%
68.1%
70.3%
70.5%
65.8%
4
25%
5.3x
63.5%
67.7%
64.0%
67.4%
64.7%
61.8%
35%
6.1
61.0%
62.7%
58.1%
62.7%
59.0%
57.7%
HQQ" which use FP16. We also adapt GPTQ (Frantar et al. 2023) for all models and also use 4-bit quantization the embedding and output layers an additional baseline.
Datasets and evaluations. We evaluate UniQL on five zero-shot tasks with batch size of 16, including HellaSwag (Zellers al. 2019), PIQA (Bisk et al., 2020). ARC (Clark et al. 2018). and
WinoGrande (Sakaguchi et al. 2020) using LM-EVAL (Gao et al. 2023). The average of Wino-
Grande, PIQA, and ARC-easy (accuracy), and HellaSwag wag and ARC-challenge (length-normalized accuracy) is reported for experiments. More evaluations are placed Appendix B
Implementations and environments. Our kernels are adapted from the 4-bit kernels (Frantar et al. 2024) and RoPE kernels (Hsu al. 2025). All computations in BF16, except for cor- relation matrices for structured sorting and Cholesky decomposition for GPTQ are calculated in FP32. Detailed parameters are placed Appendix F The weight-sorting, masked fine-tuning, and quantization are computed on an A6000 GPU with 48GB memory in one-shot for enabling adaptive pruning on the device. We profile the latency on A6000 and Orin Nano 8GB, as our experimental cloud and edge platforms. We report the average latency of twenty profiles after five warmup runs.
4.2 ZERO-SHOT DOWNSTREAM TASKS
Comparison with structured pruning. Table 2 compares structured pruning baselines against UniQI.. Without fine-tuning, UniQL. outperforms both MoDeGPT and SVD-LLL in most cases, achieving strong results such as 66.7% on Llama-2-7B and 69.1% on Qwen-2.5-7B. With fine-tuning, UniQL further boosts performance, reaching 67.2% on Llama-2-7B and 70.0% on Nemotron-H-8B. surpassing SVD-LLM consistently. MoDeGPT suffers from the ill-conditioned correlation matrices C E R Das and numerical instability when D >> Dj, with limited cal- ibration samples, resulting in large accuracy drops in Qwen-2.5-7B [Dint: D (18944, 3584). SVD-LLM truncates numbers the eigenvalues in the decomposed weight matrices according to the desired compression rates, requiring fine-tuning to recover the performance. These results highlight UniQL's effectiveness in preserving task performance while enabling efficient model compression.
Comparison with post-training quantization. Table 6 presents the comparison of weight-only post-training quantization (PTQ) methods. Across all models, UniQL demonstrates highly compet- itive performance, matching surpassing existing PTQ methods in several settings. For example, UniQL achieves 72.9% on 1.ama-3.1-8P with 4-bit layers and FP16 embedding/output layers. No- tably, while TAO-HQQ slightly edges out UniQL on Llama-2-77 and Qwen-2.5-7B, UniQL is more general to different architectures, providing adaptive pruning features on-device. UniQL surpasses equals GPTQ, the baseline we modify for all models.
One-pass adaptive pruning. Table evaluates the one-pass adaptive pruning under 4-bit quanti- zation with fine-tuning of UniQL. We compare UniQL against SVD-LLM baselines, which follow
TTee embedding and output layers use FP16 according to the official implementation.
8

---

Published as a conference paper at ICLR 2026
similar compression process but support only single compression rate per run. Without any pruming. UniQL achieves competitive results, for example, 67.6% on Llama-2-7B and 73.6% on Llama-3.1-8B, while reducing the model size by 4 x. As pruning ratios increase, UniQL main- tains graceful degradation in accuracy; at 15% pruning, it still achieves 71.4% on Llama-3. 1-88B and 70.5% Nemotron-H-8hh outperforming SVD--LLM across all comparable settings. At higher compression (e.g,, 35% pruning), UniQL still delivers reasonable performance, such as 62.7% on Llama-3.1-8B and 57.7% on Mamba2-8B. These results demonstrate UniQL's strong adaptability, generalizing to wide range of architectures.
4.3 COMPRESSION TIME
In Table 5 we compare the compression time of UniQL against MoDeGPT (Lin al. 2025). noted for being training-free, and SVD-LLM (Wang et al., 2025b), which involves fine-tuning, both state-of-the-ar algorithms for Transformer compression. Our matrix decomposition 22x (0h19m vs. 7h03m) faster than MoDeGPT and 1.8x (0h19m vs. 0h35m) faster than SVD-LLM, as UniQL avoids pseudo-inverse and SVD decomposition for large MLP weight matrices. With masked fine-tuning (FT), UniQL remains quicker (6h59m) than both MoDeGPT (7h03m) and SVD-LLM (15h57m). MoDeGPT suffers from the high computation cost of performing pseudo-inverse on large weight matrices. SVD-LLM splits weights into two successive layers, U and V, and carries out independent fine-tuning for each, leading to a longer fine-tuning duration. Lastly, post-training quantization (PTQ) takes extra forty minutes. Our compression algorithm is one-time O(1) with respect to the number of compression rates, compared to (n) for MoDeGPT and SVD-LLM.
Table 5: (Compression time.) The time is reported on an A6000 GPU. UniQL. supports all compression rates in one shot.
Table 6: (Model size.) The model size reported in GB. Our 4-bit embedding/output layers yield smaller size than TRT-AWQ and TAO-HQQ.
Method
+FT +PTQ
1lama-3.1
8B
Mamba2 8B
Method W-bit Prun. p%
Llama-3.1 8B
Qwen-2.5
7B
Nemotron
8B'
MoDeGIT
7h03m
FP16
16
0%
16.0 GB
15.2 GB
16.2 GB
0h35m
MoDeGPT
16
15%
13.9 GB
13.2 GB
SVD-LLM
V
16h25m
SVD-LLM
16
15%
14.1 GB
13.3 GB
V
16h46m
TRT-AWQ TAO-HQQ
4" 0%
5.8 GB
5.6 GB
Oh 1h1
Oh 1m
4"
0%
5.7 GB
6,0 GB
UniQL
(Ours)
6h59m
1h18m 7h50m
UniQL (Ours)
0%
4.1 GB
3.9 GB
4.1 GB
7h43m
35%
2.8 GB
2.7 GB
2.9 GB
4.4
MODEL SIZE AND LATENCY PROFILING
We evaluate the model size and latency of UniQL and compare them with AWQ (Lin et al. 2024a)
from TensorRT-MO" IVVIII 2024 2023) (TRT-AWQ) and HQQ (Badri Shaji 2023) in Tor- chAO" (torchao 2024) (TAO-HQQ). Both libraries weight-only (i.e., W4A16) quantization frameworks in production. We show the model size in Table 6 UniQL quantizes models in head- to-toe fashion, i.e., from embeddings, backbone layers, and output heads all at 4-bit, resulting in smaller model size GGB vs. 5.7GB) compared to TRT-AWQ and TAO-HQQ with minimal accu- racy drops. Our model enables all compression rates and on-device structured pruning, providing an elastic 3.9x- 5.7x memory reductions. We profile the time-per-output-ooie tion) and time-to-last-toenn (TTLT, i.e., prefilling and generation) on A6000 and Nano 8G, and show 2.7x 3.4x throughput improvements in generation, outperforming TRT-AWQ and TAO-HQQ. as shown in Table 7 and Table 8 On the Nano 8G, our model is 1.7x faster than TAO-HQQ in TPOT. By pruning 35% of weights in our 4-bit model, our models generate 2.1 x faster than TAO-HQQ.
(TPOT, i.e., genera-
5 ABLATION STUDY
We conduct an ablation study to demonstrate the effectiveness of each component of our framework. Additional ablation studies can be found at Appendix C
"The embedding and output layers use FP16 according to the official implementation.

---

Published as a conference paper at ICLR 2026
Table 7: (Lateney profiling on an A6000.) Re- ported TPOT and TTLT (1k+1k) are in mss.
Table 8: (Latency profiling on a Nano 8G.) TPOT and TTLT (256+256) shown in ms. (OOM: out-of-memory)
Prun.
Llama-3.1-88
Nemotron-H-S3
Method
W-bit
p%
TPOT
TILT
TPOT TILT
Prun.
Qwen-225-7B"
Mamba2-88
Method
W-bit
FP16
16
0%
25.0
26653.8
24.4 25889.4
psa
TPOT
TTLT
TPOT
TTLT
TRT-AWQ
4"
0%
11.4
10130.4
FP16
16
0Si
0OM
0OM
OOM
0OM
TAO-IIQQ
4"
0%
10.2
11639.5
TAO-HQQ
0%
129.8
38567.9
UniQL 0%
9.0
9944.6
8.2
9095.7
UniQL
0%
75.8
19447.0
81.6
21139.1
(Ours)
35%
7.3
8105.4
6.8
6955.6
(Ours)
35%
55.5
13792.0
58.2
14892.5
Table 9: (The fused RoPE kernel.) We profile TPOT on A6000 and report in
Table 10: (Accuracy by Components.) We report the average accuracy for the models with different settings.
Prun.
Fused
Llama-3.1
Qwen-2.5
Prun. p%
Llama-3.1 8B
Qwen-2.5
W-bit
W-bit
+FT +PTQ +QSVD
p%
RoPE
8B
7B
7B
0%
25.0
23.2
16
0%
74.0%
72.4%
16
25%
/
20.2
19.3
17.9
18.7
16
25%
:
:
69.6%
67 0%
65.8%
62. 1
9.9
9.1
y
55.2%
56.1%
0%
/
9.0
8.3
4
25%
65.0%
60.7%
8.6
7.9
60.2%
61.0%
25%
/
7.7
7.1
/
67.7%
64.0%
Fused rotary positional embedding. We compare latency with and without our fused RoPE. Since the positions are broken by our structured sorting, we have collect the corresponding indices from the rotary positional embeddings, where we fuse in a kernel to minimize memory access. The fused RoPE kernel with index gathering yields a 10% latency reduction (1.1x speedup) for Llama- 3.1-8B in 4-bit models at 0% and 25% compression, as depicted in Table
Masked LoRA fine-tuning. We show that masked LoRA fine-tuning (FT) significantly benefits pruned models. As seen in Table 10 method enhances accuracy by 2.6% (from 67.0% to 69.6%) and 3.7% (from 62.1% to 65.8%) ())oo for FP16 Llama-3. and Qwen-2.5-7B at 25% compression. For bit models, it improves accuracy by 2. 2.7% (from 65.0% 67.7%) and 3.3% (from 60.7% to 64.0%) for Llama-3.1 1-B and Q.5-72.5-7B at 25% compression.
Quantization-aware decomposition. We show the quantization-awarc SVD decomposition (QSVD) is a key design to fill the performance gaps in Table [0] Low-bit quantization (i.e., INT4) sensitive to the numerical distribution the quantization group. We decompose the weight ma- trix W = -UV and combine the onilld eigenvalues Σ with U, resulting in W (UE)V, where a column of is multiplied by the corresponding eigenvalue a.. Thus, a; acts as the group's quantization scaling factor, as shown in Figure 4 We show this simple observation leads to sig- nificant performance gains 7.5% (from 60.2% to 67.7%) and 3% (from 61.0% to 64.0%) for 4-bit Llama-3.1 and Qwen-2.5-7B the 25% pruming rate, respectively.
CONCLUSION
We present UniQL, a unified post-training compression framework that combines quantization and low-rank pruning to enable adaptive deployment of LLMs on edge. By supporting on-device con- figurable pruning and a one-shot cloud compression pipeline, UniQL addresses the key challenges posed by dynamic workloads. Through structured weight-sorting, quantization-awar decomposi- tions, and fused rotary kernels, UniQL achieves substantial gains in memory and throughput across Transformers, SSMs, and hybrid models. Our results demonstrate that the compressed models can elastically adapt runtime constraints.
10

---

Published as a conference paper at ICLR 2026
IMPACT STATEMENT
The UniQL framework has the potential to make large language models more accessible by en- abling their deployment on edge devices with limited resources. This could broaden the scope of applications beyond high-end servers, potentially benefiting settings such as education, accessibility tools, or low-resource regions. At the same time, the increased availability compact models raises concerns about potential misuse, including privacy risks and the generation of harmful or mislead- ing content on widely distributed devices. Reducing the computational and memory footprint may also lessen the environmental costs of running large models, though the overall impact depends the scale of adoption and usage patterns. We emphasize that UniQL itself does not mitigate so- cietal risks associated with language model outputs, and responsible deployment practices remain necessary. By releasing our code and models, we aim to facilitate further research on efficient adap- tation while encouraging the community carefully consider both the benefits and risks of enabling lightweight edge deployment.
ACKNOWLEDGMENTS
This work supported in part by NSF CCF Grant No. 2107085, iMAGINE the Intelligent Machine Engineering Consortium at UT Austin, UT Cockrell School of Engineering Doctoral Fel- lowships, NSF CAREER Grant No. 2339084, Nvidia research gift, and Taiwan's NSTC Grant No. 11-2221-E-449-148-MYY3
REFERENCES
Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305. 22023305.1345.. 13245, 2023.
Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations, 2024a.
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated 1lms. Advances in Neural Information Processing Systems, 37:100213-1002400 2024b.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski. David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv: 108.077872 2021.
Hicham Badri and Appu Shaji. Half-quadratic quantization of large machine learning models, November 2023.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAT), 2020.
Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Alek- sander Ficek, Alexis Bjorlin, Ali Taghibakhshi, Amala Sanjay Deshmukh, Ameya Sunil Maha- baleshwarkar, al. Nemotron-h: family of accurate and efficient hybrid models. arXiv preprint arXiv:2504.03622 2025.
Ruisi Cai, Saurav Muralidharan, Greg Heinrich, Hongxu Yin, Zhangyang Wang, Jan Kautz, and Pavlo Molchanov. Flextron: Many-in-one flexible large language model. In International Con- ference on Machine Learning, pp. 5298-5311 PMLR, 2024.
Ruisi Cai, Saurav Muralidharan, Hongxu Yin, Zhangyang Wang. Jan Kautz, and Pavlo Molchanov. Llamaflex: Many-in-one Ilms via generalized pruning and weight sharing. The Thirteenth International Conference Learning Representations, 2025.

---

Published as a conference paper at ICLR 2026
YuenYu Chi, Hung- Yueh Chiang, Chi-Chih Chang, Ning- Chi Huang, and Kai-Chiang Wu. V "mean" ba: Visual state space models only need hidden dimension. In Workshop on Machine Learning for Systems Advances in Neural Information Processing Systems (NeurIPSW), 2024.
Hung- Yueh Chiang, Chi-Chih Chang, Natalia Frumkin, Kai-Chiang Wu, Mohamed S. Abdelfattah, and Diana Marculescu. Quamba2: A robust and scalable post-training quantization framework for selective state space models. In International Conference on Machine Learning (ICML), 2025a.
Hung-Yueh Yueh Chiang, Chi-Chih Chang, Natalia Frumkin, Kai-Chiang Wu, and Diana Marculescu. Quamba: A post-training quantization recipe for selective state space models. In International Conference Learning Representations (ICLR), 2025b.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the A12 reasoning challenge. CoRR, 2018.
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. Forty-first International Conference on Muchine Learning. 2024.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International conference on machine learning, pp. 10323-10337. PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training
compression for generative pretrained transformers. In International Conference on Learning Representations (ICLR), 2023.
Elias Frantar, Roberto L Castro, Jiale Chen, Torsten Hoefler, and Dan Alistarh. Marlin: Mixed-precision auto-regressive parallel inference on large language models. arXiv preprint arXiv:2408.1177 11743, 2024.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 2023. URL httpp:://eenod: org/ records/ 10256836
Martin Genzel, Patrick Putzky, Pengfei Zhao, Sebastian Schulze, Mattes Mollenhauer, Robert Sei- del, Stefan Dietzel, and Thomas Wollmann. Compressing large language models any size with- out re-computation. In ES-FoMo IlI: 3rd Workshop on Efficient Systems for Foundation Models, 2025.
Yuxian Gu, Qinghao Hu, Shang Yang, Haocheng Xi, Junyu Chen, Song Han, and Han Cai. Jet-nemotron: Efficient language model with post neural architecture search. arXiv preprint arXiv:2508.15888 15884, 2025.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.00 001::20099.3300, 2020.
Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song. Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, Yanning Chen, and Zhipeng Wang- Liger-kernel: Efficient tri- ton kernels for LLM training. In Championing Open-source DEvelopment in ML Workshop @ net. ICML25, 2025. URL https:tt//// http//ppeeevvieeeee httpps://operreiieeeeeee htt.rrriwww..tttfooorrr/.oa3-///
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Alle Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. International Confer- ence on Learning Representations (ICLR), 1 1(2):3, 2022.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang. Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409, 12186, 2024.
12

---

Published as a conference paper at ICLR 2026
Bamba IBM. Bamba-9b-v2 fast and powerful!
bttp:///uugginofacccco//LL.
1bm-as adlpplatform/bamblee 9b 2025.
Toshiaki Koike-Akino, Xiangyu Chen, Jing Liu, Ye Wang, Matthew Brand, et al. Latentllm: intention-aware joint tensor compression. arXiv preprint arXiv:2505. 2025.
Yun Li, Lin Niu, Xipeng Zhang, Kai Liu, Jianchen Zhu, and Zhanhui Kang. E-sparse: Boost- ing the large language model inference through entropy-based n: M sparsity, arXiv preprint arXiv:2310. 15929, 2023.
Chi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. Modegpt: Modular decomposition for large language model compression. In International Conference on Learning Representations (ICLR), 2025.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awan aware weight quantization for on-device IIm compression and acceleration. Proceedings of machine learning and systems, 6:87-100, 2024a.
Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. Qserve: W4a8kv4 quantization and system co-design for efficient Ilm serving. arXiv preprint arXiv:2405. 2455522400.00552, 2024b.
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Llm quantiza- tion with learned rotations. In The Thirteenth International Conference on Learning Representa- tions, 2025.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruned On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-21720, 2023.
Shannon McCurdy. Ridge regression and provable deterministic ridge leverage score sampling. Advances in Neural Information Processing Systems, 31, 2018.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXhv:2003.38853, 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations. 2017.
AI Meta. Llama 3: Open foundation and instruction-tunae language models, 2024.
Mohammad Mozaffari, Amir Yazdanbakhsh, and Maryam Mehri Dehnavi. Slim: One-shot quan- tization and sparsity with low-rank approximation for Ilm weight compression. In Forty-second International Conference on Machine Learning, 2025.
J Pablo Muñoz, Jinjie Yuan, and Nilesh Jain. Mamba-shedder: Post-transformer compression for efficient selective structured state space models. arXiv preprint arXiv:2501. 17088, 2025.
NVIDIA. TensorRT-LLMM: High-performance inference for Large Language Models.
https:
wtttttttbccon////iii///esssttt- 2023. Accessed: 2025-09-13.
NVIDIA. TensonRTT-Model-Oppiiizee Quantization and Optimization Toolkit for LLMs. https:
2024. Accessed: 2025-09-13
Yeonhong Park, Jake Hyun, Sanglyul Cho, Bonggeun Sim, and Jae W Lee. Any-precision Ilm: Low-cost deployment of multiple. different-sizen sized 1lms. In International Conference on Machine Learning, pp. 39682-39701. PMLR, 2024.
Roger Penrose. A generalized inverse for matrices. In Mathematical proceedings of the Cambridge philosophical society, volume 51, pp. 406-413. Cambridge University Press, 1955.
13

---

Published as a conference paper at ICLR 2026
Wang Qinsi, Jinghan Ke, Masayoshi Tomizuka, Kurt Keutzer, and Chenfeng Xu. Dobi-SVD: Differ- entiable SVD for LLM compression and some new perspectives. In The Thirteenth International Conference Learning Representations, 2025.
http://ee////ttteeeeeeee forum?
1dd=kw76615xBB
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. In The Thirty-Fourte AAAI Conference on Artificial Intelligence (AAAI), 2020.
Ibne Farabi Shihab, Sanjeda Akter, and Anuj Sharma. Efficient unstructured pruning of mamba state-space models for resource-constraiig environments. arXiv preprint arXiv:2505.08299 2025.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: en- hanced transformer with rotary position embedding. arxiv. arXiv preprint (XXi:2214.098860 2021.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Ali Taghibakhshi, Sharath Turuvekere Sreenivas, Saurav Muralidharan, Marcin Chochowski, Yashaswi Karnati, Raviraj Joshi, Ameya Sunil Mahabaleshwarkar, Zijia Chen, Yoshi Suhara, Oluwatobi Olabiyi, et al. Efficient hybrid language model compression through group-aware ssm pruning. arXiv preprint arXiv:2504. /1409, 2025.
Endri Taka, Ning- Chi Huang, Chi-Chih Chang, Kai-Chiang Wu, Aman Arora, and Diana Mar- culescu. Systolic sparse tensor slices: Fpga building blocks for sparse and dense ai acceleration. In Proceedings of the 2025 ACM/SIGDA International Symposium on Field Programmable Gate Arrays, pp. 171, 2025.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: an instruction foollooion Ilama model (2023), 2023.
torchao. Torchao: Pytorch-native training-to-serving model optimization, oct 2024. URL https:
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 207iv:2307.09288, 2023.
Kaiwen Tuo and Huan Wang. Sparsessm: Efficient selective structured state space models can be pruned in one-shot. arXiv preprint arXiv:2506.0661 03611, 2025.
Xin Wang. Samiul Alam, Zhongwei Wan, Hui Shen, and Mi Zhang, Svd-lim v2: Optimizing singu- lar value truncation for large language model compression. In Proceedings the the 2025 Conference of the Nations of the Americus Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 4287-4296, 2025a.
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD-LLM: Truncation aware singular value decomposition for large language model compression. In International Conference Learning Representations (ICIR), 2025b. URL. httph:tt////c.... http::://openrevieee ew. net. forum? i
LNYTUouhdE
Xinghao Wang, Pengyu Wang, Bo Wang, Dong Zhang, Yunhua Zhou, and Xipeng Qiu. Bitstack: Aizesize compression of large language models in variable memory environments. In The Thir- feenth International Conference on Learning Representations, 2025c.
Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu, Yong Li, Wei Lin, and Shuaiwen Leon Song. Flash-llm: Enabling cost-effective and highly-efficient large gen- erative model inference with unstructured sparsity. Proceedings of the VLDB Endowment, 17(2): 211 111-224, 2023.
14

---

Published as a conference paper at ICLR 2026
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: quantization for large language models. In International
Accurate and efficient conference on machine learning, pp. 38087-38099, PMLR, 2023.
Zukang Xu, Yuxuan Yue, Xing Hu, Dawei Yang, Zhihang Yuan, Zixu Jiang, Zhixuan Chen, Sifan Zhou, et al. Mambaquant: Quantizing the mamba family with variance aligned rotation methods. In The Thirteenth International Conference on Learning Representations, 2025.
Zhihang Yuan, Yuzhang Shang, Yue Song. Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.0582 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics (ACL), 2019.
Zheng Zhan, Zhenglun Kong, Yifan Gong, Yushu Wu, Zichong Meng, Hangyu Zheng, Xuan Shen, Stratis loannidis, Wei Niu, Pu Zhao, et al. Exploring token pruning in vision state space models. Advances in Neural Information Processing Systems, 37:50952-50971. 2024.
Yilong Zhao, Chien- Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krish- namurthy, Tianqi Chen, and Baris Kasikei. Atom: Low-bit quantization for efficient and accurate Ilm serving. In Proceedings of Machine Learning and Systems (MLSys), volume 6, pp. 196- 209, 2024. URL
wrgp:///www.....////////....../. http/pppper_fllles/pee///222 f  Fi22-paabb7b00000000eeeeeeeeeee2 Paper Paper-Confereece..ddd
15

---

Published as a conference paper at ICLR 2026
A DETAILED STRUCTURED SORTING ALGORITHMS
A.I MULTI-LAYER PERCEPTRON
A Doutiinss the structured sorting algorithm for an MLP. The MLP comprises up and down projections, with optional gate projection. To obtain S. for sorting weight matrices in MLP layer, we gather the intermediate activation Xint from the calibration set and compute the channel correlation. We then average this correlation matrix over the set and compute the ridge leverage score. Using these scores, a sorting matrix Sn that sorts the output columns of Wa and W 9 as W,Sm and WySm- The rows of Wa are arranged as S Wa accordingly. We set ridge lambda in our experiments.
Algorithm Structured sorting for MLP.
1: Input: Up projection Wu. E IR Dr Dins, gate projection W, E Dr.xx down matrix WD RDimxDD, and hidden states X E RRDxs from N calibration samples 1,..., N, and ridge
intensity X.
2: X = (X) W
X W.. i= i.
int: C E R Dime 4: s + diag (C(C + AT)
3: C E X X
D Average the correlation matrix over the samples > Compute ridge leverage scores
s E Rim
5: Sm IID. It argsort(s) (s)]. Sm E Dine Dint
Get the sorting matrix based on the vector
6: Pseudo-inverss-free (Ours)
7: return (W., W Wa) t (W.Sm, Wysm: S Wa) > Output the structured sorted weights
A.2 MULTI-HEAD SELF-ATTENTION QUERY-KEL KEY
Algorithm describes the structured sorting for the query-key for MHSA. We obtain the activations for the W 9 and W and compute the channel correlation C₁ and Ct2- The sorting scores calculated as
12 o IIc 1/2 I. and averaged over the calibration samples. To support the fused RoPE, we split the dimension of by half such that (5,,5) = and apply sorting to $1 + s2. As such, we construct the final sorting matrix Syk that sorts the output columns as W and S 9k'
Algorithm 2 Structured sorting key-query for MHSA with H heads.
I: Input: MHSA query matrices W, E P Dia), key matrices We E P Dua), hidden states X E RTxD, from N calibration samples i = 1,..,, N, and the function of rotary positional embedding p(-).
> Apply sorting to each head independently
2: for I..... H do
3:
W pXX W C. # M E W (()
Ci = N E
C. E DiaxDan
Query correlations
4:
C E PR
Key correlations
- |/2 C |IcE 1/2 ||. E Das
5:
Calculate the norm score
6:
Symmetric sorting for fused RoPE kernel (Ours) (81, 82) + 8, {81, 82) E Pha/2
7:
b Split the norm score vector by half
8:
Get the symmetric sorted indices
9:
idxaym & [argsort(s1 + 82), Dan/2 argsort(s) + 82)], idXsym E Pad
10: Spk + I F. idxeym], S.N E Daxx Dan & Get the sorting matrix based on the vector
11: (W). W <- (WiS) gbo w(s) )k)
12: end for
13:
return (W,, Wx) + (ww 91
WH q
[W..... w
P Concatenate the sorted heads
16

---

Published as a conference paper at ICLR 2026
A.3 MULTI-HEAD SELF-ATTENTION VALUE-OUTPLT
Algorithm presents the quantization-awarr SVD decomposition for W and W; in MHSA. With the input correlation matrix C, we perform joint decomposition by two consecutive SVD operations: Cá C:W: w U.UEVT. The SVD decomposition ranks the eigenveetors by eigenvalues. We decompose the weight matrix W = UEV in a quantization-friendly fashion by merging the long- lailed eigenvalues Σ with U, such that W (UEVV.
Algorithm 3 Structured sorting value-output for MHSA with H heads.
1: Input: MHSA value matrices W, E (RDxxxXxxaa)) output matrices W. E R(HxDDa)xDo, hidden states X E Tx from N calibration samples = 1,.... N.
2: C= - X Ce P
Apply sorting each head independently
3: for ==
H do
4:
(U, E. ) t SVD(C1/2 W
5:
SVD(E, V W (W2, 01 W < (,-1//U,,U)) V
(U,E, V
6:
> Quantization-awaree SVD (Ours)
7: end for
8: return (W.. Wo) + (W! WVl. ! W
Concatenate the sorted heads
A.4 GROUP-QUERY ATTENTION: QUERY-KEY
Algorithm 4 describes the structured query-key sorting for GQA, featuring H, self-attention heads and H&o key-value heads, where H, > Hkv- Firstly, we determine activations for W and then compute the channel correlation Cg for key-value head The correlation C is shared among group of self-attention heads. We then compute the channel correlation C for the group of self- attention heads, and sum the norm scores of the group. The sorting matrix S.p is obtained simi- larly MHSA enable RoPE. In GQA, Se sorts the output columns of self-attention heads as (W]s) glg ......, w Me St ykh and the key-value head as W gk-
Algorithm 4 Structured sorting key-query for GQA with H, heads and Hg. key-value heads.
1: Input: MHSA query matrices W,, E D Daa), key matrices W. E RDx XH Dra), hidden states X E RTxD,, from N calibration samples i 1,... N, and the function of rotary positional embedding p(-).
2: for j 1,.... Hkv do
3:
C/ N 1==1
1 ΣE
X W T ((X) C/ CR
Key correlations > Initialize with zeros
4:
- [0, ..,
SER Daa
5:
for l,..., He
H.
do
6:
CK = 1 Ei p W p X WH), C. E Daa
N
Group query correlations
7:
s + | |
Calculate the norm score
8:
end for
9:
[81, 82) 88, 8, (81,82) 82) E Dra/2
D Split the norm score vector by half
10:
Get the symmetric sorted indices
11:
idxaym + [argsort(s: + 82), Dan/2 + argsort(s: + +2)], idxsym E Pha
12:
S + --Daa [t. idxeyml. S E R Pai Dan
> get the sorting matrix based on the vector w H. Sy W/S
([W1 q: H../HHe HKo w (Wis) gb g.r.
13:
14: end for
15: return (W,, WR) (m ..., WH.), [W].
w wHkk))
Concatenate the sorted heads
17

---

Published as a conference paper at ICLR 2026
A.5 GROUP-QUERY ATTENTION: VALUE-OUTPUT
Algorithm [S] outlines structured value-output sorting for GQA. Since GQA has H. self-attention heads and HAu key-value heads, where H3 HRu. a single SVD decomposition is performed using the input correlation matrix CW; = U. E V v. We also incorporate quantization-awan SVD for W: by integrating Eg with U, for W The V is shared across attention heads such that V, W for K E [1, ,., H
Algorithm Structured sorting value- value-output for GQA with H, heads and Hyu key-value heads.
1: Input: MHSA value matrices W. Dr x Dna), output matrices W. R Dr, hidden states X E IR 1h from N calibration samples = I,..,NN. N.
2: C X X CE Reaxxaa
3: for 1 ..... HAv do
4:
(U,, 2., V. + SVD(CWW))
5:
W f C U,D
P Quantization-awarr SVD
6: for l,..,, H do
Heo
7: W + W
8: end for
9:
end for
10: return (W.. W.) + (WL) 00
WHr)), w . > Concatenate the sorted heads
A.6 MAMBA: B-C STATE MATRICES
We show the algorithm that sorts the weights (W w at Algorithm 6 For the SSM group with H Hm SSM heads in the group, we obtain the activations B and C B is discretized using the input-dependent variable A by broadcasted outer product (AB)² A & B As result, we compute the channel correlation ACy and Cg. The sorting scores E R. are calculated as 8 Σ C 1/2 sorting matrix Se with that sorts the output columns of W and W as WD and W S
G,
@ |C2 1/211, and averaged over the calibration samples. We construct the
Algorithm 6 Structured sorting B-C for Mamba with Hm, heads and G2 SSM groups
1: Input: matrix WB E PLGx D.) and matrix We E IR Pa, xG.xx hidden states X; RTxx,, and input-dependent step size A TT from N calibration samples
i= 1,..., N, and the function of 1D causal convolution ((.).
2: for 1,.... G5 do
3:
B = (X, W (
B E TTD. C = o W9 C E RTD,
B
4:
5:
(AB) Aia @ Bía (AB) E H2
Broadcasted outer product
6:
Average over the calibration samples
ACE N EN (AB) As (AB)'9 ACE (.,,/G.)xD.xx
7:
8:
CO Σ i= C C C2 D.xD.
9:
Compute group correlations Rh.
10:
[0. .o.....00 0] for k l...., Hm do
Initialize with zeros
11:
C = (AC2) RT (Acy) )*
12:
B
13:
8= 8 + C 1/2
о ||c
1/2
Calculate the norm score
end for
15:
S/c + Ip, [t. (s)] BC E D,D,, & get the sorting matrix based on the vector
16: (W) W (W/S) BO s BC)
17: end for
return (WB) Wc) < (Ww,.... WC B. (W...., w Concatenate the sorted states
18:
18

---

Published as a conference paper at ICLR 2026
A.7 MAMBA: Z-X AND OUTPUT MATRICES
We describe the structured sorting for weights (W:, W W in Algorithm We collect the correlations C2 from the SSM states Ht and average over the calibration samples. The ridge leverage score diag (CH(CH AT)- computed the MLP layer. We create column sorting matrix S, E IR Pau Dad using ridge leverage scores, then arrange the output columns via W and W S The input rows for W are sorted accordingly by S, W We set ridge lambda À 1 in our experiments.
Algorithm 7 Structured sorting z-x-0 for Mamba with H, heads and G, SSM groups.
Input: X projection Wz RDDx(HH projection W₂ R Dia), out matrix W. E R x P Ph, and HC R (Tx D2)x Dad from N calibration samples I,
and ridge intensity X.
2: for =1 1..... Hm do
3:
State-aware (Ours)
4:
C = + Ei HitTHi C DraxDDDa i= I,,.,, N
D Average over the samples compute ridge leverage scores S - I-Ip argsort(s)), S E D & get the sorting matrix based on the vector
5:
< diag (C(C + A7)-1), SE RDd
6:
7: (W W W + (W?S) W S w
8: end for
9: return (W W., Wo) + ([w) 21 W [W] m W [W].... WH"])
Concatenate the sorted heads
B BROADER EVALUATION RESULTS
B.1 COMPARISON WITH ADDITIONAL BASELINES
we include more common baselines in Table I with all methods evaluated FP16 to ensure fair and controlled setting. We follow the experimental setup used in MoDeGPT (Lin et al. 2025) and append our Llama-3.1-8B results at the 25% pruning rate to those reported in their work. The numbers, except for UniQL., transcribed from MoDeGPT. The results show that UniQL consistently outperforms prior pruning-based methods at the 25% compression level, and UniQL-ft further boosts accuracy, achieving the strongest performance across all evaluated tasks.
Table 11: (Comparison with Additional Baselines.) All models are evaluated in FP16 for a fair and consistent comparison following the experimental setup used in MoDeGPT.
Compress. % Method
ARC-e
ARC-c
PIQA
WinoG.
HellaS.
Average
0%
Dense
77.69%
53.58%
80.63%
72.69%
79.16%
72.75%
25%
ShorGGF--AI Allpaca
38.13%
31.40%
60.94%
54.22%
31.52%
43.24%
SliceGPT-Alpaca
44.44%
29.27%
57.56%
58.48%
41.08%
46.17%
MoDeGPT-Alpaca
67.05%
41.13%
75.52%
69.61
66.49%
63.96%
UniQL. 70.37%
46.33%
74.16%
71.82%
70.12%
66.56%
UniQL-ft 76.05%
50.00%
76.55%
72.93%
73.37%
69.78%
B.2 EVALUATION THE MMLU DATASET
We test Llama-3.1 3.1 1--B 2.5--2.,- and Nemotron-H-8B, and report five-shot accuracy on the MMLU dataset (Hendrycks al., 2020) with batch size of eight. The MMLU dataset large multitasking dataset, covering 57 subjects of varying difficulty. Our pruned models maintain com- petitive accuracy on the challenging dataset and outperform MoDeGPT (Lin et al. (025) and SVD- LLM WWng et al. 2025b). Nemotron-H. the Mamba-Transformn hybrid model, shows a large accuracy drop, but recovered with our low-cost masked fine-tuning. We compare our method with 2024a) implemented in the TensorRT framework (NVIDIA) 2023 2024) (TRT- AWQ) and HQQ (Badi & Shaji 2023) in TorchAO (torchao) 2024) (TAO-HQQ). Our 4-bit models perform comparably these -fate-of-the-art PTQ frameworks while offering broader model support.
AWQ Lin et al.
19

---

Published as a conference paper at ICLR 2026
Table 12: (Five-shot MMLU.) We compare UniQL against baselines under different settings. represent the FP16 embeddings and output layers per the official implementation. denotes the GPTQ (Frantar et al. 2023) implemented on all models as an additional baseline.
Method +FT +PTQ W-bit Prun. p%
Llama-3.1 3.1-
Qwen-2.5-7B*
Nemotron-H-aee
FP16
16
0%
65.6%
74.2%
67.6%
MoDeGPT
16
15%
59.5%
23.1% %
SVD-LLM
15%
28.4%
49.9%
UniQL (Ours)
16
15%
60.2%
55.9%
37.5%
SVD-LLM UniQL (Ours)
16
15%
41.5%
61.1%
16
15%
59.2%
59.9%
56.1 1%
TRT-AWQ
0%
63.0%
72.5%
TAO-HQQ
4"
0%
62.9%
72.5%
GPTQ
0%
61.5%
70.5%
64.0%
UniQL (Ours)
0%
63.2%
70.3%
67.5%
SVD-LLM
15%
34.8%
56.3%
UniQL (Ours)
15%
56.9%
52.7%
52.6%
B.3 EVALUATION ON CODING TASKS
We present the evaluation results the MBPP+ (Austin et al. 2021 Gao al. 2023) coding bench- mark in Table We apply UniQL to Llama-3. -nst--tt and compare its performance against SVD-LLM (Wang et al. 2025b) and MoDeGPT (Lin al.] 2025) under various pruning ratios and bit-width configurations. The MBPP+ results obtained under batch size I and O-shot settings following common practice. These results demonstrate UniQL's ability maintain competitive performance compared to SVD-LLM and MoDeGPT while reducing model size.
Table 13: (Evaluation results on the MBPP+ coding tasks.) We compare UniQL with existing compression baselines under different pruning ratios and bit-width settings. Results are reported on the MBPP+ (instruct) benchmark using batch size and O-shot evaluation.
Method
One-pass
+FT
W-bit
Prun.
R,size (x)
Llama-3.1-8D
FP16
16
0%
0x
75.4%
MoDeGPT
x
x
16
15%
0.15 x
42.3%
SVD-LLM
X
/
4
15%
4.7x
24.0%
V
4
0%
4x
64.8%
UniQL
V
4
15%
4.7x
54.2%
(Ours)
4
25%
5.3x
33.8%
C ADDITIONAL ABLATION STUDIES
C.1 ABLATION STUDY ON CALIBRATION SETS
Table 14 presents the performance of Llama-3.1-88 under different combinations of calibration sets at a fixed 25% pruning rate. Following our setting, we report the average accuracy of five zero-shot downstream tasks. For each configuration, all hyperparameters and the number of calibration sam- ples strictly follow the settings in Table 11 of our manuscript, ensuring a controlled and consistent comparison. Using the Alpaca dataset (Taori al.) 2023) for all stages results in the best average accuracy. We follow MoDeGPT and use WikiText2 as the calibration set for pruning-rate allocation ensure fair comparison with prior work in all experiments.
20

---

Published as a conference paper at ICLR 2026
Table 14: (Ablation study calibration sets.) We report results for Llama-3.1-8B 25%
pruning rate under different combinations of calibration sets used for weight-sorting, masked fine- tuning, and post-training quantization (PTQ).
Prun. Ratio Alloc. Weight-Sort. Masked FT
PTQ
W-bit Prun. Rate
Avg. Acc
16
0%
74.0%
wikitext2
wikitext2
wikitext2
wikitex12
25%
60.8%
wikitex12
wikitex12
alpaca
wikitex12
4
25%
65.5%
wikitext2
alpaca
alpaca
wikitex12
4
25%
67.7%
alpaca
alpaca
alpaca
wikitex12
4
25%
68.6%
C.2 ABLATION STUDY ON 3-BIT UNIQL
Our framework supports post-training quantization with various bit-widths, including 8, 6, 4, and even 3 bits. To support this claim, we include additional experiments exploring 3-bit UniQL. in Table The 3-bit precision is simulated by FP16 only for proof of concept purposes. These results demonstrate stable performance trends as the precision decreases, highlighting UniQL applicable various bit-widths. Notably, even the 3-bit variant retains competitive accuracy across multiple models, underscoring the effectiveness of our weight-sortng and recovery fine-tuning procedure. Overall, these results highlight the flexibility of UniQL and confirm that remains reliable even in resource-constrainct on-device deployment scenarios.
15
Table 15: (Experimental results of 3-bit UniQL.)
One pass
W
Prun.
R.size Llama-2 .lama-3.1
Qwen-2.5
Bamba-v2 Nemotron-F Mamba2
Method
+FT
bit
(x)
7B
8B
7B
9B
8B
8B
FP16
0x
68.8%
74.0%
72.4%
74.6%
76.0%
70.6%
MoDeGPT
16
15%
0.15x
66.2%
72.4%
52.1%
SVD-LLM
4.7x
63.2%
60.6%
66.8%
0%
4x
67.6%
73.6%
72.4%
75.1%
73.3%
69.3%
UniQL (Ours)
V
v
15%
4.7x
65.6%
71.4%
68.1 68.1%
70.3%
70.5%
65.8%
V 25%
5.3x
63.5%
67.7%
64.0%
67.4%
64.7%
61.8%
V v
0%
5.3x
62.8%
64.5%
67.4%
67.8%
71.3%
67.8%
UniQL
15%
6.2
60.2%
63.5%
63.0%
64.4%
67.8%
64.0%
(Ours)
V
25%
7.1x
58.9%
59.4%
58.7%
61.6%
62.1%
60.3%
D PARETO-FRONT ANALYSIS
illustrate the Pareto-frontier trade-offs between accuracy and latency across diverse set of Transformer hybrid, and state-space models on A6000 and Nano 8G, respectively. Each sub- plot groups models by: (a) pure Transformers, (b) hybrid and SSM-based models, and (c) the union of both. We compare UniQL (W4A16, starred markers), GPTQ (Frantar et al.) 2023) (W4A16, cir- cular markers), and FP16 baselines (squares), with circle sizes indicating memory consumption on A6000. For the Nano 8G, we use HQQ (Badri & Shajil 2023) that is supported in the TorchAO framework (torchao 2024) (TAO-HQQ) as our baseline. Across all architectures, UniQL. consis- tently yields better latency- accuraay trade-offss under the same memory budget, especially in the 2-4GB regime critical for edge deployment. For example, UniQL significantly improves latency for Qwen-2.5-7B and Mamba-2-8B while maintaining accuracy close to the FP16 baseline. Notably, UniQL achieves Pareto-dominant points for models like Llama-3.1-2 outperforming GPTQ and TAO-HQQ in both latency and accuracy. Our analysis underscores UniQL's advantage for high- performance LLM inference under tight latency and memory constraints.
Figure 6
E LAYER-WISE PRUNING RATES
We adopt the approach from (in et al. 2025 Men et al. 2024) to determine layer-wise pruning
rates r using Block Influence (BI) scores for specified global pruning rates. The BI score is given by
21

---

Published as a conference paper at ICLR 2026
UNIOL (W4A16)
Qwen-2.5-7B
Nematron-HH H-BB-B---BBBB
GPTO (M4416)
Liamm--3.1-88
Bamba-98-v2
Mamba-22
1716
Aug. Acc. 291
Avg. ACC (16)
10
70
2008
AGB
4GB
2001
2014
55
55
30 (a) Transformers
(sec.)
35
lime-to-int-tttee (sec.)
20
30
10
(sec.)
Hybrids SSMs
(c) Hybrids and Transformers
Figure 5: (Pareto-front analysis on A6000.) We evaluate the trade-off off between average accuracy (%) and time-to-last-tokee (sec.) for various LLMs under different quantization and pruning config- urations. Circle, square, and star markers denote GPTQ (W4A16), FP16, and our proposed UniQL (W4A16), respectively. Marker size indicates memory footprint.
Liama 2-78
Qwen-2.5-78
Nemotron-HH --atrrooo----------
UniqL (W4A16)
(ELL)
Llame-3.1-20
Bamba-9B-v2
Mamba-2-88
TAO MOO (W4A116)
OOM
20
70
70
Aug. Aic
4GD
4G2
268
55
12
24
122
32
64
220
54
125
Tinnetoo-bstttoken (sec.
to-keet-t-tknnn (sec.)
Time-to-lasi-oken (sec.)
Figure 6: (Pareto-front analysis on Nano 8G.) We evaluate the trade-off between average accuracy (%) and time-to-last-token (sec.) for diverse LLMs with different quantization and pruning settings. Circle, square, and star markers represent TAO-HQQ (W4A16), FP16, and our UniQL (W4A16), respectively. Marker size reflects memory usage.
8= 1-E | |x y: with xy and y as the input and output of the block (e.g.. Transformer or Mamba block) at layer I. Using the closed-form solution from Lin et al. (2025). ). we smooth the layer-wise pruning rate allocations obtain p' Ir rf ...., I. such that P LPavg Softmax(-s// where 8; represents the importance score of layer i, and Pavg denotes the target global sparsity, In our experiments,
set 0.1, and we present the pruning rates for all models in Figure 7 self-attention layers in hybrid models have low pruming rates, indicated by high BI scores in the Figure. In Bamba-9B-v,, layers 9, 18, and 27 are self-attention layers with lower pruning rates than nearby layers. Similarly, Nemotron-H-8B shows this pattern in layers 7, 18, 29, and 40. Pruning self-attention layers in hybrid models leads to sigmificant accuracy drops.
F IMPLEMENTATION DETAILS
F.1 CALIBRATION SETS
Table 16 lists the calibration set, number of samples, and the sequence length we use our ex- periments. We collect BI scores and assign layer-specific pruning rates using 128 samples with sequence length of 2048 from wikitext-2 (Merity et al.) 2017). Various global pruning rates, such as Pis. Pas. and Pas. computed using the same setting. With 128 samples and a sequence length of 2048, we compute channel correlations from the Alpaca dataset (Taori et al. 2023), as detailed in (Lin et al. 2025). Our masked LoRA fine-tuning conducted the Alpaca dataset with sequence
22

---

Published as a conference paper at ICLR 2026
Qwen-2-5-78
0.8
0.8
0.8
10%
10%
10%
0.7
15%
0.7
0.7
20%
25%
25%
25%
Prunking Retee
35%
35%
0.4
0.4
Pruring
10.4
0.3
0.3
0.3
0.0
0
10
35
20
25
30
10
15
20
25
30
0
5
10
35
20
25
Layer Indles
layer Indlao
Tayer Index
Damba- Mitaa-DDv22
Nemotron- H1-0D
Mambe2-88
10%
SelfAAtnn
Salf-Attin
0.7
15%
0.7
15%
0.7
15%
0.6
25%
0.6
0.6
25%
Pruning Renes
30%
0.5
30%
30%
Praning
0.3
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0.1
0.0
15 Layer Indeo
100
30
10
20
30
40
50
Layer Inblex
Layar Index
Figure 7: (Layer-wise pruning rates.)
length of 256 to reduce training memory usage. Lastly, we calibrate post-training quantization on the wikitext-2 dataset using 256 samples with sequence length of 2048.
Table 16: (Calibration sets.)
dataset
#data
seq. len.
Layer-wise prun. ratio alloc.
wikitext2
128
2048
Structured weight-sorting
alpaca
128
2048
Masked LoRA fine-tuning
alpaca
51800
256
Post-training quantization
wikitext2
256
2048
F.2 MASKED LORA FINE-TUNING
Table 17: (Hyperparameters for fine-tuning.)
We list the hyper-parameter for our masked
Hyperparameter
Value
LoRA fine-tuning in Table 17 We follow the prior work to perform instruction tuning on the Alpaca dataset (Taori et al. 2023) for five epochs. Specifically, we adopt a relatively small sequence length of 256 to reduce training cost, and set the LoRA rank r = 8 with scaling factor a = 16. warmup of 100 steps is applied with the AdamW optimizer, and micro-batching is used to accom- modate GPU memory limits. All models ex- periment with are using the same hyperparame- ters, and we do tune the parameters for the model and experiments.
Learning rate
x 10-4
Batch size Micro batch size
Optimizer
AdamW
LoRA rank (r)
LoRA scaling (a:)
LoRA dropout
0.05
Warmup steps
100
Max sequence length
256
Training epochs
F.3 HADAMARD TRANSFORM FUSION
To enable the flexibility of the model size, our framework does not apply Hadamard rotations to the pruned channels. Importantly, the hidden dimension, iee,, the dimension propagated across layers,
23

---

Published as a conference paper at ICLR 2026
remains unchanged. This design choice enables efficient on-device pruning for adaptive deployment, and avoid the mismatch shapes of the pre-fused Hadamard matrices after pruning the channels. We provide the detailed Hadamard fusion configurations for Transformer block in Table [8 and Mamba block in Table [9] where pruned channels are indicated with a "*", All other models follow the same Hadamard fusion pattern these two examples. For Qwen-2.5-7B, we empirically find that applying Hadamard matrices degrades accuracy, so we remove all Hadamard matrices in our contiguration.
Table 18: (Hadamard matrix fusion for Transformer blocks.)
Table 19:
(Hadamard matrix
fusion for
Mamba blocks)
Operator
Input Had.
Output Had.
Operator
Input Had.
Output Had.
q-proj
/ Yes
No
z.proj
/ Yes
No*
k.proj
/ Yes
No
v-proj
/ Yes
X No
x proj
/ Yes
No*
o.proj
No
V Yes
B_proj
/ Yes
No*
up up-proj
/ Yes
X No*
gale proj
V Yes
No*
C-proj
/ Yes
No*
down proj
X No
V Yes
out proj
X No*
V Yes
G ENERGY PROFILING
To assess the practical efficiency of our quantization and pruning strategies, we conduct energy profiling on an A6000 GPU and an Orin Nano 8G, both of which are representative of cloud and edge platforms. On Orin Nano, each request is prefilled with 512 tokens and generates 512 new tokens, where record the total energy consumption in Joules-per-request (J/req.). As shown in Table 20, full-precision (FP16) models exceed the device's 8 GB memory limit, resulting in out- of of-memory (OOM) eπrоrs during inference. In contrast, quantized methods substantially reduce energy consumption while maintaining deployability. Without pruning, UniQL (W4A16) reduces the energy per request 208.23 J and 224.56 J on Qwen-2.5-7B and Mamba2-8B!, respectively. When combined with structured pruning (p=35%) the energy further decreases to 143.12 J and 153.64 J.
Table 20: (Energy profiling on Nano.) Joules-per pequusst (J/req.) is reported. Each request is prefilled with 512 tokens and 512 generated tokens. Lower is better (4). represent the FP16 embeddings and output layers as per the official implementation.
Prun.
Qwen-2.5-7B
Mamba2-8Bt
Method
W-bit
p%
J/req. &
J/req. +
FP16
16
0%
OOM
OOM
TAO-HQQ
4"
0%
381.13
UniQL
0%
208.23
224.56
(Ours)
35%
143.12
153.64
On cloud GPUs, we evaluate energy efficiency in terms tokens-per-GGggavatt to align with the industrial computing power metric. In Figure & visualize the energy efficiency of a Transformer model (i.e.. Llama-3.1-8B) and a Hybrid model (i.e. Nemotron-H-888 on an A6000 GPU with 48GB memory. Each request prefilled with 1024 tokens and generates 1024 new tokens. Under different batch sizes, we report the total number of tokens for a Gigawatt per second. Nemotron-H adopts SSM blocks reduce the memory needs for KV cache. Also. UniQL consistently achieves higher throughput-per-con establishes UniQL. as an effective deployment framework for both resource-constrained and energy- aware scenarios.
across both Transformer-bass and SSM-based architectures. This
"The embedding and output layers use FP16 according to the official implementation.
24

---

Published as a conference paper at ICLR 2026
PP16
UniQU (Ours, WCA16)
UNIQL (Ours, VACTO P35%)
V.IDO
3.155
8.046
7,30G
0.09G
5.660
5.756
Tokenstiigg Gigneeett
4.99G
4.945
5.27
4x13
3.59
3.489
3.95
4.16
2417
1.87
buiee
Llama33 3.1-88
Nemetran 8B
Figure 8: (Energy efficiency analysis A6000.) Nemotron-H incorporates SSM blocks to de- crease KV cache memory requirements. UniQL. continually offers superior energy efficiency for both Transformer and SSM architectures.
25


Reviews:
[{"review_id": "5QHEGPcmXZ", "reviewer": "Reviewer_Duby", "summary": "The paper introduces UniQL, a unified post-training compression framework designed to efficiently deploy large language models (LLMs) on edge devices. It integrates structured pruning and quantization in a single-shot pipeline that supports adaptive on-device compression based on real-time resource availability.", "questions": "- With masked fine-tuning (FT), UniQL remains faster (6h 59 m) than both MoDeGPT (7h 03 m) and SVD-LLM (15h 57 m). I do not understand why masked fine-tuning would be faster — could the authors clarify the reason behind this behavior?\n\nI am open to discussing this further during the rebuttal and will be happy to increase my score if my concerns are addressed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "- Unified Framework: UniQL supports Transformers, State Space Models (SSMs), and hybrid architectures, addressing a wide range of LLM structures.\n\n- On-device Adaptive Pruning: Enables users to prune the model at inference time based on the current device memory state.", "weaknesses": "- While the results are generally strong, some inconsistencies exist in the evaluation setup:\nIn Tables 1 and 2, the latency results for different models and methods are evaluated on different hardware platforms (Llama-3.1-8B and Nemotron-H-8B on A6000; Qwen-2.5-7B and Mamba2-8B on Nano 8G). Additionally, Table 2 lacks baseline comparisons such as TRT-AWQ for some models. This raises concerns about the consistency and comparability of latency evaluations across models and methods. Can the authors clarify why all models and methods are not evaluated uniformly across both platforms, and whether such comparisons are fair and meaningful under these mixed settings?\n\n- UniQL is compared to SVD-LLM both with and without fine-tuning, which is helpful. However, the comparison with MoDeGPT is conducted only without fine-tuning, despite UniQL including fine-tuning in its best-performing configuration. Furthermore, all comparisons are conducted at only one sparsity level (15%), which limits the ability to assess how robust each method is across different compression regimes (e.g., 25%, 35%).\n\n- In Table 7, UniQL is evaluated under single-pass adaptive pruning across multiple pruning rates and compared only with SVD-LLM. However, MoDeGPT, another key baseline used in Table 5 and throughout the paper, is not included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0NDuIKxpYS", "reviewer": "Reviewer_oUGV", "summary": "This paper proposes a unified quantization and on-device structural pruning method for edge LLMs, including Transformers, State Space Models (SSMs), and hybrid models. To enable the on-device structural pruning, weight sorting is designed for different model blocks, and a LoRA-based recovery fine-tuning (FT) is conducted on the sorted model. 4-bit quantization and pruning with different rates are applied to the model. SVD decomposition is used to reduce quantization error. The evaluation presents different pruning rates that offer a memory reduction of 4x–5.7×.", "questions": "1. The finetune is applied after the weight sorter. Why does it apply after quantization? Have you experimented with different orders of the applied methods?\n2. The pruning method is similar to SliceGPT [1]. What’s the key difference between SliceGPT and the proposed sorting method?\n\n[1] Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari Do Nascimento, Torsten Hoefler, James Hensman. SliceGPT: Compress Large Language Models by Deleting Rows and Columns. ICLR 2024", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The adaptive LLM memory problem discussed in this paper is important and interesting.\n2. The proposed methods are evaluated on different model structures.", "weaknesses": "1. The contribution is limited, and the proposed methods are very incremental.\na) Methods applied to different model structures appear more as a systematic engineering effort than a novel algorithmic advancement.\nb) The quantization, pruning combination has been explored.\n\n2. Insufficient Empirical Evaluation.\na) The paper claims adaptive deployment of LLMs on the edge, but there is no real deployment with different workloads. Crucially, the paper does not address the core systems challenge: how the memory footprint can be dynamically managed at runtime without first loading the entire unpruned model.\nb) Unignored performance drop. The reported performance drop is substantial, especially at lower pruning rates (e.g., a 6% drop for Llama2-7B at only 35% pruning).\nc) Lack of comparison with related works: Quantization or pruning methods for SSM and hybrid models. Structure pruning methods, such as SliceGPT [1] .\n\n3. The paper writing should be improved.\na) Explain why you do it before the detailed introduction. For example, the reason for weight sorting should be given in the introduction. \nb) Too many mathematical symbols make the methods part hard to follow. The algorithms in the Appendix are better for understanding.\nc) All of the components should be included in the overview Figure.\n\n[1] Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari Do Nascimento, Torsten Hoefler, James Hensman. SliceGPT: Compress Large Language Models by Deleting Rows and Columns. ICLR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z0gPddUpja", "reviewer": "Reviewer_8Vsv", "summary": "This paper proposes UniQL, a unified post-training quantization and low-rank compression framework designed to support adaptive deployment of large language models on resource-constrained edge devices. UniQL achieves integrated compression for Transformer, SSM, and hybrid models through structured weight ranking, quantization-aware SVD decomposition, state-aware SSM compression, and the integration of RoPE kernels. The framework performs one-time compression in the cloud and supports dynamic on-device pruning (up to 35%) based on current load. Experiments demonstrate that UniQL achieves significant memory reduction and inference acceleration across multiple models and hardware platforms, with manageable accuracy loss.", "questions": "Calibration Data Sensitivity:\nHow sensitive is UniQL to the choice of calibration data? Have you tested its robustness across domains (e.g., code, math, dialogue) or with out-of-distribution samples?\n\nComparison with Unstructured Pruning:\nHow does UniQL compare with unstructured pruning methods like SparseGPT, especially in terms of accuracy-efficiency trade-offs and hardware friendliness?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Wide Model Architecture Coverage: This approach systematically supports post-training quantization and structured pruning for Transformer, SSM, and hybrid models for the first time.\n\nStrong On-Device Adaptability: This approach dynamically adjusts model size based on memory and compute resources after deployment to adapt to the dynamic load of edge devices.\n\nHigh Compression Efficiency: This approach significantly accelerates the compression process (up to 22x faster than MoDeGPT) by avoiding pseudo-inverses and optimizing SVD decomposition.\n\nSystem-Level Optimization: This approach integrates RoPE kernels and quantization-aware decomposition to significantly improve inference speed and reduce quantization error.\n\nExtensive Experimentation: This approach demonstrates its effectiveness and versatility across a variety of models (e.g., Llama, Qwen, Mamba) and hardware (e.g., A6000, Nano 8G).", "weaknesses": "Accuracy-compression tradeoff not yet optimal: At high pruning rates (e.g., 35%), the accuracy of some models (e.g., Mamba2) drops significantly (down to 57.7%).\n\nSensitive to calibration data: Structured ordering and quantization depend on the calibration set; their robustness to different data distributions is not analyzed.\n\nLack of comparison with unstructured methods: No comparison with popular unstructured pruning methods (e.g., SparseGPT) or hybrid sparse methods is provided.\n\nDevice-side pruning overhead not quantified: While device-side pruning is supported, its runtime overhead (e.g., memory rearrangement, index lookup) is not analyzed.\n\nLimited interpretability: No visualization or interpretable analysis is provided for the \"state-aware\" or \"quantization-aware\" mechanisms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YMg6SXhxzA", "reviewer": "Reviewer_ZDeG", "summary": "The paper targets dynamic, resource-constrained edge deployment of LLMs. It proposes a one-shot, post-training pipeline that produces a single INT4 model which can be adaptively pruned on-device at different rates depending on runtime constraints. Overall, the paper is a well-executed systems integration that addresses a timely deployment problem and demonstrates tangible engineering wins. However, for ICLR, the lack of clear algorithmic novelty and the absence of a rigorous case for “why combine pruning with quantization instead of simply going lower-bit” (especially W3) are significant gaps. Clarifying the scope of Hadamard/rotation, aligning it with pruning, and adding stronger/fairer baselines and ablations would strengthen the submission.", "questions": "1. Please provide matched-budget comparisons of W4 + {25}% pruning versus W3 (with Hadamard/rotation, e.g., QuaRot-style) on the same models, reporting accuracy–memory Pareto curves. Under what budgets does “W4 + pruning” strictly dominate “W3 alone”?\n\n2. Which layers/axes are rotated (Q/K/V/O proj in attention; MLP up/gate/down; SSM B/C/Z/X/O)? Are rotations applied to channel dimensions that will be pruned on-device?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Practicality and scope: A unified, one-shot post-training pipeline that supports Transformers, SSMs, and hybrids, and enables on-device adaptive pruning without retraining is very relevant for edge deployment.\n2. Solid engineering: the fused RoPE kernel, quantization-aware SVD, and avoidance of pseudo-inverses show careful engineering; the export path quantizes even embeddings/LM head to 4-bit, reducing footprint beyond common W4A16 libraries.", "weaknesses": "1. Most core techniques (importance-driven structured pruning, Hadamard/rotation-based quantization smoothing, W4 PTQ) are known. The main novelty is system integration and some engineering refinements (QSVD, fused RoPE, SSM-aware sorting). This level of originality feels marginal for ICLR unless paired with stronger conceptual/theoretical advances or broader evidence.\n\n2. No justification for “quantization + pruning” vs “lower-bit quantization alone”: The paper does not compare W4 + k% pruning to a lower-bit alternative (e.g., W3) under matched memory/latency. Prior results (e.g., QuaRot with 3-bit) suggest that W3 may match or exceed the accuracy of W4 + 25% pruning at similar or lower budgets. Without this key comparison, the necessity of combining pruning with quantization remains unproven.\n\n3. Weak PTQ baselines : PTQ comparisons are largely against framework built-ins (TRT-AWQ, TAO-HQQ) and a basic GPTQ variant; missing stronger recent baselines such as AWQ/QServe, Quarot/SpinQuant, etc.\n\n4. The scope of rotations is not clearly specified: If rotations are applied on input channel axes of layers whose channels will later be pruned (e.g., O_proj or MLP down-proj input channels), this mismatch can harm pruning efficacy. There is no ablation on restricting rotations to non-pruned axes or aligning pruning boundaries with quantization/rotation groups.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)