Optimal Brain Restoration for Joint Quantization and Sparsification of LLMs
Authors: Hang Guo
Luca Benini
Yawei Li
Venue: ICLR 2026 Poster (2026)
Keywords: Quantization
Pruning
LLMs
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper introduces a compensation-based framework for joint quantization and sparsity, and is the first to enable W4A4KV4 quantized + 50% sparse LLMs.

Abstract: Recent advances in Large Language Model (LLM) compression, such as quantization and pruning, have achieved notable success. However, as these techniques gradually approach their limits, relying on a single method for further compression has become increasingly challenging. In this work, we explore an alternative solution by combining quantization and sparsity. This joint approach, though promising, introduces new difficulties due to the inherently conflicting requirements on weight distributions: quantization favors compact ranges, while pruning benefits from high variance. To attack this problem, we propose Optimal Brain Restoration (OBR), a general and training-free framework that aligns pruning and quantization by error compensation between both. OBR minimizes performance degradation on downstream tasks by building on a second-order Hessian objective, which is then reformulated into a tractable problem through surrogate approximation and ultimately reaches a closed-form solution via group error compensation. Experiments show that OBR incurs only a 1.4 perplexity degradation on Llama2-7B to enable aggressive W4A4KV4 quantization with 50% sparsity, delivering up to 4.72x speedup and 6.4x memory reduction compared to the FP16-dense baseline.

Published as a conference paper at ICLR 2026
OPTIMAL BRAIN RESTORATION FOR JOINT QUANTIZA- TION AND SPARSIFICATION OF LLMs
Hang Guo, Luca Benini,
Yawei Li
ETH Zürich
httpp:///atthub..co///////
ABSTRACT
Recent advances in Large Language Model (LLM) compression, such as quanti- zation and pruning, have achieved notable success. However, as these techniques gradually approach their limits, relying on a single method for further compression has become increasingly challenging, In this work, we explore an alternative solu- tion by combining quantization and sparsity. This joint approach, though promising, introduces new difficulties due to the inherently conflicting requirements on weight distributions: quantization favors compact ranges, while pruning benefits from high variance. To attack this problem, we propose Optimal Brain Restoration (OBR), a general and training-free framework that aligns pruning and quantization by error compensation between both. OBR minimizes performance degradation on downstream tasks by building on second-order Hessian objective, which is then reformulated into a tractable problem through surrogate approximation and ultimately reaches a closed-form solution via group error compensation. Exper- iments show that OBR incurs only a 1.4 perplexity degradation on Llama2-7B to enable aggressive W4A4KVA V4 quantization with 50% sparsity, delivering up 4.72x speedup and 6.4x memory reduction compared the FP16-denss baseline.
INTRODUCTION
Large Language Models (LLMs) (Brown et al., 2020; Achiam et al., 2023; Dubey et al., 2024) have demonstrated remarkable capabilities across wide range tasks. However, as I.LMS continue to grow in size with increasing parameter counts, efficiently serving them, especially in resource- constrained edge devices, remains a significant challenge (Dettmers et al., 2022).
To meet the demand for efficient LLM deployment, variety of methods have been proposed. One prominent line of work focuses on LLM quantization (Nagel et al., 2021), whose main objective is to remove outliers inherent in the LLM weights. To this end, existing works introduce either smooth- ing (Lin et al., 2024a; Xiao et al., 2023) or Hadamard rotation as a preprocessing step (Ashkboos al., 2024; Liu et al., 2024) to redistribute outliers before quantization. Thanks the resulting flat distributions, recent state-of-theearts (Liu et al., 2024; Sun et al., 2024; Hu et al., 2025) can achieve even 4-bit cright-activation-KY cache (W4A4KV4) inference while preserving performance comparable to the FP16 counterparts. Besides quantization, LLM pruning (Ma et al., 2023; Frantar & Alistarh, 2023) often considered another popular solution for compressing LLMs. Recent LLM pruning works (Sun et al., 2023; Zhang et al., 2024) have shown promising results on 50% unstructured and 2:4 struuutrut sparsity by additionally considering the statisties of activations during pruning.
Despite the promising results at moderate compression, relying on a single technique for further reduc- tion is becoming increasingly difficult. As shown in Fig. (a)), the quantization method QuaRot (Ashk- boos et al., 2024) achieves competitive perplexity at moderate bit-width, but suffers from severe degradation under 4-bits. Similarly, pruning alone also faces analogous limitations, where aggressive sparsity inevitably leads to substantial degradation. beyond current LLM compressions by jointly leveraging quantization and sparsification. The intu- ition arises from the observation that low-bit and sparse representations can cocxist. We empirically find an average of 14.28 unstructured sparsity in the W4A4K V4 quantization-only Llama2-7B
this work, we explore an alternative path to go

---

Published as a conference paper at ICLR 2026
(a) Performance scaling with #bits
(b) Efficiency comparison between different GiMMs
18
Pruning-onlyy(W. (AND))
Timec
9.54
WikiText2 Parplexity
Quantization-olly QuaRR)
2.73
Qparsity+Quan((Uu
4.72x speedup
approaching
Memocyy(A)) TOPS
504.62
10
compression limits
1090.78
8
32,00
6
robust under 4bil
5.00
16-bit
8bit 6bit 4-bit 3-bit
FP16-Dense
INT4-Dense
INT4-Sparsc
Equivalent Weight Bit-widths
Figure I: (a) Single compression techniques (Sun et al., 2023; Ashkboos et al., 2024) rapidly reaches limits under sub-4 bits while the joint counterpart can enable further compression. To enable a unified comparison in single figure, pruning is represented using equivalent bit-widthe (b) INT4 2:4 sparse GEMM can achieve faster inference speed, higher throughput, and lower memory usage.
model, suggesting potential combination of quantization and pruning. Furthermore, recent hardware advances, such as NVIDIA's Ampere and Hopper architectures, have introduced native support for INT4-sparse GEMM kernels (Mishra al., 2021; NVIDIA, 2022; 2021), making the combination of quantization and sparsity increasingly practical for efficient LLM inference.
However, achieving effective joint quantization and sparsification is non-trivial, primarily due to the inherent conflict between their objectives. Specifically, quantization favors a narrow numerical range in the weights to minimize quantization erгоr, whereas pruning benefits from large variations in weight magnitudes to reveal naturally sparse patterns. For instance, Hadamard rotation is common practice existing methods smooth outliers for W4A4KV4 quantization. However, as evidenced by Sec. 5.1, using existing pruning methods to force zeros on the Hadamard-rotated weights leads to unacceptable performance degradation.
Our approach. In this work, we propose Optimal Brain Restoration (OBR), general framework to enable joint quantization and sparsification. The core idea of our OBR is to intervene between pruning and quantization by computing compression on downstream tasks, thereby reconciling their conflicting requirements on weight distributions. To achieve this, we begin by formulating the second-order Hessian objective minimize the impact of weight perturbations on downstream task performance. To make the optimization problem tractable, this objective then approximated through row-wise decoupling, which eliminates inter-row correlations. Building on this surrogate, we further introduce group error compensation, which redistributes distortions from pruning and quantization minimize overall error, yielding an explainable closed-fome solution. By reconciling the confficting requirements between quantization and sparsity, OBR provides an efficient and practical solution for LLM compression.
optimal compensation that minimizes the impact of
To the best of our knowledge, OBR is the first to enable W4A4KV4+505 sparsity LLMs, without requiring any additional retraining. We apply the proposed framework on Llama2 (Touvron et al., 2023), Llama3 (Dubey et al., 2024), and Qwen2.5 (Qwen'Teae 2024) families, and demonstrate promising performance with OBR. particular, our highly compressed model narrows the perplexity gap to merely 1.37 wrt its full-precision Llama2-70B counterpart. Furthermore, we evaluate the inference efficiency using INT4 sparse GEMM kernels. As shown in Fig. 1, OBR achieves up to 4.72x speedup and 6.4x memory reduction compared to FP16-dense baselines. We hope our work can serve as solid baseline and stimulate further research towards sparse low-bit LLMs.
RELATED WORK
Network Quantization for LLMs. Network quantization aims to accelerate inference by converting the full-precision representations into low-bit representations (Nagel et al., 2021). With the thriving of LLMs, many efforts (Tseng et al., 2024; Lin et al., 2024b) have focused on adapting quantization tech- niques for LLMs. For example, GPTQ (Frantar al., 2022) improves upon the classic OBQ (Frantar Alistarh, 2022) by enabling efficient post-training quantization on large-scale parameters and can outperform the common RTN baseline. Moreover, L.L.ss also contain outliers, where small number
2

---

Published as a conference paper at ICLR 2026
of elements exhibit disproportionately large magnitudes and heavily influence downstream perfor- mance. To address this, LLM.int80) (Dettmers et al., 2022) introduces mixxddppecision scheme that preserves outliers in higher precision. Later, AWQ (Lin et al., 2024a) proposes employ smoothing factors to transfer outliers from weights to activations, thus allowing for 8-bit weight quantization. SmoothQuant (Xiao et al., 2023) further trades off smoothing between weights and activations to achieve W8A8 quantization. To push toward even lower bit-widths, recent works (Chee et al., 2023; Hu al., 2025; Sun al., 2024) have predominantly leveraged the Hadamard transformation to flatten the weight distributions before quantization. For instance, QuaRot (Ashkboos et al., 2024) applies random rotation as a preprocessing step, enabling quantization even to W4A4KV4 while maintaining performance. SpinQuant (Liu et al., 2024) and FlatQuant (Sun et al., 2024) further extend this idea by introducing learnable rotation matrices to further enhance quantization performance.
Network Pruning for LLMs. Network pruning reduces computational and memory costs by eliminating weights that contribute little to the final prediction (LeCun et al., 1989; Han et al., 2015; Frankle & Carbin, 2018; Zhang et al., 2024). Early pruning methods primarily relied on magnitude- based criteria, which proved effective for small-scale networks. However, these simple approaches often struggle to maintain accuracy when applied to LLMs. To address this, a variety of methods have been developed to either refine the pruning process or introduce more advanced selection criteria. For instance, LLMM-Pruner (Ma al., 2023) proposes to remove coupled components followed by LoRA (Hu et al., 2022) finetuning to restore accuracy. SparseGPT (Frantar & Alistarh, 2023) introduces a one-shot pruning method based on OBD (LeCun et al., 1989). enabling efficient pruning without additional retraining. WANDA (Sun et al., 2023) demonstrates that information contained in activations crucial for LLMs pruning, and introduces simple yet effective scoring metric for activation-awa -warre sparsity.
Joint Quantization and Sparsification. Before the rise of LLMs, several early works explored joint quantization and pruning on small networks. For instance, DJPQ (Wang et al., 2020) solves an optimization problem via gradient descent to balance sparsity and quantization error. OBQ (Frantar & Alistarh, 2022) proposes unified framework that simultaneously considers both pruning and quantization. In the context of LLMs, JSQ (Guo et al., 2024) adopts simulated annealing to identifyy optimal activation editing policies, and can achieve W8A8 quantization with 50% sparsity. Moreover, one recent work (Harma et al., 2024) also provides a theoretical analysis suggesting that pruning followed by quantization is the optimal compression order. Despite these advancements, existing techniques still fall short in achieving aggressive compression levels such as W4A4KV4 with 50% sparsity, leaving room for further improvement in this domain.
MOTIVATION
As shown in Fig. I(a), relying on single method such as quantization or pruning is rapidly approach- ing its compression limits. For instance, solely decreasing the quantization wit-w or increasing the pruming ratio leads to drastic performance degradation. In contrast, since different compression techniques are largely orthogonal in nature (Guo et al., 2024), combining them effectively presents potential direction to "squeeze out" additional efficiency. For instance, as shown in Appendix B, the W4A4KV4 quantized Llama2-7B model in QuaRot (Ashkboos et al., 2024) naturally exhibits 14.28% average layer sparsity. Moreover, recent hardware advances have already supported INT4 sparse GEMM, which can achieve faster execution than dense INT4 kernels in practice. These observations motivate us to explore how to jointly leverage quantization and sparsity for more aggressive and practical LLM compression.
However, realizing an effective joint quantization and sparsification scheme is notoriously challeng- ing due to their inherently conflicting nature. Specifically, quantization typically favors a compact numerical range of weights to minimize quantization error. For example, recent b-t quantization methods (Ashkboos et al., 2024; Liu et al., 2024; Sun et al., 2024) commonly adopt Hadamard transformation to rotate weights into smoother distributions for suppressing outliers before quantiza- tion. While such rotation is beneficial for quantization, is detrimental to sparsity, which instead prefers weight distributions that exhibit large numerical disparities to better encourage sparsity. As demonstrated in See. 5.1, naively applying sparsification on top of rotated weights leads to severe performance degradation.
3

---

Published as a conference paper at ICLR 2026
0.8
0.6
0.2
0.0
W
printe W)
praan(poiate W) Wode
quass/pruee 2ww
Figure 2: Given original L.M weights W, we first apply a rotation to smooth out outliers, followed by pruning introduce sparsity. The proposed OBR is employed to compute optimal compensation, which is added to the unpruned elements to mitigate the conflict between pruning and quantization. Finally, quantization is applied to obtain the sparse and quantized LLM weights.
OPTIMAL BRAIN RESTORATION
In this work, we propose the Optimal Brain Restoration (OBR) framework, which adjusts weight distributions to reconcile the conflicting demands of pruning and quantization. Following previous practices (Harma et al., 2024; Guo et al., 2024), we adopt an order pruning-then- then-quantization. As shown in Fig. 2, the overall process generate low-bit and sparse weights using the proposed OBR can be formalized as:
(uant(pruueerotaaee (W)) WOBR),
where W is the original LLM weights, AWORR is the compensation derived from OBR. In the following, we start in Sec. 4.1 by defining the necessary notations and objective function. Then we detail the generic formulation of the proposed OBR in Sec, 4.2, followed by the specific instantiations for quantization and pruning in Sec. 4.3.
4.1 OBJECTIVE APPROXIMATION
Given the weight matrix W E Cout in one standard linear layer and E RCCx being the input activation representing the dataset's statistics, our work employs the following classic optimiza- tion objective (LeCun et al., 1989; Frantar & Alistarh, 2022) which minimizes the perturbation of downstream task loss:
min
B[AL] E[[(X, W + AWW L(X, W)).
where A W is the perturbation on W. C is the downstream task loss.
To solve the optimization problem Eq. (2), we first simplify the objective function. In detail, applying Taylor series on L(X, W + AW) at W drives:
1
AC = (V w L(X, W), AW) + voc (OI)AW)
+ O(JAW])
(3)
where V L(X, W) is the gradient, vec(-) CCoe BlxCoue is the vectorisation operator, and Haall (rec(W)@vec((W)
E Cout Cont the layer-wise Hessian.
Assume that the model has been fully trained and reaches local minima, the VwL(X, W) 0. Further ignoring the last high order terms, Eq, (3) can be approximated into:
AL vec( A wenl
(4)
Despite the above preliminary approximation, computing Ha exactly is still infeasible in LLMs due to the O( Cn) complexity, we thus following previous works (Frantar & Alistarh, 2022) and estimate Hiul
Ham ~ G & H,
(5)
where G E R Cout is the output-side curvature matrix which depicts the second-order sensitivity among output channels, H 2 2XXT E C x C.n is the empirical Fisher matrix, and @ denotes the Kronecker product.
4

---

Published as a conference paper at ICLR 2026
Based on Eq. (5), propose to decouple the row-wise correlation of output channels in Hqul by approximating G as an Identity matrix to make Hfull = I & H completely tractable. Finally, the original objective can be simplified into the following Cot independent optimization sub uub-problems:
Cout
min ve & (live(AW) 1
Σ ElAw,HAw,]
(6)
where Aw; E RIxCin the i-th row of AW. Intuitively, Eq. (6) quantifies the impact of weight changes on the final downstream performance. For example, when H is large, even a small change in weights can result in large differences for downstream tasks.
4.2 SOLUTION AND FRAMEWORK
To solve the simplified objective in Eq. (6), proposed OBR employs the Group Error Compensation to optimally adjust weight distributions by shifting information from error-sensitive groups to the other robust ones. Since the rotation matrix acts both W and X, and thus cancels out during multiplication, in the following sections, we will omit the rotation operation and directly denote W the rotated matrix for notational clarity.
Let Jj = 1 Aw, HAw, denote the i-th sub-problem, we begin by partitioning the elements of the i-th row Aw, into two disjoint groups using two index sets, i.e., the retain set R& and the eviction set E,, where Rg U,E fl,..., Cin J and Rg n E; = 0. The retain set RG collects weights that are less affected by compression, e-g., unpruned or less quantization-distoree corresponds to the indices of elements that are susceptible to compression effects, For clarity, we will omit the row index in the following.
whereas the eviction set E;
With this grouping, our key idea is to compensate for compression-indue -induucd errors eg in eviction set E by transferring its lost information to a more robust retain set R. To enable this, we reorder the perturbation vector Aw into [wwR, Awg]. Then the sub-problen becomes:
HRR Hae]
Aw ej
arg min AWR
J AwHAw (AWR egl
(7)
HER
HEE
Since Eq. (7) is an unconstrained optimization problem, solution by taking the partial derivatives w.r.t. AWR, i.e., VawRJ HRRAWR + HREeE & 0. Then the optimal solution for AWR which minimizes the row-wise error can derived as:
directly obtain the closed-form
Aw - Hnge.
(8)
In Fig. 3(a), we give an example on how to extract sub-Hessian HRR and HRE from H. According to the above formulation, the error in set E is theoretically zero guaranteed by the closed-form solution. Since the retain set R is assumed to be robust against remless errors, the total error can be decreased through transferring information from to Notably, Eq. (8) also offers strong explanation that the Hessian actually serves as a "bridge" for error propagation between different groups. Specifically, Eq. (8), the eg is first projected from E's space to the shared space via HRE. followed by the mapping the R's space through H RR. and the negative sign denoting the correction direction.
4.3 SPECIFIC IMPLEMENTATION
In this section, apply the generic closed form- solution in Eq. (8) to the specific implementation for sparsification and quantization.
OBR for Sparsification. As shown in Fig. 3(b), given the 0-1 mask from existing pruning algorithms, we define retain set R1 as the unpruned slots, and eviction set Fg as the pruned ones. In this way, the information loss due pruning on set Ej be compensated by transferring to set R1. Formally, since the pruning eтror on set Ej is E WE, using Eq. (8), the optimal OBR compensation for pruning can be derived
prune
R R, HR, WE, to the unpruned elements w WR, to obtain the OBR-compensated sparse weight Brune w [wR, + Aw nt O]. After that, we perform another round of OBR on w to further consider the incoming quantization error. Details given below.
Aw prume H
(9)
We then add A prune R.
5

---

Published as a conference paper at ICLR 2026
(a) Sub-Hessian Extraction
(b) OBR for pruning
(C) OBR for quantization
I
0
0
h2x
fuz
Aza
hz4
his
retain set R= II, 3, 43
eviction E=
pruning mask
prumed weights
hzr
hzc
{2,
Wel
We.
Wes
Mox
Wid
Wini
h3
h,32
hxs
h3
lsss
hi
his
h1s
h12
h1s
rotated weights W
comminiin
hai
huz
Asss
har
h3r
h39
h12
1sss
Wi5
Mi2
Wid We
Wil Wel
hss
Asz
Asa
hss
hes
he
h4s
WR:
Es
Ra
Has
-H Hit Was
H Rilt, H (Wa. quaant Way))
Figure 3: (a) Given a Hessian approximation H, we extract the submatrices HRR and HRE based on the index sets R and E. (b) The rotated dense weights partitioned into R1 and E₁ according to the binary pruning mask, followed by OBR to transfer information from WE, to wR,- (c) The unpruned index set Rg is further divided into groups: the first fraction assigned to set Ez, the remaining 1 - a to set R2. OBR is used to compensate for quantization error in E.z.
OBR for Quantization. Different from pruning where the retain set and eviction set can be naturally obtained from the pruning mask, in quantization, we need manually assign the grouping to obtain R2 and Eg for compensation with OBR. Thanks to the flat distribution introduced by Hadamard rotation, we find the discrepancy among unpruned elements is actually small (see Fig. 6). Inspired by this observation, we propose to take the first proportion of elements in set R₁ as the set E2, and the remaining
a proportion of elements as the set R2. other words, R2 + |E2] = R2),
where | is the number clements. In Fig. 3(c), given quantization error on set Eg as e guant WE, quant W,,, we can obtain the OBR compensation for quantization as follows:
Aw quant
- Rt H Eg (w quant (w E2)).
(10)
Considering both quantization and pruning, the overall OBR-processed weights can be formalized as: w = quant + AwR. prume + Aw
quant R2
WE, + AAw prume
O)),
(11)
where Awerune and A Aw Aruune denote indexing from Awerume using R2 and Ez, and w is the final R joint low-bit and sparse weights. Algo. I provides more details of our proposed OBR.
CUDA Kernel Implementation. After transforming LLMs to both sparse and low-bit using the proposed OBR, we implement corresponding GEMM with the CUTLASS library'. To align with hardware supported quantization and sparsity, we perform 2:4 semi-structured sparsity and INT4 quantization on the weights W, and use INT4 quantization for activations X and KV caches.
5 EXPERIMENTS
Datasets and Models. We evaluate the proposed OBR framework on various open-source LLM fam- ilies, including Llama2 (7B/13B/70B) (Touvron al., 2023), Llama3 (8B/70B) (Dubey et al., 2024), and Siven255-nssruuc((BB33333
( (Qwen'eem,, 2024). To comprehensively assess the effectiveness of our method, we conduct experiments on both zero-shot classification and language modeling tasks. For zero-shot evaluation, we report accuracy on commonly used benchmarks including PIQA (Bisk et al., 2020), BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), ARC-easy (Clark et al., 2018), ARC-challenge (Clark et al., 2018), and WinoGrande (Sakaguchi et al., 2021). In addition, we also follow prior LLM compression works (Sun et al., 2023) and evaluate the perplexity on the WikiText2 test set (Merity et al., 2016).
Baselines. We compare our method against a range of competitive baselines under sub-4-bit com- pression settings. Specifically, the full- nell-precision model is included
upper bound for reference. We also evaluate against quantization-oni onll baselines (Ashkboos et al., 2024: Liu et al., 2024) under equivalent bit-widths, é.g., W4A4 model with 50% sparsity is compared model. In addition, we include a simple baseline that directly combines existing quantization and pruning techniques without any specially designed compensation. Furthermore, following the exten- sion described in (Frantar & Alistarh, 2023), we adopt SparseGPT combined with GPTQ as a strong joint
W3A4 quantized
baseline for comparison.
'https:/ //g chub.cco// DI cut

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of perplexity score on WikiText2 and accuracy on zero-shot common sense reasoning tasks with L and Llama3(8B/70B) model families. iSince the Llama3- 70B is sensitive to quantization as demonstrated in (Ashkboos et al., 2024), we keep the KV cache
being 16-bit for acceptable performance. The best and the second best results are in red and blue.
Model
Method
#Bits W.A-KY
Sparsity ratio
PIQA
BoolQ
HellaS.
Arc-e (t)
Are-c (t)
WinoCi.
Avg. (7)
Wiki2
(*)
(1)
(t)
(?)
(J)
Ploating point
16-16-16
0%
79.11
77.71
76.02
74.49
46.33
66.14
70.47
5.47
QuaRot(quant-only
3-4-4
0%
51.96
39.72
29.25
31,36
23.46
52.33
38.01
132.97
2-7B
QuaRot+ WANDA
4-4-
50%
50.27
37.83
25.81
25.00
27.73
49.25
35,98
5868.24
SpareCGPT+GPTQ
4-4-4
50%
63.38
63.27
47.71
50.93
29.44
54,70
51.57
12.94
OBR_RIN
4-4-4
50%
68.77
66.39
55.46
55.98
32.17
60.22
56.49
9.23
OBR_GPTQ
4-4.4
50%
68.99
67.31
58.22
55,93
34.22
61.48
53.45
8.40
Floating point
16.16.16
0%
80.52
80.55
79.37
77.48
49.15
72.14
73.20
4.88
QuaRoi(quant-only)
3-4-4
0%
55.01
62.26
30.00
31.10
22.44
$1.07
41.98
72.53
QuaRot-WANDA
4-4-4
50%
51.36
38.29
26.40
26.18
27.56
49.49
36.54
2289.41
2-13B
SparseGPT+GPTQ
4-4-4
50%
71.27
70.83
60.99
61.87
36,60
62.90
60.74
7.89
OBR_TTN
4.4.4
50%
72.74
69.17
63.85
65.95
38.31
64.17
62.37
7.29
OBR GPTQ
4-4-4
50%
72.91
64.74
65.57
37.88
62.60
7.06
Floating-point
16-16-16
0%
82.70
83.76
83.81
81,06
57.25
77.98
77.76
3.32
QuaRot(quant-on only)
3-4-4
0%
67.74
66.27
56.55
50.67
30.63
62.43
55.72
8.19
2-70B
QuaRot+ WANDA
4-4-4
50%
51.52
38.56
27.67
27.06
23.21
50.04
36.34
169.67
SponseGTT+GPTQ
4-4-4
50%
79.11
76.79
77.20
77.61
51.19
73.95
72.64
4.78
OBR RTN
4-4-4
50%
78.67
75.93
76.09
77.57
51.96
74.51
72.45
4.84
OBR.GPTQ
4-4-4
50%
79.
76.91
77.23
77.53
50.68
74.11
72.61
4.69
Fleating-poiin point
16-16-16
0%
80.85
80,98
79.17
77.74
53.24
73.40
74.23
6,13
QuaRoi(quantonnl))
3-4-4
0%
55.28
39.72
30.78
30.72
21.76
50.36
38.10
196.23
3-8B
Spart
QuaRot+ WANDA
4-4-4
4.4.4
50%
50%
66.21
49.62
65.41
37.95
53.58
26.42
27.02
50.67
23.98
29.52
47.83
57.22
53.77
35.47
1927.29
16.40
ОBR_RTN
4-4-4
50%
67.95
64,98
54,06
52.57
30.89
55,96
54.40
14.47
OBR GPTQ
4-4.4
50%
66.87
65.23
55.41
54.63
30.03
58, 80
55.16
13.92
Floating-poii
16-16-16
0%
84.49
85,38
84,96
86.11
64.16
80.51
80.93
2.85
QuuRoi(qunt-only)
3--16
0%
52.77
51,99
30.65
31,23
23.12
50.51
40.05
80.25
DuaRot+WANDs
4-4-16
50%
50.82
37.83
26.25
25.38
26.96
45.70
35.49
23245.17
3-70BB
SparsePT+GPPQ
44-16
60.12
52.81
35.02
38.30
23.29
53.51
43.84
41.39
OBR_RIN
444-6
50%
61.92
56.54
37.81
43.77
25.17
52.01
46
33.38
OBR GPTQ
4--16
50%
67.36
64.40
55.26
55.64
33.11
50.59
55.96
16.69
Implementation Details. Since our OBR framework, as well most other pruning and quantization methods (Frantar et al., 2022; Frantar & Alistarh, 2023; Sun et al., 2023), require calibration data to estimate input statistics, we follow standard practice and use 128 samples from WikiText2 with a sequence length of 2048 as the calibration set. For the Hadamard transformation, we test our OBR rotation matrices from various existing works, including QuaRot (Ashkboos al., 2024), SpinQuant (Liu et al., 2024), and FlatQuant (Sun et al., 2024). In addition, as our OBR treats pruning mask and quantizer as givens, it is potentially compatible with different pruning and quantization methods. Therefore, for pruning, we adopt the 0-1 mask generated by WANDA (Sun et al., 2023) as the default setting due its strong performance and training-free nature. We will further discuss OBR's generality across other pruning algorithms in Sec. 5.2. For the grouping ratio a in OBR quantization, we simply use a 50% as the default setting for all setups. For quantization, we include both the simple Round To-nd-TT--errsss (RTN) quantizer to obtain OBR RTN, and the more advanced GPTQ (Achiam et al., 2023) quantizer for OBR GPTQ as an extension.
5.1 EXPERIMENT RESULTS
Main Results. As shown in Tab. 1, the QuaRot (quant-only), which relies solely on quantization for compression, suffers from severe performance degradation under 4-bit, e.g., 132.97 perplexity for W3A4KV4 quantized Llama2-7B model. Furthermore, effectively combining quantization and sparsity is non-tiivial. For example, directly combining the existing quantization method Quarot (Ashkboos et al., 2024) with the pruning method WANDA (Sun et al., 2023) leads to unacceptable performance. For joint quantization and sparsification comparison, our OBR with a simple RTN quantizer can achieve even better performance than SparseGPT+GPTQ in most cases. For example, our OBR_RTY achieves even 3.71 better perplexity compared to SparseGPT+GPTQ on the Llama2-7B model. When using the more advanced quantizer GPTQ, our OBR_GPTQ can achieve a further 0.83 perplexity improvement. These experimental results demonstrate the effectiveness of the proposed OBR framework across different L.LMs and tasks.
7

---

Published as a conference paper at ICLR 2026
PPie-laae Dere- INTe-ppune
1450
4.0
2NT4-Spamc
1330
Rarkane (08))
12.5
TO11S
Delo
1200
700
10.0
200 350
Sequence Length
Sequence Length
Length
Figure 4: Comparison on runtime, FLOPs, and TOPS across different sequence lengths. We evaluate the performance of FP16-Dense, INTs-Dense, and INT4 2:4 Sparse GEMM single NVIDIA 1000-SXM4-800E GPU. The GEMM computation follows a typical LLM inference setting, where the weight matrix is W E R 1096xx0096 and the input activation is X E p=22xee.Jenx
Table 2: Comparison under other quantization wit-wss on WikiText2 perplexity (wiki2) and average zero-shot accuracy (0-shot) using the Llama2- 7B model.
W4A8KV8
W4A16KV16
Method
sparisty
wiki2]
0-shot
wiki2,
0-shot
Quarot(quant-ong
0%
80.525
39.98
80.25
40.04
Quarot+ WANDA
50%
5278.13
35.95
5272.07
35.92
SparseGPT=GPTQ
50%
8.53
59.41
8.53
59.47
OBR_RTH
50%
7.24
62.16
7.24
62.27
OBR_GPTQ
50%
6.87
63.39
6.86
63.33
Practical Speedups. Given that recent GPU architectures such Ampere and Hopper have naively supported INT4-sparse GEMM kernels, we compare the efficiency on batched matrix multiplication with other two baselines, i.e., INT4-dense and FP16-dense GEMM, in terms of latency. FLOPs, and TOPS. In Fig. 4, as input token length increases, the latency advantage of INT4+2:4 sparse pronounced. For example, sequence length of 4096, the INT4+2:4 sparse GEMM achieves a 5.9 speedup over FP16-dense and a 1,4x speedup over INT4-dense GEMM. Furthermore, thanks the 50% sparsity, INT4+2:4 sparse GEMM reduces theoretical FLOPs by 2x compared to its dense counterpart. Finally, when the GPU compute resources are fully saturated, i.e., with sequence length> 2048, the INT4+2:4 GEMM also achieves higher throughput in terms of TOPS. These results highlight the efficiency potential of low-bit sparse GEMM in real-world deployment compared to classic dense low-bit matrix multiplication.
GEMM becomes
Comparison on other Bits. We further evaluate the OBR framework under more bit-width con- figurations. Given that LLMs are known to be memory- we keep the weights quantized to low precision, i.e., 4-bit, while varying the activation and KV cache bit-width. Tab. 2 presents the results for W4ASKV8 and W4A 166V16 (weight-only quantization) settings. One can see that our OBR consistently outperforms all competitive baselines. Notably, OBR RTN with W4A8KV8+50% sparsity even surpasses weight-only quantization of SparseGPT+GPTQ by 1.29 perplexity. These results demonstrate the generality and effectiveness of OBR across different quantization bit-widths.
Results with SpinQuant. To further validate the generality of other rotation schemes, we apply OBR SpinQuant (Liu al., 2024), which introduces learnable rotation matrices for improved performance. Similar to the setup of QuaRot, we treat the rotation matrix as given and do not learn a dedicated rotation matrix for the joint crantization-ssarsiffeeee method achieves notable improvements over other competitive baselines e,g., OBR RTN achieves 7.69% average accuracy improvement against SpPr on zero-shot evaluation with 1lBma2-7B. Since the quantization-on only W3A4KV4 baseline employs the rotation matrices specifi- cally trained for quantization, our method is slightly inferior due the task gap. We believe learning rotation matrices specifically for low-bit and sparse setups holds potential for further improvement.
setting. As shown in Tab. 3, our
Other Sparsity Patterns. Semi-structured pruning, such as 2:4 sparsity, now well-supported by modern hardware to achieve practical acceleration. To this end, we further include comparisons under semi-structured pruning settings in Tab. 4. One can see that the advantages of our OBR become more apparent the compression becomes more challenging. In detail, both OBRRTN and OBR OBR_GPTQ consistently outperform other baselines under given setups. For example, under the challenging W4A4KV442:4 sparse setup, our OBR_RTN reduces perplexity by 18.8 and improves the average accuracy on zero-shot evaluation by 5.86% over the SparseGPT+GPTO. These promising results demonstrate the effectiveness of OBR in joint low-bit quantization and semi-structured sparsity.
8

---

Published as a conference paper at ICLR 2026
Table 3: Comparison of perplexity on WikiText2 and average accuracy on 0-shot commonsense reasoning tasks under SpinQuant (Liu et al., 2024) rotated weights.
Llama2-7B
Llama2-13B
Llama2-70B
Llama3-8B
Llama3-70B
Method
bits
spursity
wiki2]
0-shott
wikiz]
0-shott
wiki2, 0-shot?
wiki2.
0-shor"
wiki2]
0-shot
SpinQuant(quant-oouu
3-4-4
0%
8.24
58.95
6.39
66.78
4.21
74.09
10.50
60.29
9.64
63.64
SpinQuant+ WANDA
WANDA
444-
50%
1589.54
36.17
648.59
35.94
26,99
43.77
703.05
39.05
18565.64
36.27
SpurseGT--GPTQ
4-4-4
50%
22.57
45.42
8.47
57.39
4.75
72.75
16.37
53.67
21.74
51.14
OBR RTN
4-4-4
50%
10.40
53.11
7.57
60.72
4.71
72.85
13.10
55.22
18.18
49.30
GBRO
4-4-
50%
10.70
53.45
7.17
61.50
4.60
72.88
13.34
55.28
11.60
60.64
Table 4: Comparison on 4:8 and 2:4 sparsity with Llama2-7B models. The included base- lines have all been quantized using QuaRot W4A4KV4 configuration.
Figure 5: Applying the proposed OBR to
WANDA (Sun et al., 2023) pruning algo- rithm in single compression tasks.
70
WANDA
Method
sparsity
wiki2
0-shot
Wikitex22 Perplexity
60
WANDA+OUTS
Floating-point
5.47
70.46
50
SparseGPT+GPPQ
4:8
20.29
44.99
40
OBR_RIN
11.45
51.60
OBR_GPTQ
10.61
52.02
30
SparseGPT+GPTQ
2:4
34.76
40.52
20
OBR_RIN
2:4
15.96
46.38
10
GPTQ
50%
55%
60%
65%
70%
Table 5: Ablation on different pruning masks under W444KV4+50% sparsity using Llama2- 7B and QuaRot rotation.
Table 6: Ablation on partition ratio a.
Llama2-7B
Llama2-13B
1
wiki2]
0-shott
wiki2
0-shotf
pruning metire
wiki2,
0-shorf
75%
25%
9.96
53.56
7.70
60.22
Magnitude: [W]
8.92
56.51
50%
50%
9.23
56.49
7.29
62.37
SparseGPT: [[W] [ (/diog(H"))
9.28
55.45
25%
75%
9.07
57.06
7.09
63.20
WANDA: /
8.40
53.45
20%
80%
8.89
56.79
7.43
61.53
5.2 ABLATION STUDIES
Different Pruning Masks. In the proposed OBR framework, the pruning mask is treated given, making our method compatible with various existing pruning algorithms. the above main experiments, primarily adopt masks generated from WANDA (Sun et al., 2023) pruning. To further evaluate the effectiveness of other pruming metrics, we report in Tab. 5 the results using magnitude- based, SparseGPT-bbase (Frantar & Alistarh, 2023), and Random pruning masks. Thanks the error compensation from OBR, even the naive magnitude metric can achieve satisfactory performance. These results demonstrate the robustness of the proposed method across different pruning metrics.
For quantization error compensation in OBR, we adopt a simple strategy that splits weights into two groups with the first proportion as the eviction set Eoz and the remaining as the retain set R2. followed by the OBR error transfer. To further understand how the partitioning ratio affects error compensation, we conduct an ablation study with different a. As shown in Tab, 6, transferring the error from 20% elements to the remaining 80% leads to performance drop due an insufficient compensating number. Conversely, migrating 75% of the error to only 25% of the elements also yields suboptimal results due to low-quality compensation. As a trade-off, we adopt a 50% partitioning ratio for constructing E2 and Rt as our final design.
Partition Ratios for OBR Quantization.
5.3 DISCUSSION
OBR for Pruning Only. As shown in Sec. 4.3, the proposed OBR can be potentially applied to a single compression task to compensate for errors produced by given compression algorithm. To this end, we first extend our OBR framework to the pruning-only task. Specifically, we apply the proposed OBR to WANDA (Sun et al., 2023) by compensating for post-pruning weight distortions. The perplexity results on WikiText2 under different sparsity ratios are reported in Fig. 5. Equipped with OBR, WANDA consistently achieves lower perplexity under given sparsity levels. For instance, at 60% sparsity, WANDA+OBR improves perplexity by 0.53 compared to the original WANDA, and this performance gain becomes more pronounced when sparsity increases. These

---

Published as a conference paper at ICLR 2026
Rotated weight: rotate(w)
Pruned Weight: prune(rotate( w))
OBR for pruning: A Awarme
erom
0re1t
2.014
0/005
0.817
65627
01005
0.00%
11603
01822
5105
01003
00000
0.002
E2003
0
-12
-024
1.5%
2048
3553
3072
3524
4005
312
-004
1,56
2.46
2280
3812
3356
4000
3.000
312
1824
3356
2866
2280
3012
3354
405
OBR for quantization: Aw-uunt
Total OBR compensation: A Aw
Low-bit and sparse weight: W
0.8330
5.014
0.012
0,8225
0.9325
0.100
21095
0.330
08014
03313
38300
0.300
2.0022
0.300
0.300
512
1024
1536 2049
35.00
3072
3504
-002
0.0003
512
-834
1556
2046
2560
3572
3554
4000
0.300
512
1034
1556 2068
2542
3072 3554
and
Figure 6: Distribution visualization of different stages in the OBR framework. The weight matrix taken from the layer. O proj layer from the Llama2-7B model. Due the row-wise decoupling design in OBR, we visualize the distribution of the first row here and give full matrix visualization Appendix D. The z-axis represents the C.n. channel index, and the y-axis denotes the absolute value of weight elements.
results suggest that OBR can potentially serve as a generic post- processinn enhancement for existing pruning algorithms to improve performance without retraining.
OBR for Quantization Only. We further apply the pro- posed OBR to pure quantization- based compression scenario. Specifically, similar to the process described in Sec. 4.3, we first redistribute the rotated weights using OBR compensation to prepare weights more suitable for subsequent quantization. Then, we use the RTN quantizer to obtain low-bit weights. We compare this variant with the baseline that directly applies RTN quantization to the rotated weights without OBR. The results are shown in Tab. 7. As can be seen, the compensation from OBR significantly improves RTN quantization, e.g., 2.17 reduction in perplexity and a 3.88%
Table 7: Results of OBR for RTN quan- tizer in quantization-only tasks.
Methods
W-A-KV
wiki2,
0-shott
Floating-poin
16-16-16
5,47
70.47
GPTQ
4-44
6.33
66.09
RTN
4-44
9.04
60.10
OBR+RTN
4-4
6.87
63.98
gain in zero-shot accuracy. Although OBR is not specifically designed for quantization, OBR+RTN still achieves comparable results to GPTQ with a 0.54 perplexity gap. These results demonstrate the potential of our proposed method in quantization-onl tasks.
Illustrative Visualization of OBR. In Fig. 6, we visualize the weight distribution at different stages of the proposed OBR pipeline. The AwPrune can effectively recover the information loss caused by pruning while preserving the original sparsity. Moreover, the compensation AWOBR does not introduce additional outliers, and this flat distribution facilitates the subsequent quantization process. At last, the magnitude of the compensation introduced by OBR is comparable to that of the original weights, indicating that our OBR compensation is not noise but structured information capable of restoring the knowledge lost during compression.
6 CONCLUSION
In this work, we propose Optimal Brain Restoration (OBR), a unified framework that jointly performs pruning and quantization by computing an optimal compensation to reconcile the conflicting require- ments of different compression methods. We begin by formulating a second-order Hessian-based objective that minimizes downstream task degradation. To make the optimization tractable, we introduce a row-wise decoupling approximation. Furthermore, we develop group error compensation, which redistributes compression-induces errors through a closed-form solution. By aligning the weight distribution with the distinct demands of each compression technique, OBR is among the first methods to support INT4 quantization combined with 50% sparsity for LLMs. Experimental results demonstrate that our approach significantly outperforms existing methods and achieves up to 4.72 x practical speedup over the FP16-dense baseline.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. GPT-4 technical report. arXiv preprint TEXii:2303.08774 2023.
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated LLMs. Advances in Neural Information Processing Systems, 37:100213-100240 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. PIQA: Reasoning about physical commonsense natural language. In AAAI Conference on Artificial Intelligence, volume 34, pp. 7432-7439, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. QUIP: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36: 4396 4429, 2023.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 10044 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 180.00557. 2018.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. GPT3.int8 Q: 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 30318-30332, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.008: 1803.0333, 2018.
Elias Frantar and Dan Alistarh. Optimal brain compression: framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475-4488, 2022.
Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323- 10337.. PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.1732: 2022.
Jinyang Guo, Jianyu Wu, Zining Wang, Jiaheng Liu, Ge Yang, Yifu Ding, Ruihao Gong, Haotong Qin, and Xianglong Liu. Compressing large language models by joint sparsification and quantization. In International Conference on Machine Learning, 2024.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in Neural Information Processing Systems, 28, 2015.
Simla Burcu Harma, Ayan Chakraborty, Elizaveta Kostenok, Danila Mishin, Dongho Ha, Babak Falsafi, Martin Jaggi, Ming Liu, Yunho Oh, Suvinay Subramanian, et al. Effective interplay between sparsity and quantization: From theory to practice. arXiv preprint arXiv:2405.20933 20935. 2024.
11

---

Published as a conference paper at ICLR 2026
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations, 1(2):3, 2022..
Xing Hu, Yuan Cheng, Dawei Yang, Zukang Xu, Zhihang Yuan, Jiangyong Yu, Chen Xu, Zhe Jiang, and Sifan Zhou. Ostquant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting. arXiv preprint arXiv:2501.13987 2025.
Yann LeCun, John Denker, and Sara Solla.
Optimal
brain damage.
Advances in Neural Information Processing Systems, 2, 1989.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang. Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-awan weight quantization for on-device IIm compression and acceleration. Proceedings of machine learning and systems, 87-100, 2024a.
Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. Qserve: W4A8KV4 quantization and system co-design for efficient LLM serving. arXiv preprint Axiv:22405.04532 2024b.
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: LLM quantization with learned rotations. arXiv preprint arXiv:2405. 16406, 2024.
Shuming Ma, Hongyu Wang, Shaohan Huang, Xingxing Zhang, Ying Hu, Ting Song. Yan Xia, and Furu Wei, Bitnet b1. 58 2b4t technical report. arXiv preprint arXiv:2504. 122.00..1228,.. 12285, 2025.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-pruner: On the structural pruning of large language models. Advances in Neural Information Processing Systems, 36:21702-21720, 2023.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07888 2016.
Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08377 2021.
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tij- men Blankevoort. A white paper on neural network quantization. arXiv preprint arXiv:2106.08995, 2021.
NVIDIA. NVIDIA Hopper Architecture In-Depth, 2021. URL / ham/bbog/nviidke htwp:/iiddaaaapppeeeeeeeeeeeeeee www.pppprrrrcitttttrreereeeeeeee in-depth/.
NVIDIA. Structured sparsity in the NVIDIA Ampere architecture, 2022. URL https: /ttp:////eeeeeeeeeeeeeeeeeeeeeet .nviii. httpa///c///ooo//t/t/teeeeeeeeee wtt.rrruuuuuuuddrrssstt-------ee
aatddii-ampereee ampere aww.rreccccctttttueeee and- and-anpddlaeeea applications in -awetriiooo----snac-----ee-e-
QwenTeam. Qwen2.5: A party of foundation models, September 2024. URL https:///weell.. qwen2. 5/.
github.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9):99-106.
2021.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1169,, 2023.
Yuxuan Sun, Ruikang Liu, Haoli Bai. Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xi- anzhi Yu, Lu Hou, Chun Yuan, et al. Flatquant: Flatness matters for LLM quantization. arXiv preprint 21Xiv:2410.09426, 2024.
12

---

Published as a conference paper at ICLR 2026
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09228, 2023.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. QUIP#: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint urXiv:2402.04396. 2024.
Ying Wang, Yadong Lu, and Tijmen Blankevoort. Differentiable joint pruning and quantization for hardware efficiency. In European Conference Computer Vision, pp. 259-277. Springer, 2020,
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference Machine Learning, pp. 38087-38099 PMLR, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 1iciv:1900 (91199905779880 2019.
Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannis- traci. Plug-and-play: efficient post-training pruning method for large language models. International Conference on Learning Representations, 2024.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A SUMMARY OF OBR ALGORITHM
In Algo. I, we provide a detailed pseudocode to illustrate the process of obtaining joint low-bit and sparse LLM weights in the proposed OBR framework.
Algorithm 1
Optimal Brain Restoration (OBR)
Input: Hadamard rotated weight matrix W R Conux Hessian approximation H E Cin partitioning ratio
Output: Low-bit and sparse weight W gCoexCinn
// Step1 Pruning
M E {0, 1 prune(W)
Wprune W M
1/ Step2 OBR compensation Initialize A AWOBR as zero matrices in RCuu XCCn C.n.
for c = 1 1... Cont do
// OBR for pruning
R1 < ii M 11,
E (i Mcj 0}
b + HR,F WT
c.E, Prune --HR, bj
Ri
W + Worune Awuree
CB /1 OBR for quantization
equnnt t w quantize(W)
+ aa. |
Eg + (ri....,reeh rth,
R2 + frtt1, r|R
b2 + HRRE, E
guant
Aw guant
R2 - Rg b
Compensation ion Gathering
A AWOBR +
Aw prune
e,
A AWOBR + Aw quant
e,
ft
end for
Wouent Werume AWOBR
// Step3 Quantization
W + quantize(W!uan))
B COEXISTENCE OF QUANTIZATION AND PRUNING.
A key motivation behind the proposed OBR is the compatibility of low-bit quantization and sparsity in the Hadamard-rotate LLMs. In this section, we provide empirical evidence to justify this motivation. Specifically, visualize the sparsity distribution of 1lama2-7B and Qwen2.5-7B models quantized by different rotation frameworks, Le., QuaRot (Ashkboos et al., 2024), SpinQuant (Liu et al., 2024), and FlatQuant (Sun al., 2024). Fig. 7 offers the results. Interestingly, even without any explicit pruning operations, the quantized LLMs inherently exhibit non-trivial sparsity. For instance, Llama2- 7B with QuaRot reaches average sparsity of 14. 14.28%. Based on the observation of this coexistence. we design our OBR to achieve more aggressive LLM compression.
C MORE EXPERIMENTS
Comparison with BitNet. BitNet-22B44 (Ma al., 2025) is recently proposed 1.58-bit LLM that trained from scratch to achieve aggressive compression with strong performance. In this section, give brief comparison between the BitNet-2B-4T model and Qwen2.5 compressed
14

---

Published as a conference paper at ICLR 2026
(a) Llama2-7B with Quarot
(b) Llama2-7B with SpinQuant
(c) Qwen2.5-7B with FlarQuant
80
40
60
Couits
30
5.5
20
20
30
20
10
13.00 13.25 13.50 13.75 1400 14.25 14,50
14.75
15,00
1500 15.25
15.50
13.75
14,00
14,25
14.50
15 15.00
5.00
15.00
20.00
25:00
30.00
Laver Sparsiy
Ratio
Layes Spansity Ratio
Figure 7: Distribution of layer-wise sparsity across LLMs under different rotation methods, All models are quantized with W4A4K RTN quantizer.
Table Comparison between Bit and our OBR compressed @0wn2.55-Insruue models.
methods
quantization
sparisty
PIQA
BoolQ
HellaSwag
ARC-E
ARC-C
WinoGrande
Avg.
Wiki2
BiiNet-2B-4T
W1.58A8KV16
0%
76.55
80.43
68.39
74.66
49.40
72.22
70.27
13.67
Swen2..511.5B + OBR
W4A8KV16
50%
68.99
66.88
$2.68
62.50
35.24
60.77
$7.84
15.06
Qwen2.5-1.5B + OBR
W4A4KY4
50%
67.25
68.01
51.18
56.99
32.94
55.96
55.38
14.92
Owen2.5-3B
W4A8KV16
50%
77.19
62.86
60.06
62.90
63.06
11.07
Qwen2.5-3B OBR
W4A4KV4
50%
72.14
76.67
60.43
60.69
41.13
65.59
62.77
11.79
Table 9: Ablation experiments other calibration dataset. We change the calibration set to the C4 (Raffel et al., 2020) dataset for the generation of activation statistics and keep other setups the same.
Llama2-7B
Llama2- 13B
Llama3-8B
dataset
method
perplexity|
0-shot
perplexity
0-shot
perplexity.
0-shot
Spars
12.94
51.57
7.89
60.74
16.40
$3.77
wikitext2
OuT
9.23
56.49
7.29
62.37
14.47
$4.40
Ours_GPTQ
8.40
53.45
7.06
62.60
13.92
55.16
SpunseGPT+GPTQ
18.36
51.18
9.69
60.48
23.02
$3.87
Ours Our__RTN
10.74
58.00
8.74
62,88
18.23
56.02
Ours. GPTQ
10.40
57.95
8.22
63.16
17.90
57.12
using our OBR. As shown in Tab. 8, our post-training method achieves comparable performance. To be specific, Dwwen..5-3B+OB
WT4A4KV4450%$paraaa achieves better perplexity on WikiText2 and comparative performance zero-shot accuracy. It should be noted that the performance of OBR can be further boosted when future, more advanced base LLMs are proposed. Moreover, the resulting W444KV4+50% sparse LLMs can be seamlessly deployed. such as in NVIDIA Ampere and Hopper, whereas BitNet requires specially designed kernels and customized implementations. At last, our method provides stronger generalization and flexibility. BitNet currently offers only model size and typically requires training from scratch, which is computationally expensive and impractical for users with domain-specific or confidential data. In contrast, our OBR framework is a general post-training compression approach that can be directly applied to existing models of different sizes, enabling users to efficiently adapt their own LLMs without re-training.
Allaioon on other Calibration Set. In the proposed OBR, we use the WikiText-2 (Merity et al., 2016) dataset to obtain activation statisties. To further verify the robustness across different calibration sets, we additionally experiment with the C4 (Raffel et al., 2020) dataset for calibration. The results are shown in Tab. slight performance degradation on WikiText perplexity due to the train-test shift. However, models calibrated with C4 achieve better results on zero-shot tasks, and this advantage is more pronounced with our OBR. For example, in the Llama3-SB experiment with C4, SparseGPT+GPTQ achieves only a 0.1%% accuracy improvement, whereas the proposed GPTQ delivers a 1.96% gain. Moreover, both OBR_TN and OBR_GPTT consistently outperform the GparseGPT+GPTQ baseline across all calibration sets and base models under the same compression settings. The above results demonstrate the generalization of our method under other calibration sets.
can be seen, when switching to the C4 dataset, all compared methods suffer a
15

---

Published as a conference paper at ICLR 2026
Table 10: Comparison of perplexity score on WikiText2 and accuracy on zero-shot common sense
reasoning tasks using the rotation matrix from FlatQuant (Sun et al., 2024).
Model
Method
#Bits (W-A-KV)
Sparsity ratio
PIQA
BoolQ
HellaS,
Are-e
Are-c
Wino., (1)
Avg. (*)
Wiki2
(1)
(1)
(1)
(1)
(1)
(L)
Floating-Pun point
16-16-16
0%
79.11
77.71
76.02
74.49
46.33
69.14
70.47
5.47
FlatQuant( PlatQuantqquant-one quant-only)
4-4-4
77.48
74.62
73.64
72.56
43,00
68.27
68.26
5.79
Llama2-7B
FlatQuantquant--on
3-4-4
0%
75.68
73.94
69.44
67.85
40.96
64.17
65
6.74
SpurseGPT+ GPTQ
4-4-4
50%
73.56
50.40
65.36
61.11
34.73
62.75
57.99
7.75
OTN
4.4-4
50%
74.32
72.91
65.88
64,94
37.88
65.82
63.62
6.88
Ours GPTQ
4-4-4
50%
74.37
71.41
65.92
64.06
38.82
66.38
63.49
6.87
Floating-point
16-16-16
0%
80,52
80.55
37
77.48
49.15
72.14
73 20
4.88
FlatQuanti quatQuaniquanttoonn only)
4-4-4
79.00
79.39
77.44
76.47
70.17
71
5.11
Llama2-13B
FlatQuantquuantoonly
3-4-4
0%
78.56
78.04
75.35
70,66
44.97
70.09
69.61
5.70
SparseGPT-
4-4-4
50%
75.90
74.53
69.81
68 86
40.19
67.09
66.06 06
6.13
Ours Ours .PTO
4.4.4
76.66
73.94
71.44
71
5.84
4-4-4
50%
76.61
73.27
71.39
72.10
42.49
68.43
67.38
5.84
Floating-poit
16-16-16
0%
80.85
80.98
79.17
77.74
53.244
73.40
74.23
6.13
FlaQuaan(quant-only))
4-4-4
0%
79.33
79.36
76.64
75.21
48.46
72.06
71.84
6.97
Llama3-8B
Hlat HarQQuantqquant-only
3-4-4
0%
75.68
69.42
71.21
67.47
39.85
67.44
65.17
9.14
SpaseGGTT-GPTQ
4-4-4
50%
69.97
74.95
63.59
57.03
34.64
65.19
60.89
13.32
Ours KTN
4-4-4
50%
74.16
77.61
66.86
68.81
40.78
0.6661
65 80
9.12
Ourx_GPTQ
4.4.4
50%:
73.99
77.16
66.74
69.11
41.30
68.19
66.08
9.10
Floating-point
16-16-16
0%
80.14
85,96
79.57
76.47
51.19
69.46
73.78
8.35
Mlt(Quantquant-onyy
4.4-4
0%
73
85.87
78. 48
77.23
$1.02
68.82 82
73.25
8.40
Qwen2.5-7B
PlatQuantqquant-ong only)
3-4-4
73.23
82.20
74.51
69.78
48.29
10.08
SparscCPT-GPTQ
4-4-4
50%
73,56
83.70
68.50
68.10
42.49
64.01
66.72
14.53
Ours RTN
4.4.4
50%
74.70
85.41
71 12
74.49
49.83
66.30
70.32
9.55
Ours GPTQ
4-4-4
50%
76.66
85.08
70.68
74.12
50.85
67.56
70.82
9.51
Floating-point
16-16-16
0%
81.39
90.54
85 25
77.
$8.62
73.16
77.66 66
5.32
FlaQQuan(quaat-only
4-4-4
0%
80.96
89.39
83.86
79.17
57.94
73.95
77.54
5.82
Qwen2.5-32B
FatQuantquunt-only)
3.4-4
78.94
87.83
81 45
74.87
54.69
67.64
74.23
6.79
SparseGPT+CPPQ
4-4-4
50%
80.20
89.94
0.7986
73.78
$2.65
72.14
74.76
8.06
Ours RTN
4-4-4
50%
77.86
90.00
80.00
78.45
57.17
72.77
76.04
6.81
Ours GPTQ
4-4-4
50%
79.11
89.45
80.00
77.31
59.22
72.61
76.28
6.79
Performance on FlatQuant. In the main paper, we present the application of our OBR on the LLMs rotated by QuaRot (Ashkboos al., 2024) or SpinQuant (Liu et al., 2024). To further evaluate the generalization ability of our method on other Hadamard rotation frameworks, we additionally include the comparison results with the FlatQuant (Sun al., 2024) method. The experimental results are shown in Tab. 10. As can be observed, OBR continues to deliver strong performance compared to the SparseGPT+GPTQ bascline across various base models. Interestingly, comparing with QuaRot and SpinQuant, when using stronger rotation matrix from FlatQuant, the W4A4KV4 + 50% sparsity LLMs using our OBR can achieve performance on par with their FP16 counterparts. For example, the perplexity gap on Llama2-7B is merely 1.4, compared with the gap of 2.93 in QuaRot. This result further indicates the potential that our OBR can scale in parallel with a more advanced rotation framework.
Results on Qwen Families. In this section, we take Qwen2.5- -0wen2.5-Instrucc (7B/3BB) as a representative demonstrate the generalization capability of the proposed OBR on other LLMs. The experimental results are presented in Tab. 10. Given Qwen as the base models, OBR consistently outperforms other strong baselines across different scales. For instance, OBR RTN surpasses SparseGPT+GPTO by 4.98 perplexity on the Qwen2.5-7B model. In addition, OBR_RT also outperforms the quantization- only W3A4KV4 baseline by 0.53 perplexity. These results demonstrate the strong generalization ability of the proposed OBR across different L.M families.
Calibration Time Cost of OBR. Tab. 11 reports the time cost for compressing models of different scales using OBR. As one can see, for smaller models such as the 7B model, OBR can produce W4A4KV4 + 50% LLMs in about 2 hours. For even larger models, such as the 70B, the proposed OBR completes in roughly 36 hours. Since our OBR adopts row-wise decoupling strategy, it requires solving a linear equation for each row, making it slower than SparseGPT+GPTQ. Nevertheless, we emphasize that post-training compression needs to be performed only once per model. As result, this cost has only minimal impact on large-scale deployment. Moreover, the promising performance of OBR against other baselines under aggressive compression further justifies its advantages.
16

---

Published as a conference paper at ICLR 2026
Table 11: Calibration time results for Llama model family.
The reported times correspond to
QuaRot (Ashkboos et al., 2024) rotation single A A100 GPU.
Llama family
2-7B
2-13B
2-70B
3-8B
3-70B
SparseGPT+GPT
45min
54min
1h53min
48min
2h9min
OBR_RTN RTN OBR_GPTO
2h10min
4212min
35h30min
2h30min
35h28min
2h18min
4h30min
35h45min
2h40min
35h47min
Rotated weight: rotate( W)
Pruned Weight: Bright::runn(eig pruno W)
OBR pruning: A AWrrun
OBR for quantization: A Wyront
Total OBR compensation: AWOBR
Low-bit and sparse weight: W
Figure 8: Visualization of the full weight matrix at different stages in the proposed OBR pipeline. The x-axis corresponds to the Cn dimension, and the y-axis is the Cout dimension. The weight matrix is taken from the layer. 0. proj layer from the Llama2-7B model, and absolute values are used to enhance visual clarity.
D MORE VISUALIZATION
In Fig. 8, we present visualizations of the full weight matrices at different stages of OBR processing. can be observed that the rotated weight matrix inherently exhibits strong row-wise independence, as indicated by the similarity patterns across rows in rotate(W) W). Morcover, the compensation terms Warune and Aquant produced by OBR clearly contain useful information, since they share a similar magnitude with the prune(rot (ooee(tttte ato (Wa)atteWWW..... Therefore, if the OBR compensation were merely noise, perturbations of this magnitude would lead to significant errors. In addition, the overall compensation A ABR also demonstrates row-wise independence, where some rows have large magnitudes while others have small ones, yet column dimensions instead exhibit similar patterns. This observation further justifies proposed row-wise decoupling strategy.
E LIMITATION AND FUTURE WORK
While the proposed OBR can effectively redistribute weights to reconcile the differing distributional requirements quantization and pruning, there are several avenues for further improvement. First, OBR relies on row-wise decoupling strategy estimate the full Hessian. This approximation renders the original objective tractable, but it requires solving a linear system for each row of the weight matrix. Although this overhead is acceptable in model compression tasks, where the compression algorithm needs to run only once, further accelerating the compression process for large-scale L.MMs remains meaningful. Second, the current implementation of OBR treats the pruning
17

---

Published as a conference paper at ICLR 2026
mask and quantization rotation matrix as fixed given imputs. However, recent quantization studies (Liu et al., 2024; Sun et al., 2024) suggest that introducing gradient-bart based optimization can further boost performance. Thus. designing learnable pruning masks and rotation matrices compatible with our OBR framework could lead to additional gains. Third, although OBR significantly outperforms individual compression methods under equivalent sub-4-bit settings, its advantage narrows at higher bit-widths, where standalone methods have not yet reached their performance limits. Developing more advanced algorithms to maintain superior performance across various bit-widths is also promising direction, and we leave it for future work.
18


Reviews:
[{"review_id": "K7odWE08gv", "reviewer": "Reviewer_JF8r", "summary": "This paper propose Optimal Brain Restoration (OBR), combining the sparse pruning and quantization with error compensation for model compression. Experiments show that OBR incurs only a 1.4 perplexity degradation on Llama2-7B to enable aggressive W4A4KV4 quantization with 50% sparsity, delivering up to 4.72× speedup and 6.4× memory reduction compared to the FP16-dense baseline.", "questions": "1. Maybe the training aware method by combining the quantization and pruning is more practical, especially for the use of W4A4 + 4:2 sparsity.\n2. What's dominant application of 4:2 sparsity? Since few examples of 4:2 sparsity are released in the industry.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A training free method with the combination of pruning and quantization is achieved, by the intervene between pruning and quantization by computing an optimal compensation that minimizes the impact of compression on downstream tasks. \n\nThe experiments show SOTA results, indicating the first algorithm to enable W4A4KV4+50% sparsity LLMs, without requiring any additional retraining.", "weaknesses": "The sparse pruning can be used based on 4:2 sparsity, the hardware implementation is efficient, but the accuracy drop of W4A4KV4+50% sparsity LLMs is larger than 5%, which hinders the application of this method. The experimental results are based on simple QA tasks, and lack of long context tasks and long-cot tasks, so the results are not convincing.\n\nAlso the algorithm should verify the results on the new data formats such as NVFP4 or MXFP4, which are popular on the modern hardware.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qKb1Kw4f35", "reviewer": "Reviewer_ruw5", "summary": "This paper proposes Optimal Brain Restoration (OBR), which reconciles the conflict between pruning and quantization via a second-order approximation and a closed-form group error compensation scheme: errors introduced on pruned/quantized weights are projected—through Hessian sub-blocks—onto the retained weights to minimize accuracy loss under joint compression. On models such as Llama-2/3 and Qwen-2.5, OBR achieves better zero-shot accuracy and WikiText-2 perplexity than strong baselines (e.g., SparseGPT+GPTQ and naïve “quantization+pruning”) under the W4A4KV4 + 50% sparsity setting. For efficiency, the authors implement an INT4 + 2:4 sparse GEMM using NVIDIA CUTLASS and report kernel-level latency/throughput curves (e.g., a 5.9× speedup over dense FP16 and 1.4× over dense INT4 at sequence length 4096).", "questions": "same as the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Clear and well-justified motivation. The paper precisely frames the problem: quantization and pruning each approach their limits, and naïvely combining them causes mutual interference, motivating a method that explicitly resolves this conflict. It also provides empirical evidence that non-trivial sparsity naturally emerges after quantization, supporting the coexistence and complementarity of the two representations and thus the direction of joint compression.\n\n2. Sound theoretical development. Starting from a second-order (Hessian-based) objective, the paper introduces group error compensation, derives a closed-form optimum, and explains the Hessian’s bridging role in transferring errors—offering good interpretability.\n\n3. Methodological generality. The proposed approach can be combined with popular sparsity/quantization techniques such as SparseGPT, OBR, QuaRot, SpinQuant, and FlatQuant, making it a broadly applicable compression method.\n\n4. Custom kernel implementation. The authors implement dedicated GEMM CUDA kernels with NVIDIA CUTLASS (INT4 with 2:4 sparsity), which strengthens practical usability and reflects substantial engineering effort.\n\n5. Comprehensive setup and strong results. Experiments span multiple sizes of Llama-2/3 and Qwen-2.5, evaluating zero-shot benchmarks and WikiText-2 perplexity, with comparisons against quantization-only and strong prune+quantize baselines (e.g., SparseGPT+GPTQ). The method shows high speedups and substantial memory reduction.", "weaknesses": "1. Lack of end-to-end inference metrics. The paper does not report TTFT (time-to-first-token), tokens/s, or throughput under concurrency. Although kernel-level gains are provided, their translation to full-pipeline performance is unclear.\n\n\n2. Narrow evaluation tasks. The experiments could be broadened to include more task types—such as code generation or long-context workloads—to better demonstrate generality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ZBWDp29EX", "reviewer": "Reviewer_vBwQ", "summary": "This paper proposes Optimal Brain Restoration (OBR), a general training-free framework for jointly achieving quantization and sparsification of Large Language Models (LLMs). Its core motivation stems from the fundamental conflict between quantization and pruning: quantization prefers compact weight distributions (to suppress outliers) while pruning favors high-variance distributions (to identify important weights). OBR reconciles these conflicting objectives by introducing a Hessian-based error compensation mechanism between pruning and quantization, which \"transfers\" the information lost due to pruning or quantization distortion to more robust weights. This method transforms the original second-order optimization problem into a closed-form solution via row-wise decoupling and group error compensation, making it efficient and interpretable. Experiments show that OBR achieves LLM compression of W4A4KV4 + 50% sparsity, incurring only a 1.4 perplexity increase on Llama2-7B while delivering 4.72× inference speedup and 6.4× memory reduction.", "questions": "1.Table 6 demonstrates that employing different α values across various models can lead to superior performance. Is it feasible to design an adaptive α strategy?\n\n2.OBR’s compensation relies on the calibration dataset; if the calibration data distribution differs significantly from that of the target downstream task (e.g., calibrating on general-domain text but deploying on mathematical reasoning), could the compensation amplify errors rather than mitigate them?\n\n3.Is the row-wise decoupling assumption in Section 4.1 of the paper reasonable? Could the authors provide an ablation study comparing the error accumulation under the decoupling assumption versus partially coupled alternatives?\n\n4.The paper compares OBR against baselines such as SparseGPT+GPTQ, but does not include comparisons with other joint compression methods like JSQ. Under the W4A4KV4 + 50% sparsity setting, is OBR’s advantage over JSQ and similar methods truly significant?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.Applying Classical Hessian Optimization to Modern LLM Joint Compression Scenarios: Closed-Form Solution Design Balances Efficiency and Performance\n\n2.OBR functions as a post-processing module, compatible with various pruning methods, quantizers, and rotation schemes.\n\n3.This work systematically extends the classic Optimal Brain Damage/Surgeon (OBD/OBQ) theory, generalizing it from single-task scenarios to joint compression settings.\n\n4.This work benchmarked the latency, FLOPs, and TOPS of INT4 2:4 sparse GEMM on the A100 GPU and implemented a custom CUTLASS kernel.", "weaknesses": "1.The calibration phase requires solving a system of linear equations for each row, resulting in significant time overhead during calibration.\n\n2.The partitioning ratio α in the quantization compensation phase lacks theoretical grounding. Although validated through ablation studies, the optimal α may vary across layers and models.\n\n3.OBR’s calibration data sensitivity is insufficiently analyzed—it is only evaluated within general-domain text corpora (WikiText2 and C4) and lacks cross-domain robustness experiments, making it difficult to guarantee the stability of its compensation mechanism in real-world, domain-specific deployment scenarios.\n\n4.The inter-row decoupling hypothesis, due to ignoring the correlations between channels, may lead to underestimation of error propagation and accumulation of non-orthogonal errors in LLM compression.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hElOhdI3eP", "reviewer": "Reviewer_Y5Wm", "summary": "The key idea is to model the effect of weight perturbations due to pruning + quantization via a second-order Hessian objective, approximate it tractably (row-wise decoupling, approximate Hessian via Fisher × identity factorization, then derive a closed‐form compensation term that “restores” performance by shifting information from more error‐sensitive weights to more robust ones.", "questions": "1 For pruning, you adopt the mask from WANDA as default. Have you tested OBR using other pruning algorithms", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "Model compression for LLMs is a hot area and bridging quantization + pruning is of high practical relevance", "weaknesses": "1 The general idea of combining quantization + pruning is not new (the authors themselves reference earlier works on small networks like DJPQ (Wang et al., 2020) and OBQ (Frantar & Alistarh, 2022) for the joint problem). \n\n2 The Hessian‐based analysis for pruning quantization has precedent (e.g., OBD/OBQ for pruning, Fisher/Hessian analyses, quantization with perturbation bounds). The authors are extending to LLM scale, but one might argue that the core conceptual novelty is limited: the compensation term is derived, but the big leap is engineering (applying it to huge models) rather than algorithmic novelty.\n\n3 Some sections feel dense and hard to follow", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)