SliderQuant: Accurate Post-Training Quantization for LLMs
Authors: Shigeng Wang
Chao Li
Yangyuxuan Kang
Jiawei Fan
Zhonghong Ou
Anbang Yao
Venue: ICLR 2026 Poster (2026)
Keywords: Large language models
post-training quantization
low-bit neural networks
model compression
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper presents SliderQuant, a new post-training quantization framework for LLMs, which is superior to existing methods.

Abstract: In this paper, we address post-training quantization (PTQ) for large language models (LLMs) from an overlooked perspective: given a pre-trained high-precision LLM, the predominant sequential quantization framework treats different layers equally, but this may be not optimal in challenging bit-width settings. We empirically study the quantization impact of different layers on model accuracy, and observe that: (1) shallow/deep layers are usually more sensitive to quantization than intermediate layers; (2) among shallow/deep layers, the most sensitive one is the first/last layer, which exhibits significantly larger quantization error than others. These empirical observations imply that the quantization design for different layers of LLMs is required on multiple levels instead of a single level shared to all layers. Motivated by this, we propose a new PTQ framework termed **Sliding**-lay**er** **Quant**ization (SliderQuant) that relies on a simple adaptive sliding quantization concept facilitated by few learnable parameters. The base component of SliderQuant is called inter-layer sliding quantization, which incorporates three types of novel sliding window designs tailored for addressing the varying quantization sensitivity of shallow, intermediate and deep layers. The other component is called intra-layer sliding quantization that leverages an incremental strategy to quantize each window. As a result, SliderQuant has a strong ability to reduce quantization errors across layers. Extensive experiments on basic language generation, zero-shot commonsense reasoning and challenging math and code tasks with various LLMs, including Llama/Llama2/Llama3/Qwen2.5 model families, DeepSeek-R1 distilled models and large MoE models, show that our method outperforms existing PTQ methods (including the latest PTQ methods using rotation transformations) for both weight-only quantization and weight-activation quantization under diverse bit width settings. Code is available at https://github.com/deep-optimization/SliderQuant.

Published as a conference paper at ICLR 2026
SLIDERQUANT: ACCURATE POST- QUANTI- ZATION FOR LLMs
Shigeng Wang 2 Chao Li Yangyuxuan Kang', Jiawei Fan', Zhonghong Ou?, Anbang Yao
IItel Labs China, BUPT [shigeng. wang, chao3, 11, yaagyuxua kang, jiawei, fan, anbang.yaoleinnee yao)eintes..com
ABSTRACT
In this paper, we address post-training quantization (PTQ) for large language mod- els (LLMs) from an overlooked perspective: given a pre-trained high-preodt eigh-precisoon LLM, the predominant sequential quantization framework treats different layers equally, but this may be not optimal in challenging low bit-width settings. Based on three major categories of sequential PTQ methods, we empirically study the quantization impact of different layers on model accuracy, and observe that: (I) shallow/deep layers are usually more sensitive to quantization than intermediate layers; (2) among shallow/deep layers, the most sensitive one is the first/last layer, which exhibits significantly larger quantization error than others. These empirical observations imply that the quantization design for different layers of LLMs is re- quired on multiple levels instead of a single level shared to all layers. Motivated by this, we propose new PTQ framework termed Sliding-layer Quantization (Slid- erQuant) that relies on a simple adaptive sliding quantization concept facilitated by few learnable parameters. The base component of SliderQuant is called inter- layer sliding quantization, which incorporates three types of novel sliding window designs tailored for addressing the varying quantization sensitivity of shallow, intermediate and deep layers. The other component is called intra-layer sliding quantization that leverages an incremental strategy to quantize each window. As a result, SliderQuant has strong ability to reduce quantization errors across lay- ers. Extensive experiments on basic language generation, zero-shot commonsense reasoning and challenging math and code tasks with various LLMs (including waw.aaaaaaaaa//////...
model families, DeepSeek-R1 distilled models and large MoE models) show that method outperforms existing PTQ methods (including the latest PTQ methods using rotation transformations) for both weight- only quantization and weight-activatin quantization under diverse bit width set- tings. Code is available at
INTRODUCTION
Transformer- -baeedee----eedddddee large language models (LLMs) /Vaswani et al. 2017 Brown ct al. 2020 Achiam al. 2023; Anil et al., 2023 Liu et al.] 2024a Jacch et al. [022]] Guo et al.) 2025 have demon- strated extraordinary performance on a wide range of natural language processing tasks. However, deploying them in real-world scenarios poses great challenge due to their huge model sizes. Post- training quantization (PTQ) is a practically appealing way to reduce memory and computation de- mands of LLMS at inference. It approximates pre-trained high-precision models with low-precision replacements conditioned on a small number of calibration samples. without the need of the expen- sive retraining pipeline used in quantization-awee awaae training (Zafrir al. 2019 Ma al. 2024a Xu et al. 2024). Because of this, PTQ research has gained increasing attention in the LLM community.
Existing PTQ methods for LLMs generally use a sequential quantization framework: splitting a pre- trained LLM into the same-sized disjoint parts, and then quantizing them from the first to the last part separately. Early seminal works, such as (LM..in8() (Dettmers et al. (022), ZeroQuant (Yao) et al. 2022), GPTQ (Frantar et al.] (023)) and SmoothQuant (Xiao et al. (023). assume that differ- ent layers of a pre-trained LLM are independent to each other, and use the layer- wise quantization.
"This work was done when Shigeng Wang was intern at Intel Labs China. These authors contributed cqually the writing of the paper led by Anhang Yao who conceived the project. Corresponding author.

---

Published as a conference paper at ICLR 2026
L7mm22-78
Llama--1 138
Qwen? 5-148
Peplosity WaiTess
Staodklaaae QraaiQuaa
Perplesity WhiTew?
SxeetiQ
Peplesitt an WileTeer2
SeoetQquet
Ooeelqeet CBQ
A.-
(B2
SiskrQustt
CBQ
FiN6
PPis
FP15
3.K.
3.6
30
20
20
40
Layer index
Layer index
Layer index
Llmm-27B
L- 1-B
Qwen2. 5-14B
Perploiity on WilTTeur2
Perplexity on Wild WilTTer22
Perplsiity on Wild
8
100
75
Number quantized layers
Number quantized layer
Number of quantized layors
Figure 1: Illustrations on the quantization impact of different layers to model accuracy: (1) quantiz- ing a single layer (the first row) and (2) quantizing the first I layers (the second row) of LJama2-7B, Llama2-13B and Qwen2.5-14B, Here, we select representative layer-wise, block-wise and multi- block-wise quantization methods, SmoothQuant, OmniQuant and CBQ, and examine them under 4-bit weight-activation (W4A4) quantization on WikiText2. In the Appendix, Figure C provides more illustrations on Llama3-8B, Qwen2.5-7B and Qwen2.5-32B, showing similar observations.
This simple framework has been popularly adopted by many subsequent works (Lin al. 2024b
Ashkboos et al., 2024b Lin et al. 2024a, Dettmers et al. 2024 Duanmu et al. 2024). Instead, Om- niQuant (Shao et al. 2024) and FlatQuant (Sun et al., (025) employ the block wise quantization in which layers within each attention block are quantized simultaneously. To explore longer-distance dependencies than the block-wise quantization, QLLM (Liu et al. 2024b) and CBQ (Ding et al.) 2025) utilize the multi-block-wir the insights of prior works (Li et al. 2021; Zheng et al. 2022) tailored for quantizing convolutional neural networks in computer vision. These methods have significantly advanced the post-training quantization research for LLMs. However, in formulation, they typically treat different layers of a pre-trained LLM equally, no matter using the layer-wise or block-wise
diice-wise
quantization based
fixed-size sliding window, resembling
multi-block--wse quan- tization. Such an assumption appears reasonable under moderate quantization settings, e.g., 8-bit weight quantization, as the total quantization error tends to be small. However, we conjecture may be not optimal under challenging quantization settings, e.g., 4-bit weight-activation quantization.
For the sequential quantization framework described above, some previous works (Nagel et al., 2020 Li et al.) 2021 Frantar et al., [202]] Shao et al. 2024 Ding et al. 2025) have established a solid theoretical foundation based on second-order Taylor expansion to get better approxima- tion solution for post-training quantization. This paper moves one step further: we revisit the pre- dominant sequential quantization framework via questioning whether different layers of currently prevailing LLMs have similar quantization impacts on model accuracy. To explore this question, we select three representative layer-wise, block-wise and multi-block-wise quantization methods, Smoothhuant, OmniQuant and CBQ, and examine them on lot of popular LLMs in the challeng- ing 4-bit weight-activation quantization regime. Of particular interest, we have observed several properties. Firstly, for each of tested LLMs, intermediate layers usually have smaller quantiza- tion impacts on model accuracy compared to shallow/deep layers. This implies that shallow/deep layers are more sensitive quantization than intermediate layers which are relatively easy to quan- tize. Secondly, among shallow/deep layers, the first/last layer has the largest quantization impact on model accuracy, exhibiting significantly larger quantization error than others. This implies that the first and last layers are greatly important in the quantization process, as they are responsible for the very basic feature extraction and the final feature abstraction, respectively. Thirdly, as more layers are sequentially quantized, the quantization impact on model accuracy will be magnified gradu- ally. However, SmoothQuant, OmniQuant and CBQ show unsatisfactory abilities to suppress this issue, suffering from the underlying premise that all layers treated equally in their layer-wise, block-wise and salli-bbock-wiss quantization frameworks. Figure [ illustrates these properties on Llama2-7B, Llama2-13B (Touvron et al. 2023bb and Qwen2.5-14B Yang al. 2024).
These empirical properties highlight two ingredients that are essential to formulate an improved se- quential quantization framework adopting fixed bit-width: (1) special attention during quantization is required on shallow and deep layers, particularly the first and last layers; (2) the quantization syn- ergy of successive layers is required to reduce quantization errors across layers. Driven by this,
2

---

Published as a conference paper at ICLR 2026
present sequential quantization framework, Sliding-layer Quantization (SliderQuant) shown in Figure which relies on a simple adaptive sliding quantization concept. In principle, by adopting sliding window, the layers of pre-trained LLM are sequentially divided into overlapping windows with the same size first, and then the quantization is performed window by window facilitated by few learnable quantization parameters. The overlaps of consecutive windows establish basic quan- tization synergy path to reduce quantization errors across layers. However, simply using fixed-size sliding window (Duanmu al., 2024, Ding et al. 2025) still has a large gap endow SliderQuant with the desired two ingredients, as shallow, intermediate and deep layers will be quantized with the same window size and moving interval per step and treated equally important. We fill this gap by presenting two novel sliding quantization components. Our base component, inter-layer sliding quantization, incorporates three types of sliding window designs tailored for adaptively quantizing shallow, intermediate and deep layers with smart optimization relay across them. Specifically, it first allocates a progressively expanded sliding window along shallow layers, a fixed-size sliding window along intermediate layers and progressively contracted sliding window along deep layers, and then performs the sliding quantization progressively. With these three types of sliding window designs, our inter-layer sliding quantization component leverage the aforementioned empirical properties about the varying layer sensitivity to quantization. To exploit these empirical properties further, we present another complementary component, called intra-layer sliding quantization. extends the progressively expanded sliding design within each window of inter-layer sliding quan- tization component, by which all layers in each window are jointly quantized in an incremental manner. Coupling these two components in this way forms a neat implementation of SliderQuant.
Overall, SliderQuant is a flexible PTQ framework which can be used for both weight-only and weight-activation quantization. Our experimental results show that SliderQuant outperforms exist- ing PTQ methods across broad range of quantization settings (W4A16, W3A16, W2A16, W4A4), model families (Llama, Llama2, Llama3, Qwen2.5, Qwen3) and model sizes (7B, 8B, 13B, 14B, 32B, 65B, 70B) on 2 basic language generation and 6 commonsense reasoning benchmarks. Incor- porating advanced techniques (e.g., rotation transformations) into SliderQuant further improves its performance. In addition, we validate the effectiveness of our SliderQuant on the advanced MoE model Qwen3-30B-AS
Notably, we also apply SliderQuant to the recently popular DeepSeek- RI (uuo et al.] 2025) distilled models with strong thain-of-thhouhh reasoning abilities, achieving loss-lsssless accuracy under 4-bit weight-only quantization 5 challenging math and code tasks.
2 RELATED WORK
Many PTQ works focus on weight-only quantization. Early works, such as Q-BERT (Shen et al., 2020) and GOBO (Zadeh et al. 2020), use a mixed-precision decomposition scheme in which the weight outliers (i.e., a small fraction of weights causing large quantization errors) are retained in high- gighprrccssion format while the other weights are quantized into low-precision format. They con- sider small language models like BERT (Devlin et al. 2019). GPTQ (Frantar et al. 2023 formu- lates an efficient weight-only quantization approach using approximate second-order Hessian matri- ces. Compared to the popular round-to- -o-eressttt (RTN) quantization method (Dettmers et al.) 2022 GPTQ shows much better performance when quantizing weights to 3-bit/4-bit. QulP (Chee et al. incoherence-drive
scheme promote GPTQ, especially for the 2-bit quantiza- tion of weights, and an improved variant is further presented in TTeng et al. (2024). SpQR (Dettmers) al. 2024) and AWQ (Lin et al., 2024b) extend the peeecisinn decomposition scheme through designing more effective strategies to identify weight outliers, achieving superior performance to GPTQ. To avoid the inefficiency of the mixed-precision implementation on hardware systems (Cai et al. 2020), AWQ searches channel- wwie factors to scale down weight outliers, enabling full-weight quantization. Other works, including but not limited to [Shen et al. (2020). Tang et al. (2023). Park
2024 introduces
et al. (2024) and Kim et al. (2024), try to advance the weight-only PTQ research from other aspects.
Compared to weight-only quantization, weight-activation quantization can bring more significant reductions in both compute and storage costs at inference, but it is more challenging due to its higher sensitivity to quantization error. LLM.int8() (Dettmers et al. 2022). a pioneering work for weight-activation quantization, uses mixed-precision scheme that quantizes all weights and most of activations into INT8 format, but isolates activation outliers (i.e., a small fraction of activations that have large magnitudes than the others) into FP16 format. To suppress activation outliers, the authors of Wei et al. (202)) employ the non-scaling layer normalization and the token- wise clipping, making activations be more friendly for 8-bit quantization. Unlike LLMMint80 using the vanilla vector-
3

---

Published as a conference paper at ICLR 2026
wise quantization with RTN, ZeroQuant (aao et al., 2022) applies fine lineggrinned INT-8 quantization scheme consisting of group-wiss wise quantization for weights and token- wise quantization for activa- tions. Based linear equivalent transformation, SmoothQuant (Xiao et al. 2023) uses per-channel smoothing factors to scale down activation outliers and scale up the corresponding weights, mitigat- ing the quantization difficulty from activations to weights which are easier to quantize. SmoothQuant calculates channel-wise smoothing factors over a randomly sampled calibration set in an offline manner, and is tailored for 8-bit weight-activatie factors and weight-clipping thresholds, and consid-
quantization. Instead, OmniQuant (Shao et al.)
2024) dynamically learns activation-smoothine ers more diverse bit-width settings down to b-b QUIK (Ashkboos et al. (024a)) addresses 4-bit weight-activation quantization by extending the mixed-precision scheme. QLLM (Liu et al. (0024b)
formulates a channel disassembly and channel assembly scheme facilitated by the low-rank adapta- tion (Hu et al. 2022) to suppress outliers in some channels. However, this scheme modifies LLM architectures, and thus introduces extra inference-timg costs. QuaRot (Ashkboos et al., 2024b) uses random rotation transformations to remove outliers from the hidden state. Some subsequent works improve QuaRot by making rotation transformations learnable (Liu et al. 2025: [Sun et al. 2025) or combining rotation and permutation transformations (Lin et al.) (2024a). Similar to QLLM, these rotation-based methods also introduce extra computational costs at inference, as rotation transfor- mations added to some layers are not absorbable due non-linear operations.
3
METHOD
In this section, we describe the formulation of our SliderQuant and detail its implementation.
3.1 BASIC CONCEPT: FIXED-SIZE SLIDING QUANTIZATION
Given a pre-trained high-precision LLM having L layers, we start with the vanilla sliding quantiza- tion (Duanmu et al. 2024 Ding et al. 2025). It uses a fixed-size sliding window {s, i) moving along the layer direction of the given model and performs the sequential quantization in window-wise manner, where denotes the window size and i denotes the moving interval per step. The overlap between two consecutive windows is i. Let W [W1.- ... ..... the pre-trained weight matrix set for s layers in the current window and let X be its input feature corresponding to a small set of e calibration samples. Then, for weight-only quantization, the optimization objective is defined as
argmin |[W,, F(W, X)12
W
where F(:))) denotes the output feature of the current window, and W (W.,... ........ denotes the low- low-precision weight matrix set needs to be determined. For weight- quantization, the low-precision input feature X is obtained from the quantization of X beforehand, and then its optimization objective can be defined by simply replacing F(W, X) in 1 F(W, X).
According the above definition, when the window size is one layer or one attention block or multiple attention blocks and the moving interval per step is equal to the window size s (i.e., there is no overlap between two consecutive windows), we will get layer-wise, block-wise and multi-block- wise quantization frameworks popularly used in existing PTQ works (Yao et al. 2022 Frantar et al., 2023 Xiao et al., 2023 Shao et al. 2024 Liu et al. 2024b). That is, they are special cases of fixed- sized sliding quantization. For sliding PTQ methods including ours, a larger window size leads to increased memory cost but enjoys much better accuracy (see Table compared to layer-wise PTQ methods like GPTQ (the most efficient PTQ method tailored for weight-only quantization).
3.2 SLIDER QUANT
Recall that our empirical observations underlie two ingredients that are crucial to improve the se- quential quantization framework. Firstly, special attention during quantization is required on shallow and deep layers, particularly the first and last layers, as they are more sensitive to quantization than intermediate layers. Secondly, the quantization synergy of successive layers is required to reduce quantization errors across layers. For fixed-size sliding quantization (we use it as the baseline slid- ing quantization design in our ablations), the existence of an overlap i > between any two consecutive windows builds a basic synergy path to reduce quantization errors across layers. How- ever, when using fixed-size sliding window, all layers of a pre-trained LLM will be quantized with the same window size and moving interval per step. That is, shallow, intermediate and deep layers still treated equally great extent, leading to large gap have the desired quantization
4

---

Published as a conference paper at ICLR 2026
Inter-layer sliding quantization
Intra-layer sliding quantization
shellow
intemndiate
ary quartieing window
layer
aldfing windlww
Stage
quentined group
leal
quarritttio ooder
progressively expanded sliding
fixod-sine shiing
peogressivel contrccted sliding
Stage
quentinet group
Quantiaad
Quastizing
Unquartized
Figure 2: Overview of SliderQuant consisting of two components based on a simple adaptive slid- ing quantization concept. The base component of SliderQuant, inter-layer sliding quantization, has three sliding window designs along shallow, intermediate and deep layers of any pre-trained high- precision LLM, which are tailored for addressing their varying layer sensitivity to quantization. To establish a smooth sliding quantization relay from shallow to intermediate layers then from interme- diate to deep layers, we set one overlapped layer between shallow and intermediate layers and one overlapped layer between intermediate and deep layers. Besides, this also makes each intermediate layer have an even quantization frequency. The other component of SliderQuant, intra-layer sliding quantization, is applied within the current window of inter-layer sliding quantization component, by which all layers in the current window are jointly quantized in an incremental manner.
design. We present SliderQuant to fill this gap via designing a more adaptive sliding quantization framework, which consists of two novel sliding quantization components, as shown in Figure 2
Inter-Layer Sliding Quantization. Our base component, inter-layer sliding quantization, incorpo- rates three types of sliding window designs tailored for adaptively quantizing shallow, intermediate and deep layers with smart optimization relay across them. For I,, shallow layers, progressively expanded sliding window (PESW) is designed, which starts from quantizing the first layer with the window size of 1, then takes the first layer as the anchor layer and gradually increases the window size by I per step until including all shallow layers. With PESW, the first layer is always involved in the quantization process with every expanded sliding window, building dense local to global syner- gies to ease the quantization of shallow layers. Reversely, a progressively contracted sliding window (PCSW) is designed for La deep layers, which starts from quantizing all deep layers, and then grad- ually decreases the window size by 1 per step until to only include the last layer. With PCSW, the last layer is always used as the anchor layer, and it is involved in the quantization process with every contracted sliding window, building dense global to local synergies to ease the quantization of deep layers. For I; intermediate layers, we adopt a fixed-size sliding window {s = 2, 1) (FSSW) and an even optimization frequency across layers. Specifically, we set one overlapped layer be- tween shallow and intermediate layers, and also set one overlapped layer between intermediate and deep layers. Alternatively, when the window size of fixed-size sliding quantization at intermediate layers is larger than (i.e.,& 8 > 2), we can easily ensure an even optimization frequency by chang- ing the number of overlapped layers between intermediate layers and shallow/deep layers, e.g., overlapped layers when s = 3. Sequentially applying these three types of sliding window designs establishes the desired optimization relay across all layers of any pre-trained LLM,, flexibly leverag- ing our identified empirical properties about the varying layer sensitivity to quantization. According to the ablative experiments, we set L. 4, L as default for an accuracy-efficiene trade-off.
Intra-Layer Sliding Quantization. To exploit our identified empirical properties further, another complementary component, called intra-layer sliding quantization, is presented. This component is applied into the current sliding window of our base component. Specifically, it extends the pro- gressively expanded sliding design to each window of lnter-layer sliding quantization, where its s layers parallelly apply the progressively expanded sliding by a ratio along weight and activation dimensions. As a result, the joint quantization of all s layers is completed incrementally in N = 1/> sliding stages. We set = 0.5, N = 2 as default, as illustrated in the right part of Figure 2 In the first stage, the first half of weight/activation matrices in the current window of inter-layer sliding quantization is quantized jointly. In the second stage, the whole of weight/activation matrices (in- cluding the first half) in the current window of inter-layer sliding quantization is quantized jointly. As result, intra-layer sliding quantization builds a local to global parameter synergy across layers within the current sliding window of inter-layer sliding quantization to reduce quantization error. Coupling these two components in this way also leads to a neat formulation of SliderQuant.
5

---

Published as a conference paper at ICLR 2026
Learnable Parameters and Quantizer. SliderQuant builds a new sequential quantization frame- work by a flexible schedule of multiple sliding window designs, enabling dense synergies among shalloo//itermedditeeddegp layers and smart quantization relay across them. Next, we describe how SliderQuant quantizes weights and activations at each layer in a sliding window. It is well known that effectively removing outliers in weights and activations at each layer of pre-trained L.MM is important to reduce the total quantization error. Prior works (Xiao et al. 2023 Shao et al.. 2024 Liu et al., 2024b Ding et al. (025)) popularly use channel scaling (CS) (Meller et al.) 2019) and low-rank adaptation (LoRA) (Hu et al. 2022] Dettmers et al. 2023) to handle this issue. Inspired by them, we simply combine CS and LoRA in our method. Let W. Ra*m be the weight matrix of the 2thh layer in sliding window, and let X; E R be its input feature corresponding to a small set of e calibration samples (c = 128 as default). Then, the quantization process is defined as
X, = X; ag
W, W. A,B,, X,+1 quantizer( X.) -uantizer( WW), (2)
where OG E Rn denotes learnable channel-wise scaling vector to scale X; and reversely scale W., A; E [Raxr and B, E IRrxm ( [we sel r = 4 in experiments) are two low-rank matrices to get a re- fined weight matrix W; E RR for quantization, @ and @ denote element-wise division and multi- plication operations, respectively. With the refined weight matrices W and input X defined by Eq the corresponding quantized weight matrices W = quantizer(W] and input X = quantizer( (X) are obtained to minimize the mean square error defined in Eq. []] In implementation, we use uniform quantizer for both weights and activations, for simplicity and fair performance comparisons with existing post-training quantization methods. We put its definition the Appendix B./
EXPERIMENTS
Models. We first select widely used Llama (Tuvron et al. 2023a). Llama2 (Touvron et al. 2023b), Llama3 (Dubey et al. 2024) and Qwen2.5 (Yang et al. (024) families for experiments. To further explore the potential of SliderQuant, we evaluate it on more advanced LLMs, including a Mixture of Experts (MoE) model Qwen3-30B-A3B (Yang al. 2025) and the recently popular DeepSeek-
RI (Guo et al. 2025) distilled models with chain-of-thought reasoning abilities.
Evaluations. For most models (including the Llama series, Qwen2.5, and Qwen3), following the commonly adopted settings in post-training quantization research, we evaluate their two fundamen- tal capabilities: basic language generation and commonsense reasoning. For language generation, we report perplexity on WikiText2 (Merity et al. (2077) and C4 (Raffel et al.) 2020). For common- sense reasoning, we report zero-shot accuracy on widely adopted benchmarks: PIQA (Bisk al., 2020), ARC (Clark et al., 2018). HellaSwag (HS) (Zellers et al. 2019). Winogrande (WG) (Sak-) aguchi et al.] 2021). BoolQ CClak et al.] 2019), and MMLU (Hendrycks et al.] (020). For the DeepSeek-R1 distilled models with cfain-of-thought reasoning abilities, we evaluate their per- formance on challenging mathematical reasoning (MATH-500 (Hendrycks et al., 202T), AIME- 2024 (Jia 2024), and GSM8K (Cobbe et al., 2021) and code generation benchmarks (HumanEval+ and MBPP+ (Liu et al. 2003]). To ensure fair and reproducible results, we adopt standard evalua- tions: LM Evaluation Harness (Gao al. 2024 for commonsense reasoning. OpenCompass (Con-
tributors 2023) for mathematical reasoning, and EvalPlus (Liu et al.) 2023) for code generation.
Counterpart Methods. Recent top-performing PTQ methods usually introduce architectural mod- ifications, some of which are not absorable at inference. For fair comparison, we categorize ex- isting PTQ methods into two distinct groups based on whether they introduce extra inference-time costs. Notably, as a flexible quantization framework, SliderQuant maintains compatibility with both paradigms. For quantization without extra inference-time costs, our experiments cover both weight- only and weight-activation quantization: for weight-only quantization, we compare SliderQuant with round-to-nearest quantization (RTN), GPTQ (Frantar et al. 2023), AWQ (Lin et al. 2024b), and QulP (Chee et al. [024); for weight-activation quantization, we compare SliderQuant with RTN, SmoothQuant (Xiao et al. 2023), OmniQuant (Shao et al., 2024), AffineQuant (Ma et all 2024b), and CBQ (Ding et al. 2025). For quantization with extra inference-time costs, we im- plement a variant of SliderQuant named SliderQuant+ for a fair comparison, which additionally incorporates rotation transformations. We compare it with QLLM (Liu et al. (024b). Atom (Zhao et al. 2024). DuQuant (Lin et al. 2024a), QuaRot (Ashkboos et al. (2024b), SpinQuant (Liu et al. 2025 and FlatQuant (Sun et al. 2025). Implementation details are provided in the Appendix

---

Published as a conference paper at ICLR 2026
Table I: Results comparison of different quantization methods without extra inference- costs on the language generation tasks. The metric is perplexity. Best results shown in bold.
Llama2-7B Wiki 4. C4.
Lluma2-138
Llumu2-70B Wiki 4.
Llama3-8B
Qwen2,5-7B Wiki J. C4
Qwen2.5-14B Wilk 4 C4.)
ABits
Method
Wiki
C4.)
C4.4
Wiki 4.
C4.
W16A16
5,47
6.97
4.88
6.46
3.33
5.54
6.13
8.93
7.73
11.55
5.30
9.11
6.11
7.71
5.20
11.85
10.39
14.83
6.78
10.35
AWQ
6.15
7.68
5.12
6.74
3.60
5.70
8.09
11.23
8.54
12.78
6.43
9.89
GPTQ
5.83
7.37
5.13
6.70
3.58
8.01
11.34
8.64
12.98
6.45
10.01
W4A16
OmniQQunt
5,74
7.35
5.02
6.65
3.47
5.65
7.28
10.59
8.23
12.25
5,94
9.67
CBQ
5.67
7.23
5.02
6.67
3.46
5.64
6.93
10.27
7.92
11.77
5.83
9.54
StiderQuant
5.61
7.19
5.00
11.59
5.80
RTN
3.804
4.8c4
5.604
7.2c4
2.0c4
2.4c4
2,466
2.5c6
6.904
6.9c4
6.0c6
4.4c6
AWQ
2.2c5
1.7e5
1.2c5
9.4c4
9.1e1
5.1e1
5.6e5
3.1e5
1.5e2
2.7e2
1.4e3
2.7e3
GPTQ
7.7c3
NAN
2.1c3
3.2c2
77.95
48.82
7,8c5
9.7c5
1.262
3.1c2
1.2c3
1,3c3
W2A16
QulP
55.00
13.75
OmniQuant
37.37
90.64
17.21
26.76
7,81
12:28
2.8e5
3.9e5
56.45
89.13
67.84
89.56
CBQ
12.10
18.91
9.32
21.93
7.23
91.83
404.31
18.65
37.10
13.65
25.55
SliderQuant
9.59
13.83
7.71
11.21
6.53
9.39
27.59
56,98
17.15
31,08
12,68
21.91
RIN
5.3c2
5.4c2
5,8c2
5.3c2
8,904
2.3c2
2.0c2
3.6c5
3.7c5
4.0c3
3.0c3
SmnotthQuant
83.12
77.27
46.62
43.19
33.40
43.28
2.0c2
1.5e2
1.3e2
2.9c2
1.3e2
1.4e2
W444
AffineQuant
OmniQuant
12.69
15.76
12.30
11.45
14.55
13.97
11,54
2.1c3
3.5e3
1.4e2
93.73
2.9e2
34.70
61.75
CBQ
12.73
14.45
8.48
11.71
7.56
11.04
35.97
32.64
35.00
72.09
18.20
27.96
SliderQuant
8.34
11,10
7.62
10.26
6.87
9.67
15.47
21.74
13.81
21.52
11.00
16.60
Table 2: Results comparison of different quantization methods without extra inference-time costs on the zero-shot commonsense reasoning tasks. The metric accuracy (%%
Model
ABits
Method
PIQA
ARC-e 1
ARC-E 1
HS 1
WG
BoolQ
MMLUU 1
Avg 1
W16A16
80.41
77.40
49.15
79.37
72.14
80.55
52.77
70.26
W4A4
SmoothQuant
61.10
44.87
27.47
41.03
50.67
58.50
21.14
43.54
Llama2-138
W4A4
OumiQuaat
69.21
57.37
34.56
61.95
56.91
65.44
23.56
52.71
W444
CBQ
71.00
61.57
35.84
65.15
57.93
66.39
24.78
54.67
SliderQuant
71.65
71.22
27.04
W16A16
82.10
82.95
85.26
77.58
77.42
W1A4
SmoothQuant
54.57
35.14
24.66
35.29
51.46
56.45
24.90
40.35
Qwen2.5-14B
OmniQuant
59.45
48.
W444
CBQ
67.52
60.86
36. 18
60.12
58.09
60.1 15
31.61
$3.50
W4A4
SliderQuant
71.16
66.96
40,96
63.38
62,51
64.25
43.50
58.96
4.1
MAIN RESULTS
Quantization without Extra Inference-Tinee Time Costs. We first report the results for quantization methods without extra inference-time costs. shown in Table SliderQuant consistently achieves lower perplexity on WikiText2 and C4 than counterpart PTQ methods across broad range of quan- tization settings, model families and model sizes. Under the extremely low-bit configuration of W4A4, SliderQuant gets more prominent performance. These results demonstrate the robustness and effectiveness of SliderQuant in preserving generation quality even under aggressive quantiza- tion. In Table we provide the results comparison 6 commonsense QA benchmarks. We can observe that SliderQuant consistently surpasses counterpart PTQ methods. These results validate the effectiveness of SliderQuant in preserving various capabilities of LLMs under low-bit quantization.
Quantization with Extra Inference-Time Costs. The comparative results of quantization methods with extra inference-time costs are shown Table 3 We can see that SliderQuant+ achieves the best results on average across difffrrntt models and benchmarks. While SliderQuant demonstrates strong quantization capabilities, incorporating rotation transformations further enhances its effec- tiveness, enabling it to handle more precision-senen Table
scenarios. Combining the results in Table 2 and Table 3 we demonstrate the generalizability of our sliding- layer quantization frame- work, which consistently gets superior performance compared to the state-of-the-ar PTQ methods with or without extra inference-time costs. Experimental details are provided in the Appendix
Extension to Mixture of Experts Architectures. As shown in Table when applying SliderQuant to Doven3-30B-oe A3B, an advanced MoE architecture, we can observe a similar performance improve- ment trend on dense LLMs. The results further validate the generalizability of SliderQuant.
Challenging Math and Code Tasks with Reasoning Language Models. Most existing PTQ works limited to evaluating LLMs basic language generation and commonsense reasoning tasks. However, for real-world applications, it is also crucial to assess their complex reasoning abilities on more challenging tasks. In the experiments, apply SliderQuant the state-of-the-ar DeepSeek-

---

Published as a conference paper at ICLR 2026
Table Results comparison of different quantization methods with extra inference time costs. Slid- erQuant+ denotes SliderQuant using rotation transformations.
Model
Methods
WikiText2
ARC-
ARC-c
WG
W16A16
5.77
6.97
78.
74.62
46,42
75.90
69.05
W4A4
QLLM
11.75
13.26
67.68
44.40
30.89
58.45
56.59
$1.60
W4A4
Atom
6.96
9.12
69.75
47.35
34.22
63.21
56.51
54.21
Llama--88
W444
DuQuant
6.08
7.79
73.68
50.00
37.46
69.74
63.93
59.36
W4A4
QuaRor
6,10
8.69
76.77
69.87
40.87
72.16
63.77
64.69
SpinQuant
41.72
W4A4
QuanQuant
5.79
7.79
77.
72.05
43.26
73.64
69.53
67. 15
W4A4
SuaderQuant+
5.71
7.68
77.97
73.15
43.35
73.71
69.74
67.58
W16A16
4.88
5,46
W4A4
QLLM
9.09
11.13
70.46
48.48
34. 39
62.88
55.41
$4.31
W4A4
Atom
6.96
9.12
71.16
50.89
37.88
67.51
58.40
$7.17
Llama2-13B
W444
DuQuant
5.33
7,02
77.26
56.23
42.15
73.66
62.95
W4N4
QuaRoe
6,10
8.67
77 .9.69
69.95
42.83
73.54
67..85
66. 38
SpinQuant
W4A1
FlatQuant
5.12
7,09
79.38
76.64
18.04
77.59
70.24
70.38
W4A4
Slider SuanerQuant+
5.07
7.04
48.95
71.22
W16A16
3.32
5.71
82,70
$1.02
$7.17
83.81
76.54
W444
QuaRoc
3.79
6.12
81.83
79.76
$5.46
81.58
76.09
74.94
Lluma-2-70B
W4A4
SpinQuant
3,70
6.07
82.37
79.04
55.38
82.57
78.22
75.52
W444
FlatQuant
3.55
5.91
82.75
80,30
56.14
83.01
77.90
76.02
W4A4
Slider QuaenQuant+
3.50
5.87
82.75
81.23
$6.57
83.12
77.93
76.32
W16A16
6.13
8,93
80.79
77. 0969
53.41
79.13
72.77
72.76
30.12
W4A1
DuQuant
8.06
11.29
76.22
70.41
43.69
73.87
67.80
66.40
Llama3-8B
W4A4
Qualoot
8.16
13.38
43.34
65.
W4A4
SpinQuant
7.39
12.19
77.37
74.20
47.27
74.55
68.51
68.
W4A4
FlanQuant
6.98
11.13
79.16
75,80
50.00
76.80
72.69
70.89
W4A4
SuaderQuant++
6,87
11.04
79.22
77.53
50.60
77.31
72.82
71.50
Qwee2.5-7B-
W16A16
8.36
14.37
80.20
75.80
51.37
79.57
69.95
71.37
W4A4
FlanQuant
8.46
13.94
76.93
77.69
51.71
78.42
69.53
70.86
SliderQuant
52.27
Table Exploration of applying SliderQuant to the Mixture Experts (MoE) model Qwen3-30B-
A3B. We take OmniQuant the baseline using its public code for comparative evaluation.
#Bits
Methods
WikiText2
C4.4
PIQA
ARC- t
ARC-e
HS
WG
BoolQ
MMLU
W16A16
8.71
12.08
80.14
79.25
56.23
77.66
69.95
88.56
77.74
75.64
W4A16
OmniQuant
9.25
12.54
79.43
77.23
52.22
76.36
69.46
87 68.68
76.21
74.08
W4A16
SliderQuant
9.04
12,47
79.87
77.44
55.20
76.75
70.96
87.92
76.83
75.00
W3A16
OmniQuant
10.27
13.52
79:00
76.73
52.73
74.77
67.25
86.94
73.63
73.01
W3A16
SliderQuant
9.92
80.20
79.00
54.61
74.80
70.48
87.61
74.63
W2A16
OmniQuant
33.25
27.12
70:78
$4.97
34.04
$6.59
54,
69
32.67
53.31
W2A16
SliderQuant
23.84
21.60
72.36
67.51
40.53
60.79
62.04
77.61
47.19
61.15
W4A4
OmniQuant 52.43
41.98
67.41
55.64
33.28
52.34
$2.09
64.86
25.17
50.11
SliderQuast
15.49
75.52
63.14
82.26
Table Exploration of applying SliderQuant to the DeepSeek-RI distilled models with chain-of- thought reasoning abilities on the challenging mathematical reasoning and code generation tasks.
Mathematical Ressoning(passe11
Code Generation(guss@0
Model
#Bits
Method
AIME-2024 GSMSK
HumonEvall+ MBPP,
W16A16
95.00
73.33
91.50
73.17
61.11
78.82
DeepSeek-Ri R1 -isiiiill--enn-11ee
W2A16
OmniQuant
0.00
0.00
2.20
0.00
0.00
0.44
SliderQuant
$4.28
12.80
25.53
DeepSeek-Ri Bia-Seck--I-Distill---- --iisill-QQen-148
W4116
OmniQuant
90.29
70.12
W4A16
Quant
94.60
70.00
91.35
72.56
60.32
77.77
W16A16
94.60
76.67
93.02
81:71
69 84
83.17
DeepSenk-R ---eea------iiiei A---Distll--ee H-DistillQQen-322
W2A16
OmniQuant
13.40
0.00
26.83
0.00
0.00
8.05
W2A16
StiderQuant
58.60
16.67
73.69
12.80
21.16
36.59
DeepSeek-RR R1-Dissill- Qwen-32B
W4A16
OmniQuant
93.00
56.66
92.64
75.00
65.61
76.58
W4A16
QuiderQuant
94.40
76.67
92.94
80.49
69.05
82.71
RI distilled models, exploring its performance on challenging mathematical reasoning and code generation benchmarks. The results are shown in Table 5 Under the W4A16 setting, SliderQuant remains near-lossless relative to FP16 on both 14B and 32B models, while consistently outperform- ing OmniQuant. Under the more aggressive W2A16 setting, SliderQuant achieves substantially higher accuracy than OmniQuant across all benchmarks and model scales. These results indicate that SliderQuant preserves reasoning fidelity 4-bit setting and remains robust 2-bit setting.
8

---

Published as a conference paper at ICLR 2026
Table 6: Results comparison of SliderQuant with mixed-precision quantization methods on Llama2- 13B. In the table, (1) for LLM-MQ/QUIK, 0.5% in FPI6 or 5% in FP16 indicates the ratio of weight or weight-activation outliers at each layer stored in FPI6; (2) for 2.1-bit weight quantization by Slid- erQuant, we simply use 4-bit quantization to the first and last layers while quantizing all other layers to 2-bit, and SliderQuant+ denotes our best version of SliderQuant using rotation transformations.
#BitS
Method
WikiText2 4
PIQA
ARC-e
HS
WG
1
WIGA16
4.88
80.41
77.40
79.37
72.14
77.33
W4A16 (0.5% FP16)
LLM-MQ
8.03
79.49
58.50
76.31
69.30
70,90
W4A16
SliderQuant
5.00
80.41
76.73
78.30
72.14
76.90
W3.4A16 (0.5% FP16)
LLM-MQ
8.61
79.49
58.12
74.77
69.61
70.50
W3A16
SliderQuant
79.11
74.16
76.78
69.76
74.95
W2.2A16 (0.5% in FP16)
LLM-MQ
10.80
76.77
55.26
70.83
67.09
67.49
W2,1A16
SliderQuant
7.64
77.21
62.58
71.05
67.51
69.59
W2A16 (0.5% FP166
LLM-MQ
12.17
75.84
54.29
68.32
65.51
65.99
W2A16
SliderQuant
7.71
73.56
67.47
64.75
63.22
67.25
W2A16 (0.5% in FP16)
SliderQuant
7.64
76.13
67.51
69.25
66,12
69.75
W4.9A4 (5% FP16)
QUIK
79.22
74.92
78.36
71.90
76.10
W4A4
SliderQuant+
5.07
79.96
77.27
77.96
71.98
76.79
Table 7: Results comparison of SliderQuant with the mixed-precision quantization method SpQR on Llama- 7B and Llama- 13BB
Model
#Bits
Method
WikiText2
C4
PIQA
ARC-e
ARC-e
HS
WG
W16A16
5.68
7.08
79.43
73.15
45.05
76.16
70.24
68.81
Llama-7B
W3.4SA16
SpQR
7.28
78.13
65.87
38.05
55.27
W3A16
SliderQuant
5.82
7.13
77.42
69.70
40.36
71.74
67.96
65.44
W16A16
5.09
5.62
80.41
74.71
47.95
79.08
73.09
71.05
Lluma-133 13B
W3445166
SpQR
5.22
6.72
78.73
73.27
42,75
58.22
68.90
64.37
W3A16
SliderQuant
6.78
79.33
73.15
45.73
76.19
70.72
69.02
Comparison with Precision Methods. In Table 6 and Table 7 we compare SliderQuant with the state-of-the-art
quantization methods including LLM-MQ (Li et al. 2023) and SpQR (Dettmers et al. 2024) for weight-only quantization, and QUIK for weight-activation quantization. Across all settings, SliderQuant consistently achieves higher accuracy, even with lower bit widths. Notably, SliderQuant surpasses LLM-MQ and QUIK with higher bit widths preserving fixed portion of FP16 outliers, and also yields clear gains over SpQR. These results show that sliding-layer quantization mechanism is superior to existing mixed-precision quantization schemes.
4.2 ABLATION STUDIES
To have better understanding of our proposed SliderQuant, further conduct lot of ablative experiments under both weight-only quantization and weight-activation quantization with Llama2- 7B. In the ablations, use the fixed-size sliding quantization {s 2, 1} the baseline.
Overall Design. We first conduct experiments to study the two core components of SliderQuant, inter-layer sliding quantization (Inter-S) and intra-layer sliding quantization (Intra-S). The results are shown in Table g Recall that compared to the baseline fixed-size sliding quantization, Inter-S additionally introduces progressively expanded sliding window (PESW) for shallow layers and progressively contracted sliding window (PCSW) for deep layers. We can find that both PESW and PCSW significantly improve the performance, demonstrating the importance consider the varying layer sensitivity to quantization of any pre- pre-trained LLM. By extending the progressively expanded sliding design within each window, Intra-S further enhances the performance. Coupling Inter-S and Intra-S leads to the best results, confirming the effectiveness of SliderQuant's multi-level design.
Effeet of I. and Laa in Inter-Layer Sliding Quantization. After validating the effectiveness of Inter-S and Intra-S, we next study the effect of different settings for them independently. In Table we provide results of Inter-S with different L. and La, namely the number of shallow layers and the number of deep layers in our design. For simplicity, we always keep L, and La the same in the ex- periments. As L. and La gradually increase from 2 to 6, the model performance steadily improves. While it also leads to larger memory and computational costs during quantization accordingly. Con- sidering the trade-off between quantization performance and efficiency, we set L, = L = 4 as default, further scaling beyond this point tends to bring marginal performance improvements.

---

Published as a conference paper at ICLR 2026
Table 8: Ablation of inter-layer sliding quantization (Inter-S) and intra-layer sliding quantization (Intra-S).
Table 9: Ablation of inter-layer sliding
quantization with different La and La-
PESW
Inter
PCSW
Intra-8
WikiTeat2.
W444
C4
WikiText2
W2A16
C4
W444
W2A16
18.91
I
WikiText22
C4
Wikileet2 1
C34 L
10.34
13.46
10.71
16.10
10.30
13.78
10:67
16.76
3
9.66
12.87
10.83
16.94
9.84
13.31
10.92
17.31
9.13
10.53
15.15
9.13
11.78
10.53
15.15
8.98
11.72
10.50
14.96
8.94
11.67
10.43
14.75
Table 10: Ablation of intra-layer sliding quan-
Table 11: Ablation of fixed-size sliding quantiza- tion with larger s.
tization with different y. Note N 1/7.
Ratio #Stage
W444
W2A16
Method
L. Le
W2A16
7
N
WikiText2
C4
WikiTeex2
C4
WikiText2
C4.
WikiText2
C4
1.0
12.73
14.45
18.91
0.5
2
10.34
13.46
10.71
16.10
Baseline
11.18
13.94
11.48
17.23
10.56
13.52
0.25
11.32
14.10
10.83
16.45
Inter
9.13
11.78
10.53
15.15
Effect of y in Intra-Layer Sliding Quantization. In this set of ablative experiments, we study the performance of Intra-S with different settings of the sliding ratio 7. For given value of 7, all layers in each window of Inter-S are parallelly quantized along the weigh'/activation dimension incremen- tally, and the quantization process completed in N 1/7 stages. As shown Table I0 Intra-S with N > I always achieves better performance than the baseline (N 1). For SliderQuane we set Y 0.5, N 2 as default, considering the performance and the simplicity of implementation.
Effeet of in Fixed-Size Sliding Quantization. To mitigate the performance differences potentially caused by varying window sizes, increase the fixed window size for the baseline. As shown in Table 11 the performance of the baseline improves increases from 2 to 4. Nevertheless, substantial performance gap still remains compared Inter-S which uses the same window size of 4 only in shallow and deep layers. This highlights the limitation of fixed-size sliding quantization, where a uniform strategy overlooks the varying quantization sensitivity across layers. These results further validate the necessity of our adaptive sliding window designs.
Channel Scaling and LoRA. Table 12 shows the effect of Channel Scaling (CS) and LoRA under W4A4 quanti- zation. We observe that neither component alone achieves the best performance. Regardless of whether applied to the baseline or our SliderQuant, using both CS and LoRA together consistently yields better results than using ei- ther one individually. This demonstrates the complemen- tary nature of the two techniques under our proposed slid- ing quantzation framework, well suppressing outliers in weights and activations different layers.
Table 12: Effect of channel scaling (CS) and LoRA.
#Bits
Methods
|CS LoRA
WikiText2 C4
20.41
29.67
Baseline
12.73
14.45
W444
/
/
13.92
18.44
9.18
12:21
SliderQuant
12.95
17.27
8.34
11.10
We include further implementation details and more ex- perimental results the Appendix, covering the following aspects: (1) the analysis of SliderQuant's quantization efficiency: (2) the ablation studies on the number of calibration samples and the group- wise quantization; (3) extended evaluation across diverse model families and quantization settings; (4) design details of SliderQuant with rotation transformations; (5) other results and visualizations; (6) other experiments and discussions conducted for the rebuttal.
Additional Experiments and Analyses.
CONCLUSION
In this paper, propose SliderQuant, new post-training quantization framework that explicitly accounts for the varying layer sensitivity to quantization large language models. By coupling inter-layer and intra-layer sliding quantization components, SliderQuant reduces quantization rors across layers. Extensive experiments across various model families, quantization settings (c.g., W2A16, W4A4), and benchmarks demonstrate the effectiveness and generalizability of our method.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint
2023.
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptisk Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: A family of highly capable multimodal models. arXiv preprint (xXi:2332.1//80) 2023.
Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, and Dan Alistarh. Towards end-to-end 4-bit inference on generative large language mod- els. In EMNLP, 2024a.
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. In NeurIPS, 2024b.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense in natural language. In AAAI, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurlPS, 2020.
Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In CVPR, 2020.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. NeurlPS, 2024.
Christopher Clark, Kenton Lee. Ming-Wa Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: (003:0v58000055577 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reüichir Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
OpenCompass Contributors. Opencompass: A universal evaluationn platform for foundation models. https:///ithhii ctt/
http://ttthuubcccoooeeeeeeooee httppaaoooooassppppcccoppeeeeeee 2023.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.nn800: 8-bit matrix multi-
plication for transformers at scale. In NeurIPS, 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llm.. In NeurlPS, 2023.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized repre- sentation for near-lossless Ilm weight compression. In ICLR, 2024.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Xin Ding, Xiaoyu Liu, Zhijun Tu, Yun Zhang, Wei Li, Jie Hu, Hanting Chen, Yehui Tang, Zhiwei Xiong, Baoqun Yin, et al. Cbq: Cross-block quantization for large language models. In ICLR, 2025.
11

---

Published as a conference paper at ICLR 2026
Haojie Duanmu, Zhihang Yuan, Xiuhong Li, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. Skvq: Sliding-window key and value cache quantization for large language models. In COLM, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 1783, 2024.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. In ICLR, 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 2024.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. arXiv preprint 1tt.:20033038744 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.1677 2024.
Maxwell Jia. Aime 2024 dataset. AIME_ 2024 2024.
Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. In ICMI., 2024.
Shiyao Li, Xuefei Ning, Ke Hong, Tengxuan Liu, Luning Wang, Xiuhong Li, Kai Zhong, Guohao Dai, Huazhong Yang, and Yu Wang. Llm-mq: Mixed-precision quantization for efficient llm deployment. In NeurlPS Workshop, 2023.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. ICLR, 2021.
Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quan- tized llms. In NewrIPS, 2024a.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. In MLSys, 2024b.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseck-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024a.
12

---

Published as a conference paper at ICLR 2026
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In NeurlPS, 2023.
Jing Liu, Ruihao Gong. Xiuying Wei, Zhiwei Dong, Jianfei Cai, and Bohan Zhuang. Qlim: Accurate and efficient low-bitwidth quantization for large language models. In ICLR, 2024b.
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Lim quantiza- tion with learned rotations. In ICLR. 2025.
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang. Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The of 1-bit Ilms: All large language models are in 1.58 bits. arXiv preprint arXiv:2402. 17764, 2024a.
Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. Affinequant: Affine transformation quantization for large language models. In ICLR, 2024b.
Eldad Meller, Alexander Finkelstein, Uri Almog, and Mark Grobman. Same, same but different: Recovering neural network quantization error through weight factorization. In ICML, 2019.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR, 2017.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In ICML, 2020.
Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. Lut-gemm: Quantized matrix multi- plication based on luts for efficient inference in large-scale generative language models. In ICLR, 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 2021.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. In ICLR, 2024.
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In AAAI, 2020.
Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan, et al. Flatquant: Flatness matters for Ilm quantization. In ICML, 2025.
Hanlin Tang, Yifu Sun, Decheng Wu, Kai Liu, Jianchen Zhu, and Zhanhui Kang. Easyquant: An efficient data-free quantization algorithm for llms. In EMNLP, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava. Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint w:Xiv:2307.09288 2023b.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#:: Even better Iim quantization with hadamard incoherence and lattice codebooks. In ICMI., 2024.
13

---

Published as a conference paper at ICLR 2026
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, 2017.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Feng- wei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer lan- guage models. In NeurIPS, 2022.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post- post-training quantization for large language models. ICML, 2023.
Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang. Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models. In NewrIPS, 2024.
An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024.
Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint :xXi::5505.09388, 2025.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In NewrIPS, 2022.
Ali Hadi Zadeh, Isak Edo, Omar Mohamed Awad, and Andreas Moshovos. Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference. In MICRO, 2020.
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. In EMC2-NeurIPS, 2019.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In ACL, 2019.
Yilong Zhao, Chien Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate Ilm serving. In MLSys, 2024.
DanDan Zheng, Yuanliu Liu, Liang Li, et al. Leveraging inter-layer dependency for post-training quantization. In NeurlPS, 2022.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
Section
Datasets used in experiments.
Section
Implementation details of SliderQuant.
Section
Implementation details of SliderQuant with rotation transformations.
Section
Quantization efficiency of SliderQuant.
Section
More ablation studies.
Section
More results across diverse LLM families and quantization settings.
Section
Visualizations of the quantization impact of different layers to model accuracy.
Section
Visualizations of weights and activations in SliderQuant.
A DATASETS USED IN EXPERIMENTS
WikiText2 (Merity et al. 2017 is a popular language modeling benchmark consisting of over million tokens from verified Wikipedia articles.
C4 (Rafell al. 2020 Clean Crawled Corpus) is a large-scale dataset primarily used for language modeling tasks, comprising 156 billion clean tokens. It sourced from cleaned web pages, originally from Common Crawl.
PIQA Bisk et al. 2020) contains 16,000 training and 3,000 validation samples. It focuses on physical reasoning through multiple-choice questions, where models select the most appropriate solution from two options, with exactly one correct answer.
ARC CClar et al. 2018) is a dataset of 7,787 genuine grade-school level, multiple-choice science questions, assembled to encourage research in advanced
wansernn.n The dataset is parti- tioned into a Challenge Set and an Easy Set, where the former contains only questions answered incorrectly by both retrieval-based algorithm and word co-occurrence algorithm.
HellaSwag Zellers et al. 2019) contains 70,000 training and 10,000 validation samples. It focuses commonsense reasoning by predicting the most plausible sentence continuation, sourced from crowdsourced captions and activity descriptions.
Winograde (Sakaguchi al., 2021) is collection of 44,000 problems, which is formulated fill-n-a-blank task with binary options. The goal is to choose the right option for given sentence which requires commonsense reasoning.
BoolQ (Clark et al. 2019) is a dataset comprising 15,942 naturally occurring yes/no questions paired with Wikipedia passages. Each example consists of question, passage, and binary answer, aiming to evaluate reading comprehension and entailment-like reasoning.
MMLU (Hendrycks al. 2020) benchmark designed evaluate the multitask accuracy of lan- guage models across 57 diverse subjects, including elementary mathematics, U.S. history, computer science, law, and more. The dataset consists of multiple-choice questions and is intended assess models' world knowledge and mooblem-solving abilities in zero-shot and few-shot settings.
MATH-500 (Hendrycks et al. 2021) comprises 500 challenging competition-leve mathematics problems sampled from the MATH dataset. These problems span various topics such as algebra, geometry, number theory, and probability, and are designed to test model's ability to perform complex mathematical reasoning and generate step-by-step solutions.
ATME-22024 (Tia, 2024) includes problems from the 2024 American Invitational Mathematics
Examination (AIME), prestigious high school mathematics competition. The dataset serves as a benchmark for evaluating models' capabilities in solving advanced mathematical problems that require deep understanding and creative problem-solving skills.
GSM8K (Cobbe et al. 2021) is a dataset of 8,792 high-quality, linguistically diverse grade school math word problems created by human problem writers. The dataset is segmented into 7,473 train- ing problems and 1,319 test problems, each requiring multi-step reasoning and basic arithmetic operations to solve.
15

---

Published as a conference paper at ICLR 2026
HumanEval+ (Liu et al. 2023) is an extension of the HumanEval dataset, consisting of 164 original programming problems designed to assess the functional correctness of code generated by language models. Each problem includes function signature, docstring specifying the intended function- ality, and multiple test cases for evaluation.
MBPP+ (Liu al. 2023 is augmented version of the Mostly Basic Programming Problems (MBPP) dataset, comprising approximately 378 crowd-sourced Python programming tasks. Each task includes natural language description, a reference solution, and three test cases, aiming to evaluate models' abilities in basic programming and problem-solving.
B IMPLEMENTATION DETAILS OF SLIDERQUANT B.I QUANTIZATION DETAILS
By default, we adopt uniform quantizer for both weights and activations to maintain simplicity and efficiency, while ensuring fair comparison with other post-training quantization methods. Specifi- cally, apply per-channel quantization for weights and per-token quantization for activations, using the WikiText2 dataset for calibration. For particularly challenging tasks where quantization errors can have a larger impact, specifically the MoE models evaluated in Table 4 and the mathe- matical reasoning and code generation tasks Table 3| we employ group-wise quantization with a group size of 128 to preserve performance, instead using the C4 dataset for calibration.
Z quantizer ()) = clamp (1=
0, 0, 0%
Zanax
Zmin
Zunin
(A)
2
a
where clamp(x., Qmin Qmax) clips x to the interval 1Qmin Qmarl; [x] denotes rounding to the nearest integer; the bit-width; and Zmin, Zmax are the minimum and maximum values of Z, respectively.
B.2 CHANNEL WISE SCALING
Recall that, in our SliderQuant, we adopt popular used channel scaling and low-rank adaptation (LoRA) effectively remove outliers in weights and activations at each layer of pre-trained LLM. For the channel scaling, we simply follow the implementation
OmniQuant (Shao et al. 2024). For the Llama (Touvron et al.) 2023a) and Llama2 (Touvron et al.] (023b) families, we introduce learnable scaling factors for the Qproj. Kproj. Veroj. Oproi, U Uproj, and Downproj operators. For the Llama3 (Dubey et al.) 2024) and Qwen2.5 (Yang et al.] (024) families, due to the presence of Group Query Attention, we align the dimensions of the Query and Key by replicating the scaling fac- tor. All scaling factors are initialized to and absorbed into the adjacent weights after quantization, introducing no additional inference overhead.
B.3 LOW-RANK ADAPTATION
For every LLM tested in experiments, ap- ply LoRA to all the linear layers to reduce quantiza- tion loss through learnable weight adjustments. The rank of LoRA is set to 4 (i.e., r=4), which introduces significantly fewer learnable parameters compared full parameter fine-tuning. After quantization, the ad- ditional learnable parameters introduced by LoRA are absorbed into the model weights, resulting in no extra computational overhead during inference.
Table A: The detailed tings of SliderQuant.
set-
Configuration
Selting
Calibration set Number of calibration samples Tokens sample
WikiTear2 C4
128
2048
L,, La 8,3
4,
2, I
0.5
Rank LoRA (r) Quantization group size
4 channel-wise 128
B.4 HYPER-PARAMETE SETTINGS
Batch size
3 AdamW
For inter-layer sliding quantization, we set both the number of shallow layers L, and deep layers La to 4 default. For the remaining intermediate layers, we adopt a fixed-size sliding window with s and i == 1. For intra- sliding quantization, set the sliding ratio y to 0.5. All the ppramettrsmeters are provided in Table
Optimizer
Epochs W22116 others)
60 20
Learning rate of scaling factor
0.001
Learning of LoRA
0.0001
Learning rate schedule
lincar decay to 2e1o
16

---

Published as a conference paper at ICLR 2026
B.5 QUANTIZATION HARDWARE AND DEPLOYMENT ACCELERATION
Post-training quantization experiments for models up to 32B parameters are performed on a single NVIDIA RTX A6000 (48GB) GPU, while larger models such as the 65B and 70B variants require two A6000 GPUs. Since the A6000 serves as our primary experimental platform, its deployment results are reported in Table To provide a broader evaluation of hardware efficiency, we addition- ally benchmark the quantized models on other GPUs, including a consumer- consumer-grade NVIDIA RTX 4090 (24GB) and data-center NVIDIA A100 (40GB), with results summarized in Tables C and D
Importantly, our method does not rely customized quantization kernels. All deployment tests are conducted with the widely adopted 1 ama. epp framework using a batch size of 1, a prompt length of 512 tokens, and generation length of 128 tokens. While specialized implementations such as GPTQ (Frantar et al., 2023) or AWQ (Lin et al. 2024b) kernels may yield higher acceleration ratios, results demonstrate that the proposed method can be directly integrated into mainstream frame- works without additional engincering efforts, ensuring practical applicability and fair comparability across hardware platforms.
Table B: Inference efficiency of quantized models on one NVIDIA RTX A6000 using llama. cpp. W M_Memory: weight storage; R Memory: peak runtime memory; Tokens/s: generated tokens per second.
Llama2-7B R.Memory
Llama2-13B
Llama2-70B R.Memory
#Bits
W_Memory
Tokens/s
w W_Memory
R.Memory
Tokens/s
W W_Memory
Tokens/s
FP16
12.55 GB
12.67 GB
45.89
24.24GGB
2B
24.71
128.48 GB
0OM
W4A16
GB
391 GB
116.56
7.64
36.55
GB
15.42
W3A16
2.41 GB
3.00
125.66
GB
4.99 GB
74.15
G4
25.04
17.90
W2A16
LB
2.34GB
135.80
GB
3B
GB
G7.62
19.62
Table C: Inference efficiency of quantized models on one NVIDIA RTX 4090 using llama. cpp. W Memory: weight storage; R Memory: peak runtime memory; Tokens/s: generated tokens per second.
Llama2-7B
Llama2-13B R Memory
Llama2-70B
#Bits
Memory
Memory
Tokens/s
W Memory
Tokens/s
W Memory
R Memory
Tokens/s
FP16
12:55 GB
G3
62.22
24.24
00M
128.48 GB
OOM
W4A16
GB
4.26GB
156.09
GB
92.99
36.55 GB
W3A16
2.41 GB
3.18
188.06
4.62 GB
5.32
117.20
2B.76GB
0OM
W2A16
L.3GB
2.42GB
226.73
3.29
3.71 GB
143.51
GB
GB
38.67
Table D: Inference efficiency of quantized models on one NVIDIA A100-40GB using 1 1lama. cpp. W W_Memory: weight storage; R M.Memory: peak runtime memory; Tokens/s: generated tokens per second.
Llama2-7B Memory
Llama2-13B
Lluma2-70B
#Bits
W W.Meorry
Tokens/s
W Memory
R Memory
Tokens/s
W Memory
R M.Meory
Tokens's
FP16
12.55 GB
12.71 GB
81.79
24.24
24.41 GB
44.73
128.48 GB5
0OM
W4A16
3.98
138.14
GB
36
20.16
W
2.41 GB
3.12GB
133.79
GB
5.
79.64
24.76
GB
18.96
W2A16
L.3GB
GB
146.35
3.
GB
87.33
17.03
18.01 GB
21.19
C IMPLEMENTATION DETAILS OF SLIDERQUANT WITH ROTATION TRANSFORMATIONS
Recall that in the Experiments section of the main paper, we apply our quantization framework SliderQuant with and without extra inference-ting costs. In the default settings, all the additional parameters introduced by SliderQuant during quantization are merged into the original weights at inference. To further demonstrate the versatility of our approach, we also design a variant named SliderQuant+ that incorporates rotation transformations, corresponding to the results shown Table 3 of the main paper. Our implementation is consistent with the QuaRot (Ashkboos et al. 2024b) codebasd The detailed design is illustrated in Figure A The non-absorbable Hadamard trans- formations are added after query projection (Qproj), ) key projection (K proz) and before the output projection Oproj in the head self-attention module. In the feed-forward network (FFN), the transformations are added before the down-projection (Downproj).
https: thub. com/ httmm/pcc//Quu com/spc1/quaactttttcccccceoo /Quaket
17

---

Published as a conference paper at ICLR 2026
W
Onginal Model Operator
Additional Rotation Operator
Intermediate A ctivation
Wg
softmax
Q" Wap
QTWG.
Watwnn
@ Wauce
@ww,
Mullii- Head Self-Attention
FEN
Figure A: Structural illustrations on additional rotation transformations added in SliderQuant+.
D QUANTIZATION EFFICIENCY OF SLIDER QUANT
To evaluate the quantization efficiency of SliderQuant, we measure its perplexity performance under different quantization time costs. As shown in Table D. Slid- erQuant achieves remarkably low perplexity after only 10 epochs. We adopt 20 epochs the default setting to strike a balance between time overhead and perfor- mance. Figure B further illustrates the detailed relation- ship between quantization time and perplexity. demon- strating that our method converges rapidly. OmniQuant achieves 14.26| 18.02 on WikiTex(2/C4 C4 with the training time of 4.75 hours. Comparatively, with the training time less than hour, SliderQuant achieves significantly better performance on both benchmarks 12.5 on WikiTex12 C44. The results demonstrate the high quantization efficiency of SliderQuant. Even under extremely limited time budgets, the language models quantized with SliderQuant still maintains good performance. Extending the quantization duration can further improve performance.
Table E: The quantization efficiency of SliderQuant in terms of GPU hours. Ex- periments are conducted with Llama2- 7B under W4.4 quantization gle NVIDIA RTX A6000. The metric is perplexity.
sin-
AEpoch
Wikitext2
C4
Time Cost (h)
10
8.92
11.59
3.24
30
8.30
11.05
9.22
11.04
12.81
DimiQhont
Orreiquare
Siderruarr StderQuantikpear 20)
SliderQuont
StikrOuanttkkpech
Slode-Quanhhepeee 2)
P91. on Wieffet22
14.5 14:3
PPL 04
Flatee 04.24
12-
Tedoce 10 IY..
12-
(614.1100
0031
16.1 3,30
4 Quantization Time Cost (GPU Hours)
2
Quantization Time Cost Cost
(a) WikiTex12.
(b) C4.
Figure The perplexity of SliderQuant on Llama2-7B under W4A4 quantization with different quantization time costs. Experiments conducted single NVIDIA RTX A6000.
E MORE ABLATION STUDIES
The Number of Calibration Samples. To ensure fair comparison with prior methods, use the calibration set with 128 samples as default. Here, we study the effect of the number calibration samples to SliderQuant. The results are shown in Table Notably, erQuant achieves the perplexity of 9.39 11.69 on Wikitext2/C4, demonstrating strong performance with much smaller number of calibration samples. As the number of samples increases, perfor-
with only 32 samples, Slid-
18

---

Published as a conference paper at ICLR 2026
mance gradually improves, reaching the perplexity of 8.34 on Wikitext2 C4 with 128 samples. Further increasing the number of calibration samples leads to consistent performance improvement. These results highlight the efficiency and robustness of SliderQuant, which maintains competitive performance even under a small number of calibration samples.
SliderQuant with Group-Wist Quantization. To ensure inference efficiency, we channel-wise quantization for SliderQuant in the default settings. In T we provide the results of SliderQuant with group-wise quantization. Compared to channel-wise quantization, group-wise quantization leads to better performance but increased parameter storage and slower inference speed, due to the need to maintain more quantization parameters. We can find that the performance gradually improves as the group size decreases, demonstrating trade-off between performance and efficiency. When inference cost is not a limiting factor, applying finer- grained group-wise quantization within the SliderQuant framework can lead to even better performance, showcasing its flexibility and strong quantization capability.
Table F: Effect of the number of calibration sam- ples. We apply SliderQuant on Llama2-7B under W4A4 quantization.
Table G: Ablation of SliderQuant with group-wise quantization on Llama2-7B un- der W2A16 quantization.
#Samples
Wikitex12
C4 +
Time Cost (h)
Group Size
Wikitest2 1
C4
32
9.38
11.69
1,87h
32
8.20
10.90
64
8.87
11.45
3,68
64
8.78
11.45
8.34
11.10
6.14h
9.15
12.30
256
8.30
11.01
11.76h
256
9.23
13.11
10.93
23.16h
channel wise
9.59
13.83
F MORE RESULTS ACROSS DIVERSE LLM FAMILIES AND QUANTIZATION SETTINGS
About the Results for Counterpart Methods. To demonstrate the effectiveness of SliderQuant, we compare it with other counterpart post-training quantization methods (e.g., AWQ, GPTQ, Om- niQuant). Here, we clarify the details about our reported results for the methods. For perplexity on WikiText2 and C4, we adopt the evaluation results of Llama and Llama2 models as reported in the respective papers of the counterpart methods. For models and settings not covered in prior work-such as Llama3, Qwen2.5, and certain quantization configurations (e.g.. W2A16) eval- uate the results using public code and train the models to the best of our ability. For downstream tasks, we follow prioritized strategy: reported results in the original papers, followed by evaluation using official checkpoints, and finally, reproduction via open-source code when necessary. For a fair comparison, we implement the fixed-size sliding quantization method CBQ with the same sliding window setting as we used in SliderQuant for intermediate layers.
The Results on Llama Model Family. In the main paper, report results the Llama2, Llama3, Qwen2.5 model families. Here, we further provide results on the first version Llama family, including Llama-7B, Llama- 13B and Llama-65B, evaluated both language generation and zero- shot commonsense reasoning tasks. As shown in Tables and SliderQuant consistently achieves the best performance under various quantization settings, outperforming existing methods such as RTN, AWQ (Lin et al. 2024b), GPTQ (Frantar et al. 2023 SmoothQuant (Xiao et al., (2023), OmniQuant (Shao et al. 2024) and CBQ (Ding al. (025). Notably, in challenging low-bit con- figurations like W4A4 and W2A16, SliderQuant maintains strong generation quality and reasoning accuracy, demonstrating its robustness and precision-preserving preserving capability.
More Results with Weight-Activation Quantization. In the main paper, we provide the results of Allama2-13B and Qwen2.5-144 14B with aeight-activation quantization on zero-shot commonsense reasoning tasks. Here, we provide the additional results of other four models with different scales, including Llama2-7B, Llama2-70B, Qwen2.5-7B and Qwen2.5-322 As shown in Table]] our Slid- erQuant consistently outperforms existing methods such as SmoothQuant, OmniQuant and CBQ across all tested models and tasks under W4A4 quantization. Even for smaller models like Llama2- 7B and Qwen2.5-7B, which generally more sensitive quantization, SliderQuant achieves the best performance on average. For instance, on Llama2-7B, it reaches the average accuracy of 59.30% notably higher than CBQ (56.26%) and OmniQuant (54.95%). Similarly, for the larger-
19

---

Published as a conference paper at ICLR 2026
Table H: Results comparison of different quantization methods without extra inference-time costs on Llama model family for the language generation tasks. The metric is perplexity.
#Bits
Method
Llama-7B Wiki 4 C4
Llama-13B
Llama-65B
Wiki
C4
Wad 4
W16A16
3.68
7.08
5.09
6.61
3.53
5.62
RTN
6,43
7.93
5.55
6.98
3.87
5.85
AWQ
7.52
5.34
6.98
3.76
GPTQ
6.13
7.43
5.40
6.84
3.83
5.80
OmniQQuant
5.86
7.34
5.21
6.76
3.71
5.73
CBQ
7.33
5.22
6.77
StiderQuam
5.81
7.26
5.19
6.73
3.65
5.70
RIN
1.1e5
1.3c5
6.8c4
5.6c4
2.2c4
2.2c4
AWQ
2.6e5
1.9e5
2.8e5
2.3e5
75.43
56.34
GPTQ
2.1e3
690
5.5c3
2.5c3
55.91
40.58
OmniQuant
13.47
24.89
13.21
18.31
7.58
10:77
CBQ
13.45
7.96
SliderQuant
9.00
12.91
7.51
10.43
5.95
8,36
RTN
2.7e2
4.0c2
2.4c3
1.8c3
3.7e4
8.9e3
SmoothQuant
25.25
32.32
40.05
47.18
2.8c2
2.4c2
W4A4
OmniQuant
14.51
10.87
13.78
CBQ
10.39
13.41
9.69
12.55
7.23
9.45
SliderQuant
9.52
Table I: Results comparison of different quantization methods without extra inference-time costs Llama model family for the zero-shot commonsense reasoning tasks. The metric is accuracy (%).
Model
#Bits
Method
PIQA
ARC+1
ARC-et
HS 1
WG 1
BoolQ
Avg 1
W16A16
79.43
73.15
45.05
76.
70.24
75.17
69.87
W4A4
SmoothQuant
49.80
30.40
25.80
27.40
49.10
38.42
Llama-7B
W444
OmniQQuant
66.15
45,20
31.14
56.44
53.43
63.51
52.65
W444
CBQ
70.51
55.81
31.74
60.03
57.93
64.85
56.81
W444
SliderQuant
71.3
59.05
63.26
59.16
W16A16
74.71
79.08
77.92
W4A4
Smooth-Quat
61.04
39.18
30.80
$2.29
51.06
61.80
49.36
Lisma-133
W444
Quant
69.69
47.39
33.10
$8.96
55.80
62.84
$4.63
W444
CBQ
71.00
61.57
35.84
65.15
57.93
66.39
58.48
W4A4
SliderQuant
75.19
62.79
36,26
68.49
64.01
67.43
62.36
W16A16
82.37
79.76
55.38
$4.13
76.95
84,92
77.25
SmonthQuant
Llama 65B
W444
OmniQuant
74.54
65.61
40.61
69.30
59.35
70.24
63.28
W4A4
CBQ
76.01
68.45
42.56
73.45
62.89
71.23
65.76
W444
SliderQuant
76.77
70.92
75.65
64.48
72.39
67.51
scale Llama2-70B, our method reaches 65.24%, outperforming CBQ (62.70%) and OmniQuant (59.93%) with clear margins.
More Results with Weight-Only Quantization. In the main paper, we provide the results with weight-only quantization language generation tasks. In Table KJ we further provide the results of SliderQuant with weight-only quantization on zero-shot commonsense reasoning tasks, show- casing the performance of SliderQuant under W4A16 and W2A16 quantization settings across five representative models selected from the Llama and Qwen model families. We can see that SliderQuant achieves nearly lossless performance under W4A16, with accuracies closely matching the full-precision (W16A16) counterparts. For example, on Qwen2.5- 14BB, SliderQuant in W4A16 16 achieves the average accuracy of 76. 76.74% compared 77.40% for the FP16 counterpart. Under the more aggressive W2A16 setting, performance degradation becomes more noticeable but remains within an acceptable range.
As an additional complement to the results in the main paper, the comprehensive experiments across different model families (Llama, Llama2, Llama3, Qwen2.5). model scales (7B, 8B, 13B, 14B, 32B, 65B, 70B), quantization setting (W4A4, W2A16, (W4A4,W2A16, W4A16), and benchmarks(language genera- tion and zero-shot commonsense reasoning) further highlight the stable and superior performance of SliderQuant.
20

---

Published as a conference paper at ICLR 2026
Table J: Results comparison of different quantization methods without extra inference-time costs on the zero-shot commonsense reasoning tasks under weight-activatien quantization. The metric is accuracy (%).
Model
WBits
Method
PIQA
ARC-e
ARC-
HS
WG
BoolQ
W16A16
78. 84.84
69.46
78.01
W4A4
Sunoethuuant
60.88
39,77
27.13
41.32
51.54
51.07
45.29
Llama2-17 7B
W444
QuniiQuant
68.44
54.17
31.91
55.95
55.56
63.67
54.95
W4A4
CBQ
78.18
56.40
33.45
60.46
55.09
61.96
56.26
W4A4
SliderQuant
71.38
$9.64
32.94
62.33
61.72
67.77
89,30
W16A16
82.64
80.47
57.34
83.32
78, 14
84,10
77.67
SmoethQuant
61.37
46.21
31.23
50.91
57.28
Llama2-70B
W4A4
OmniQuant
71.71
59.55
37.20
66.63
58.17
66.30
59.93
W444
CBQ
73.24
62.45
37.30
69.84
61.23
72.13
62.70
W4A4
SliderQuant
75.79
64.14
37.71
73.02
65.75
75.02
65.24
W16A16
79.71
76.05
71.27
84.71
W4A4
Smocthhuant
50.44
25.97
26.02
25,92
$3.20
38.81
36.73
Qwen2.5-7B
W4A4
Qmni(Quant
53.32
33.84
24.40
33.75
52.80 $2.80
38.78
39.48
W4A4
CBQ
62.57
48.99
31,06
47.31
54.54
53.67
49.69
W4A4
SliderQuant
66.92
59.30
33.62
55.06
59.43
62.54
56.15
W16A16
82,26
78.03
55.63
84,07
75.45
87.43
77.15
SmoothQuant
61.59
Qwen2.5-32B
W4A4
OmniQuant
76.16
61.24
39.93
65.48
59.27
65.26
60.39
W4A4
CBQ
71.44
63.47
36.86
59.41
63.93
69.63
60.79
W444
SliderQuant
72.09
68.10
40.70
62.12
63.30
70.14
62.74
Table K: sero-shot commonsense reasoning results under weight-only quantization of SliderQuant. The evaluation metric is accuracy (%).
Model
#Bits
Method
PIQA
ARC-e
ARC-c
HS
WG
BaolQ
W16A16
46.42
Llama2-78
W4A16
SliderQwant SliderQuant
78.67
71.55
42.49
74.77
69.14
74.89
68.59
W2A16
70.78
31.06
57.15
66.12
57.17
W16A16
80.41
77.40
49.15
79.37
72.14
80.55
73.17
Llama2-133
W4A16
W2A16
StiderQuant
73.56
67.47
37.54
64.75
63.22
70.67
62.87
W16A16
79.71
76.05
49.57
78.13
71.27
84.71
73.24
Qwen2.5.77
W4A16
SliderQuant SliderQuant
79.22
75.47
49.20
76.97
71.27
83.15
72.55
W6A16
68.50
60.31
34,56
$3.02
59.91
65.05
56.89
W16A16
Qwen2.5-14B
W4A16 16
SliderQuant SliderQuant
81.23
79.12
58.43
81.65
75.03
84.95
76,74
W2A16
72.09
40.70
62.12
63.30
70.14
62.74
16
82.26
$1.03
58.45
84.13
77.61
87.49
78.50
Qwen2.5.32B
W4A16
W2A 16
SliderQuant
74.59
72.01
46.42
65.12
66.30
56.85
63.55
G MORE EXPERIMENTS AND DISCUSSIONS FOR THE REBUTTAL
In this section, we provide more experiments and discussions conducted for the rebuttal.
G. STUDY ON FOUR SLIDING WINDOW SCHEDULE KNOBS
Our SliderQuant has four schedule knobs including a progressively expanded sliding window (PESW) for L, shallow layers, fixed-size sliding window withe the size of for L; intermediate layers and a progressively contracted sliding window (PCSW) for Lş deep layers used in inter-layer sliding quantization (Inter-S), and an incremental quantization ratio for intra-layer sliding quan- tization (Intra-S) within each window of Inter-S. In the Subsection [ of the main paper, we used Llama2-7B: to study their choices from the perspectives of their individual roles and combined roles, and chose L. 4, s 2, La = 4, = 0.5 as the default setting our SliderQuant applied to all different LLMs tested in paper, for simple implementation. Indeed, this default setting is not optimal even for Llama2-7B, let alone other LLMs. To better compare their choices, we addition- ally conducted multiple sets of ablative experiments with larger and recently released Qwen2.5- 14B under W4A4 quantization. Detailed results are summarized in the below Table L Table|M] Table and Table O from which see that SliderQuant: (1) with default setting, four sched-
21

---

Published as a conference paper at ICLR 2026
ule knobs are complementary to each other (see Table D; (2) for each individual knob, its default setting is usually not the best for both Llama2-7B and Qwen2.5-14B (see Table M and Table N) (3) combining it with PESW and PCSW is significantly better than merely using fixed-size slid- ing quantization to all layers, both on LJama2-7B and Qwen2.5- 14B (see Table Oj. Although our SliderQuant with the default setting already achieves superior results to existing PTQ methods,
Table L: Ablation of inter-layer sliding quanti- zation (Inter-S) and intra-layer sliding quanti- zation (Intra-S) on Qwen2,5-14B under W4.44 quantization.
Table M: Ablation of inter-layer sliding quan- tization with different choices of I., and La on Qwen2.5-145 14B under W4A4 quantization.
PESW
Inter-S
PCSW
Intra-S
Wikitext2
C4
L,
La
Wikitex12
C4 |
17.41
25.20
2
2
14.95
20.30
16.01
23.71
3
3
13.83
19.07
15.42
20.71
4
4
12.99
18.40
14.08
21.48
5
5
12,45
18.
12.99
18.40
6
6
12.22
18.01
11.00
16.60
Table N: Ablation of intra-layer sliding quanti- zation with different choices of y Qwen2.5- 14B under W4A4 quantization.
Table O: Ablation of fixed-size sliding quanti- zation with different choices window size s on Qwen2.5-14B under W4A4 quantization.
Method
L,
Wikitext2 L
C4
Ratio 7
#Stuge N
Wikitext2
C4
17.41
25.20
1.0
17.41
25.20
Baseline
15.94
24.15
0.5
23.71
14.75
21.41
0.33
3
15.67
23.03
0.25
16.07
22.71
Inter-S
2
12.99
18.40
these ablations indicate that there is still room to get improved quantization performance by choosing better settings of these four schedule knobs for different LLMs.. Underlined entries indicate our default hyperparameter settings and their corresponding experimental results.
G.2 ADAPTIVE WIINNW QUANTIZATION REPEATED OPTIMIZATION
In our SliderQuant, the first and the last layers are used as the anchor layer when quantizing shallow and deep layers, which means that they are quantized more times than the other layers. Naturally, critical question is whether performance gain comes more from repeated optimization or our adap- tive sliding quantization. To explore this question, we conducted an ablation to compare the roles of the sliding-window quantization design and the repeated optimization (quantization of all layers in each window by multiple times) on Llama2-7B and Qwen2.5- 14B under W4A4 quantization. Detailed results are summarized in the below Table from which we see that, on both Llama2- 7B and Qwen2.5-144 14BB: (1) for fixed-size sliding quantization with the window size of = 2 or 4 applied to all model layers, performing multiple times of quantization of all layers in each window can improve the quantization performance, but the gain against its corresponding single time baseline is marginal (< 1 perplexity); (2) comparatively, retaining fixed-size sliding window 8 = 2 for intermediate layers, by introducing a progressively expanded sliding window (PESW) for L, 4 shallow layers, progressively contracted sliding window (PCSW) for Laa 4 deep layers, our SliderQuant with this default setting achieves significant perplexity reductions (1.54 to 3.60 for Llama2-7B and 1.23 4.42 for Qwen2.5-14p on Wikitext2, with similar reductions observed on C4) over all fixed-sized sliding quantization baselines and its repeated variants. These experimen- tal results indicate that our adaptive sliding quantization contributes significantly more the final improvement of quantization performance compared to merely applying fixed-size size sliding window to all layers of FP16 LLMs (even with repeated optimization), which well echoes the importance of our empirical observations and method's 's motivation.
G.3 TRAINING TIME COST AND MEMORY OVERHEAD
We also conducted ablation to compare the training time and the memory overhead of our Slid- erQuant (including our default version and a fast version), the fixed-size sliding quantization base- line (CBQ, which is closely related to our SliderQuant) and OmniQuant both Llama2-7B and
22

---

Published as a conference paper at ICLR 2026
Table P: Ablation of comparing SliderQuant with repeated fixed-size sliding quantization Llama2-7B under W4A4 quantization. The repeated baseline applies quantization to all layers in each window multiple times with window size s. Left: Llama2-7B. Right: Qwen2.5-14B.
L,8
Repetitions
Wikitex12
C4
L,&La
# Repetitions
Wikitext2
12.73
14.45
2
17.41
25.20
12.58
14.23
3
12.41
14.15
2
17.01
24.97
12.11
14.08
2
24.81
1
11.13
13.52
4
14.75
21.41
10.94
13.39
4
14.54
20.93
10.79
13.25
4
14.35
20.71
10.67
4
9.13
11.78
2
12.99
18.40
Qwen2.5-14D under W4A4 quantization. In the experiments, we always use the same number of calibration samples (128), the same batch size (1) and the same number of training epochs (20) for all methods unless otherwise stated. Note GPTQ is known as an efficient PTQ method but is tailored to weight-only quantization and is less related to method, so it is not compared here. Detailed results are summarized the helow Table from which we can see that, on both Llama2-7B and Qwen2.5-14B under W4A4 quantization: (I) in terms of training time, SliderQuant (with the de- fault setting of L. 4, 2, La 4, 0.5) is 1.08x to 1.55 slower than OmniQua and the fixed-sized sliding quantization (CBQ) with the window size
2, but faster than the fixed-sized sliding quantization with the window size of = 4 or 3, and our SliderQuant- Fast (the only change the number of training epochs is 10 instead of 20) is the fastest (1.21x x 1.78x faster than OmniQuant and the fixed-sized sliding quantization (CBQ) with the window size 2) and achieves slightly worse quantization results compared to SliderQuant; (2) terms of memory over- head, our SliderQuant (with the default setting of T,g 4, = 2, La 4, 7 0.5), SliderQuant- Fast and the fixed-sized sliding quantization (CBQ) with the window size of 8 = have the same memory usage, which is around 2x compared to the memory usage by the fixed-sized sliding quan- tization (CBQ) with the window size of SliderQuant (with the default setting of L, 4, 2, La 4,7 0.5) and SliderQuant- Fast always achieve significantly better perplexity than other counterpart methods, showing perplexity reductions ranging from 2.21 to 23.70 on Wikitext-2 and 1.93 to 41.15 on C4. Furthermore, Fig- crfformance-efffciiinn trade-offs across different training time configurations. These results demonstrate that SliderQuant can achieve even greater training efficiency, showcasing up to 6.42x speedup compared to Omni- Quant on Llama2-78 7B while maintaining better perplexity on both datasets. Summarily, these results show that the training cost of SliderQuant is decent, which guarantees the practicality of our method to quantize different LLMs, also thanks to its simplicity.
and OmniQuant; (3) in terms of model accuracy, both
ure B in this Appendix presents comprehensive view of SliderQuant's
Table Q: Comparison of the training time and the memory overhead of our SliderQuant, fixed- size sliding guantization baseline and OmniQuant on Llama2-7B and Qwen2.5-14P under W4A4 quantization. All experiments are conducted on single NVIDIA A6000-48G GPU, and we set the number of calibration samples to 128 popular choice in the quantization community). the batch size to 1, and the number of training epochs to 20 for all methods except our SliderQuant-Fae with 10 training epochs. Best results are bolded.
Model
Method
Wikitext2
C4
Training Time (GPU Hours)
Memory Overhead (GB)
OmniQuant
14.26
18.02
4.75
15.04
Fixed-sizee sliding (s=2, CBQ)
12.73
14.45
5.66
16.24
Fixed-sized sliding (==3, CBQ)
11.18
13.94
7.65
23,02
Llama2-7B
sliding (s=4, CBQ)
11.13
SiiderQuant-Fast
8.92
11.59
3,24
29.80
SliderQuant
8.34
11.10
6.14
29.80
OmniQuant
34.70
61.75
7.38
22.12
sized sliding (s=2, CBQ)
17.41
25.20
Qwen2.5.
Fixed sized sliding (s=3, CBQ)
15.94
24.15
13.56
30.23
Fixed-sized sliding (sm4, CBQ)
14.75
21.41
15.36
SllderQuant-Fast
12.19
17.53
6.08
43.71
SliderQuant
11.00
16.60
11.43
43.71
23

---

Published as a conference paper at ICLR 2026
G.4 AITA-EFFCCEENCY AND CALIBRATION ROBUSTNESS
In the below Table R we provide an ablation to study the performance of our SliderQuant under smaller number of calibration samples 32, 64 and a larger number of calibration samples 256, besides the default number of calibration samples 128, and we compare our results with the reported results in the original papers of Omni and DuQuant (a top rotation- based PTQ method), We can see that: (1) when using a smaller or larger number of calibration samples, our SliderQuant and
Table R: Comparison of calibration data efficiency under W4A4 quantization. Left: Llama-7B, com- paring SliderQuant and OmniQuant with different numbers of calibration samples. Right: Llama2- 7B, comparing SliderQuant+ (SliderQuant with rotation) and DuQuant with different numbers of calibration samples. Underlines denote the default configuration; bold indicates the best results.
#Samples
Method
Wikitext2
C4
#Samples
Mothod
Wikitex22
C4
32
OmniQuant
11.48
14.80
32
DuQuant
6,31
7.99
32
SliderQuant
8.71
11.19
32
SliderQuant+
5.83
7.85
64
OmniQuant
11.40
14.57
64
DuQuant
6.29
7,88
SliderQuant
SliderQuant+
5.81
128
OmniQuant
11.23
14.61
128
DuQuant SliderQuant+
6.28
7,90
128
SliderQuant
8.01
10.58
128
5.71
7.68
OmniQuant SliderQuant
256
DuQuant SliderQuant+
6.23
256
8.49
11:26
2566
5.64
7,53
SliderQuant+ (using rotation transformations) achieve significantly better results than OmniQuant and also better results than DuQuant (using learnable rotation transformations) consistently; (2) intriguingly, even with 32 calibration samples, our SliderQuant and SliderQuant+ show superior performance than OmniQuant and DuQuant with 256 calibration samples, respectively.
H VISUALIZATIONS OF THE QUANTIZATION IMPACT OF DIFFERENT LAYERS TO MODEL ACCURACY
Llame3.88
Qwen2 5.7B
Owen2 5.32B
Pempleiit Wikltee2
8.0
Porplexity on WikTTxx2
5.6
SeowthQuaat Quaniqunt
Qunniqunt
CuaniQani
LBQ
10
CBQ
5.4
7.0
FP16
FP16
FP16
Perplcsity
5.0
Layer
Llama38B
Owen1
Owen2 5-32B
300
Perpiexity on WikITeN12
Perpiesity Wokllest2
Perplisiit on WWkTToo12
1.5-
300
70
100
100
20
30-
20
20
Nomber of quantiieddd layers
Number of quaatizod layens
Nomber of quaaniized layers
Figure C: Iusstrations on the quantization impact of different layers to model accuracy: (1) quantiz- ing a single layer (the first row) and (2) quantizing the first / layers (the second row) of Llama3-8B, Qwen2.5-7B and Qwen2.5-32B. Here, we select three representative layer-wise, block-wise and multi-block-woor quantization methods, SmoothQuant, OmniQuant and CBQ, and examine them in 4-bit weight-activation (W4A4) quantization on WikiTex12.
To further validate our empirical observation that different layers in LLMs exhibit varying sensitivity to guantization, we provide additional visualizations on Llama3-8B, Qwen2.5-7B, and Qwen2.5- 32B, shown in Figure C Consistent with the observations discussed in the main paper (illustrated in Figure []. these results reveal several important trends. First, for all tested LLMs, intermediate layers tend be less sensitive quantization, incurring smaller accuracy degradation compared to
24

---

Published as a conference paper at ICLR 2026
shallow and deep layers. This confirms that shallow and deep layers more difficult to quantize and require special attention. Second, the first and last layers exhibit the highest quantization sensi- tivity, leading to the most significant increases in perplexity when they are quantized. This highlights their critical role in maintaining model fidelity. Third, as more layers are quantized sequentially from shallow to deep, the cumulative quantization error increases gradually, further demonstrating the compounding effect of poor quantization in sensitive layers. Among all methods, SliderQuant con- sistently achieves the lowest perplexity in both the single-layer and cumulative-layer quantization settings. This is because it explicitly focuses on reducing quantization errors in the more vulnerable shallow and deep layers, effectively controlling overall errors propagation. These additional visual- izations provide further empirical evidence for layer-wise differences in quantization sensitivity, and reinforce the design rationale of SliderQuant. They demonstrate the necessity of quantization-aave strategies that account for such sensitivity variation, especially when targeting low-bit quantization.
VISUALIZATIONS OF WEIGHTS AND ACTIVATIONS IN SLIDERQUANT
To better understand how SliderQuant improves the quantization process, we visualize the numeri- cal ranges both activations and weights before and after applying it. Specifically, we compute the range as the difference between the maximum and minimum values-per channel for weights and per token for activations. Large value ranges are known to complicate quantization, especially under low-bit settings, they increase the risk of information loss. Therefore, reducing these ranges can significantly ease quantization and improve accuracy. Taking Llama2-7B under the W4A4 quantiza- tion setting as example, we examine the value ranges after merging the learnable parameters from OmniQuant and SliderQuant into the original weights, without actually applying quantization. This allows us isolate the effect of these methods on the intrinsic distribution of the weights and acti- vations, offering a clearer view of the quantization difficulty induced by each approach. As shown in Figures F SliderQuant consistently reduces the channel-wise weight ranges across shallow, middle, and deep layers, clearly outperforming OmniQuant. Similarly, as shown in Figures [ to the token-wise activation ranges after applying SliderQuant are significantly smaller than those in the original model and OmniQuant across wide range of representative samples and layers. By simultaneously compressing the value ranges of both activations and weights across different model depths and inputs, SliderQuant reduces quantization difficulty and enables effective low-bit quanti- zation with minimal performance degradation. This dual-range suppression is a key factor behind SliderQuant's robustness and near-lossless performance in challenging low-bit regimes.
Ser
Qpenj
K pros
pnaj
Oyruj
U Uprus
Gate
Figure D: Visualization of channel- weight ranges (max min) in the 1st layer of Llama2-7B under W4A4 quantization.
25

---

Published as a conference paper at ICLR 2026
Blcerjonnt
K proj
proj
proo
Uprra
Gatéprot
Figure E: Visualization of channel-wise weight ranges (max-min) in the 18th layer of Llama2-7B under W4A4 quantization.
Trides
pros
K pros
Vproj
Opros
U Uprea
Gatépros
Figure F: Visualization of channel-wise weight ranges (max-min) in the 31st layer of Llama2-7B under W4A4 quantization.
prud
K proj
prro
Upros
Gatepros
Figure G: Visualization of token-wise activation ranges (max-min) in the 1st layer of Llama2-7B for the first sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.
26

---

Published as a conference paper at ICLR 2026
K proj
Uerpol
Opro
U Uppros
Gatepros
Figure H: Visualization of token-wise activation ranges (max-min) in the 1st layer of Llama2-7B for the second sample of samples randomly selected from Wikitext2 under W4A4 quantization.
Oridict
Driped
Q prra
prox
proj
Uigeoul
DeerQven
Opros
Uprea
Gotéproy
Figure Visualization of token-wise activation ranges (max-min) in the 1st layer of Llama2-7B for the third sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.
pray
K proj
Sec
U Ppras
Gatepros
Figure J: Visualization of token-wise activation ranges (max-min) in the 1st layer of Llama2-7B for the fourth sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.
27

---

Published as a conference paper at ICLR 2026
K proj
Uprus
Gategruj
Figure K: Visualization of token-wise activation ranges (max-min) in the 18th layer of Llama2-7B for the first sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.
Hky
Dogputt
K pros
Vpros
Clipal
U Ppros
Gatepros
Figure L: Visualization of token-wise activation ranges (max-min) in the 18th layer of Llama2-7B for the second sample of samples randomly selected from Wikitex12 under W4A4 quantization.
prra
K proj
Ypra
Inquutt
Ppros
Gateprot
Figure M: Visualization of token-wise activation ranges (max-min) in the 18th layer of Llama2-7B for the third sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.
28

---

Published as a conference paper at ICLR 2026
K proj
prod
Ppros
Gatepros
Figure N: Visualization of token-wise activation ranges (max-min) in the 18th layer of Llama2-7B for the fourth sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.
Qpros
K pray
Ondeaa Miterhan
Oeldeet
Opros
U Ppras
Gatepros
Figure O:
Visualization of token-wise activation ranges (max-min) in the 31st layer of Llama2-7B for the first sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.
Qyevj
proy
Doulteos
Ppros
Gatepros
Figure P: Visualization of token-wise activation ranges (max-min) in the 31st layer of Llama2-7B for the second sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.
29

---

Published as a conference paper at ICLR 2026
Ordiat
Orided
Qprns
K proy
Vpraj
Popputt
Opros
Ppros
Gatepras
Figure Visualization of token-wise activation ranges (max-min) the 31st layer Llama2-7B for the third sample of 4 samples randomly selected from Wikitext2 under W4A4 quantization.
Qpraa
K proy
Dosdbce
Ogrez
Uprru
Gate
Figure R: Visualization of tosen-wise activation ranges (max-min) in the 31st layer of Llama2-7B for the fourth sample of samples randomly selected from Wikitext2 under W4A4 quantization.
30


Reviews:
[{"review_id": "EbLHWylMoK", "reviewer": "Reviewer_ZBQd", "summary": "The paper improves post-training quantization by making the sliding-window reconstruction approach depth-aware. Instead of a fixed window, they expand the window in shallow layers, keep it fixed in the middle, and contract it near the end so that the sensitive early/late layers are quantized with enough context. This simple schedule combined with incremental quantization within each window, outperforms common PTQ baselines (SmoothQuant/OmniQuant/CBQ) on several models(e.g. LLaMA/Qwen/MoE) especially at 4-bit and 2-bit. The main value is practical and better results without changing inference while at the same time the limitations are extra calibration cost and only incremental novelty over existing depth/adaptive PTQ ideas.", "questions": "- Results against at least one rotation-based PTQ(e.g. SpinQuant[1]) to contextualize the gains would be very useful.\n- Is there a way to automatically estimate the window schedule from measures e.g. per-layer sensitivity.\n- What is the calibration-time required and memory overhead compared to the simplest fixed-window baseline?\n- Do the improvements still hold with a smaller calibration set or if the calibration set is slightly domain-mismatched?\n\nI'd be happy to increase my score if these questions are answered!\n\n[1] Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Krishnamoorthi, R., Chandra, V., Yuandong, T., & Blankevoort, T. (2025). SpinQuant: LLM quantization with learned rotations. arXiv:2405.16406.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Depth-aware sliding window actually makes early and late layers easier to quantize, instead of treating all layer depths the same. \n- Inter-layer and intra-layer sliding reinforce each other, so you get denser cross-layer synergy as compared to a fixed window. \n- On MoE (Table 4) it improves over OmniQuant at every bit setting, which helps generalizable, not tuned for one model claim. \n- Generation Table 5 is especially strong, 2-bit OmniQuant nearly collapses on DeepSeek-R1 distilled models, while 2-bit SliderQuant keeps usable pass@1 on math and code. That’s very useful in order to have a useful model in the end. \n- The method stays compatible with standard PTQ methods(scaling, re-quantizing overlaps), so it is easy to slot into existing pipelines.", "weaknesses": "- The method adds several schedule knobs (expand depth, contract depth, window size, γ), and robustness to non-ideal choices is not fully presented in the paper. \n- Comparisons are mostly against fixed-window, non-rotated post training quantization techniques. It’s unclear how much of the gains remain vs the strongest rotation/equivalent methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RUvC5E3R72", "reviewer": "Reviewer_R2bM", "summary": "The paper proposes SliderQuant, a quantization framework. It introduces an adaptive sliding-layer strategy that assigns progressively expanded, fixed, and contracted windows to shallow, middle, and deep layers, respectively, addressing the unequal quantization sensitivity across layers—higher at the top and bottom. The method also includes intra-layer sliding quantization to further improve performance. Experiments show that SliderQuant achieves lower perplexity and higher reasoning accuracy than existing PTQ baselines, especially under challenging low-bit configurations.", "questions": "1. Missing baseline results in Table 1: Some baselines in Table 1 have missing or abnormal values. Could the authors clarify the cause?\n\n2. Source of improvement: sliding vs. repeated optimization: The proposed method combines a sliding-window design and more times of optimization of the first and last layers. Which factor contributes more to the final improvement? An additional ablation that isolates “adaptive sliding” from “frequency of re-quantization” would help clarify the main source of gain.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear and strong motivation: The paper is motivated by an empirically grounded observation on layer-wise sensitivity to quantization in LLMs. The motivation is clearly presented and addresses an overlooked aspect in post-training quantization.\n\n2. Comprehensive experiments: The evaluation covers multiple model families and various bit-width settings, demonstrating the generality of the proposed framework.\n\n3. Intuitive and well-written method: The proposed sliding-layer quantization framework is easy to follow and clearly described, with both figures and ablation studies supporting the core design choices. The overall paper is well written and organized.", "weaknesses": "1. Uneven optimization frequency of middle layers: According to Figure 1 and the default hyperparameter setting, the 4th and 5th layers appear to be quantized only once. This means that some middle layers receive fewer optimization passes than their neighbors. Could this uneven optimization frequency introduce instability or suboptimal performance? In particular, when the middle-layer window size is larger than two, how do you ensure that all middle layers are optimized an equal number of times?\n\n2. Training cost comparison with baselines is unclear: Although Table 7 usefully extends the baseline window size from (s=2) to (s=4), the training (quantization) cost of SliderQuant versus the baselines remains unclear. Please report quantitative efficiency metrics such as GPU hours. Ideally, a main table could compare performance under equal training budgets to ensure fairness.\n\n3. Hyperparameter generalization is under-discussed: The paper introduces several hyperparameters, but the discussion of their sensitivity is limited to a single model (LLaMA-2-7B). It remains unclear how these settings generalize across models of different sizes and depths. Please analyze or at least discuss how to scale or tune these parameters for larger models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KrEBh5wWfm", "reviewer": "Reviewer_DdkN", "summary": "This paper discovers the distinct quantization sensitivity among model layers, and proposes SliderQuant, a novel learnable quantization framework which contains two level sliding quantization concept. Experiments indicate that this method becomes SOTA sliding-based PTQ method.", "questions": "1. More detailed description about intra-layer sliding quantization is needed (W1).\n2. More training details should be provided (W3).\n3. Although the appendix mentions that the training can be completed on a single A6000 GPU, providing explicit comparative data (e.g., against OmniQuant, CBQ, and the more efficient GPTQ) would make the analysis more intuitive. If the training cost is excessively high, the practicality of SliderQuant may be questioned.(W4).\n\nI promise to increase my score if all my concerns are addressed.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The authors identify varying sensitivities of different layers to quantization and improve the quantization performance of layers with different sensitivities through a sliding-window design, rather than directly adopting a mixed-precision approach. This provides a novel and interesting perspective.\n2. The writing is clear and well-structured, the experiments are thorough, and the figures and tables are elegantly designed.", "weaknesses": "1. The description of intra-layer sliding quantization is the main weakness of the paper. As one of the core innovations, its explanation is too brief, which makes it confusing. Does it mean that the weights/activation matrices are also partitioned and quantized sequentially within each layer?\n2. I'm afraid that whether the effectiveness of both learnable low-rank matrices A and B will be influenced after quantization because they have been integrated into weights before quantization during inference.\n3. The authors ignores to describe the training details, such as loss function, supervision information. Is it followed by OmniQuant?\n4. The authors ignores to provide the memory usage and runtime of SliderQuant. As described, SliderQuant requires loading multiple blocks into the GPU simultaneously, which can lead to substantial memory overhead. Based on prior experience, training a 7B LLM with OmniQuant takes approximately 0.8 hours and 15 GB of memory on an A100 GPU, while SliderQuant is likely to demand even more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h6Z5kdlV3N", "reviewer": "Reviewer_BEoY", "summary": "This paper introduces a new PTQ framework that accounts for varying layer sensitivities in large language models. The authors observe that the first and last layers are far more quantization-sensitive than intermediate ones and propose SliderQuant, which adaptively applies inter-layer and intra-layer sliding quantization on weights or weights+activations to mitigate these effects. This design tailors window sizes and quantization granularity across layers, effectively reducing accumulated quantization errors. Experiments on several models and benchmarks show its superior performance at low bitwidths.", "questions": "- Could the authors provide more theoretical or intuitive explanations for the observations that the first / last layers are much more sensitive to quantization, beyond the empirical observations? I believe these could provide insights to future studies and add more novelty to this paper.\n- While the bitwidth is set to be the same for all layers in SliderQuant, mixed-precision quantization is also a popular way for LLM quantization that could utilize the characteristics that different layers have different sensitivity to quantization  (e.g. [[2]](https://nicsefc.ee.tsinghua.edu.cn/%2Fnics_file%2Fpdf%2F5c805adc-b555-499f-9882-5ca35ce674b5.pdf)). Could the authors provide some comparison with such kind of methods, to show whether the sliding-layer mechanism could complement or outperform mixed-precision strategies?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is clearly written and well-organized, with intuitive figures and tables that make the methodology and results easy to follow.\n- The experimental evaluation is comprehensive, covering a wide range of model families including Llama, Qwen, and MoE architectures. The authors conduct extensive tests not only on standard language generation and reasoning benchmarks but also on more challenging tasks like mathematical reasoning, providing rich empirical evidence for the method’s effectiveness. The ablation studies are also sufficient for understanding different factors that could affect the performance.", "weaknesses": "- The empirical observations and methodological novelty are limited. Specifically:\n  - The finding that different layers of LLMs exhibit varying sensitivity to quantization—particularly that the first and last layers are most sensitive—has already been clearly identified in several prior works (e.g. [[1]](https://arxiv.org/abs/2412.03599?utm_source=chatgpt.com), [[2]](https://nicsefc.ee.tsinghua.edu.cn/%2Fnics_file%2Fpdf%2F5c805adc-b555-499f-9882-5ca35ce674b5.pdf)), so this observation might not be suitable to be regarded as a novel contribution.\n  - The proposed method, while effective and technically sound, is largely incremental. Its design combines previously explored ideas, such as sliding-window quantization (e.g. [[3]](https://arxiv.org/abs/2405.06219), [[4]](https://arxiv.org/abs/2312.07950)) and layer-wise sensitivity-aware quantization (e.g. [[1]](https://arxiv.org/abs/2412.03599?utm_source=chatgpt.com), [[2]](https://nicsefc.ee.tsinghua.edu.cn/%2Fnics_file%2Fpdf%2F5c805adc-b555-499f-9882-5ca35ce674b5.pdf))), and the concept of quantization synergy across successive layers somewhat resembles extensions of the GPTQ framework's concepts.\n- Some experimental results have limited practical significance. For instance, in Table 2, all methods (including SliderQuant) show more than 10% accuracy degradation under W4A4 quantization on almost all the benchmarks, and similar issues appear in Table 5 under W2A16. Although SliderQuant achieves numerically better results than baselines, such large performance drops render these configurations impractical, raising doubts about the meaningfulness of these ultra-low-bit experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)