NoWag: A Unified Framework for Shape Preserving Com- pression of Large Language Models
Authors: Lawrence Ray Liu
Inesh Chakrabarti
Yixiao Li
Mengdi Wang
Tuo Zhao
Lin Yang
Venue: COLM 2025 (2025)
Keywords: Quantization
Vector Quantization LLMs
Compression
Sparsity
Pruning
TL;DR: NoWag is a unified framework for zero-shot shape preserving compression of LLMs, achieving state-of-the-art quantization and competitive pruning on Llama models.

Abstract: Large language models (LLMs) exhibit remarkable performance across various natural language processing tasks but suffer from immense computational and memory demands, limiting their deployment in resource-constrained environments. To address this challenge, we propose NoWag (Normalized Weight and Activation Guided Compression), a unified framework for one-shot shape preserving compression algorithms. We apply NoWag to compress Llama-2 (7B, 13B, 70B) and Llama-3 (8B, 70B) models using two popular shape-preserving techniques: vector quantization (NoWag-VQ) and unstructured/semi-structured pruning (NoWag-P). Our results show that NoWag-VQ significantly outperforms state-of-the-art one-shot vector quantization methods, while NoWag-P performs competitively against leading pruning techniques. These findings highlight underlying commonalities between these compression paradigms and suggest promising directions for future research. Our code is available at https://github.com/LawrenceRLiu/NoWag

Published as a conference paper at COLM 2025
NoWag: A Unified Framework for Shape Preserving Com- pression of Large Language Models
Lawrence Liu'
Inesh Chakrabarti Yixiao Li?
Mengdi Wang²
Tuo Zhao²
Lin F. Yang!
University of California, Los Angeles Georgia Institute of Technology Princeton University (lawrencerliu inesh33] @ucla edu, yixiaoli @egatech........ edu mengdiweprincee inceton...oo....oo edu, tourzhao@gatech. edu, linyangêee. ucla.edu
Abstract
Large language models (LLMs) exhibit remarkable performance across various natural language processing tasks but suffer from immense com- putational and memory demands, limiting their deployment in resource- constrained environments. To address this challenge, we propose NoWag (Normalized Weight and Activation Guided Compression), a unified framework for one-shot shape preserving compression algorithms. We apply NoWag to compress Llama-2 (7B, 13B, 70B) and Llama-3 (8B, 70B) models using two popular shape-preserving techniques: vector quanti- zation NNWWa--VO) and unstructured //ssmmm-sstuuu struuturdd pruning (NoWag- P). Our results show that NoWag-Va VQ significantly outperforms state-of- the-art one-shot vector quantization methods, while NoWag-P performs competitively against leading pruning techniques. These findings high- light underlying commonalities between these compression paradigms and suggest promising directions for future research. Our code avail- able at https://githhb. www.:tt.....wwwweeeeeee.eeeeeeee cow/LawwenceRL iunillauu///aa
1 Introduction
Large language models (LLMs) (Brown et al., 2020) have demonstrated remarkable capa- bilities across a wide range fields and tasks (Huang & Yang. 2025; Wei al., 2022; Park et al., 2023), but their immense computational and memory requirements during inference pose significant challenges for deployment. Consequently, post-training compression tech- niques have emerged promising tool to reduce model size and computational overhead while maintaining accuracy. Two promising families of methods for post-training compres- sion are Pruning (LeCun et al., 1989; Hassibi et al., 1993; Han et al., 2015) and Quantization (Yao et al., 2022; Dettmers et al., 2022; Ahmadian et al., 2023; Li et al., 2025).
Pruning aims to remove redundant parameters from LLMss while preserving performance. We will focus on two forms of pruning, unstructured pruning (Liao et al., 2023), which removes/ zeros out individual elements without any structure, and N:M semi-structured pruning (Huang et al., 2025), where strictly N of every M elements are zeroed out. SparseGITT (Frantar & Alistarh, 2023) introduced structured pruning method that leverages Hessian-based weight updates to minimize per- formance loss. More recently, Wanda (Sun et al., 2023) demonstrated simple yet effec- tive unstructured and semi-structured pruning method that requires no weight updates or hessian computation, making it significantly faster and easier to apply than SparseGPT. However, current hardware only supports 2:4 semi-structured sparsity, which leads to sig- nificant performance degradation after compression.
efficient, unstructured and semi-
A more effective compression method is quantization, which reduces the number of bits used to store each weight (Kuzmin et al., 2023). In this paper, we focus on weight-only Post-Training Quantization (PTQ), common form of quantization. Pioneering works (Frantar et al., 2022; Lin et al., 2024; Kim et al., 2023) focused on scalar quantization. For extreme compression (e.g., 4 bits per weight), Vector Quantization (VQ), where groups

---

Published as a conference paper at COLM 2025
Individual Weight Matricies
Normalized Weights
Compressed Model
Original LLM
VQ LLM
K-means Vector
Quantization
Greedy Pruning
Pruned LLM
Activation Guidance diag (Xx*)
Figure 1: Illustration of proposed NoWag (Normalized Weight and Activation Guided Compression). Given LLM, compression of each weight matrix W is performed inde- pendently. Vectors y(1) and y(2) are used normalize W. With the second moment of the activations diag (Xx)) guiding the importance of each weight for the compression algo- rithm, such as K-means VQ (NoWag-VQ) and Pruning (NoWag-P)
of d consecutive weights are quantized together, has demonstrated superior performance because the codebook can be shaped to the distribution weights (Egiazarian et al., 2024; Van Baalen et al., 2024; Tseng et al., 2024a; Liu et al., 2024). However, most current algo- rithms share at least one of the following two drawbacks. First, an expensive weight- update process necessitating matrix inversion, similar to SparseGPT. Second, sampling sufficiently accurate Hessian for quantization can require up to 25 million tokens and TB of CPU memory for model like Llama-3 70B, introducing new computational bottle- neck.
In this work, we address these issues by formulating a unifying framework for shape- preserving compression algorithms, where the compressed weight matrix has the same shape as the original uncompressed counterpart but can be stored with less memory. This method is weight update free, less dependent on calibration data, and features a novel nor- malization technique that benefits both pruning and quantization. We term this family of compression methods NoWag (Normalized Weight and Activation Guided Compression). We show that the VQ variation of NoWag, NoWag-VQ (NoWag for Vector Quantization), outperforms the SOTA one-shot VQs Qull# (Tseng et al., 2024a), at bits per value, while us- ing 48x less calibration data. Furthermore, we show that the pruning variation of NoWag, NoWag-P (NoWag for Pruning), outperforms Wanda, SOTA pruning algorithm at retain- ing language modeling performance.
Problem Formulation
Given a trained LLM, our goal is to obtain a compressed model that significantly reduces the computational and memory requirements while retaining much general perfor- mance as possible. Due to the large number of parameters, using global optimization for compression is computationally infeasible. As result, common approach is to indepen- dently perform data aware compression of each linear layer (Nagel et al., 2020), as well as to finetune the compressed LLM, (Egiazarian al., 2024; Malinovskii et al., 2024; Tseng et al., 2024a) to recover performance.
More formally, the objective for shape preserving compression algorithms is: For linear layer in an LLM with weight matrix W E do Xiln, find compressed weight matrix W e IR xdn to replace W that requires less memory while minimizing the deviation from the original model's behavior.
2

---

Published as a conference paper at COLM 2025
In standard notation, given input activations E R4m, the output is computed as y Wx, where y E R To incorporate data awareness, sample sequences of length I from a calibration dataset and collect the corresponding activation samples X R=xdd where m x
3 Related Works
Many existing works for shape preserving compression algorithms aim to minimize the expected error between the outputs of the original layer and the compressed layer (Nagel et al., 2020; Frantar & Alistarh, 2023; Frantar et al., 2022):
((W) = I., [I(W- w)xl|]] tr( tr(((w (wW- W) Ex(x") [ w)"
(1)
Because the underlying distribution of is unknown, the expected value of the hessian Ex [xx]] of the activations is estimated through the sample activations A 1,xx" E R Xmn, allowing for the following approximation, l' of E:
e' (()) = (WW f (W (w-w)")
(2)
The challenge of minimizing such an objective that it cannot be broken down into inde- pendent element wise subproblems. As a result, solving for the optimal pruned or quan- tized W is NP-hard. Even greedy approximations to these solution are themselves compu- tationally expensive; for example, a common approach is a group wise greedy approach with linear feedback updating (Frantar et al., 2022; Tseng et al., 2024a; Liu et al., 2024; Van Baalen et al., 2024; Chee et al., 2023; Frantar & Alistarh, 2023). In such an approach, groups of rows are iteratively quantized pruned in greedy fashion, and the remaining non compressed rows are adjusted to compensate. Calculating the optimal compensation requires calculating the inverse of fi, which carries a computational overhead of (df,).
However, recent works appear to show that simplified approximations of equation 2 are sufficient, and even superior. For example, Wanda (Sun et al., 2023), only considers the diagonal elements of the sample hessian. This results based on score metric Sij = |Wyl Xiil2 without any feedback. Furthermore, for unstruc- tured pruning, pruning is performed independently in per-output groups. In other words, l
pruning algorithm that prunes
instead of constraining each weight matrix, each individual row is constrained to x% spar- sity. Although this constraint is more restrictive, Wanda reports that it results in superior performance competitive with SparseGPT. Likewise, for quantization, Squeeze (Kim et al., 2023), replaces the Hessian with the diagonal of the Fischer information matrix, al- lowing for K-means to be used to cluster the weights without any computationally expen- sive weight updates. SqueezeLLM is able to outperform some linear feedback based scalar quantization algorithms such as GPTQ Frantar et al. (2022).
4 NoWag: (Normalized Weight and Activation Guided Compression)
Compression Objective To ensure numerical stability and enhance compression effi- ciency, we first normalize W as per Algorithm 1 obtain W E Rdo xd using normal- ization vectors ((1) E di and (2) E Roout
2
Wi
1
W
(1)
dout Σ
W
VjE [dn),
pp==
din
W
Vi E [dout].
(2)
The compressed weight matrix W is obtained by minimizing the following weighted Frobe- nius norm:
7(W)= || w w 2
(W,, -W,
(3)
3

---

Published as a conference paper at COLM 2025
Algorithm 1 NoWag Normalization
Algorithm 3 NoWag-VQ
Require:
Weight matrix E R
Require: Weight matrix E Roonn activation samples
Ensure: Normalized weight matrix W. normalization vectors y(l y(
X E Rusde, subvector dimension d, target bits per value mumber of iterations faur
Compute column norms: p v /LiT W for (1,
Ensure:
Quantized weight matrix Rlourdi codebook
C, Asssignments Nollag Nornalization(W) (See Algo-
Nornalize columns: Wi- Wi Vi.i
(f0), W)
rithm 1}
Compute row nurms: !,
Li Wt,, for [1,dal
(W (Ppideed padded)
VOPadding {See Algo-
rithm 4]
Normalize rows: Wiy
7- Vi,
[X.lB. [1, (Diagonal of sample hessian) if dm padied then
return W, (0, y!?)
Pad H with zeros to length di, padded
end
Reshape Woodted into subvectors: Wee E RRdd where N fout sfnn padded
Algorithm NoWag Inference
Reshape H into subvector weights: Huu RNNd
Require:
Input activation x é Réin, compressed weight ma- trix W é Rdouttnn, normalization vectors ,() RAf,,
Mortrods 2"his
(C,A) +- Weighted Wae.. Hab, Movetrode Noeer [See Algorithm 6}
y(8 IRoout
Ensure:
Output yctivation Riout
Wabb Map each subvector to its assigned centroid using
A
Normalize input: 82 + xy/r/" Xj E [1,
Reshape Wax back to matrix form and remove padding get W Reout {Reverses padding from Algorithm 4} return W,C. C,
Compute intermediate output: 4- Wox, Vi E (l,
Denormalize output: y1 +- 31 1 2 E [o,d return
Here, x E R" represents the calibration activations for the jth input channel, and || acts as a weighting term that prioritizes important elements of W. key benefit of such an objective is that it can be decomposed into independent subproblems at the element level, subject to global constraints imposed by the chosen compression algorithm. This allows for compression algorithms that do not require any feedback methods.
Denormalization can be done on the fly by multiplying the inputs by y(2) and the out- puts by p(1) as noted in Algorithm 2. The additional computational cost of these opera- tions scales linearly with the input and output dimensions respectively, They are therefore largely neglible compared with the overall computational cost of Matrix multiplication. Furthermore, even though these normalization vectors vectors stored in fp16, the ad- ditional overhead is minimal: in terms of bits per value, they account for < 0.01 bits per value.
Paradigms of Compression The above formulation unifies the following two paradigms of shape preserving compression.
1. Quantization (NoWag-VQ): With the discretization constraints imposed by quan- tization, optimizing equation 3 can be approximated by weighted K-means, where the weights are determined by diag( XXT). For initialization, modified version of K-means++ was used, with the possible centroids sampled from a random subsam- ple of the subvectors. The computational complexity of d dimensional K-means is (dNTK) where N is the number of data points, K is the number of clusters, and T is the number of iterations. Therefore the computational complexity of d di- mensional NoWag-VO for quantization to Abppy bits per value is (dindout ). Note that in our experiments T << min dout). The pseudo-code for this algo- rithm is detailed in Algorithm and a detailed formulation of VQ can be found in Appendix A.
2. Unstructurad/Smm-Sttuuuue pruning pattern, where x% of the weight matrix entries (i, are zeroed out, our method selects the x% of entries with the smallest W1l1X thereby minimizing Equation 3. For N:M Semi-Structured Pruning, our method selects the N entries in each group of M with the smallest W W 1113- Through the quickselect algorithm
Pruning (NoWag-P): For an a%-unssrrctured
4

---

Published as a conference paper at COLM 2025
Figure 2: A sample weight from the first attention layer of Llama-2-7B, From left to right: visualization of the absolute values of the weights, normalized weights, importance scores, and normalized importance scores all down-sampled to 1:4 scale by max pooling. Individ- ual elements are visualized in log scale, with blue implying larger value.
(Hoare, 1962), we can find the threshold for x% of the entries linear time on aver- age (Mahmoud et al., 1995). Thus, NoWag-P has computational complexity of (dindout) on average. The pseudo code for NoWag-P is detailed in Algorithm 5 in Appendix F.
4.1 Why this works
The critical step in our approach is the normalization of W. Our normalization method effectively normalizes Wij by both the input and output group. This removes the biases on the compression algorithm focus on magnituem rows/columns, leading better retention of the overall performance of an LLM.
To illustrate this, we visualize W and its normalized counterpart, W for an example weight matrix in figure 2. In addition, to understand the effects of data awareness, also vi- sualized the element wise importance scores Sij Wij Xil2 derived from equation 3. Likewise, for comparison, we also visualized the naive scores Sij Wij 2;l22 without considering normalization.
We observe that non-normalized weights exhibit a structured pattern, with specific outlier rows and columns, with larger magnitudes. These structures be attributed to several phenomenons, such as sensitive attention heads, rotatory embedding patterns, and out- lier features (Dettmers et al., 2024; Su et al., 2024; Dettmers et al., 2022; Vig, 2019; Olsson et al., 2022). In comparison, the normalized weights do not exhibit this patterns. This is highly beneficial for vector quantization, it projects the d dimensional distribution of consecutive weights into bounded [0, "ball shaped" distribution, visualized in Figure 3
The importance visualizations in Figure 2 once again exhibits these row and column wise structures. Thus, when pruning is applied, the removed elements will be concentrated away from these rows and columns on the non-outlier columns. In turn, this effectively removes entire input/output channels, reducing the performance the compressed LLM. Normalization largely removes the rowise outlier structures from the importance scores. In addition some of the columnwise structure removed, while some still remains. The remaining structure is due to the ||2 component of the scores 5; = Wij 1
4.2 Comparisons with Other Compression Algorithms
The average computational cost of both NoWag-P and NoWag-VQ scales linearly with the size of the weight matrices, and thus the size of the LLM. Since din and dout are of roughly the same magnitude in a modern LLM, NoWag-P and NoWag-VO offers signif- icant speedup for compression over linear feedback based pruning methods, whose com- putational complexity scales cubically with din-

---

Published as a conference paper at COLM 2025
Figure 3: 2d PCA visualization of the distribution of d 6 grouped entries from W and W. Densities are plotted log scale. Normalization reshapes the distribution into a more "ball-shaped distribution.
Pruning: Several parallels can be drawn between our approach and Wanda. First, with- out normalization, NoWag-P is equivalent to Wanda without output grouped pruning. Second, Wanda takes advantage of the same phenomenon described in Section 4.1 through output group pruning, only normalizing along dout. However, output grouped pruning used by Wanda is not as strong as our normalization method in two aspects. First, nor- malization is only performed on the output dimensions; this breaks up the row wise struc- ture, but some column-wise structure will still remain. Second, group-wise normalization cannot be used with semi-structured pruning malization method directly breaks up both row and column structures. Furthermore, our normalization method is applied directly to the, allowing for support for semi-structured pruning and other compression algorithms such as quantization.
quantization. In comparison, our nor-
Quantization Kmeans has been explored for LLM PTQ in several works. In many VQ algorithms, it is used to initialize before optimizing the quantization (Van Baalen et al., 2024; Liu et al., 2024; Egiazarian et al., 2024). For scalar quantization, SqueezeLLM (Kim et al., 2023) has employed weighted K-means using the diagonal of the fisher information as weights. Our algorithm has several key differences from those aforementioned. First, K-means only, without any computationally expensive optimization procedures required by previous VQ algorithms. Second, our weights are simply the second moment of the sample activations which be calculated without backwards pass.
Experiments
Models. We evaluate NoWag on two popular families of models Llama 2 (Llama-2 7B/13B/70B) (Touvron et al., 2023) and Llama-3 8B/70B (Grattafiori et al., 2024). Due to resource and time constraints, we did not apply NoWag-VQ to the Llama-3 70B model; therefore, for the Llama family, we report quantization performance only for the Llama-3 8B model.
Baselines particular focus the research community "one-shot" compression meth- ods for large language models (LLMs). Here, "one-shot" refers to directly compressing the model based on the calibration data without fine-tuning to adjust the compressed model parameters. Such compression methods are desirable because they have minimal computa- tional overhead associated with it. However in more recent works, for "extreme" compres- sion, roughly bits per value, it has been demonstrated that specialized finetuning post quantization can significantly recover performance lost during compression (Malinovskii et al., 2024; Egiazarian et al., 2024; Tseng et al., 2024a).

---

Published as a conference paper at COLM 2025
The focus of our paper is how to compress. As result, primary focus on "one-shot" compression algorithms. The SOTA for one-shot VQ for LLMs at bits per value is QulP# (Tseng et 2024a), we make comparisons against it. QulP# incorporates VQ with Ham- mard incoherence matrices and an Eg structured codebook. We did not compare against QTIP (Tseng et al., 2024b) our focus was VQ rounding methods, and because Trellis coding can be extended to any VQ rounding method (Iseng et al., 2024b).
For pruning, compare NoWag-P against Wanda (Sun et al., 2023), an unstructured and semi-structured SOTA pruning algorithm. As discussed previously, the key difference be- tween Wanda and NoWag-P is our normalization method. Wanda does no normalization, and only does output group wise pruning for unstructured pruning only, while for semi- structured pruning, Wanda uses just simple greedy pruning based on the element-wiss importance scores. As such, comparison between NoWag-P and Wanda serves to high- light the impact of our normalization scheme in both a unstructured and semi-structured pruning scheme.
Calibration dataset We use 128 samples at the model's native sequence length (4096 for the Llama 2 family and 8192 for the Llama 3 family) of the RedPajama 1T dataset (Weber et al., 2024) as our calibration data for both pruning and quantization. This is the same dataset used by QulPH, which uses 6144 samples, or 48x times more data.
Evaluation To evaluate NoWag-VO, we follow standard evaluation metrics for quantized models of measuring the perplexity the test splits of the C4 (Dodge al., 2021) and Wikitext2 (Merity et al., 2016). Furthermore, we performed task specific sequence classifi- cation zero shot accuracy through the Eleuther AÍ LM Harness (Gao et al., 2024). For equal comparison with Qull#, we used the same version of LM Harness (Version 0.3.0). The exact tasks are listed in appendix B. It worth noting that the zeroshot tasks have some intrinsic randomness, even FP16 numbers can disagree by up to 0.5%, due to their element of randomness.
5.1
Quantization Evaluation
Method
Bits
Wiki2 (L)
C4 (1)
fp16 (2-7B)
Method
Bits
Wiki2 (4)
C4 (L)
fp16 (2-13B)
16
4.57
6.05
fp16 (2-7B)
16
5.12
fp16 (2-70B)
16
3.12
4.97
fp16 (2-13B)
16
4.57
6.05
fp16 (3-8B)
5.54
7.01
fp16 (2-70B)
16
3.12
4.97
QUIP (2-7B)
8.23
10.8
AQLM (2-7B)
2.02
6.59
8.54
NoWag VQ (2-7B)
2.02
7.07
9.12
NoWag-Vel VQ (2-7B)
2.02
6.51
8.50
QulP (2-13B)
8.07
AQLM (2-13B)
1.97
5.60
7.49
NoWag VQ (2-13B)
2.01
5.93
7.94
NoWag-VQ (2-13B)
2.01
5.53
QulP # (2-70B)
4.16
6.01
AQLM (2-70B)
2.07
3.94
5.72
No (2-70B)
2.02
NoWag-VQ (2-70B)
2.02
3.99
5.77
QulPW (3-8B)
13.8
15.6
NoWag-VQ (3-8B)
2.02
10.68
11.92
Table 2: Perplexities for WikiText2 and
C4 with blockwise finetuning for 2-bit Quantized Llama-2 7B/ TB/13B/700 com- pared with AQLM.
Perplexity on WikiText2 and C4 for 2-bit quantized Llama 2 (7B/13B/70B) and Llama (8B) mod- els without fine-tuning. were performed at the models' native sequence lengths (4096 for Llama and 8192 for Llama 3).
Table 1:
Evaluations
For Llama-2 7B/ 7B/13B and Llama-3 8B, VQ was performed with groups of d 6 elements together. This means that the codebook will be able fit inside the L1 cache of an Nvidia A6000 GPU, enabling fast decoding. For Llama-2 70B, VQ was performed with groups of = 7 elements together, since the relative overhead of the codebook was diminished
QuiP# accuracies taken from CALDERA (Saha et al., 2024).
7

---

Published as a conference paper at COLM 2025
Bits
Wino (1)
RTE (t)
PIQA (1)
ArcE (1)
AreC (1)
Avg Ace (1)
FP16 (2-7B)
67.3
63.2
78.5
69.3
40.0
63.66
FP16 (2-13B)
16
69.5
61.7
78.8
73.2
45.6
65.76
FP16 (2-70B)
16
77.0
67.9
81.1
77.7
51.1
70.96
FP16 (3-8B)
73.5
68.6
79.7
80.1
50.2
70.42
QulP# (2-7B) )
61.7
57.8
69.6
61.2
29.9
56.04
NoWag- (2-7B)
2.02
64.4
54.5
73.6
60.7
31.7
56.99
QulP # (2-13B)
2
63.6
54.5
74.2
68.7
36.2
59.44
NoWag-VQL (2-13B)
2.01
68.1
62.5
75.9
67.3
37.9
62.34
QuIP # (2-70B)
74.2
70.0
78.8
48.6
69.9
NoWag-VQ (2-70B)
2.02
74.5
69.0
79.4
75.4
46.2
68.9
QulP # (3-8B)
2
63.2
52.7
67.6
57.6
28.2
53.86
NoWag-Von (3-8B)
2.02
67.7
53.0
72.3
68.4
33.2
58.93
Table Zeroshot sequence classification accuracies (%) across 5 tasks and the average accuracies of Quantized Models without finetuning.
Wikitext2 PPL (1)
C4 PPL (1)
Method
Bits
2-7B
2-13B
2-70B
2-7B
2-13B
2-70B
Dense
0%
5.47
4.88
3.32
6.97
6.47
5.52
One Shot
GPTVQ
2.125
8.23
4.64
ClusComp
< 2.01
52.38
22.9
9.84
50.08
24.47
13.96
NoWag-Va VQ
< 2.02
7.59
6.37
4.41
9.28
8.16
6.34
Blockwise Finetuned
ClusComp
2.01 7.5
6.17
4.83
10.29
8.49
7.02
NoWag-VC VQ
< 2.02
7.01
5.93
4.25
8.07
7.64
6.18
Table 4: Wikitext2 and C4 Perplexities of NoWag-VOL VQQ, GPTVQ, and ClusComp at bits
per value for Llama-2 7B/13B/70B at 2048 context length. One shot and blockwise fine- tuned results are provided
with the larger model size. While larger, this codebook is still able fit inside the L1 cache of Nvidia H100 GPU. In Table 3 we compare the Zero Shot accuracies of NoWag- VQ against QulPH. Table we compare the perplexities of NoWag-VO, against QulPH, both at 2 bits per value. NoWag-VQ outperforms Qull# in perplexity for all models for both Wikitext2 and C4. In terms of zero shot accuracies, NoWag-VO beats QulP# for all models beyond Llama-2 70B. We would like to draw attention to how NoWag-Va VQQ is able to significantly improve the quantized performance of Llama-3-8B compared with QulP#. This is worth noting the Llama-3 family of models is significantly harder quantize (Huang et al., 2024).
We also examine the performance of NoWag-VQ beyond the "one-shot" compression regime. Existing literature has proposed several methods for post quantization finetun- ing. One popular method is finetuning the remaining continuous parameters of each transformer block to minimize the block output errors (Egiazarian et al., 2024). Another model-wise finetuning minimize the overall Kullback-Leible divergence with the original model, optimizing over the continuous (Tseng et al., 2024a), and the discrete pa- rameters (Malinovskii al., 2024). Because of our limited computational resources, we were only able to perform block-wise finetuning. We compare the perplexities of those models against those AQLM (Egiazarian al., 2024) in table 2. NoWag- VQ outperforms AQLM for Llama 2 7B and 13B, but falls short for Llama-2 70B. We suspect this is due to AQLM using d VQ rather than = 7, which allows for more than 4x the parameters. However, codebook fits into the L1 cache of an H100 and AQLM's does not.
For additional baselines, we compared the performance of NoWag-VO against two other recent LLM VQ methods, GPTVQ (Van Baalen et al., 2024), and ClusComp (Liao & Monz,
8

---

Published as a conference paper at COLM 2025
2025) in Table 4. ClusComp consists of several variants, including a zero-shot compres- sion variant ClusComp and blockwise finetuned variant, ClusComp. We included our one-shot and blockwise finetuned variants to provided an apples to apples comparison. We believe these comparisons offer additional clarity for the benefits for the two main differences between NoWag and ClusComp. First, ClusComp does not incorporate any normalization, unlike our method. Second, ClusComp performs vanilla vector K-means, rather than weighted K-means, meaning that ClusComp has no data awareness and treats all input channels with equal importance. The benefits of these methods are especially evident in the one-shot regime.
5.2
Pruning Evaluation
Wikitext2 PPL (4)
C4 PPL (4)
Method
Sparsity
2-7B
2-13B
2-70B
3-8B
3-70B
2-7B
2-13B
2-70B
3-8B
3-70B
Dense
0%
5.12
4.57
3.12
5,54
2.58
6.63
6.05
4.97
7.01
5.78
Wanda
50%
6.46
5.58
3.97
9.06
5.34
8.39
7.47
5.77
10.19
7.00
NoWag-P
50%
6.37
5.49
3.89
8.32
4.95
8.27
7.35
5.71
9.67
6.81
Wanda
8.07
6.55
4.49
13.39
6.50
10.19
8.68
6.39
13.95
7.95
NoWag-P 4:8
8.04
6.47
4.45
12.66
6.24
10.17
8.67
6.38
13.86
7.69
Wanda
2:4
11.35
8.36
5.20
22.42
8.29
13.80
10.96
7.19
21.63
9.63
NoWag-P
2:4
11.14
8.28
5.17
24.0
7.52
13.91
11.05
7.23
23.5
9.18
Table 5: Wikitext2 and C4 Perplexities NoWag-P and Wanda at 50% unstructured, and 4:8
and 2:4 semistructured pruning for Llama-2 1B//1B//70B and Llama-3 8B/70B. Context length was at the model's 's native context length, 4096 for Llama-2 and 8192 for Llama-3.
Avg Zero Shut Accuracy (t)
Method
Sparsity
2-7B
2-13B
2-70B
3-8B
3-70B
Dense
0%
63.66
65.76
70.96
70.42
75.89
Wanda
50%
60.24
63.66
70.16
63.49
73.45
NoWag-P
50%
60.48
63.57
70,28
62,93
72.3
Wanda
4:8
58.27
61.32
68.7
57.84
70.42
NoWag-P 4:8
56.71
60.6
68.43
57.46
70.76
Wanda
2:4
55.37
66.73
52.59
NoWag-I
2:4
54.3
58.14
66.95
51.21
67.71
Table 6: Zeroshot Accuracies for Llama-2 7B/13B/70R and Llama-3 8B/70B pruned using NoWag-P and Wanda for 50% unstructured, 2:4 semi-structured, and 4:8 semi-structured
Because Wanda did not report C4 perplexities, we modified the code to compute them, furthermore we added support for pruning the Llama-3 family of models. Because of this, several libraries had to be upgraded from what the original Wanda paper used, resulting perplexities Wikitext2 that are slightly different those reported in Wanda.
Table 5 shows a comparison of the language modeling abilities of NoWag-P and Wanda pruned models, measured through Wikitext2 and C4 perplexity. Three different lev- els of pruning patterns were evaluated, 50% unstructured, 4:8 semi-structured, and 2:4 semi-structured pruning. NoWag-P uniformly outperforms Wanda at 50% and 4:8 semi- structured pruning. This empirically demonstrates the benefits of the NoWag normalizer. However at 2:4 semi-structured pruning, NoWag-P only roughly matches the performance of Wanda. We believe that this is due to the more structured pattern, which negates the impact of the normalizer. To confirm, we conducted a sweep of pruning using NoWag- P and Wanda for range of N:M structures for Llama-2-13B and Llama-3-8B, the results are visualized in Figure 4. The relative reduction of C4 perplexity between NoWag- and rapidly diminishes the pattern becomes more structured (N becomes smaller).
Table 6 shows comparison of the average zero shot task accuracy for NoWag-P and Wanda pruned models. Once again, three different levels of pruning patterns were eval-

---

Published as a conference paper at COLM 2025
1ame-2-136 1amo-3-8
Improvemest qrer Worde 196 Hel/er
6.0%
Heplexitt
4.0%
2.0%
50
150
200
250
from N:2N spersity pettern
Figure 4: Relative difference in C4 perplexity NoWag-P between Wanda: (NoWag Perplexity) (Wanda Perplexity) 1. Calculated for range of semi struc- tured patterns for Llama-2-13B and Llama-3-8B
uated, 50% unstructured, 4:8 semi-structured, and 2:4 semi-structured pruning. For most models across most pruning methods NoWag-P and Wanda produce models similar aver- age zero shot task accuracy.
Conclusion
In this work, we introduced NoWag, a novel framework unifying pruning and quantiza- tion under a common normalization-base approach. Our experimental results demon- strate that NoWag-P improves upon existing pruning techniques in maintaining language modeling accuracy, while NoWag-VQ achieves superior quantization performance with substantially less calibration data. By leveraging structured normalization strategy, NoWag reduces the sensitivity of compression to outlier weights and enhances the effi- ciency of both pruning and quantization. These findings establish NoWag as scalable and adaptable compression paradigm for LLMs, facilitating their deployment in real-world ap- plications with reduced computational costs.
References
Arash Ahmadian, Saurabh Dash, Hongyu Chen, Bharat Venkitesh, Zhen Stephen Gou, Phil Blunsom, Ahmet Üstün, and Sara Hooker. Intriguing properties of quantization at scale. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances Neural Information Processing Systems, volume 36, pp. 34278-34294. Curran Associates, Inc., 2023. URL htwp:tts://rrooeedig...eeeeee neuripsss.....n..oooommmmmm.m... cc/ cc/paper.a... wtteerrfille/ppae///00/ pdf.
David Arthur and Sergei Vassilvitskii. k-means++: the advantages of careful seeding. In Proceedings of the Fighteenth Annual ACM-SIAM Symposium on Discrete Algoritiums, SODA '07, pp. 1027-1035, USA, 2007. Society for Industrial and Applied Mathematics. ISBN 9780898716245..
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Lan- guage models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
10

---

Published as a conference paper at COLM 2025
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. Quip: 2-bit quanti- zation of large language models with guarantees. Proceedings of the 37th International Conference Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ar 0tt::1003.545701, 2018.
Ido Dagan, Oren Glickman, and Bernardo Magnnii. The pascal recognising textual entail- ment challenge. In Joaquin QuinooneroCanndll. Ido Dagan, Bernardo Magnini, and Flo- rence d' d'Alché Buc (eds.), Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pp. 177-190, Berlin, Heidel- berg, 2006. Springer Berlin Heidelberg. ISBN 978-3-540-33428-6.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8(): 8-bit matrix multiplication for transformers at scale. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
Tim Dettmers, Ruslan A. Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Fran- tar, Saleh Ashkboos Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. SpQR: A sparse-quantized representation for near-lossless LLM weight compression. In The Twelfth International Conference on Learning Representations, 2024. URL https:// wttp:/iwwwnne/oooonidii-i---uuu
Jesse Dodge, Maarten Sap, Ana Marasovié, William Agnew, Gabriel Ilharco, Dirk Groen- eveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1286-1305, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/vi /v1//vvv/// / 1ee.211emll-mae main.98. emnlp-main.9.. 98/.
URL httpss:/acclanthologg) org/2021 emnlp-main.
Myroslava O. Dzikovska, Rodney D. Nielsen, Chris Brew, Claudia Leacock, Danilo Gi- ampiccolo, Luisa Bentivogli, Peter Clark, Ido Dagan, and Hoa Trang Dang. Semeval- 2013 task 7: The joint student response analysis and 8th recognizing textual entail- ment challenge. In International Workshop on Semantic Evaluation, 2013. URL https:
semanticccaaal httprriiccccooaooooror//oooeeeee wrrg/CCorpusD/ 48974455. 4897445.
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization. ar Xiv preprint ar arXio:2401.06118, 2024.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately
pruned in one-shot. In International conference machine learning, pp. 10323-10337.
PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate
post-training quantization for generative pre-trained transformers. arXio preprint ar arXiv:2210.1732. 2022.
Leo Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopí, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL. https://zenodo. argg/recodss/156e 12686602.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B. Dolan. The third pascal recognizing textual entailment challenge. In ACL-PASCAL@ACL, 2007. URL https:// wtt.s//oo.....oo///oooorss///2a.
11

---

Published as a conference paper at COLM 2025
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et The llama 3 herd of models. ar preprint ar XXiv:240.221783, 2024.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
B. Hassibi, D.G. Stork, and G.J. Wolff. Optimal brain surgeon and general network pruning. In IEEE International Conference on Neural Networks, pp. 293-299 vol.1, 1993. doi: 10.1109/ (CNN.1993.298577,
C. A. Hoare. Quicksort. The Computer Journal, 5(1):10-16, Jan 1962. doi: 10.1095/comjnl/5. 1.10.
Wei Huang, Xingyu Zheng, Xudong Ma, Haotong Qin, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. An empirical study of llama3 quanti- zation: from Ilms to milms. Visual Intelligence, 2(1), December 2024. ISSN 2731-9008. doi: 0441007/s4425--02--00000
URL h/tp:///xp.ttt////// doi. org/10 00007//4227-024-0000
Weiyu Huang, Yuezhou Hu, Guohao Jian, Jun Zhu, and Jianfei Chen. Pruning large lan- guage models with semi-structural adaptive sparse training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 24167-24175, 2025.
Yichen Huang and Lin F Yang. Gemini 2.5 pro capable of winning gold at imo 2025. aiv preprint ar fii::2507.1585.. 2025.
Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-spars quantization. Xrv preprint ar Xtt:2306.07629, 2023.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
URL https: //axxiv./////////////// orggabs//1110 6980.
Andrey Kuzmin, Markus Nagel, Mart van Baalen, Arash Behboodi, and Tijmen Blankevoort. Pruning vs quantization: Which better? In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neu- ral Information Processing Systems, volume 36, PP- 62414-62427. Curran Associates, Inc., 2023. URL https://proceedingg/ neuripsp cc/ wtt.f////ppaee///////aa.. pdf.
c48bc80aa5d3cbbbdd77 htt.:///watttttteeooooo///////oo
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touret- zky (ed.), Advances in Neural Information Processing Systems, volume 2. Morgan- Kaufmann, 1989. URL https: https::/prroceedings nwwrips.....eee cc cc/paper...... waw.:668---app.p.......
File//6c882bbccc7783322222
881277658
Xinlin Li, Osama Hanna, Christina Fragouli, and Suhas Diggavi. Icquant: Index coding enables low-bit 11m quantization. arXiv preprint axXio:2505.00880, 2025.
Baohao Liao and Christof Monz. Cluscomp: A simple paradigm for model compression
and efficient finetuning. 2025.
Zhu Liao, Victor Quétu, Van-Tam Nguyen, and Enzo Tartaglione. Can unstructured prun- ing reduce the depth in deep neural networks? In 2023 IÉEE/CVF International Conference on Computer Vision Workshops (ICCVW), Pp. 1394-1398. IEEE, October 2023. doi: 10.1109/ iccvw60793.200 210.993..2220000110 URL http:://xx... doi wtw.://doooooo///...oo org/10. 1tt.1/11//////0099990000 0823.00111
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for 11m compression and acceleration. In MI.Sys, 2024.
Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li Lyna Zhang, Ting Cao, Cheng Li, and Mao Yang. Vptq: Extreme low-bit vector post-training quantization for large language models. In The 2024 Conference Empirical Methods in Natural Language Processing, 2024.
12

---

Published as a conference paper at COLM 2025
Hosam M. Mahmoud, Reza Modarres, and Robert T. Smythe. Analysis of quickselect: algorithm for order statistics. RAIRO Theurefical Informatics and Applications Informa- fique Théorique et Applications, 29(4):255-2) 1995. URL httpp::/www. numdam....mmmmmmmmmmmmmmmmmm.. org/ tem/ ITA 1995 29 4 255 0/.
Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin Burlachenko, Kai Yi, Dan Alistarh, and Peter Richtarik. Pv-tuning: Beyond straight-througe estimation for extreme Ilm compression. Advances in Neural Information Processing Systems, 37:5074- 5121, 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. ar preprint Xiiv:1609.07833 2016.
Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization, 2020. URL https://arxiv org./os/////////ooooo 2204 10568.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895 2022.
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pp-
1-22, 2023.
Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea Goldsmith, and Mert Pilanci. Com- pressing large language models using low rank and low precision decomposition. Ad- vances in Neural Information Processing Systems, 37:88981-89018, 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9): 99-106, 2021.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Ro-
former: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024. ISSN 0925-2312. doi: httt:////eeeeeeettccc////////13/
URL
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. Xiv preprint ar Xix:::3306.1166 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXio:2307.09255 09228, 2023.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. QulPş) #S: Even better LLM quantization with hadamard incoherence and lattice code- books. In Forty-first International Conference on Machine Learning, 2024a. URL https: //openreview. new. wet//formmmiii d-BryuuU-u-rrrccoooee
Albert Iseng, Qingyao Sun, David Hou, and Christopher De. Qtip: Quantization with trellises and incoherence processing. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neu- ral Information Processing Systems, volume 37, pp. 59597-59620. Curran Associates, Inc., 2024b. URL https://proceeding: www.igs.eeups....omm cc/ cctpsccr.ppp....ceeeeeeeeeeee. wwtpppaflleepppaaeeaeaapppeeeppe ffilee//ppper/22///fllee pdf.
Mart Van Baalen, Andrey Kuzmin, Ivan Koryakovskiy, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. Gptvq: The blessing of dimensionality for Ilm quantization. aiv preprint ar 2rxi::4002.53319 2024.
13

---

Published as a conference paper at COLM 2025
Jesse Vig. A multiscale visualization of attention in the transformer model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demon- strations, pp. 37-42, 2019.
Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athi- waratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christo- pher Ré, Irina Rish, and Ce Zhang. Redpajama: an open dataset for training large lan- guage models. NeurIPS Dalasels and Benchmarks Track, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thouggh prompting elicits reasoning in large language mod- els. Advances in neural information processing systems, 35:24824-24837 2022.
Omry Yadan. Hydra- a framework for elegantly configuring 2019. URL https: /
Github,
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yux- iong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances neural information processing systems, 35:27168-27188, 2022.
14

---

Published as a conference paper at COLM 2025
Appendix
A Detailed Formulation of Quantization
To quantize model, we map each weight entry or contiguous vector of weight entries to a codebook. Without loss of generality, we assume that use vector quantization algorithm where we quantize every parameters, Wijj+td together. Then quantization results in the following:
A codebook = fc E IR*)
A mapping M(W;fjid = E.
when we represent the quantized weights for inference, these mappings become a string of bits of length [|CI]. Therefore the resulting in a bits per value of (log, ([[CIT) + é) à, where is the bits needed to represent the overhead of normalization parameters, codebooks, etc. Note that we can inverse this relationship to find that if we have a target bits per values
Nhyp. the size of the codebook should be [C| 2nboad The main benefit of vector quan- tization is that it allows for the quantization codebook to be better shaped to the weights. However, the size of the codebook increases exponentially with the dimension d, which leads to c/d longer becoming a negligible quantity compared with the bits used to code each value. Furthermore, for fast inference, C must fit inside the L1 cache of a GPU. This has lead to a line of work on more efficient encoding schemes, such as trellis encoding schemes pioneered by (Tseng et al., 2024b).
A.1 Weighted Vector K-Means Formulation
As discussed in section 4, for quantization we used weighted vector K-Means, this con- sists of two steps, assignment step and update step, below explicitly write each step. Traditionally, K-Means run with several random initializations, and allowed to converge for each initialization. However, in the interests of runtime, for each layer, only initialized once using the K-means++ algorithm (Arthur & Vassilvitskii, 2007), and only performed 100 assignment update step pairs. We did observe increasing performance scaling from 20 to 100 assignment steps, therefore we believe that the results reported in tables 3 do not demonstrate the full performance of NoWag-VQ.
Assignment step: For each vector Wi a we select the mappings to such that the weighted 12 norm is minimized.
M ((W.fj+a) = arg min (Wirzta-c) - -i'" (diag( Xxx) jijtd O (Winittd- - -4)
(4)
GEC
Update step: For each vector in the codebook, take the weighted average of the assign- ments:
Σ {
Σ
diag(XX)) Djjtd Wisgtd o
diag(XX Xx7 )
=
(5)
B Zero Shot Task Descriptions
NoWag-P was evaluated on zero-shot accuracy noted Tables 10, 11, and 12. The classifi- cation sequence classification tasks are as follows:
1. WinoGrande (Sakaguchi et al., 2021) - WINOGRANDE is a large-scale dataset of 44k problems based on the Winograd Schema Challenge, specifically crafted to minimize biases training data. It features a two-choice fill-in-thebllak format that requires deep commonsense reasoning.
15

---

Published as a conference paper at COLM 2025
2. RTE (Dagan et al., 2006; Giampiccolo et al., 2007; Dzikovska et al., 2013) - The Recognizing Textual Entailment (RTE) Challenge has participants determine se- mantic relationships like entailment, contradiction, neutrality between sentence pairs.
3. PIQA (Bisk al., 2020) -200)-PQAA is benchmark dataset for physical commonsense reasoning, having Al answer questions about everyday interactions without direct physical experience.
4. ARC-e (Clark al., 2018) A subset of the A12 Reasoning Challenge (ARC), ArcE consists of multiple-choice questions designed to assess grade-school level knowl- edge and represents the "Fasy" portion of the dataset.
5. ARC-c (Clark et al., 2018) - The ARC-Challenge subset follows the same format as ARC-Easy but includes only questions that baseline algorithms previously failed to answer correctly.
C Quantization Additional Evaluations
We include additional comparisons with three more VQ algorithms, AQLM (Egiazarian et al., 2024), VPTQ (Liu et al., 2024) and CALDERA (Saha et al., 2024). AQLM (Additive Quantiztation for Language Models) performs quantization through additive multi code- book VQ. VPTQ combindeds are vector quantization extension of GPTQ (Frantar et al., 2022) with residual quantization. CALDERA builds ontop of QulP# by adding additional quantized low rank matrices, and as a result, requires higher bits per value.
For AQLM and VPTQ, their appendices included zero shot ablation results for perplexity of Wikitext2 and C4. For VPTQ, because a very detailed ablation was chosen, we simply chose the compres-
chose the best performing zero-shot quantization. For CALDERA, sion with the least equivalent bits per value. Perplexities are shown in table 7, and zero shot results with CALDERA are show in table 8. We can see that our algorithm performs competitively to these modern VQ algorithms as well.
Method
Bits
Wiki2 (1)
C4 (1)
fp16 (2-7B)
16
5.12
6.63
fp16 (2-13B)
16
4.57
6.05
fp16 (3-8B)
16
5.54
7.01
fp16 (2-70B)
16
3.12
4.97
AQLM (2-7B)
2.02
8.18
10.59
QUIP (2-7B)
2
8.23
10.8
CALDERA (7B)
2.1
7.37
9.74
NoWag- VQ (2-7B)
2.02
7.07
9.02
QulP # (2-13B)
2
6.06
8.07
CALDERA (2-13B)
2.08
6.04
7.98
VPTQ (2-13B)
2.07
6.02
7.96
NoWag-Ve VQ (2-13B)
2.01
5.93
7.94
QulP# (3-8B)
2
13.8
15.6
CALDERA
2.1
10.6
11.8
NoWag VQ (3-8B)
2.02
10.68
11.92
QulP# (2-70B)
2
4.16
6.01
CALDERA (2-70B)
2.1
4.11
5.95
NoWag VQ (2-70B)
2.01
4.15
5.94
Table 7: Performance comparison of different methods on Wiki2 and C4 datasets.
16

---

Published as a conference paper at COLM 2025
Bits
Wino (1)
RTE (1)
PIQA (1)
ArcE (1)
AreC (1)
Avg Ace (1)
FP16 (2-7B)
67.3
63.2
78.5
40.0
63.66
FP16 (2-13B)
16
69.5
61.7
78.8
73.2
45.6
65.76
FP16 (3-8B)
16
73.5
68.6
79.7
80.1
50.2
70.42
FP16 (2-70B)
77.0
67.9
81.1
77.7
51.1
70.96
QulP# (2-7B) )
61.7
57.8
69.6
61.2
29.9
56.04
Caldera (2-7B)
2.1
63.7
62.1
72.3
60.9
31.7
58.14
NoWag- VQ (2-7B)
2.02
64.4
54.5
73.6
60.7
31.7
56.99
QulP # (2-13B)
63.6
54.5
74.2
68.7
36.2
59.44
Caldera (2-13B)
2.08
66.9
61.0
76.0
69.5
37.2
62.12
NoWag-VQ(2-130) (2-13B)
2.01
68.1
62.5
75.9
67.3
37.9
62.34
QulP # (3-8B)
2
63.2
52.7
67.6
57.6
28.2
53.86
Caldera (3-8B)
66.9
58.5
71.8
34.3
NoWag-VQ (3-8B)
2.02
67.7
53.0
72.3
68.4
33.2
58.93
QulP # (2-70B)
2
74.2
70.0
78.8
77.9
48.6
69.9
Caldera (2-70B)
2.1
75.5
69.3
79.8
76.9
47.7
69.84
NoWag- (2-70B)
2.02
74.5
69.0
79.4
75.4
46.2
68.9
Table Zeroshot accuracies (%) across 5 tasks and the average accuracies of Quantized Models without finetuning.
D Additional Pruning Results
Wikitext2 PPL (1)
Sparsity 50%
2-7b
2-13b
2-70b
3-8b
3-70b
SparseGIT
6.51
5.63
3.98
8.53
5.31
NoWag-P 50%
6.37
5.49
3.89
8.32
4.95
SparseGPT 4:8
7.99
4.59
10.92
6.56
NoWag-P 4:8
8.04
12.66
SparseGPT 2:4
10.23
8.29
5.38
14.31
8.62
NoWag-P
2:4
11.14
8.28
5.17
24.0
7.52
Table 9: Comparison of NoWag-P and SparseGPT (Frantar & Alistarh, 2023) perplexities
for Llama-2 7B/13B/70B and Llama-3 8B/70B
Sparsity
Wino (t)
RTE (t)
PiQA (1)
ArcE (1)
ArcC (1)
Avg Acc (t)
FP16 (2-7B)
0%
67.3
63.2
78.5
69.3
40.0
63.66
FP16 (2-13B)
78.8
FP16 (2-70B)
0%
77.0
67.9
81.1
77.7
51.1
70.96
FP16 (3-8B)
0%
73.5
68.6
79.7
80.1
50.2
70.42
FP16 (3-70B)
0%
80.7
69.0
82.5
86.8
60.4
75.89
Wanda (2-7B)
50%
66.9
55.6
75.6
66.2
37.0
60.24
NoWag- (2-7B)
50%
65.7
60.7
75.7
65.5
34.9
60.48
Wanda (2-13B)
50%
68.9
58.5
78.4
41.0
63.66
NoWag-P (2-13B)
50%
68.9
59.6
77.8
71.3
41.0
63.57
Wanda (2-70B)
50%
76.9
69.3
80.5
75.9
48.2
70.16
NoWag (2-70B)
50%
76.6
80.7
75.5
47.4
70.28
Wanda (3-8B)
50%
71.0
59.9
74.9
71.4
40.3
63.49
NoWag-P (3-813)
50%
70.0
56.7
75.8
71.7
40.4
62.93
Wanda (3-70B)
50%
78.0
70.0
81.3
NoWag (3-70B)
50%
76.7
67.9
81.2
82.8
52.9
72.30
Table 10: Zeroshot accuracies for each task for 50% Pruning
17

---

Published as a conference paper at COLM 2025
Sparsity
Wino (t)
RTE (1)
PIQA (1)
ArcE (1)
ArcC (t)
Avg Acc (f)
HP16 (2-7B)
0%
67.3
63.2
78.5
69.3
40.0
63.66
FP16 (2-13B)
0%
69.5
61.7
78.8
73.2
45,6
65.76
FP16 (2-70B)
77.0
67.9
81.1
77.7
70.96
FP16 (3-8B)
0%
73.5
68.6
79.7
80.1
50.2
70.42
FP16 (3-70B)
0%
80.7
69.0
82.5
86.8
60.4
75.89
Wanda (2-7B)
4:8
65.35
58.12
73.61
62.46
31.83
58.27
NoWag P (2-7B)
4:8
64.7
54.2
72.7
61.7
30.2
56.71
Wanda (2-13B)
4:8
68.98
55.96
75.79
67.47
38.4
61.32
NoWag- (2-13B)
69.0
75.0
65.5
Wanda (2-70B)
4:8
74.9
67.87
79.54
74.71
46.5
68.7
NoWag-P (2-70B)
4:8
75.6
67.2
79.3
74.0
46.2
68.43
Wanda (3-8B)
4:8
66.69
53.07
71.0
64.31
34.13
57.84
NoWag-P (3-8B)
4:8
65.0
54.5
70.7
63.6
33.4
57.46
Wanda (3-70B)
4:8
73.8
66.06
80.09
80.89
51.28
70.42
NoWag-P (3-70B)
4:8
76.1
65.7
79.7
81.4
50.9
70.76
Table 11: Zeroshot accuracies for each task for 4:8 Pruning
Sparsity
Wino (1)
RTE (1)
PIQA (1)
ArcE (1)
ArcC (t)
Avg Acc (f)
FP16 (2-7B)
0%
67.3
63.2
78.5
69.3
40.0
63.66
FP16 (2-13B)
0%
69.5
61.7
78.8
73.2
45.6
65.76
FP16 (2-70B)
0%
77.0
67.9
81.1
77.7
51.1
70.96
FP16 (3-8B)
73.5
79.7
70.42
FP16 (3-70B)
0%
80.7
69.0
82.5
86.8
60.4
75.89
Wanda (2-7B)
2:4
60.5
58.5
70.1
57.6
30.2
55.37
NoWag (2-7B)
2:4
69.3
55.8
27.9
54.30
Wanda (2-13B)
2:4
65.8
54.5
73.1
63.8
34.0
58.2
NoWag-P (2-13B)
2:4
65.6
58.1
72.4
62.9
32.7
58.14
Wanda (2-70B)
2:4
73.6
64.6
78.9
72.9
43.2
66.70
NoWag P (2-70B)
2:4
75.1
66.8
77.6
72.5
42.7
66.95
Wanda (3-8B)
2:4
59,9
52.7
67.5
56.9
25,9
52.59
NoWag-P (3-8B)
2:4
58.1
52.7
66.6
54.4
24.2
51.21
Wanda (3-70B)
2:4
71.7
63.9
78.1
78.5
48.2
68.08
NoWag-P (3-70B)
2:4
72.9
62.5
78.5
77.8
46.9
67.71
Table 12: Zeroshot accuracies for each task for 2:4 Pruning
E Inference Speedup
A primary objective of model compression to reduce the memory footprint during in- ference. NoWag-P,, at 50% sparsity, achieves 2x reduction in memory required to store weights, while NoWag- VQ compresses weights to 2 bits per value-resultin in an 8x re- duction compared to fp16 precision.
In addition to memory savings, our method also leads to inference speedups due to the alleviation of memory bandwidth bottlenecks and the use of structured sparsity. To illus- trate this, in Table we report the matrix multiplication runtimes for 2:4 NoWag-P on an NVIDIA A6000 GPU, matching the evaluation setup used in prior work such as SParseGPT, Wanda, and AQLM. Specifically, we benchmark matrix-vector multiplication on the gate projection layers of L.LMA-2 models of varying sizes (7B, 13B, and 70B).
18

---

Published as a conference paper at COLM 2025
2-7B
2-13B
2-70B
FP-16
0.137ms
0,205ms
0.673ms
NoWag-P
0.117ms (1.173x)
0.15ms (1.373x)
0.401ms (1.679x)
Table 13: Benchmarked matrix-vector multiplication on the gate projection layers of LLaMA-2 models of varying sizes (7B, 13B, and compared between the original dense FP-16 matrices and the NoWag-I' compressed matrices
F
Additional Algorithms
Algorithm VQ Padding
Require: Normalized weight matrix W E Rdouttddn input dimension din, subvector di- mension d
Ensure: Padded weight matrix W Wpadddd padded input dimension d
padded
1: if din mod d + then
2: pad_size < - (din mod d)
3: pad_value + mean(W) (W)
4: Pad W with pdd_ value shape (dout, dn + size)
ppadedd + din pad_size in
5:
6: else
7: W padded t W
8: ppadddd +- din
9: end if
10: return W padded, padded
Algorithm NoWag Pruning (NoWag-P)
Require:
Weight matrix W R activation samples X E Raddd, target sparsity s, sparsity pattern @ [unstructured,NNM)
Ensure: Pruned weight matrix W
1: (22)),,(2) W) < NoWag _NormalizationWW) (Apply normalization) [1,dout] Vj E [1, dm)] (Importance scores)
2: Sij < W/ ||
3: if Pattern is unstructured then
4: T T {s-th percentile of all elements in S}
5: M M+115> (Binary mask where Mi = if Si₃ > T, else 0}
6:
else Pattern is N:M then
7:
for each row E [1,dout] do
8: for g = 0 to d - do
9:
Ig t- {gM g + ..2,....gM.......... M) [Indices of group g)
10:
Tg top-N((Si j 1x)) (Indices of top N scores)
Mij - { i Tg
11:
Vie lg (Set mask]
otherwise"
12: end for
13: end for
14: end if
15: W+ W M (Element-wise multiplication)
16: return W
19

---

Published as a conference paper at COLM 2025
Algorithm 6 Weighted K-Means with K-Means++ Initialization
Require:
Subvectors Wsub E R Nxd, importance weights Hsub E R RNx number of cen- troids #centroids number of iterations Hiter
Ensure: Codebook C E R"ceentuids Xd assignments A E RN
1: Initialize codebook C R"Mentrvids: xd.
2: Select random indices
in. from [1, without replacement
3: C f Suubi 1I E [1, centtoodds
4: for = to Riter do
5: Assignment step:
6: for each subvector i E [1, do
7:
Di + -i-1 Haub..j (Wsub..) ( VA E [1, centroids
8:
4 arg D
9:
end for
10:
Update step:
11:
for each centroid E [1, Icenrroid do S (i:A: A, 13
13:
if S: + 2 then
for dimensionj jE [1,d] [ do
Lies, Hauh,y Weuhhdf (Weighted average) Lies, Haubij
15:
Cī +
16:
end for end if
17:
18: end for
19: assignments haven't changed since last iteration then
20: break
21: end if
22:
end for
23: return C, A
20

---

Published as a conference paper at COLM 2025
G Implementation Details
G.1 Finetuning
Finetuning was performed in blockwise fashion. For each transformer block, objec- tive was minimizing the 12 norm between the outputs of the original block and those of the quantized blocks. The parameters to optimize over were the codebooks, and the normal- ization vectors of each quantized layers, and the RMS norm parameters. In addition we initialized bias vector, set to all zeros, for each linear layer in the block.
For Llama-2 7B/ 13B,, finetuning was done using 128 samples of Red Pajamas (Weber et al., 2024), with held out as a validation set. Optimization was done through Adam Kingma & Ba (2017), without any weight decay and learning rate of 10-4, For Llama-2 70B, 256 samples Red Pajamas was used with a learning rate 10-6, Configuration management for both one-shot and finetuning was done through hydra (Yadan, 2019).
H Ablations
H.1 K-means iterations for NoWag-VQ
Wikitext2 PPL (L)
C4 PPL (4)
(2-7B)
(2-13B)
(3-8B)
(2-7B)
(2-13B)
(3-8B)
QuiP NoWag-VO T
8.23
6.06
13.80
10.80
8.07
15.60
7.32
6.20
12.44
9.58
8.36
13.96
No T 10
7.04
6.00
11.08
9.19
8.03
12.35
NoWag-VG VQ T 40
7.10
5.99
10.86
9.15
7.98
12.07
NoWag- VQ T 100
7.07
5.93
10.68
9.12
7.94
11.92
Table 14: Ablation for T= 1,10, 40 against the default 100 for NoWag-VOQ for Llama-
2-7b/13b and Llama-3 8B at bits per value. We listed the Wikitext2 and C4 perplexities below, bolding the smallest for which NoWag outperforms Qull#. For Llama-2 7b and Llama-3 8B NoWag with only one iteration of K-means was able to outperform QulP#, for Llama-2 13B, after 10 iterations, NoWag-VQ was able to outperform QulP#. We also note that the performance of the model continues increase we increase the number of iterations.
H.2 Row and Column Normalization
Across all models and all compression methods, normalizing along row and column out- performs normalizing along row or along column alone.
Wikitext2 PPL
C4
(2-7b)
(3-8B)
(2-7b)
(3-8B)
NoWag- col only
7.45
11.45
9.54
12.53
NoWag-VO row only
7.68
11.29
10.10
12.53
NoWag- -QQ row and column
7.07
10.68
9.12
11.92
Table 15: Ablation for NoWag-VQ at 2 bits per value with row or col only normalization against the default of row and column normalization for Llama-2 7B and Llama-3 8B
21

---

Published as a conference paper at COLM 2025
Wikitext2 PPL
(4)
C4 PPL (4)
(2-7b)
(2-13B)
(3-8B)
(2-7b)
(2-13B)
(3-8B)
NoWag-I col only NoWag-P row only
7.01
5.53
9.26
9.22
7,44
10.60
6.42
5.82
8.49
8.37
7.88
9.78
NoWag-P row and column
6.37
5.49
8.32
9.12
7.35
9.67
Table 16: Ablation for NoWag-P at 50% unstructured sparsity with row or col only nor-
malization against the default of row and column normalization for Llama-2 7B/13B and Llama-3 8B
H.3 Calibration Dataset
Wikitext2 PPL (4)
PPL (1)
Calibration dataset
(2-7b)
(3-8B)
(2-7b)
(3-8B)
Wikitext2
6.9
10.9
9.16
12.55
C4
7.19
11.22
9.12
12.055
RedPajamas (default)
7.07
10.68
9.12
11.92
Table 17: Ablation for different calibration datasets, training splits of Wikitext2 and C4, compared with the default of RedPajamas, conducted on NoWag- VQ for Llama-2 7B and Llama-3 8B. 128 samples were used for each dataset.
22


Reviews:
[{"review_id": "Mh2Crk9kb1", "reviewer": "Reviewer_5R73", "summary": "This paper introduces NoWag (Normalized Weight and Activation Guided Compression), a framework for post-training compression of Large Language Models (LLMs). The core idea is a two-step normalization process applied to weight matrices (first column-wise, then row-wise) followed by compression guided by the second moment of activations (specifically, diag(XX^T)). The authors propose two instantiations: NoWag-VQ for vector quantization using weighted K-means, and NoWag-P for un-/semi-structured pruning by removing elements with the smallest normalized weight magnitude multiplied by activation norms. Experiments on Llama-2 and Llama-3 models show that NoWag-VQ significantly outperforms SOTA zero-shot VQ methods like QuIP# (especially with much less calibration data), and NoWag-P is competitive with or slightly better than SOTA pruning methods like Wanda in terms of perplexity, while being comparable in zero-shot task accuracy. The paper argues that the proposed normalization makes the weight distribution more amenable to compression and reduces sensitivity to outliers.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. **Clarity**: The paper is generally well-written and easy to follow. The core normalization idea is explained clearly, supported by algorithms and illustrative figures (Figs. 1, 2, 3) that effectively convey the intuition. The problem formulation and connection to existing work are well-established.\n\n2. **Originality**: The main originality lies in the specific two-step normalization procedure (column-then-row normalization) and its application as a unified pre-processing step for both VQ and pruning, coupled with an activation-guided importance score. While components like weighted K-means or magnitude pruning with activation scaling are known, their combination with this particular normalization to improve compression is novel. The observation that this normalization creates a more \"ball-shaped\" distribution (Fig. 3) is an interesting insight.\n\n3. **Significance**: The work is significant for several reasons: (1) It presents a simple yet effective technique that improves zero-shot VQ performance, notably outperforming SOTA (QuIP#) while using 48x less calibration data. This is a substantial practical contribution. (2) It offers a competitive pruning method (NoWag-P) that slightly improves upon Wanda in perplexity. (3) The analysis and visualizations provide valuable insights into how weight normalization can benefit LLM compression. It suggests a common underlying principle that can be beneficial for different compression paradigms, potentially inspiring future unified approaches.", "reasons_to_reject": "1. \"Unified Framework\" Scope: The term \"unified framework\" is used, but the actual compression algorithms (K-means for VQ, greedy selection for pruning) remain distinct. The unification primarily comes from the shared normalization pre-processing step and the conceptual objective (Eq. 3). This is a minor wording concern rather than a fundamental flaw.\n\n2. Performance on Llama-2 70B with Finetuning (Table 2): NoWag-VQ falls slightly short of AQLM on Llama-2 70B with finetuning. The authors attribute this to AQLM using d=8 vs. NoWag's d=7. While plausible, it's a point where NoWag-VQ isn't outperforming. This is a minor point as overall performance is strong.\n\n3. Clarity on Pruning Score: There's a slight ambiguity in the pruning score formulation.\nore). This should be clarified: is the pruning score based on |W_bar_ij| ||X_j||_2 or (W_bar_ij)^2 ||X_j||_2^2 (or its square root)? The former is used in Wanda-like methods, the latter aligns more directly with minimizing the squared error in Eq. 3. The current text and Figure 2 legend imply the former (|W_bar_ij| ||X_j||_2). If so, the connection to Eq. 3 for pruning is less direct than for VQ.", "questions_to_authors": "1. Could you please clarify the exact pruning score used for NoWag-P? Is it |W_bar_ij| ||X_j||_2 (as implied by line 134 and Fig 2 legend) or is it derived more directly from minimizing Eq. 3, e.g., by pruning based on (W_bar_ij)^2 ||X_j||_2^2? If it's the former, how does this align with the objective in Eq. 3 for pruning\n\n2. (minor) The paper mentions (line 130) T << min(d_in, d_out) for the number of K-means iterations. For NoWag-VQ, 100 iterations were used (line 587). Could you comment on the sensitivity to T? Could fewer iterations still yield strong results, further speeding up the compression?\n\n3. The computational complexity for NoWag-VQ is given as O(d_in d_out T 2^(n_bpv d)) (line 129). This is O(N_{total\\_weights} T 2^{k}) where k is bits per subvector. Can you briefly compare this complexity to QuIP# or other VQ methods if readily available, beyond the Hessian-based methods which are O(d^3)? The main benefit seems to be fewer calibration samples rather than algorithmic complexity of the VQ step itself, correct?", "extra_scores": {}}, {"review_id": "L5RafHogAF", "reviewer": "Reviewer_sxCa", "summary": "In this paper, the authors propose a model compression method, NoWag, that can be applie to both quantization and pruning. Unlike proor works that compress the original weight matrices, the authors propose to first normalize the weight matrices row-wise and column-wise, and then compress it. In this way, the elements in the weight matrix has similar scalar range that are friendly to compression. After the normalization, the weight matrix can be compressed by using the same methods proposed earlier, i.e. kmeans for VQ and Wanda for pruning.\n\nAfter extensive experiments, the authos show:\n\n1. Compared to VQ methods, NoWag outperforms QUIP#, and is comparable to AQLM.\n\n2. Compared to Prunning method, NoWag outperforms Wanda for the 50% unstructured setting, comparable or mixed to Wanda for the 4:8 and 2:4 settings.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method is very easy to implement. Only one additional step is required prior to existing methods, i.e. normalization. And the overhead is neglegible. \n\n2. The motivation for this normalization step is clear to me. It's similar to store the salient weight element in a two vector per-column and per-row.\n\n3. The paper is clearly written, easy to follow.", "reasons_to_reject": "1. **The results are a bit mixed**. For quantization, NoWag clearly outperforms QUIP#, but is comparable to AQLM. For pruning, NoWag clearly outperforms Wanda for the 50% setting, but is comparable for the 2:4 and 4:8 settings.\n\n2. **Lack of baselines**. Inspired by the first step, I would suggest to include more VQ and Pruning baselines to show the benifit of NoWag. For example, VQ baselines could be GPTVQ [1] and ClusComp [2]. [2] seems a concurrent work, but a related discussion is also benificial. Pruning baselins could be DARE[3] and SLERP[4].\n\n[1] GPTVQ: The Blessing of Dimensionality for LLM Quantization\n\n[2] ClusComp: A Simple Paradigm for Model Compression and Efficient Finetuning\n\n[3] Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch\n\n[4] Spherical Linear Interpolation and Text-Anchoring for Zero-shot Composed Image Retrieval", "questions_to_authors": "See reasons to reject for clarification.", "extra_scores": {}}, {"review_id": "mKkqoq1cqv", "reviewer": "Reviewer_yN5X", "summary": "This paper studies the model compression. The paper clearly states the problem of model compression using calibrated datasets and the proposed approach. The main idea to to normalize the weight and then using the calibration dataset's activation as a weight to guide the VQ approach to compress the model weight more efficiently. The paper is well written and with a discussion section describing why the proposed approach works.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is very well written and the problem statement is very clear.\n2. The proposed approach makes a lot of sense and has illustration and explanation for why the proposed approach works.\n3. The model achieves quite good performance for compressing the small size model.", "reasons_to_reject": "None", "questions_to_authors": "This paper is so well and clearly written. It's a pleasure to read from the beginning to the end. Although I am not an expert in model compression, this paper is quite strong by providing both empirical results and discussions for describing why the proposed approach works. The figure 3 helps quite a lot for understanding the approach.\n\nJust one quick question, how sensitive of the approach to the calibration dataset?", "extra_scores": {}}, {"review_id": "Bmi4R1XK42", "reviewer": "Reviewer_H7VY", "summary": "The paper introduces NoWag, a framework combining vector quantization (NoWag-VQ) and pruning (NoWag-P) to compress large language models (LLMs). It achieves better performance than existing methods, reducing computational and memory requirements while maintaining accuracy.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This framework achieves good performance.\n\n2. This paper is well-written.", "reasons_to_reject": "1. There's no inference speedup evaluation.\n\n2. The technique contribution is insufficient, as I believe the only contribution is that normalization can help gain good performance for VQ and pruning. \n\n3. Lack of ablation comparisons of proposed methods w/ and w/o column(row) normalization.\n\n4. Comparisons with more advanced pruning approaches could help validate its SOTA performance, e.g., [SparseGPT](https://github.com/IST-DASLab/sparsegpt), [https://arxiv.org/abs/2409.17372](https://arxiv.org/abs/2409.17372).", "questions_to_authors": "N/A", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces NoWag (Normalized Weight and Activation Guided Compression), a framework for post-training compression of LLMs. The authors provided a strong rebuttal, which led to two reviewers increasing their scores. After the rebuttal, the paper received scores of 6678. All reviewers expressed positive opinions about the paper, highlighting several strengths: (1) The paper is very well written with a clear problem statement; (2) The proposed approach is well-motivated, with intuitive explanations and illustrations of why it works; (3) The method achieves strong performance in compressing smaller-scale models. Given these merits, the AC recommends accepting the paper.
Decision: Accept