RWKVQuant: Quantizing the RWKV Family with Proxy Guided Hybrid of Scalar and Vector Quantization
Authors: XUCHEN
Yuxuan Yue
Zukang Xu
Xing Hu
JiangyongYu
Zhixuan Chen
Sifan Zhou
Zhihang Yuan
Dawei Yang
Venue: ICML 2025 poster (2025)
Keywords: RWKV
Quantization
Primary area: deep_learning->large_language_models

Abstract: RWKV is a modern RNN architecture with comparable performance to Transformer, but still faces challenges when deployed to resource-constrained devices. 
Post Training Quantization (PTQ), which is a an essential technique to reduce model size and inference latency, has been widely used in Transformer models.
However, it suffers significant degradation of performance when applied to RWKV.
This paper investigates and identifies two key constraints inherent in the properties of RWKV:  (1) Non-linear operators hinder the parameter-fusion of both smooth- and rotation-based quantization, introducing extra computation overhead. (2) The larger amount of uniformly distributed weights poses challenges for cluster-based quantization, leading to reduced accuracy.
To this end, we propose RWKVQuant, a PTQ framework tailored for RWKV models, consisting of two novel techniques: (1) a coarse-to-fine proxy capable of adaptively selecting different quantization approaches by assessing the uniformity and identifying outliers in the weights, and (2) a codebook optimization algorithm that enhances the performance of cluster-based quantization methods for element-wise multiplication in RWKV.
Experiments show that RWKVQuant can quantize RWKV-6-14B into about 3-bit with less than 1\% accuracy loss and 2.14$\times$ speed up.

RWKVQuant:
Quantizing the RWKV Family with Proxy Guided Hybrid of Scalar and Vector Quantization
Chen Xu "1 Yuxuan Yue
Zukang Xu Zhihang Yuan
Xing Hu
Jiangyong Yu Zhixuan Chen'
Sifan Zhou'
Dawei Yang"
Abstract
RWKV a modern RNN architecture with com- parable performance to Transformer, but still faces challenges when deployed to resource- constrained devices. Post Training Quantization (PTQ), which is essential technique reduce model size and inference latency, has been widely used in Transformer models. However, it suf- fers significant degradation of performance when applied to RWKV. This paper investigates and identifies two key constraints inherent the prop- erties of RWKV: (I) Non-linear operators hinder the parameter-fusin of both smooth- and rotation- based quantization, introducing extra computation overhead. (2) The larger amount of uniformly distributed weights poses challenges for cluster- based quantization, leading to reduced accuracy. this end, we propose RWK WWKKQuant, PTQ framework tailored for RWKV models, consist- ing of two novel techniques: (1) coarse-to-fine to-fine proxy capable of adaptively selecting different quantization approaches by assessing the unifor- mity and identifying outliers in the weights, and (2) codebook optimization algorithm that en- hances the performance of cluster-basse quantiza- tion methods for Dlement-wist multiplication in RWKV. Experiments show that RWK Vuannt quantize RWKV-6-144 1BB into about 3-bit with less than 1 %% accuracy loss and 2. 2.14x speed up.
60-
Accuracy
45
FP16
S0-GPTQ-J.500
VO-GPTVO-3.55x
Hybrid wWw.Quant.3.uu
10a
10a
10a
Model Size (MB)
Figure I mccuracy-model size curve. Results of zero-shot accuracy are evaluated on the LAMBADA dataset (Radford et al., 90. Our proposed RWKV Quant outperforms the individual utilization SQ and VQ methods for sizes of models.
It has a comparable capacity to Transformer-basee Large Language Models (T-LLMs) while retaining the efficient inference feature of RNNs, positioning promising foun- dational architecture for both language (Peng et al., 2024) and vision (Zhou & Chen, 2024) tasks.
Despite the advantages, its vast size of parameters have posed a significant barrier to the deployment on resource- constrained devices. For instance, the compute-to tompuutr----mmmory access ratio (FL.OP//Bytes) of RWKV-6-7B (Peng al., 2024) is 0.97, while that for the decoding phase of LLaMA- 2-7B (Touvron al., 2023) is 4.88 (detailed in A.3), Sec- ondly, large RWKV models demand substantial memory resources. For instance, RWKV-6-14H requires approx- imately 30GB of memory to be loaded, which typically exceeds the capacity edge devices.
1. Introduction
RWKV (Peng et al., 2023) is modern sequence model that integrates the strengths of both Recurrent Neural Networks (RNNs) (Elman, 1990) and Transformer (Vaswani, 2017).
Post Training Quantization (PTQ), including Scaler Quanz- ization (SQ) and Vector Quantization (VQ), is widely adopted approach to reduce model size and inference la- tency for T-LLMs (Shao et al., 2023: Ashkboos et al., 2024; ost, 2024; Yuan et al., 2024). However, directly applying the most advanced quantization frameworks to RWKV mod- els leads to severe performance degradation. For instance,
'Equal contribution Technology (Shenzhen).
'Houmo AI Harrin Institute of Correspondence to: Dawei Yang
<dawei.yange hiumo.ai>.
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

RWKVQuant: Proxy-Guided Hybrid Scalar- Quantization for RWKV Models
Table 1: The average relative cluster loss of weights for the RWKV and LLaMA family, computed by KMeans (Lloyd,
families, including RWKV-6 (Peng et al., 2024) and R (Bo, 2021) for RWKV-based language tasks, as well as VRWKV (Duan et al., 2024) for RWKV-based vision tasks, As shown in Figure 1, RWKVQuant quantizes weights into about 3-bit and achieves superior accuracy compared to the individual utilization of SQ and VQ. Additionally, RWKVQuan demonstrates remarkable efficiency. For in- stance, it can quantize RWKV-6- 14B with less than 1 1% accuracy loss, 2.83 memory saving, and 2.14x speed up. Lastly, our contributions can be concluded as follows.
1982).
Family
Model
8 Clusters
16 Clusters
6-7B
2.01
0.78
RWKV
6-14B
1.98
0.78
2-7B
0.96
0.65
LLaMA
2-14B
0.89
0.64
We reveal that both smooth- and rotation- -aaed PTQ methods are not well-suitable for RWKV, primarily due to the unavoidable runtime overhead. Further, cluster- based PTQ methods suffer severe accuracy drop, owing to the larger amount of uniformly distributed weights.
applying QuaRot (Ashkboos et al., 2024) (belongs to SQ) to RWKV-7 series models increases the overall FLOP by more than 99%, and applying VPTQ (Liu et al., 2024a) (be- longs to VQ) to RWKV-6-7B model leads to more than 16% accuracy decline.
We propose RWKVQuant, which enhances VQ by par- tially adopting compensation- based SQ methods. introduces a coarse-to- proxy to guide the hybrid strategy. It further enhances the VQ for the unique element-wist multiplication modules in RWKV.
In depth, we investigate and identify two primary limitations inherent in the properties of RWKV. ! Non-linear opera- fors hinder the parameter-fusion of both smooth- and rotation-based methods Typically, these SQ approaches fuse the introduced parameters, i.e., smoothing vectors and orthogonal matrices, into neighbored normalization layers and linear layers of T-I.LMs. However, the RWKV structure employs several non-linear operators along the fusion path, including token-shift, Sigmoid function, and exponential function. These modules can block the linear fusion process, inevitably leading to additional runtime overhead. @ The larger amount of uniformly distributed weights poses challenges for cluster-based quantization. While such VQ methods benefit from distinctly categorized distribution, RWKV tends have more uniform weights compared to T-LLMs (detailed in Section 4.4), which complicates the clustering process as shown in Table 1.
RWKVQuant can effectively and efficiently quantize weights into about 3-bit and outperforms both SQ and VQ methods as shown in Figure 1.
To the best of our knowledge, RWKVQuant is the first comprehensive PTQ framework for the RWKV family. As a pioneering study, we will publish the code in the hope of promoting further research and facilitating advancements in this field.
2. Preliminaries
2.1. RWKY Structure and Models
Referring to Figure 2, RWKV structure contains two key modules, including Time Mixing and Channel Mixing (de- tailed in A.1). With the previous word æ;-1, the current word x can be derived by token-shift operator:
To this end, we propose RWKVQuant, an effective and effi- cient post-training quantization (PTQ) framework tailored for RWKV models. Our core insight is to enhance VQ by partially applying the classic compensation- based SQ methods like GPTQ (Frantar et al., 2022), which are more suitable for uniformly distributed weights. Specifically, we propose coarse-to-fine proxy to optimize the hybrid strat- egy. (1) The coarse- graised proxy is established on the basis of Information Entropy (IE) (Shannon, 1948), which evaluate the overall uniformity. For non-uniform weights, VQ is directly applied. (2) For uniform weights, we fur- ther introduce a fine-grained proxy, computed by weighted high-order central moments, detect local outliers. VQ is applied when outliers emerge; otherwise, SQ is applied. In addition to the hybrid, we further optimize VQ for the unique eiement-wise multiplication operator of RWKV.
2 = concat :,),0),
(1)
where 0 denotes an all-zero vector. RWKV models make use of Time Mixing for seizing the relationship among tokens and utilize Channel Mixing to probe the dimensions within the hidden layer that are relevant to individual tokens.
Compared to T-LLMs, these modifications enables RWKV to decrease substantially the computational overhead and memory demands while effectively retaining the capacity to model long-term dependencies. Thereby, the RWKV family has already manifested its potential in a diverse array of real-world applications (Li et al., 2024), includ- ing QQ (Cryscan, 2023), WeChat (MrTom34, 2023; Le- oLi42258, 2024), and Telegram (spion, 2023). For natural
Experiments show that RWKVQuamt outperforms existing methods across various tasks on different RWKV model
2

---

NWKKQuuant Proxy-Guided Hybrid Scalar-Vector Quantization for RWKV Models
SQ is widely-adopted by most of the PTQ frameworks (Yang al., 2024). For instance, the classic compensation- based GPTQ (Frantar et al., 2022) to 3-4 bit with slight accuracy drop based on approximate second-order information. To address outliers, AWQ (Lin et al., 2023), SmoothQuant (Xiao et al., 2022), and Omni- Quant (Shao et al., 2023) explore the scheme of smoothing by detecting the importance of different activation channels. Recent works (e.g., Quarot (Ashkboos al., 2024), Spin- Quant (Liu al., 2024b), and OSTQuant (ost, 2024)) further suppress outliers by rotating the variables to be quantized with orthogonal matrices.
Nonlinear Operator
Quantitative Operator
FP Operator
Softmax
quantize weights
Layernorm
relu
W
Add
Mixing
Token Shift
out
Layernorm
Nx
Add
exp
WKV
Vector Quantization (VQ) VQ quantizes several vectors into a finite subset, which commonly referred to as a codebook C (Gersho, 1979). Typically it has shape (2 d), where & is the bits of the index and d is the vector dimension. Given tensor with shape (m, n) to be quantized, VQ first transforms it into x' with dimensions (m n//d, d). Second, for each d-dimensional vector in ø', VQ replaces it with the k-bit index of the nearest vector from the codebook. For instance, if we use the Euclidean distance (calculated by the Frobenius normalization || (I ) measure similarities. the quantization process can be expressed as:
Mixing
W
W
W3
W.
W
Token Shift
Add
Layernrrm
W
W
W
W. W.
Input
Embedding
Figure 2: Model Structure of RWKV-7. It contains several blocks and each block has two key modules: Time Mixing and Channel Mixing.
Q(x') {argmin | C5llp p 1, 1,.., m=n//d). (3)
jE2
Compared to SQ, this scheme takes the advantage main- taining the shape of the source distribution, especially under lower bit-width. For example, VPTQ (Liu et al., 2024a) and GPTVQ (van Baalen et al., 2024) combine VQ with GPTQ, achieving advanced performances under 2~3 bits. To obtain the codebook, they cluster the source vectors by K-Means Algorithm (Lloyd, 1982) and Expectation-Marxe -Maximizztiinn Algorithm (Moon, 1996), respectively. AQLM (Egiazar- ian et al., 2024) further utilizes layer-wise training for the codebook obtain optimal accuracy.
language understanding tasks, RWKV-v6 (Peng al., 2024) has achieved remarkable advancements in accuracy, attain- ing comparable performance to those of larger models like SomILM and Qwen (Chen et al., 2024). For vision tasks, BSBP-RWKV (Zhou & Chen, 2024) excels in the domain of image segmentation.
2.2. Post Training Quantization (PTQ)
PTQ serves as a potent strategy for model compression. By converting the high- eigh-precision variables of pre-trained mod- els into low-bit integers, achieves reduction in memory usage and acceleration of inference speed. Typically, PTQ can be divided into two main approaches: SQ and VQ.
3. Method
3.1. Coarse-to-fine Proxy for Hybrid Quantization
Hybrid of SQ and VQ Given inputs æ, weights 0, number of weights M, and the model f(.), the optimization goal is to minimize the expectation [[[] of the Mean Square Error (MSE) of the model output:
Scalar Quantization (SQ) SQ maps the original data to the quantized range by scaling factor, and subsequently rounds floating points to the For a tensor æ to be quantized, it can be uniformly quantized b-bits as follows (Jacob et al., 2018):
integers.
argmin E [[fo(x)) - for (x)
((x) = clamp( x 1 + 2 1),
(2)
t. (dm {00 m = 2,,
M}
(4)
Ø' SQ(em)) + (1-mm)) ( 10, E 6, m 1,2
where Q(-) represents the quantization function, = (max(x) min(x))/2b - 1 the scale factor, 2 = min(x)/s is the zero point, 1.1 denotes the rounding- to-nearest operator, and clamp is the clipping function.
Here, & represents the collection of options for SQ and VQ. Although the optimal solution of Equation can be found by
3

---

RWKVQuant: Proxy-Guided Hybrid Scalar- Quantization for RWKV Models
the exhaustive algorithm, its complexity increases exponen- tially with the number of weights, i.e., O Considering the computational cost, we construct an effective proxy by evaluating the uniformity and outliers of each weight from both coarse- and fine-grained perspectives, whose complex- ity decreases to (M).
system, small amount of local outliers does not signifi- cantly effect Pa. However, in case of SQ, the accuracy is highly dependent to the data scale. Such outliers can cause more minimal values to be mapped to the same integer, thus increasing the rounding error. For instance, Figure 3(b) and Figure 3(c) have close Pa values, while the former contains obvious outliers and is more accurate under VQ.
Coarse-grained Proxy Information Entropy (IE) (Shan- non, 1948) one of the most common approaches to eval- uate uniformity. However, it measures the probability dis- tribution, rather than the original data that concerned by quantization. To take advantage of its effectiveness, we perform a series of transformations on the model weights.
Fine-grained Proxy To mitigate the issue that Pa is not sensitive enough local outliers of relatively uniform data, we further introduce fine- ane-grrined proxy. Specifically, we perform the Taylor expansion (Taylor, 1717) to Equation 9 to evaluate the minor disturbances ö around G'.
Given a weight W E Rocxic it is first flattened, then sorted in ascending order to formulate W' E goeric, Subsequently, the intervals G (( of all adjacent positions in W can be calculated by:
Step 1 The gap between G' and G' can be written as:
(ō: = 6 -6 - n 1.2,..., n). (10)
1
According to Equation 6, it should be satisfied that:
G [ [ -1].
(5)
For the clarity of expression, the term (oc- ic) 1 is denoted by the symbol n in the following contents. Next, G is transformed to G' by:
-
36:-2 G 1 =0
n
(11)
71
Step 2 The Taylor expansion can be formulated as:
G. G'= E.G. G.
ii 1,2,,,....,
(6)
Pa Pa(G') +
(kl) (R!)-1 Pp: P: (G')) + t((8k)
Considering that G' satisfies that Σn G. = 1, it can be treated as discrete probability distribution. Consequently, its IE (denoted by H) can be obtained by:
(12)
K
P
Σ (k!)
8 +
UG k
G
H(G') - Σ G G₁,
Taking the Fuler's number the base of the log function in Equation 9, the k-th order partial derivative of P. with respect to G¹, can be expressed as:
According to the property of IE, Equation 7 measures the concentration of G'. Since G' are the intervals, its concen- tration can equivalently reflect the uniformity of the original weight W.
gh Pc aG* = fInG +1
k ==1 1 k 2
(13)
Assuming an absolutely uniform weight W with fixed in- tervals, it can be transformed to G' following the above process, which finally should be:
Step 3 Taking Equation 11 and 13 into consideration, Equa- tion 12 can be transformed into:
K
(-1) k-1
P.(G')
Σ 0 )
(14)
G = (G',
i 112,,,,..,. n..
(8)
2= k(k 1)
Step 4 Omitting the term o(8*), ), Equation 14 can be refor- mulated as:
Owing to the property of IE, only if W = W does Equa- tion 7 take the maximum value. Finally, the coarse-grained proxy Pa. can be obtained by computing the gap between the IE of G' and G::
(C RJ (82. sK| O [02 ..., uk] o [M2, .... Mk],
n
Em 5%
where 8k = (-1)*, Uk k(k - Mr
= ((G')=H(G')-11)
(9)
n
(15)
Here, "6' reppressnts element-wise multiplication. M₂ is the k-th order central moment of G', which is defined as:
By introducing a threshold T. non-uniform weights can have larger values of Pa, indicating the usage of VQ as shown in Figure 3(a). Since IE is measure of the entire
Ma = E (G' E E
(16)

---

RWKVQuant, Proxy-Guided Hybrid Scalar- Vector Quantization for RWKV Models
layer Inner
layer.S.ttt.inned
layor..attt.cy
0.3
100
0
3
200
400 300
100
03
400
300
-0.
100
39.33
Pc=2.65
5Qac:199.33
0.96
:
/P. 0,94
VQace :39.54
/
VQace 39.46 pp 79.98
VQace 39.14
P: 16.78
(a) Non-uniform weight
(b) Unifrom weight with outliers
(c) Unifrom weight without outliers
Figure 3: Zero-shot accuracy when applying different quantization methods to specific weights. For the weight in each sub-figure, SQace denotes the performance when SQ is applied, VQace denotes the performance when VQ is applied, while all other weights are quantized with VQ. Pa and Pj are coarse- and fine-grained proxy, while pe and f are their corresponding thresholds.
The central moment can serve as metric for assessing the local features of data. This is because the difference between outliers and other data points is magnified by the k-th power. For instance, when & 2, Equation 16 yields the variance, indicating the spread of the data. For k- = 3, the central moment corresponds to skewness, assessing the symmetry of the data. For i = 4, it represents kurtosis, revealing the data's long-tail characteristic.
the application of VQ. Notably, the fine one-gnained proxy is only utilized in condition that P.(G',) < Te-
3.2. Codebook Optimization for Element-wise Multiplication
Different from Transformer-bans LLMs, the element-wise multiplication "Q" betteeee the input and the weight fa is applied in all projection layers of the RWKV structure, as shown in Figure 2. In accordance with the proxy introduced in Section 3.1, VQ is expected to be applied to most of them. However, existing VQ methods are primarily tailored for matrix multiplication modules. We thereby propose to optimize the VQ codebook specifically for element-wise multiplication modules.
Step 5 Considering that GG' signifies the overall uni- formity, 8ą and 2 can be regarded the direction and the significance of the local feature Mr. Since only the magni- tudes of features are considered when it comes to outliers, the fine-grained proxy can be defined as:
K
Pj(
Σ
un Me],
(17)
Given weight μ E Rm*x. it is first transformed to jı' E ((m-n//d) xd, where d is the hidden dimension. Following the VQ process stated in Equation 3, it can be quantized into Qu4'). Typically, the quantization loss L can be written as:
where K is hyper-parameter. By introducing threshold Ty, outliers can be detected by larger values of Py. indicating the usage of VQ as shown in Figure 3(b).
L= || @ Op' X Deq(Q(n'))
Finally, our proxy solution of Equation can be obtained by the combination P. and py:
m.n//e //d d
(19)
Σ Σ (D))))
(G < To and ()( <Tf 0 ( ) < and (G 2 Tf (18)
where X is representative of the calibration activations, Deq the de-quantization process, and Aµı denotes the quantization error of the weight. To minimize Equation 19, larger activation value should correspond to a smaller Ajr', indicating the significance of this position. Thus, we employ the term X to direct the weighted K.Means algorithm in the generalization of codebooks.
3 = (Ø, =
or Pa (P))) 2 To
2 2= 1,2, 1.2,...MM) M).
where G¹ denotes the m-th weight after the transformation. Only if both the coarse-graines proxy and the fine-grained proxy are lower than their corresponding threshold will SQ be applied, as shown in Figure 3(c). Otherwise, the weight distribution is supposed to be generally uneven, or relatively uniform but with local outliers, which indicates
Due to the nature of element-wise multiplication, X must have the same shape as μı' which further introduces an issue of integrating batches of data. The most straightfor ward approach is to simply average samples. However,
5

---

RWKVQuant: Proxy-Guided Hybrid Scalar-Vector Quantization for RWKV Models
Table 2: Comparison of perplexity on LAMBADA and averaged accuracy on nine Zero-Shot tasks. For all methods except ours and floating-porns point, we report metrics under both bpw settings of 3.5 and 3.25.
RWKV7-0.IB
RWKV7-0.5B
RWKV7-1.47B
RWKV6-IB
RWKV6-3B
RWKV6-7B
RWKV6-14B
Bpw.
Method
0-shot
LambA.
O-
LambA.
0-shot"
LambA.
0-shot" LambA.
O-shot" LambA.
0-shot" LambA.
O-shor LambA.
Avgi(t)
(4)
(Avg.(t)
(4)
Avg.(())
())
Avg.(t)
(1)
Avg.(t)
())
Avg.(t))
(U)
Avg.(1)
(1)
16
PloatingPoint
43.02
14.21
48.67
7.21
55.08
4.80
54.39
4.60
58.32
3.83
61.69
3.21
63.65
3.02
RTN
36.22
152.22
39.99
57.11
45.46
11.43
49.84
6.39
54.17
4.71
58.34
3.87
61.16
3.34
GPTQ
37.92
63.54
41.16
23.29
51.15
7.93
50.55
6,43
53.94
4.88
59.28
3.72
60.18
3,43
AWQ
36.20
132.06
! 68.92
5.92
43.62
15.27
44.41
17.97
47.24
11.97
49.48
8,33
49.35
8.18
3.25
QuaRot
34,53
243.99
40.17
76.89
50.81
9.39
45.02
29.38
48.24
22.67
$4.29
8.81
54.76
14.05
kMeans
38.21
87.06
44.59
20.19
52.77
6.57
47.02
15.93
53.06
8.27
S5.72
4.69
61.40
4.61
GPTVQ
40.25
23.75
1 43.64
14,15
52.06
5.54
49.86
6,11
54.23
4,31
58.57
3.49
59.63
3.15
VPTQ
35.78
128.59
40.14
30.63
45.83
11.13
43.89
14.67
48.22
7.77
53.06
4.75
57.87
3.62
RTN
38.08
81.14
43.02
25.09
51.74
7.89
51.76
5.83
54.42
4.50
59.20
3.59
60.84
3.31
GPTQ
39.33
40.16
45.73
13.07
52.28
6.55
51.23
5.86
54.89
4.54
60.07
3.68
61.14
3.29
AWQ
38.31
55.72
42.40
16.98
44.61
10.71
51.20
10.07
48.40
8.77
50.50
7.07
50.86
6.06
3.5
QuaRot
37.26
126.19
40.84
40.38
51.98
7.94
47.00
16.29
51.01
16.99
56.95
6.44
56.33
10.63
kMeans
39.55
36.26
43.07
17.05
52.77
6.57
50.03
8.24
54.99
6.28
59.51
3.96
61.15
3.84
GPTVQ
40.10
25.82
44.13
10.88
52.13
5.51
50.29
5.74
55.12
4.12
59.70
3.30
59.76
3.34
VPTQ
37.07
74.70
41.06
25.03
47.38
9.52
43.82
14.74
48.86
8.62
52.95
4.47
57.93
3.75
3.275
Ours
41.10
18.41
46.01
9.39
52,40
5.24
51.69
5.29
55.79
3,88
60.19
3,23
62.69
2.89
this method is not sufficiently effective because it is highly susceptible to the influence of small number of outliers. Given that the activations of RWKV models typically fol- lows an approximately normal distribution, we introduce a percentile- based clipping operation to limit the range of samples prior to averaging, thereby alleviating this issue.
Baselines and Implementation Details. In addition to com- paring with SQ methods such as RTN, GPTQ (Frantar et al., 2022), AWQ (Lin et al., 2023), and Qurot (Ashkboos et al., 2024), we also benchmark our approach against VQ meth- ods like K-Means, GPTVQ (van Baalen et al., 2024) and vPTQ (Liu al., 2024a) for weight-only quantization. To ensure faimesss, report the performance of each method under two configurations, where the average number of bits per weight (bpw) is set to 3,25 and 3.5. For SQ methods, we take the scale size into account when calculating the bpw. To achieve 3.25 and 3.5 bits per weight, we set the group size for quantization to 32 and 64 respectively. For VQ methods, we consider not only the bit size occupied by the quantized weights but also the bit size required for storing the codebook to achieve the corresponding bpw. In our method, we dynamically set Ta and Ty according to dif- ferent models, ensuring that SQ with a bpw of 3.25 is used in nine-tenths of the layers, while VQ with a bpw of 3.5 is used in one-tenth. For example, in RWKV7, Te set to 1.54, while 7y is set to 30. For both vision and language tasks, we select 128 samples from the corresponding test datasets for calibration.
4. Experiments
4.1. Experimental Settings
Models and Datasets. We evaluate the RWKVQuant frame- work on RWKV6 (Peng et al., 2024), RWKV7, and VR- WKV models (Duan et al., 2024). For vision tasks, we utilize ImageNet (Deng et al.. 2009) for image classifi- cation, Coco (Lin et al., 2014) for object detection, and ADE20K (Zhou et al., 2019) for segmentation. Aligned with the accuracy evaluation methods used in the VRWKV experiments, we report Top-1 Accuracy for classification tasks, Box Average Precision (AP) for detection tasks, and Mean Intersection over Union (MIoU) for segmentation tasks. For language tasks, consistent with the VW pa- per, we report the perplexity (PPL) on the Lambada dataset. We also evaluate the models on up to nine zero-shot tasks using the AM-evaluation--hmmess (version 0.4.4), including LAMBADA(OpenAI) (Radford et al., 2019), HEADQA (EN) (Rogers et al., 2023), HellaSwag (Zellers et al., 2019), OpenBookQA (OBQA) (Mihaylov et al., 2018), PIQA (Bisk et al., 2020), SCIQ (Pedersen et al., 2020), Winogrande (Sakaguchi et al., 2021), ARC-Challenge and ARC-Easy (Boratko et al., 2018).
4.2. Overall Results
Performance Comparison on Language Tasks. As shown in Table 2, on language tasks our method consistently out- performs other approaches across nearly all models. Com- pared to methods with a bpw of 3.25, regardless of whether they based on SQ or VQ, our method demonstrates signifi-
6

---

RWKVQuant: Proxy-Guided Hybrid Scalar- Quantization for RWKV Models
Table 3: Comparative results under different quantization settings for Vision RWKV models.
Table 5: Ablation study on the impact of hybrid quantiza- tion on LAMBADA PPL and zero-shot" score for language
RWKV models.
RWKV-T
RWKV-S
Bpw.
Method
GPTQ
GPTVQ
Ours
Cls. Det.
Seg.
Cls.
Det.
Seg.
16
FloatingPoint
75.10 41.70 43.3 80.10 44.8 47.2
Model
O-shot
LambA
O-shot
O-sho, LambA.
Avg.(1)
(L)
Avg.(t)
(1)
Avg.(t)
(L)
GPTQ
69.74 39.85 41.20 20 78.30 43.37 45.50
RWKV7-0.1P IB
39.33
40.16
38.49
55.30
40.69
24.71
AWQ
68.50 39.03 38.88
78.00
42.90 42.88
3.5
RWKV7-0.5B
45.36
13.07
43.85
20.16
45.03
13.49
GPTVQ
70.31 40.14 41.65
78.65
44.03 45.00
RWKV7-147B
52.28
6.55
51.31
6.85
52.23
6,54
VPTQ
67.21 39.02 40.14
76.40
42.01 43.54
RWKV6-IB
51.20
5.86
49.70
5.52
51.44
5.32
3.275
Ours
70.41 40.22 41.70 78.74 43.95 46.09
RWKV6-3B
55.24
4.54
54.86
4.41
55,40
3.97
RWKV6-7B
59.20
3.59
48.29
3.41
60.18
3.21
cant improvements in both PPI. and accuracy zero-shot tasks. Compared to methods with a bpw of 3.5, our method consistently achieves lower PPL. Except for slightly lower accuracy on RWKV7-0.5B and RWKV6-1B with certain methods, achieves the highest accuracy across all other be observed that the smallest O.IB model, the PPL of other methods increases by at least 10 points, whereas our method results in an increase of only 4.2 points. On larger models such as RWKV6-7B and RWKV7-14B, our method results in almost no increase in PPL, while the accuracy decreases by less than point.
RWKV6-14B
61.14
3.29
59.86
3.31
62.03
2.89
the model's ' inference speed, as shown in Table 4.
4.3. Ablation Study
models.
Hybrid Quantization. We conduct series of ablation stud- ies on the hybrid quantization method proposed in Section 1.1, comparing its performance on the RWKV model with that of employing single quantization methods. For fairesss, the weights of all multiplication operations are quantized using the RTN method. Our method leverages the proposed coarse-grained and fine-grained proxy to hybridize GPTQ and GPTVQ. While GPTQ and GPTVQ use bpw of 3.5, our method achieves bpw of 3.275 by applying GPTVQ (bpw 3.5) to one-tenth of the layers and GPTQ (bpw 3.25) to the remaining nine-tenths. The ablation study results in Table highlight the effectiveness of the hybrid quan- tization method. In nearly all RWKV models, the hybrid method achieves better metrics compared to both GPTQ and GPTVQ.
Performance Comparison on Vision Tasks. Table 3 presents the results of the quantized RWKV models applied to various vision tasks, including classification, detection and segmentation. Our method achieves the highest scores in both segmentation and classification tasks. For detection tasks, although the precision of RWKV-S is not the highest, it is very close to the best-performing method.
Memory Occupancy and Computational Cost. Our method incurs only negligible loss in 3.275-bpw quanti- zation, making 3.275-bpw inference feasible. As described in the section 1, models based on the RWKV architecture differ from those built on GPT or LLaMA architectures. Whether in the pre-fill or decoder stage, RWKV models exhibit a lower camputett--memoy-aaee quently, quantizing the weights lower bit-widths can sig- nificantly reduce memory access time, thereby accelerating
Proxy Strategy. Table 5 shows that our hybrid quantization improves accuracy but still lags behind floating-poin preci- sion. We then apply the proxy described in Section 3.1, com- bining coarse-grained and fine-grained proxies to determine the quantization method for each layer. The ablation results,
ratio. Conse-
Table 6: Ablation study on the impact of different proxies for hybrid quantization in language RWKV models.
Table 4: Comparison of generation speed and memory usage before and after 3.275-bpw quantization on RWKV6 models. All tests were conducted on an NVIDIA A6000 GPU.
RWK7-0.1B
RWK7-0.5B
RWK7-1.478
Method
0-shot LambA.
0-shot" LambA.
0-shot LambA. Avg.(t)
Avg.(t)
(4)
Avg.(t)
(4)
(L)
Variance
40.67
20.80
42.51
9.74
51.29
5,90
speed (tokens/sec)
Memory (GB)
CV
39.09
22.36
40.92
10.10
51.58
5.79
Model Size
Range
39.92
23.78
40.24
10.41
$1.37
5.82
FP
Quantized Speed up
FP
Quantized Mem. saving
MAD
38.97
22.65
42.33
10.02
51.95
6.04
3B
32.95
51.29
1.55x
5.88
1.65
3.56x
MSE
37.99
28.56
42.60
10.22
51.05
6.87
7B
30.75
62.42
2.03x
13.91
4.25
3.27x
IE
41.01
20.03
45.12
9.67
52,12
5.31
14B
16.02
34.32
2.14x
26.07
9.21
2.83x
Ours
41.04
19.70
45.54
9.55
52.32
5.24
7

---

RWKVQuant: Proxy-Guided Hybrid Scalar-Vect Quantization for RWKV Models
Table 7: Ablation study on the impact of codebook opti-
WWSY Modes
mization for wlement-wise multipiication.
LLAMA Model
0.99
5756
56%
wo.
w.
90%
Model
0-shot
Lambda
O-shot
Lambda
Avg.(t)
(L)
Atg.(t)
())
Proportion
RWKV7-0.1B
40.69
24.71
41.09
18.41
RWKV7-0.5B
45.03
13.49
46.01
9.39
30%
RWKV7-1.47B
52.24
6.54
52.41
5.24
RWKV6-IB
51.45
5.32
51.69
5.29
RWKV6-3B
55.48
3.97
55.79
3.88
10%
8%
10%
RWKV6-7B
60.18
3.21
60.19
3.23
RWKV6-14B
62.03
2.89
62.69
2.89
0%
LLAM6-148
comparing the use of different proxies such Variance, Coefficient of Variation (CV) (Abdi, 2010), Range (Gum- bel, 1947), Mean Absolute Deviation (MAD) (Konno & Koshizuka, 2005), Mean Squared Error(MSSE, and IE, are presented in Table 6. Notably, MSE denotes making se- lections between SQ and VQ by directly comparing their MSE of each weight. The other metrics are used in the same manner as described in method, focusing the transformed weights G". Intuitively, the MSE method is the local optimum for each weight. However, our coarse-to-fine proxy attains the best results across all three models from the global perspective.
Figure 5: Comparison of SQ proportion between RWKV and LLaMA Models.
4.4. More Uniform Weights in RWKV
Table in Section presents the average relative clustering loss of weights using K-Means methods (Lloyd, 1982) for the RWKV family and the LLaMA family respectively. In depth, we conduct experiments leveraging the proposed coarse-to-fing proxy in Section 3.1 to investigate the usage proportions of SQ and VQ. Under the settings of T. 1.5 and Ty 50, Figure 5 shows that approximately 60% of the layers in the RWKV family are categorized suitable for scaler quantization, whereas the proportion is only about 10% for the LLaMA family. This further demonstrates that the RWKV models have a significantly higher number of uniform weights.
Codebook Optimization. We conduct ablation experiments on the codebook optimization for the element-wise multi- plication proposed in Section 3.2, across all RWKV models. The results are presented in Table 7. It can be observed that using the codebook optimization for the element- wise multi- plication operator generates better accuracy across all mod- els compared to not applying the optimization. Specifically, we also visualize the effectiveness of the clipping- per- centile technique within this codebook optimization. From Figure 4, it can be clearly observed that the input activation approximately follows the normal distribution. However, the outliers make the representative feature to leave far from the center point, thereby decreasing the overall performance. By clipping these outliers, a more close-to-cente, feature can be obtained, thus enhancing the calibration process.
5. Conclusion
In this paper, we focus on introducing the quantization tech- niques into the realm of RWKV models. Our investigation reveals that applying SQ or VQ individually may not be optimal for RWKV. We have subsequently identified that enhancing VQ with conventional compensation basedd SQ holds great promise. To this end, we propose NWKKQuant comprehensive post training quantization framework es- pecially designed for RWKV models. The core idea is to design optimal strategy that indicates the choice between SQ and VQ for each weight. Specifically, we propose a guidance that employs a coarse soasee-raiied proxy to evaluate uniformity and a fine-grained proxy to identify outliers. We also optimize the codebook generation for element-wise multiplication modules, which are unique to the RWKV models. Our proposed RWKVQuant advances in accuracy for both RWKV-based vision and language tasks compared to existing methods, making RWKV models more practical for deployment in resource-constraingg environments. As pioneering study on quantization within the RWKV family, we will publish the code in the hope of promoting further research and facilitating advancements in this field.
layar.Oxxmaa mul
700
A06
Mean:-0.107
800
Meancerpet:0.000
Fequeecy
500
LambA.Acc:4.dd
Clip_valus Min Max Distribution
400
J00
200
100
Value
Figure 4: Effectiveness of clipping for batch integration.
8

---

RWKVQuant: Proxy-Guided Hybrid Scalar- Quantization for RWKV Models
Impact Statement
Elman, L. Finding structure in time. Cognitive science, 14(2):19-211, 1990.
This paper aims to promote the application of the RWKV family, mainly focused on the post training quantization methods. By introducing RWKVQuant, our approach enables the deployment of RWKV models resource- constrained devices.
Frantar, E., Ashkboos. S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arXiv:221001732.. 2022.
Gersho, A. Asymptotically optimal block quantization. IEEE Transactions on information theory, 25(4):373-388 1979.
References
Ostquant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting. OpenReview, 2024. URL https:
Gumbel, J. The Distribution of the Range. The Annals of Mathematical Statistics, 18(3):384 412, 1947. doi: 10. 10.1221440 ///14////m/117//0000 URL / 1114/a/m//117773088
Abdi, H. Coefficient of variation. Encyclopedia of research design, 1(5):169-171. 171, 2010.
Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integer- arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference rotated llms. arXiv preprint arXiv:2404. 00456, 2024.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense natural language. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Konno, and Koshizuka, Mean-absolute deviation model. lie Transactions, 37(10):893-900, 2005.
Langley, Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (CCM20000) pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Bo, P. Blinkdl/rwky-Imm 0.01, August 2021. URL https: //doi. org/10. 0 5281 51 96577.
Boratko, M., Padigela, H., Mikkilineni, D., Yuvraj. P., Das, R., McCallum, A., Chang, M., Fokoue-Nkoutche, A., Kapanipathi, P., Mattei, N., et al. A systematic classifica- tion of knowledge, reasoning, and context within the are dataset. arXiv preprint wxXvv18066000358 2018.
LeoLin4258. Infofusion. Github, 2024. URL https:// github.c httpc///LLeLLii22////ffiiee
Li, Z., Xia, T., Chang, Y., and Wu, Y. A survey of rwkv. arXiv preprint arXiv:2412.1488/ 2024.
Chen, Z., Li, C., Xie, X., and Dube, P. Onlysportslm: Optimizing sports- pporis-domain language models with sota performance under billion parameters. arXiv preprint AXXi::2409002866 2024.
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., and Han, S. Awq: Activation-aware weight quantization for 11m compression and acceleration. arXiv preprint arXiv:2306.0099 00978, 2023.
Cryscan. eloise. Github, 2023. URL https:://gttthub. com//cryscan/eloise.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Deng, J., Dong, W., Socher, Li, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, leee, 2009.
Duan, Y., Wang, W., Chen, Z., Zhu, X., Lu, L., Lu, T., Qiao, Y., Li, H., Dai, J., and Wang. W. Vision-rwkv: Efficient and scalable visual perception with rwkv-like architectures. arXiv preprint arXiv:2403.02300 2024.
Liu, Y., Wen, J., Wang, Y., Ye, S., Zhang, L. L., Cao, T., Li, C., and Yang, M. Vptq: Extreme low-bit vector post- training quantization for large language models. arXiv preprint arXiv:2409. 2024a.
Egiazarian, V., Panferov, A., Kuznedelev, D., Frantar, E., Babenko, A., and Alistarh, D. Extreme compression of large language models via additive quantization. arXiv preprint Arvivv2401. 06/ 18, 2024.
Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Kr- ishnamoorthi, R., Chandra, V., Tian, Y., and Blankevoort, T. Spinquant- llm quantization with learned rotations. arXiv preprint arXiv:2405. /6406, 2024b.

---

RWKVQuant: Proxy-Guided Hybrid Scalar- Vector Quantization for RWKV Models
Lloyd, S. Least squares quantization in pem. JEEE Trans- actions Information Theory, 28(2) 1982.
Taylor, B. Methodus incrementorum directa & inversa. Inny, 1717.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint TrXiv:180.0027350 2018.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama Open foundation and fine- tuned chat models. arXiv preprint MXii:2337.009886 2023.
Moon, K. The expectation-maaiiizatioo algorithm. IEEE Signal processing magazine, 13(6):47-60, 1996.
van Baalen, M., Kuzmin, A., Nagel, M., Couperus, P., Bas- toul, C., Mahurin, E., Blankevoort, T., and Whatmough, P. Gptvq: The blessing of dimensionality for Ilm quanti- zation. arXiv preprint arXiv:2402.153199 2024.
MrTom34,
Chatwwk-inn-eecht--eeiiio- Github,
2023. URL https://github. httpl//co////////// MelaRwvv-i--wecat--eeee
Vaswani, A. Attention all you need. Advances Neural
Pedersen, C., Otokiak, M., Koonoo, I., Milton, J., Mak- tar, E., Anaviapik, A., Milton, M., Porter, G., Scott, A., Newman, C., ct al. Sciq: an invitation and recommenda- tions to combine science and inuit gaujimajatuqanii for meaningful engagement of inuit communities in research.
Information Processing Systems, 2017.
Xiao, G., Lin, J., Seznec, M., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211 104388, 2022.
Arctic Science, 6(3):326-338) 339, 2020.
Yang, D., He, N., Hu, X., Yuan, Z., Yu, J., Xu, C., and Jiang, Z. Post-training quantization for re-parameterization via coarse & fine weight splitting. Journal of Systems Archi- tecture, 147:103065, 2024.
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Grella, M., et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13044. 2023.
Yuan, Z., Shang, Y., Zhou, Y., Dong, Z., Xue, C., Wu, B., Li, Z., Gu, Q. Lee, Y. J., Yan, Y., Chen, B., Sun, G., and Keutzer, K. Llm inference unveiled: Survey and roofline model insights, 2024.
Peng, B., Goldstein, D., Anthony, Q., Albalak, A., AI- caide, E., Biderman, S., Cheah, E., Du, X., Ferdinan, T., Hou, H., al. Eagle and finch: Rwkv with matrix- valued states and dynamic recurrence. arXiv preprint arXiv:2404.05892. 2024.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint 13XXv:1905007830 2019.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Zhou, B., Zhao, H., Puig. X., Xiao, T., Fidler, S., Barriuso, A., and Torralba, A. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 22:330--321, 2019.
Rogers, A., Gardner, M., and Augenstein, I. Qa dataset explosion: A taxonomy of nlp resources for question answering and reading comprehension. ACM Computing Surveys, 55(10):1-45,200)
Zhou, X. and Chen, T. Bsbp-rwkv: Background suppression with boundary preservation for efficient medical image segmentation. In Proceedings of the 32nd ACM Interna- tional Conference on Multimedia, pp. 4938 4946, 2024.
Sakaguchi, K., Bras, R. L.., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Shannon, C. E. mathematical theory of communication. The Bell system technical journal, 2793):379-423, 1948.
Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L., Li, Z., Zhang, K., Gao, P., Qiao, Y., and Luo, P. Omniquant: Ommidirectionally calibrated quantization for large lan- guage models. CoRR, abs/2308. 13137, 2023.
spion. notgpt. Github, 2023. URL https: htt/g/gtthhb..c. com/ spion/notgpt.
10

---

RWKVQuant: Proxy-Guided Hybrid Scalar- Quantization for RWKV Models
A. Appendix
A.I. Structure of Time- and Channel-Mixing
The RWKV model, similar to Transformer networks, composed of multiple identical blocks, cach containing a Time Mixing component. The Time Mixing process can be written as:
Tt W, (Er + æz + - (1-1,) ④ xx-1),,
(20)
k; W: (n ① 0z (1 p4)) ② 02-1))
(21)
6 W. (Mo ① Ft + p.) 3 x1-))
(22)
1-1 21=1 f(i) @ v; exp(u + kt) @
wkn;
Σ f(i) + exp(u k,)
(23)
wheref (i)) exp( (t - i)w + kj),
0, W. (o(r)) @ wkv.).
(24)
Here, the symbol '0 represeents element-win wise multiplication, while the symbol 'stands for matrix multiplication. Both W and y are parameters. the context of RWKV, the terms rt, kG, and o bear an analogy to the Q K, and components found in the attention mechanism of Transformers. Notably, the input x in RWKV is simply the embedding of the current token. Rather, it signifies the weighted sum of the embedding of the current token and that of the previous token. Subsequently, the Channel Mixing module performs:
r/ W; (u, @ Hr + (1 (1-1²)= @ xx-1),
(25)
K W (Hk @ + (1- - p4) @ x111)).
(26)
of o(r)) @ (W; max(k', 0)
(27)
A.2. RWKV Weight Distribution
Figue 6 shows layers with relatively uniform weight distributions in the RWKV7-0.1B model, which are classified as layers that should use SQ based on our proposed coarse-to-fine proxy. In contrast, Figure 7 illustrates layers with uneven weight distributions, which are typically classified as layers that should use VQ. Furthermore, although the weights in Figure 8 appear generally uniform, the presence of local unevenness still leads to their classification as layers that require VQ.
Ityen: att.key
byyen ww..tt..uttt output
layerD at: receptannee
ayyrr..ffn.kkg
lyyan&.tt.veuu
-03
100 / <3s 933 231
2k: 88: de 160 120
201 128 851 we me
tie sec
42c 150
Ven Dan 1100 Jo1n 2546 1031
Figure 6: Unifrom weights without outliers in RWKV7-0.1 IB different layers.
layer
layer
layer
120 620 540 600 300 200
120 C0
1C0
Figure 7: Non-uniform weights RWKV7-0.1D IB different layers.

---

RWKV RWKVQuant: Proxy-Guided Hybrid Scalar- Vector Quantization for RWKV Models
byyerO.att.iimee
lytr.S.attllncar
Sayer
8-
27
Figure 8: Unifrom weights with outliers in RWKV7-0.18 different layers.
A.3. Compute-to-memory Ratio
The figure compares the compuut-to-Memovy Ratio (FLOPs per byte) across various models and highlights that RWKV consistently exhibits the lowest ratio, indicating that its operations rely more memory access rather than intensive computations, compared to models like GPT-3 and LLAMA. This characteristic makes RWKV particularly well-suited for acceleration through weight quantization, as its lower computational demands relative to memory usage allow for more significant gains in inference speed, especially when optimizing for resource-constrainge environments.
Compute-to to-tto -Memory Ratio (FLOPs/Bytes)
RWKV
3.5e+04 04 2. 2.7e+04
1.70 2.0e+04 1.8 1.80+04
GPT3 prefill
10
LLAMA
LLAMA_prefil prefill
1. 1.
GPT3 decoder
decoder
10a
16,98
13,56
13.12
101
8 8.25
9.77
8.05
8.64
4. 88
0.8
0.9
0.9
1.0
1.0
1.0
0.17
0.43
1.5
14
0.35
2.7
6.7
13
175
65
Model size (B)
Figure 9: Compute-to-mmemorr--naio different models.
Table 8: Complete comparative results under different quantization settings for Vision RWKV models. For classification tasks, we report the Top-1 Accuracy on ImageNet. For detection tasks, the Box AP is evaluated on Coco, while for segmentation tasks, the mloU measured on ADE20K.
RWKV-T
RWKV-S
RWKV-B
RWKV-I
RWKV6-T
RWKV-L
RWKV6-T
Bpw.
Method
Cls.
Det.
Seg.
Cls.
Det.
Seg.
Cls.
Seg.
Det.
Seg.
Cls.
Det. Seg.
Cls. Det. Seg,
Cls. Det. Seg.
16
FloatingPoint
75.10 41.70 43.30
80.10
44.80 47.20 82.00 46.80 49.20
50.6 53.50 76.60
81.10
82.60
GPTQ
69.74 39.85 41.20
78.30
43.37 45.50 81.42 46.14
48.64
50.30 53.26
72.79
80.13
82,31
AWQ
68.50 39.03 38.88
78.00
42.90 42.88
81.15 45.70 48.55
50.19 53.18
71.46
79.74
82.09
3.5
GPTVQ
70.61 40.14 41.65
78.65
44.03 45.00
81.37 46.23 48.70
50.28 52.90
73.22
80.23
82.25
VPTQ
67.21 39.02 40.14
76.40
42.01 43.54
80.29
45.02
48.68
49.10 51.45
70.36
77.75
81.31
3.275
Ours
70.41 40.22 41.70
78. 74 43.85 46.09
81.58
46.40 48.49
50.31 52.95
73.13
80.24
82,32
12

---

RWKVQuant: Proxy-Guided Hybrid Scalar- Quantization for RWKV Models
A.4. Additional Results
We provide comprehensive presentation of our results across various datasets to complement the main paper. Specifically, the results include:
Complete comparison of the results under different quantization settings for Vision RWKV models.((Tble 8).
Complete comparison of the perplexity score on LAMBADA and averaged accuracy on zero-shot common sense reasoning tasks on RWKV7 (Tab 9) and RWKV6 (Table 10).
Validate the effectiveness of codebook optimization for element- multiplication. (Table 11).
Table 9: Complete comparison of perplexity LAMBADA and averaged accuracy nine Zero-Shot tasks. For all methods except ours
poin,, we report metrics under both bpw settings 3.5 and 3.25.
ARC-c
ARC-
HQA.
HellaS.
Lam.
OBQA
PIQA (7)
SCIQ (7)
WinoG. (1)
Avg. (1)
Wiki2
Model
Bpw.
Method
(1)
(1)
(1)
(1)
(1)
(1)
(4)
16
Full Precision
19.7
47.90
25.12
31.59
45.62
17.2
65.61
81.8
52.6
43.02
14.21
RTN
19.53
40.66
23.70
30.12
12.83
14.4
62.73
70.2
51.93
36.22
152,82
GPTQ
20.39
42.17
23.52
30.59
20.47
15.40
62.62
73.60
52.50
37.92
63.54
AWQ
20.13
41.16
22.72
29.88
15.20
25.44
61.15
76.70
52.40
38.31
132.06
QuaRot
19.96
39.94
22.46
29.38
12.28
13.80
60.93
58.80
53.19
34.53
243.99
3.25
KMeans
18.94
43.56
23.81
29.85
19.93
15.00
63.49
75.70
53.59
38.21
87.06
GPTVQ
18.94
43.93
22.75
30.04
38.06
16.40
63.36
77.00
51.77
40.25
23.75
VPTQ
18.94
39.60
21.84
28.59
17.50
13.20
59.90
71.90
50.51
35.78
128.59
RTN
19.36
43.47
24.50
30.17
18.13
16.20
64.09
74.40
52.48
38.08
81.14
GPTQ
18.68
44.06
24.14
30.69
26.74
17.80
63.76
74.30
53.82
39.33
40.16
RWKV-7 O.IB
AWQ
19.88
40.40
22.97
29.52
14.92
14.00
59.99
71.50
52.64
36.20
55.72
3.5
QuaRot
19.88
44.40
22.72
30.16
15.36
15.60
62.78
72.40
52.09
37.26
126.19
KMeaas
19.19
44.02
24.14
29.98
29.44
15.00
63.54
77.40
53.27
39.55
36.23
GPTVQ
19.11
42.88
22.68
30.25
38.35
15.80
63.11
74.40
54.30
40.10
25.82
vPTQ
18.51
40.74
22.42
28.82
23.17
13.80
61.31
72.90
51.93
37.07
74.70
3.275
Ours
19.73
42.17
24.14
30.02
42.25
17.00
63.10
79.10
52.30
41.10
18.41
16
Full Precision
23.80
57.11
28.55
38.28
57.85
20.80
69.31
86.6
55.72
48.67
7.21
RTN
21.84
45.28
23.66
32.64
26.45
T9.20
64.96
73.4
52.56
39.99
57.11
GPTQ
22.61
52.77
26.14
35.92
42.52
20.80
67.46
79.60
55.40
43.69
15.97
AWQ
19.62
43.09
22.93
32.64
39.34
17.00
62.57
79.20
54.14
41.16
23.29
QuaRot
22.18
48.69
24.76
33.85
20.22
19.00
65.01
73.80
54.06
40.17
76.89
KMeans
23.46
52.48
26.29
36.19
39.41
20.00
67.35
81.50
$4.69
44.59
20.19
GPTVQ
21.50
51.30
26.14
34.80
46.19
19.40
66.64
82.40
54.38
43.64
14.15
VPTQ
18.68
42.97
22.86
31.15
34.81
16.40
62.73
78.80
52.88
40.14
30.63
RIN
18.13
19.36
43.47
24.50
33.77
T6.20
64.09
74.40
52.488
38.08
81.14
GPTQ
22.26
53.11
25.41
36.51
45.78
21.40
67.51
83.10
56.51
45.73
13.07
RWKV-7 0.5B
AWQ
21.33
43.60
22.57
33.06
45.14
18.80
61.75
82.60
$2.80
42.40
16.98
3.5
QuaRot
28.33
20.13
46.38
24.54
32.30
18.80
65.23
80.40
51.46
40.84
40,38
KMeans
20.73
47.55
25.05
33.58
41.39
17.40
65.83
82.00
54.14
43.07
17.05
GPTVQ
21.92
52.18
25.38
34.90
52.05
17.80
67.51
81,60
53.82
44.13
10.88
vPTQ
19.79
44.73
23.12
31.62
37.78
16.40
63.98
78.90
53.27
41.06
25.03
3.275
Ours
22.61
53.11
25.41
34.45
53.39
19.60
66.53
84.20
54.80
46.01
9.39
16
Full Precision
31.65
65.40
32.12
46.55
66.97
26.00
72.68
90.00
64.40
55.08
4.84
RTN
21.69
53.44
25.65
35.81
46.63
17.90
65.39
85.14
57.55
45.46
11.43
GPTQ
29.01
59.93
28.99
42.74
55.52
23.60
70.34
87.30
62.98
51.15
7.93
AWQ
22.35
44.91
22.72
35.22
45.04
16.00
61.26
85.50
59.58
43.62
15.27
QuaRot
29.43
63.04
30.01
42.24
52.30
22.80
70.78
88.10
58.64
50.81
9.39
3.25
KMeans
29.86
60.39
28.92
43.07
57.75
23.20
70.78
90.20
62.90
51.89
7.04
GPTVQ
27.21
61.48
28.51
42.95
64.84
22.80
70.34
88.40
62.03
52.06
5.54
VPTQ
22.78
51.55
25.23
36.13
49.31
18.20
66.21
85.40
$7.69
45.83
11.13
RIN
29.94
61.82
30.45
43.81
54.84
24.2
71.27
88.1
61.32
51.74
7.89
GPTQ
30.54
61.65
29.79
43.96
59.23
23.80
71.65
88.90
61.06
52.28
6.55
RWKV-7 1.47B
AWQ
22.44
46.00
21.95
35.30
52.65
17.20
61.86
84.50
$9.66
44.61
10.71
QuaRot
30.80
62.28
30.85
43.27
55.64
23.20
71.87
89.20
60.77
51.98
7.94
KMeans
29.69
61.44
30.48
43.92
61.58
24.60
71.32
89.90
62.03
52.77
6.57
GPTVQ
28.32
61.57
29.24
42.63
64.18
21.20
70.83
88.60
62.66
52.13
5,51
VPTQ
23.54
54.25
26.44
37.70
52.24
19.04
67.73
86.90
58.64
47.38
9.52
3.275
Ours
28.15
61.32
29.35
43.24
65.38
22.60
71.70
88.90
61.01
52.40
5.24
13

---

RWKVQuant: Proxy-Guided Hybrid Scalar- Vector Quantization for RWKV Models
Table 10: Complete comparison of perplexity on LAMBADA and averaged accuracy on nine Zero-Shot tasks. For all methods except ours and floating-pon point, we report metries under both bpw settings of 3.5 and 3.25.
ARC-c
ARC-e (1)
HQA.
HellaS.
Lam. (1)
OBQA
PIQA
SCIQ
WinoG. (1)
Avg. (1)
Wiki2
Model
Bpw.
Method
(T)
(1)
(t)
(1)
(t)
(1)
(U)
16
Full Precision
31.22
64.40
30.45
46.34
67.11
25.20
74.26
89.60
60.93
54.39
4.60
RTN
25.25
55.43
27.13
42.59
59.73
24.00
71.00
84.80
58.64
49.84
6.39
GPTQ
25.17
58.62
27.06
42.07
60.18
24.00
71.16
86.70
60.06
50.55
6.43
AWQ
23.72
50.16
23.12
37.34
40.03
19.60
65.12
83.50
57.14
44.41
17.97
3.25
QuaRot
27.30
51.89
26.65
37.42
32.08
21.20
67.84
85.00
55.80
45.02
29.38
KMeans
27.98
55.21
26.80
40.03
41.24
20.20
69.53
85.20
57.06
47.02
15.93
GPTVQ
24.65
57.82
27.20
39.67
62.33
22.00
70.02
86.30
58.80
49.86
6.11
VPTQ
21.33
49.45
23.59
34.65
42.95
17.80
65.56
84,40
55.32
43.89
14.67
RTN
29.43
60.81
27.46
43.64
61.11
22,60
72.52
88,40
59.90
51.76
5,83
RWKV-6 IB
GPTQ
26.53
59.55
27.53
43.63
61.61
23
71.76
87.9
59.58
51.23
5.86
AWQ
24.57
$3.07
22.90
38.57
50.48
20.40
66.43
85.20
58.87
51.20
10.97
3.5
QuaRot
26.19
54.67
26.91
39.21
41.34
23.60
68.60
85.70
56.82
47.00
16.29
KMeans
28.07
60.26
28.11
41.84
51.50
22.40
70.23
87.80
60.14
50.03
8.24
GPTVQ
27.81
58.71
26.18
40.41
61.96
22.20
70.83
87.30
57.22
50.29
5.74
vPTQ
20.87
49.99
23.41
34.37
43.41
17.80
64.20
85.50
54.85
43.82
14.74
3.275
Ours
27.90
60.35
28.08
42.51
63.87
23.00
71.38
87.90
60.22
31.69
3.29
16
Full Precision
35.58
71.33
33.29
50.53
71.32
28.00
76.15
92.30
66.45
58.32
3.83
RTN
30.54
64.52
28.53
47.21
66.87
24.8
73.34
88.88
62.98
34.17
4.71
GPTQ
31.14
65.06
28.77
46.92
65.65
25.40
73.83
85.1
63.61
53.94
4.88
AWQ
26.02
55.13
23.48
40.42
47.53
17.00
66.05
87.80
61.79
47.24
11.97
3.25
QuaRot
29.43
57.82
28.55
41.59
38.35
24.00
69.74
83.80
60.93
48.24
22.67
KMMans
31.99
66.41
30.99
44.62
52.26
25.60
72.41
92.10
61.24
53.06
8.27
GPTVQ
29.77
64.52
28.84
44.56
69.92
24.40
72.57
91.20
62.35
54.23
4.31
VPTQ
25.00
56.56
25.20
37.93
55.95
18.40
67.57
86.20
61.24
48.22
7.77
RTN
31.56
66.75
30.78
47.91
68.02
26.00
74.75
90.50
53.53
54.42
4.50
RWKV-6 3B
GPTQ
31.48
67.17
29.80
47.65
67.49
25.80
74.26
90.30
63.22
55.24
4.54
AWQ
28.41
54.67
24.03
39.96
53.06
19.20
66.15
88.20
61.95
48.40
8.77
3.5
QuaRot
31.22
61.48
29.46
43.04
42.99
29.20
71.21
89.10
61.48
51.01
16,99
KMeans
34.72
67.46
31.43
46.63
58.53
25.60
73.77
91.90
64.95
54.99
6,28
GPTVQ
30.04
65.61
29.57
45.39
69.84
24.60
73.12
89.90
65.58
54.85
4.12
vPTQ
25.59
56.81
25.52
38.09
53.72
19.60
68.49
87.70
60.69
48.46
8.62
3.275
Ours
30.97
65.74
30.12
46.90
71.18
25.00
74.53
92.20
65.5T
35.79
3.88
16
Full Precision
41.70
75.25
35.66
55.82
75.35
31.4
78.18
93.80
68.11
61.69
3.21
RTN
35.66
70.32
32.53
31.85
70.04
28.20
77.20
92.60
66.69
38.54
3.87
GPTQ
38.56
72.51
31.69
51.98
71.78
29.40
77.14
93.50
67.00
59.28
3.72
AWQ
29.01
57.40
23.92
40.72
52.44
20.60
67.3
88.10
65.91
49.48
8.33
3.25
QuaRot
32.25
67.46
31.36
47.39
51.43
28.40
74.59
89.60
66.21
54.29
8.81
KMeans
38.73
72.72
34.71
51.41
61.85
29.00
76.93
81.50
54.69
55.72
4.69
GPTVQ
36.34
71.63
32.09
49.78
74.17
28.40
74.64
92.40
67.71
58.57
3.49
VPTQ
28.83
63.80
28.18
42.15
66.23
21.20
71.32
91.10
64.79
53.06
4.75
RTN
37.37
70.79
32.53
53.26
73.06
29.80
76.93
68.58
59.20
3.59
RWKV-6 7B
GPTQ
39.33
73.40
32.96
52.95
71.96
30.60
77.2
94.20
68.11
60.07
3.68
AWQ
29.01
57.40
23.92
40.72
61.54
20.60
67.3
88.10
65.91
50.50
7.07
3.5
QuaRot
35.32
67.76
31.36
49.68
60.18
31.40
75.46
94.50
66.92
56.95
6.44
KMeans
39.84
74.03
34.35
67.48
28.60
76.98
93.90
67.48
59.51
3,96
GPTVQ
38.50
72.50
31.69
51.98
75.04
29.40
77.71
93.50
67.00
59.70
3.30
vPTQ
28.78
62.33
27.83
42.45
67.86
21.00
71,49
90.40
64.48
52.95
4.47
3.275
Ours
37.62
72.93
33.63
552.03"
75.7ō
30.40
76.60
95.10
67.7ī
60.19
3.23
16
Full Precision
44.70
76.97
37.05
58.76
76.23
33.6
79.59
94.7
71.27
63.65
3.02
RTN
35.76
73.98
32,93
"53.66
74.03
31.80
70.10
93.00
70.27
6T.16
3.34
GPTQ
38.22
69.99
33.36
55.28
74.00
31.40
78.18
91.50
69.69
60.18
3.43
AWQ
28.15
55.34
23.77
38.82
58.59
20.60
65.61
88.90
64.40
49.35
8.18
3.25
QuaRot
36.51
69.27
32.78
50.38
40.09
28.20
76.06
91.60
67.95
54.76
14.0S
KMeans
43.85
74.16
35.55
54.84
67.81
32.40
78.18
94.80
71.27
61.40
4.61
GPTVQ
35.65
71.66
30.45
52.33
76.46
30.14
76.31
93.24
70.44
59.63
3.15
VPTQ
36.27
69.54
29.46
50.08
73.01
27.35
76.38
91.40
67.39
57.87
3.62
RTN
39.59
72.22
33.55
56.31
74.29
31.4
78.01
92.3
69.92
60.84
3.31
RWKV-6 14B
GPTQ
38.90
72.72
34.46
56.34
75.08
31.20
78.67
91.70
71.19
61.14
3.29
AWQ
29.09
57.82
23.59
39.99
64.47
20.20
67.19
90.70
64.71
50.86
6.06
3.5
QuaRot
38.05
69.06
33,80
50.96
44.56
30.80
76.38
93.00
70.40
56.33
10.63
KMeans
41.29
74.07
34.82
54.76
70.66
31.60
78.23
94.30
70.63
61.15
3.84
GPTVQ
36.65
70.87
31.91
52.43
75.04
30.00
77.31
93.40
70.24
59.76
3.34
VPTQ
37.10
69.87
29.69
50.31
72.31
27.45
76.56
91.60
66.55
57.93
3.75
3.275
Ours
4T.80
76.50
34.09
55.26
78.17
32.80
78.61
95.40
71.60
62.69
2.89
14

---

RWKVQuant: Proxy-Guided Hybrid Scalar- Vector Quantization for RWKV Models
Table 11: Complete ablation study on the impact of codebook optimization for clement-wise multiplication.
ARC-c
ARC-e
HQA.
HellaS.
Lam.
OBQA
PIQA
SCIQ
WinoG.
Avg. (t)
Wiki2
Model
Method
(1)
(1)
(T)
(1)
(t)
(1)
(1)
(t)
(f)
(4)
w.
19.79
42.17
24.14
30.02
42.25
17.00
63.10
79.10
52.30
41.09
18.41
RWKV7-0.1B
wO.
20.15
44.45
24.47
30.32
38.09
17.80
64.08
73.95
52.91
40.69
24.71
22.61
53.11
25.41
34.45
53,39
19.60
66.53
84.20
54.80
46.01
9.39
RWKV7-0.5B
wO.
21.61
53.02
25.31
34.55
47.10
18.90
66.53
83.7
54.61
45.03
13.49
28.15
61.32
29.35
43.24
65,38
22.60
71.70
88.90
61.01
52.40
5.24
RWKV7-1.47B
wO.
29.15
61.90
29.99
42.92
59.31
24.10
70.13
88.52
64.14
52.24
6.54
27.90
60.35
28.08
42.51
63.87
23.00
71.38
87.9
60.22
$1.69
5.29
RWKV6-1B
27.38
60.05
27.42
42.31
63.71
23.2
71.72
87.5
59.75
51.44
5.32
30.97
65.74
30.12
46.90
71.18
25.00
74.53
92.20
65.51
55.79
3.88
RWKV6-3B
30.97
68.00
29.72
46.84
70.46
25.20
74.21
90.80
63.14
55,48
3.97
W.
37.62
72.93
33.63
52.03
75.70
30.40
76.6
95.1
67.71
60.19
3.23
RWKY6-7B
WO.
37.97
73.19
33.44
52.04
75.18
30.80
76.61
93.8
68.59
60.18
3.21
RWKV6- 14B
W.
41.80
76.50
34.09
55.26
78.17
32.80
78.61
95.40
71.60
62.69
2.89
WO.
40.53
73.82
34.13
56.19
77.72
32.00
78.44
94.00
71.50
62.03
2.89
Table 12: Our method's performance on RWKV6, using the LambA training dataset, shows advantages in PPL and ACC metrics under consistent experimental settings.
GPTQ
GPTVQ
Ours
Avg-ace
LambA. PPL.
Avg-acc
LambA. PPI.
Avg-ace
LambA. PPI.
RWKV6 IB
51.28
5.87
49.69
5.51
51.41
5.35
RWKV6 3B
55.25
4.56
54.85
4.42
55.38
3.97
RWKV6 7B
59.18
3.59
48.30
3.41
60.18
3.21
RWKV6 14B 61.15
3.29
59.86
3.31
62.03
2.89
Although using the test set for calibration is a common practice in Post Training Quantization (PTQ), we conducted additional comparative experiments based the LambA training dataset for RWKV6 in Tab. 12. The results show that our method outperforms both GPTVQ (Vector Quantization) and GPTQ (Scalar Quantization). This verifies the effectiveness and robustness of our approach.
In Equation 17, the byperparameter K represents the order of Taylor decomposition. Higher-order terms in the Taylor expansion are close to zero and have minimal impact on the final result. In our previous paper, we used K = 4 as the configuration. To further validate the influence of on our method, we conducted comparative experiments on RWKV6 as shown in Tab. 13. Under the same configurations of Ty and Tex we compared the model accuracy under different values of K. It can be observed that when K increases from 2 to 4, the model accuracy significantly improves. However, as K further increases, the accuracy does not continue to increase and even starts decline. Therefore, conclude that excessively large K is unnecessary: it not only fails to improve accuracy but also introduces additional computations.
A.5. Limitations and Future Work
Our proposed RWKVQuant framework relies on the coarse-to-fie to-ss-tt--fie proxy introduced in Section 3.1, where To and Ty play important role in determining the appropriate quantization method for each layer. In our experiments, these values were empirically set based on different model configurations, ensuring that the 3.25 bpw SQ proportion is approximately one-tenth and nine-tenths for 3.5 bpw VQ across different models.
Actually, as shown in algorithm 1, Te and Ty are automatically set for each individual model and do not require adaptation. Specifically, we obtain their values in the following steps:
15

---

RWKVQuant: Proxy-Guided Hybrid Scalar-Vect Quantization for RWKV Models
Table 13: The detailed performance metrics for the RWKV6 model across different K, Ty and configurations. the percentile M (384) values of Pe used to determine Te. is the percentile of M C² values Py used to determine TY-
Lambda accuracy
SQ:VQ Proportion
K = 2 =
K
K = = 6
Quantization Bits
63.11
63.82
63.87
63.88
63.87
F = C = 0.5
10:90
10:90
10:90
10:90
10:90
3.275
3.275
3.275
3.275
3.275
62.95
63.71
63.78
63.74
63.78
== 0 0.5
20:80
20:80
20:80
16:84
20:80
3.300
3.300
3.300
3.290
3.300
61.96
62.87
62.98
62.95
62.91
= 0.6.6, 0.5
30:70
30:70
30:70
30:70
30:70
3.325
3.325
3,325
3.325
3.325
60.98
61.35
61.56
61.68
61.54
0.8 = 0.5
40:60
40:60
40:60
40:60
40:60
3.350
3.350
3.350
3.350
3,350
1. Compute the coarse-grained proxy 15(Eq.15) for each layer to be quantized.
2. Set T. to the value at the 50th percentile of all Pe-
3. Compute the fine-grained proxy Pf 17q.17) for each layer whose Pa < Te-
4. Set to the value at the 20th percentile of all Py-
Although fine-tuning the percentile values for each network may further improve accuracy, the percentile values (i.e., 20% and 50%) -used in all our experiments-alreade delivers strong performance across all RWKV networks, shown in Table 2.
Table 14:
Comparison of perplexity LAMBADA and averaged accuracy nine Zero-Shot tasks under different and Ty configurations for the RWKV models.
RWKV7-0.1B
RWKV7-0.5B
RWKV7-1447B
Tc
Tf
0-shot g
LambA.
0-shot
LambA.
9-shot
LambA.
Avg.(t)
(4)
Avg.(t)
(4)
Avg.(t)
(4)
20.00
40.27
23.65
43.72
13.98
52.09
5.52
25.00
40.27
23.65
43.72
13.98
52.09
5.52
1.00
30.00
40.27
23.65
43.62
14.06
52.09
5.52
35.00
40.25
23.75
43.64
14.15
52.06
5.54
40.00
40.25
23.75
43.64
14.15
52.06
5.54
20.00
-40.56
18.68
45.93
9..8
52.37
5.24
25.00
41.01
18.31
46.01
9.46
52.40
5.24
1.50
30.00
41.10
18.41
46.01
9.39
52.40
5.24
35.00
40.86
18.71
46.01
9.45
52.40
5.24
40.00
40.78
18.41
45.88
9.67
52.34
5.24
20.00
39.86
28.12
45.93
9.88
52.34
6.12
25.00
39.86
28.12
45.93
9.88
52.34
6.12
2.00
30.00
39.65
31.34
45.87
10.07
52.26
6.32
35.00
39.37
37.25
45.83
13.01
52.28
6.54
40.00
39.37
37.25
45.83
13.01
52.28
6.54
16

---

RWKVQuant: Proxy-Guided Hybrid Scalar-Vecs Quantization for RWKV Models
Table 15: The detailed performance metrics for the RWKV6 model under various configurations of the parameters and Te- C the percentile of M (384) values of Pa used to determine Te- determine Ty.
the percentile
values of Py used to
Lambda accuracy
SQ:QQ Proportion
C' = 0.1
C 0.2
C = 0.3
C = 0.4
C = 0.5
C' = 0.6
C 0.7
C = 0.8
Quantization Bits
61.20
62.14
61.78
61.65
61.21
60.84
60.75
60.54
wo F WO Tf)
10:90
20:80
30:70
40:60
50:50
60:40
70:30
80:20
3.275
3,300
3.325
3,350
3.375
3.400
3,425
3,450
61.28
61.31
61.54
62.45
62.44
62.47
61.25
60.84
F= 0.1
1:99
2:98
3:97
4:96
5:95
6:94
7:93
8:92
3,253
3.255
3,258
3,260
3,263
3.265
3.268
3,270
61.29
61.52
61.88
62.63
63.87
63.88
61.34
60.88
F = 0.2
2:98
4:96
6:94
8:92
10:90
12:88
14:86
16:84
3,255
3,260
3,265
3.270
3,275
3,280
3.285
3,290
61.39
61.64
62.62
62.73
63.83
62.86
60.95
60.89
F= 0.3
3:97
6:94
9:91
12:88
15:85
18:82
21:79
24:76
3,258
3.265
3,273
3,280
3,288
3.295
3.302
3.310
61.42
61.94
62.65
62.21
62.78
62.32
61.36
61.26
F = 0.4
4:96
8:92
12:88
16:84
20:80
24:76
28:72
32:68
3,260
3,270
3,280
3.290
3,300
3.310
3,320
3.330
61.43
62.37
62.54
61.95
61.18
60.92
60.73
60.54
F= 0.6
6:94
12:88
18:82
24:76
30:70
36:64
42:58
48:52
3,265
3,280
3.295
3.310
3.325
3,340
3,355
3.370
61.44
61.47
61.13
60.88
60.56
60.53
60.33
60.34
F = 0.8
8:92
16:84
24:76
32:68
40:60
48:52
56:44
64:36
3,270
3,290
3.310
3.330
3.390
3,410
3,430
3.450
However, this allocation might not reflect the most balanced or effective proportion. As shown in Table 14, we conducted multiple comparative experiments on the RWKV language model with varying To and Ty. The results demonstrate that when To is larger than the optimal value, the final accuracy approaches that of directly using uniform quantization, whereas a smaller т. yields results closer to codebook quantization. Moreover, with an appropriately chosen Te. the setting of Ty has a direct impact on the final accuracy. Therefore, in future work, we will further explore how to determine appropriate values for T. and Ty:
Based on proper initialization, we plan to utilize fine-tuning to achieve the optimal configuration for different models.
At the same time, we will remove the fixed constraint of bpw being 3.275 and, when selecting and Ty, consider the trade-off between compression rate and post-quantization model performance to meet different accuracy requirements across various scenarios.
In addition, we plan to use the proposed coarse-to-fine proxy to determine whether specific features finer granularity are better suited for SQ or VQ quantization, such channel-level or block-level finer granularity.
17

---

RWKVQuant: Proxy-Guided Hybrid Scalar- Quantization for RWKV Models
Algorithm RWKV Model Quantization Algorithm
input RWKV model with weights 0, number of weights M, hyperparameters C' and F output Quantized model with weights 0'
1: Initialize: C_Lisl +- D.F_List + D. Φ + 0 {Compute T..
2: for m = M - do
3: Apply Equations (5) and (9) Om get Pa
4:
C_LLastaapendd(P
5: end for
6: To f- ttopk(C_LLise List, k = = C M, largest False)
[Compute T
7: for m = 0 O.M-- - do
8: if CCL_ist(mm] < Te then
9: Apply Equations (10) and (17) to mm to get Pf F (_List.oppend(p)
11: end if
12: end for
13: Ty took(F___aaa k = F. C M, largest False)
(Perform quantization)
14: for m = 0 M do
if P. T. then
Apply Vector Quantization (VQ)
else
18:
if P/ 2 Ty then
19:
Apply Vector Quantization (VQ)
20:
else
21:
Apply Scalar Quantization (SQ)
22:
end if
end if
24:
end for
25:
return A' =0
18


Reviews:
[{"review_id": "3vmLYgAytr", "reviewer": "Reviewer_qMhV", "summary": "This paper introduces RWKVQuant, a post-training quantization framework for the RWKV model family. The main contributions are: 1) revealing the limitations of existing Scalar Quantization (SQ) and Vector Quantization (VQ) methods on RWKV; 2) proposing a coarse-to-fine proxy strategy to guide the hybrid use of SQ and VQ; 3) optimizing the codebook for RWKV's unique element-wise multiplication operation. Extensive experiments show that the method can quantize RWKV-6-14B to about 3-bit with less than 1% accuracy loss while achieving 2.14x speedup.\n\n## update after rebuttal\nThank you for the author's response. After referring to the author's response and the comments of other reviewers, I decided to keep my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable time and efforts in reviewing our manuscript. We have addressed each comment and made the necessary revisions to improve the quality and clarity of our manuscript.\n>Only empirical setting of $\\tau_c$ and $\\tau_f$ mentioned in experimental section.\n\nSorry for causing this confusion. Actually, $\\tau_c$ and $\\tau_f$ are automatically set for each individual model and do not require adaptation. Specifically, we obtain their values in the following steps:\n- Compute the coarse-grained proxy $P_c$ (Eq.15) for each layer to be quantized. \n- Set $\\tau_c$ to the value at the 50th percentile of all $P_c$.\n- Compute the fine-grained proxy $P_f$ (Eq.17) for each layer whose $P_c$ < $\\tau_c$.\n- Set $\\tau_f$ to the value at the 20th percentile of all $P_f$.\n\nAlthough fine-tuning the percentile values for each network may further improve accuracy, the percentile values (i.e., 20% and 50%) —used in all our experiments—already delivers strong performance across all RWKV networks, as shown in Table 2 of the manuscript. \nWe have also included more results in various configurations in our latest manuscript. Here is an example of RWKV-6-1B  in [Table R1](https://anonymous.4open.science/r/RWKVQuant-E375/Table_for_Tr_Tf.MD). \n\n\n>Insufficient analysis of optimal quantization strategies across different model scales. No related discussion in experimental section.\n\nIn line 347-353 of the manuscript, we have disscussed this point. Here is a takeaway for your convenience. \nAs shown in Table 2 in original paper, on small-scale models, our method reduces the error by 14.5% compared to the SOTA; on larger-scale models, our method performs almost consistently with floating-point precision and reduces the error by 62.34% compared to the SOTA. The following table compares the average zero-shot accuracy between origin models and our quantized models. It can be observed that our method is only of slight accuracy decline across different model sizes . \n\n|Model |RWKV-6-1B|RWKV-6-3B|RWKV-6-7B|RWKV-6-14B|\n|:----:|:----:|:----:|:----:|:----:|\n| FloatingPoint|54.39%|58.32%|61.69%|63.65%|\n| FloatingPoint|51.69%|55.79%|60.19%|62.69%|", "claims_and_evidence": "The paper's main claims are well supported by both experimental and theoretical evidence:\n\n1. Table 1 verifies the more uniform weight distribution characteristic of RWKV through clustering loss comparison\n2. Figure 3 validates the effectiveness of the proxy strategy through visualization analysis\n3. Tables 2-3 demonstrate the method's superiority through comprehensive comparative experiments", "methods_and_evaluation_criteria": "The methodology is sound and evaluation criteria are comprehensive:\n\n1. Method design progressively develops solutions based on RWKV characteristics\n2. Evaluation metrics include model performance indicators such as accuracy and perplexity\n3. Considers practical metrics like memory usage and inference speed\n4. Thorough validation on multiple benchmark datasets", "theoretical_claims": "The theoretical analysis is complete:\n\n1. Detailed analysis of SQ and VQ limitations in RWKV\n2. Complete mathematical derivation for the coarse-to-fine proxy strategy\n3. Codebook optimization method designed based on element-wise multiplication characteristics", "experimental_designs_or_analyses": "The experimental design is comprehensive and well-structured:\n\n1. Validation across multiple RWKV model scales\n2. Comparison with existing mainstream quantization methods\n3. Rich ablation studies\n4. Both qualitative and quantitative analyses provided", "relation_to_broader_scientific_literature": "Clear articulation of relationships with existing work.", "essential_references_not_discussed": "All major related works are discussed, with no significant omissions noted.", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. First systematic study of RWKV model quantization. First paragraph of Introduction explicitly states this is the first comprehensive quantization framework for RWKV family.\n2. Novel and effective method design. \n    - Table 2 shows superior performance across multiple models.\n    - Ablation studies validate the necessity of each component\n3. Rigorous theoretical analysis. \n    - Complete mathematical derivation in Section 3\n    - Intuitive visualization explanation in Figure 3\n4. Comprehensive experimental validation.\n    - Validation on 7 different scale models\n    - Comparison of multiple evaluation metrics in Tables 2-3\n\n**Weaknesses:**\n\n1. Lack of theoretical guidance for threshold selection in proxy strategy. Only empirical setting of $\\tau_c$ and $\\tau_f$ mentioned in experimental section.\n2. Insufficient analysis of optimal quantization strategies across different model scales. No related discussion in experimental section.", "other_comments_or_suggestions": "None", "questions_for_authors": "Please refer to Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bkaN0pPyah", "reviewer": "Reviewer_iLYA", "summary": "RWKV is a modern RNN architecture that faces deployment challenges on resource-constrained devices. RWKVQuant, a post-training quantization (PTQ) framework, is proposed to address the limitations of applying existing quantization methods to RWKV models. RWKVQuant uses a coarse-to-fine proxy to adaptively select quantization approaches and optimizes codebook performance for element-wise multiplication. Specifically, RWKVQuant can quantize weights to approximately 3-bits with less than 1% accuracy loss, while providing up to a 2.14× speedup in inference", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable time and efforts in reviewing our manuscript. We have addressed each comment and made the necessary revisions to improve the quality and clarity of our manuscript.\n\n> The paper would benefit from a broader comparative analysis with other advanced quantization methods. \n\n1. Directly applying SOTA methods on Transformer-based LLMs on RWKV models can lead to significant accuracy degration. For example, applying QuaRot [5] to LLaMA series has only 1% accuracy drop, while that for RWKV series is over 8%. \n\n2. There are few works that apply quantization for RWKV models. We are the first to combine SQ and VQ, achieving effective performance on the RWKV family. \n\n>Concerns of Overfitting:\n\nThe characteristic of PTQ is that it enables rapid quantization deployment with a small calibration dataset. Current SOTA methods of Transformer-based LLMs do not encounter this issue. \n\nWe also care about the overfitting challenges on PTQ for RWKV.  We perform experiments on nine datasets, **which are not used for calibration (i.e., zero-shot)**. As shown in Table 2 of the manuscript, RWKVQuant does not overfit. \n\n>Lack of pseudo-code:\n\nThanks for you adivce, we've added an [Algorithm](https://anonymous.4open.science/r/RWKVQuant-E375/Table_for_New_Method.MD) to our latest manuscript.\n\n>Can this method be applied to activation quantization as well?\n\nConsidering that bandwidth is the main bottleneck of LLMs' inference speed, this work is put forward to tackle the challenges associated with weight quantization. Moreover, applying this quantization approach to activations represents a highly promising and valuable area of research. We intend to conduct in-depth explorations of this aspect in our forthcoming research endeavors.\n\n>In Table 4, why does the speedup increase with model size? With weight quantization alone, what measures should be taken to achieve actual speedup?\n\nThe actual speed up can be effected by various factors, such as instruct latency, memory access, computational time, optimization of de-quantization kernal, and so on.\n\nGenerally, memory access takes the most importance across all above factors, since the decode phase is typically memory-bound. With the model size increase, memory access will account for more latency. Considering that weight quantization can reduce this cost, the speed up should also increase with the model size. \n\n>What is the actual cost of the codebook optimization step?\n\nThe codebook optimization is a lightweight **offline** process; for instance, it only takes about 15 minutes for a 70-parameter model.\n\n>Could you provide specific strategies for tuning the parameters $\\tau_c$ and $\\tau_f$?\n\nActually, $\\tau_c$ and $\\tau_f$ are automatically set for each individual model and do not require adaptation. Specifically, we obtain their values in the following steps:\n- Compute the coarse-grained proxy $P_c$ (Eq.15) for each layer to be quantized. \n- Set $\\tau_c$ to the value at the 50th percentile of all $P_c$.\n- Compute the fine-grained proxy $P_f$ (Eq.17) for each layer whose $P_c$ < $\\tau_c$.\n- Set $\\tau_f$ to the value at the 20th percentile of all $P_f$.\n\nAlthough fine-tuning the percentile values for each network may further improve accuracy, the percentile values (i.e., 20% and 50%) —used in all our experiments—already delivers strong performance across all RWKV networks, as shown in Table 2 of the manuscript. \nWe have also included more results in various configurations in our latest manuscript. Here is an example of RWKV-6-1B  in [Table R1](https://anonymous.4open.science/r/RWKVQuant-E375/Table_for_New_Method.MD). \n\nReferences:\n\n[1] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.\n\n[2] GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.\n\n[3] GPTVQ: The Blessing of Dimensionality for LLM Quantization.\n\n[4] VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models.\n\n[5] QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs", "claims_and_evidence": "This paper presents strong claims about RWKVQuant’s effectiveness in improving post-training quantization for RWKV models, backed by experimental results. It demonstrates weight quantization to 3 bits with less than 1% accuracy loss and a 2.14× speedup, using a hybrid SQ-VQ approach guided by Information Entropy and higher-order moments. The method shows promising performance across language and vision tasks, with reduced memory and faster inference — supported by ablation studies. However, clearer comparisons with other leading quantization frameworks and more detailed performance metrics would further strengthen the claims.", "methods_and_evaluation_criteria": "### Proposed Methods:\n\n1. RWKVQuant Framework: Combines Scalar Quantization (SQ) and Vector Quantization (VQ), leveraging RWKV’s architecture for better performance.\n2. Coarse-to-Fine Proxy: Uses Information Entropy to handle weight uniformity and outliers, adapting the quantization approach to the weight characteristics.\n3. Codebook Optimization: Tailors codebook generation to RWKV’s element-wise multiplication modules for improved efficiency.\n\n\n### Evaluation Criteria:\n\n1. Benchmark Datasets: Uses standard datasets like LAMBADA (language) and ImageNet, Coco, ADE20K (vision) for fair comparisons.\n2. Performance Metrics: Measures perplexity (PPL) for language and Top-1 Accuracy, AP, and MIoU for vision tasks — suitable for assessing quantization impact.\n3. Comparative Analysis: Benchmarks against existing SQ, VQ methods, and strong baselines to validate performance improvements.", "theoretical_claims": "This paper does not explicitly present formal proofs for theoretical claims; rather, it primarily focuses on empirical results and the introduction of the RWKVQuant framework. However, it does make several theoretical assertions and claims, particularly about the quantization methods and their effects on the RWKV models.", "experimental_designs_or_analyses": "1. Choice of Datasets and Tasks: Uses LAMBADA (language) and ImageNet, Coco, ADE20K (vision), covering both NLP and CV tasks.\n\n2. Quantization Methods and Baselines: Compares RWKVQuant against SQ methods (RTN, GPTQ) and VQ methods (K-Means, GPTVQ).  Comparisons are broad, but ensuring equivalent configurations (e.g., bit representation, group sizes) across methods is essential for fairness. More transparency on parameter choices (e.g., bpw = 3.25/3.5) would improve clarity.\n\n3. Performance Metrics: Uses perplexity (PPL) for language and Top-1, Box AP, MIoU for vision tasks.\nDiscussion: Metrics are appropriate, but reporting should include mean performance across runs, standard deviation, or confidence intervals to account for random variations.", "relation_to_broader_scientific_literature": "Post-Training Quantization (PTQ): Builds on prior PTQ methods, highlighting the poor performance of traditional SQ and VQ on RWKV, similar to challenges in other architectures.\nHybrid Quantization Strategies: Introduces a hybrid SQ-VQ approach, aligning with prior research suggesting hybrid methods improve quantization performance.\nModel Efficiency: Focuses on efficiency for resource-constrained devices, achieving ~3-bit quantization with high accuracy, supporting findings on T-LLM deployment", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "### Strengths:\n\n1. Novel Hybrid Quantization Framework: RWKVQuant introduces a significant advancement by combining Scalar Quantization (SQ) and Vector Quantization (VQ) tailored specifically for RWKV models. This hybrid approach effectively addresses inefficiencies in previous methods, optimizing quantization for non-linear operators in RWKV.\n2. Empirical Results: The experiments show impressive outcomes, such as less than 1% accuracy loss while quantizing RWKV-6-14B to about 3 bits and achieving a 2.14x speedup. These benchmarks validate the framework’s real-world applicability, making it highly relevant for deploying large models on resource-constrained devices.\nThorough Understanding of RWKV Architecture:\n3. The paper demonstrates a deep understanding of RWKV’s unique characteristics, such as Time Mixing and Channel Mixing, and tailors the quantization strategy accordingly. This approach is more effective than generic quantization methods, ensuring optimal performance.\n4. Clarity and Structure: The paper is well-organized, with a clear flow from the problem statement to the methodology, experiments, and discussions. The inclusion of figures like the accuracy-model size curve effectively communicates complex concepts, making it easy to follow for researchers and practitioners.\n### Weaknesses:\n\n1. Limited Comparison with State-of-the-Art: While RWKVQuant shows promising results, the paper would benefit from a broader comparative analysis with other advanced quantization methods, such as Adaptive Weight Quantization or mixed precision methods, to establish its relative strengths more clearly.\nConcerns of Overfitting:\n\nThe favorable experimental results raise concerns about overfitting, particularly with the Lambada dataset. Acknowledging this limitation and conducting experiments on more diverse datasets or employing cross-validation would provide a clearer picture of the method's generalization capability.\nPractical Implementation Guidance:\n\nAlthough the methodology is clear, the paper lacks practical implementation details. Providing examples, pseudo-code, or insights into hyperparameter tuning and the application of the coarse-to-fine proxy would make the framework more accessible for practitioners.", "other_comments_or_suggestions": "Please refer to the weakness part.", "questions_for_authors": "1. Can this method be applied to activation quantization as well?\n2. In Table 4, why does the speedup increase with model size? With weight quantization alone, what measures should be taken to achieve actual speedup?\n3. What is the actual cost of the codebook optimization step?\n4. Regarding practical implementation, could you provide specific strategies for tuning the parameters τ_c and τ_f? Were any heuristic approaches or guidelines identified during your experiments that could assist other researchers in achieving optimal performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}, {"review_id": "TEyw5IMsVn", "reviewer": "Reviewer_D8uK", "summary": "This paper aims at reducing the memory usage and inference latency of RWKV through post-training quantization (PTQ) techniques. However, authors find that  non-linear operators and  larger amount of uniformly distributed weights hinder the effectiveness of previous PTQ methods. Therefore, authors propose RWKVQuant to quantize weights of RWKV models to extremely low-bit with less than 1% accuracy loss,  2.83$\\times$ memory saving and 2.14$\\times$ acceleration rate, which includes a coarse-to-fine proxy to adaptively select weight outliers and a codebook optimization algorithm to further enhance the performance of quantized RWKV models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable time and efforts in reviewing our manuscript. We have addressed each comment and made the necessary revisions to improve the quality and clarity of our manuscript.\n\n>Why authors choose to use hybrid quantization instead of only use VQ since the outliers in applying SQ to RWKV is unsolvable.\n\nVQ and SQ are suitable for differently distributed weights. VQ excels in managing outliers but underperforms when dealing with uniform distributions. For weights without outliers, SQ can achieve better results. Thus, we are motivated to hybrid them. \n\n>Concerns of Eq. 11.\n\nThanks for pointing it out. We have fixed this problem in our latest manuscript. \n\n>Why the $s_k$ in Eq.15 can be omitted.\n\nEq.15 is an equal expression of the Taylor expansion, but Eq.17 is obtained by performing an absolute operation to Eq.15. Thereby $s_k$ can be omitted. We explain the reason for this operation in step5 of the fine-grained proxy (line 241-245).\n\n>Concerns on the fine-grained proxy.\n\nFine-grained proxy is not just a Taylor expansion version of coarse-grained one. It extracts the outliers features by performing an absolute function to specific terms of the Taylor expansion, as stated in step5 of the fine-grained proxy (line 241-245).\n\n>Authors should add more comparison about different setting of $\\tau_c$ and w/ w/o $\\tau_f$.\n\nActually, $\\tau_c$ and $\\tau_f$ are automatically set for each individual model and do not require adaptation. Specifically, we obtain their values in the following steps:\n- Compute the coarse-grained proxy $P_c$ (Eq.15) for each layer to be quantized. \n- Set $\\tau_c$ to the value at the 50th percentile of all $P_c$.\n- Compute the fine-grained proxy $P_f$ (Eq.17) for each layer whose $P_c$ < $\\tau_c$.\n- Set $\\tau_f$ to the value at the 20th percentile of all $P_f$.\n\nAlthough fine-tuning the percentile values for each network may further improve accuracy, the percentile values (i.e., 20% and 50%) —used in all our experiments—already delivers strong performance across all RWKV networks, as shown in Table 2 of the manuscript. \nWe have also included more results in various configurations in our latest manuscript. Here is an example of RWKV-6-1B  in [Table R1](https://anonymous.4open.science/r/RWKVQuant-E375/Table_for_Tr_Tf_wo_f.MD). \n\n>It would be better to further explain the novelty and differences between the codebook optimization and previous arts.\n\nFirstly, our approach is designed for element-wise multiplication operations (widely-applied in RWKV), instead of matrix multiplication (widely-applied in Transformer-based LLMs). Compared to matrix level, element-wise poses a challenge of batch integration for calibration.\n\nSecondly, we discover that the activations of RWKV follow the Guassian distribution as shown in Figure 4. Thus, our codebook optimization can be well applied. \n\nThirdly, directly applying previous arts to RWKV leads to significant accuracy decline as shown in Table 7 of the manuscript. Our method can effectively enhance performance. \n\n>How the performance changed with different settings of proportion of SQ and VQ? \n\n1. VQ and SQ are suitable for differently distributed weights. VQ excels in managing outliers but underperforms when dealing with uniform distributions. For weights without outliers, SQ can achieve better results. Thus, we are motivated to hybrid them. \n\n2. It is possible to improve accuracy by adjusting the percentiles for each network. However, this default configuration can already achieve effective results across all RWKV networks and typical tasks as shown in Table 2 of the manuscript.  \n\n>Why are VQ methods much better than all other previous work?\n\nGenerally, SQ is not good at handling outliers for its scaling characterstic, especially in aggressive bitwidth (e.g., lower than 4 bit). In contrast, VQ has the ability to capture the distribution of the original sequence and offers an advantage in terms of compression ratio [1].\n\n>Results on other bit-width settings.\n\nPlease refer to the  Table R1 in the above response. \n\n>The effectiveness of codebook optimization is much weaker with larger-scale RWKV. \n\nGenerally, larger-scale models are less sensitive to quantization, primarily due to their larger amount of redundant parameters as mention in [2,3]. Thus, the optimal quantization strategy may be less obvious for these models.\n\n>Symbol conflict in Eq. 6.\n\nThanks for pointing it out. We have fixed this problem in our latest manuscript. \n\n>The explanation of superscript k in Eq. 12 is missing.\n\nWe explain it next to Eq. 12 in line 188-190.\n\n>Symbol reuse: m.\n\nThanks for pointing it out. We have fixed this problem in our latest manuscript. \n\nReferences:\n\n[1].  QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks\n\n[2]. A Survey of Quantization Methods for Efficient Neural Network Inference\n\n[3]. LLM Inference Unveiled: Survey and Roofline Model Insights", "claims_and_evidence": "Claims in the \"introduction\" section is supported by Figure 1&5 and Table 1.", "methods_and_evaluation_criteria": "I've checked all theoretical and qualitative analysis and claims in this paper. See \"Other Strengths And Weaknesses\" part of this review for my major & minor concerns about the methodology and equation derivation.", "theoretical_claims": "I've checked all theoretical and qualitative analysis and claims in this paper. See \"Other Strengths And Weaknesses\" part of this review for my major & minor concerns about the methodology and equation derivation.", "experimental_designs_or_analyses": "I've checked all experimental settings, comparison and results in this paper. See \"Other Strengths And Weaknesses\" part of this review for my major & minor concerns about the experimental part.", "relation_to_broader_scientific_literature": "All contributions are technical and all datasets used for experiments are open-sourced. Thus no key contributions of this paper related to the broader scientific literature.", "essential_references_not_discussed": "All necessary references are discussed.", "other_strengths_and_weaknesses": "## Strengths:\n1. The bottleneck of quantizing RWKV and the differences between RWKV models and Transformer-based LLMs are both clearly claimed and explained. Thus the motivation of this paper is clear and well-explained. \n2. Figures are fancy and easy to follow, all legends in figures are clear and captions provide detailed information about the corresponding figure. \n3. Preliminary is concise. The process of proposed method is clearly described, although the symbol is a little bit confusing. \n4. Experiments are extensive and prove the effectiveness of the methods proposed in this paper. \n\n\n## Major weaknesses:\n1. In section 2.2, authors introduce SQ and VQ, but fail to claim the pros and cons of these two techniques. Therefore, I'm a little bit confused by why authors choose to use hybrid quantization instead of only use VQ since the outliers in applying SQ to RWKV is unsolvable. \n2. Eq. 11 is weird, since $$\\\\sum_{i=1}^n G'_i = 1$$ \n\nand $$\\\\sum_{i=1}^n \\\\frac{1}{n} = 1$$, which means Eq. 11 is always satisfied. Maybe authors mean \n\n$$\\\\sum_{i=1}^n \\\\delta= \\\\sum_{i=1}^n (G'_i - \\\\frac{1}{n}) = 0$$, which means conducting the element-wise subtraction first then conducting the sum operator. \n\n3. Why the $s_k$ in Eq. 15 can be omitted? if so, the Taylor expansion won't be satisfied, then the $P_f(G')$ does not equal to the k-th expansion of $P_c(G')$. Maybe authors can add some experimental comparison of w/ & w/o $s_k$ for double-check and further clarification since $s_k$ is easy to calculate. \n4. Seems like the fine-grained proxy is just a Taylor expansion version of coarse-grained one. What if just set a smaller $\\\\tau_c$? As shown in Figure 3 (b) & (c), if $\\\\tau_c$ is set as 0.95 (0.94(Fig. 3c) <0.95 <0.96(Fig. 3b)) and w/o $\\\\tau_f$, then it will also utilizes VQ when the condition depicted in (b) happened. Authors should add more comparison about different setting of $\\\\tau_c$ and w/ w/o $\\\\tau_f$. \n5. The section of \"codebook optimization for element-wise multiplication\" seems like just applying percentile-based clipping operation into a traditional VQ, instead of simply average all samples, thus solve the outlier phenomenon. This part lacks novelty. It would be better to further explain the novelty and differences between authors' method and previous arts. \n6. (1). Why the setting of bpw of previous methods are 3.25/3.5, while authors' method is 3.275? Just for ensuring that SQ is used for 9/10 of the layers and VQ for 1/10? How the performance changed with different settings of proportion of SQ and VQ? Lack relevant experiments and analysis. (2). Meanwhile, results in Table 2&3 both prove that the performance of VQ method GPTVQ is much better than all other previous work, is it because VQ methods often provide better performance with latency overheads? Then authors should also compare the flops and performance with different settings of proportion of SQ and VQ. \n7. Authors only provide results with bpw around 3-bit, how about other bit-width, e.g. 2-bit & 4-bit. \n8. As shown in Table 11, it seems like the effectiveness of codebook optimization is much weaker with larger-scale RWKV on Wiki2. I'm curious about this phenomenon, is it because with the model scale becomes larger, the outliers affect the performance less significantly? Maybe  authors can explain this part shortly. \n\n## Minor weaknesses:\n1. Symbol conflict in Eq. 6: since subscript $i$ is used in the numerator $G_i$, the subscript in denominator $\\sum_{i=1}^n G_i$ should be a new one, which can be formulated as $\\sum_{j=1}^n G_j$. \n2. The explanation of superscript $k$ in Eq. 12 is missing. If I'm not missing anything, does it represent k-th  order partial?\n3. Symbol reuse: $m$ in section 3.1 represents index of number of weights, while in section 3.2 & 2.2 represents shape of a tensor or weight $\\\\mu$. .", "other_comments_or_suggestions": "See \"Minor weaknesses\" part above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1DNiPiiLeT", "reviewer": "Reviewer_RAuy", "summary": "This paper proposed RWKVQuant, a method for PTQ of RWKV. They propose a hybrid method combining both scalar and vector quantization, and a decision rule for assigning different layers to these two methods. For appropriate values of hyperparameters (such as $\\tau_c$ and $\\tau_f$), they demonstrate strong performance on a wide range of benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable time and efforts in reviewing our manuscript. We have addressed each comment and made the necessary revisions to improve the quality and clarity of our manuscript.\n\n>Concerned about the setting of hyperparameters.\n\n1. For hyperparameters  $\\tau_c$ and $\\tau_f$\n\n    We have explicitly added the sentence \"the performance of RWKVQuant can vary depending on the setting of $\\tau_c$ and $\\tau_f$, see Appendix A.5\" to our latest manuscript after line 274 (first column). \n\nActually, $\\tau_c$ and $\\tau_f$ are automatically set for each individual model and do not require adaptation. Specifically, we obtain their values in the following steps:\n- Compute the coarse-grained proxy $P_c$ (Eq.15) for each layer to be quantized. \n- Set $\\tau_c$ to the value at the 50th percentile of all $P_c$.\n- Compute the fine-grained proxy $P_f$(Eq.17) for each layer whose $P_c$ < $\\tau_c$.\n- Set $\\tau_f$ to the value at the 20th percentile of all $P_f$.\n\nAlthough fine-tuning the percentile values for each network may further improve accuracy, the percentile values (i.e., 20% and 50%) —used in all our experiments—already delivers strong performance across all RWKV networks, as shown in Table 2 of the manuscript. \nWe have also included more results in various configurations in our latest manuscript. Here is an example of RWKV-6-1B  in [Table R1](https://anonymous.4open.science/r/RWKVQuant-E375/Table_for_ablation_study.MD). \n\n2. K represents the order of the Taylor expansion, we set it to 4 for all models.\n\nA larger number of K offers a closer result to the actual value (i.e., K=+∞). Experimentally, we find that K=4 is an ideal trade-off between accuracy and quantization cost under various $\\tau_c$ and $\\tau_f$. For instance, in RWKV-6-1B [Table R2](https://anonymous.4open.science/r/RWKVQuant-E375/Table_for_ablation_study.MD). \n\n>Writing Errors.\n\nThanks for pointing them out. We have fixed them in our latest manuscript. \n\n>Concerns about calibrating from the test dataset.\n\n1. RWKVQuant does not include any training process. All samples that are used for calibration are not used to monitor performance. They are only used to generate activations, rather than supervising.\n\n2. For a fair comparison, we randomly sample 128 sequences from Lambda, following previous work like GPTQ [1] and GPTVQ [2]. In addition, we evaluate on various datasets that are not used for calibration, as shown in Table 2 in our manuscript.\nTo further address your concern, we provide [Table R3](https://anonymous.4open.science/r/RWKVQuant-E375/Table_for_ablation_study.MD), where calibration is performed using only a small subset (128 samples) of the training dataset under the same conditions. The results indicate that our approach still maintains an advantage.\n\n>Does \"calibration\" here mean the procedure for setting $\\tau_c$ and $\\tau_f$ such that 90% of the layers use SQ and 10% of the layers use VQ? \n\nCalibration and proportion are two distinct concepts. Calibration is an common practice of PTQ, which samples data and feed it to models to generate activations. Making use of such activations, PTQ can optimize the weight quantization [1,2]. As for the proportion of SQ and VQ, it is fixed to our default configuration in this work corresponding to the first reply.\n\n>I would like to see an ablation where this percentage of layers using SQ vs VQ is changed and the performance of RWKVQuant is tracked. \n\nPlease refer to the [Table R1](https://anonymous.4open.science/r/RWKVQuant-E375/Table_for_ablation_study.MD). \n\n>Why is the setting of $\\tau_c$ and $\\tau_f$ are not the same?\n\nThe values of $\\tau_c$ and $\\tau_f$ can vary across different models. This variation occurs because these values are dictated by the percentile of the proxy specific to each individual model.\n\nReferences:\n\n[1] GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.\n\n[2] GPTVQ: The Blessing of Dimensionality for LLM Quantization.", "claims_and_evidence": "The claims are supported by clear evidence, but the setting of hyperparameters needs to be addressed more carefully.", "methods_and_evaluation_criteria": "The benchmark datasets make sense for the problem at hand. However, much more sensitivity analyses are needed for the setting of hyperparameters.", "theoretical_claims": "Section 3 is not so much a theoretical claim, but rather an explanation for the development of a heuristic that the authors then use.", "experimental_designs_or_analyses": "I did not check the experimental designs.", "relation_to_broader_scientific_literature": "PTQ of recurrent models has been much more difficult than that of Transformers, likely because recurrent computation allows errors to compound. It is very impressive that RWKVQuant is able to attain such strong performance in light of the known difficulty of this problem in the literature.", "essential_references_not_discussed": "I am not aware of essential references not discussed.", "other_strengths_and_weaknesses": "**I am very concerned about the setting of hyperparameters in this paper**.\n\nThe hyperparameters I'm most concerned about are $\\tau_c$ and $\\tau_f$ , as well as $K$ from equation 17. I would like to see a detailed section in the Appendix discussing in detail all decisions about how to set these hyperparameters in the different experiments, and detailed ablation studies / sensitivity analyses showing what happens to the performance of RWKVQuant as different values for these hyperparameters ($\\tau_c$, $\\tau_f$, and $K$ are used). \n\nAppendix A.5 is an OK start, but it should have been explicitly pointed to in the main section. There needs to be a sentence somewhere in Section 3 saying that \"the performance of RWKVQuant can vary depending on the setting of $\\tau_f$ and $\\tau_c$, see Appendix A.5\".\n\nI am also concerned by the admission of the following:\n> Therefore, in future work, we will further explore how to determine appropriate values for $\\tau_c$ and $\\tau_f$. \n\nIn my opinion, the derivations in Section 3 (Method) are not sufficiently principled to warrant publication on their own, and so the merit of this work is based on the strong practical and empirical performance of RWKVQuant in Section 4 (Experiments). However, **a practical method is only useful if its hyperparameters can be set robustly.** I cannot recommend publication of this work until the bullet points laid out by the authors at the end of Appendix A.5 are satisfactorily addressed. I believe the answers are vital to widespread use of this method.", "other_comments_or_suggestions": "* The short title has not been changed from \"Submission and Formatting Instructions for ICML 2025\"\n* typo: line 097: \"hight-order\"\n* typo: line 066 (second column): \"we propose RWKVQuant, which **enhance** VQ...\"\n* typo: line 088 (second column): \"the current word $x_t$ **and** can be derived by...\"\n* typo: line 202: \"dose\" should be \"does\"", "questions_for_authors": "1. > For both vision and language tasks, we select 128 samples from the corresponding test datasets for calibration\n\n1a. Do you think that choosing the samples from the **test** dataset corresponds to a slight form of test set leakage? I would like to see an ablation where the samples are chosen from a different dataset (or at least where those 128 samples that are used for calibration are not used to monitor performance)\n\n1b. To clarify: does \"calibration\" here mean the procedure for setting $\\tau_c$ and $\\tau_f$ such that 90% of the layers use SQ and 10% of the layers use VQ? I would like to see an ablation where this percentage of layers using SQ vs VQ is changed and the performance of RWKVQuant is tracked).\n\n2. Lines 316-7 (second column) say\n> For example, in RWKV7, $\\tau_c$ is set to 1.54, while $\\tau_f$ is set to 30.\n\nHowever, Figure 5 is constructed \n> Under the settings of $\\tau_c=1.5$ and $\\tau_f=50$.\n\nWhy is there this discrepancy?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a post-training quantization (PTQ) method for compressing pre-trained RWKV models. The authors first point out two limitations of popular PTQ methods for LLMs when applying them to RWKV models. Scaler quantization (SQ) methods relying on smoothing factors/rotation matrices lead to extra computational overhead because of non-linear operators of RWKV models, vector quantization (VQ) methods relying on codebook and vector clustering cannot well handle uniformly distributed weights occupying a large portion of weights in an RWKV model. Therefore, the authors argue that applying SQ or VQ individually is not optimal for RWKV model, and thus propose a hybrid SQ-VQ method called RWKVQuant. It leverages a coarse-grained proxy to evaluate weight uniformity and a fine-grained proxy to identify outliers, so that SQ and VQ can be combined to achieve better performance on RWKV models. Experiments on both vision tasks and language tasks are conducted to showcase the effectiveness of the proposed method.

The paper initially|finally got scores (3,3,3,2)|(3,3,3,3) by four reviewers, who mostly recognized the motivation, the basic idea and its performance. Meanwhile, the reviewers also raised some concerns about 1) method formulation and writing (a lot of errors and typos); 2) lack comparisons with some state-of-the-art methods; 3) lack experiments on more diverse bit-width settings, like 2/4-bit and mix-precision; 4) missing ablations on the choice of hyper-parameters, calibration samples, etc.; 5) some experiments on language tasks may have overfitting issue; 6) lack sufficient experimental details.

The authors provided detailed responses to these concerns. All reviewers acknowledged the authors' rebuttal. Reviewer qMhV recognized the rebuttal and kept the same score for weak accept, and reviewer RAuy also recognized the rebuttal and increased the score to weak accept, and the other two originally positive reviewers kept the same score for weak accept. The AC read the paper, the reviews, the rebuttal and the reviewers' feedback, and mostly agree with reviewers' assessment. As the novelty of the propose method is not very strong and the writing and ablations of paper need a lot of improvements, I recommend "weak accept" to this paper. The authors are encouraged to carefully consider the reviewers' comments/suggestions and their rebuttal in the final paper revision.
Decision: Accept (poster)