SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models
Authors: Haojie Duanmu
Zhihang Yuan
Xiuhong Li
Jiangfei Duan
Xingcheng ZHANG
Dahua Lin
Venue: COLM (2024)
Keywords: quantization, KVCache compression, model compression
TL;DR: This paper proposes SKVQ,  a sliding-window KV cache quantization strategy that achieves high compression ratios while maintaining accuracy, enabling efficient handling of long context lengths in large language models.

Abstract: Large language models (LLMs) have demonstrated the capability to process extended token sequences, enabling complex tasks such as book comprehension and long-form text generation. However, as context length increases, the key-value (KV) cache required for LLMs consumes substantial memory, becoming a bottleneck for deployment.
This paper introduces SKVQ (Sliding-window KV cache Quantization), a strategy designed to address the challenge of extremely low bitwidth KV cache quantization. SKVQ rearranges the channels of the KV cache to enhance channel similarity within quantization groups and applies clipped dynamic quantization at the group level. Furthermore, SKVQ maintains high precision for the most recent window tokens in the KV cache, preserving accuracy for a small yet critical portion of the cache.
Our evaluation of LLMs demonstrates that SKVQ achieves high compression ratios while maintaining accuracy, outperforming previous quantization methods. SKVQ enables the quantization of the KV cache to 2-bit keys and 1.5-bit values with minimal accuracy loss. This advancement allows processing context lengths of up to 1M tokens on an 80GB GPU for a 7B parameter model, resulting in up to 7 times faster decoding.

Published as a conference paper at COLM 2024
SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models
Haojie Duanmu"
Zhihang Yuan*
Xiuhong Lit Peking University
Shanghai AI Laboratory Shanghai Jiao Tong University
Houmo AI
Jiangfei Duan
Xingcheng Zhang Shanghai Al Laboratory
Dahua Lin
CUHK
CUHK
Shanghai Al Laboratory
Shanghai Al Laboratory
Abstract
Large language models (LLMs) have demonstrated the capability to process extended token sequences, enabling complex tasks such as book comprehen- sion and long-form text generation. However, as context length increases, the key-value (KV) cache required for LLMs consumes substantial mem- ory, becoming a bottleneck for deployment. This paper introduces SKVQ (Shiding-window KV cache Quantization), a strategy designed to address the challenge of extremely low bitwidth KV cache quantization. SKVQ re- arranges the channels of the KV cache to enhance channel similarity within quantization groups and applies clipped dynamic quantization at the group level. Furthermore, SKVQ maintains high precision for the most recent window tokens in the KV cache, preserving accuracy for a small yet critical portion of the cache. Our evaluation of LLMs demonstrates that SKVQ achieves high compression ratios while maintaining accuracy, outperform- ing previous quantization methods. SKVQ enables the quantization of the KV cache to 2-bit keys and 1.5-bit values with minimal accuracy loss. This advancement allows processing context lengths of up 1M tokens on an 80GB GPU for a 7B parameter model, resulting in up to 7 times faster decoding.
1 Introduction
Large Language Models (LLMs) have recently demonstrated remarkable success in artificial intelligence. As LLMs advance, the demand for extended context support has increased. For instance, OpenAl GPTT- Turbo can handle 128k tokens (Achiam et al., 2023), and Google Gemini 1.5 can process up to 1 million tokens (Team et al., 2023). This expanded token capacity enables LLMs to address more complex tasks, including book comprehension, large image analysis, and video processing, enhancing their versatility. LLM inference operates in an rego-seeeee manner, generating sentences token by token. To reduce the computation overhead, inference system always store key and value activations in memory known as the key-value (KV) cache, for reuse during subsequent token generation. With the increasing popularity of utilizing LLM for long sequence tasks, the KV cache consumes a significant amount of memory. On the other hand, the large amount KV cache can also bring large amount of memory access in the attention mechanism when generating the output tokens. The system will be stuck on the memory access, known as the memory-bound problem in LLM inference (Yuan et al., 2024).
To tackle the problem of large KV cache size in language models, several compression techniques have been proposed. One approach is KV eviction (Zhang et al., 2023), which
"Equal Contribution
I *Correspondence to: lixiuhong@pku..duu.c

---

Published as a conference paper at COLM 2024
involves removing less important key-value pairs from the cache to free up space. However, this may impact the accuracy of inference. Another method is KV offloading (Sheng et al., 2023), which transfers a portion of the KV cache to slower but larger storage devices like main memory and even secondary storage. However, this may slow down the system due to the low bandwidth of these devices. Scientists have recently been studying the compression of KV cache using quantization. This involves converting floating point KV cache, which initially utilizes a large number of bits, into format that uses fewer bits. Several novel approaches have been developed to accomplish this, including KVQuant (Hooper et al., 2024), WKVQuant (Yue al., 2024), and KIVI (Liu et al., 2024). Previous quantization methods have been successful in reducing memory requirements and the number of memory accesses. However, they faced a challenge when using very low-bitwidth quantization because it led to a significant decrease in accuracy, as shown in Figure 1.
In this paper, we observe that there is sig- nificant difference in the distribution of differ- ent channels during the quantization process. This has great impact on quantization accu- racy, especially in extremely low-bitwidth sce- narios. To alleviate this problem, we propose the clipped dynamic quantization with channel reorder. First, we use transformation invariant permutation to group similar channels based on their statistical characteristics. Second, we apply clipped dynamic quantization to further miti- gate the outlier problem. In this way, we greatly reduce the quantization error within each group, thus improving the accuracy of the quantized model.
35
25
Score
20
15
SKVO
10-
KVI
SmoothQuant
2.0 2.1
2.2
Average bits of element in KV Cache
Figure 1: Results on GovReport and MultiFieldQA-zh (Miistra-76-Insttuut--o222 We count the storage for meta data includ- ing quantization params and reorder index.
Meanwhile, we discover that the protecting the
accuracy of these small portion of but more important caches in KV cache quantization is critical. Due to the locality of attention, these recently generated KV caches are highly likely to be attended to with high probability. We propose sliding window quantization strategy. This mechanism preserves a small portion of the most recently generated KV cache from being quantized. After generating new tokens, the probability of attending to the old tokens' KV cache decreases significantly, so the accuracy loss caused by quantizing them is minimal. The proposed method is named as bliding-window KV cache quantization (SKVQ). It is efficient and easy to implement in existing inference system, which makes it practical for real-world deployment.
To evaluate the effectiveness of our method, we experiments on models of LLaMA (Touvron et al., 2023) and Mistral (Jiang et al., 2023) family. The experiments show that our methods can quantize the key cache into 2 bits and value cache into 1.5 bits with almost no accuracy drop. Compared with the previous quantization method, our approach can achieve optimal performance under different average bit widths as shown in Figure 1. Our performance analysis shows SKVQ enables 1M context length in a single A100-80GB for a 7b model. As for the inference latency, in the case of batch size 128 and sequence length 200k, the theoretical 7x speedup in decoding phase can be achieved
2 Related Work
There are many multi-billion scale transformer quantization methods designed for LLMs. A main branch of LLM quantization is weight-only quantization, which only involves the quantization of model weights to lower precision. For instance, GPTQ (Frantar et al., 2022) uses second-order approximation to quantize weights, enabling the weight quantization of LLMs into 4-bit. AWQ (Lin et al., 2023) quantizes model weights to 4bits whith an activation- aware manner. SqueezeLLM (Kim et al., 2023) adopts the concept of sensitivity-
'The performance analysis be found in Appendix 9
2

---

Published as a conference paper at COLM 2024
Original Key Cache
Channel Reorder
Group Clipping
g2
94
94
14
99
13
81
94
10
93
93
g1
92
g2
Charrall
100 120
0
20
Channel
00
100
120
23
Channeld
50
200
120
Figure 2: Visualization of the key cache going through channel reorder and group clipping in sequence. The elements in the red/green green box will be placed in the same group to share the quantization parameters.
non-uniform quantization along with Dense-and-Sparee decomposition. This line of work is orthogonal to ours, as they can be combined together.
Another line of work focuses on weight-activation quantization. Ilm.int8() (Dettmers al., 2022) retain outlier channel to full precision, so that other parts can be better compressed to 8bits. SmoothQuant (Xiao et al., 2022) uses equivalent transformations to balance the quantization complexity for both activation and weight, making the activation easier to quantize. RPTQ (Yuan et al., 2023) reorder the channels to reduce the variance in one quantization cluster, further enhancing the accuracy. ATOM (Zhao et al., 2023) improves quantization performance and reduces inference latency by using finer-grained quantization with efficient kernel. However, since these works are not specifically designed for KV cache quantization, even applying the best results from such works still results in significant losses in KV cache compression. We compare with these works in the experimental section.
Recently, as natural language tasks require processing longer contexts, researchers have focused on quantizing key-value caches. Several new methods have been developed, such as KVQuant (Hooper et al., 2024), WKVQuant (Yue et al., 2024), and KIVI (Liu et al., 2024). Quantizing the KV cache can significantly reduce both the memory requirements and the number of memory accesses needed. Our experimental results show that the performance of our method on long context tasks performs the best this type of work.
There are also a series of work dedicated to the design of KV cache eviction strategy (Liu et al., 2023; Ge et al., 2023; Zhang et al., 2023; Xiao et al., 2023). Unlike KV cache quantization, which retains all caches but compresses them to low precision, these methods selectively retain part of the KV cache and discard other caches directly. These methods usually allocate fixed-size buffer for KV cache When the generated KV cache exceeds the buffer limit, some tokens considered less important will be evicted from the buffer. These methods are inevitably and irrecoverably discarding KV pairs deemed, in one way another, less important than others. Our approach is inspired by and can be well integrated with such work.
3 Method
3.1 Clipped Dynamic Quantization with Channel Reorder
Quantization is to transform the high-bitwidth float values into low-bitwidth integer values. - 1), where X the
( The quantization process can be formulated as clamp X-=1, float values and Jt is scaling factor and is zero point. Previous studies have highlighted significant variations in numerical values among activation channels (Xiao et al., 2022; Wei et al., 2022; 2023). As shown in Figure 2, we also observe substantial variations between channels and tokens in the KV cache (high channel variance). Therefore, directly quantizing the KV cache leads to substantial quantization errors. If values in different channels share the scaling factor and zero point, the value from outlier channels skew the quantization range. Especially in the low-bitwidth case, this makes almost all elements except outlier channel
0, 2N
3

---

Published as a conference paper at COLM 2024
drop. to the same value, and this loss of information leads to significant performance drop. To tackle this issue, we introduce channel transformation based quantization.
To address this problem, some methods have proposed using additional quantization parameters or keeping certain channels in float format to handle outliers (Dettmers et al., 2022). However, we have noticed that the concept of outliers is relative. The channels with the highest values are outliers compared to the medium-sized channels, and the medium-sized channels are outliers compared to the small-sized channels. Other methods propose smoothing the difference between channels by multiplying an extra factor before quantization (Shao al., 2023; Yue et al., 2024). However, these methods do not take into account the differences in token dimension(sequence length dimension) of KV cache. The magnitude of values can vary between different tokens. We have observed that the variation in magnitude of non-outlier channels is relatively high. Specifically, some channels experience magnitude changes of several times or even dozens of times. Smoothing is not effective in addressing this phenomenon, especially in extremely low bitwidth quantization.
Channel Reorder. Inspired by RPTQ(Yuan et al., 2023), we employ a permutation invariant transformation and then apply group clipping to solve the problem of extremely low bitwidth quantization for KV cache The permutation invariant transformation allows us to change the order of computation without changing operation's output. For example, when execute the matrix multiplication S = Q x KT, we can rearrange the columns of Q and the rows of K (which represent their channel dimension) in the same order without affecting the result of the computation.
We perform channel reorder on KV cache to make channels with similar data distribution are grouped together for quantization. Values in channels with similar distribution are quantized together. By this way, we can greatly reduce the quantization error of channels with smaller ranges. The same as Yuan et al. (2023), we do the corresponding equivalent permutation for Q and W, to avoid explicit reorder operation. The calculation of attention module O = (rfttmax(QKT] V. Wa is transformed as:
O Softmax(P;Q (KTPZ) P.V. -W.P?
where Px E IRCixxaa and P. E RC are channel reorder matrix of Key and Value respectively In our algorithm, the index is calculated based on the statistical characteristics of each channel. Specifically, we extract the distribution feature of each channel and then use the KMeans algorithm to cluster channels with similar characteristics into the same group. We also compared channel reordering with mathematical equivalent smoothing, and the results in Appendix 10 demonstrated the effectiveness of the former.
Clipped Dynamic Quantization. Dynamic per-token quantization is widely used method for quantizing the activations in LLMs (Xiao et al., 2022). Different with static quantization that use the static and z, dynamic quantization will compute new / and 2 using the
max(X))min(XX
min(X)
maximum value and minimum value for each token:
2
,
Previous work about weight quantization (Lin et al., 2023; Shao et al., 2023) has shown that introducing clipping when quantizing weights can improve the quantization performance. According to the second picture in Figure 2, even though we have grouped similar channels together, there are inevitably some outliers within quantization group. In order to reduce the impact of these outliers on other values in the same group, we propose the clipped dynamic quantization, which can be formulated
X 2, 1,,,2 1), where / a a(max () min (X))
mmmnnX) (2)
f(a, x) clamp(L
2
We introduce a clipping scale a E (0, 1] for each group to compute h and z. In order to get the best clipping scale, for each transformer block we try to minimize the MSE of the output of the attention module before and after quantization, i.e, the optimization objective:
a arg min L(x), ((a) MSE(O9,0)
a
We fuse the channel reorder index into the the projection weight matrix of attention module. We describe the fusion in the Appendix 6.
4

---

Published as a conference paper at COLM 2024
Clipped Dynamic Quantization
With Channel Reorder
Quantized Cache
reordering
group diipping
per token quantisation
Full-Precision Cache
Time Step
Sliding Window
X,
K
Time Step t-1
atteation
Key Cache
Filter Rules
sttention sink hitters
X
Qt
Qt
Figure 3: Overview of sliding window quantization Strategy. In each time step, ensure the latest w KV cache is full precision. For a token cache that slides out of the window, we make decision based on the filter rules and choose whether to retain it to high precision.
where 09 is the output of attention module after quantizing KV cache. Unlike weight-only quantization, the KV cache is generated at runtime, it is costly to solve this optimization for each inference. Therefore, we approximate by offline calibration. By performing optimization on calibration dataset in advance, we get the approximate & for each group. We share the same å across different tokens. Using the approximate à*, we can also improve the quantization performance without introducing significant inference cost.
Using channel reorder and clipped dynamic quantization, the elements falling within the same group more fully utilize the numerical range of the quantized data type, thus reducing the quantization error. Because all the parameter Ps, Po, a is determined offline and the reorder operation be fused into linear layers, it is efficient to implement the clipped dynamic quantization with channel reorder on existing inference frameworks.
3.2 Sliding Window Quantization Strategy
Although clipped dynamic quantization with channel reorder improve the quantization performance to a large extent, extremely low-bitwidth KV cache quantization still suffers serious performance degradation, especially when the sequence length become longer. This is because the quantization errors accumulate along the sequence dimension. Because the auto-segressive manner of the LLM, the decoding of a new token depends on the previous KV generated. We realize that this not only challenge but also chance, the auto-regressive manner can be fully exploited to develop more flexible quantization strategies.
Locality. Previous research has demonstrated that attention modules exhibit strong local- ity (Kovaleva et al., 2019; Beltagy al., 2020; Ge et al., 2023). This locality implies that at each time step, the attention module focuses more recently generated tokens. We posit that in KV cache quantization, preseroing the accuracy of a small but critical portion of the cache is more important than maintaining the larger, less significant content from earlier in the sequence.
Motivated by this observation, we propose a sliding window quantization strategy that maintains high precision for the most recent KV cache of window of w tokens. The work- flow, illustrated in Figure 3, consists of two phases: 1) Prefill phase: For each transformer block, after generating the KV cache, first compute attention using full-precision KV

---

Published as a conference paper at COLM 2024
cache. We then quantize the KV cache, reserving the last w token cache pairs at full precision. 2) Decode phase: We process only the token that slides out of the window at each time step. This approach ensures lossless KV cache generation for each transformer block during the prefill phase. It also enhances the quality of generated content by leveraging the locality of the attention module in the decode phase.
Important KV Cache Filter. Beyond recently generated tokens, certain tokens are particu- larly sensitive to quantization. We explored additional methods to identify critical tokens whose KV cache should be maintained at high precision. Inspired by (Xiao et al., 2023), we recognized that the initial tokens of a prompt are crucial for the entire generation process. Consequently, we incorporated an attention sink into our filter rules, reserving the first few tokens at high precision. We observed that maintaining a small number of sink tokens at high precision is effective. Given the fixed positions of sink tokens, this approach is easily implementable and was enabled in our experiments. Some cache eviction methods, such as those proposed by (Liu et al., 2023; Zhang et al., 2023), monitor each token's cumulative attention score, treating these scores as token frequency and retaining only the most fre- quent tokens (heavy hitters) in the KV cache. While keeping heavy hitters at high precision seems intuitive, we did not implement this approach in our experiments for two reasons: 1) The improvement on prediction accuracy by keeping heavy hitters high precision is not significant. 2) When using PlashAttention (Dao, 2023), directly obtaining attention scores is challenging, making implementation in existing inference frameworks problematic.
We anticipate that superior methods for identifying important KV caches may emerge. Therefore, we have maintained this as an interface (filter rules in Figure 3) in our implemen- tation, allowing for the integration of new filters in future research.
By preserving small portion of tokens at high precision, we achieve substantial perfor- mance gains in long long-context tasks while incurring minimal additional overhead. The impact of window size on quantization performance will be discussed in Section 4.3.
4 Experiments
In this section, we introduce the detailed experimental settings and evaluate the effectiveness of the proposed SKVQ.
4.1 Settings
Models. We select a wide range of models with different architectures and different size to demonstrate the generalizability of our approach: Llama2-13b (Touvron et al., 2023), and models fine-tuned based on Llama2: Llama2-7b-chat, Llama2-13b-chat, Llama2-7b-80k (Fu (Chiang et al., 2023), LongChhat-1.5-32k (Li et al., 2023). We also evaluate models of Mistral family which recently very popular: Mistral-7b- v0.1(jiang et al., 2023), family adopt multi-head attention, mistral-7b-instruut-vd.. uses multi-query attention, and mistral-7b-v0.1 uses multi-query attention and sliding-window attention.
et al., 2024), Vicuna-v1.5-7b-1e-
Among these models, models of Llama
Tasks. We evaluate SKVQ mainly on long sequence tasks, as this is the scenario for which KV cache quantization is most suitable. We use Long Bench (Bai et al., 2023) to evaluate on various datasets. Specifically, Multi (Fl score) is a Single- task; 2WikiMultihoPQA is a Multi-Document QA task; GovReport (ROUGE score) is Summarization task; TREC (classification score) is Few-shot Learning task; and LCC (similarity score) and RepoBench-P (similarity score) is Code Completion task. We also tested SKVQ on Needle-in-a-Haasttcc (Kamradt, 2023), which is a popular test-bed for whether models can actually utilize long context length. It requires the model to recite the information in a given sentence, which is placed anywhere in a long document. Finally, to provide clearer picture of the effects of the SKVQ components and to compare with previous methods, we also measure the perplexity on wikitext2 (Merity et al., 2016) in Section 4.3.
QA

---

Published as a conference paper at COLM 2024
LCC1
RepoBenchbPP
PR-ent
TRFC!
2wikimqaa
GovReport
MQ4-zhi
Averaget
FP16
52.33
44.05
10.25
32.09
27.29
11.39
38.50
KIN
15.44
0.79
1.93
6.76
Llama-2-7B-chat
RPTQ
SmoathQuant
35.31
22.37
19.08
32:18
0.79
5
28.75
47.5
15.57
7.45
20.07
11.83
3.24
1.88
19.50
21.92
KIVI
24.73
35.91
SKVQ
27.07
10.7
FP16
50.54
52.1
15.25
68.5
13.21
27.52
7.23
38.83
RTN
20.89
18.62
0.33
0.52
1.68
0.16
10.15
Llama-211B-cchtt
Smoathiuuat
23
23.83
KIVI
48.6
48.81
13.5
68
14.32
25.7
2.01
37.21
SKVQ
49.53
49.76
12.25
67.5
14.05
26.68
6.63
37.53
FP16
27.98
2.49
Mistral-78
SmeetthQuant
40.63
35.14
3.40
30.5
6.05
5
4.12
23.
RPTQ
55.29
47:12
5.11
39.5
9.77
7,81
12:36
35.05
KIVI
68.16
58.33
12.43
13.22
13,87
42.43
SKVQ
67.81
60.54
13.21
10.91
1767
15.9
$3.47
FF16
55.07
48.96
60
70
22.63
31.18
42.74
48.65
KIN
32.36
33.23
D.67
2,25
16.03
2.3
18.02
Mistral-7B-Instruu
SmoothQuant
43.84
4,7%
39.5
23.61
8.33
29.27
RPTQ
46.85
+4.07
27.67
64.5
28
24.68
38.91
KIVI
53.13
48.6
47.5
69
20.68
29.37
33.88
45.48
SKVQ 54.86
49.05
56.42
20.94
30.82
42.4
46.23
Table 1: Evaluation of different KV cache quantization methods on LongBench. Group- size(average) 128, key-cache 2bit, value-cache 2bit, window-size 128. We abbreviated PassageRetrieval as PR and MultiFielDQA as MQA. We highlight the result of our method.
Quantization. Both channel reorder and clipped dynamic quantization requires offline calibration. For calibration dataset, we select 256 pieces of data with length 4096 from the training set of wikitex12-vl, the calibration takes about a few minutes which is quite lightweight. We perform asymmetric quantization in all experiments. We have explored the FP8(E4M3) datatype to store scale and zero-point. Our experiment results in Table 3 show that FP8 will bring almost no performance degradation, but significantly reduces overhead at extremely low bit-width and fine grained groups.
4.2 Main Results and Analysis
The performance of SKVQ in the LongBench datasets is summarised in Table 1. We compare our method with SmoothQuant (Xiao et al., 2022), RPTQ (Yuan et al., 2023) KIVI (Liu et al., 2024) and vanilla asymmetric per-token uniform
LongBench Results.
2blts,
Hatin
Lirraa
Average Score
RTN(Round To Nearest) quantization. SmoothQuant and RPTQ are LLM weight-activation quantization schemes. We use them to quantize KV cache with- out involving model weights and other activation. KIVT (Liu et al., 2024) is recent 2-bit asymmetric quantization scheme specially designed for KV cache. We set the group size of all the methods to 128. SKVQ utilizes reordering which leads to unequal size of each group. In order to ensure the fairness of the comparison, groups in SKVQ to ensure the average group size is 128. The window size in SKVQ is to 128 and the residual length in KIVI is set to 128. a in SmoothQuant is set to 1.0 to make the smooth transformation completely inclined to KV cache Table 1 suggests that SKVQ an effective method for KV cache compression that outperforms previous quantization approaches across various hard long context generation tasks. We also evaluate the results is in Appendix
20-
1P16
SKVG
RIN
1P16
SKVD
RIN
Figure Average score on LongBench SKVQ for Llama2-7b-chat and Mistral- 7b-Instruct-V022
control the number of
Vicuna-v1.5-76-116 and LongChat-v1.5-7--33
For all models tested, the accuracy drop of SKVQ is less than 5%. Towards extremely low-bitwidth KV cache quantization, we further quantize the key cache into 2 bits and value cache into 1.5 bits with group size 64. The result in Figure shows that SKVQ can compress key cache into 2 bits and value cache into 1.5 bits with almost no accuracy drop. It is worth noting that the experimental results under the setting of group size 128. SKVQ can
7

---

Published as a conference paper at COLM 2024
FP16: 268.5
KIVI-K2V2-g128. 244.5
SKVQ-K2v1.5-g122 272.2
14
70
2.0
170
710
310
1000
Token Limit
Taken Limit
Taken Limit
Figure 5: Comparison of SKVQ with KIVI on needle in haystack test. SKVQ achieved higher scores while using lower bitwidth.
4bit
3bit
2bit
Method
PPLL
avg-bits.
PPL4
avg-bits
PPL1
avg-bits.
RTN-sym 4.66
4.25
4.98
3.25
26.83
2.25
KVQuant
4.59
4.32-4.35
4.64
3.32-3.35
4.92
2.32-22335
Ours 4.60
4.25
4.63
3.25
4.87
2.25
Table 2: Ablation Study: Comparison of our channel reorder based clipped dynamic quanti- zation approach with KVQuant(best setting) and symmetric RTN per-token quantization in different quantization setting. For RTN-sym and our method, we set group-size to 64. Perplexity is Llama-2-13b test on Wikitext-2 with sequence length 4096.
also benefit from a finer-grained group, and achieve almost lossless compression, which is shown in Section 4.3.
Needle in Haystack Results. For needle in haystack test, we used Llama2-7b-80k (Fu et al., 2024) model for our experiments. We set the context to grow from 1K to 32k for a total of 20 intervals, and for each context length, we insert the needle into 15 different positions of the context. We compare SKVQ with KIVI under the setting of group size 128. For SKVQ, we set the window size to 128 and reserve 5 attention-sinks, i.e,, when the first 5 token cache pairs slide out of the sliding window, they are retained to full precision instead of quantized to 2 bits. The residual length in KIVI is set to 128. We follow the method in (Fu et al., 2024) to calculate the recall, and finally average the scores of test cases as the overall score. As shown in Figure 5, in key cache 2bits, value cache 2bits, group size 128 setting, KIVI got 244.5, while our SKVQ achieved 272.2 even with 2 bits key cache and 1.5 bits value cache in group size 128.
These results demonstrate that it is practical to quantize the key-value cache into extremely low-bitwidth for these tasks. More result on needle in haystack test can be found in Appendix7. 7.
4.3
Ablation Study
In this section, we decompose each part of SKVQ separately in detail and study the effect from each technique and different parameter settings.
Method
Avg Scoret
Breakdown of different components of SKVQ. We investigate the accuracy impact of various quantiza- tion techniques employed in SKVQ. Initially, we utilize
FP16
48.66
RTN
35.55
RTN and adopt por-token quantization with a group size of 32. Subsequently, we apply other quantization techniques used in SKVQ, including sliding window, clipping, channel reorder, attention sink, and FP8. The LongBench average scores are presented in Table 3. The attention sink size is set to 5, meaning the first 5 token cache pairs are retained at full precision. Our results indicate that windoww and channel re-
+ Window-128
45.73 (10.181)
+ Group Clipping
46.44 (0.71
+ Channel Reorder
47.99 (1.55f)
+ Attention Sink
48.14 (0.151)
FPS(F4M3)
48.04 (0.14)
Table 3: Ablation Study: The perfor- mance gain or loss by applying each technique SKVQ based on RTN method. Quantization setting: kv 2bits with group size 32.
8

---

Published as a conference paper at COLM 2024
Misttal--D-instruuct-v..22
Liama2-7b-chat
46.0
25.75
45.5
35.50
25.25
45.0
.000
44.5
34.75
44.0
34.50
43.5
43.0
34.00
WD
w1
14.2
W164
w128
w1
W32
W66
w126
Figure 6: Ablation Study: Average score of Mistral-7D-Instruct-0222 on LongBench under different window sizes. Quantization setting: KV cache 2bits with group size 128
order techniques significantly enhance accuracy. FP8 (F4M3) represents the of the FP8 datatype store per-group quantization parameters, such as scale and zero-point. In our study, using FP8 results in relatively minor accuracy decrease compared FP16. However, at extremely low-bitwidth and fine-grained group sizes, using FP8 to store quantization parameters sig- nificantly reduces the average number of bits. For example, with KV 2-bit quantization and group size of 32, using FP16 to store quantization parameters results in an average bit count 2 2/32 = 3, whereas using FP8 results an average bit count of 2 + 82/32 2.5, which is 16.7% reduction.
The effect of clipped dynamic quantization with channel reorder. To further demonstrate the effect of our quantization approach without sliding window, we perform evaluation by measuring Llama-2-13b perplexity on wikitext2. The result in Table shows that by only applying channel reorder based clipped dynamic quantization, we have outperformed KVQuant (Hooper et al., 2024). We set group size to 64 and use FP8 to store quantization parameter that the average bits is equal to asymmetric approach adopt in ATOM and FlexGen. We also reserve the first tokens to FP16 as attention sink. It worth noting that are comparing the best full-precision, which results in higher storage overhead than SKVQ.
of KVQuant iee. nuq with 1% outliers retained to
The effect of window size. To further investigate the effect of sliding window on the final results, we set up different sizes of windows and tested them on LongBench, and the average scores are shown in the Figure 6. The result shows that the average score increases as the window size increases. In general, different sub-tasks can all benefit more or less from the sliding window strategy, and the extra overhead brought by window with size of about 128 is negligible in long context scenarios, so we use window of size 128 in the main experiments.
The effect of group Size. We vary group size from 128 to 32 test SKVQ on LongBench, the average score on LongBench is as shown in Table 4. It shows that SKVQ can always benefit from finer-grained group. While finer-grained group brings better accu- racy, it increases the computation overhead for quanti- zation/ dequuntizainniio and storage overhead for quan- tization parameters, which noted as average bits. Since the performance of SKVQ on various tasks does not drop significantly when the group size is set to 128, we employ 128 group size in the main experiments.
Group size
Avg Score
Avg Bits
35.365
2.125
35.805
2.25
32
36.51
2.5
Ablation Study: age scores Mistral--b-Instruct--o0 on GovReport and MultiFieldQA-zh dataset for different group sizes. Quan- tization setting: KV cache 2bits, win- dow size 128.
Table 4:
Aver-
9

---

Published as a conference paper at COLM 2024
5 Conclusion
In this paper, we achieve accurate ultra-low precision KV cache quantization. By channel reordering, we group similar channels together, and apply group clipping further mitigate the outlier problem. We propose a sliding window quantization strategy with filter rules, which greatly improves the performance of the KV cache quantization method on long context tasks by reserving small portion of the cache to full precision. By combining theses two approaches, we successfully quantize the KV cache to Key 2bits value 1.5 bits without significant precision loss. We believe this work will further advance the design of mixed sixee-ppecisson quantization strategies for KV cache In the future, we will further optimize the filter rules and the kernel implementation.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint ar arXiv:2303.08774, 2023.
Yushi Bai, Xin Lv, Jiajie Zhang, Hong Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. ArXiv, abs/2308.14500 2023. URL https: /api. semanticscholar wrw.llarrrrr//ooossDDDD 261.:s4II:::225222220 261245264.
Iz Beltagy, Matthew F. Peters, and Arman Cohan. Longformer: The long-documeet trans- former. ArXiv, 2505000..55100, 2020. URL https://api. swwanticscholar org/CorpusID: 215737171.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric Xing, Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023, URL /oKL:tttps///nnss.o////////2/22
Tri Dao. Flashattention-22 Faster attention with better parallelism and work partition- 2023. URL https://api. www.ntisscoooaat.... org. org/CorpusID:
ing. ArXiv, 259936734.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm Lnt.(n 8-bit matrix multiplication for transformers at scale. ArXiv, abs/2208.07533 2022. URL
https://api. semanti ortg/CorpssD:221555555555
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post- training quantization for generative pre-trained transformers. ArXiv, abs abbs/2210..732,, 2022. URL https: //api: semanticscholar org//orppsIID (0:0Duu2I::223332880 253237200.
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hanna Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. ArXiv, abs/2402.10177, 2024. URL / wewppiisstatnnncooaaaaaaaaa wet.aaiiiiccc.....ooorrrrrr.reaa 2676823611.
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei
Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. ArXiv, ab3/2310.01801,, 2023. URL https: / /api. wwwanticscholaattt.eeeeeieooeeee htt.:/ccho.a....rrrrooooosl//e/e :tt.rpuuuII:22666990000 263609075.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh,, Michael Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length Ilm inference with kv cache quantization. Xrv preprint ar Xii::2401.18079, 2024.
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Deven- dra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. ArXiv, abs/2310.06825, 2023. URL https: / /api. semant wwwttniccchharrroooooo CorpuSID:263830493
10

---

Published as a conference paper at COLM 2024
Greg Kamradt. Needle in a haystack pressure testing llms. https://githhhi http:/hww...o////araaaaaaaa LLMTest Seleest.Needeenntaaytttcck 2023.
Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantiza- tion. ArXiv, Abs/2306.07666 2023. URL https://api. semanticcchoallaaaa wttpresscccco.aoooorooriiiii org/CorpusID: 259144954.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of bert. ArXiv, abs/1900.0888 08593, 2019. URL https://api. www.ppsssaaaataaaaaroooomm org/ CorpusID: 201 2416445
Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, lon Stoica, Xuezhe Ma, and Hao Zhang. How long can open-source llms truly promise on context length?, June 2023. URL http::///mmmss..rg/loog///////// 06- 26-29-
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. ArXiv, abs/2306.00978, 2023. httpst stw.ppi.semmannccoollseee wtw.amttticcccc...ooooooooooooor 258999941..
Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anasta- sios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for Ilm kv cache compression at test time. ArXiv, 2023. URL https: / /api..pp.... www.nticscholaaa.a.eee wrggCorppssDD htt.:a/rr....//rrroo...22228255 258947558.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free fiinng-fee asymmetric 2bit quantization for kv cache. Xiv preprint ar arXto:240..02750 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference Learning Representations, 2016.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. In The Twelfth International Conference on Learning Representations, 2023.
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with single gpu. In International Conference on Machine Learning, pp. 31094-31116. PMLR, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint 2rXi::2312.118000 2023.
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Subramanian, Xia Tan, Binh Tang, Ross Toy Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/ 2abs/2300 039888,,88,,, 2023. URL https: /api.. 259950998.
org/CorpusID:
11

---

Published as a conference paper at COLM 2024
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems, 35: 17402-17414, 2022.
Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression4: Accurate quantization of large language models by equivalent and effective shifting and scaling. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. ArXiv, abs/2211.10438,, 2022. URL https://api seap.issaaaniscoolarrr
wowpo//rrrrrg/oouuuiiio..o 253708271.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. ArXiv, abs/2309.17453, 2023. URL https://api.
Zhihang Yuan, Lin Jia-Wen Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models. ArXie, abs/2304.01089, 2023. URL https:
//ap... wtw.riaoooo..oor///ruosss....... 257913374.
Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, et al. Llm inference unveiled: Survey and roofline model insights. ar Xiv preprint ar XrXi:22022.66333 2024.
Yuxuan Yue, Zhihang Yuan, Haojie Duanmu, Sifan Zhou, Jianlong Wu, and Liqiang Nie. Wkvquant: Quantizing weight and key /value cache for large language models gains more. arviv preprint ar Xrv: 2024.
Zhenyu (Allen) Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark W. Barrett, Zhangyang Wang, and Beidi Chen. H2o: Heavy-hitter oracle for efficient generative inference of large language models. ArXiv, abs/ 14048, 2023. URL https: //api. semanticscholas org/CorpusID: 259263947.
Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate 1lm serving. ArXio, abs/2310.19100 2023. URL https://api. semanticscholar. wtgp/CoopusDDD22488888.
12

---

Published as a conference paper at COLM 2024
6 Detailed Implementations
We describe our algorithm as shown in Algorithm 1. The subroutine wtw.peeetttttnmmaaeee and get gett-rrupp-ae group.clipping described Section 3.1. It's worth noting that the prologue only needs to be executed once before deploying, and we do not pay for it during the inference phase.
Algorithm SKVQ Algorithm
SKVQ Parameter: window size W, group size G, filter rules < ffrffr.... processed KV cache length processed <-
Attention Module Parameter: Wa. Wg, Wo Rdxd
Prologue:
Px, Po, group indices t get permutation matrix(calibration set) clipping f- get group clipping(calibratiin set, P, group indices)
Wi + Px Wt
W. P. W.
end
Input: X E Rixd, Kaache caache E Rhxd where h is context (eneth phase h = 0),
I is current input length(prefill phase = len (prompt), decode phase! | 1)
Algorithm algo (Attenttn module with SKVQ algorithm): W.
Q = Wn. K X W
Keache dequant( cache)
Veache - dequant( caache)
Kaache + concat( Keacher K)
V Vacche < concat V cache.
S + reorder cacch)
4 S reorder( Vaach)) W.
cixlen <- len Veache)
indices + processed cfx W]
if indices Ø then
kmask + processedd: ctx len W; Falsel
vmask processed ctx_len W; False)
for filter in F do kmask filter(Kaachan Keache [indices] A kmask
vmask + filter( V cache indices] A vmask end
K Kaache [indices]
Siiping.quant(KKacrk [inddiiiess kmask)
V cachh [indices] processed += len(indices) end return O
llippingquantee Cacant(Vaache cache [indices), vmask)
end
function clipping tlipingg-quantXX (X, mask):
groups + split groups group indices) quant cache + Ø for group in groups do
minmax(gruop) group_min + hhpppinggroopp x group-min
mooup_min.groupp..t
group_max clippingigrous group) x group_man
quant sront--rrup--- + croup-quant(ge)) group min, group-max group)
coantt-roupmmmss +- concat(quant cache, qaaeee.guat.geee group)
grouplindices
end
return quant cache end
13

---

Published as a conference paper at COLM 2024
7 More Experimental Results of Needle in Haystack
KIVI
K2V2-g64: 264.2
K2V2-g128: 245.0
K2V1.5-g120: 235.1
124
10
70
80
64
80
Limit
Limit
SKVQ(Ours)
K2V2-g64: 270.5
K2V2-g128: 271.2
K2V1.5-22 5-g188: 272.2
18
10
70
36
30-
655
88
113
Limit
Figure 7: Comparison of SKVQ with KIVI on 32k context length needle in haystack test. The baseline score is 268.5. We vary the group size from 64 to 128, and vary the quantization bits from (key 2bits, value 2bits) to (key 2bits, value 1.5bits).
The results in Figure 7 show SKVQ is clearly better than KIVI, especially in K2V1.5-g128, SKVQ achieve the same level with FP16, while KIVI suffers significant accuracy loss. These results shows the robustness of our approach.
8 More Results of LongBench Evaluation
We also evaluated LongChat-vl BongChat-v1.577b-333 .5-7b-32k and Vicuna-v1.5-7b-11 which are two famous long-context models fine-tuned based on Llama2-7b. The results in Table demonstrate that our SKVQ outperformed previous methods, which highlight the generalizability of our approach.
Model
Method
LCC
RepoBench-P
PR-en
TREC
2wikimqa
GovReport
MQA-zh
Average
FP16
51.38
46.18
4.5
69
21.3
27.79
43.74
41,02
SuanthQuant
40
29.27
1.94
18:25
8.33
14.86
7.19
22.37
RPTQ
40.64
41.4
3.75
39.5
15.92
23.16
19.41
32,68
KIVI
49.32
43.35
5.56
68
24.47
38.86
39.19
SKVO
50.98
44.07
69
22.04
26,55
40.82
40.20
FP16
54.89
59.05
30.5
66.5
24.58
30.89
35.33
47.27
RTN
5.11
3.73
1.5
0
0.42
0.51
0.08
2,461
LongChat-vl -157-----k
SuanthQuant
36.21
31.91
2.45
36.5
13:94
17.21
6.59
24.70
KIVI
49.86
54.77
20.5
66
23.79
28.75
31.58
43.22
SKVQ
55.01
57.24
22
67
22.4
30.03
31.68
45.37
Table 5: Evaluation resultes of Vicuna and LongChat on LongBench. Group-size(average)) 128, key-cache 2bit, value-cache 2bit, window-size 128. We abbreviated PassaagReerieeal as PR and MultiFieldQA as MQA. We highlight the result of our method,
9 Memory and Latency Analysis
In order to further illustrate the benefits of quantizing KV cache to extremely low-bitwidth, we use LLM-Viewer (Yuan et al., 2024) to analyze the benefits in terms of memory consump- tion and inference latency. The result is shown in Table 6. When batch size and sequence
14

---

Published as a conference paper at COLM 2024
length are relatively large, KV cache dominates almost all the memory consumption, and load KV cache becomes the performance bottleneck of the entire inference system. By quantizing KV cache with SKVQ, consumption. The analysis result shows SKVQ enables 1M context length in single A100- 80GB. As for the inference latency, show the results of decoding phase, which domain the inference time in long context tasks. In the case of batch size 128 and sequence length 200k, the theoretical 7x speedup can be achieved.
significantly reduce both latency and memory
Batch Size
Seq Length
Latency(ms)
Memory(GB)
FP16
KV4
KV2
Inference Time
10.6
7.5
32k
Memory Access
21.6
15.3
14.3
Memory Consumption
29.7
17.2
15.1
Inference Time
23.1
10.8
8.7
128k
Memory Access
47.2
22
17.8
Memory Consumption
80.1
29.7
21.4
Inference Time
32.5
13.3
10
200k
Memory Access
66.3
27
20.5
Memory Consumption
118
39.2
26.1
Inference Time
274.1
76.6
43.7
32k
Memory Access
559
156
89.1
Memory Consumption
1100
282
147
Inference Time
1100
286.4
154.8
64
128k
Memory Access
2200
584
316
Memory Consumption
4300
1100
551
Inference Time
1700
443
238.1
200k
Memory Access
3400
905
485
Memory Consumption
6700
1700
853
Inference Time
541.8
146.8
81
32k
Memory Access
1100
299
165
Memory Consumption
2200
550
282
Inference Time
2100
566.4
303.1
128
128k
Memory Access
4400
1200
618
Memory Consumption
8600
2200
1100
Inference Time
3300
881.1
469.7
200k
Memory Access
6800
1800
958
Memory Consumption
13400
3400
1700
Table LLaMA-7B memory and latency analysis with roof line model. The hardware platform is A100 80G, we assume flash-attention is used.
15

---

Published as a conference paper at COLM 2024
10
Comparison Between Smooth and Reorder
To improve the accuracy of per-token quantization, we utilize the reorder to cluster similar channels together. There are also other methods improve the accuracy, one of them is smoothing, which is adopted in (Xiao et al., 2022; Shao et al., 2023; Yue et al., 2024). This approach smooth the difference between channels by multiplying an extra factor before quantization. We explore and compare smoothing with reordering, the experimental results are shown in Table 7. SKVQ-smooth represents our sliding window strategy together with smoothing and SKVQ-reorder represents the approach described in Section 3.1. The results demonstrate that reordering can effectively improve the per-token quantization performance while the smoothing cannot. This is mainly because smoothing does not take into account the differences in token dimensions.
Model
Method
LCC
RepoBench-P
TREC
2wikimqa
GovReport
MQA-zh
Average
63
27.25
HLMMA-2-7B<chat
SKVO-reorder
50.69
45.4
5.5
63
28.5
27.07
10.7
37.30
SKVO
48.93
40.12
4.75
62.5
26.75
23.19
7.93
34.77
IP16
50.54
52.1
15.25
68.5
13.21
27.52
7,23
38.83
LLaMA-2-1SB-chat
SKV SKVO-reorde
49.53
49.76
12.25
67.5
14.03
26.68
6.63
37.33
-mmothh
IT16
68.06
60.46
17.71
68
10.87
20.09
17.1
45.51
Mistral-7U
Q-reoreer
SKV V Q-smmotth
64.18
57.95
9.49
63.5
10.11
13.99
12.77
41.52
FP16
55.07
48 96.96
70
31.18
42.74
48.66
Mistral-7B-Instruct
SK V-reorder
54.86
49.05
56.42
70
20.94
30.82
42.4
46.23
SKVQ-smooth
49.83
45.74
40.42
66
17.11
28.32
30.32
42.11
Table 7: Comparison of different methods(i.e. smooth v.s. reorder) on LongBench. Group- size(average) set 128, key-cache 2bit, value-cache 2bit, window-size 128. We abbreviated P Passaae as PR and MultiFielQQA as MQA.
16


Reviews:
[{"review_id": "Ixdfz3D4W7", "reviewer": "Reviewer_QPwH", "summary": "The paper introduces a novel method, SKVQ (Sliding-window Key and Value Cache Quantization), that mitigates the memory consumption issue caused by the key-value (KV) cache in large language models (LLMs). The authors propose an extremely low bitwidth quantization strategy that deploys channel reordering, clipped dynamic quantization at group granularity, and a sliding window approach to ensure data accuracy while achieving high compression ratios. The experimental results demonstrate significant improvements over existing methods with minimal loss of precision.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1**: The full-precision prefill stage can bottleneck when the prompt length is long.\n\nQuantization is executed layer-by-layer, not all KV-cache stay in full-precision in prefill phase. Instead, there is only 1 layer's KV-cache in full-precision.\n\nAssuming prompt length is 100k,$bs=1$, for Llama2-7B with $\\text{HeadDim} = 128$,$\\text{NumHeads} = 32$, peak memory footprint in prefill phase:\n\n- FP16: $2 \\times 100000 \\times 2 \\times 32 \\times 32 \\times 128 = 50 \\text{GB}$\n\n- SKVQ: $2 \\times 100000 \\times \\left(\\frac{2 \\times 31 \\times 32 \\times 128}{9.14} + 2 \\times 1 \\times 32 \\times 128\\right) = 7.4 \\text{GB}$\n\n\nTherefore, SKVQ also **achieves a high compression rate at prefill phase**.\n\n> **Q2**:  Evaluation on long generation scenarios \n\nFirst, there is not suitable dataset for long generation task.\nTo address your concerns, we manually created some prompts with instructions to generate as long as possible (at least 3000 words). We test on Llama2-7b-chat. For each prompt, we generated responses under three settings: full precision, SKVQ, and KIVI. The SKVQ and KIVI models were quantized with K 2bits and V 1.5bits, group size set to 128, window-size for SKVQ at 128, and residual length for KIVI at 128.\n\nWe utilize GPT4-o, Gemini-1.0-Pro to judge the responses.\n\nThe data in the table represents the score of **FP16/SKVQ/KIVI** respectively.\n\n| prompt | GPT4-o   | Gemini-1.0-Pro | Claude-Sonnet |\n| ------ | -------- | -------------- | ------------- |\n| p1     | 30/34/22 | 32/40/24       | 30/36/27      |\n| p2     | 35/30/33 | 36/39/36       | 30/38/36      |\n| p3     | 26/32/19 | 18/37/30       | 9/35/22       |\n\n> **Q3**: Evaluation  on larger models?\n\nEvaluation on Llama3-70B-Instruct.\n\n|                      | FP16  | SKVQ  | Relative to FP16 |\n| -------------------- | ----- | ----- | ---------------- |\n| RepoBench-P          | 68.78 | 67.69 | 98.42%           |\n| passage_retrieval_en | 68.83 | 68.33 | 99.27%           |\n| multifieldqa_en      | 18.52 | 18.14 | 97.95%           |\n| Qasper               | 12.2  | 11.46 | 93.93%           |\n| TREC                 | 73.5  | 73.5  | 100.00%          |\n| 2wikimqa             | 18.37 | 18.05 | 98.26%           |\n| gov_report           | 32.82 | 31.96 | 97.38%           |\n| multifieldqa_zh      | 20.1  | 20.02 | 99.60%           |\n| multi_news           | 26.83 | 26.9  | 100.26%          |\n| qmsum                | 22.25 | 21.88 | 98.34%           |\n| samsum               | 44.41 | 44.78 | 100.83%          |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper tackles an important problem in deploying large language models - the memory consumption of KV caches. And the paper combines several techniques such as channel reordering and clipped dynamic quantization to minimize the quality loss.\n2. The evaluation is extensive and relatively thorough. The result shows that SKVQ surpasses previous approaches in terms of accuracy under different average bit widths.\n3. The paper is well-written and provides in-depth insights into the proposed method.", "reasons_to_reject": "1. Since the prefill stage is computed in full-precision, it could become a bottleneck especially when the generation length is short compared to the context length.\n2. It would be great if the proposed method can be evaluated in long generation scenarios, which may present different challenges than long-context scenarios.", "questions_to_authors": "How would it perform on larger models, e.g. llama-2-70b? Possibly together with tensor parallelism? It would be beneficial for future revisions to consider evaluating the proposed methods on larger models.", "extra_scores": {}}, {"review_id": "VkYpx0aDvx", "reviewer": "Reviewer_MMd8", "summary": "The paper presents a new variant of KV Cache compression for LLMs.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions on our manuscript. We find most of them to be very accurate and appreciate your careful attention. We believe your insights will significantly enhance the quality of our paper. Sincere thanks! \n\n> **Q1**: estimation about quantizing the key cache into 2 bits and value cache into 1.5 bits.\n\nThank you for highlighting this issue. In general, our estimates are based on two aspects:\n\nFirstly, we noticed that the accuracy of `KIVI`(previous SOTA) at K2V2 significantly decrease as the group-size increasing. We think this means that `KIVI` does not handle outliers within a group well. This presents an opportunity for superior performance.\n\nSecondly, our approach is driven by experimental results. As shown in Table 1 of the manuscript, with a group size of 128, key cache at 2 bits, and value cache at 2 bits, the SKVQ scores were very close to FP16. This motivated us to explore lower precision.\n\nWe will emphasize this process in the revised version of the paper. Thank you for pointing it out.\n\n> **Q2**: \"RTN\" in page 7 is unclear.\n\nYou are absolutely right; we did not explain \"RTN\" in detail in the original text. In fact, apart from \"RTN-sym\" in Table 2, \"RTN\" in our experiments refers to asymmetric per-token group quantization, which applies the round-to-nearest strategy.\n\nWe will clarify this in the revised manuscript. Thank you for pointing it out!\n\n> **Q3**: Grammar issues and other error\n\nWe sincerely appreciate your careful review and the identification of grammatical errors. We will thoroughly review the manuscript and correct these issues to improve clarity and readability. Thank you for bringing these to our attention.\n\n> **Q4**: confusion about the  phrase \"differences in token dimensions\"\n\nHere is a revised, clearer version:\n\nwe use the term \"token dimension\" to refer to the `seq_len` dimension in KV-cache. Specifically, for KV-cache with shape $[\\text{bs}, \\text{SeqLen}, \\text{NumHeads}, \\text{HeadDim}]$, we use \"token dimension\" to indicate the second dimension($\\text{SeqLen}$).\n\nWe observed that, in addition to fixed outlier channels which is observed in previous work, there is also variance between different tokens. Simply applying channel-wise smoothing does not address the issues in this dimension. To resolve this, we propose Clipped Dynamic Quantization.\n\nWe apologize for any confusion caused by our original wording, and will correct it in revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The presented compression idea is neat, and the evaluations show promising results. Ablation studies are presented that decompose the proposed approach and show the influence of its components.", "reasons_to_reject": "The paper is very condensed and would not be easy to understand for wider NLP audiences. However, for those working on KV caches, it is adequate.\n\nA major claim in the manuscript, \n\"we further quantize the key cache into 2 bits and value cache into 1.5 bits\"\nis not clear at all. \nIt is not explained in the theory part, and is mentioned as given in the experiment part. \nThis claim should be explained in detail, showing how you arrive at this estimation.", "questions_to_authors": "There are certain aspects in the manuscript that need to be improved.\n\n\"RTN\" is mentioned in tables (first time on page 7) but is not defined/explained in the text. \nIt is somewhat defined in the legend of table 2, but needs to be explained earlier in the manuscript, and in more detail.\n\nTable 1 presents evaluation, but what are the numbers that we see in it? Are those miles per hour, or salary per day? Or scores (percents) on the tasks?  Please care to define what are the values we see (what are the units) and how the scale should be interpreted (higher or lower values are better?).\nAlso mention that in the relevant part of the text.\n\nGrammar issues:\nThe word 'but' is used in several places where it is not grammatical. e.g. page 5: \"large amount of but less important content\", \"small portion of but more important caches\".\nAnother error, page 2: \"we discover that the protecting the accuracy\" (drop 'the' after 'that').\n\nPage 3: \"these methods do not take into account the differences in token dimensions.\"\nThis is not quite clear. If you are talking about text tokens, their dimensions are fixed via embeddings dimension. So what tokens are you talking about and how do their dimensions differ?  This should be clarified.\n\nThis publication needs to be cited, as it as also about sliding window  approach:\nhttps://arxiv.org/abs/2401.01325.\n\nPage 8 (bottom of page):\n\"cache is 2 + 16 ∗ 2/32 = 3, but if we use FP8, then average bits is 2 + 8 ∗ 2/32 = 2.5, which is 50% smaller.\" \nThis is incorrect, 2.5 is not 50% smaller than 3. it is only 17% smaller than 3.", "extra_scores": {}}, {"review_id": "H0eYM8PX5u", "reviewer": "Reviewer_nGVZ", "summary": "This paper proposes SKVQ, a version of KV-cache quantization that utilizes two further ideas to improve quality and efficiency: a recent-token-keeping mechanism and a channel reordering schema coupled with dynamic quantization. Each contribution is evaluated in ablation studies and in a fitting setup, with the conclusion being that SKVQ outperforms existing methods and should be adopted for LLM generation.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful review that will help us strengthen the manuscript.\n\n> **Q1**: What value of K did you choose for KMeans (p.4 center)? Did this choice make a difference?\n\nThank you for pointing this out, we will explain this part in detail in the revised paper. \n\nTo ensure a fair comparison, we set $K$ to ensure that the average group size for SKVQ matches the group size in other methods. \nFor example, with Llama2-7b-chat with **MHA** ($\\text{NumKVHeads} =\\text{NumHeads} = 32$, $\\text{HeadDim} = 128$), for the main experiment setting of \"group-size 128\", we set $K = 32$. Hence, total $\\text{NumKVHeads}\\times\\text{HeadDim}$ channels are divided into 32 clusters, each considered as a quantization group. Although each group size varies, the average group size is $\\text{NumHeads} \\times \\text{HeadDim} / K = 128$.\n\n> **Q2**: Why do you claim to have outperformed KVQuant in p.8 when Table 2 shows otherwise?\n\nPlease note that Table 2 on page 8 reports the model **perplexity(PPL)** obtained on Wikitext-2, where **lower values are better**. In the 4-bit setting, `Ours` achieves a PPL of $4.60$ while `KVQuant` achieves $4.59$, which is slightly better. However, it worth noting that the average bits of `KVQuant` is also larger than `Ours` due to the storage overhead caused by outlier index.\n\nWhen it comes to 3-bits and 2-bits, `Ours` achieved a lower PPL than `KVQuant` while maintaining lower average bits.\n\n> **Q3**: What do the colors in Figure 5 signify? There's no legend.\n\nWe sincerely apologize and appreciate you pointing out this issue. colors closer to green indicate higher scores for the model at that test point, while colors closer to red indicate lower scores.\n\nWe overlooked the legend when creating the plot. We acknowledge that the current figure is not friendly to colorblind readers. We will reconsider the result display in the revised paper and add appropriate text explanations. Thank you for your valuable suggestion.\n\n> **Q4**: Grammar issues and other bugs in paper.\n\nWe sincerely apologize for the inconvenience caused by these detailed errors. We agree that authors should ensure grammatical errors do not distract reviewers. We will correct the careless mistakes you pointed out and carefully recheck each paragraph. \n\nThank you for your meticulous feedback again!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The motivation is clear, the idea is sound, and the experiments are convincing.\n* The argument structure of the paper is good.\n* The amount of technical detail in section 3.1 is just right.", "reasons_to_reject": "Nothing that merits rejection, but there are **a lot** of grammatical errors in this paper. Frustratingly enough, they are localized to (a large number of) specific paragraphs (see below), which raises the concern that at least one of the authors has sufficient competence in English and could have easily made a pass over the malformed paragraphs but didn't bother. This is upsetting as it is not the reviewers' role to serve as copyeditors (and it distracts from evaluating the content).", "questions_to_authors": "### Questions\n* What value of K did you choose for KMeans (p.4 center)? Did this choice make a difference?\n* Why do you claim to have outperformed KVQuant in p.8 when Table 2 shows otherwise?\n* What do the colors in Figure 5 signify? There's no legend.\n\n### Required fixes\n* The figures require color display to make out, especially 1, 5, and 6 (colors being referenced from the text). This is inconsiderate towards colorblind folks and anybody printing the paper grayscale. Please make the necessary adjustments.\n* Some notions are mentioned without proper introduction, for example \"locality of attention\" in page 2 (only elucidated in p.5)\n* h is defined twice, in eq. 2 and just several lines above, for no apparent reason\n* The choice to highlight the performance of your own method in the tables, as opposed to bolding teh best-performing methd, is misleading. Your method has its own row and doesn't need highlighting. Please revert to conventional practice.\n\n### Grammar\nAs mentioned above, there are many grammatical errors in the paper, mostly with respect to determiner placement (\"a\" / \"an\" / \"the\") but also verb-subject number agreement, some peculiar phrasings (\"combined together\"; \"a chance\" where \"an opportunity\" is meant), and hyphenation.\nThere are also many spacing issues, mostly missing spaces before parentheses (especially those of citations).\n\nFoci of errors include:\n* the abstract\n* the last two paragraphs of the introduction\n* the first two of section 2\n* the first of 3.1\n* \"we perform...\" on p.4\n* footnote 2\n* most of p.5, until the end of section 3\n* first sequence of 4.3 (which is not a sentence)\n* the next three paragraphs (what does \"adopt in ATOM\" mean?)\n* Figure 6's description\n* \"the effect of group size\" paragraph\n\nJust go carefully over the whole thing, please.", "extra_scores": {}}, {"review_id": "HmB7Xfxgao", "reviewer": "Reviewer_KwiV", "summary": "This paper proposes a KV Quantization method to address the memory overhead encountered when processing long contexts in LLMs. The method, called SKVQ, combines traditional KV-cache compression strategy via a sliding window with new quantization techniques like channel re-ordering and group clipping. This approach demonstrates superior performance on LongBench, Needle in a Haystack, and PPL benchmarks compared to other KV Quantization methods (e.g., KIVI, KVQuant), even at low bit levels such as 2-bit or 1.5-bit.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1**:  Evaluation on long generation scenarios \n\nOur space is very limited, please refer to the reply to `Q2` of `QPwH`\n\n> **Q2**: novelty\n\n- sliding window: We are the first to **thoroughly exploring the potential of attention locality in KV-cache quantization** and designing an algorithm that seamlessly fuses it with KV-cache quantization.\n\n- AWQ use auto-clip for weight quantization. Our research discovered that **approximating and precomputing clip thresholds on a small calibration set effectively addresses the clipping challenge** for KV-cache and effectively improves accuracy.\n\n- KVQuant does not employ channel reordering. SKVQ conducts **per-token uniform quantization** by extracting value distribution features from each channel.\n\n> **Q3**: Start Token Retention in Attention Sink\n\nThe value here is usually an empirical setting. In our experiments, we set this number to `5`(which is `4` in streaming-llm).\n\n> **Q4**: Overhead in Prefill Stage for Very Long Contexts\n\nOur space is very limited, please refer to the refer to the reply to `Q1` of `QPwH`\n\n> **Q5**: Effectiveness of H2O\nThere are bunch of details to implement heavy hitter detection in our framework. But the rebuttal space is limit to 2500. We look forward to continuing our discussion in discussion period.\n\nHere are some basic results. **K3V2** setting, group size set to 32, window 32, `H2O` represent enabling the heavy hitters in filter rules, while `Random` represent randomly retain some token cache to full-precision. We set the ratio of full precision to **4%**.\n\n| Setting | RTN | H2O | Random |\n| ------- | ------------------ | ------ | ------ |\n| K3V2    | 257.41 | 261.58 | 261.91 |\n\n> **Q6**: RoPE and Quantization? ternary quantization?\n\n1.5bits means ternary quantization. We perform pre-RoPE quantization:\n\n- Quantization Step:  QKV_projection $\\Rightarrow$ channel-reorder $\\Rightarrow$ quantize $\\Rightarrow$ store together with `past_kv`\n\n- Decoding Step: load quantized KV $\\Rightarrow$ dequant $\\Rightarrow$ channel-reorder $\\Rightarrow$ RoPE $\\Rightarrow$ Attention\n\n> **Q8**: fake quantization? Kernel and Memory Storage Details\n\nAll the results are obtained **through fake quantization**.\nwe do have custom kernel, there is no more space to describe it in detail, we look forward to disccusion in discussion period. Basically, we adjust the output channels of the weigh to avoid explicit reordering, pack every four adjacent numbers within the same group into one `uint8` for storage.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Despite being a KV Quantization method, the sliding window quantization framework opens opportunities to incorporate insights from existing KV cache eviction methods (e.g., H2O), providing a novel and impressive aspect.\n- The evaluation in long-context scenarios, such as LongBench, (not just MMLU, CSQA, ...) where KV optimization is critical, makes this paper a relevant contribution for KV cache compression research.\n- Detailed ablation studies (examining channel re-order and group clipping effects, window size sweep experiments, and task-specific analyses) reinforce the method's effectiveness and provide additional insights to readers.", "reasons_to_reject": "- The evaluation would have benefited from testing long generation scenarios in addition to long-context scenarios. During the prefill stage, FP is used to calculate the KV Cache, making KV embeddings relatively robust to quantization. However, as the generation length increases, the impact of quantization on KV embeddings created during the generation stage remains unexplored in this paper. It would have been beneficial to assess whether SKVQ remains robust in scenarios where LLMs generate long texts, such as agent-based conversations.\n- The method has limitations in terms of novelty. The sliding window approach (from KV cache eviction works), channel re-ordering (from KVQuant), and group clipping (from AWQ) are all derived from existing methodologies. This reduces the originality of the SKVQ method.", "questions_to_authors": "- **Start Token Retention in Attention Sink**: Is the number of start tokens retained determined heuristically, or was there a specific criterion used? \n- **Overhead in Prefill Stage for Very Long Contexts**: Considering scenarios like NarrativeQA in LongBench with extremely long input contexts (over 80K), was there any notable overhead during the prefill stage when using FP to calculate KV embeddings? It seems likely that memory consumption would be significantly high in these very long contexts.\n- **Effectiveness of H2O**: It was mentioned that the impact of H2O was minimal. Could you clarify this numerically? \n- **1.5-bit Quantization**: Does the 1.5-bit specification imply the application of ternary quantization? \n- **Positional Encoding and Quantization**: There was no mention of Rotational Positional Encoding (RoPE). When considering RoPE's rotating nature, is the quantization applied pre-RoPE like in KVQuant, or is channel re-order applied after RoPE has been integrated? This missing information is crucial for understanding the interaction between RoPE and the quantization process.\n- **Quantization Kernel and Memory Storage Details**: You mentioned storing quantization parameters in FP8. Are you using a specific quantization kernel for this, and when storing KV cache at 2-bit and 1.5-bit, is fake quantization employed? More details on the kernel implementation would be informative, especially regarding how memory efficiency is achieved.", "extra_scores": {}}]

Author rebuttal:

Meta-review: Quality
- Pro
  - Convincing results, well-motivated
  - Technically and experimentally rigorous and thorough
- Con
  - Too technical for NLP audiences

Clarity
- Con
  - minor grammar issues
  - explanations of key concepts and ideas could be clearer
  - notation and results not well explained
  
Originality
- Pro
  - novel method that augments other approaches

Significance
- Pro
  - Effective solution for a highly relevant practical problem
  - novel and likely good impact on the community and field

**Decision**

All reviewers agree that this work presents a novel, effective solution that is well-supported by the rigours and extensive experiments. The only concern that almost all reviewers have is with clarity. Some concepts, notation are unclear and there are other issues like grammar. Overall, this is a clear accept. I recommend that the authors to work on clarity of the draft for the paper-ready to increase the impact of this work.
Decision: Accept