BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments
Authors: Xinghao Wang
Pengyu Wang
Bo Wang
Dong Zhang
Yunhua Zhou
Xipeng Qiu
Venue: ICLR 2025 Poster (2025)
Keywords: large language models; compression; decomposition
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: A novel,  training-free weight compression approach that enables megabyte-level trade-offs between memory usage and model performance.

Abstract: Large language models (LLMs) have revolutionized numerous applications, yet their deployment remains challenged by memory constraints on local devices. While scaling laws have enhanced LLM capabilities, the primary bottleneck has shifted from $\textit{capability}$ to $\textit{availability}$, emphasizing the need for efficient memory management. Traditional compression methods, such as quantization, often require predefined compression ratios and separate compression processes for each setting, complicating deployment in variable memory environments. In this paper, we introduce $\textbf{BitStack}$, a novel, training-free weight compression approach that enables megabyte-level trade-offs between memory usage and model performance. By leveraging weight decomposition, BitStack can dynamically adjust the model size with minimal transmission between running memory and storage devices. Our approach iteratively decomposes weight matrices while considering the significance of each parameter, resulting in an approximately 1-bit per parameter residual block in each decomposition iteration. These blocks are sorted and stacked in storage as basic transmission units, with different quantities loaded based on current memory availability. Extensive experiments across a wide range of tasks demonstrate that, despite offering fine-grained size control, BitStack consistently matches or surpasses strong quantization baselines, particularly at extreme compression ratios. To the best of our knowledge, this is the first decomposition-based method that effectively bridges the gap to practical compression techniques like quantization. Code is available at https://github.com/xinghaow99/BitStack.

Published as a conference paper at ICLR 2025
BITSTACK: ANY-SIZE COMPRESSION OF LARGE LAN- GUAGE MODELS IN VARIABLE MEMORY ENVIRON- MENTS
Xinghao Wang', Pengyu Wang', Bo Wang', Dong Zhang', Yunhua Zhou Xipeng Qiu² Fudan University, 3Shanghai Artificial Intelligence Laboratory (xinghaowang22, pywang24, bwang22, dongzhang22 (m. (6m. fudan. edu, on zhouyanhia@pjlab org. en, xpqiue fudan edu. en
ABSTRACT
Large language models (LLMs) have revolutionized numerous applications, yet their deployment remains challenged by memory constraints on local devices. While scaling laws have enhanced LLM capabilities, the primary bottleneck has shifted from capability to availability, emphasizing the need for efficient mem- ory management. Traditional compression methods, such quantization, of- ten require predefined compression ratios and separate compression processes for each setting, complicating deployment in variable memory environments. In this paper, we introduce BitStack, novel, training-free weight compres- sion approach that enables megabyte-level trade-offs between memory usage and model performance. By leveraging weight decomposition, BitStack can dynam- ically adjust the model size with minimal transmission between running mem- ory and storage devices. Our approach iteratively decomposes weight matrices while considering the significance of each parameter, resulting in an approxi- mately 1-bit per parameter residual block in each decomposition iteration. These blocks are sorted and stacked in storage as basic transmission units, with dif- ferent quantities loaded based current memory availability. Extensive periments across a wide range of tasks demonstrate that, despite offering fine- grained size control, BitStack consistently matches surpasses strong quanti- zation baselines, particularly at extreme compression ratios. To the best of our knowledge, this is the first decomposition based method that effectively bridges the gap to practical compression techniques like quantization. Code is available at httpss:/ thub. com/ x x nghaow 9 tSttcck
Zaro-shot Parformanc
0.6
0.5-
AV
AVC
KWCa
3PTG
3002
4000
0000
10000
15003
Low Memory Availobility
High labilibylity
Memory (MB)
(a)
(b)
Figure 1: BitStack enables dynamic compression of LLMs in variable memory environments a. while still matching or surpassing the performance of practical compression methods such as GPTQ (Frantar et al. 2022) and AWQ (Lin et 2024) with the same memory footprint(b)
"Corresponding Authors

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Large language models (LLMs) have demonstrated superior performance on various bench- marks (Achiam et al. 2023 Dubey et al. 2024) and are increasingly serving as practical assistants in people's daily lives, such as general language assistants (OpenA1, 2024 Google 2024 Anthropic
2024). search engines (Perplexity. Al 2024). and code assistants (GitHub 2024).
With the blessing of scaling laws (Kaplan et al.) 2020), LLMs are becoming more powerful as their sizes expand, and the main bottleneck for deploying task-capable LLMs has shifted from their capa- bility to their availability. For example, loading only the weights of the Ilama 3.1 8B model (Dubey) ct al. 2024) requires approximately 14.96 GB of RAM in FP16, not including the activations, which also consume significant memory during inference, especially for long-context tasks.
To adapt to various memory and device constraints, numerous methods have been proposed for LLM compression. such quantization (Frantar et al.) 2022 Lin et al.) 2024 Shao et al. 2023 Egiazar- ian et al., 2024] Tsengg et al. (2024), pruning (Ma et al. 2023 Xia et al. 2023|
Ashkboos et al.
2024), and distillation (Muralidharan al., 2024). These methods often compress models to pre- defined compression ratio (e.g.. specifying numerical precision, defining target structures for pruned models student models) and require running the compression procedure from scratch for every compression setting. Another line of research for compressing LLMs is weight decomposition (Hsu al. 2022 Yuan et al., 2023 Wang et al., 2024). These methods compress the model weights via low-rank decomposition but often suffer from severe performance degradation at high compression ratios.
Deploying large language models locally(e.g. on personal computers or smartphones) is common practice, as safeguards private data and enables offline functionality. However, the available RAM on these devices is often limited and variable, the total memory capacity is generally small and memory usage by other applications can fluctuate(Figure [aap. This variability in available memory poses challenge for deploying LLMs, as they require consistent and substantial RAM resources. For example, when more memory becomes available from other applications, users may want to use b-b quantized model instead of 3-bit one for better performance. However, this requires reloading the entire model, which may cause significant delays due to limited transmission band- width. Additionally, multiple versions of the model at different compression ratios need to be stored on the device, and each version requires running a separate compression process in advance, which increases the storage burden on the device and requires additional computational resources to run separate compression processes. Therefore, a compression strategy that enables dynamic tradc-offs between memory usage and performance is highly desirable.
As discussed earlier, achieving these trade-offs requires avoiding compressing towards a fixed ra- tio. Instead, we aim to compress the model once, allowing it to be dynamically loaded within any arbitrary memory budget, which leads us to weight decomposition. However, previous studies on weight decomposition for LLMS failed to match the performance with practical methods like quanti- zation (Hsu et al. 2022: Yuan et al. 2023 Wang et al. 2024). To tackle this challenge, we propose a novel training-free free, dccomposition baasdd weight compression approach called BitStack, where we decompose the original weight matrices and iteratively decompose the residuals from the previous approximation. In the decomposition process, we account for the unequal importance of weights (stemming from the high variance in activation channel magnitudes) by scaling the weights before decomposition. We then iteratively apply singular value decomposition (SVD) to decompose the magnitude of the matrices (or residuals) into vectors while preserving their sign matrix, yielding an approximately 1 bit of memory per parameter residual block in each iteration. Subsequently, the residual blocks for different weights across various layers are universally sorted and stacked based on their importance to overall performance at the current memory level, stored as basic trans- mission units in storage. Weight matrices are also treated as stacks, progressively approaching the original matrices as more blocks are added. In this way, BitStack enables memory perforrance trade-off for LLMs by dynamically loading or offloading residual blocks between running mem- ory and storage devices, making LLM deployment feasible in variable memory environments. We conduct extensive evaluations on BitStack across a wide range of tasks, demonstrating that, despite its capability to deploy in variable memory environments, BitStack consistently matches or sur- passes the performance of widely adopted compression methods like GPTQ (Frantar et al., (022) and AWQ (Lin al. 2024), especially at extreme compression ratios(Figure Tb). To the best of
2

---

Published as a conference paper at ICLR 2025
Available Memory Increases
Avai Lable Memory Decreases
Model Layer
Stacked Residual Blocks
Model Layer
Stacked Residuol Blocks
(In Menoryhh
(In Storage
CIn Menorya)
(In Storoge
Figure 2: Overview of BitStack. BitStack dynamically loads and offloads residual blocks (Figure) 3 between RAM and storage devices based on current memory availability. We can load more weight residuals from storage when available memory increases a offload them otherwise (b). The residual blocks for all weights across all layers are universally stored in the same stack on the storage device (grey blocks denote residual blocks for weights in other layers). Note that omit positional embeddings, normalization layers, and residual connections in the figure for clarity.
our knowledge, BitStack is the first decomposition-based method that closes the performance gap between decomposition-basee methods and quantization-base methods.
Our contributions can be summarized as follows: (1) We identify the challenge of deploying LLMs in variable memory environments, which existing model compression methods can not handle. (2) We propose BitStack, a training-free decomposition- based method for model compression that en- ables megabyte-leve
trade-off for modern L.MMs. (3) We conduct extensive experiments Llama 2, Llama 3, and Llama 3.1 models, ranging in size from 7/8B to 70B, demon- strating that BitStack matches or surpasses the performance of practical quantization- base- lines, particularly at extreme compression ratios.
seemory-performatt
BITSTACK
overview BitStack is illustrated in Figure 2 BitStack is able to dynamically adjust the size of each weight matrix based the available memory capacity at the time. When more RAM is freed by other applications, we can retrieve additional residual blocks from pre-sorted stack and load them into RAM. Conversely, when memory becomes limited, from the model weights (also stored as stacks) back to storage devices in reverse order, ensuring the system remains functional. In the following subsections, procedure for each weight (or residual) matrix Section [2. and then explain how we sort the residual blocks be pushed into the universal stack in Section 2.2 comprehensive overview of BitStack is provided in Algorithm ]
offload residual blocks
first introduce the decomposition
2.1 DECOMPOSING WEIGHTS IN LLMS
In weight decomposition, the objective is to break down weight matrices into sub sub-matrices to reduce the total number of parameters, with the ability reconstruct the full matrices during inference. Singular value decomposition (SVD), in particular, is a widely-used and effective method for matrix decomposition due to its ability to capture the most significant components of the weight matrices. Formally, let W mpmx" be weight matrix in linear layer, decompose W via SVD by:
W = USVT Σ a.ano
where d = min(m, n). 2 ... 2 aa are the singular values W, and u, and v; are the corresponding left and right singular vectors, respectively. We then obtain rank-k approximation
3

---

Published as a conference paper at ICLR 2025
=
Residual Block
Sign Matrix & Singular Vectors
Sign Matrix Packing
Figure 3: Illustration of residual block in BitStack. A residual block consists of sign matrix and singular vectors obtained through absolute value decomposition. The sign matrix can be packed into GPU-supported data types minimize memory usage. Mlll denotes the sign matrix while E denotes the packed sign matrix.
of W:
W = V =
Σ
o
-VVoru AB
(2)
==1
==1
where [Voju Volus and - VoTUL
2.1.1 ACTIVATION- AWARE DECOMPOSITION
Large language models are known to exhibit outliers in their activations, i.e., the channel variance in X can be high, leading to outputs dominated by these outliers. Fortunately, prior research (Dettmers) et a 2022) has demonstrated that these outliers are often systematically distributed across the ac- tivation channels, underscoring the importance of accurately restoring the corresponding weight rows. Lin et al. first proposed scaling the weight matrix using row-wise scaling vector which precomputed with calibration set to reduce the quantization error of salient weights. (uaan et al.) further adopted this method, scaling the weights before applying SVD. In BitStack, we also adopt this methodology to preserve the restoration accuracy do not incorporate any additional searching procedures or hyperparameters obtain the scaling factors as in previous studies (Lin et al. 2024 Yuan et al. (003);; instead, we compute the scaling factor using the channel-wis. l2 norm of X. Formally, let X Iexm represent the input activations for a linear layer, computed using a calibration set, and W E RRxx he the corresponding weight matrix, compute the scaling factor as follows:
salient weights. To simplify the process,
=
||| I|,, ||>2||2, ,,,
(3)
The inference computation can then be transformed
XW X di //di/g(ssssgssss (s) W X diag(1/8)W/ Wacaled
scaled
(4)
And we use W scated for the subsequent decomposition.
2.1.2 ITERATIVE ABSOLUTE VALUE DECOMPOSITION
To reduce the approximation error each decomposition process, we propose to use absolute value decomposition. In this approach, we first decompose each (scaled) weight matrix into its sign matrix and absolute value matrix; for a weight matrix W E mm this expressed as W Wsign O [W]. We then apply SVD W while retaining W Weign- This method enables to store more information than directly applying SVD on W, since we save an additional matrix Wsign which is typically large in LLMss. Since W sign consists solely +1ss, can pack W.im to GPU-supported data types for storage and unpack it for use during inference computation. We store the singular vectors in FP16, resulting an overall memory occupation of approximately I bit per parameter when k & min m, n) in each decomposition process. A similar technique was employed in recent quantization aware training research to initialize the weights for 1-bit LLM training (Xu et al., 2024).
Formally. for matrix W sition would be:
W sign C W the approximation of W after absolute value decompo-
W avd
W sign
W
W sign + (A'B"T
(5)
svd
where W lan V. A'
IV oo and B' /orpo/.... (.
4

---

Published as a conference paper at ICLR 2025
To better restore the original matrix W, decompose W iterations, progressively decom-
posing the residuals from the previous approximations. For the i-th iteration, we compute:
AW (1)
W
w
(6)
tand
1=0
W land
AW
avd
=
w sign
O (A
(7)
where ( saud = 0. Hence, the overall approximation of W after iterations is:
W land
w
W sign C (A B
soud
Generally, this approach ensures that the recovery of the original matrix is compatiie,, allowing us to dynamically load or offload w iand (termed residual blocks in this paper. illustrated in Figure 33 based on the currently available memory budget, rather than reloading an entirely new model. Additionally, it enables precise size control of the model, as each residual block typically occupies less than a few megabytes, depending on the size of the corresponding weight matrix. See details in Section
2.2 SORTING RESIDUAL BLOCKS
Having universally decomposed each weight matrix every layer, it is essential to determine the order in which these residual blocks are loaded from storage into memory to optimize model perfor- mance within a given memory budget. To this end, we utilize a small calibration set to calculate per- plexity, assessing how much each residual block influences the overall performance. However, solv- ing this sorting problem remains non-trivial, even with this comparison criterion, since the search space is large. For instance, in model with layers, each containing M linear layers, and with each weight matrix decomposed over iterations, there are n possible combinations of settings across the various linear layers.
To reduce the search space, we constrain the difference in the number of residual blocks across all weight stacks no more than This approach facilitates smooth memory-performance trade- off and promotes effective load balancing when the model is distributed across multiple devices, resulting in a significant reduction of the search space to nLM. More specifically, no stack loads the i + 1th block until all stacks have loaded the ith block. We then sort the relative order of all the i + 1th blocks based on their importance, which is measured by the perplexity score after loading this single residual block while keeping all other i + Ith blocks for other stacks unloaded. The residual blocks are then placed into a universal stack, ensuring: 1) for ith blocks, blocks with lower measured perplexity scores
top of those with higher scores; 2) all ith blocks of any + 1th ones. This allows a relatively more important block to be loaded when additional memory becomes available. We provide the pseudocode of the sorting process from Line 25 to Line in Algorithm
top
3 EXPERIMENTS
3.1 EVALUATION ON BASE MODELS
3.1.1 SETTINGS
Baselines. Since our method training-free,, we compare it with two other strong, widely adopted training-freee model compression baselines: GPTQ (Frantar et al. (022)) and AWQ (Lin et al. 2024), both of which also require only a small calibration set as our approach. Note that we do not clude the comparison to other decomposition based methods in the main paper, as they suffer from severe performance degradation under high compression ratios (1
compressed model memory, and their original model memory
reported highest compression ratios are significantly lower than those in our study. For example, the highest compression ratios are 30%. 25%, and 60% for FWSVD (Hsu et al. 2022), ASVD YYuan et al. 2023), and SVD-LLM (Wang et al. 2024), respectively. Furthermore, for the state-of-the-an art decomposition based method, SVD-LLM, the perplexity score increased by 745% and the average performance zero-shot tasks dropped by 40% at compression ratio of 60% compared to the orig-
5

---

Published as a conference paper at ICLR 2025
inal model, which falls short of matching the performance of the quantization baselines <Wang et al. 2024). Therefore, we leave the comparison to other decomposition- based methods in Appendix A.6
Table I: Evaluation results of I. Llama 3.1 8B/70B models. Perplexity scores on Wiki'exx2 test set and accuracy scores on 6 zero-shot reasoning tasks. (t): higher is better; (4): lower is better. We
denote the overall compression ratio (1
compressed model memoryy after memory consumption.
original model memory
Model
Memory (MB)
Method
Wiki2 (4)
ARC. (1)
ARC-c (f)
PIQA
HellaS. (1)
WinoG. (1)
LAMBADA (1)
Avg. (1)
(f)
15316
FP 16
6.24
81.1
78.9 53.6-13.81.2400 389403_73999122
81.2g02
73.9 75.8_09_
74.1
GPTQ
1.2c6
26.010.9
27.1 1.3 51.711.2
26.010.4
48.5
0.010.0
29.910.9
3674(285)
AWQ..
1.1c6
24.9=0.9
23.6-1.2
49.621.2
26.2+0.4
52.2+14
0.0+0.0
29.420.9
BitStack
3.3e3
29.3109
23.4_12
114_12_5,4_11
27.9
50.771.
02..11
30.810.9
2g
1.7e5
25.9+0.9
26.0-13
53.941.2
26.5+0.4
49.6+1.4
0.0+0.0
30.3+0.9
3877 (7558)
1
1,5c6
24.6:00
24.7-13
50.0=1.2
26.4+0.
46.7+1.4
0.0+0.0
28.7+0.9
BitStack
79.28
48.4-19
26.0
66.5 41.0
57.1
15:--13
42.44100
GPTQ
260.86
34.7+10
24.5-13
57.6-1.2
30.4+05
53.0+1.4
3.0+0.2
33,9+0.9
(506g)
AWQ..
17.01
67.0+1.0
42,9-14
72.6-1.0
67.3+0.5
62.6+1.4
53.3-0.7
61.0+1.0
BitStack
12.55
68.511.
39.4_
39.4
75.5
63.4
65.8115
66.2 0.7
63.111.0.
8B
38.28
55.3+1.0
33,9-14
66.9-1.1
53.1
61,9+1.4
46.9-07
53,0+1.0
4709 (09%))
AWQ..3g11
8.06
74.5=009
48.4-15
77.7=1.0
73.9
70.6+1.3
67.8-0.7
68.8+0.9
BitStack
10.91
72.710.9
41.6 1.4 76.7
65.910.5
67.8113
69.6 0.6
65.711.0
GPTQ
20.88
74.7+09
45.6-15
77.2+1.0
54.640.
64,5+13
40.9
59.6+1.0
(338
AWQ
7.12
78.4+0.8
51.1-15
79,9=0.9
77.5+04
73.3+1.2
70.6-0.4
71.8+0.9
BitStack
8.39
76.610.9
47.9
79.0=1.0
71.6104
19.611.
76.1 0.6
10.1
6.83
78.6
51.5-15
29.1
77.0+a.
71.2+1.
72.9-u.
71.7+0.9
5541
AWQ
6.63
79.3+08
51.2-15
81.0=0.9
78.2+a.4
72.1+1.
74.2-0.6
72.7+0.9
BitStack
8.14
77.6
49.7-15
79.5
72.4104
70.6
71.02..9
134570
FP 16
2.81
86.7
64.8_14
84.320.8
85.1
79,8+1.1
79.2_0.6
80.0
GPTQ 22
NaN
24.8
26.2-111
50.8
26.4
51.4
0.0
29.9+0.9
20356
AWQ..
9.6e5
25.0==.9
25.5-13
51.7=1.2
26.6404
50.4+1.4
0.0+0.0
29.9+0.9
BitStack
1.0e3
27.9-02
52.3-13
304
49.6-11-
2.640
31.1
GPTQ
4,4c5
23.9
25.6-13
51.121.2
26.4104
50.4+14
0.0.0.0
29.640.9
22531 (80%)
AWQ.22112
1.866
14.9
26.2
133
51.311.2
26.810.4
49.4114
0.0
29.8
BitStack
8.50
76.8
50.6
77.9=1.
74.2
73.7
73.2
71.1
GPTQ 3
3.7c6
24.7
26.8
51.1212
26.3204
50.5
0.0.0.0
29.940.9
28516
AWQ.. 3
10.76
57.4
57.1100
37.0
71.1111
63.8
59.0114
49.5 o.7
56.311.0
BitStack
6.38
817
20
56.7
21
81.8
7933004
76.6
76.8
76.8_99_
75.5
70B
4,4e5
24.2
24.2_13
51.7
26.0...
49.3
0.0
29.240.9
30691 (?79)
AWQu33112 28
4.68
84.0
60.6
1.4
83.1
82.5
79.2 11.1
75.8_a.
17.5
BitStack
5.94
82.6
58.3
82.9
80.9
78.8
78.4 905_
77.0
GP
NaN
24.9-09
25.3-13
51.4-12
26.8
51.1
0.0+0.0
29.940.9
(6676
AWQ...
4.24
83.4108
61.3_14
83.5
83A104
63.5
69.1_0.
74.010.9
BitStack
4.97
84.8
61.4
61.4
83.2
82.1
79.3
79.3
79.4
_
78.4 78.4099
GPTQue9g12
6.5e4
23.4
27.3-13
$1.9
26.640.4
49.9+1.4
0.0+0.0
29.840.9
38851 (71%)
AWQ..4g128
3.27
86.6
63.3
83,9
10.9
84.4
78.8
77.3_0a
79.140.8
BitStack
4.96
85.1
61.3
83.5
82.6104
78.8
78.7
78.310.9
Evaluation. We evaluate our approach alongside the baselines on the well-known Llama2, Llama3, and Llama3.1 series (Touvron et al., 2023) Dubey et al.) 2024). with model sizes rang- ing from 7/8B to 70B parameters. We conduct the evaluations by computing the perplexity score on the Wiki'eex22 testset (Merity et al. 2016), and accuracy scores on range of zero-shot reasoning tasks including PIQA (Bisk et al. 2020), HellaSwag [Zellers et al. 2019), WinoGrande (Sakaguchi 202 ARC-easy and ARC-challenge (Clark et al. 2018). LAMBADA(OpenAl) (Radford ct al. 2. The zero-shot tasks are evaluated using the LM Evaluation Harness (Gao et al. 2024)
cet al.
with default parameters.
Implementation details For both our approach and the baselines, we randomly sample 256 sam- ples with sequence length 2048 from the WikiText2 (Merity et al. 2016) training set to serve as the calibration set. The baselines are implemented using zeropoint quantization, and we report their evaluation results both with and without group quantization to ensure optimal performance and fair comparison. For group quantization, we use group size of 128, which provides significant per- formance boost to the baselines. It is worth noting that the performance of GPTQ on Llama 3 and Llama 3.1 models can be unstable and may occasionally collapse. This variability is likely due to the Llama 3 series being more sensitive to compression (Huang et al. (024b), necessitating a larger number of calibration samples in GPTQ for optimal performance. By default, in BitStack, per-

---

Published as a conference paper at ICLR 2025
form iterative absolute value decomposition (IAVD) on each weight matrix over 16 iterations, saving the sign matrix and the first 16 singular vectors in each decomposition process. Additionally, for sorting the residual blocks in BitStack, sample smaller set of 32 samples from the WikiText2 training set to evaluate the importance of each residual block. All experiments are conducted on a node with 8 NVIDIA H800 GPUs
3.1.2 RESULTS
Evaluation results of both the perplexity scores and zero-shot performance of Llama 3.1/Llama 2/Llama 3 models are presented in Table T and respectively. Since BitStack allows for megabyte-level size control, we align the model sizes of the BitStack-come cosppack-ommprssee models with those of the different baselines for fair comparison. Specifically, utilize the largest size that does not exceed the baselines' sizes.
BitStack performs better at extreme compression ratios. As shown in the tables, BitStack de- livers superior or comparable performance with strong quantization baselines across different com- pression ratios, despite having the advantage that it only needs to compress and store once and can dynamically adjust its memory consumption at megabyte level. More specifically, BitStack mod- els constantly outperform the baselines at extremely high compression ratios. For 7/8B models, BitStack constantly outperforms GPTQ models below 4-bit-level and AWQ models below 3-bit- level. For 7/8B models, BitStack outperforms the best 2-bit baselines with group quantization by an absolute margin of 12. (L.lama 3.1), 22.3(Ilama 2), 10.4([lamm 3) on average performance in zero- shot tasks. This advantage is even more pronounced in larger models; for example, on the Llama 3.1 70B, BitStack retains 89%of the performance of the original FP16 models, surpassing the best baseline by substantial margin of 41.3 on zero-shot tasks.
BitStack maintains strong performance at lower compression ratios. While quantization base- lines excel at lower compression ratios, BitStack maintains comparable effectiveness, even with group quantization, which significantly enhances the performance of these quantization methods. For instance, at the lowest compression ratio (64%) in our experiments, BitStack Llama 3.1 8B and 70B models recover 96% and 98% of the zero-shot performance of the original FP16 model, respectively. Although they exhibit slightly higher perplexity scores, they only fall short of the best baselines by negligible 1.7 and 0.8 absolute average score on zero-shot tasks. As shown in the tables, the gap consistently narrows as the model size increases. For instance, when compared to the best baseline with group quantization, the gap zero-shot tasks decreases from 1.7 1.1 to 0.2 for Llama 2 models with 7B, 13B, and 70B parameters, respectively(Table (2). It can be seen that BitStack demonstrates particularly strong performance with larger models. For 70B models, consistently outperforms the baselines without group quantization across all compression ratios in our experiments.
3.2 EVALUATION ON INSTRUCTION-TUNNE MODELS
aadncuucaorg AMD-d
STOM
Russceing
DAS AMDrd
-0000MS
-7000MB
DiStack AMDrd
708
Mith
-5000MS
55000105
Dittankg000000 vn. AMOVO
Eodino
(a) Performance with various sizes.
(b) Pair-wise comparison with AWQ.
Figure 4: Evaluation results of BitStack Ilama 3.1 Instruct 8B/70B models on MT-Bench, assessed by -pt-tp.......o 4o. @ demonstrates the single-answer grading results across various sizes of the 8B model loaded by BitStack, while (b) illustrates the pairwise comparison results against AWQ at different compression ratios for both the 8B and 70B models.
7

---

Published as a conference paper at ICLR 2025
3.2.1 SETTINGS
To assess the generalization capability of our method, we conduct further experiments on instruction- tuned models. Specifically, apply compression to the Llama 3.1 Instruct 8B and 70B models using both our approach and AWQ, which has been shown be stronger baseline in the previ- ous section. We follow the procedure in Zheng et al. (2023) and evaluate the compressed models on MT-Bench (Zheng et al. (023), which consists of 80 multi-turn common user prompts, cover- ing writing, roleplay, extraction, reasoning, math, coding, knowledge I (STEM), and knowledge II (humanitics/sociaal science). We use OpenAl gpt-4o as the judging model to evaluate the model answers.
3.2.2 RESULTS
Figure 4a illustrates the evaluation results on BitStack compressed Llama-3. 1 1-Instruct-8B model with (4000, 5000, 6000, 7000, 8000) megabytes. The results show a clear trend across all domains: increasing the model size (by loading more residual blocks from storage) consistently improves performance. This underscores that while BitStack facilitates fine-grained memory- memory-performance trade-offs, the performance improvement spans all domains comprehensively. When compared to AWQ, BitStack demonstrates a similar trend at various compression ratios as seen with the base models. As shown in Figure 4b] extremely high compression ratios appioxsappo bit level-Bittttck models can occasionally generate reasonable answers, whereas the AWQ com- pressed model fails to produce coherent text. This distinction becomes even more pronounced the 3-bit level, where the BitStack model consistently generates high-qualio, responses, while the AWQ model still outputs gibberish. At lower compression ratios (4-bit level), where quantization- methods excel, BitStack outperforms or matches the baseline on about of the samples for the 8B model and about for the 70B model. We provide extra qualitative results in Section A.3
the 2-
3.3 ABLATION STUDY AND ANALYSIS
In this section, we conduct an ablation study to evaluate the impact of each component within our proposed approach. We assess performance by plotting perplexity and average zero-shot accuracy curves to measure the model's effectiveness at different memory footprints. For these experiments, we use the BitStack Llama 3.1 8B model and evaluate performance with memory stride of 500MB. Additionally, we provide further discussion the minimal transmission units in BitStack in Sec- tion A.4 and analyze the inference throughput of BitStack models in Section A.5
10
Average Zero Zero-shon shot Perfrmance
10
Perplxxity
10
scaling with Ao3
whilcing 8.00
Memory (MB)
Figure 5: Perplexity and average zero-shot performance of BitStack Llama 3.1 8B with without activation-awarc scaling and absolute value decomposittinn(AVDD (AVD). In the "w/o scaling" experiments, scaling is applied in Eq.Ei in the "w/o AVD" experiments, vanilla SVD is used instead of AVD mxn
as in Eq. 5 For vanilla SVD, we set k' = + 16x(m+7) (for W E "=**)) to ensure the size of each residual block matches that of the main experiments. Solid lines represent average zero-shot performance, while dotted lines represent perplexity scores.
Impact of each component. As shown in Figure activation aware scaling consistently improves model performance across all compression ratios, with particularly strong effects at higher compres- sion ratios. For instance, it leads to 8-point improvement in average 7ero-shot performance at
8

---

Published as a conference paper at ICLR 2025
memory footprint of 4000MB. Regarding absolute value decomposition (AVD), ablate it by placing it with vanilla SVD, using larger number of kept singular vectors to match the sizes of residual blocks. The figure shows that when AVD is replaced with SVD, the model performance degrades significantly, collapsing until a memory footprint of 12,000MB, which corresponds to a compression ratio of 22%, even with activation-aware scaling applied. This highlights that AVD significantly enhances approximation accuracy during the decomposition process compared to SVD under the same memory constraints, enabling the model maintain strong performance at high compression ratios. When both activation-awe aware scaling and AVD are removed, the model collapses across all compression ratios, underscoring the critical importance of these components. Note that we use the same sorting approach as we proposed in Section 2.2 for all these experiments.
Average Zero shot Performance
Perplexity
Greedy
Averrage Handom
10'
-0.3
4000
9030
6000
9000
10002
11000
12000
Memory (MB)
Figure 6: Perplexity and average zero-shot performance of BitStack Llama 3.1 8B with 3 different sorting approaches for residual blocks. Solid lines represent average zero-shot performance, while dotted lines represent perplexity scores.
Impact of the sorting algorithm for residual blocks. To reduce the search space for sorting residual blocks, we propose constraining the length difference between weight stacks to no more than (as detailed in Section 2.2, referred to as Average). We compare this approach to two al- ternatives: 1) Random, which randomly shuffles the universal residual stack without any search process; 2) Greedy, which evaluates each weight stack at each level (number of residual blocks) while freezing all other weight stacks a level of 5. and utilize the current perplexity as the impor- tance for corresponding stack that level, which also has search space of nLM. We provide visualization of resulting weight stacks of the three sorting approaches in Section (A.7 As shown in Figure. the memory footprint goes up, all three approaches converge most residual blocks are loaded into the model. However, at lower memory footprints(<< 8000MB), Average significantly outperforms both baselines, surpassing the best baseline by 16, 16, and 7 points in absolute zero- shot performance at 4000MB, 4500MB, and 5000MB, respectively. In addition to excelling at high compression ratios, Average also provides better load balancing, the memory footprint of each block varies minimally, making it easier to deploy in distributed scenarios.
Ablation on calibration set size n. We compute the scaling vector 8 using various sizes of the calibration set, as shown in Figure. 7a The figure demonstrates that BitStack is robust to changes calibration set size, as the curves align almost perfectly across different sizes, particularly as the memory footprint increases. Interestingly. BitStack even performs slightly better with a smaller calibration set size of 64 in extreme compression scenarios, such as with a memory footprint of 4000MB.
Ablation on number of kept singular vectors i in each decomposition process. Generally, larger & in SVD indicates a better approximation in each decomposition process, but at the cost of increased memory usage, as the singular vectors are stored in FP16. Figure. 7b illustrates the performance when setting to 41,4,8, 8, 16, 32). As shown in the figure, keeping only the largest singular value and its corresponding vectors is insufficient for good approximation, leading to performance degradation. On the other hand, increasing & results fewer residual blocks being loaded at the same memory footprint, limiting model performance. This is evident from the figure,

---

Published as a conference paper at ICLR 2025
077
35
0.73
Iverage Zoro zoro-shot Perfoma momaaaao
Avtragge 2.00 Peroromm
0.03
Parplxtyy
Perplextyy
0.00
0.50
1045
03
5000
7320
5030
7003
8020
Memoory (MB)
Mirmory (MB)
(a)
(b)
Figure 7: Perplexity and average zero-shot performance of BitStack Llama 3.1 8B with various calibration set sizes (a) and number of singular vectors & (b). Solid lines represent average zero- shot performance, while dotted lines represent perplexity scores.
as increasing & beyond provides significant performance improvement. Overall, k 16 strikes a good balance between approximation accuracy and memory consumption.
RELATED WORK
Fixed ratio weight compression. As discussed in Section [ we categorize weight compression approaches such quantization, pruning, and distillation under fixed ratio weight compression. Quantization-basse methods compress weights by reducing precision, pruning techniques compress by directly modifying the model structure (e.g., reducing the number of layers or hidden dimen- sions), and distillation methods involve training smaller model on the outputs of the original model. The latter two approaches, as well as quantization methods for higher compression ratios, typically require extensive training (Ma et al. 2023 Xia et al. 22023] Muralidharan et al. (2044), which can be computationally expensive when compressing models for multiple compression ratios (Shao et al.) 2023 Tseng et al. 2024 Egiazarian et al. 2024 Huang al. 2024a). Furthermore, models com- pressed by these methods are poorly suited for variable memory environments due to their fixed memory usage, preventing efficient utilization of available capacity.
Adaptive ratio weight compression. Weight decomposition methods are more suitable for adap- tive ratio weight compression due their neward-ccopptibe nature, as the approximation im- proves with the inclusion of more singular vectors in SVD. However, current decomposition-baz weight compression approaches for LLMs tend collapse high compression ratios (Hsu al.] 2022 Yuan et al. 2023 Wang et al., 2024), rendering them impractical for real-world deployment. In this work, we bridge the performance gap between decomposition- based methods and practical quantization-baat based approaches, making LLM deployment in variable memory environments feasi- ble.
5 CONCLUSION
In this paper, we highlight the challenge of deploying compressed large language models in vari- able memory environments and propose BitStack, decomposition- compression approach designed to address this issue. BitStack enables megabyte-level pecmory-pefformance trade-off's in a training-free manner, requiring only small calibration set. Additionally, BitStack simple to implement, with the decomposition of 70B models being achievable single GPU. Despite its flexibility in memory footprint, BitStack consistently matches or surpasses the performance of practical baselines, making it a viable solution for real-world applications. We believe that Bit- Stack represents a new paradigm for LLM deployment on local devices, providing not only efficient memory management but also strong performance within the given memory budget.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Key Research and Development Program of China (No. U24B20181).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Anthropic. Claude ai. http::/www.attaropiccccoo///c/c/ 15.
2024. Accessed: 2024-09-
Salch Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.1502: 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1803.05500 2018.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances Neural Information Processing Systems, 35: 30318-30333 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Al- istarh. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.06118 18, 2024.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh, Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint wwXiv:2210.173222 2022.
Leo Gao. Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds. Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. http:tttss//eeooooorr///////////
12608602
GitHub. Github copilot. httpe://gttuub.cc//eetuuu//////
2024. Accessed:
2024-09-15.
Google. Google ai.
2024. Accessed: 2024-09-15.
Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Man- grulkar, Mare Sun, and Benjamin Bossan. Accelerate: Training and inference scale made sim- plc, efficient and adaptable. https::/ wwt.t/twwww.....uuuggggaaaeaeeee aww.ee/eecceeeeeeee.eee.e 2022.
wwt.
Yen- Cen-Chang Hsu, Ting Hua, Sungen Chang. Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. arXiv preprint arXi::2207.00112 12, 2022.
Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu. Michele Magno, and Xiaojuan Qi. Billm: Pushing the limit of post-training quantization for 1lms. arXiv preprint arXiv:2402.04300 2024a.
11

---

Published as a conference paper at ICLR 2025
Wei Huang, Xudong Ma, Haotong Qin, Xingyu Zheng, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. How good are low-bit quantized Ilama3 models? an empirical study. arXiv preprint arX1v:240..14047, 2024b.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. XrXiv preprint arXiv:2001.0882 2020.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration, 2024. URL httt:p/e/aaxxii.rrr/abbbbse/// 0291 00978
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-2117 2023.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture 2016.
models. arXiv preprint
Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. arXiv preprint arXiv:240.114679 2024.
OpenAI. Chatgpt. attps::/chat..peeee
2024. Accessed: 2024-09-15.
Perplexity.Al. Perplexity ai. https::/wwwpperpeexxii
2024. Accessed: 2024-09-15.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenA/ blog. 1(8):9, 2019.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprini arXiv:2307.09288 2023.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipi:: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint Stl1:24022.03996, 2024.
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-llm: Truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07377, 2024.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre-training via structured pruning. arXiv preprint Ttt::2310.06694 2023.
Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models. arXiv preprint arXiv:2402.1 / 1295, 2024.
Zhihang Yuan, Yuzhang Shang, Yue Song. Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation awwre singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.0582, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint 1905. 2019.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
12

---

Published as a conference paper at ICLR 2025
A
APPENDIX
We provide further details of our approach in this appendix as follows:
Section A.1
the overall algorithm of BitStack.
Section
A.2
evaluation results of Llama and Llama models.
Section
A.3
qualitative results of BitStack Llama 3.1 Instruct 8B and 70B models.
Section
A.4
discussion minimal transmission units in BitStack.
Section
A.5
analysis of inference throughput of BitStack.
Section
A.6
comparison to other decomposition- based approaches.
Section
A.7
visualizations of weight stack in different sorting approaches.
13

---

Published as a conference paper at ICLR 2025
A.1 OVERALL ALGORITHM OF BITSTACK
Algorithm BitStack
Input: A model M with I. layers each consists M weight matrices with weight matrices (W WLLM)) and corresponding activations (x ,... X in calibration
set X. Number of decompose iterations n, number of singular values i retained SVD. And
extra small calibration set X' for sorting.
Output: Sorted residual block stack S
1: procedure ACTIVATION-AWE
WEIGHT SCALING
2: for / + 1 to L do
4: for m 1 to MM do
3:
g(lm)
|| (1m) (|2, ||:02 ( 12,, ,1,ch (1)
Eq. (3)
W W(lm) scated diag( (1m)) ((m)) W(())
5:
Eq. (4)
6: end for
7: end for
8: end procedure
9: Now that the weight matrices scaled, we omit the scated, for clarity.
10: procedure ITERATIVE ABSOLUTE DECOMPOSITION
for 1 + do
12:
for m < to M do Initialize empty weight stack g(lm) for 4- to do 5(lm) .push (W:l)))(1) end for Initialize empty stack c(lm)
13:
15:
Eq.
sannd
16:
17:
Stemp temporarily store the elements for subsequent sorting. 5(im) Stemp- push (S0m) p))())
for
+ 11=torr
do
20:
end for
21:
end for
22:
end for
end procedure
24: Initialize an empty universal residual block stack
25: procedure RESIDUAL BLOCK SORTING
26: for i - 1 to
do for to L do
27:
for m to do (Smmp ())
29:
((lm) push ((()))
b Push a new residual block for assessing.
30:
Isaud i importance = compute.perplexityy. (M, X') Measure influence. <(lm) Stmpp push (S(mm) ())
31:
Pop before evaluating the next weight.
32:
end for end for wtw.eeddtt..i..rreiiallll....... = sort (W(i) Geond key=importance,
33:
34:
35:
for block in iti blocks do S.push (block) end for
36:
Push important blocks first.
38:
for - 1 L do
39:
for m + to M/ do ((lm)pushh (5(m) pop pop ())
40:
b Push all stacks before assessing (i + 1)th blocks.
41:
end for end
42:
43:
end for
44: end procedure
45:
return reversse Position residual blocks with lower bit levels and higher importance at the top.
Algorithm [ illustratss the pseudocode for the overall algorithm of BitStack. Specifically, Lines to
describe the activation-awwar aware weight scaling process, as introduced in Section 2.1.T This scaling applied only once before the iterative absolute decomposition. Lines to 23 detail the iterative absolute decomposition process, as explained in Section [2.1I] where each scaled weight matrix is decomposed into residual blocks and stored as stack. Finally, Lines 24 to 44 demonstrate the
14

---

Published as a conference paper at ICLR 2025
residual block sorting process, as discussed in Section 2.2 where the influence of each residual block is evaluated while keeping all other weight stacks at the same level. The blocks are then sorted by their evaluated importance and placed into universal stack.
A.2 EVALUATIONS OF LLAMA2 AND LLAMA3 MODELS
Evaluation results of Llama 2 7B/13B/70B models. Perplexity scores on WikiText2 test set and accuracy scores on 6 zero-shot reasoning tasks. (t): higher is better; (4): lower is better. We denote the overall compression ratio (1
Table 2:
compressed model memory after memory consumption.
original model momory
Model
Memory (MB)
Method
Wiki2 (4)
ARC-e
ARC-c (1)
PIQA (1)
HellaS. (1)
WinoG. (1)
LAMBADA
Avg. (1)
(1)
(1)
12852
FP 16
5.47
74.510.9
46.211s
79.1 7.. 69.1
73.9108
69.810.9
GPTQ
2.8c4
26.5+a.9
27.6+1.3
48.4+1.
25.9+04
50.3+1.4
0.0+0.0
29.8+0.9
2450
AWQ
1.8e5
26.310.9
26.711.3
50.9112
26.510.4
49.3+1.
0.0.0.0
30.0+0.9
BitStack
29.93
32.3++19
25.6+13
62.4+11
42.8
42.8:00_5336640
53.6
24.7
40.2+10.
GPTQu22112
156.37
28.2.0.
27.111.3
51.7
28.0.0.
51.1114
0.310.1
31.11099
2238 (8399)
AWQu22012
2.3c5
25.8.10.
26.711.3
50.2
26.1
49.8
1.0.0.0
29.810.9
BitStack
12.49
51188
30.1+1.
71.1
53.0
61.1
53.3
53.4
GPTQ.s m.
9.38
58.111.0
34.0
71.911.
61.7.05
60.611.
53.3
56.611.0
(882
AWQ
14.33
52.7+1.0
33.0+1.4
68.3+1.1
56.3+0.
59.3+1.4
36.3+0.7
51.01.0
BitStack
7.45
62.5
37.5
74.8
67.0
66.5
68.510.
62.811.0
GPTQu33128
922.54
26.3
25.3
52.411.
27.4
49.0
0.1
30.1
7B
(010(779)
AWQ ..:3y 128
6.14
70.2
43.7
78.0+1.0
73.9
67.6+1.3
71.4
67.5+1.0
BitStack
7.10
63.8
38.211.4
76.0
68.4
65.9
70.7
63.8
GPTQ
5.91
71.8
43.7
77.7
14.5
68.7
71.1108
67.911.0
(129)
AWQ
5,81
70.9
44.5
78.5
74.8
69.2
71.510.6
68.2
BitStack
6.36
67.0.1
414444
77.1
71.4
69.5
73.1
66.041.0
5.73
73.6
45.3+1.5
78.7
75.4
67.6-1.3
72.7+0.6
68.9
(712
AWQus4e122
5.61
73.310.
45.2
78.611.0
75.2
68.7
72.710.8
68.9
BitStack
6.27
67.8+11
43.3+1.4
77.2+1.1
72.2-04
68.6.1.
73.9
67.2+1.0
24825
FP 16
4.88
77.4400
49.1
80.5.0
79.4
72.2
76,8408
72.640.
GPTQ
1.244
26.4.0.9
28.2+1.3
50.2+11
26.3
48
48.4+14
0.0.0.0
29.9+0.9
3659
AWQ...
9.624
27.3+0.9
28.0+1.3
49.9+12
26.0-0.4
50.4+14
0.0+a.
30.3+0.9
BitStack
68.64
38.1
23.542
32.2-0
51.6
14.0
36.1
3.9c3
26.2.2.9
28.8
50.7
26.9
48.6214
0.1
30.240.9
4029(8499) (849)
2gQ 28
1.2c5
26.9
27.5
50.0
26.1
50.8
0.01000
30.2
SitStack
9.26
64.5
34.2
73.0
60.9
203
64.9
65.3
60.5
GPTQ..
6.20
68.221.0
42.8
77.1+1.0
71.400
67.6-1.3
63.1
65.0+1.0
5171 (171(799)
AWQ
6.46
71.1109
44.411.5
77.611.
71.2105
66.8
61.9
65.511.0
BitStack
6.32
74.4
45.1
77.1
71.9
69.2
74.8
68.8
GPTQ G8
5.85
23.4
45.2+1.5
78.2+1.1
74.4
68.0.1.3
67.640.7
67.8
13B
5541 (188))
AWQ...yt 128
5.29
75.3
48.5+1.6
79.4+0.9
77.1
70.8
75.1
71.0+0.9
BitStack
6.04
74.420.
46.2+1.
77.9
72.640.4
70.6
76.6.0.
69.740.9
GPTQ w4
5.09
75.8
48.0+1.8
79.610.9
77.4
72.4
74.5
71.410.9
(684
AWQ..
5.07
78.2+0s
49.7+1.5
80.4
78.6
71.6+1.3
76.1
72.4+0.9
BitStack
5.53
76.7
48.411
79.041.
75.2
71.7
77.4
71.4
4.97
76.420.9
49.2+1.5
79.9
78.8
71.721.3
76.0+o.6
72.0+0.9
7054 (129%)
AWQ
4.97
77.1
48.5
80.4
78.8
73.1
76.810.
72.510.9
BitStack
5.47
76.5
48.011.5
79.0
75.7
71.7
77.810.8
71.410.9
131562
FP 16
3.32
81.1
57.3114
82.7
83.8
78.0
79.610._
17.1
GPTQ .
152.31
26.8
26.0+1.3
49.0+1.
26.1+04
49.8+14
0.0+a.0
29.640.9
(7348(87%)
AWQwz
8,0e4
25.8
28.8+1.3
50.1
25.7
48.3+14
0.0+0.0
29.8+0.9
BitStack
9.41
67.8+11
42.1
75.9
65.1+05
67.7
65.7
64.1+1..
7.79
53.0
32.0
66.9
51.1
60.2
34.810.7
49.7
(9363
AWQu220128
7.2c4
26.0+a.
28.9+1.3
49,8+11
25.7
51.0+11
0.0+0.0
30.2+0.9
BitStack
5.30
74.5
50.0
79.7+0.8
75.1
74.4
79.3408_
72.2
GPTQ... .a
4.49
75.9
52.1118
80.7
79.210.
75.3
74.3
72.910.9
25508 (81%))
AWQ
4.30
79.8
55.4
81.4
81.2
73.6
73.1
74.140.
BitStack
4.33
78.9
54.9
81.7.09
79.9
76.611,2
80.1 80.1100
75.319.
GPTQ GPTQwa9112
55.43
27.8
27.41.
50.9
29.8
48.9+1.
9.5
32.440.
70B
27523 (7u%)
AWQ .WQQ..gie 2s
3.74
79.0
56.7
82.8
82.3
76.611.2
79.3
76.1
BinStack
4.07
79.8
55.4118
82.4
80.7
77.3
81.6103.
76.2
GPTQ .e
3.59
79.3
54.941.8
82.2
82.8
77.2+1.2
79.1
75.9+0.9
33668(749)
AWQw4
3.48
80.6
57.9
82.8
83.2.04
76.5+1.2
78.840.6
76.640.9
BitStack
3.76
19.3
574444
82.4
818
77.9
81.400
76.6
3.42
81.3
57.8114
83.0
83.6
76.8
79.410.8
77.010.9
35683
AWQ A4Quss
3.41
80.3
56.7
83.1
83,4+04
78.1
79.640.8
76.940.9
BitStack
3.71
79.7+as
57.1+144
82,2+0.
82.1+04
77.9.12
81.7+0.5
76.8+0.9
15

---

Published as a conference paper at ICLR 2025
Table 3: Evaluation results of Llama 3 8B/70B models. Perplexity scores on WikiTex12 test set and accuracy scores on 6 zero-shot reasoning tasks. (f): higher is better, (d): lower is better. We denote the overall compression ratio (1
compressed model memoory after memory consumption.
original model memory
Memory (MB)
Wiki2 (L)
ARC-e
ARC-c (1)
PIQA (1)
HellaS. (1)
WinoG. (1)
LAMBADA
Avg.
Model
Method
(1)
(1)
(1)
15316
FP 16
6.13
10.
53.3.15
80.8.02 79.2.04 72.7
76.1
73.3100
GPTQ
1.1e6
25.3+0.9
26.7+1.3
50.6412
26.4=0.4
51,0+14
0.0+0.9
30.0+0.9
(774
AWQ
1.1c6
25.2++0.9
24.1+1.2
50.7+1.2
26.2-0.4 48.6+1.4
0.0+0.0
29.1+0.9
BitStack
1,5e3
29.5
23.911.2
53.4112
27.7.0.4
50.6114
0.010.0
30.910.9
GPTQ
22g12
1.2c5
26.140.9
25.9+1.3
50.7+1.2
26.0=044
50,0+1.4
0.0+0.0
29.8+0.9
(777))75%)
AWQ
1.7c6
24.8+0.9
24,4+1.3
50.4+1.2
26.4-004
50.5+1.4
0.0.0..
29.440.9
BitStack
96.87
48.5
25.3-113
64.0.11
37.1
56.7
24401
40.2.09.
GPTQ wi
9.6e4
26.0
25.7+1.3
50,9
27.1=004
50.3+14
0.0
30.0+0.9
45066
AWQ..
12.08
61.7+1.
38.8+14
71.4+1.1
68.6-0.5
65.0+1.
51.+
59.6+1.0
BitStack
12.79
69.420.9
38.7+14
75.6
63.5.0.5
65.9
66.620.7
63.311.0
GPTQ
122
8.00
73.1
46.4
77.8+1.
74.5
71.6
68.5
68.7+0.9
8B
4709 (709()9%)
AWQ
8.09
70.7+09
44.0+1.5
77.9+1.0
73.4-04
70.5+13
69.7-0.6
67.741.0
BitSttck
11.45
77..6492
42.2
76.2
65.8-11
67.3
68.6
65.4.1.0
GPTQQ..
3.7e4
28.2209
25.3
51.041.2
28.7=0.
54.6414
0.140.0
31.340.9
5338 (65%)
AWQ
7.08
75.0+09
51.5+1.
79.5
77.8
72.1+13
71.
71.2+0.9
BitStack
8.58
74.6+09
46.2115
77.5
72.3
70.8
76.0=0.6
69.640.9
GPTQ 1g112
1.2e4
31.7
23.8+11
55.1
29.3
56,4
0.7
32.840.9
5541 (541()64))
AWQ
6.54
76.910.9
$2.4
79.9
78.1
73.611.
73.610.6
72.410.9
BitStack
8.26
75.8
47.1
78.7
73.1
70.8
26.3-0.
70.3
134570
FP 16
2,85
85.9..9
64.3
84.5
84.9
80.7
79.8
80.0gos_
GPTQ
3.7e5
24.7.09
26.3.1.3
51.5412
26.3-04
50.0+114
0.0
29.8+0.9
20356 (85%)
AWQ
8.6e5
25.1
25.9+1.3
52.3+1.
26.6-0.4
47.8+1.4
0.0.0.9
29.640.9
BitStack
59.37
46.5+1
27.3-13
65.2+1
39.1-055
51.9+1
9.2-0-4
39.9+1.0
GPTQ
4.0e5
25.3
24.7
49.3
26.0
10.4
0.0
29.210.9
22531 (85%)
AWQ.22122
1.7e6
24.9
26.4.1.3
51.4+1.
26.8
51.8+14
0.0+0.0
30.2+0.9
BitStack
8.86
74.2
48.4
78.1
73.5
10.4
73.6
71.8 _
69.9 1oo9
GPTQ ma
NaN
24.6409
25.441.3
51.0
50,4+14
0.0.0.0
29.640.9
28516 (8516(799) (19%)
AWQ
14.04
65.5+10
41.2114
73.1
64.3
$7.4114
46.92.
58.1
BitStack
6.88
79.8
54.8-15
80.8
79.6-1
77.0+12
75.3-05_
74.5
G0TQu33122 129
4.8e5
25.5
26.5
51.5 26.3
48.8
0.0
29.810.9
70B
30691 (17%)
AWQ..sg 128
4.59
82.2
60.6
82.8
82.9
78.4+1.2
76.820.
77.340.9
BitStack
5.69
81.6108
$7.8
82.410.9
81.2
78.5
79.7
76.910..
GPTQQs
NaN
25.2+209
25.3+1.3
$1.6412
26.3-04
50.1+1.4
0.0+0.0
29.8+0.9
36676 (138))
AWQ a0
4.16
77.5+09
54.4
81.5
80.0=
60.5+1.4
67.420.7
70.240.9
BitStack
4,88
82.3
61.1114
83.410.9
82.5
79.9
80.1
78.2
GPTQ 122
7.8c5
25.0
26.3.1.3
49.9+12
26.8
47.4+14
0.0+0.0
29.2+0.9
38851 (71%)
AWQ Lodky 128
3.23
85.9
63.5+14
84,2
84.5-0.
80.1
78.1
79.4
BitStack
4.80
82.8
60.2114
82.910.9
82.810.4
79.611.
80.1
18.9
Here, we include the detailed evaluation results of Llama and Llama models in Table and Table 3 respectively.
A.3 QUALITATIVE RESULTS
In Table and we compare BitStack and AWQ across different compression ratios on the Llama 3.1 Instruct 8B and 70B models. As shown in Table 4 which illustrates a math reasoning task, at a relatively lower compression ratio (5338MB), both compressed models produce correct an- swers with several reasoning steps. As the compression ratio increases (e.g., 4506MB), the AWQ- compressed model starts with a correct reasoning path but collapses midway through inference, model still manages to reach the correct answer. At extreme com- pression ratios, such as 76% (3674MB), neither model produces a correct answer; the AWQ model outputs gibberish from the beginning, while the BitStack model is able to generate coherent text but omits the reasoning steps, ultimately arriving at an incorrect conclusion. For the 70B models in As shown in Table 5 where the models are asked write rhyming proofs, the trend is similar to the 8B models. As compression ratios increase, AWQ models begin to output gibberish, while BitStack models continue to produce fluent text, albeit with lower answer quality.
whereas the BitStack-compresst
A.4 DISCUSSION ON MINIMAL TRANSMISSION UNITS IN BITSTACK
In this section, we discuss the minimal transmission units, i.c., residual blocks, in BitStack. As detailed in Section 2.1.2 we decompose the approximation residuals in each iteration into their sign
16

---

Published as a conference paper at ICLR 2025
Table 4: Qualitative results of BitStack 1atma-3.--8B-Instie at different compression ratios com- paring to AWQ.
A tech startup invests $8000 in software development in the first year, and then invests half of that amount in software development in the second year. What's the total amount the startup invested in software development over the two years?
QUESTION:
## Step 1: Identify the amount invested in the first year. The startup invests S8000 in software development in the first year. ## Step 2: Calculate the amount invested in the second year. The amount invested in the second year is half of the first year's ' investment, which is $8000 / $4000. ## Step 3: Calculate the total amount invested over the two years. Add the investments of the first and second year get the total amount invested: $8000 $4000 $12000. The final answer is: $12000.
AWQ-w4:
(5335ack(5338MBB) ## Step I: Calculate the amount invested in the first year. The startup invests $8000 in software development in the first year. ## Step 2: Calculate the amount invested in the second year. The startup invests half of the first year's investment in the second year, which $8000 2 = $4000. ## Step 3: Calculate the total amount invested over the two years. The total amount invested is the sum of the first year's investment and the second year's investment, which is $8000
$12000. The final answer is: 12000
AWQ-w3: ## Step 1: Calculate the total investment in the first year The total amount invested in the first year is 8000. ## Step 2: Calculate the total investment in the second year The total amount invested in the second year is half of the first year's s amount, which is 4000. ### Step 3: Calculate the total investment over both periods. To find the total amount invested in software development over the two years, we need to add the investment in the first year and the second year. The total amount 8000 (half of pound eight thousand five hundred six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six six Six six six six six six six six six six six six six six six six six six six six six (-.. concatenated.)
BitStack(4506MBB To find the total amount invested in software development over the two years, we need to add the amount invested in the first year to the amount invested in the second year. The amount invested in the first year $8000, and the amount invested in the second year is half of that amount, which is $4000. So, the total amount invested in software development over the two years is $8000 + $4000, which is $12000.
AWQ-w2: blank blank blank blank blank blank blank blank blank blank
blank blank blank blank blank blank blank blank blank blank blank blank blank blank blank blank blank blank blank blank
blank
blank
blank
61111111111
concatenated.)
Bi1Stack(3674MBB) The total amount of the startup is $8000.
matrix and singular vectors of the absolute value matrix (Eq. 6 Since the sign matrix requires bit of memory per parameter after packing, and the singular values are stored in FP16, for a weight matrix W E mRmx,, the overall memory of each residual block in the stack can be calculated as follows:
5w = (m n) + 16 x x (m + n) bits
where A is the number of singular vectors kept in SVD.
Since the shape of weight matrices in LLMs can vary (for example, in Llama 3 8B, Wa.proj e R44096x4096 and Wk.proj E R 14496x1021) the size of each residual block for different weight stacks may also differ. We present the sizes of residual blocks for each weight stack in the Llama2, Llama3, and Llama3. models in Table 6 As shown in the table, BitStack enables dynamic loading at a megabyte level, with minimum block size of 0.66MB in the Llama 3 8B model. We further verify BitStack's 's capability for fine- -nained neemory-performnn zoom in and use small memory stride, i.e., 10MB, to evaluate the model. As shown in the figure, BitStack successfully achieves a fine- grained trade-off, with perplexity consistently decreasing as more memory is allocated by loading additional pre-sorted residual blocks from storage.
trade-offs. In Figure
17

---

Published as a conference paper at ICLR 2025
Table Qualitative results of BitStack Llama-3.1-2 3.1-70B-Instruce different compression ratios com- paring to AWQ.
USER: Suppose you are mathematician and poet. You always write your proofs short poets with less than 10 lines but rhyme. Prove the square root of 2 is irrational number.
AWQ-w4: A delightful challenge! Here's my poetic proof: Assume, if you will, that v is rational too, fraction, a ratio, with a finite clue, Let a/b be its value, in simplest form, so fine, With a and b integers, and b not divine. Now multiply by b, to clear the fraction's sway, And square both sides, to chase the square root away. You'll get b have common factor anew. This contradicts assumption, logical flaw, I'm afraid, conclude: / is not rational trade.
2b".. a simple equation true, But this implies a and
BitStack(36666MMB) Here's s my poetic proof: numbers, a truth we must find, The square root of two, irrational to bind. Assume it's rational, contradiction to make, Then and b, two integers at stake. a/b 22, perfect ratio to claim, But if a² 2b2, then contradiction to see, So /2 is irrational, a truth decree. No rational form, it will never
must be the name, And
be.
AWQ-w3: PRO- PRO PRO PRO PRO has PRO: PRO PRO was PRO PRO. PRO PROPRO - is PRO-PRO PROSUB is PROPROSUB PRO PRO PRO.PRO PROSUBPRO- PROPROPROSUB PROPROPROSUB -OODRROPRORO (-.. concatenated.) CHistack(25116NM))
A delightful challenge! Here's s my poetic proof: In numbers, tale is told, Of a root, that's not whole. Two's square root, a number rare, Cannot be expressed with simple care. For if it were, fraction fine, A and B, with and in line, A times b, square make, Would be two, number to partake. But no such pair, with and b to share, Proves the root of two, is irrational spare. (Note: The traditional proof of the irrationality of the square root of 2 based on the fact that if it were rational, would have to be an integer or fraction, and then we can find a and b such that a² 2b². which leads to contradiction. This poetic proof is rhyming version of this idea.)
AWQ-w2: Pitaquetsalisalis. Dataiaaaalisaaaayppppepp Thick-transitionall Thickquets///aiss Pita.xra adt Thick tionalis Thickutowessleranitiee anitizea sublicensealisakeennngg. dnerequisite Pit dnMigration Pit Thick Pit Thick dnEGIN concatenated.)
BitStack(2336GMBB) In the realm of math, number so fine, A square root of two, number divine, number so fine, so divine, square root of two, so fine. number so fine, so divine, number so fine, so divine, A square root of two, so fine, A number so fine, so divine, A number so fine, so divine. number so fine, divine, number fine, so fine, number so fine, so fine, number so fine, divine. A number so fine, so divine, A number so fine, so fine, A number so fine, so divine, (Note: The number square root two is fine.) hope you can understand the proof.
Table 6: Size of residual block in various weight matrices in BitStack (k = 16), measures in MMg)
Model
W g_proj
W WK orooj
W V.proo
W Vo.proj
W gate.proj
W up appppro
W dowwn_proj
Llama 2 7B
2.25
2.25
2.25
2.25
5.84
5.84
5.84
Llama 13B
3.44
3.44
3.44
3.44
9.02
9.02
9.02
Llama 2 70B
8.50
1.28
1.28
8.50
29.13
29.13
29.13
Llama 3(3.1) 8B
2.25
0.66
0.66
2.25
7.56
7.56
7.56
Llama 3(3.1) 70B
8.50
1.28
1.28
8.50
29.13
29.13
29.13
18

---

Published as a conference paper at ICLR 2025
Figure 8: Perplexity scores on the WikiText2 test set for the BitStack Llama 3.1 8B model. We plot the perplexity scores for memory usage ranging from 4000MB to 5000MB, with stride of 10MB, to assess BitStack's 's capability for fine-grained trade-offs.
A.5 INFERENCE WITH BITSTACK MODELS
A.5.1 ANALYSIS OF INFERENCE OVERHEAD OF BITSTACK
Infurunca (4)
50
20
Overhcdd2
Osemant
Namber residual blocks weight
Figure 9: Generation time for 50 tokens with BitStack Llama 3.1 8B using different lengths of weight stacks(setting the same number of loaded residual blocks for all stacks). Results are evaluated on an NVIDIA H800 GPU.
In this section, we provide an analysis of the inference overhead of BitStack models to support the future deployment of these models in real-world scenarios. Similar other weight-only compres- sion methods (e.g., weight-only quantization), the restoration of weights is performed on the fly, introducing an inference overhead. illustrated in Figure 9 we roughly divide the total inference overhead into two parts: Overhead 1 and Overhead 2.
Overhead / represents the residual block restoration time, including unpacking the sign matrix and the multiplication time as in Eq. [5. This overhead can be substantially reduced by leveraging fused kernels, which help minimize the time-consuming I/O costs.
Overhead 2 refers to the additional time required to restore more residual blocks for weight stacks that load more than one block. As shown in the figure, Overhead increases linearly as more residual blocks loaded. This occurs because, in our PyTorch implementation, the residual blocks are restored sequentially when computing Eq. In practice, however, all residual blocks in the stack can be computed in parallel, they are independent of one another, making Overhead eliminable.
19

---

Published as a conference paper at ICLR 2025
A.5.2 THROUGHPUT OF BITSTACK MODELS
Theer faned and ee
lnter lused evd
Tiken fused ave
Icec. v fytrch
2 Pytnth
Titenn funed now
Titten luaod lod
Speeduo
3
Gencration Time
Number Revidual Blocks Weight Stack
Number Residual Blocks Weightt Stark
(a)
(b)
Figure 10: Comparison of generation time between Triton and PyTorch implementations. (a) illus- trates the time taken to generate 50 tokens using BitStack models with different implementations. while (b) depicts the corresponding speedup. Experiments are conducted on an Nvidia H800 GPU with the Llama 3.1 8B model.
To further validate the practicality of BitStack, we implemented inference kernels for BitStack mod- els using OpenAI Tritor and compare their throughput with the original Py PyTorch implementation. The results are shown in Figure TO We fuse the weight restoration operations at 5 different levels:
Level 1: Fuse the unpacking operations of the sign matrices with Triton and stack the singular vectors for parallel restoration of the absolute values of the weights with PyTorch.
Level 2: Fuse the unpacking operations and the restoration of absolute values of the weights with two different Triton kernels.
Level 3:
Fuse the restoration of the weights in a single Triton kernel, and sequentially process each residual block inside the kernel.
Level 4: Fuse the restoration of each residual block into a single Triton kernel to enable parallel processing the residual blocks.
Level 5: Fuse the unpacking operations in one kernel, and fuse the weight restoration and multiplication with the activations in another kernel.
As shown in Figure 10 level achieves the highest throughput in most scenarios by maximizing the parallelism of simple operations, such unpacking. and performing weight restoration with an optimal parallel configuration. Level 4 performs worse than level 3, likely due to the introduc- tion of atomic operations and resource contention during weight restoration, despite offering higher parallelism. Level 5 is not performant as well as it introduces redundant computations for weight restoration, even though it is the most I/O-efficient option.
As illustrated in Figure 10a the inference overhead is negligible at high compression ratios (fewer residual blocks), where BitStack models continue to maintain strong model quality. Figure T0b shows that simple Triton kernels can significantly speed up BitStack models by factor of 3x to 10x. We believe this performance can be further enhanced with more optimized CUDA kernels, which we plan to explore in future work. We further compare the efficiency of BitStack with quantization- based methods such as GPTQ. Specifically,
the well-optimized Triton inference kernels in GPTQMode to measure the practical inference latency. As shown in Table 7 the latency difference between BitStack and quantization-base methods is insignificant, especially given that the inference kernels for GPTQ are highly optimized by the community.
To evaluate the efficiency of BitStack in memory-constrained scenarios, we compare its perfor- mance with model offloading, system-level memory management approach that enables large model inference by offloading unused parts of the model to storage, thus allowing functionality in variable memory environments. We implement this baseline using the Hugging Face Accelerate library (Gugger al. 2022).
20

---

Published as a conference paper at ICLR 2025
Model
W2
W2g128
W3
W3g128
W4
W4g128
GPTQ
1.58
1.78
3.99
4.31
1.97
2.10
BitStack
1.79
1.95
2.32
2.34
2.60
2.69
Table 7: Latency generating 50 tokens on an H800 with Llama 2 7B at different compression settings with GPTQ and BitStack. The compression ratios of the BitStack model are aligned with those the GPTQ models at various compression settings.
BitStack (Generation Time)
132.74
Weight Offloading (Generation Time)
BirStack (Perplexity)
118.35
Full Model (Perplexity)
Generation Time (secondss
223
54 86c
Perplexity
72.236
59.415
-1.00s
11.915
2.76
3.45
4.00%
5:215
5.7k
5.40
6.90K
21R8
Available Memory (GiB)
Figure 11:
Comparison of generation time and WikiText2 perplexity between BitStack and model offloading in memory-constrained environments. The generation time is measured by generating 50 tokens with the Llama 7B model on an Nvidia H800 GPU.
We measure the throughput of BitStack (using the level Triton implementation) and model of- floading across various memory budgets, shown in Figure [1] The figure clearly demonstrates that BitStack significantly outperforms the model offloading method in terms of throughput in all cases where offloading occurs. Note that the full Llama 2 7B model requires approximately 12.55 GiB memory, which less than 13 GiB, meaning no model offloading occurs at 13 GiB point. The figure shows that BitStack provides speedup over model offloading by factor of 1.6x to 61.4x across different memory budgets, while maintaining comparable quality at most memory budgets (available memory 3 GiB). Additionally, the throughput of model offloading is heavily constrained by I/O bandwidth, and latency can become more pronounced on edge devices with lower memory bandwidths. This further highlights the advantages of BitStack in memory-constraine en- vironments.
A.6 COMPARISION TO OTHER DECOMPOSITION-BASHI APPROACHES
Model
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
ASVD
5.89
9.88
NaN
NaN
NaN
NaN
NaN
NaN
NaN
SVD-LLM
7.27
8.38
10.67
16.14
33.28
89.92
253.22
570.56
1474.09
SVD-LLM
7.60
8.84
11.15
16.11
27.20
54.19
125.17
356.43
966.83
BitStack
5.74
5.75
5.78
5.79
5.85
5.92
6.26
8.23
Table 8: WikiText2 perplexity of compressed Llama 2 7B at different comparison ratios. V for SVD-LLM without parameter update detailed in Wang et al. (2024)); for SVD-LLM with parameter update.
In this section, we compare BitStack with other decomposition- model compression ap- proaches. We reproduce ASVD Yuuan al. 2023) and L.M-SVD (Wang al. 2024) using their
21

---

Published as a conference paper at ICLR 2025
official implementations and evaluate the compressed Llama 2 7B models different compres- sion ratios. As shown in Table BitStack significantly outperforms these methods across all com- pression ratios, particularly at high compression ratios. It is important note that this comparison even unfair to BitStack, as the compression ratios are calculated differently these baselines. Specifically, the compression ratio in these baselines is derived from the concatenated ratio of sin- gular values, whereas BitStack's compression ratio is strictly calculated
compresed model memory original model memory accounting for the memory consumption of uncompressed weights in layers such as the embed- ding and layer normalization. For instance, an SVD-LLM checkpoint at a 0.8 compression ratio consumes 5.44 GiB of memory, while BitStack only consumes 2.51 GiB. At compression ratio of 0.9, the actual available memory for BitStack would be 1.26 GiB, which is insufficient for one residual block per weight stack, while the SVD-LLM model requires 3.02 GiB, corresponding to the BitStack model at compression ratio of 0.76.
A.7 VISUALIZATIONS OF WEIGHT STACKS
In Figure 12, we provide the visualization of the weight stacks in BitStack for three different sort- ing approaches, as detailed Section 2.2] The Average approach, which we adopt in BitStack, exhibits minimal variance in the memory consumption of different stacks, benefiting load balancing distributed deployment. Moreover, demonstrates excellent performance in our experiments, particularly at extreme compression ratios.
22

---

Published as a conference paper at ICLR 2025
Figure 12: Visualization of the weight stacks in BitStack Llama 3.1 8B with three different sorting approaches. We plot the number of residual blocks in each weight stack in the BitStack model, ranging from 4000MB 6000MB, with a stride of 500MB, due to space constraints.
23


Reviews:
[{"review_id": "13B6vMCtHG", "reviewer": "Reviewer_HY8o", "summary": "Authors propose BitStack, a training-free weight compression method using weight decomposition. The method performs well on extreme compression ratio. The method also allows model of an arbitrary size for whichever the GPU memory allows.", "questions": "1. How long does it take to dynamically load/off-load a residual block? What is the strategy to achieve the best performance/latency tradeoff? Is more or less adjustment preferred during inference?\n2. Based on the comparison between 8B and 70B, it seems that one should use llama-8B at FP16 instead of llama-70B at an extreme compression ratio. At least in this case, the extreme quantization does not seem to be needed. And I assume that the latency of a quantized larger model is also worse than the latency of a smaller model at FP16.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Paper is clearly written and well-motivated.\n2. A method that adapt models to different sizes dynamically should be very useful in practice.", "weaknesses": "See questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bTTqsW3bh7", "reviewer": "Reviewer_9SjC", "summary": "This paper introduces BitStack, a novel training-free compression approach for Large Language Models (LLMs) that addresses the challenge of deploying models in environments with variable memory constraints. Unlike traditional compression methods such as quantization that require predefined compression ratios and separate compression processes for each setting, BitStack enables dynamic adjustment of model size at the megabyte level through an innovative weight decomposition technique. The method works by iteratively decomposing weight matrices based on parameter significance, creating approximately 1-bit per parameter residual blocks that are stored and can be loaded flexibly based on available memory. BitStack not only provides fine-grained control over the memory-performance trade-off but also matches or exceeds the performance of established quantization methods like GPTQ and AWQ, particularly at extreme compression ratios. This approach is particularly valuable for local deployment scenarios where available memory may fluctuate due to other applications running concurrently.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel Problem Framing: The paper addresses a practical and important problem of deploying LLMs in variable memory environments\nIt identifies a gap in existing compression methods that typically require fixed compression ratios.\n- Practical Utility: Enables dynamic trade-offs between memory usage and model performance eliminating the need for multiple compressed versions of the same model. This reduces storage burden by avoiding separate compression processes for different ratios\n- A comprehensive set of experiments across different model scales with convincing results and good ablations to justify different design choices.", "weaknesses": "### Major\n- This paper is motivated to work in variable memory environments where the amount of memory changes and a compression algorithm is provided. It would be great to have some results on how this compares in efficiency to other memory management techniques like paging. In what scenario is and how more effective is BitStack compared to a systems level optimization that allows to offload parts of the model not currently being used ?\n\n### Minor\n- In Table 1, 2,3 it might be a good idea to make the best method's results in bold, to make it easier for the reader.\n- Could you clarify what the multiple folders and smaller llama in figure 1a actually refer to ?\n- In the abstract the paper references that the primary bottleneck for LLMs is capability and not availability. How does this factor in with other LLM problems like hallucination, knowledge cutoffs etc?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yEyH2zJFxn", "reviewer": "Reviewer_g6Mj", "summary": "The paper introduces BitStack, an iterative weight decomposition method to compress the Large Language Models (LLMs). By adjusting the number of iterations, BitStack can compress LLMs with various compression ratio. Experimental results demonstrate that BitStack achieves comparable performance to quantization-based methods.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.It is the first decomposition-based method that achieves comparable performance to quantization approaches.\n\n2.The motivation of variable RAM on devices sounds is interesting.\n\n3.The model compressed by BitStack is more scalable for deployment.", "weaknesses": "1.The evaluations should be enhanced. Both AWQ and GPTQ are no longer considered state-of-the-art quantization methods as of the current time. Even AWQ was submitted to arXiv over a year ago (June 2023).\n\n2.Despite this, the performance gap between BitStack and AWQ is relatively small. For instance, the average accuracy scores of BitStack surpass those of AWQ-w2 by 1.4 points (30.8 vs. 29.4) across the LLaMA 3.1 8B model. Given that the full-precision model achieves an accuracy score of 74.1, this improvement is quite marginal.\n\n3. When the compression ratio is below 70%, BitStack's performance tends to be weaker than AWQ on most benchmarks.\n\n4. What is the real-world inference speed of BitStack? This is a crucial aspect for a compression method targeting large language models (LLMs), yet it is not addressed in the paper.\n\n5. The font size in Figures 1, 2, 5, 6, and 7 is too small", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UzWYU4ZGaP", "reviewer": "Reviewer_GmCn", "summary": "This paper presents BitStack, a training-free weight compression method for large language models that adeptly navigates memory constraints on local devices. By utilizing weight decomposition and stacking blocks in storage as basic transmission units, BitStack enables dynamic adjustments to model size, offering megabyte-level trade-offs between memory usage and performance. The method achieves comparable or superior results to traditional quantization techniques.", "questions": "- Certain terms, like 'residual block', need to be defined more clearly as it seems to differ from its usage in the context of CNNs.\n- The choice of \"zeropoint quantization\" as a baseline needs to be justified since simple zero point quantization is known to be ineffective in LLMs.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper addresses a crucial problem in memory allocation for large language models (LLMs) in edge computing scenarios with limited RAM.\n- The research is backed by solid experimental evidence, enhancing its credibility.", "weaknesses": "- The writing could be improved for clarity. The rationale for sorting SVD matrices is not well-explained, and the process of loading/offloading within and between layers needs better elucidation.\n- The paper doesn't provide details about the caching process, which seems to be an integral part of the method. Discussion on cache size and caching strategy is missing.\n- The impacts of SVD on model performance aren't discussed. The paper should elaborate on the effects of various decomposition parameters and distinguish the proposed SVD from those used in previous model compression techniques.\n- The sorting algorithm and strategy need better description and justification. It's unclear how different matrices in LLMs should be decomposed and sorted using the proposed method.\n\nThe rebuttal has solved most of my concerns therefore I'd like to raise my score. However, I still think the scenario defined in this paper can be changed a bit. As the author claimed in their response, \"BitStack is proposed as a general weight compression method for LLMs\".  If the paper emphasize it is well suited for mobile/edge scenarios, I'd like to see more system design involved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)