LittleBit: Ultra Low-Bit Quantization via Latent Factorization
Authors: Banseok Lee
Dongkyu Kim
Youngcheon you
Young-Min Kim
Venue: NeurIPS 2025 poster ()
Keywords: Large language models
Low-bit quantization
Low-rank factorization
Sub-1-bit model compression
Binarization
Primary area: deep_learning
TL;DR: This paper presents LittleBit, a novel framework that combines latent matrix factorization and a multi-scale compensation mechanism to compress Large Language Models (LLMs) to ultra-low bit levels.

Abstract: The deployment of large language models (LLMs) is frequently hindered by prohibitive memory and computational requirements. While quantization mitigates these bottlenecks, maintaining model fidelity in the sub-1-bit regime remains a persistent challenge. In this paper, we introduce LittleBit, a novel framework for extreme LLM compression. We target quantization rates as low as $0.1$ bits per weight (BPW), achieving a memory reduction of approximately $31\times$, which effectively compresses Llama2-13B to under $0.9$ GB. We represent weights via low-rank latent matrix factorization and subsequently binarize the resulting factors. To counteract the information loss inherent to such drastic precision reduction, we integrate a multi-scale compensation mechanism that learns importance parameters across row, column, and latent dimensions. Two primary contributions enable effective training: Dual Sign-Value-Independent Decomposition (Dual-SVID) for quantization-aware training (QAT) initialization, and Residual Compensation to minimize approximation errors. Extensive experiments confirm the superiority of LittleBit in the sub-1-bit domain; for instance, our method at $0.1$ BPW surpasses the performance of leading techniques operating at $0.7$ BPW on Llama2-7B. We establish a new size-performance trade-off---unlocking a potential $11.6\times$ inference speedup relative to FP16---and render powerful LLMs practical for resource-constrained environments. Our code is available at https://github.com/SamsungLabs/LittleBit.

LittleBit: Ultra Low-Bit Quantization via Latent Factorization
Banseok Lee"
Dongkyu Kim
Youngcheon You Youngmin Kim
Samsung Research
(bs93.1es dongkyu. y01000. you, ym1012. (e112.iimmae wwiikkmmmaaaaa.... (aammmssmmuunn..mmmmmmmmm com
Abstract
The deployment of large language models (LLMs) frequently hindered by pro- hibitive memory and computational requirements. While quantization mitigates these bottlenecks, maintaining model fidelity in the sub-1 regime remains a per- sistent challenge. In this paper, we introduce LITTLEBIT, novel framework for ex- treme I.I. compression. We target quantization rates as low as 0.1 bits per weight (BPW), achieving memory reduction of approximately 31x, which effectively compresses Llama2- 13B to under 0.9 GB. We represent weights via low-rank latent matrix factorization and subsequently binarize the resulting factors. To counteract the information loss inherent to such drastic precision reduction, we integrate a multi-scale compensation mechanism that learns importance parameters across row, column, and latent dimensions. Two primary contributions enable effective training: Dual Sign- Sign-Value Value- Independee Decomposition (Dual-SVID) for quantization-awe aware training (QAT) initialization, and Residual Compensation to minimize approxima- tion errors. Extensive experiments confirm the superiority of LITTLEBIT in the sub-1-bit domain; for instance, our method at 0. BPW surpasses the performance of leading techniques operating at 0.7 BPW on Llama2-7B. We establish new size- performance trade-off- uflloccoo potential 11.6x inference speedup relative to FP16 and render powerful LLMs practical for resource-constrained environments. tigh........ www hotm///ansunggabb/iittteeee.
Our code is available at https::/// //github..
Introduction
Large language models (LLMs) based on the Transformer architecture [1] have fundamentally transformed natural language processing. However, the scale of these models, often reaching hundreds of billions or trillions of parameters [2, 3], results in prohibitive computational and memory costs. In particular, the high demand for GPU VRAM hinders widespread deployment on consumer or edge devices [4-6].
Model quantization, which reduces numerical precision, serves primary technique address these bottlenecks [7]. While post-training quantization (PTQ) methods such as GPTQ [8] and AWQ [9] effectively compress models to approximately 4 bits, achieving further compression (e.g., to 1-bit) typically requires quantization-awanl training (QAT) [10, 11] to maintain performance. QAT has demonstrated the capability enable 1-bit compression while preserving model fidelity [12-14).
Nevertheless, even 1-bit models (e.g., approximately 15.4 GB for 70B parameters (7]) may exceed the capacity of highly resource-constrained devices [15]. This limitation motivates the exploration of sub- quantization. Although prior work [16] has investigated this area, maintaining performance at extremely low effective bits (e.g., 0.55 bits per weight (BPW)) while ensuring computational
"qqual contribution Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

2100.00
Method
RTN
10a
GPTQ
893.82
OmniQuant
BILLM
STELLM
Perplexity (PPL) 1
BinaryMoS
LittleBit (Ours)
122,08
102
93.08
27.05
23.63
23.71
16.88
14,00
11.90
15.09
10:
4,98
5.51
8:25
8,18
8.52
9.24
10.48
4,88
4.98
6.42
47.41
8.76
4.88
4.95
3:28
0.5
0.25
Quantization Bit-width
Figure 1: Low-bit quantization perplexity Llama2-13B (WikiText-2). LITTLEBIT surpasses the state-of-the-art sub-1-bit quantization technique. Below 0.5 BPW, where the leading prior method degrades sharply, ours remains robust down to 0.1 BPW.
efficiency without restrictive hardware dependencies remains challenging [16]. Consequently. it is necessary to develop new techniques that achieve such extreme model compression, aiming for bit rates around 0.1 BPW, while preserving model performance.
We derive our approach from two observations. First, LLM weight matrices often exhibit pronounced low-rank structure [17, 18). This observation suggests that factorization methods such as Singular Value Decomposition (SVD) [19] may offer a more stable compression pathway than pruning [20], particularly under extreme compression regimes. For instance, at compression ratios exceeding 50%, SVD-based approaches [21] consistently outperform pruning methods such as Wanda [22] (see Appendix E for details). Second, binarization inherently causes information loss [23]. Recent high-performing 1-bit methods [13, 24, 25) demonstrate that comprehensive scaling over multiple dimensions (e.g., row and column) is crucial for mitigating this loss and stabilizing performance [25].
Building on these insights, we introduce LITTLEBIT, a novel method for extreme sub-I-bit quan- tization (e. 0.1 BPW). In LITTLEBIT, we first represent weights via low-rank factorization (W UV and subsequently binarize these factors. counteract errors from binarization, we leverage multi-scale compensation mechanism that applies learnable scales across rows, columns, and an additional latent dimension. We complement this architecture with Residual Compensation.
Extensive experiments demonstrate the superiority of LITTLEBIT over STBLLM [16], the leading sub-1 1-bit technique, on LLMs ranging from 1.3B 32B parameters. Notably, LITTLEBIT achieves competitive performance on Llama2-133 13B at 0.1 BPW (Fig. 1), surpassing STBLLM at 0.55 BPW. Furthermore, the 32B LITTLEBIT model at just 0.3 BPW maintains strong performance, substantially outperforming STBLLM and establishing new state-of-the-art for sub-l-bir quantization. In summary, contributions
follows:
We propose LITTLEBIT, a novel framework unifying latent matrix factorization with multi- scale compensation enable extreme sub-l-bit quantization. This method achieves effective bits down to 0.1 BPW while preserving model performance, establishing a viable size- performance off for deploying LLMs in resource- coouurceconnstriieed environments.
We introduce Dual-SVID for the effective initialization of factorized structures and integrate Residual Compensation to mitigate approximation errors. These techniques collectively stabilize quantization-aware training (QAT) and counteract the information loss inherent in drastic precision reduction, ensuring robust learning dynamics.
We provide extensive empirical validation demonstrating that LITTLEBIT consistently out- performs STBLLM [16], the prior leading sub- 1-bit technique, across various model scales. Our results confirm that LITTLEBIT maintains superior fidelity the deep compression regime, where baseline methods typically suffer severe degradation.

---

2 Related Works
2.1 Binarization and Quantization
Network binarization aims for extreme compression and acceleration by constraining weights or activations to +1. This enables efficient bitwise operations [26, 27]. While early studies such as BinaryConnect [28] and BNNs [26] demonstrated feasibility, they encountered substantial accuracy
degradation [29]. Researchers partially mitigated this degradation by introducing scaling factors [30] and employing the Brraight-Througee Estimator (STE) [31] to handle the non-differentiable quantiza- tion function during training [32].
The direct application of these early techniques to large language models (LLMs) typically results
in severe performance loss [33]. Consequently, L.M--pecific quantization strategies have emerged. These are primarily categorized into post-training quantization (PTQ) and quantization-awan training (QAT) [34]. PTQ adapts pre-tras rre-trained models and achieves competitive results at approximately 4-bit precision with methods such as GPTQ [8] and AWQ [9]. However, these methods generally struggle to maintain performance below 2-bit precision [33, 7J. The l-bit regime presents particular challenges for PTQ methods due to severe information loss [35]. Recent work such as STBLLM [16] demonstrates that incorporating structured binarized compression can extend PTQ into the sub- 1-bb--bit regime, but a non negligige accuracy gap persists.
In contrast, QAT integrates the quantization process directly into the training loop. This enables the model to learn and adapt to the low-precision format [36]. This approach generally yields superior performance compared to PTQ at very low bit-widths. Successful QAT approaches for low-bit scenarios often employ sophisticated scaling mechanisms alongside adaptive training, as evidenced by recent methods [13, 24]. Despite these advancements, consistently achieving high model fidelity when quantizing to less than 1.0 BPW remains a considerable challenge. This persisting difficulty underscores the need for novel architectures and training strategies specifically designed for such extreme compression regimes.
2.2 Low-Rank Approximation in Quantization
Beyond quantization, the inherent parameter redundancy within deep neural networks [35] can be effectively exploited using low-rank approximation methods. This redundancy is particularly evident in LLMS [37, 38). Techniques such as Singular Value Decomposition (SVD) [19] allow large weight matrices to be represented as products of smaller factors. This offers an alternative compression strategy that may be more resilient than pruning, especially under high compression regimes [20]. Recognizing the potential for complementary benefits, researchers have explored combining low-rank factorization reduce parameter count with quantization to lower parameter precision [39]. Initial studies applied SVD in various roles within the LLM context. Some approaches used activation statistics to guide decomposition [21], while others integrated SVD principles into initialization or parameter-efficite
updates [40, 41, 13]. Further works applied transformations prior to factorization [42] or employed low-rank structures to model quantization errors [43]. more integrated approach involves incorporating low-rank concepts directly within QAT methods. For example, DL-QAT [44] jointly learns representation amenable both low-rank approximation and low-precision quantization. However, this method is currently limited to 3-bit quantization. Optimizing structural properties such as rank and numerical precision during training represents promising path toward effective LLM compression. There is potential for further exploration of even lower precision quantization.
3 Methodology
This section details LITTLEBIT. It describes the factorized architecture with multi-scale com- pensation (Section 3.1), the Dual-SVID initialization for stable QAT (Section 3.2), and Residual Compensation for enhanced fidelity (Section 3.3). These components are optimized via QAT, as detailed in Section 4.

---

Linear
Input
W
pri
Linwar Layer (Q projection)
Linear Layer projection)
Linier projecion)
W
Dual-SVID
W
W nes
DuakSViD
prixsign
Spri
Vies.sign
Bres
Activation
MatMuul
layer (FC2)
O
O
Activation
f
R
Linear
Linerr Layer (O projectio))
prisign
U res,iggn
Activation
hpri
Prres
Primary
Residual
Feed Fond-Foowward
Multi- Head Salf Self-Attantion
Block
Figure 2: Comparison of a standard Transformer layer (left) and the LITTLEBIT architecture (right). LITTLEBIT performs linear transformation using parallel Primary and Residual pathways. The Pri- mary path employs binarized factors (Usign: Vaign) and FP16 scales (h, g, ê) on input X, initialized from W via Dual-SVID. Simultaneously, the Residual path computes correction with its parameters (U reg,s V ses, hres, Bres: fres) from the approximation residual. Their outputs sum to form Y, eliminating storage of the effective weight matrices W pri and W res-
3.1 LITTLEBIT Architecture
The LITTLEBIT architecture, depicted in Fig. 2, redesigns linear layers for extreme compression (e.g., 0.1 BPW) by synergistically combining low-rank factorization and binarization. The method leverages the observed low-rank structure LLM weights [45] to approximate the weight matrix W e ddou den as:
UV (U
V E Rdurr, r < min(dd.: dout))
(1)
We prioritize this factorization-bass approach over other structural compression methods, such as pruming, due its superior robustness in extreme compression regimes. As the comparative analysis against methods like Wanda demonstrates (Appendix E), SVD-based compression maintains model integrity far more effectively at high compression ratios. This provides more stable foundation for ultra low-bit bit quantization. The resulting factors U and V are then binarized:
U sign sign(U). Vsign = sign V) E (-1,+1}
(2)
To compensate for the significant information loss incurred by binarization [23], LITTLEBIT in- corporates learnable FP16 scales. Beyond standard row (h) and column (g) scaling [33, 13, 25], the architecture introduces an additional latent scale (e E R'). This latent scale addresses the factorization structure by learning the relative importance of each of the latent dimensions, which correspond to the columns of the binarized factors Usign and V sign:
h Rddu
g RAn. i R"
The primary effective weight, W pri is implicitly constructed from these components:
pei diag(h) Usign (e ( JTn diag(g)
Instead of storing or optimizing W pri directly, the model learns latent full-precine precisiioonnn factors U and V. These are binarized during the forward pass using Eq. (2), along with the FP16 scales h, g, l. This factorized representation enables efficient computation during the forward pass.
Proposition Let X Raeq be the input matrix. The forward computation Y XW the primary approximation (Eq. (4)) is efficiently computed as:
pri using
Y ((((X + (g)) V sign) O L) U sign) c h
(5)

---

where C denotes element-wise multiplication with broadcasting.
This decomposition (Eq.. (5)) replaces large high-precision General Matrix Multiply (GEMM) operation with two smaller binary matrix multiplications and element-wise scaling operations. This offers computational advantages and reduces memory requirements. The effective bits per weight (BPW) is determined by the storage cost of these learnable parameters relative to the original matrix size (see Appendix D for details).
3.2 Dual-SVID Initialization
Naively initializing the highly constrained LITTLEBIT structure can lead to unstable QAT. To circumvent this, we propose Dual-SVID, SVD-based initialization method designed provide a starting point where the initial effective weight W Wriio closely approximates W. Dual-SVID aims to preserve essential sign and magnitude information from the optimal low-rank SVD factors (obtained from W U'V then truncated) and map this information to the learnable LITTLEBIT parameters (U Vsign, h, L). The process involves three steps: (1) Obtaining U', V via a truncated SVD. (2) Setting initial binary factors based on signs (Eq. (6)). (3) Decomposing the magnitudes IU' E Rdomx x and V'| E Rd xr, To initialize the scales, a rank- I SVD (or equivalent rank-1 approximation) is performed on each of these magnitude matrices separately. For [U'], its best rank-1 approximation row scale ho. and lu,0 E R is a component contributing to the latent scale. Similarly, for [V', its rank-1 1 approximation $y,o(Lr.)) provides the initial column scale go = Sv.u E R and another latent scale component L..0 R". These steps effectively disentangle multi nulti-dimensional magnitude variations and allow for the estimation of initial row (ho), column (gn), and the final latent scale (Co Lu,0 @ Lu by element-wise product), shown in Eqs. (7) and (8):
be written (t;o(Lw.o)) T where SU.0 E R becomes the initial
U sign.0 sign(U'),
V Vaign. sign (V))
(6)
UU| ho (T to = Lu,0 & Lo,0
(o
(7)
(8)
The resulting initial effective weight W pri,o = diag (ho) ) ssign, dddag ( VT diag preserves key structural information. The learnable parameters are initialized follows: the latent factors U and are initialized with the factors U' and V obtained from SVD, respectively, and the scales h, g, l are initialized to ho, go, to
3.3 Residual Compensation
To further improve fidelity without increasing the primary path's rank, we introduce Residual Compensation. This technique is motivated by the theoretical insight that two-stage error correction can be more effective than a single, larger approximation (Appendix A). Crucially, this method does not increase the model's total parameter budget; instead, the fixed bit budget of a single, higher-rank approximation strategically reallocated into two lower-rank paths: a primary and a residual path. The term 'residual' primarily describes its role during initialization, where the auxiliary path is configured to model the eтror of the primary approximation. During QAT, both paths are optimized jointly to collectively represent the original weight. The advantage of this dual- -allpathh approach over single path is empirically validated in our ablation studies (Appendix B.1).
To implement this, Residual Compensation employs a parallel auxiliary path that learns approx- imation W res of the residual error. This auxiliary path mirrors the structure of the LITTLEBIT path:
res (hres) Ures, Uressiggn diag Cres) V sen diag(gres)
The parameters of this auxiliary path (Ures.sign> V Vres.sign: hres gres, fres) are also learnable during QAT. They are initialized using the Dual-SVID strategy applied to the initial residual error W Vresoo W Wrri, Employing a separate path allows the model to specifically learn and compensate for the potentially distinct characteristics of this residual error. The final effective weight is the sum of both approximations:
W
W pri W res
(10)
5

---

W pri,0
0.22
W res,0
-0.01
Wo
W
0.1 0.3
0.55
0.7
0.8
1.0
Figure 3: Visualization of Dual-SVID initialized weight components for selected layer in Llama2- 7B (Query Weight, Layer 0). Columns, from left to right, represent effective bits of 0.1, 0.3, 0.55, 0.7, 0.8, and 1.0 BPW. Rows display the primary approximation (W pei,o), the residual approximation (W o)) and their sum (Wu W pei.U + Wee,,)). The rightmost image shows the corresponding crop of the original weight matrix (W) for reference.
This targeted error correction is often advantageous for maintaining performance extreme compres- sion levels, as demonstrated by the ablation studies presented in Appendix B.1. It is important to note that all learnable parameters from both the primary and residual paths are included when calculating the final BPW (see Appendix D for calculation details).
Figure 3 visualizes the initial state of weight components (W pri.0. W Wres.o, and their sum Wo) for Llama2-78 query weight, derived via Dual-SVID across effective bits from 0.1 to 1.0, compared against the original weight W (far right). Even low 0.1 BPW, the primary approximation W pri.o (top row, leftmost) captures the dominant low-rank structure of W, albeit with considerable simplification. As effective bits increase, W progressively refines details and more effectively suppresses disruptive approximation noise, allowing underlying weight patterns emerge with greater clarity. A key observation is that Wo (bottom row) at a modest 0.3 UPW-benefiting from the initialized residual component W res.0 (middle rowe-often presents more faithful initial representation of W than pri,o alone even higher 1.0 BPW. This underscores that Residual Compensation, even at the initialization stage, is vital for capturing complexities missed by the primary low-rank approximation, thereby providing more faithful starting point for QAT.
4 Experiments
4.1 Settings
Evaluation Setup We evaluated LITTLEBIT across diverse LLM families, including Llama [46], Llama2 [47], Llama3 [48], OPT [49], Phi-4 [50], and QwQ [51]. These models span parameter scales from 1.3B to 32B. Perplexity (PPL) on the WikiText-2 [52] validation dataset served as the primary performance metric. Appendix provides additional results on the C4 [53] and PTB [54] datasets. Furthermore, we assessed zero-shot accuracy common reasoning benchmarks: WinoGrande [55], OpenBookQA (OBQA) [56], HellaSwag [57], BoolQ [58], ARC-Fasy (ARC-e), ARC-Challenge (ARC-c) [59], and PIQA [60].
Training Details We optimized the LITTLEBIT model parameters, initialized via the Dual-SVID method (Section 3.2), using QAT with knowledge distillation (KD) [61, 36, 62]. The original pre- trained full-precision model functioned the teacher (T) for the LITTLEBIT student model (S). The QAT objective combines the standard output Kullback-Lebllr (KL) divergence loss, Couts and an intermediate layer mean squared error (MSE) loss, Linter- to match hidden representations. We weighted these terms using empirically determined coefficient. X = 10:
LQAT Lout + XLinter
(11)

---

Table I: Perplexity (PPL) comparison WikiText-2 across various LLMs and quantization methods. Lower PPL indicates better performance. BPW denotes effective bits per weight. Methods marked with use quantization-avee training (QAT). STBLLM [16] utilizes N:M sparsity (ratio in parenthe- ses). LITTLEBIT demonstrates strong performance, particularly in the lubit regime.
Settings
OPT
Llama
Llama2
Llama3
Phi-4
QwQ
Method
Block Size
1.38
7B
13B
7B
13B
14.78
32B
FullPrecision
14.62
5.67
5.09
4.88
6.10
6.67
6.34
OmniQuant
28.82
9.75
7.83
11.20
8.25
349.27
12.09
10.19
GPTQ
128
119.98
39.96
15.08
52.22
23.63
1.5e3
24.96
67.16
BILLM
128
1.1
74.07
41.95
13.95
29.00
23.71
54.29
16.95
15.4
OneBit
20.27
8.21
7.37
8.36
7.41
13.09
9.92
9.86
BinaryMoS'
18.09
7.84
7.05
7.74
6.95
10.83
9.51
8,99
LITTLEBIT
20.33
9.03
8.17
9.08
8.18
16.30
11.28
12.08
STBLLM
128
0.80 (6:8)
52.13
15.19
9.43
13.81
11.90
30.90
12.12
12.19
STBLLM
128
0.70 (5:8)
73.42
19.52
11.47
19.17
14.00
59.83
14.57
13.78
STBLLM
128
0.55 (4:8)
123.03
38.73
16.88
30.67
27.05
241.95
21.99
18.32
STBLLM
128
0.30 (2:8)
2.3c3
1.6c3
592.06
1.8c3
893.82
1.7e5
761.05
512.01
LITTLEBIT
0.80
21.32
9.44
8.50
9.53
8.52
17.28
11.70
12,46
LITTLEBIT
0.70
21.99
9.66
8.71
9.85
8.76
18.01
12.05
13.22
LITTLEBIT
0.55
23.35
10.09
9.16
10.47
9.24
18.47
12.80
13,57
LITTLEBIT
0.30
28.30
11.50
10.33
12.00
10.48
20.34
14.71
16.48
LITTLEBIT
0.10
53.76
15.58
13.71
15.92
15.09
26.11
19.73
35.26
Adhering to the protocol in [24]. the training data combined WikiTeext22 with selected partitions from
C4. The configuration included sequence length of 2048 tokens, epochs, the Adam optimizer (81 = 0.9, 82 = 0.999) [63]. and a cosine learning rate decay with 2% warm-up (see Appendix G for details). For models employing Srouped-Query Attention (GQA) [64]. such as Llama3, Phi-4, and QwQ we specifically adjusted the latent ranks for key (K) and value (V) projection layers (see Appendix B.3) maintain performance under extreme quantization. We employed SmoothSign (forward: sign(r); backward gradient: tanh(kz), & = 100) for backpropagation, as it demonstrated superior performance compared to the Brright-Thhrogll Estimator (STE) [31] (see Appendix B. B.2). Although proxy gradients are established in the literature, this work validates the specific application of the tanh(100r) derivative as a stable gradient for QAT in the ultra-low-bit regime. Our ablation study confirms the effectiveness of this approach (Appendix B.2).
Baselines We benchmarked LITTLEBIT across effective bit rates ranging from approximately 1.0 down to 0.1 bits per weight (BPW). In the challenging u-bit regime, we compared our method against STBLLM [16], a post-training quantization (PTQ) method that employs N:M sparsity. For settings approximating 1.0 BPW, we compared against the 1-bit PTQ method BiLLM [7] and quantization-ave training (QAT) approaches such as OneBit [13] and Bonary [24]. We included results from standard low-bit methods, such as GPTQ [8] and OmniQuant [65], to provide broader context.
4.2 Main Results
Superior Perplexity in Sub-1-bit Regime Table I presents the perplexity (PPL) results of LIT- TLEBIT compared to baseline methods. The proposed method demonstrates robust performance, particularly within the sub-1-bit regime. Relative to STBLLM [16], LITTLEBIT achieves markedly superior PPL scores 0.8, 0.7, and 0.55 BPW. For instance, on Llama2-7B, LITTLEBIT 0.55 BPW achieves a PPL of 10.47, which represents substantial improvement over the 30.67 score reported for STBLLM. We observe comparable gains across other models and bit-widths, For 1laam2- 13B at 0.8 BPW, LITTLEBIT records PPL. of 8.52, whereas STBLLM records 11.90.
Extreme Low-BPW Stability A primary advantage of LITTLEBIT lies in its exceptional stability and performance in the extreme sub-0.5 BPW range, specifically at 0.3 BPW and 0.1 BPW. In this regime, methods such as STBLLM exhibit severe performance degradation. For example, Llama2-7B yields PPI. of 1.8 x 10 at 0.3 BPW. Conversely, LITTLEBIT maintains strong PPL scores, such as 12.00 for Llama2-7B 0.3 BPW and 15.92 at an unprecedented rate of 0.1 BPW. When configured for effective bits approximating 1.0 BPW by adjusting the latent rank LITTLEBIT yields perplexity
7

---

Table 2: Zero-shot accuracy (%) comparison common reasoning benchmarks. Compares Full Precision (FP16) models against STBLLM [16] and LITTLEBIT at 0.55 and 0.3 BPW.
Models
Method
WinoGrande
OBQA
HellaSwag
BoolQ
ARC-e
ARC-c
PIQA
Average
FullPrecision
67.16
40.80
72.95
40.95
STBLLM (0.55)
$2.80
33.00
36.94
62.17
37.33
25.34
62. 46
44,29
Llama2-7B
STRILM (0.3)
50.43
31.80
26.20
37.83
25.88
25.09
53.26
35.78
LITTLBIIT (0.55)
$1.62
34,00
44.57
61.38
44.15
26.96
68.12
47.26
LITTLEBIT (0.3)
51.30
34,80
37.61
61.80
39,98
25.09
65.83
45.20
FullPrecision
69.45
41.80
76.58
69.29
73.19
44.53
78.61
64.78
STBLLM (0.55)
55.25
31,20
35.10
62.23
42.51
27.
61.75
45.04
Llama2-13B
STBLLMM (0.3)
$1.54
28,80
26.01
55.63
27.53
24.74
53.32
38.22
LIT (0.55)
53,04
35,60
51.06
30.58
46.09
29.10
70.78
48.03
LITTLEBIT (0.3)
51.30
34.00
43.81
53.96
42.59
25.17
68.77
45.94
FullPrecision
72.92
45,00
79.18
81.25
80.21
52.98
79.54
70.15
STBLLM (0.55)
52.17
25.80
30.61
57.16
30.35
23.72
57.56
39.62
Llama3 8B
STBLLM (0.3)
49.57
26.60
26.36
51.62
26.05
24.40
51.74
36.62
BIT (0.55)
50.12
30.20
36.78
57.55
46.80
22.95
66.27
44.38
Lar (0.3)
51.93
28.20
33.91
57.98
43.48
24.32
64.64
43.49
WinoGrande
WinoGrande
Average
OpenbookOA
Average
OpenbookOA
50
PIQA
ellaSWag
PIQA
ellaSWag
ARC-c
BaaiQ
ARC-c
Boo
ARC-e
ARC-e
0.55
Ltt599t00.55
STBLLM
0.55
STBLLM--.3 0.3 UPW
Littlei--.. BPW
STULLM-O. 0.0 UPW
Littielit-0.. 0.J UPW
STBLLM
LittceBt-0.1
STBLLM
LittieBit-0.1
(a) Phi-4 14B
(b) QwQ-32B
Figure 4: Zero-shot accuracy (%) on 7 common sense reasoning tasks. The comparison highlights the
robustness of LITTLEBIT (solid lines) against STBLLM (dashed lines) in the sub-0.5 BPW regime. While STBLLM degrades sharply, LITTLEBIT preserves viable reasoning capabilities for both (a) Phi-4 and (b) QwQ models.
scores comparable to specialized 1-bit QAT methods. For example, on Llama2-7B, LITTLEBIT achieves 9.08 compared to 8.36 for OneBit [13]. It also remains competitive with PTQ methods such as BiLLM [7]. Although BinaryMoS [24] may exhibit slightly lower PPL in certain 1-bit scenarios (e.g., Llama2-7B: Mon 7.74), this performance difference is attributable to its use of more complex mechanisms, such as dynamic scaling, which represent distinct set of architectural and computational trade-offs. Our analysis of the trade-off involved in adjusting the latent rank to tune effective bits from 1.0 down to 0.1 BPW indicates that performance degradation minimal down to approximately 0.55 BPW. However, quantization cliff appears between 0.3 and 0.1 BPW. This observation positions the 0.3-0.55 BPW range as an optimal sweet spot for balancing compression and accuracy.
Model Generalization & Zero-Shot We consistently observe the advantages of LITTLEBIT across a diverse set of model architectures and sizes. This includes the Llama3 family, which is typically considered challenging for quantization, well other contemporary models such as Phi-4. For example, Llama3-8B quantized by LITTLEBIT to 0.55 BPW yields a PPL. of 18.47. This result substantially outperforms the STBLLM score of 241.95 at 0.55 BPW with 4:8 sparsity. The zero-shot

---

Table 3: Memory footprint comparison (GB) for Llama2 [47] models under different quantization methods. LITTLEBIT is evaluated at various BPWs, achieved by adjusting the latent rank r. Com- pression factors relative to FP16 are shown in parentheses. The gap between the effective BPW and the compression factor arises because quantization is applied only to the Transformer blocks, while components like the embedding layer and the Im_head remain in FP16.
Model
FP16
BILLM
OneBin [15)
LIT
BPW
16
0.8
0.55
0.3
1ama2278
13
L,0GB (843x)
GB (9.92x)
G. (11.340)
0.98 (13.77%)
0.79 GB (17.08x))
0.63 (21.41x)
Llama2-33 13B
GB.06GB
288GB (9.31x)
GB28 (11.43x)
1.95 GB (13.36x)
1.51 GB (17 (17.26x)
1,15 GB (6)666)
0.84 GB (51.02x)
Llama2 70B
GB
GB (8.96x)
9.72 (14.20x)
7.97 (17.31x)
5.83 (23.68x)
3.70 (37.31x)
GB (69.72xx
Table 4: Estimated KV cache memory reduction for Llama2-7B using LITTLEBIT, with reduc- tion factor of approximately dmodel/T.
Pip
w
X
BPW
KV Latent Rank
KV Cache
(r)
Reduction Factor
Figure 5: Conceptual view of KV Cache storage: the standard method (left) stores the full hidden dimension (dmodel), whereas LITTLEBIT (right) caches reduced latent dimension (r)
0.80
1,624
=2255x
0.55
1,112
r33.7
0.30
600
*=6.8
0.10
192
1,21,3x
evaluation results presented in Table 2 corroborate these caplexxty-baseed findings. For instance, Llama2-7B compressed by LITTLEBIT to 0.55 BPW achieves mean accuracy of 47.26%, and the model maintains 45. 20% mean accuracy even at an extreme 0.3 BPW. These results compare favorably with those of STBLLM at similar or even higher effective bits. STBLLM on Llama2-7B at 0.55 BPW scores 44.29%. This outcome suggests superior preservation of the intrinsic reasoning capabilities of the models even under extreme compression. It highlights the efficacy of LITTLEBIT in creating highly compact yet powerful LLMs.
4.3 Reasoning Performance of Extremely Compressed LLMs
The capability of LITTLEBIT to achieve sub- 1-bit compression while maintaining robust model fidelity prompts an evaluation of its reasoning abilities, particularly for large-scale models under extreme compression. We investigated the Phi-4 14B model and extended our analysis to the QwQ-32B model, comparing their performance against STBLLM across zero-shot reasoning tasks.
When compressed with LITTLEBIT to 0.55 BPW, Phi-4 achieved an average accuracy of approxi- mately 52.3%. Although this performance is marginally lower than STBLLM at a similar compression level (54.2% at 0.55 BPW), LITTLEBIT demonstrates substantially more graceful degradation. At the more aggressive 0.3 BPW setting, the LITTLEBIT-Compresee of 48.7%, which markedly outperforms STBLLM at 0.3 BPW (40.3%). Even at extreme 0.1 BPW, Phi-4 retained a notable accuracy of 43.6%. We observed a consistent trend with the QwQ-32B model, as illustrated in Figure 4. While STBLLM exhibited severe performance collapse in the sub-0.5 BPW regime, LITTLEBIT maintained a stable trajectory down to 0.1 BPW. These results highlight the ability of LITTLEBIT to preserve robust reasoning capabilities across varying model scales as compression becomes increasingly extreme.
Phi-4 maintained an average accuracy
5 Analysis
Memory Footprint Reduction LITTLEBIT is designed for resource-constraincd environments, such as on-device deployment, and achieves substantial reductions in model memory footprint. As summarized in Table 3, quantizing a Llama2-7B model (originally 13.49 GB in FP16) an effective bit rate of 0.3 BPW using LITTLEBIT reduces its required storage to 0.79 GB. This corresponds to compression factor exceeding 17x. At the extreme setting of 0.1 BPW, the footprint is further reduced to 0.63 GB (over 21 x compression). We observe comparable reductions for larger models, such as Llama2-70B, where the memory footprint decreases from 138.04 GB to under GB at 0. BPW, achieving nearly 70x compression. These substantial memory reductions significantly expand the range of devices capable of hosting large-scale language models, thereby enabling complex language tasks on hardware with limited VRAM or storage capacity.

---

0.30
11.6x
0.2111
0.25
Average Lateney (ms)
10
Speedup FP16 (x)
7.8
0.15
5.2%
3.1x
3.9x
0.10-
0.054
0.05
0.073
0.065
-2
0.037
FP16 OneBit LB(L,00)
L8V0.55)
LB(0.30) LB(0.10)
Kernel Configuration
Figure 6: Kernel-level latency (bars, left axis) and speedup relative to FP16 (dashed line, right axis) on an NVIDIA A100 GPU for a Llama2-70B MLP layer (8, 192 x 28, 672). The benchmark compares the 6P16 GEMM baseline with OneBit and the proposed LITTLEBIT (LB) kernel across various BPW configurations ranging from 1.0 to 0.1.
KV Cache Compression The proposed factorization (Eq. (4)) inherently compresses the KV cache when applied to key and value projection matrices. Because the forward computation (Eq. (5)) operates on latent states, intermediate rank r vectors are cached rather than full dmodel daode--dimensiona -imensional vectors, as illustrated in Fig. 5. Consequently, KV cache memory requirements are reduced by factor of approximately dmoddd/T /r. For example, a reduction of up to 21.3x observed for Llama2-7B at 0.1 BPW (Table 4). While this effect aligns with explicit compression methods such as MIA [66] and ASVD [42]. the approach described herein achieves this via unified factorization within attention layers, thereby reducing both weight and activation memory simultaneously.
Latency Considerations In addition to memory optimization, inference latency presents a critical performance metric. The factorized architecture (Fq. (5)) facilitates computational acceleration through low-rank binary operations, distinct from methods solcly prioritizing memory requiring specific hardware. To evaluate this capability, a custom 1-bit GEMV CUDA kernel was developed to accelerate the primary computational path. As illustrated in Fig. 6 for Llama2-70B MLP layer, the kernel demonstrates speedups that inversely correlate with bit-width, peaking at 11.6x over an optimized FP16 baseline at 0.1 BPW.
These findings suggest the proposed method enables efficient deployment with reduced latency. The discrepancy between theoretical computational gains and practical latency is likely attributable to memory access dominance during small-batch inference. The factorized approach involves multiple memory-bound operations, and the custom kernel has not yet reached the optimization level of industry-standard libraries. Appendix H provides a comprehensive analysis covering theoretical FLOPs, practical latency, and end-to-end decoding throughput.
Conclusion
This study proposed LITTLEBIT, a method designed to advance LLM compression into the sub- 0.5 BPW regime while maintaining viable performance 0.1 BPW. Performance improvements were realized through SVD-inspired latent matrix factorization, factor binarization, and a multi- scale compensation mechanism encompassing row, column, and latent dimensions. Furthermore. Dual-SVID initialization was introduced to stabilize quantization-aware training (QAT), alongside Residual Compensation for error mitigation. Empirical results demonstrate that the proposed method markedly outperforms prior sub- 1-bit techniques and preserves fidelity across LLMs scaling up to 32B parameters. Consequently, LITTLEBIT suggests a favorable size- perreperffrmacee trade-off. thereby facilitating LLM deployment in resource- conouurc-coostraieedd environments. Future research may address practical deployment through hardware co-design for edge devices, such as neural processing units (NPUs). Additional improvements in model fidelity could be examined by integrating advanced techniques such non-uniform bit allocation, applied either across model layers or within single layers via dynamic bit budget distribution between primary and residual paths.
10

---

Acknowledgements
We are grateful to Dr. Heonjae Ha, Dr. SangJeong Lee, Minseop Choi, Changdong Kim, and Hyochan Chong for their insightful discussions and generous support in this work.
References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All You Need. Advances in Neural Information Processing Systems, 30, 2017.
[2] Deepak Narayanan, Mohammad Shoeybi, Jared Casper. Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient Large-Scale Language Model Training GPU Clusters Using Megatron-LMM. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-15, 2021.
[3] Mikhail Isaev, Nie McDonald, and Richard Vuduc. Sealing Infrastructure to Support Multi Trillion Parameter LLM Training. In Architecture and System Support for Transformer Models (ASSYST@ ISCA 2023). 2023.
[4] Yue Zheng, Yuhao Chen, Bin Qian. Xiufang Shi, Yuanchao Shu, and Jiming Chen. A Review on Edge Large I. Language Models: Design, Execution, and Applications. ACM Computing Surveys, 57())1333, 2025.
[5] Xiurui Pan, Endian Li, Qiao Li, Shengwen Liang, Yizhou Shan, Ke Zhou, Yingwei Luo, Xiaolin Wang, and lie Zhang. InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Contexi LLM Inference. arXiv preprint arXiv:2409.049/) 2024.
[6] Libo Zhang, Zhaoning Zhang, Rui Li, Zhiliang Tian, Songzhu Mci, Dongsheng Li, et al. Dovetail: A CPU/GPU Heterogencous Speculative Decoding for LLM inference. In Proceedings of the 2025 Conference on Empirical Methods Natural Language Processing, pages 17393-17406 2025.
[7] Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. BiLLM: Pushing the Limit of Post-Training Quantization for LLMs. arXiv preprini arXiv:2402.0429) 042999 2024.
[8] Elias Frantar, Salch Ashkboos, Torsten Hoefter, and Dan Alistarh. GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers. arXiv preprint arXiv:2210.1733333 173323,, 2022.
[9] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang. Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
[10] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and Training Neural Networks for Efficient Integer- Arithmetic-Only Inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pages 2704-2713, 2018
[11] Jangho Kim, Yash Bhalgat, Jinwon Lee, Chirag Patel, and Nojun Kwak, QKD: Quantization-awwre Knowledge Distillation. arXiv preprint 19Xiv: 12491,,
[12] Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang. Huaijie Wang, Lingxiao Ma, Fan Yang. Ruiping Wang. Yi Wu, and Furu Wei. BitNet: Scaling 1-bit Transformers for Large Language Models. arXiv preprint arX:::2310.1 1453, 2023.
[13] Yuzhuang Xu, Xu Han, Zonghan Yang. Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. OneBit: Towards Extremely Low-bit Large Language Models, Advances in Neural Information Processing Systems, 37:66357-66338 2024.
[14] Liqun Ma, Mingjie Sun, and Zhiqiang Shen. FBI-LLM: Scaling up Fully Binarized LLMs from Scratch via Autoregressive Distillation. arXiv preprint arXiv:2407.0700 2024.
[15] Yue Zheng, Yuhao Chen, Bin Qian. Xiufang Shi, Yuanchao Shu, and Jiming Chen. A Review on Edge Large Language Models: Design, Execution, and Applications. ACM Computing Surveys, 57(8):1-35 2025.
11

---

[16] Peijie Dong, Lujun Li. Yuedong Zhong, Dayou Du, Ruibo Fan, Yuhan Chen, Zhenheng Tang, Qiang Wang. Wei Xue, Yike Guo, et al. STBLLM: Breaking the 1-bit Barrier with Structured Binary LLMs. arXiv preprint MXXx:2448.008000 2024.
[17] Ajay Jaiswal, Lu Yin, Zhenyu Zhang. Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang. From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications. arXiv preprint arXiv:2407.11239, 2024.
[18] Qinsi Wang, Jinghan Kc, Masayoshi Tomizuka, Yiran Chen, Kurt Keutzer, and Chenfeng Xu. Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives. arXiv preprint arXiv:2502.02723, 2025.
[19] Gene H Golub and Christian Reinsch. Singular Value Decomposition and Least Squares Solutions. In Linear algebra, pages 134-151. Springer, 1971.
[20] Song Han, Huizi Mao, and William Dally. Deep Compression: Compressing Deep Neural Networks with Pruning. Trained Quantization and Huffman Coding. arXiv preprint arXiv:1510.00149, 2015.
[21] Xin Wang, Samiul Alam, Zhongwei Wan, Hui Shen, and Mi Zhang. SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression. arXiv preprint arXiv:2503.12340 2025.
[22] Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A Simple and Effeettive Pruning Approach for Large Language Models. arXiv preprint 2r36v 1695, 2023.
[23] Ruihao Gong, Yifu Ding, Zining Wang, Chengtao Lv, Xingyu Zheng, Jinyang Du, Yang Yong, Shiqiao Gu, Haotong Qin, Jinyang Guo, al. A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms. Neural Networks, page 107856, 2025.
[24] Dongwon Jo, Taesu Kim, Yulhwa Kim, and Jae-Joon Kim. Mixture of Scales: Gemory-Efficieet Token- Adaptive Binarization for Large Language Models. Advances in Neural Information Processing System, 17:137474-137494 2024.
[25] Zhiteng Li, Xianglong Yan, Tianao Zhang. Haotong Qin, Dong Xie, Jiang Tian, Linghe Kong, Yulun Zhang, Xiaokang Yang, et al. ARB-LLM: Altemattnn Refined Binarizations for Large Language Models. arXiv preprint arxiv:2410.03129, 2024.
[26] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran EI- Yaniv, and Yoshua Bengio. Binarized Neural Networks. Advances in Newral Information Processing Systems, 29, 2016.
(27] Cheng Fu, Shilin Zhu, Hao Su, Ching-En Lec, and Jishen Zhao. Towards Fast and Energy-Efficient Binarized Neural Network Inference FPGA. arXiv preprint 11000288, 2018
[28] Matthieu Courbariaux, Yoshua Bengio, and Jean- Pierre David. BinaryConnect: Training Deep Neural Networks with Binary Weights during Propagations. volume 28, 2015.
[29] Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song. and Nicu Sebe. Binary Neural Networks: A Survey. Puttern Recognition, 105: 107281,, 2020.
[30] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks. In European Conference on Computer Vision, Springer, 2016.
pages 525-542.
[31] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or Propagating Gradients through Stochastic Neurons for Conditional Computation. arXiv preprint arXiv:/3088 300.:3308.333 3432, 2013.
[32] Matt Schoenbauer, Daniele Moro, Lukasz Lew, and Andrew Howard. Custom Gradient Estimators are Straight- Through Estimators in Disguise. arXiv preprint arX1v22405.0517, 2024.
[33] Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. PB-LIM: Partially Binarized Large Language Models. arXiv preprint 21X00:2310.00034, 2023.
[34] Amir Gholami, Schoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. Survey of Quantization Methods for Efficient Neural Network Inference. Low-power computer vision, pages 291-326. Chapman and Hall/CRC, 2022.
[35] Misha Denil, Babak Shakibi, Laurent Dinh, Mare' Aurelio Ranzato, and Nando De Freitas. Predicting Parameters in Deep Learning. Advances in Neural Information Processing Systems, 26, 2013.
12

---

[36] Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang. Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. LLM-QAT: Data-Free Quantization Aware Training for Large Language Models. In Findings of the Association for Computational Linguistics; ACL 2024, pages 467 667-484,27 2024.
[37] Xin Men, Mingyu Xu, Qingyu Zhang, Qianhao Yuan, Bingning Wang. Hongyu Lin, Yanjie Lu, Xianpei Han, and Weipeng Chen. ShorGPT: Layers in Large Language Models are More Redundant Than You Expect. In Findings of the Association for Computational Linguistics; ACL 2025, pages 20192-20204, 2025.
[38] Shwai He, Guoheng Sun, Zheyu Shen, and Ang Li. What Matters in Transformers? Not All Attention is Needed. arXiv preprint (tt:::24061178866 2024.
[39] Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea Goldsmith, and Mert Pilanci. Compressing Large Language Models using Low Rank and Low Precision Decomposition. Advances in Neural Information Processing Systems, 37:88981- -89018, 2024.
[40] Qingru Zhang. Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng. Weizhu Chen, and Tuo Zhao. AdaLoRA: Adaptive Budget Allocation for Parameter- Parameter-Efticient Fine-Tuning. arXiv preprint arXiv:2303. 10512, 2023.
[41] Fanxu Meng. Zhaohui Wang, and Muhan Zhang. PISSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models. Advances in Neural Information Processing Systems, 37:121038- 121072, 2024.
[42] Zhihang Yuan, Yuzhang Shang, Yuc Song, Dawei Yang, Qiang Wu, Yan Yan, and Guangyu Sun. ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models. arXiv preprint urXiv:2312.05821, 2023.
[43] Cheng Zhang, Jianyi Cheng, George A Constantinides, and Yiren Zhao. LQER: Low-Rank Quantization Error Reconstruction for LLMs. arXiv preprint arXiv:2402.02444 2024.
[44] Wenjing Ke, Zhe Li, Dong Li, Lu Tian, and Emad Barsoum. DL-QAT: Quantization- Training for Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 113-119, 2024.
Low-Rank
[45] Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD-LLM: Truncation-aware Singular Value Decomposition for I Large Language Model Compression. arXiv preprint 2430072403..77378, 2024.
[46] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302. 19771, 2023.
[47] Hugo Touvron. Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open Foundation and Fine-tuned Chat Models. arXiv preprint arXiv:2307.0928 2023.
[48] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 Herd of Models. arXiv preprint arXiv:240..2178, 2024.
[49] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open Pre-trained Transformer Language Models. arXiv preprint arXiv:2205.01 01068, 2022.
[50] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi. Piero Kauffmaan, et al. Phi-4 Technical Report. arXiv preprint arXi::44/2.08900 2024.
[51] Qwen Team. QwQ-32B: Embracing the Power of Reinforcement Learning, March 2025. URL https: /uqpg//qqq-aa
//queell.giattteeeeee github. tt/bloo////oaeeee
quq-qqqu202 32b/.
[52] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer Sentinel Mixture Models. arXiv preprint 1609.07833 2016.
[53] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu, Exploring the Limits of Transfer Learning with a Unified Text-to Text Transformer. Journal of Machine Learning Research, 21(140): 67, 2020.
13

---

[54] Mitch Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson. Karen Katz, and Britta Schasberger. The Penn Treebank: Annotating Predicate Argument Structure. In Proceedings of the Workshop on Human Language Technology. HLT '94, page 114-119, Association for Computational Linguisties, 1994.
[55] Keisuke Sakaguchi. Ronan Le Bras, Chandra Bhagavatula. and Yejin Choi. WinoGrande: Adversarial Winograd Schema Challenge at Scale. Communications of the ACM, 64(9):99-106, 2021.
[56]
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can Suit Armor Conduct Electricity? A New Dataset for Open Book Question Answering. arXiv preprint 18X.:::80.007788 2018.
[57]
Rowan Zellers, Holtzman. Youatan Bisk, Ali Farhadi, and Yejin Choi, HellaSwag: Can Machine Really Finish Your Sentence? arXiv preprini arXiv: 07830, 2019.
[58] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. arXiv preprint arXiv: 19010:9905.10444 2019.
[59] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think You Have Solved Question Answering? Try ARC, the A12 Reasoning Challenge. arXiv preprint 103.v:800.004555 2018.
[60] Yonatan Bisk, Rowan Zellers. Jianfeng Gao. Yejin Choi, et al. PIQA: Reasoning about physical common- sense in natural language. In Proceedings the pages 7432-7439, 2020.
the AAAI Conference on Artificial Intelligence, volume 34,
[61] Geoffrey Hinton. Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.0000 523313 2015.
[62] Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. TemaryBERT Distillation-awer aware Ultra- low Bit BERT. arXiv preprint arXiv:2009. 12812, 2020.
[63] Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proceedings of the Third International Conference on Learning Representations (ICLR), 2015.
[64] Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training Generalized Multi Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv preprint arXiv:2305. 13245, 2023.
[65] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang. Peng Xu. Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. OmniQuant: Onnidircctionally Calibrated Quantization for Large Language Models. arXiv preprint axXiv:2308.33111 2023.
[66] DeepSeek-AL. DeepSeek- -2: A Strong Mixture-of- ofpertss Language Model with Expert Attention. arXiv preprint Ar0:::2405.0 044444444, 2024.
[67] Noam Shazeer. Fast Transformer Decoding: One Write-Head All You Need. arXiv preprint arXiv:1911.0211 2019.
14

---

A Mathematical Analysis and Proofs
This section provides mathematical details for the LITTLEBIT computation and explores theoretical aspects related low-rank quantization errors, offering motivation for certain design choices.
Proposition (LITTLEBIT Forward Pass Computation (Proof Detail)). As stated in Section 3.1 (Proposition 1), for input E Reoeedee the primary quantized weight matrix is W pri diag U diag()(( sign diag(g). The forward pass Y = -WW can be computed as shown Eq. (12).
Y= x=((((X O Vsign) OUU sign o
(12)
Terms are defined in Section 3.1.
Proof. Starting with Y XWT.
pri
Y = X (diag(h)Usignndie () sign ding(g)) T
X diag(g) (g) V sign ( Sign diagg(hh)))
= ((((X o V o E)Usigal @ h.
(13)
This yields Eq. (12), where @ implies element-wise multiplication with broadcasting.
Claim 1 (Quantization Error vs. Factor Rank). Consider simplified model where rank-r ma-
trix ((r) (V(r) approximated. Let U(r) sign sign(U(r) ((r)
sign sign(V(r)
magnitudes are crudely approximated using fixed rank-1 SVDs of [ and |V(r)) to form scales (r)
si .sV
Wp:= (U o (6)) (v
sign o (6) 1,))
(14)
The error Err) = W(() Wr might be non-decreasing with r.
Proof Sketch / Heuristic Argument. As rank (and thus complexity of w UO), v(r) increases, the fixed-structure rank-1 magnitude approximation (via sy (r) s(()) may become relatively poorer fit for the increasingly complex actual magnitudes of U(r), (rr). This degradation in scaling accuracy could lead to (rr not decreasing, or even increasing, motivating more sophisticated scaling as in LITTLEBIT.
Claim 2 (Quantization Bias vs. SVD Component Structure). Let W's rank-r SVD approximation be
Wry W, = Wr, + Rr, (primary W r1 residual R..).. Consider a quantization operator Q. The relative quantization error [[Q(A) Al|F/|||I ||| might be larger for matrices A with "flatter" singular value spectra or less energy concentration (e-g., R vs. ). This suggests that Q applied to R might be less effective (i.e., higher relative quantization error) than when applied to W
Proof Sketch. W r. has dominant singular values, while R,, has smaller ones, often resulting in lower energy concentration for R, If Q's relative error is sensitive to this (e.g., higher for lower concentration), quantizing Wr. and R,,, separately may be better than quantizing W, jointly, as allows adapting Q to their differing structures.
Proposition 2 (Potential Advantage of Two-Stage Quantization of SVD Components). Using the setup of Claim 2, let W single Q(W,) and Wtwo-stage Q(W,,) + @(R.g). The two-stage can be less than the single-stage error Esingle- This occurs if the quantization error of the sum is greater than the sum of (vectorial) quantization errors parts:
error Eewo Etwo-stage stage
W Witwo-stage arrwo-stade
W Weingle
(((((,,)) Wr.) (Q(Rr)) - RR ([@(W,) W.
(15)
15

---

Proof Outline and Discussion. Let Etrunc- = W Wr ErTOrS are A1 = Q(Wn) W m
2 ( Rey, Asum @(W,) W,. Then Stwo-stage
Etrune (Aj + 2
and Esingle-stage
Etrune - Sum |F. The condition Eq. (15) makes Etwo-stage < Esingle-ster stage likely. Non-linear 2 (e.g., with adaptive scaling) can satisfy this by better handling the distinet characteristics of W,, r,, and Ry.. This motivates LITTLEBIT's separate residual handling, though its residual is learned differently.
B
Ablation Study
B.1 Residual Compensation
To provide further intuition behind the benefits of Residual Compensation, we visualize the output activations of selected Transformer layers under different quantization schemes. Specifically, we extract the final outputs from Transformer layers 5, 10, and 15 of an OPT-1.3B model, evaluated on two randomly sampled input sequences. For each layer and input, we display set of three activation maps corresponding to the full full-precision baseline, as well as two quantized variants wwit and without residual compensation. These groupings allow us to directly observe the structural differences induced by the presence or absence of the residual compensation.
As shown in Fig. 7, residual compensation preserves a high-fidelity resemblance to the full-precision baseline across all inspected layers. In the no-residual setting, the output feature distributions exhibit noticeable deformation relative to the full-precision baseline. In contrast, applying residual compensation consistently restores structural similarity to the original distribution, preserving both directional patterns and magnitude diversity.
Table 5 presents the PPL results for the OPT-1.3B model, comparing configurations with and without residual compensation. As indicated in the table, at the extremely low bit of 0.1 BPW, the version with residual compensation (PPI. 60.011) performed worse than the version without it (PPL 48.512). This suggests that for relatively small models (1.3B), when pushing effective bits to such extreme lows, the additional complexity introduced by the residual compensation mechanism might outweigh its benefits and instead hinder performance. However, for the same OPT-1.3B model at other effective bits evaluated (ranging from 0.3 BPW to 1.0 BPW), the inclusion of residual compensation consistently resulted in better (lower) PPL. Moreover, experiments conducted with other, larger models generally showed that residual compensation tended to provide superior performance. In light of these overall advantages observed across various settings and particularly for larger models, residual compensation was adopted as a standard component in all experiments presented in this study.
Table 5: Perplexity (WikiText-2) comparison between Residual and Non-Residun for OPT-1.3B at various BPWs. Learning rate was 8c-5, Lower PPL better.
BPW
Residual PPL
Non-Residual PPL
Difference (Residual Non-Residual)
1.00
20.329
21.691
-1.662
0.80
21.338
22.688
-1.350
0.70
22,001
23.313
-1,312
0.55
23.457
24.788
-1.331
0.30
28.984
29.724
-0.740
60.011
48.512
11.499
B.2 SmoothSign versus Thrighh Estimator
For backpropagating through the non-differentiable sign(æ) function, compared the Straight- Through Estimator (STE) [31] with our proposed SmoothSign technique. SmoothSign employs the sign(x:) function for the forward pass. For the backward pass, it utilizes smooth proxy gradient, specifically the derivative of tanh(kr) where
100 (illustrated in Fig. 8).
Table 6 (OPT-1.3B results) shows SmoothSign yielding better PPL as effective bits decrease, with a notable advantage 0.1 BPW. Due to its superior stability and performance in the ultra-low bit regime, SmoothSign was adopted for all QAT experiments (consistent with Section 4.1).
16

---

Input
Input 2
layer-0 output
layer-S output
CHAPAN
layer- 10 output
layer- 15 output
Figure 7: Visualization of Transformer layer outputs for layers 0, 5, 10, and 15 under the LITTLEBIT quantization scheme on OPT-1.3B. For each of two randomly sampled input sequences (annotated at the top), we show horizontally grouped triplets of activation maps per layer: full full-precision baseline (left), quantized with residual compensation (center), and quantized without residual compensation (right).
Forward Pass and Proxy Function Basis
Backward Pass: Gradient from Proxy
pra (ectud Forwerd
yo mach/lok0 Procc Sace!
100
1000 mumh (2080))
50
1/2 [ascumine 5
60
40
-0.
20
-1.0
-15
(a) Forward pass: sign(x) and tanh (100x)
(b) Backward pass: gradient
de tanh(100x))
Figure 8: The SmoothSign activation technique. (a) Functions relevant to the forward pass: Smooth- Sign uses the sign(x) function (actual forward pass). The plot also shows tanh (anh(100z), which serves as the basis for deriving the smooth proxy gradient shown in (b). (b) The smooth proxy gradient, d tanh(100z)). utilized for the backward pass.
de
B.3 Grouped Query Attention
Modern LLMs often Grouped Query Attention (GQA) [64] Multi-Query Attention (MQA) [67], where key (K) and value (V) projection layers have smaller output dimensions than query (Q) projections. Applying LITTLEBIT at ultra-low bits (e.g., 0.1 BPW) these GQA/MQA models can result in an extremely small latent dimension r for K/V layers (e.g.. r ze 20 for Llama3-8B K/V layers 0.1 BPW overall), potentially creating information bottleneck and hindering QAT or performance.
To investigate, we performed an ablation on Llama3-8B (~0.1 BPW target), increasing the latent rank r for K/V layers by 2x, 4x, and 8x over the standard calculation (Appendix D), while other layers remained unchanged. Table 7 shows that a 4x factor for K/V rank yielded the best PPL on WikiText-2
17

---

Table Perplexity (WikiText-2) comparison between STE and SmoothSign for OPT-1.3B at various effective bits (BPW). Learning rate was 8e-5. Lower PPL is better.
BPW
STE PPL
SmoothSign PPL
Difference (SmoothSign STE)
1.00
20.322
20.329
+0,007
0.80
21.344
21.338
-0.006
0.70
22.100
22.001
-0,099
0.55
23.528
23.457
-0.071
0.30
29.058
28.984
-0.074
0.10
60.401
60.011
-0.390
Table 7: Ablation study adjusting the latent dimension factor (r) for key (K) and value (V) projection layers in Llama3-8B with GQA, targeting an overall effective bit of 0.1 BPW. Performance measured by perplexity (PPL, lower is better) on WikiText-2 and C4 validation sets, and average zero-shot accuracy (%) on common sense reasoning tasks (higher is better). The 'KV Factor' indicates the multiplier applied to the K/V layers' latent rank compared to the standard calculation for 0.1
BPW.
KV Factor
Approx. BPW
WikiTexx-2 PPL
C4 PPL
Avg. Zero-shot Acc (%)
1x (Baseline)
00098
25.80
34.04
44.85
2x
0.101
25.22
33.20
44.92
4x
00107
24.93
32.69
44.70
8x
0.119
25.31
33.21
44.87
and C4, with minimal impact on overall BPW (e.g., ~0.098 BPW to ~0.107 BPW for 4x). Average zero-shot accuracy also remained competitive. An 8x factor showed diminishing returns.
Given the significant PPL improvement with the 4x K/V rank factor (nearly point on WikiText-2 vs. 1x) at negligible BPW cost (<10% relative increase in effective bits for transformer layers, <4% of total parameters), this strategy was adopted for GQA/MQA models (Llama3-8B, Phi-4, QwQ-32B) in our main results. For these models, the latent rank for K and V projection layers was calculated based on a 4x multiplier compared to the standard calculation for the target BPW, while all other layers used the standard rank calculation. This approach helps mitigate potential information bottlenecks in GQA/MQA layers under extreme quantization without significantly altering the target compression level.
C Perplexity on C4 and PTB
To further assess LITTLEBIT's generalization, we evaluated its performance on additional bench- marks: the diverse C4 web text corpus and the distinct PTB news corpus (Table 8). Focusing on Llama2-7B/133 against STBLLM, the results on these datasets confirm the trends observed on WikiText-2. LITTLEBIT consistently achieves lower perplexity than STBLLM at comparable sub-1-bit regimes. This performance advantage becomes more significant below 0.55 BPW, where STBLLM's performance degrades considerably, particularly PTB.
Notably, LITTLEBIT maintains robust performance and stability even at the extreme 0.1 BPW level across all tested datasets. This consistent ability to maintain strong language modeling capabilities under severe compression highlights the generalizability of our approach, which integrates latent factorization, multi-scale compensation, and QAT. It reinforces LITTLEBIT's potential for effectively deploying capable LLMs in resource-limiter environments.
D Average Bits Per Weight Calculation
This section details the calculation of the average effective bits per weight (which we denote in the following equations) for linear layer implemented using the LITTLEBIT architecture, including the parameters from the primary (W pri) and residual compensation (Wres) pathways, described
18

---

Table 8: Perplexity (PPL) comparison on C4 and PTB validation sets for Llama2 models. Lower PPL
indicates better performance. STBLLM [16] utilizes N:M sparsity (ratio in parentheses).
Model
Method
BPW
C4
PTB
FullPrecision
16
6.97
37.91
STBDL (6.8)
0.80
15.42
2.4e3
STBLLM (4:8)
0.55
30.99
527.56
STBLLM (2:8)
0.30
808.98
2.3e4
Ilama2-7B
LITTLEBiT
0.80
T1.77
40.75
LITTLEBIT
0.55
12.94
50.53
LITTLEBIT
0,30
14.78
59.17
LITTLEBIT
0.10
19.73
83.09
FullPrecision
16
6,47
50.94
STBLLM (6:8)
0.80
12.96
165.51
STBLLM (4:8)
0.55
27.38
424.82
STBLLM (2:8)
0.30
442.49
1.7e3
Alama2-13B
LITTLEBiT
0.80
10.60
44.87
LITTLEBIT
0.55
11.53
53.52
LITTLEBIT
0.30
13.07
51.66
LITTLEBIT 0.10
18.88
87.59
in Sections 3.1 and 3.3. The calculation demonstrates how to determine the latent rank r required to achieve target value for b within the range of approximately 0.1 to 1.0.
LITTLEBIT linear layer replaces the original weight matrix W E Rod din with two parallel structures, each comprising:
Two binary sign matrices: Usign E (ttlddout xr and Vsign E {=1]dmxr, These require 1 bit per element.
Three FP16 scaling vectors: h E Rdout, g E Rd, and l RT. These require 16 bits per element.
Since both the primary and residual paths have this structure, the total number of bits required to store the parameters for LITTLEBIT layer is:
Total Bits 2 x (Bits for Usign + Bits for Vsign + Bits for h + Bits for + Bits for E)
x ((dout X r x 1) + (din x r x 1) + (dout x 16) + (dn x 16) + (r 16)) 2r(oou dun) + 32 + dua) + 32r
Note: Residual Compensation is used, the initial factor of should be removed from the calculation.
The average bits per weight, denoted as b, is calculated by dividing the total bits by the number of parameters in the original FP16 weight matrix (dout x din):
2r(dou + din) + 32 + dmn) - 32r
(16)
dout X dm
To achieve a target value for b, we rearrange Eq. (16) to solve for the required latent rank r:
(b x dout x drn) 32(dout + din)
(17)
2(dout + d 32
Since must be an integer, we typically round the calculated value to the nearest suitable integer. This chosen integer r then determines the actual value of achieved, which will be very close the target value.
19

---

Example 1: Linear Layer (4,096 x 4,096) Let dout = 4,096 and dm 4,096. The original number of parameters is 4,096 4,096 16,777,216. dout + din- 8,192. For a target Rs 0.55: Using Eq. (17) with 6 = 0.35:
(0.55 x 16,777,2100 -3(18,19]
2 546
(8,, -32
We choose integer r 546. The actual value of b for 546, using Eq. (16), is:
b (8,11)))))12) + 32(8,192) + 32(546) = 0.5498
16,777,216
Example 2: Linear Layer (4,096 x 11,008) Let dout = 4,096 and dm = 11,008. The original number of parameters is 4,096 x 11,008 45,090,816. dout + dn 15,104. For a target z 0.1: Using Eq. (17) with 0.1:
(0.1 x 45,090, 816) - 32(11,104) 2 133
2(15,104) 32
We choose integer
133. The actual value of b for 133, using Eq. (16), is:
= (1133)))),104) + (2,1 + 32(1333 ~ 0.0999
45,090,816
By following this procedure for each linear layer in the model, to achieve a desired overall value for (average bits per weight), thereby controlling the trade-off between model compression and performance. The specific ranks used achieve the average bits per weight (BPW) figures reported in the main paper (e.g., in Tables and 3) are determined using this calculation methodology for for each layer type within the respective models.
select appropriate ranks (r)
E
Pruning versus SVD
To select base compression method to achieve sub-0.5 BPW, compared pruming (Wanda [22]) with SVD-based compression (SVD-LLMv2 [21]) on Llama-7B (aable 9). SVD showed superior robustness at extreme compression ratios (e.g., 25% parameter retention).
Consequently, low-rank factorization was chosen to design LITTLEBIT due to its performance resilience and the deployment advantages of dense factorized matrices over sparse pruned models (which may require specialized hardware or overhead). Although SVD-LLMv2 itself performs well, we found that initializing LITTLEBIT's Dual-SVID with vanilla SVD was sufficient. The subsequent QAT process effectively recovered performance, rendering the added complexity of SVD-LLMv2 for initialization unnecessary. Accordingly, we employ the simpler vanilla SVD within our Dual-SVID initialization procedure.
Table 9: Perplexity (WikiTexx-2) comparison between Pruning (Wanda) and SVD-based compression (SVD-LLMv2) on Llama-7B different parameter retention ratios. Lower perplexity is better. FP16 PPL is 5.68.
Remaining Parameter Ratio
Pruning [22]
SVD [21]
50%
8.7
13.6
37.5%
46.0
20.1
25%
1842.7
52.6
F Analysis of Generated Samples
The below generated samples illustrate qualitative differences as the BPW of the LITTLEBIT model decreases. Two main observations are:
1. Impact on Specificity. Factual Detail, and Coherence: As the effective bits decrease from 0.8 BPW towards 0. BPW, the generated text exhibits significant changes detail, factual accu- racy, and coherence. At 0.8 BPW, the Mona Lisa sample, while attempting to provide details
20

---

(e.g., "complex facial features"), becomes verbose and repetitive, and anachronistically refers to Da Vinci and the Mona Lisa the context of the "20th century." The Turing sample at this effective bit, however, offers a standard, factually sound definition. As the effective bit drops to 0.55 BPW, the Mona Lisa sample loses coherence significantly, expressing uncertainty about the painting's s location and bizarrely shifting to a help-seeking persona ("Could you help me find the Mona Lisa painting..." "). The Turing sample 0.55 BPW begins to broaden its definition, losing some precision but remaining somewhat related.
2. Degradation in Factual Accuracy and Relevance at Extreme Compression: Extreme compres- sion, especially at 0.1 BPW, leads to severe degradation in factual accuracy and relevance, as vividly demonstrated by both samples. The Mona Lisa sample at 0.1 BPW starts with generally acceptable statement (e.g.. "iconic masterpiece") but quickly devolves into hallu- cination, referring to the Mona Lisa as a "statue" and associating it with "Milan's fashion world." Similarly, the Turing sample 0.1 BPW deviates drastically from the core definition of computer science, incorrectly defining it as "the study of the theory and methods of how the human mind and the brain work" and "the science of mind." " These examples underscore that while LITTLEBIT might maintain superficial grammatical structure very low bits, the ability to recall and generate accurate, relevant factual information is compromised, often leading the generation of plausible-sounding but nonsensical statements.
Sample 1: Mona Lisa
Prompt: Leonardo da Vinci painted the Mona Lisa. The Mona Lisa is
BPW Generated Output (Phi-4 Model)
0.80
Leonardo da Vinci painted the Mona Lisa. The Mona Lisa is famous for complex facial features and the artist's depiction of her complex expression. It is considered one of the most important paintings of the 20th century. The Mona Lisa is a painting by the Italian artist Leonardo da Vinci, who was a painter and sculptor. He is best known for his realistic and expressive style of painting, which often compared to that of the natural world around him. Leonardo da Vinci is considered century. He famous for his complex facial features and the artist's depiction of her complex expression. The Mona Lisa is
of the most important painters of the 20th
0.55 Leonardo da Vinci painted the Mona Lisa. The Mona Lisa is in the Louvre Museum in Paris, France. I am not sure if it is in the museum or not. Could you help me find the Mona Lisa painting. am searching it everywhere. It is hard to find a picture of it. am looking for a Mona Lisa painting with Leonardo da Vinci and the Mona Lisa by the side. Thanks in advance!
0.10 Leonardo da Vinci painted the Mona Lisa. The Mona Lisa is an iconic masterpiece of Italian art, and it is one of the most photographed and photographed works of art in the world. Its execution and interpretation are breathtaking. This statue is also one of the most popular works in Milan's fashion world, and is one of the most photographed of the entire collection of the Mona Lisa.
Sample 2: Turing
Prompt: Turing is called the father of computer science. Computer science is
21

---

BPW Generated Output (Phi-4 Model)
0.80
Turing called the father of computer science. Computer science is a branch mathematics that studies the design and analysis of computational systems. It is the study of the methods designing and implementing computational systems and the application of mathematics to these systems. The most basic concepts of computer science the concepts of computation, algorithms, data structures, and programming.
0.55 Turing is called the father of computer science. Computer science a branch of engineering that focuses the design, implementation, and maintenance of systems and technologies using computers. This includes designing, implementing. and maintaining the algorithms that underpin the software, hardware, and communication systems that are used in information processing, and the Internet. Computer science is multidisciplinary field that draws on engineering, mathematics, and business. Computer science often divided into four broad areas
0.10
Turing is called the father of computer science. Computer science is the study of the theory and methods of how the human mind and the brain work. is also called the science of mind. This is very complex field, and many people are just not Some people think that computer science is the most important and most vanace field in the world, and that it is the most important branch of science in the world. This is just a myth. But there a reason for this.
what to think.
G Hyperparameters
Across all QAT experiments, models were trained for 5 epochs using the Adam optimizer with cosine decay schedule. The training corpus mixture of C4 and WikiText-2 - contains approximately I billion tokens when the text is tokenized with the Llama-2-7B tokenizer (~0.2 billion tokens per epoch). The raw number of training samples (i.e., the amount of original text) is identical for every model. As detailed in Table 12, for each model and BPW configuration, swept the learning rate within the range of 4.0c-5 to 2.4c-4. We then selected the learning rate that maintained numerical stability while minimizing validation perplexity. For example. OPT-1.3B used learning rate of 8.0e-5 at 1.0 BPW and 2.0c-4 at 0.1 BPW, while larger BPW models generally adopted slightly lower values for stability. Training was conducted using four H100 GPUs for all models except QwQ-32B, which required 4 x 8 A A100 GPUs. In this GPU configuration notation, the first number signifies the number of nodes, and the second indicates the number of GPUs per node; thus, 8 represents total of 32 GPUs.
Table 12: Knowledge Distillation Training Details
Training Sctup
OPT
Llama
Llama2
Llama3
Phi-4
QwQ
BPW
Target
1.3B
7B
13B
13B
8B
14.7B
32B
Learning Rate
8.0e-5
4.0e-5
4.0e-5
4.0e-5
4.0e-5
4.0e-5
4.0e-5
4.0e-5
1.00
GPUs
x
x
1
1 4
x
x
Learning Rate
1.2e-4 1 x 4
4.0e-5
4.0e-5
4.0e-5
4.0e-5
4.0e-5
4.0e-5
1.0e-4
0.80
# GPUs
x
xx4
1 4
1 4
x 4
1 1 4
4 x
Learning Rate
1.2c-4 1 x
8.0e-5 x
4.0e-5 x
4.0c-5
4.0c-5 1 x 4
4.0e-5
4.0e-5
4.0c-5
0.70
# GPUs
1 1 4
1x44
1 x 4
4 x 8
Learning Rate
1.2c-4
8.0e-5 1 x
8.0e-5
4.0e-5
4.0e-5
8.0e-5
8.0e-5
1.0c-4
0.55
# GPUs
1 x 4
x
1x 4
1 x 4
xx44
1 [x 4
4 x
Learning Rate
2.0e-4
1.2e-4
8.0e-5
8.0e-5
8.0e-5
1.2e-4
8.0e-5
1.0e-4
0.30
# GPUs
1 1x44
x
x 4
1 x
1 x 4
x
1 x 4
4 x
Learning
2.0e-4
1.2e-4
1.2e-4
1.2e-4
4.0e-5
1.2e-4
1.2e-4
1.6e-4
0.10
GPUs
1 1x44
x
x 4
1 4
1 1 4
X
1 x 4
x
22

---

H Inference Efficiency Analysis
This section provides a detailed account of LITTLEBIT's inference efficiency, complementing the kernel-level latency results presented in the main text. We analyze the theoretical computational cost, present detailed kernel latency benchmarks, and report end-to-end decoding throughput.
H.1 Theoretical Cost Analysis
To complement the empirical latency results, this section provides a theoretical analysis of LIT- TLEBITT' computational cost. The core principle behind LITTLEBIT's efficiency is the replacement (MAC) operations with a smaller volume of
of large number of expensive FP16 multiply-accumal cheaper FP16 additions and highly efficient bitwise operations (BOPs). To illustrate this, we analyze the operational cost for single forward pass through Llama2-7B MLP layer, where din = 11,008 latent rank of 431.
and dout. 4,096, quantized 0.3 BPW, which corresponds
standard FP16 matrix-vector multiplication for this layer involves dm x dout MAC operations. Counting each MAC as two floating Noating-ppintt operations (1 multiplication, addition), the total com- putational cost for the FP16 baseline x du x dout TLEBIT's computation is divided into floating-pont point and bitwise components. The FLOPs arise primarily from FP16 additions during accumulation and scaling multiplications, totaling approxi- mately 2 (dan + dout) ) binary weights (+1),, which are executed as sign-bit XOR operations, amounting to approximately 2 2xrx (din dous) 13.0 million BOPs. This analysis reveals a nearly 7x reduction in expensive FLOPs, which are replaced by an equivalent number of BOPs. As bitwise operations are substantially faster than floating-point operations modern hardware, this theoretical breakdown provides strong justification for the empirical latency improvements detailed below.
90.2 million FLOPs. In contrast, Lrt-
13.0 million FLOPs. The BOPs stem from multiplications with the
H.2 Kernel-Level Latency
Custom Kernel Implementation To empirically assess the latency of LITTLEBIT's factorized linear layers (Eq. (5)), we developed a custom CUDA kernel. The kernel implements the two- stage computation involving the binary matrices Vaign and U sign. Key implementation details for performance include:
Bit-level Parallelism: Binary weights (#1) are packed into uint32_t t data types. Multipli- cation by these weights is efficiently realized by directly manipulating the sign bit of the FP16 imput values via bitwise XOR operations, avoiding costly floating-point multiplications.
Warp-Lever Reductions: To accelerate the accumulation step, the kernel employs warp- level reduction primitives (varpReduceSum) This technique efficiently sums partial results within a CUDA warp (a group of 32 threads), significantly reducing memory traffic and latency compared to naive reduction methods.
FP16 Arithmetic: The scaling factors (g, h) are maintained in FP16 precision, and their application leverages native half-precision Arithmetic Logic Units (ALUs).
All latency benchmarks were performed on an NVIDIA A100 GPU with a batch size of 1. The primary baseline for comparison is standard FP16 GEMM operation as implemented by torch, matmul, which leverages highly optimized libraries like CUBLAS.
Latency Results and Discussion Table 13 presents the detailed latency comparison. The results demonstrate that LITTLEBIT, accelerated by our custom kernel, can offer significant inference acceleration. For instance, in a Llama2-70B ML.P-like layer, LITTLEBBIT achieves up 11.6x speedup relative to the FP16 baseline at an effective BPW of 0. While our custom kernel is proof-of-concept and not as exhaustively optimized as mature libraries like CuBLAS, these findings are promising. They show a clear path to substantial latency reduction at ultra-low bits, confirming that LITTLEBIT's architecture is well-suited for gegh-performacc deployment.
H.3 End-to-End Decoding Throughput
To assess real-world application performance, we benchmarked the end-to-end decoding speed of Llama2-7B model, measured in tokens per second (TPS). It is important note that these throughput
23

---

Table 13: Kernel-level latency (ms) on an NVIDIA A GPU (Batch Size 1). Compares PyTorch
FP16 GEMM with the LITTLEBIT (LB) kernel and baseline 1-bit (OB) kernel. Dimensions (N, M. R) denote output, input, and latent features, respectively.
Layer Type (Model Ref.)
Dimensions (N, R)
Method
BPW
Latency (ms)
Relative Speedup
FP16 Baseline
16.0
0,2882
1.0x
OneBit
1.0
0.0713
4,04x
LITILEBIT
1.0 (r 6,400)
0.0938
3.07
LITTLEBIT
0.8 (7 5,120)
0.0734
3,93x
Lama22-UBB 7UB MI.P-lke
(8,192, 28,672,
LITTLEBIT
0.7 4,480)
0,0648
4.45x
LITTLEBIT
0.55 (r 3.520)
0.0555
5.19%
LITTLEBIT
0.3 (r 1,320)
0.0372
7.75x
LITILEBIT
0.1 (r 640)
0.0249
11.57%
FP16 Baseline
16.0
0.0896
1.00x
OneBit
1.0
0.0285
3.14x
LITTLEBiT
10 4,096)
0.0440
2,04X
Llama2-70B ATTN-like
(8, 192, 8,192,
LITTLEBIT
0.8 (r 3,296)
0.0340
2,64x
0.7 2,880)
0.0330
2.72x
LITTLEBIT
0,55 (r 2.272)
0.03002
2.97x
LITTLEBIT
0.3 (r 1,248)
0.0238
3.76x
LITTLEBIT
0.1 (r
0.0207
4.33x
FP16 Baseline
16.0
0.0620
1,00
OneBit
1.0
0.0226
2,74x
LITTLEBIT
(r 3,008)
0.0238
2.61
Llama2-7B ML.P-ike
(4,096, 11,008, v)
LITTLEBIT
0.8 (r 2,400)
0.0220
2.82x
LITTLEBIT
0.7 (r 2,112)
0.0211
2.4x
0.55
1.664)
0.0192
3.23x
LITTLEBeT
0.3 (r 896)
0.0192
3.23x
LITTLEBIT
0.1 (r === 320)
0.0190
3.26X
gains are achieved by accelerating only the nn. Linear layers with our custom kernel; other modules, such as attention and layer normalization, remained their original FP16 implementations. Despite this partial acceleration, the results shown in Table 14 demonstrate substantial impact overall performance. When generating 128 new tokens, the 0.1 BPW model achieves 203.20 TPS, a 2.46 speedup over the FP16 baseline. This confirms that the significant kemel-level efficiencies of LITTLEBIT translate into tangible end-to-end acceleration, even when other parts the model leverage standard implementations.
Table 14: End-to-end decoding throughput (tokens/sec) for Llama2-7B on an NVIDIA A100 GPU. The benchmark was run 10 times and averaged.
128 New Tokens
256 New Tokens
Method
BPW
Avg. (tok/s)
Speedup
Avg. TPS (tok/s)
Specdup
Baseline
1,00x
78.16
1.00x
1.83x
139,70
LITTLEBIT
0.8
160.43
1.94x
147.42
1.89
LITTLEBIT
0.55
174.24
2.11x
160.67
2.06x
LITILEBIT
190.43
2.31x
174.82
2.24x
LITTLEBIT
0.1
203.20
2.46%
185,39
2.37
I Limitation
Despite the promising results of LITTLEBITT in achieving extreme compression, several limitations warrant discussion. Firstly, the current LITTLEBIT method primarily focuses on compressing the parameters within the Transformer blocks. The language model head (Im_ (im_head), which typically consists of a linear layer projecting to the vocabulary size, is not subjected to the same aggressive factorization and binarization. At ultra-low bits, such 0.1 BPW for the Transformer blocks, the Im. head can become a significant bottleneck in terms of overall model size, especially for models with large vocabularies. Thus, developing specialized compression techniques for the _ead that are compatible with LITTLEBIT' low-rank and binary principles is an important avenue for future work to fully realize the potential of extreme model quantization.
Secondly, while quantization-awas training (QAT) crucial for maintaining performance such low bits, it is computationally intensive and can be challenging to scale to extremely large models (e.g..

---

70B parameters and beyond). Our own experiments faced resource constraints when attempting QAT for models of this magnitude. Exploring more resource-efficient QAT strategies investigating post- training quantization (PTQ) approaches that can effectively adapt LITTLEBIT'S factorized structure to these massive models would enhance the practical applicability of our method in real-world scenarios with limited computational budgets.
Finally, the remarkable ability of models quantized with LITTLEBIT to perform complex tasks even when retaining only a tiny fraction of the original weight information (e.g., at 0. BPW, which can correspond to less than 1% of the original parameters' information) calls for deeper investigation. While our empirical results demonstrate efficacy, more fundamental understanding of how such aggressively compressed models retain their capabilities, perhaps through the lens of information theory or by analyzing changes in learned representations, would be valuable. This could lead to even more effective extreme compression techniques in the future.
Societal Impact
Our work on LITTLEBIT enhances the accessibility of large language models (LLMs) by reduc- ing their computational and memory costs, which can foster positive societal impacts in research, on-device applications. However, we acknowledge that easier
education, and deployment of capable LLMs may also inadvertently lower barriers potential misuse, such as the spread of disinformation or the amplification of societal biases. Our research responsibly utilizes existing, often publicly available models in accordance with their licenses and ethical guidelines. We strongly advocate for ethical development and deployment of any models compressed using LITTLEBIT, emphasizing the critical need for safeguards such as content filtering, bias mitigation techniques, transparency regarding Al-generated content, and the continuous development of robust detection mechanisms.
25

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] pprovided proper justificaiion is given (e.g., "error bars are not reported because would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract and Section clearly state the contributions of LITTLEBIT, including the novel factorization architecture, Dual-SVID initialization, and Residual Com- pensation. Experimental results supporting these claims are provided Section and Section 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
26

---

Answer: [Yes]
Justification: We explicitly discuss limitations, such as the lack of compression for the language model head and the computational cost of QAT, in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Mathematical details for the LITTLEBIT forward pass and proofs/claims regarding quantization errors are provided in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
27

---

We provide full details on model architectures, training settings, and hyperpa- rameters in Section 4.1 and Appendix G (including Table 12). Specifics on initialization, average bit calculation, and GQA adjustments are in Section 3.2, Appendix D, and Appendix B.3.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our code provided as supplemental material and will be publicly released. We use open datasets (WikiText-2, C4) and standard open-source models (Llama, OPT, etc.).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
public/ cablic/guides//
ccy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: // //nips. httppp/wwwwpggiieee//eeeeeiiaaaa
for more details.
28

---

The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
reproducible, they
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Comprehensive training details, including optimizer settings, learning rate schedules, and GPU configurations, are provided in Section 4.1 and Appendix G.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the high computational cost of performing QAT on large language models (up 32B parameters), we report results from single runs, consistent with standard practices in LLM quantization research (see Table 1).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
29

---

For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We specify the hardware used (NVIDIA A100/H100 GPUs) and the number of GPUs per experiment in Section 4.1, Section 5, and Table 12.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httppp::///orrps...cuai/e//////e hc/ppbbiic/thhicssudedel.....eel
Answer: [Yes]
Justification: Our work involves compressing public open-source models and uses standard datasets. It does not involve human subjects or sensitive personal data.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the benefits of efficient LLM deployment and the potential risks of easier access to powerful models in Appendix J.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
30

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: address responsible use and safeguards our Societal Impact discussion in Appendix J. We do not release new base models but compress existing ones.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual- use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all original models (Llama, OPT, etc.) and datasets (WikiText-2, C4) in Section and throughout the text. We respect the licenses these open-source assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the werrswithhcdee package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code for LITTLEBIT (our primary new asset) is documented and included in the supplemental material.
Guidelines:
31

---

The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This research did not involve human subjects or crowdsoureing.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This research did involve human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs are the subject of our study (we are compressing them), but we did not use LLMs to generate the scientific content or methodology of this paper.
Guidelines:
32

---

The answer NA means that the core method development in this research does not involve LLMs as any important. original, non non-standard components.
Please refer to our LLM policy (https : /ttp//eeurppppppeeee / /LLM) for what should or should not be described.
33


Reviews:
[{"review_id": "MVRJICMjKi", "reviewer": "Reviewer_8rfk", "summary": "The paper introduces LittleBit, a novel quantization method for compressing LLMs to ultra-low bit widths, achieving as low as 0.1 bits per weight. By combining low-rank matrix factorization with binary representations and a multi-scale compensation mechanism, LittleBit significantly reduces memory and computational costs without severely sacrificing model performance. Key technical contributions include: 1. The Dual-SVID initialization for stable quantization-aware training (QAT) and 2. Residual Compensation to further mitigate binarization errors. Experimental results show that LittleBit outperforms prior state-of-the-art methods across a range of LLMs.", "questions": "Please refer to the Weakness section.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. This paper pushes the boundary of model quantization to ultra-low 0.1bit per weight, offering a practical path for deploying LLMs on resource-constrained devices.\n2. The integration of low-rank factorization with binary quantization and the multi-scale compensation mechanism, particularly the latent dimension scaling, is novel.\n\nWeaknesses:\n1. Performance at <1 BPW, while notable, still shows a steep degradation, making them unlikely to be deployed in real-world applications.\n2. The use of both a primary and residual factorized path (each with learnable scales and binary matrices) increases architectural complexity and parameter count.\n3. The decrease of average latency is far below optimal. In the ideal case, 1 BPW can offers 16x speedup. However, in the paper, the speedup can only be demonstrated when BPW is lower than 1.", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the thoughtful feedback. We are particularly grateful for your positive assessment of our success in pushing quantization to the ultra-low 0.1 BPW regime and the novelty of our multi-scale compensation mechanism.\n\nWe would like to address the weaknesses you have pointed out:\n\n1.  Regarding Performance Degradation:\n\n    We agree that a performance gap exists compared to full-precision models, a trade-off inherent to such extreme compression. However, we wish to respectfully highlight that LittleBit's primary contribution is not to match full-precision performance, but to maintain unprecedented robustness in the sub-0.5 BPW regime, where prior methods like STBLLM fail catastrophically (as shown in Table 1). Our work establishes a new, viable size-performance trade-off, enabling powerful LLMs to operate in previously inaccessible, highly resource-constrained environments. We believe this is a significant step forward.\n\n2.  Regarding Architectural Complexity and Parameter Count:\n\n    Thank you for this insightful comment. The dual-path architecture was a deliberate design choice to mitigate the substantial information loss from extreme binarization. Crucially, this approach does not increase the model's memory footprint for a given bits-per-weight (BPW) target.\n\n    -   Fixed Bit Budget: The model size is constrained by a fixed bit budget defined by the target BPW. When we introduce the residual path, we do not add new parameters on top of a single-path model. Instead, we re-allocate the existing bit budget across both the primary and residual paths. To maintain the target BPW, the total latent rank is distributed between the two paths, ensuring the number of binarized parameters remains constant. For instance, instead of one path with rank $r$, we use two paths with ranks $r_{pri}$ and $r_{res}$ such that the total storage cost is equivalent. This allows us to dedicate capacity to model the residual error without increasing the final model size.\n    -   Necessity for Performance: Our ablation studies in Appendix A.2 (Table 5) demonstrate that this Residual Compensation path is vital for achieving the reported performance. The results show that allocating the bit budget across two paths is more effective than concentrating it in a single, higher-rank path. We will clarify this important design rationale in Section 3.3 of the revised manuscript.\n\n3.  Regarding Latency and Speedup:\n\n    The gap between the theoretical speedup and our measured speedup stems from the critical difference between compute-bound theory and real-world inference latency.\n    -   Theoretical Computational Gains: A theoretical analysis based on floating-point operations (FLOPs) indeed shows a massive reduction. For instance, for a Llama2-7B MLP layer at 1.0 BPW, our method reduces FLOPs from $\\approx 90.2$ million to $\\approx 44.6$ million ($\\approx 2 \\times$ reduction), replacing the remaining work with highly efficient bitwise operations (BOPs). This confirms the fundamental computational efficiency of our architecture.\n    -   Practical Latency Bottlenecks: However, practical latency at small batch sizes is often dominated by memory access, not raw computation. Our factorized method involves multiple memory-bound steps. This overhead, combined with the fact that our proof-of-concept CUDA kernel is not as exhaustively optimized as the industry-standard cuBLAS FP16 baseline, explains why the practical speedup doesn't reach the theoretical maximum.\n\n    Given these real-world constraints, we believe achieving up to a $5.5 \\times$ speedup against a highly-optimized baseline is a very promising result that validates the potential for substantial, practical acceleration. We will add a detailed discussion clarifying the distinction between theoretical FLOPs and practical, memory-bound latency in Section 5 and the appendix.\n\nThank you once again for your constructive feedback, which will help us significantly improve the clarity and impact of our paper."}, {"review_id": "wQuSQvd8BX", "reviewer": "Reviewer_dapu", "summary": "The authors introduce a novel method for compressing LLMs in the ultra-low-bit (as low as 0.1 bits per parameter) regime.\nThis is the first method to achieve stable performance at such a high compression ratio.\nAt a high level, compression is achieved via a binarized low-rank decomposition (with row-wise, column-wise, and latent FP16 scales), followed by quantization-aware training under a knowledge distillation framework.\nAccuracy of this method relies on some specialized algorithmic decisions: initialization of the scaling factors based on a rank-one approximation of the SVD factor magnitudes, and residual compensation (decomposing the weight matrix into two separate additive components).\nAblation studies and some theoretical intuition are provided to justify these design choices.\n\nEvaluation is performed on a large range of LLMs, comparing perplexity and task accuracy to a structured sparse baseline (in the $<1$ bit/parameter regime) and state-of-the-art one-bit quantization methods. \nAlthough LittleBit does not beat the baselines at the $\\\\approx1$ bit/parameter regime, it is the only method to achieve good accuracy and stability at under $\\\\approx0.5$ bits/parameter.\nThe authors develop custom CUDA kernels for compressed matrix multiplication, achieving an average $5.5\\\\times$ speedup over FP16 matrix multiplication at 0.1 bits/parameter.", "questions": "(1) Is there additional theoretical justification that can be provided for the two lemmas in Appendix A, or numerical examples that can be shown to strengthen the intuition provided?\n\n(2) Can latency and bandwidth results be provided for an end-to-end setting (such as autoregressive generation), as opposed to just on the level of a single matrix multiplication?\n\n(3) Have potential algorithmic features like a calibration-data-aware initialization for the low-rank factors and non-uniform bit allocation been considered?\nDo you think they would make a decent impact on the accuracy of the method, or would they just unnecessarily increase the computational cost?\n\n(4) Do the authors have ideas to improve performance at the 1-bit regime (note: this is a minor question, as I understand that the point of the work is the $<$ 1-bit regime)?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Quality**\n\n_Strengths_.\nThe work is technically sound, with careful attention placed to almost every component of the algorithm in order to compress LLMs to the very aggressive $0.1$ bit/parameter regime with reasonable degradation.\nMany strategies are introduced in all parts of the compression pipeline (e.g., DualSVID initialization, residual quantization, SmoothSign, larger rank for grouped query attention matrices), and all are justified via ablation studies.\nIn addition, the benefit of residual quantization, which is perhaps surprising, is justified via theoretical intuition.\n\nComprehensive evaluation is performed: perplexity on three common datasets, zero-shot task accuracy, latency (with custom CUDA kernels for compressed matrix multiplication), memory footprint reduction, and estimated KV cache footprint reduction.\nResults are provided over a variety of LLMs.\nAt ultra-low bit rates of $<0.5$ bits/per parameter, the proposed method maintains reasonable accuracy and is very stable up to $0.1$ bits per parameter, whereas baselines experience significant degradation in that regime.\n\nThe discussion of limitations is nuanced and broad, touching on output degradation due to lossy compression, computational complexity of the QAT component, limitations in compression ratio, and limitations in theoretical understanding.\n\n_Weaknesses_.\nThe two lemmas (and, by extension, the theorem) in the appendix are substantiated by theoretical intuition rather than rigorous proofs.\nThough the intuition is sound, I believe it would be more accurate to reframe the lemmas as \"claims.\"\nThey could also be strengthened via concrete examples where they hold (e.g., finding some neural network weight matrix where the conditions of the lemmas hold, or even synthetic data).\n\nAlso, the accuracy could possibly be improved by non-uniform bit allocation between layers, or an initialization of the low-rank factors that takes into account calibration data (which should have a closed-form solution via results from [Xiang et al, \"Optimal exact least squares rank minimization\"]).\n\n**Clarity**\n\n_Strengths_.\nThe paper is well-organized and well-written, and its primary contributions, as well as their motivation, are quite clear.\nAll algorithmic and experimental details appear to be provided.\nThe discussion of the results is accurate and highlights the benefits of the proposed method.\n\nAlso, though the SmoothSign approach to quantization-aware training seems to be novel, this is not clear from reading the paper.\n\n_Weaknesses_.\nAt the introduction of residual compensation, the motivation was a little unclear; it's not immediately obvious that increasing the primary rank is an issue that would hurt accuracy.\nAs this is an important contribution, I would recommend refining the discussion introducing residual compensation to summarize the key insights from Appendix A. \n\nAlso, a minor note: Figure 1 chooses BiLLM for the one-bit method instead of the better performing OneBit or BinaryMoS baselines.\n\n**Significance**\n\n_Strengths_.\nThis paper pushes the bounds of ultra-low-bit compression of LLMs to 0.1 bits/parameter, and is the first work to do so.\nThis could encourage further work in this particularly-challenging compression regime.\nThe authors introduce several algorithmic nuances, any of which would be quite useful in future LLM compression algorithms. \n\nIn addition, the the empirical results and theoretical intuition on the benefit of residual compensation could encourage more exploration to this extent.\n\n_Weaknesses_.\nThe theoretical results presented are somewhat lacking; as stated under the \"Quality\" header, they could be improved by providing some concrete examples and/or rigorous analysis. \n\n**Originality**\n\n_Strengths_.\nMany components of the algorithm are novel (e.g., the Dual-SVID initialization, residual compensation, and SmoothSign QAT), and others are combined in novel ways.\nThe review of prior literature is comprehensive and appropriately contextualizes the work.\n\n_Weaknesses_. N/A.", "was_revised": true, "final_justification": "The authors have clarified all of my questions on the scope and contributions of the work. I will maintain my score.", "rebuttal": "We sincerely thank you for your exceptionally thorough and positive review. We are greatly encouraged by your appreciation for the technical soundness, comprehensive evaluation, and novelty of our proposed methods. Your insightful feedback on our work's quality, clarity, and significance is invaluable for refining our paper.\n\nWe would like to address your questions and suggestions:\n\n1.  Regarding Theoretical Justification (Lemmas in Appendix A):\n\n    This is an excellent suggestion. We agree that reframing the lemmas as \"claims\" or \"heuristic arguments\" more accurately conveys their role in providing intuition for our design choices, rather than serving as rigorous proofs. We will revise the appendix accordingly in the final version. As you suggested, we will also investigate adding concrete numerical examples to further strengthen this intuition and make the concepts more tangible.\n\n2.  Regarding End-to-End Latency:\n\n    This is a great point. Our work focuses on demonstrating potential at the kernel level, as linear layers are the primary computational bottleneck. We will add a note to our discussion acknowledging that end-to-end autoregressive generation benchmarks are an important and natural direction for future work.\n\n3.  Regarding Algorithmic Features (Calibration-aware init, non-uniform bit allocation):\n\n    These are indeed promising research directions. For this work, our goal was to first establish the fundamental viability of our combined factorization and binarization approach. We agree that advanced techniques like calibration-aware initialization and non-uniform bit allocation could be integrated into our framework. We appreciate the insight and will certainly consider these for future research to further enhance performance.\n\n4.  Regarding Performance at the 1-bit Regime:\n\n    This is an important point. Our primary focus was on establishing a robust and stable method in the challenging sub-1-bit regime, where existing methods falter. As our priority was demonstrating stability in the largely unexplored sub-1-bit regime, we did not focus on optimizing performance specifically at the 1.0 BPW. However, we agree that improving performance at 1.0 BPW is important for creating a robust method. This is a planned area for our future work, where we intend to explore various optimization strategies to enhance competitiveness at this level.\n\n5.  Regarding Clarity Points:\n\n    Finally, we appreciate your careful reading and thank you for these actionable suggestions. We will update Figure 1 to include a stronger 1-bit baseline (OneBit & BinaryMoS) to provide a more competitive comparison. We will also revise the introduction to Residual Compensation in Section 3.3 to better summarize the key insights from the appendix, clarifying its motivation upfront. Thank you for helping us improve the paper's clarity.\n\nThank you again for your valuable and supportive review, which will significantly improve the final version of our paper."}, {"review_id": "iWBNFp6D5N", "reviewer": "Reviewer_c6YK", "summary": "In this paper, author proposed LittleBit, a novel method advancing LLM compression to the extreme sub-0.5 BPW regime, with viable performance at 0.1 BPW. Gains were achieved via SVD-inspired latent matrix factorization, binarization of these factors, and a multi-scale (row, column, latent) compensation mechanism. Proposed method also introduced Dual-SVID initialization for stable quantization-aware training (QAT) and integrated Residual Compensation for error mitigation.", "questions": "Please see Weakness.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Integrates low-rank decomposition with binarization processing and introduces a multi-scale residual compensation module, effectively mitigating accuracy degradation under ultra-low-bit quantization.\n\n2. Conducts comprehensive comparative experiments across models of various architectures and scales, covering multiple metrics including perplexity and zero-shot accuracy.\n\n3. Performs thorough ablation studies, providing quantitative validation for core components (e.g., Dual-SVID and residual compensation) with rigorous logic and substantial data support.\n\nWeaknesses:\n\n1. While the paper compares parameter counts across models of diverse architectures and scales, it fails to provide corresponding computational cost comparisons.\n\n2. Despite claiming applicability for lightweight and edge devices, the work implements no prototype on FPGA, NPU, or other edge hardware, nor does it evaluate inference latency or quantify computational costs on general platforms (e.g., GPU).\n\n3. The paper states that decomposing W<sub>pri</sub> reduces parameters to facilitate lightweight deployment, yet no ablation study validates the effectiveness of this decomposition strategy.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive comments and are encouraged that you recognized the novelty of our method, the comprehensive nature of our experiments, and the rigor of our ablation studies. We would like to address the identified weaknesses, as we believe these clarifications will further strengthen our paper.\n\n1.  Regarding Computational Cost Comparisons (FLOPs):\n\n    This is an excellent point. While we focused on empirical latency—which is often memory-bound and more representative for edge deployment—we agree that a theoretical FLOPs analysis provides a valuable complementary perspective. To offer a more granular view of our method's computational efficiency, we provide a sample operational cost analysis below.\n    -   Calculation Process:\n        Our method replaces expensive FP16 multiply-accumulate (MAC) operations with cheaper FP16 additions and highly efficient bitwise operations (BOPs). We analyzed a forward pass for a Llama2-7B MLP layer ($d_{in} = 11,008$, $d_{out} = 4,096$) at 0.3 BPW ($r = 431$).\n        1.  FP16 Baseline Ops: A standard matrix-vector multiplication consists of $d_{in} \\times d_{out}$ multiply-accumulate operations. Counting each MAC as 2 FLOPs (1 multiplication, 1 addition), the total is $2 \\times d_{in} \\times d_{out}$. For our example, this is $2 \\times 11,008 \\times 4,096 \\approx 90.2$ million FLOPs.\n        2.  LittleBit Ops: The computation is divided into floating-point and bitwise operations, considering both the primary and residual paths.\n            -   FLOPs: These arise mainly from FP16 additions during accumulation and scaling multiplications. The total is approximately $2 \\times (d_{in} + d_{out}) \\times r$, which for our example is $2 \\times (11,008 + 4,096) \\times 431 \\approx 13.0$ million FLOPs.\n            -   BOPs: Multiplication with the binary weights (±1) is replaced by a sign-bit XOR operation. The total is approximately $2 \\times r \\times (d_{in} + d_{out})$, which for our example is $2 \\times 431 \\times (11,008 + 4,096) \\approx 13.0$ million BOPs.\n\n      Based on this, the operational cost comparison is as follows:\n      | **Metric** | **FP16 Baseline** | **LittleBit (0.3 BPW)** | **Reduction** |\n      | --- | --- | --- | --- |\n      | Floating-Point Ops (FLOPs) | ≈ 90.2 million | ≈ 13.0 million | 6.9 × |\n      | Bitwise Ops (BOPs) | 0 | ≈ 13.0 million | - |\n\n    As the analysis shows, LittleBit significantly reduces the number of floating-point operations. Although it introduces bitwise operations, these are much faster on modern hardware. This provides a strong theoretical justification for our empirical latency results. We will include a full breakdown for the entire model in the revised appendix.\n\n2.  Regarding Applicability and Implementation on Edge Devices:\n\n    We completely agree that deployment on physical edge hardware is the ultimate goal. Our work's primary contribution is to overcome the most critical first barrier: the prohibitive memory footprint that makes loading such models impossible on most edge devices. By reducing model sizes by over $30 \\times$, LittleBit makes it feasible to load these models onto memory-constrained devices for the first time.\n\n    We thus position our GPU latency results as a hardware-agnostic proof-of-concept demonstrating the fundamental computational efficiency of our approach. By replacing large FP16 matrix multiplies with smaller binary operations, we demonstrate a principle of acceleration that is fundamental and translatable to various hardware backends, including NPUs and FPGAs.\n\n    We agree that hardware-specific implementations are a vital and exciting next step. We will clarify in our conclusion and limitations that our work provides the essential algorithmic foundation, with hardware co-design being a crucial direction for future research.\n\n3.  Regarding the Ablation Study for the Decomposition Strategy ($\\mathbf{W} \\approx \\mathbf{U}\\mathbf{V}^\\top$):\n\n    Thank you for this sharp observation, which allows us to better highlight our core design rationale. While we did not present an ablation removing the $\\mathbf{W} \\approx \\mathbf{U}\\mathbf{V}^\\top$ decomposition (as it is the foundation of our method), we validated this strategic choice in Appendix A.5 (Table 9).\n\n    This section directly compares low-rank factorization (SVD) against a leading alternative for structural compression, magnitude pruning (Wanda). The results unequivocally show that at the extreme compression ratios targeted by LittleBit ($>50\\%$ reduction), low-rank factorization is fundamentally more robust and preserves model performance where pruning methods fail. This preliminary study served as the justification for building our method upon a low-rank backbone. We will make this link more explicit in Section 3 of the revised paper to ensure this design choice is clearly motivated.\n\nOnce again, we thank you for your thoughtful and valuable feedback. We are confident that incorporating these clarifications will significantly improve the paper and more clearly articulate the importance of our contributions."}, {"review_id": "PQjt7idSh5", "reviewer": "Reviewer_y8CC", "summary": "LittleBit is a sub-1-bit quantization method for LLMs that compresses weights to as low as 0.1 BPW via low-rank factorization and binarization. It introduces Dual-SVID, a QAT method, for stable training and multi-scale compensation to reduce quantization errors, outperforming prior methods like STBLLM across multiple model sizes.", "questions": "1. In Table 3, there is a gap between BPW and compression factor. Is this because of embedding? Any other factor? Need to explain this in the article.\n2. If I'm understand right, the \"Residual Compensation\" part has the same architectual as the primary part. It may act like residual during initialization, but may not be residual anymore after training. I could understand that two components is not equivalent to double rank, as h, l, g is different. But I don't understand quite well the design philosophy of this. Or another question, if this work, why isn't it reasonable to repeat this component to 3 or 4 times? Why just 2 times?\n3. The number of tokens comsumed by QAT should be reported for 5 epochs. \n4. I found that the 1 bit kernel implementation is not included in current code in supplementary. It would be good if authors could provide the code. Or at least more details about the implementation is expected. For example, does CUDA (or Triton) provide a built-in function to compute the outer product of two uint32_t matrices using XOR directly, or do you compute the XOR one vector pair at a time in a loop, or one value at a time? Is it on cuda core or tensor core?", "limitations": "Addressed in appendix.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:  \n1. The experiment result is impressive. With 0.1 BPW, LittleBit could still achieve ~15 PPL. \n2. The overall algorithm design is reasonable and understandable.\n3. Authors implemented a new 1 bit GEMV CUDA kernel. The idea of using XOR for 1 bit multiplication seems make sense.\n4. The SmoothSign is interesting, although the improvement is not quite large compared with STE.\n\nWeakness:\n1. 5x speedup is reported in Table 11. However, the rank (640) is already about 2% of dimension (28672). This means that even with 16 bit precision, the FLOPs of LittleBit is much less than 1/5 of baseline.\n2. Please check the question.", "was_revised": true, "final_justification": "After considering the rebuttal and the discussion among reviewers and the AC, I maintain my assessment and recommend a weak accept.", "rebuttal": "Thank you for the detailed review and for noting the impressive aspects of our results. Your insightful questions help us improve the clarity of our paper, and we address each point below.\n\n1.  On the 5x Speedup vs. Theoretical FLOPs\n\n    This is an excellent observation that correctly identifies the gap between our current results and the method's theoretical potential. Our architectural design intentionally trades a single, compute-intensive operation for a sequence of smaller, computationally simpler steps.\n\n    The primary goal of our custom kernel in this work was to validate that this decomposed architecture could yield a practical speedup against the highly-optimized FP16 baseline. The measured $5.5 \\times$ speedup successfully demonstrates this, confirming the viability of our approach.\n\n    This result should be viewed as a strong baseline, not the final optimized performance. We believe that significant further speedups are achievable through advanced, low-level optimizations such as operator fusion, which would combine the scaling and matrix multiplication steps into a single, more efficient kernel to reduce memory-access overhead. This optimization is a promising direction for future engineering work. We will revise Appendix A.8 to clarify this context, positioning our current results as a strong validation of the architecture's potential while acknowledging the path to further optimization.\n\n2.  On the Gap Between BPW and Compression Factor (Table 3)\n\n    You are correct. The gap arises because, like many quantization methods, LittleBit is applied only to the transformer blocks (i.e., the linear layers), while the token embedding and lm_head layers remain in FP16. These non-quantized components constitute a fixed-size overhead. We will clarify this in Section 5.\n\n3.  On the Design Philosophy of \"Residual Compensation\"\n\n    This is a very insightful question about our design philosophy.\n    -   The Two-Path Architecture: The decision to use two paths was based on empirical results. We found that the two-path architecture offered the optimal trade-off between performance and parameter efficiency. While we tested a three-path variant, it provided diminishing returns and, in some cases, slightly degraded performance, making the two-path design the clear sweet spot. We will add a note about this empirical validation in our revised paper.\n    -   Role of the Second Path After Training: You are correct to question its role post-training. The term \"Residual Compensation\" primarily reflects its function at initialization, where the second path explicitly models the residual error ($\\mathbf{W} - \\widehat{\\mathbf{W}}_\\mathrm{pri}$). During QAT, both paths are jointly optimized and learn to cooperate in a more complex manner to minimize the final loss. While it may no longer be a pure \"residual\" in the mathematical sense, this two-path architecture, seeded by the initial residual, proved consistently more effective than simply increasing the rank of a single path. Thank you for this point. We will revise Section 3.3 to clarify this important distinction between its initialization role and its jointly-optimized function post-QAT.\n\n4.  On QAT Token Consumption\n\n    Thank you for noting this. Based on the Llama-2-7B model, for our 5-epoch QAT process, the dataset (C4 + WikiText-2) consists of about 0.2 billion tokens per epoch, resulting in a total of about 1 billion tokens over 5 epochs. We will update Appendix A.7 with this precise detail.\n\n5.  On the 1-bit Kernel Implementation Details\n\n    We are unfortunately unable to share the complete kernel code at this pre-publication stage due to our organization's policy, but we are happy to provide more technical details as you requested.\n    Our custom CUDA kernel does not use a built-in function for this operation. It is designed to maximize throughput by leveraging bitwise logic on standard CUDA cores. Here is the process:\n    -   Bit-Packing: We pack 32 binary weights (representing values of +1 or -1) from the $\\mathbf{U_\\mathrm{sign}}$ and $\\mathbf{V_\\mathrm{sign}}$ matrices into a single 32-bit unsigned integer (`uint32_t`). This dramatically reduces the memory footprint and bandwidth required for weight loading.\n    -   Kernel Execution: The computation is performed in a CUDA kernel where thread blocks process parts of the matrix-vector multiplication. The kernel loops over the packed `uint32_t` weight data.\n    -   Core Operation: The \"multiplication\" by +1 or -1 is achieved by a bitwise XOR operation. For each FP16 activation value, we XOR its sign bit with the corresponding weight bit from the packed integer. This is functionally equivalent to a sign flip but is executed much faster by the hardware's logic units. The results are then accumulated.\n    -   Hardware Utilization: These operations (XOR, shifts, accumulation) run on the standard CUDA cores. Our kernel does not use Tensor Cores, as their architecture is optimized for dense matrix multiply-accumulate operations.\n\nWe hope this detailed explanation provides the necessary clarity. We thank you again for your detailed and constructive review, which has been invaluable in helping us strengthen our paper."}]

Author rebuttal:

Meta-review: This paper introduces techniques for extreme LLM compression down to 0.1 bits per weights, enabling deployment of LLMs in resource-constrained environments. This approach compresses weights into a low-rank form using latent matrix factorization combined with binarization. The method is validated on Llama models, demonstrating improved compression rate while maintaining accuracy compared to prior work.

The paper received favorable reviews from all reviewers. It is well written, with details clearly presented. To achieve aggressive 0.1bit-per-weight compression, every component - algorithm, architecture, and training- is carefully designed using novel techniques such as DualSVID initialization, residual quantization compensation, and SmoothSign, supported by thorough ablations on those core components. 

The evaluation is comprehensive, showing strong results in both compression rate and accuracy, as well as kernel level speedup. 

Therefore, this paper is recommended for acceptance.
Decision: Accept (poster)