Title: LittleBit: Ultra Low-Bit Quantization via Latent Factorization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large language models

Low-bit quantization

Low-rank factorization

Sub-1-bit model compression

Binarization
Abstract: The deployment of large language models (LLMs) is frequently hindered by prohibitive memory and computational requirements. While quantization mitigates these bottlenecks, maintaining model fidelity in the sub-1-bit regime remains a persistent challenge. In this paper, we introduce LittleBit, a novel framework for extreme LLM compression. We target quantization rates as low as $0.1$ bits per weight (BPW), achieving a memory reduction of approximately $31\times$, which effectively compresses Llama2-13B to under $0.9$ GB. We represent weights via low-rank latent matrix factorization and subsequently binarize the resulting factors. To counteract the information loss inherent to such drastic precision reduction, we integrate a multi-scale compensation mechanism that learns importance parameters across row, column, and latent dimensions. Two primary contributions enable effective training: Dual Sign-Value-Independent Decomposition (Dual-SVID) for quantization-aware training (QAT) initialization, and Residual Compensation to minimize approximation errors. Extensive experiments confirm the superiority of LittleBit in the sub-1-bit domain; for instance, our method at $0.1$ BPW surpasses the performance of leading techniques operating at $0.7$ BPW on Llama2-7B. We establish a new size-performance trade-off---unlocking a potential $11.6\times$ inference speedup relative to FP16---and render powerful LLMs practical for resource-constrained environments. Our code is available at https://github.com/SamsungLabs/LittleBit.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLittleBit is a sub-1-bit quantization method for LLMs that compresses weights to as low as 0.1 BPW via low-rank factorization and binarization. It introduces Dual-SVID, a QAT method, for stable training and multi-scale compensation to reduce quantization errors, outperforming prior methods like STBLLM across multiple model sizes.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:  \n1. The experiment result is impressive. With 0.1 BPW, LittleBit could still achieve ~15 PPL. \n2. The overall algorithm design is reasonable and understandable.\n3. Authors implemented a new 1 bit GEMV CUDA kernel. The idea of using XOR for 1 bit multiplication seems make sense.\n4. The SmoothSign is interesting, although the improvement is not quite large compared with STE.\n\nWeakness:\n1. 5x speedup is reported in Table 11. However, the rank (640) is already about 2% of dimension (28672). This means that even with 16 bit precision, the FLOPs of LittleBit is much less than 1/5 of baseline.\n2. Please check the question.\n\nQUESTIONS:\n1. In Table 3, there is a gap between BPW and compression factor. Is this because of embedding? Any other factor? Need to explain this in the article.\n2. If I'm understand right, the \"Residual Compensation\" part has the same architectual as the primary part. It may act like residual during initialization, but may not be residual anymore after training. I could understand that two components is not equivalent to double rank, as h, l, g is different. But I don't understand quite well the design philosophy of this. Or another question, if this work, why isn't it reasonable to repeat this component to 3 or 4 times? Why just 2 times?\n3. The number of tokens comsumed by QAT should be reported for 5 epochs. \n4. I found that the 1 bit kernel implementation is not included in current code in supplementary. It would be good if authors could provide the code. Or at least more details about the implementation is expected. For example, does CUDA (or Triton) provide a built-in function to compute the outer product of two uint32_t matrices using XOR directly, or do you compute the XOR one vector pair at a time in a loop, or one value at a time? Is it on cuda core or tensor core?\n\nLIMITATIONS:\nAddressed in appendix.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-05"}

comments: SUMMARY:
In this paper, author proposed LittleBit, a novel method advancing LLM compression to the extreme sub-0.5 BPW regime, with viable performance at 0.1 BPW. Gains were achieved via SVD-inspired latent matrix factorization, binarization of these factors, and a multi-scale (row, column, latent) compensation mechanism. Proposed method also introduced Dual-SVID initialization for stable quantization-aware training (QAT) and integrated Residual Compensation for error mitigation.

STRENGTHS AND WEAKNESSES:
Strengths:

1. Integrates low-rank decomposition with binarization processing and introduces a multi-scale residual compensation module, effectively mitigating accuracy degradation under ultra-low-bit quantization.

2. Conducts comprehensive comparative experiments across models of various architectures and scales, covering multiple metrics including perplexity and zero-shot accuracy.

3. Performs thorough ablation studies, providing quantitative validation for core components (e.g., Dual-SVID and residual compensation) with rigorous logic and substantial data support.

Weaknesses:

1. While the paper compares parameter counts across models of diverse architectures and scales, it fails to provide corresponding computational cost comparisons.

2. Despite claiming applicability for lightweight and edge devices, the work implements no prototype on FPGA, NPU, or other edge hardware, nor does it evaluate inference latency or quantify computational costs on general platforms (e.g., GPU).

3. The paper states that decomposing W<sub>pri</sub> reduces parameters to facilitate lightweight deployment, yet no ablation study validates the effectiveness of this decomposition strategy.

QUESTIONS:
Please see Weakness.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a novel method for compressing LLMs in the ultra-low-bit (as low as 0.1 bits per parameter) regime.\nThis is the first method to achieve stable performance at such a high compression ratio.\nAt a high level, compression is achieved via a binarized low-rank decomposition (with row-wise, column-wise, and latent FP16 scales), followed by quantization-aware training under a knowledge distillation framework.\nAccuracy of this method relies on some specialized algorithmic decisions: initialization of the scaling factors based on a rank-one approximation of the SVD factor magnitudes, and residual compensation (decomposing the weight matrix into two separate additive components).\nAblation studies and some theoretical intuition are provided to justify these design choices.\n\nEvaluation is performed on a large range of LLMs, comparing perplexity and task accuracy to a structured sparse baseline (in the $<1$ bit/parameter regime) and state-of-the-art one-bit quantization methods. \nAlthough LittleBit does not beat the baselines at the $\\\\approx1$ bit/parameter regime, it is the only method to achieve good accuracy and stability at under $\\\\approx0.5$ bits/parameter.\nThe authors develop custom CUDA kernels for compressed matrix multiplication, achieving an average $5.5\\\\times$ speedup over FP16 matrix multiplication at 0.1 bits/parameter.\n\nSTRENGTHS AND WEAKNESSES:\n**Quality**\n\n_Strengths_.\nThe work is technically sound, with careful attention placed to almost every component of the algorithm in order to compress LLMs to the very aggressive $0.1$ bit/parameter regime with reasonable degradation.\nMany strategies are introduced in all parts of the compression pipeline (e.g., DualSVID initialization, residual quantization, SmoothSign, larger rank for grouped query attention matrices), and all are justified via ablation studies.\nIn addition, the benefit of residual quantization, which is perhaps surprising, is justified via theoretical intuition.\n\nComprehensive evaluation is performed: perplexity on three common datasets, zero-shot task accuracy, latency (with custom CUDA kernels for compressed matrix multiplication), memory footprint reduction, and estimated KV cache footprint reduction.\nResults are provided over a variety of LLMs.\nAt ultra-low bit rates of $<0.5$ bits/per parameter, the proposed method maintains reasonable accuracy and is very stable up to $0.1$ bits per parameter, whereas baselines experience significant degradation in that regime.\n\nThe discussion of limitations is nuanced and broad, touching on output degradation due to lossy compression, computational complexity of the QAT component, limitations in compression ratio, and limitations in theoretical understanding.\n\n_Weaknesses_.\nThe two lemmas (and, by extension, the theorem) in the appendix are substantiated by theoretical intuition rather than rigorous proofs.\nThough the intuition is sound, I believe it would be more accurate to reframe the lemmas as \"claims.\"\nThey could also be strengthened via concrete examples where they hold (e.g., finding some neural network weight matrix where the conditions of the lemmas hold, or even synthetic data).\n\nAlso, the accuracy could possibly be improved by non-uniform bit allocation between layers, or an initialization of the low-rank factors that takes into account calibration data (which should have a closed-form solution via results from [Xiang et al, \"Optimal exact least squares rank minimization\"]).\n\n**Clarity**\n\n_Strengths_.\nThe paper is well-organized and well-written, and its primary contributions, as well as their motivation, are quite clear.\nAll algorithmic and experimental details appear to be provided.\nThe discussion of the results is accurate and highlights the benefits of the proposed method.\n\nAlso, though the SmoothSign approach to quantization-aware training seems to be novel, this is not clear from reading the paper.\n\n_Weaknesses_.\nAt the introduction of residual compensation, the motivation was a little unclear; it's not immediately obvious that increasing the primary rank is an issue that would hurt accuracy.\nAs this is an important contribution, I would recommend refining the discussion introducing residual compensation to summarize the key insights from Appendix A. \n\nAlso, a minor note: Figure 1 chooses BiLLM for the one-bit method instead of the better performing OneBit or BinaryMoS baselines.\n\n**Significance**\n\n_Strengths_.\nThis paper pushes the bounds of ultra-low-bit compression of LLMs to 0.1 bits/parameter, and is the first work to do so.\nThis could encourage further work in this particularly-challenging compression regime.\nThe authors introduce several algorithmic nuances, any of which would be quite useful in future LLM compression algorithms. \n\nIn addition, the the empirical results and theoretical intuition on the benefit of residual compensation could encourage more exploration to this extent.\n\n_Weaknesses_.\nThe theoretical results presented are somewhat lacking; as stated under the \"Quality\" header, they could be improved by providing some concrete examples and/or rigorous analysis. \n\n**Originality**\n\n_Strengths_.\nMany components of the algorithm are novel (e.g., the Dual-SVID initialization, residual compensation, and SmoothSign QAT), and others are combined in novel ways.\nThe review of prior literature is comprehensive and appropriately contextualizes the work.\n\n_Weaknesses_. N/A.\n\nQUESTIONS:\n(1) Is there additional theoretical justification that can be provided for the two lemmas in Appendix A, or numerical examples that can be shown to strengthen the intuition provided?\n\n(2) Can latency and bandwidth results be provided for an end-to-end setting (such as autoregressive generation), as opposed to just on the level of a single matrix multiplication?\n\n(3) Have potential algorithmic features like a calibration-data-aware initialization for the low-rank factors and non-uniform bit allocation been considered?\nDo you think they would make a decent impact on the accuracy of the method, or would they just unnecessarily increase the computational cost?\n\n(4) Do the authors have ideas to improve performance at the 1-bit regime (note: this is a minor question, as I understand that the point of the work is the $<$ 1-bit regime)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper introduces LittleBit, a novel quantization method for compressing LLMs to ultra-low bit widths, achieving as low as 0.1 bits per weight. By combining low-rank matrix factorization with binary representations and a multi-scale compensation mechanism, LittleBit significantly reduces memory and computational costs without severely sacrificing model performance. Key technical contributions include: 1. The Dual-SVID initialization for stable quantization-aware training (QAT) and 2. Residual Compensation to further mitigate binarization errors. Experimental results show that LittleBit outperforms prior state-of-the-art methods across a range of LLMs.

STRENGTHS AND WEAKNESSES:
Strengths:
1. This paper pushes the boundary of model quantization to ultra-low 0.1bit per weight, offering a practical path for deploying LLMs on resource-constrained devices.
2. The integration of low-rank factorization with binary quantization and the multi-scale compensation mechanism, particularly the latent dimension scaling, is novel.

Weaknesses:
1. Performance at <1 BPW, while notable, still shows a steep degradation, making them unlikely to be deployed in real-world applications.
2. The use of both a primary and residual factorized path (each with learnable scales and binary matrices) increases architectural complexity and parameter count.
3. The decrease of average latency is far below optimal. In the ideal case, 1 BPW can offers 16x speedup. However, in the paper, the speedup can only be demonstrated when BPW is lower than 1.

QUESTIONS:
Please refer to the Weakness section.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01