Title: Efficient Multi-bit Quantization Network Training via Weight Bias Correction and Bit-wise Coreset Sampling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-bit quantization network

once-for-all training

coreset sampling
Abstract: Multi-bit quantization networks enable flexible deployment of deep neural networks by supporting multiple precision levels within a single model. However, existing approaches suffer from significant training overhead as full-dataset updates are repeated for each supported bit-width, resulting in a cost that scales linearly with the number of precisions. Additionally, extra fine-tuning stages are often required to support additional or intermediate precision options, further compounding the overall training burden. To address this issue, we propose two techniques that greatly reduce the training overhead without compromising model utility: (i) Weight bias correction enables shared batch normalization and eliminates the need for fine-tuning by neutralizing quantization-induced bias across bit-widths and aligning activation distributions; and (ii) Bit-wise coreset sampling strategy allows each child model to train on a compact, informative subset selected via gradient-based importance scores by exploiting the implicit knowledge transfer phenomenon. Experiments on CIFAR-10/100, TinyImageNet, and ImageNet-1K with both ResNet and ViT architectures demonstrate that our method achieves competitive or superior accuracy while reducing training time up to 7.88×.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThey proposes two techniques to reduce the training overhead of multi-bit quantization networks. The first technique is weight bias correction, which aligns the quantized weights to match the activation outputs across different bit-widths, enabling shared batch normalization and eliminating the need for an extra training stage. The second technique is bit-wise coreset sampling, which dynamically selects informative data subsets for each child model based on gradient-based importance scores, reducing training redundancy. Experiments on CIFAR-10/100, TinyImageNet, and ImageNet-1K with ResNet and ViT architectures demonstrate that the proposed methods achieve competitive or superior accuracy while significantly reducing training time.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper introduces two novel techniques, which are well-motivated and supported by both theoretical insights and empirical results. It is worth noting that this method is applicable to both ViT and CNNs and is not restricted by the BN layer.\n2. The proposed methods achieve up to 7.88× reduction in GPU hours, making multi-bit quantization networks more practical for deployment. \n3. The authors conduct extensive experiments across multiple datasets and architectures, demonstrating the effectiveness and generalizability of their approach.\n4. The paper is well-organized and clearly written. The authors provide detailed explanations of their methods, supported by intuitive figures and comprehensive tables, making it easy for readers to understand and reproduce the work.\n\nWeaknesses\n1. The evaluations are restricted to computer vision tasks. While this is understandable given the computational cost of training multi-bit networks, extending the approach to other domains such as generative LLMs would further validate the broad applicability of the methods. Of course, this is merely encouragement rather than a requirement.\n2. Eq. 2 was not properly introduced and was somewhat abrupt. The paper does not provide a formal theoretical analysis or explanation of the proposed techniques of Eq. 2. Why Eq.2 reduce the errors?\n3. Furthermore, does the [29] mentioned in line 165 have been utilized in the ViT network? This might need some ablation experiments.\n4. Implementing the bit-wise coreset sampling strategy might be complex and require significant modifications to existing training pipelines. The authors could provide more detailed guidance or tools to facilitate adoption by the research community.\n\nQUESTIONS:\nSee Strengths And Weaknesses\n\nLIMITATIONS:\nSee Strengths And Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper proposes a method to reduce the computation required to train multi-bit quantization models. The authors observed that the quantization error in the model comes from a bias between the different quantization widths and propose to address this bias by directly modifying the weights. This results in a lighter model that does not need a batch normalization layer per quantization width. Additionally, the authors also propose an efficient strategy to calibrate the model that leverages core set selection to reduce the number of samples that the model needs to visit, further reducing the computation required to train the model.

STRENGTHS AND WEAKNESSES:
Strengths:
- The paper is well written and easy to follow. The visualizations effectively support the claims in the text and help understand the proposed approach.
- The approach is well designed: each element tackles a weakness of the task in an efficient manner. As a result, the proposed approach surpasses SOTA approaches in accuracy and computation efficiency.
- The experimental section is comprehensive and well-designed.


Weaknesses:
- The novelty of the proposed approach is limited, and the design is somewhat simple. The contribution of the paper could be stronger with a more detailed discussion or a theoretical framework that explained the effect of the different elements of the model. This would bring valuable insights to the community.
- The efficiency benefits of the coreset sampling strategy should be better highlighted. This is an important contribution, and there is no formal definition of the computation required in relation to a standard approach.

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes an efficient training framework for producing multi-bit neural networks. The authors introduce a weight bias correction technique to align activation distributions, enabling the use of shared batch normalization during training and shifting the calibration burden to a more simple post-training adaptation step. Additionally, they propose a novel bit-wise dynamic coreset sampling method to significantly reduce training time. Empirical validation and comparisons with existing methods demonstrate the effectiveness of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:  \n1. Building on observations, the authors propose a novel approach to handle activation misalignment and introduce coreset sampling dedicated for multi-bit training.\n2. The method demonstrates significant improvements in wall-clock training time while maintaining quality, validating its effectiveness.  \n\n  \nWeaknesses:\n1. If my understanding is correct, sample-wise importance changes throughout the training process. However, the proposed sampling strategy calculates importance only once, prior to coreset selection, without updating it over time. Even though temperature-based sampling is used, samples with initially high probability are more likely to be selected consistently, which seems misaligned with the motivation of dynamically capturing changing important samples.\n2. The proposed weight bias correction is claimed to enable shared batch normalization during training. However, a separate BN layer is still used for the 1-bit setting. Does this imply that the bias correction mechanism is less effective or insufficient for 1-bit quantization?\n3. In the ImageNet-1K experiments, the paper mentions that the Any-Precision model is used for fine-tuning. Does this mean that fine-tuning is performed starting from a pre-trained Any-Precision model after the calibration or training phase? Clarification on the training pipeline would be appreciated.\n\nQUESTIONS:\n- Regarding weakness1, could the authors clarify this design choice and provide result of applying dynamic coreset sampling on spearman correlation?\n- Could the authors elaborate concerns on weakness 2 and 3?\n- Could the authors provide GPU hours with more detailed granularity— the time spent separately on training, calibration, and adaptation phases? This would help better assess the efficiency claims of the proposed method.\n\nLIMITATIONS:\nLimitations and potential societal impact are not addressed in the paper", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on reducing the large training overhead of multi-bit quantization networks, caused by training separate batch-normalization parameters for each bit-width to correct activation distributions and full dataset updates for each bit-width. To that end, the authors propose: 1) weight bias correction, by correcting quantization-induced bias and aligning activation distributions, enabling shared BN across bit-widths, and 2) a bit-wise coreset sampling strategy that accounts for sample importance, obtained via a novel bit-wise training scheme, across bit-width and training epoch. The method is demonstrated using multiple datasets and models and achieves consistent GPU hours speedup while matching the performance of other works.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well written and provides clear background information.\n2. The observations made throughout the paper are deeply insightful and clearly motivate the solution approach.\n3. The paper introduces a novel bit-wise training scheme for score evaluation that addresses the limitations of traditional methods aimed at single precision networks.\n4. The results are positive, showcasing a reduction in GPU hours while matching the performance of existing methods.\n\nWeaknesses:\n1. Activation alignment is showcased for the inefficient separate BN parameters method. However, the authors do not show the alignment achieved by the proposed bias correction for quantized weights method making the impact of the method unclear.\n2. The impact of BN adaptation at the final training stage is not clear. Ablation studies are needed to determine how much the alignment and final accuracy is impacted by bias correction for quantized weights vs. BN adaptation. \n3. Resampling is not addressed in the results. While it has been shown that sample importance changes over training epochs, the paper does not provide guidelines on how to choose the frequency of coreset sampling and its impact on accuracy and GPU hours.\n4. Some of the figures use symbols that are not explained anywhere, making it difficult to understand (ex. Figure 1 with D and S)\n\nQUESTIONS:\n1. How important is the BN adaptation performed at the last training stage? Why is bias correction for quantized weights not enough for activation alignment? Are most of the accuracy gains due to BN adaptation or bias correction?\n2. How often is coreset sampling performed? What is the impact of the frequency at which it is performed on accuracy and GPU hours? Also, from the results, it seems that most of the speedup is due to the coreset sampling method. What would the accuracy and speedup be if only coreset sampling is performed?\n3. In Section 3, how is the statement “achieving optimal performance without additional overhead” true? It is clear that BN adaptation will incur additional overhead that needs to be considered. \n4. For Table 3 and 4, it is mentioned that the results are obtained by fine tuning a pretrained Any-Precision model. Why are the GPU hours for Any-Precision model training then not included in the reported hours for the proposed method?\n5. Some results in Table 1, 3 and 5 are bolded while they are not the best. Why?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}