Efficient Bayesian DNN Compression through Sparse Quantized Sub-distributions

Authors: Ziyi Wang, Guang Lin, Qifan Song
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: Bayesian Deep Neural Networks, Quantization, Pruning, Variational Inference

Abstract:
This paper presents a novel method that simultaneously achieves model pruning and low-bit quantization through Bayesian variational inference to effectively compress deep neural networks (DNNs) while suffering minimal performance degradation. 
Unlike previous approaches that treat pruning and quantization as separate, sequential tasks, our method explores a unified optimization space, enabling more efficient compression. 
By leveraging a spike-and-slab prior combined with Gaussian Mixture Models (GMM), we can achieve both network sparsity and low-bit representation. Experiments on CIFAR-10, CIFAR-100, and SQuAD datasets demonstrate that our approach achieves compression rates of up to 32x with less than a $1.3\\%$ accuracy loss on the CIFAR datasets and a 1.66 point decrease in F1 score on SQuAD. Additionally, we show that the Bayesian model average of neural networks can further mitigate the impact of quantization noise, leading to more robust compressed models. Our method outperforms existing techniques in both compression efficiency and accuracy retention, offering a promising solution for compressing DNNs.

Decision: reject
Reviewer scores: 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729747061291

thread_type:
official_review

<thread_object>
creator: reviewer_kfpy

<content>
summary:
This paper proposes a framework that integrates quantization and pruning into a unified process, rather than applying them sequentially. This study reformulates the weight quantization issue as the selection of a sub-distribution within the Gaussian Mixture Model, while the pruning problem is approached as a probability sampling problem from a Bernoulli Distribution. The authors established an approximate ELBO objective that consolidates the optimization processes for pruning and quantization into a unified optimization framework. Results for the ResNet and BERT models in image classification and question answering tasks demonstrate a significant compression rate with low performance degradation.

strengths:
1. The idea of combining the optimization processes of quantization and pruning into a unified optimization space is interesting, and the optimization objectives have been thoroughly theoretically derived.
2. The paper is easy to read.

weaknesses:
1. While the authors perform experiments on ResNet and BERT models, they restrict their testing to small-sized datasets (CIFAR10/100, SQUAD V1.1). Furthermore, the evaluated tasks are standard (image classification), in contrast to the previous work DGMS compared in this paper, which conducted experiments on a large dataset (ImageNet) and an object detection task. I think the proposed method may have certain inherent limitations.
2. The idea is not novel.
3. The main limitation of this paper is that proposed method lacks more experimental results on larger datasets (i.e., Imangenet, COCO2017, VOC), larger networks (i.e., VIT, EfficientNet, LLMs, etc), and compared with more competitors (i.e., OMPQ [1], EMQ [2], etc).
4. The statement "HAQ is suboptimal" is not accurate in Section I. You should provide empirical results to prove your statement.
5. The format of references is not correct, i.e.,  "Deep learning with limited numerical precision"
6. There exist some typos/grammatical errors in the paper and should be revised.
7. The presentation of the paper is bad.
8. The paper claims that the method outperforms state-of-the-art methods, but a more comprehensive comparative analysis is needed. Detailed comparisons with other existing approaches, along with discussions about the reasons behind the performance differences, would strengthen the argument for the superiority.
9. Exploring the reasons behind the success of these techniques and providing intuitive explanations would contribute to the overall scientific contribution of the work.
10. Provide a more thorough discussion of the generalization ability, robustness, and potential applications of the proposed approach.



[1] OMPQ: Orthogonal Mixed Precision Quantization
[2] EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantization

questions:
1. In line 176, is $T$ used before definition?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730599068508

thread_type:
official_review

<thread_object>
creator: reviewer_ecze

<content>
summary:
This paper introduces a novel method that simultaneously achieves model pruning and low-bit quantization through Bayesian variational inference, effectively compressing deep neural networks with minimal performance degradation. This method explores a unified optimization space using a spike-and-slab prior combined with Gaussian Mixture Models, achieving significant (up to 32x) compression rates with negligible accuracy loss on CIFAR-10, CIFAR-100, and SQuAD datasets. The proposed approach outperforms existing techniques and demonstrates that Bayesian model averaging can further mitigate quantization noise for more robust compressed models.

strengths:
Pros:
1.	The proposed method unified the pruning and quantization, further shrinking the model size with less model accuracy drop.
2.	Approximating the sub-distribution problem and leveraging the variational learning to solve the problem reduces the complexity of optimizing the quantization and sparsity in all weight data.
3.	Bayesian average training is introduced to enhance the performance.

weaknesses:
Cons:
1.	The experiments are not sufficient to demonstrate the effectiveness of the LLM model. The Bert model and GPT-2 model are not widely used LLM models.  
2.	The baselines are not the SOTA work. Some SOTA works, e.g. AWQ, are expected to be compared in evaluation.
3.	Can you provide the evaluation in Table 5 with some baselines to show the proposed framework outperforms the existing works?

questions:
See weeknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692782959

thread_type:
official_review

<thread_object>
creator: reviewer_9lab

<content>
summary:
This paper presents a novel method called Sparse Quantized Sub-distribution (SQS) that unifies neural networks pruning and quantization through Bayesian variational inference. The key innovation is treating both compression techniques as part of a single optimization problem rather than sequential steps. The method leverages spike-and-slab priors combined with Gaussian Mixture Models (GMMs) to achieve both network sparsity and low-bit representation, while using Bayesian model averaging to improve robustness to quantization noise.

strengths:
- The unified treatment of pruning and quantization as a single single optimization problem is novel and well-motivated. 
- The mathematical framework combining spike-and-slab priors with GMMs is theoretically sound. 
- The preliminaries are rigorously explained. 
- Mathematical notation is consistent and well explained. 
- The methodology is presented in a reproducible manner.
- Strong results for ResNet/BERT on CIFAR/SQuAD. Upto 32x compression rates with minimal performance degradation. The baselines are well defined and diverse for a fair comparison.

weaknesses:
- Some hyperparameter choices could be better justified.
- The results presented are technically on just 2 different architectures : ResNet and BeRT, which are very limited. 

Based on the suggestions/questions below, I am more than willing to increase my score as the paper does show promise.

questions:
1. I would like to see similar results across other computer vision models, on other datasets than just CIFAR. Maybe include results on ImageNet1k with a wider variety of models. 
2. Similarly for the NLP task, I would like to see performance on other tasks like GLUE/SuperGLUE/MMLU etc with a wider variety of models and not just BERT. 
3. Consider adding an ablation study on the impact of different prior choices other than spike-and-slab. It might not work and that is fine, but having a better understanding on why only this prior works would be helpful. 
4. An analysis on scalability to larger models would also be helpful.
5. For Table 1 and Table 2, consider adding in the captions the type of test/dataset on which you evaluate the model on. Otherwise the captions and the results are a bit ambiguous if one does not go over the experiment section in detail.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730744335498

thread_type:
official_review

<thread_object>
creator: reviewer_vowk

<content>
summary:
The authors proposed a variational inference-based method for pruning and quantizing DNNs.

strengths:
+ Well-motivated idea. 
+ Novel ELBO approximation enabling the method in practice.
+ Experiments on both vision and natural language networks.

weaknesses:
- Computational expenses and data requirement not well discussed.  
- The existence of extra hyperparameters requires further tuning, and the nontrivial balance between the aggressiveness of pruning and of quantization is not well addressed.

questions:
* It is not really clear to a practitioner, after reading the paper, how to tune the knobs of this new method, at what cost, and how to factor in the specific information of the target hardware the model is to be deployed on.
</content>
</thread_object>

</thread>
<thread>
creation date:
1760337159256

thread_type:
public_comment

<thread_object>
creator: ~thomas_pfeil1

<content>
comment:
How does this work relate to
Achterhold, Jan, et al. "Variational network quantization." ICLR. 2018.
https://openreview.net/forum?id=ry-TW-WAb
?
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary**

This paper introduces Sparse Quantized Sub-distribution (SQS), a novel framework that integrates model pruning and low-bit quantization into a unified optimization process using Bayesian variational inference. By treating weight quantization as selecting sub-distributions within a Gaussian Mixture Model and pruning as sampling from a Bernoulli Distribution, the method leverages spike-and-slab priors combined with GMMs. This approach consolidates the optimization processes for pruning and quantization, allowing for significant network compression (up to 32x) with minimal performance loss, as demonstrated on ResNet and BERT models across CIFAR-10, CIFAR-100, and SQuAD datasets. The method outperforms existing techniques by mitigating quantization noise through Bayesian model averaging, enhancing the robustness of the compressed models.



**Strengths**


* The paper is well-written and easy to follow.
* The unified treatment of pruning and quantization as a single optimization problem is well-motivated, resulting in shrinking the model size with less model accuracy drop.
* Approximating the sub-distribution problem and leveraging the variational learning to solve the problem reduces the complexity of optimizing the quantization and sparsity in all weight data.
* The mathematical framework combining spike-and-slab priors with GMMs is theoretically sound and interesting.

**Weaknesses**

* While the authors perform experiments on ResNet and BERT models, they restrict their testing to small-sized datasets (CIFAR10/100, SQUAD V1.1).  The proposed method lacks more experimental results on larger datasets for the results to be conclusive. 
* The baselines used in this paper are not the current state-of-the-art methods.
* The paper's presentation could be significantly improved.

**Conclusion** 

The paper received borderline reviews, primarily due to the lack of robust baselines and substantial scale experiments, as noted by most reviewers. After reviewing the manuscript, the feedback from the reviewers, and the authors’ responses, I recognize the merit in the approach of integrating pruning and quantization through variational inference using spike-and-slab priors and Gaussian Mixture Models. However, the limited scope of the experiments and the inadequate comparison with more recent relevant literature make it challenging to assess the practical impact of this work. Based on these considerations, I recommend rejection as the paper does not seem ready for publication.