Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes to simultaneously learn model compression and sparsity in a latent space. Experiments on CIFAR-10 and ImageNet show that the proposed methods significantly outperforms previous competitive methods on compression and pruning only.
strength_and_weaknesses: Pros.
1) I think the core idea is interesting and reasonable: building a latent space of quantized model parameters for simultaneously learning model compression and sparsity (both unstructured and structured) with continuous weights maintained for sparsity learning.
2) Experimental results on CIFAR-10 and ImageNet in Table 1 are impressive. The proposed method achieves much better compared to previous methods, from perspective of both compression rate and accuracy.

Cons.
My major concern is that the whole paper (including both the method description and experiments) does not clearly introduce contribution of quantization to the proposed method. To be specific: 
1) After reading Section 3, I am still not clear how the model weights are quantized during training. The Equation (8) does not show processing on quantization. 
2) In Section 4 and Section 5, the role/contribution of quantization to the experimental results is not formally analyzed. Maybe ablation studies are needed.

The paper clearly describes unstructured and structured sparsity but not quantization. According to my experiences on DNN model compression, quantization performs well on both model compression and acceleration, while structured spasity performs better on acceleration compared to unstructured sparsity. Anyway, I am not clear about the specific contribution of the latent space of quantized weights.
clarity,_quality,_novelty_and_reproducibility: 1) Clarity. As I mentioned above, the whole paper does not clearly introduce contribution of quantization to the proposed method
2) Quality. The writing quality is acceptable.
3) Novelty. The code idea, a latent space based combination of compression and sparsification, is novel.
4) Reproducibility. Code is provided. The paper itself includes sufficient implementation details.
summary_of_the_review: The code idea is novel and experimental results are impressive, though contributions of the latent space of quantized weights are not clearly introduced.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new framework to jointly optimize the model size and inference acceleration.  Compared to previous methods, This paper can train the sparse model considering the model size and computation efficiency simultaneously using an end-to-end manner. The unified framework is novel and elegant.
strength_and_weaknesses: Strength:

The paper is well-written, and code is also provided for reproductivity.

Compared to the previous pruning method, this paper is more practical, because mode deployment needs to consider model size and memory.

This paper analysis part is Thorough.




Weaknesses:

In terms of the model's size, the author are encouraged to discuss different sparse formats (CSR, COO, N:M) [1] to encode the non-zero elements.





[1]. STen: An Interface for Efficient Sparsity in PyTorch 

clarity,_quality,_novelty_and_reproducibility: Na
summary_of_the_review: Na
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an algorithm to optimally compress a model while preserving performance and improving inference speed. The algorithm achieves compression by modelling the weights as quantized latent representations sampled from a Gaussian prior which are further optimized using an entropy penalty. In order to ensure inference speedup, the algorithm also leverages a structured sparsity based loss. The authors show the efficacy of their algorithm on CIFAR-10/100 and Imagenet and claim better model compression due to the analogous behavior of the two constraints.
strength_and_weaknesses: **Strengths**

1. The problem of model compression which preserving inference speed is important for realtime and edge applications, and therefore interesting to the community.
2. The authors show comparable results to existing compression approaches (Oktay et al. especially) while also adding sparsity as a constraint. 
3. The algorithm cleverly uses a zero mean prior to further enforce sparsity.

**Weaknesses**
1. I am not sure why the loss contains both a group sparse and sparsity enforcing constraint. These are obviously analogous constraints, and ideally the algo should be able to achieve the same result by adjusting the structured sparsity constraint. Fig. 3 shows some effects of sparsity on the model size, but it does not seem to be due to any change in slice sparsity. Could the authors clarify what Fig.3 is supposed to represent?

2.  The paper also does not compare with more modern sparsification methods (for eg. Continuous Sparsification [1], structured continuous sparsification[2]) which achieve similarly sparse models while retaining performance. It would be fair to compare model sizes and accuracies of standard sparsification algorithms as well. 

3. I also suggest the authors add bit-quantization based approaches to the comparisons so that their results can be placed in the context of common compression methods. A few simple baselines of 8-bit or 4-bit quantization could help readers validate the need for training models with the proposed algorithm.

Refs

[1] Savarese et al., Winning the Lottery with Continuous Sparsification

[2] Yuan et al., Growing Efficient Deep Networks by Structured Continuous Sparsification
clarity,_quality,_novelty_and_reproducibility: Overall the paper is well written and easy to follow. The algorithm itself is of limited novelty (combining a prior based quantization approach from Oktay, 2020 ,and a common structured sparsity based loss.). However, I did not find any previous work which tackles model compression and structured sparsity simultaneously explicitly. The paper may therefore be of independent interest to some in the community. The authors have provided the setup and code to reproduce their results, which I was able to partially validate. 
summary_of_the_review: Given the limited novelty and missing comparisons with modern structured sparsity enforcing algorithms, I am currently inclined towards a weak reject. The performance boosts due to the weight sampling mechanisms are unclear given that recent results in bit level compression have shown almost no loss in accuracy with significant speedups. However, i am open to changing my mind if the authors can provide reasonable rebuttals to my concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work adopted a combination of losses that jointly regulate model size, weight decay, and structured sparsity to achieve reduced weight encoding bits and slice sparsity. The authors show promising results in reducing model storage size in their experiments.
strength_and_weaknesses: Strength:
The authors show that their approach can give a significantly better storage size compression ratio when compared with earlier works.

Weakness:
Some statements in the manuscript are not well supported. Moreover, there is a lack of comparison with a proper baseline, especially for the acceleration experiments.

It is not true: "Most quantization works only optimize for model compression and not for inference speedups.". It is a standard industry practice to use quantized models, and the use case of an optimized model only for storage size compression does exist but is limited. 

It is unclear how much of the performance gain is from the training procedure and fine-tuning. Additional ablation experiments would make the paper stronger. 

The slice-sparsity approach can be considered a particular case of block sparsity, and I believe there is probably a need for additional justification for this new approach. 

The uncompressed MobileNet-v2 accuracy presented in the paper is incorrect. It should be 28 instead of 32.7
clarity,_quality,_novelty_and_reproducibility: The paper is of sound clarity.
summary_of_the_review: Overall this work presented promising results but can use extra improvement.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The manuscript proposes to simultaneously learn model compression and sparsity in a latent space. The algorithm achieves compression by modelling the weights as quantized latent representations sampled from a Gaussian prior which are further optimized using an entropy penalty. Such a latent space of quantized model parameters is used for simultaneously learning model compression and sparsity (both unstructured and structured) with continuous weights maintained for sparsity learning. The authors show the efficacy of their algorithm on CIFAR-10/100 and Imagenet. Reviewers unanimously agree that the current manuscript is above the accept line, so is the decision.
justification_for_why_not_higher_score: Three reviewers agree that the paper is marginally above the accept line, while one reviewer likes the latent space modeling quantization idea and thinks it a good paper. So it is a solid poster paper. 
justification_for_why_not_lower_score: The reviewers unanimously suggest to accept the current manuscript, so is the decision.
Decision: Accept: poster