Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an efficient CNN-based neural image compression algorithm. Their efficient CNN framework achieves a high speed while maintaining a high RD performance on par with SoTA works, and also can handle variable RD trade-offs through a single model. They also propose to effectively compress the large teacher model through knowledge distillation using a sparse channel mask technique along with a decaying sparsity regularization. During the mask decay based knowledge distillation, they further apply residual representation learning on encoder for higher scalability.
strength_and_weaknesses: * Strength

Their CNN only framework is more efficient and GPU friendly, compared to recent Transformer based models, while the performance is on par with more complex SoTA neural image compression models.

With an adjustable quantization step, their model handles variable RD trade-offs through a single model unlike previous works.

They successfully enabled knowledge distillation on an image compression model through mask decay with a new sparsity loss that regularizes more effectively than Lp losses.

Their residual representation learning technique enables scalable encoder design with various complexities while maintaining an overall performance. 


* Weakness

More visual results would better present their results, currently none in the main paper.

There are several points they insist as their technical contributions, which are rather scattered thus hard to get a clear picture. It would be better to clarify each more clearly in an overview.

Mask merging can be better clarified within the main paper. Also, does the Fig.4(a)-Pruned and the derivation in Eqs.(5-12) in Appendix B match?

Figure 7 seems to be an important study, but hard to read due to small size.

The intention of discussions on channel importance at p8 is rather confusing, considering their knowledge distillation mainly targets channel reduction, which was successful according to their experiments.
clarity,_quality,_novelty_and_reproducibility: The paper is well written, and conducted rich ablation studies to sufficiently prove an effectiveness of each new building block.

The mask decay based knowledge distillation, the sparsity regularization loss specialized for this KD, and their CNN framework based on resnet and depth-wise convolution based encoder and encoder along with an adjustable quantization control design seem to be novel.

They plan to make the code available in public, so full details of their internal network design that is not described in the paper may also be available.
summary_of_the_review: This paper proposes a pure CNN-based efficient neural image compression model and a knowledge distillation algorithm based on mask decay. The network is more efficient and GPU friendly while maintaining an on-par performance compared to recent Transformer based models. The evaluation and analysis have been made thoroughly enough, proving the effectiveness of the proposed algorithm properly. In conclusion, the proposed work successfully solved a hard challenge on efficiency-performance trade-offs in image compression.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an efficient single-model variable-bit-rate network, which is able to run at 30 FPS with 768x512 input images and still outperforms VVC for RD performance. By further reducing both encoder and decoder complexities, the small model even achieves 30 FPS with 1920x1080 input images. To bridge the performance gap between different capacities models, the authors meticulously design the mask decay, which transforms the large model’s parameters into the small model automatically. And a novel sparsity regularization loss is proposed to mitigate the shortcomings of Lp regularization.
strength_and_weaknesses: Strength: The proposed method is novel and effective.

Weaknesses: The reviewer has some concerns about the experiments:

(a) The authors should compare the RD performance, speed, etc. of the proposed method with the SOTA neural image compression methods, e.g., Xie et al., Enhanced Invertible Encoding for Learned Image Compression. ACM MM 2021; Cheng et al., Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules. CVPR 2020, etc. 

(b) The result of Tecnick should be added to Table 2.

(c) HEVC dataset is a video dataset, which is rarely used for image compression. Did the authors test their image compression methods on all frames of each video in the HEVC dataset? Instead, CLIC is a popular dataset for image compression.

clarity,_quality,_novelty_and_reproducibility: The proposed method is novel and effective. The supplementary provides the details that are helpful for reproducing the proposed method. The authors promised to release codes in the abstract.
summary_of_the_review: The proposed method is novel and effective. The reviewer recommends to accept the paper, however, the authors should address the questions regarding the experiments.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a real-time neural image compression solution. This approach utilizes the knowledge distillation approach to obtain low-complexity codecs through the mask decay design. Furthermore, the paper also introduces the scalable encoder for neural image compression and achieves the dynamic complexity for different latency requirements. Experimental results on several benchmark datasets are provided to demonstrate the superiority of the proposed approach. In summary, the paper focuses on practical neural image compression based on knowledge distillation and slimmable design.
strength_and_weaknesses: **Strengths**

1. The proposed method is useful for practical neural image compression. The Mask decay solution seems to be reasonable and interesting. 
2. The inference speed reported by this paper is encouraging.  And dynamic complexity is an interesting design in practical applications. 

**Weaknesses**

1. Several strong baseline methods[1,2,3] should be discussed and compared. In particular, ELIC[1] method is also proposed for practical image compression and the authors are suggested to make further comparisons. 

     [1] He, Dailan, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang. “ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding.” arXiv, March 29, 2022. http://arxiv.org/abs/2203.10886.

      [2] Xie, Yueqi, Ka Leong Cheng, and Qifeng Chen. “Enhanced Invertible Encoding for Learned Image Compression.” ArXiv:2108.03690 [Cs, Eess], August 8, 2021. http://arxiv.org/abs/2108.03690.

      [3] Kim, Jun-Hyuk, Byeongho Heo, and Jong-Seok Lee. “Joint Global and Local Hierarchical Priors for Learned Image Compression.” In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5982–91. New Orleans, LA, USA: IEEE, 2022. https://doi.org/10.1109/CVPR52688.2022.00590.

2. Using the transformer based NIC methods as baseline methods is not a good choice. Most transformer based approaches suffer from the high complexity cost. 

3. For the proposed mask decay approach, the improvements on the kodak dataset are not significant. It brings 1-3% bitrate savings. 

4. The comparison between the proposed method and SlimCAE is not fair since they use different backbone networks. 

5. The novelty of dynamic complexity is not significant as several approaches like SlimCAE and CBA-NET[4] have been proposed. The authors should make further explanations. 

     [4] Cbanet: Towards complexity and bitrate adaptive deep image compression using a single network. 

6. The training strategy seems a little tricky. For example, how to decide the termination of mask decay. The authors are suggested to provide more details. 




clarity,_quality,_novelty_and_reproducibility: The novelty of this paper is not significant considering the dynamic complexity that has been investigated by existing research.
summary_of_the_review: The paper focuses on practical neural image compression based on the KD technique and dynamic complexity. The selection of baseline methods is not convincing and several strong baseline methods are missing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper explores neural image compression and focuses on improving existing methods in several ways:

1. A single model that allows variable bit rate encodings, variable encode compute, and faster run time. Although previous research has explored these items in isolation, I do not know of a model that incorporates all of them.

2. Better model pruning through a novel sparsity loss ("mask decay"). This loss (applied via weight decay) leads to better rate-distortion (RD) loss than training models with identical architectures from scratch or by using simpler sparsity losses like L1 and L2.

3. Scalable encoding, in terms of computation, is achieved through residual representation learning (RRL), where the encoder is iteratively refined by learning a mapping from the input image to a residual in latent space (Fig 5). At inference time, the encoder can decide how many of the residual encoders to run to trade off runtime for RD performance.


strength_and_weaknesses: The paper is well-motivated, and the claims are empirically evaluated and compared against strong/appropriate baselines.

The mask decay idea, which combines a novel sparsity loss applied via weight decay, addresses a significant problem for neural image compression. Other papers have looked at methods for speeding up neural codecs by reducing the channel depth of conv layers (the primary runtime bottleneck), but the RD penalty is typically quite high. The authors show that mask decay has a much smaller penalty (Fig 1b) and thus provides a new approach to close the runtime gap between learning-based and hand-engineered codecs.

The main weakness of the paper is empirical. The authors should compare against this paper from CVPR 2022:

ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding
Dailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, Yan Wang
https://arxiv.org/abs/2203.10886

This paper explores a combination of channel-wise and spatial decomposition for the entropy model that's different than what's used in the paper under review (primarily, it uses five channel-wise groups with uneven depths). The authors show that this allows them to reduce the computational complexity of the analysis and synthesis transforms and achieve a net benefit in terms of RD performance vs. runtime. In particular, I think the results is both faster and gives a slightly better compression rate than what's presented here (it's hard to be 100% sure without overlaying the RD graphs, and runtime comparisons are difficult since different hardware was used (2080Ti vs Titan Xp)).

Regardless, I think there are enough novel ideas with promising impact on the neural image compression subfield that this paper should still be accepted.
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is clearly written with high quality in terms of the presentation and evaluation. I don't have major concerns about reproducibility, especially since the authors say code will be released (mentioned at the end of the abstract).

I do think clarity could be improved in terms of the scalable encoder. Although the technical approach seems clear from the description and Fig 5, it's not clear to me how it relates to the small/medium/large architectures (discussed at the end of Section 3.1) and mask decay. Are these different methods -- that is, you *either* use RRL or mask decay? Or is mask decay applied on top of RRL as the arrows seem to imply in Fig 1b? If the latter, more detail on how mask decay (a version of student-teacher learning) is applied is needed, e.g. it would seem that the teacher (the "cumbersome" model) would need to learn N residual encoders and then each of those would be masked to yield a smaller student.

Also, I think that where the authors say the method "achieves 30 FPS for the 1920 x 1080 inputs", the results should be given as megapixel/s (mp/s) instead. This is an image compression model so there are no FPS per se, and mp/s (or mb/s though I assume all results here use 8 bits per channel) is the standard unit.
summary_of_the_review: I'm recommending that this paper be accepted based on promising results from the novel sparsity loss and mask decay. This approach can be applied to other models and appears to give much better results than previous pruning methods applied to compression models.

As mentioned above, a comparison to ELIC should be added, and a favorable comparison (runtime and RD) would strengthen my recommendation based on the empirical results.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposed a model for real-time neural image compression that works well for different rate distortion tradeoffs. 

Strength:
The model achieves high compression speeds, high performance, and can handle different rate-distrotion tradeoffs in a single model. In addition, the paper is well written and contains a variety of ablation studies.

Weaknesses: 
Code is not yet available, and a few additional comparisons, e.g., to ELIC, would be useful. Those are not critical, however, the current experiments give a good idea about the merits of the proposed method.
justification_for_why_not_higher_score: This paper provides a solid contribution, but the ideas are not sufficiently novel to warrant a spotlight or oral.
justification_for_why_not_lower_score: The paper proposed a model for real-time neural image compression that works well for different rate-distortion tradeoffs, and contains new elements. The method looks like it is practically useful, and it contains useful ideas for researchers and practitioners interested in learning-based image compression.
Decision: Accept: poster