Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a method to improve the coding efficiency of the baseline variable bitrate model based on adaptive quantization (quantization steps controlled by quality scaling factors) while at the same time reducing entropy coding time, thus one step closer to closing the gap to the fixed bitrate models. Specifically, it proposes to generate 3D importance binary map for the latents to be selectively coded. The 3d map generation is based on side information (decoded hyper latent) and a target bitrate. Element-wise mask without much overhead is the key to bridge the gap to fixed bitrate models.
strengths_and_weaknesses: Strengths
- Almost close the gap between variable bitrate models with separately optimized fixed-bitrate models with element-wise 3D importance map. The RD improvement over the baseline method (Cui 2020) mainly over the lower bitrates is convincing.
- The techniques induces small overhead (a single 1x1 conv, and parameterized importance adjustment curves), can be applied to many existing models, as showed in the paper: hyperprior, mean-scale, context-based.
- The paper is well organized with detailed description of various parts of the whole compression system.

Weaknesses
- Lack of ablation on the importance adjustment, which is one of the main contribution of this paper.
- The presentation on reducing decoding time in Sec 4.2 is a bit misleading. line 295 - "Furthermore, SCRpsnr Hyp294 and SCRpsnr MS showed lower decoding times compared to the original reference compression models over all target quality levels except q = 8..." Fig  9 only shows the decoding time for reference models for q=8 instead of all target quality levels.
- Also the way to show comparison results with BD rate can be more fair when all variable bitrate models are compared to the reference model, instead of like line 248-249.
- The third contribution claim on continuous bitrate through interpolation already exists in Cui 2020?
- Training procedure becomes more complicated and expensive - Three stages, in total 9.4 M steps.
questions: 1. please confirm that even through results in Cui 2020 (e.g. Fig 2) looks like it already closes the gap to fixed rate reference models, according to your evaluation, it is still worse at lower bitrates. As stated in line 87-88?
2.  For your ablation, what is exactly `the SCR variant without the selective compression`?
3.  Do you have ablation on the importance adjustment? e.g. with only adaptive quantization from Cui 2020 + 3d importance map; or quality q as input to 3d map generation besides the second to last features in hyper decoder.
4.  line 126: just to clarify, for mean-scale hyperprior, masked out latent is just set to mean value predicted by hyper decoder, right? same for line 253.
5.  Do you also have decoding time profiling for Context-based models? it is not shown in Sec 4.2
6. Fig 9: 
   * the Y-axis is second?
   * why reshape time increases as rate goes up?
   * why hyper decoder takes comparable/ much longer than decoder network? It should be around/less than 1/10 of the decoder? 
   * 15ms for the decoder (yellow) seems too small for Kodak images. What GPU exactly are you profiling on?
   * any details on entropy coding? any parallel impl?



limitations: - The paper does not mention the *memory* increase due to generation of 3D importance map in encoding / decoding. Should profile this in decoding time section as well.
- I like the study in figure 5. Is it possible to extend to embedded bitstream? i.e. encode an image to a single bitstream, to get lower bitrates, just truncate to a prefix of this stream.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: - The authors propose NN-based variable-rate image compression method that selectively compresses the representations with generated 3D binary mask.
strengths_and_weaknesses: Strengths
- Figure 6 shows that the proposed selective compression scheme appears to improve R-D performance at low rates compared to without it.
- According to Figure 9, the proposed method seems to have an advantage in decoding time by reducing entropy decoding time.
- Figure 3 provides an interesting visual representation of the importance of features.

Weaknesses
- The authors mention that the conventional method [1, 17, 18, 19, 20, 21, 22] requires additional network modules, layers, and inputs, which may result in complex overhead. But the proposed method may also requires additional processing, the authors present experimental results for processing time with respect to decoding, but not for encoding.
- There have been several studies in the past to generate importance maps for representations in deep image compression [a, b]. These are not cited in the current paper, but comparison with these would be good. Also, since the supp. material in [b] mentions the application of the importance map to multiple compression rates, I think it would be good to clarify the novel point and superiority of the proposal.
- It would be great to make direct experimental comparison of R-D curve or B-D rate with previous studies of variable-rate compression in deep image compression [1, 17, 18, 19, 23, 20, 21, 24, 22] with the authors' proposed method and demonstrate superiority.

[a] M. Li, et al. "Learning Convolutional Networks for Content-weighted Image Compression", CVPR 2018

[b] F. Mentzer, et al. "Conditional Probability Models for Deep Image Compression", CVPR 2018
questions: - Is it possible to show the encoding time and its computing overhead when the proposed method is applied?
- Figure 6 shows that both SCR (w/o selective compression) and SCR (Full) above 0.6 bpp and below 1.2 bpp have slightly better R-D performance than baseline [6], and above 1.5 bpp, the R-D performance appears to be slightly worse than baseline [6]. Similar characteristics are exhibited for the baseline [7]. Why this characteristic?
- As a direct comparison with previous work on variable-rate compression, is it possible to experimentally compare the R-D curve with previous work (e.g. [19][23])?
limitations: - In practice, I believe there are many different neural image compression network architectures. I thought it would be good to discuss the architectures to which this method cannot be applied (e.g., without using hyperprior) and future works.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Currently used variable-rate image compression model usually adds additional modules to the base model which increases the complexity while those using adaptive quantization to achieve variable-rate compression face degradation in coding efficiency. To address this problem, this paper introduces selective compression of representations (SCR), which embeds selective compression into variable-rate models based on adaptive compression. The selective compression mechanism first generates a 3D importance map according to the output of the hyper decoder and then adjusts it using the channel-wise parameter , and finally, it is binarized to indicate whether the element is selected for compression. Also, this paper introduces a interpolation-based mothed to support continously variable-rate compression when the quality level q is a value between two discrete target quality levels.

strengths_and_weaknesses: Strengths: 
1. According to the experiment, the proposed selective compression mechanism can distinctly increase the compression efficiency when using "Hyperpiror [1]" and "Mean-scale [2]" as the base model. Also, because the proposed method only performs entropy coding for some of the elements, the decoding time decreases, especially at low bitrate. The proposed method only uses a single 1*1 convolutional layer to generate the 3D importance map and the parameter has a limited number of parameters, so it adds few parameters to the base model.
Weaknesses: 
1. This article argues that variable-rate compression methods like [3] may increase the compression complexity by adding additional modules to the base model, so the author chooses to implement selective compression on the adaptive-quantization-based method. Also, the author chooses the model mentioned in [1] and [2] as the base model. However, the overhead complexity of the added module may not be that high compared to the base model itself. Also, it lacks a comparison between the proposed method and other variable-rate models which add modules to the base model in compress efficiency and complexity.
2. According to Appendix A.3, the selective compression method has little impact when implemented on "Context [2]". While the state-of-the-art learned image codecs usually adopt context model in their entropy engine, the proposed method may be unable to increase the compression efficiency when using the state-of-the-art codec like [4] as the base model.



[1] J. Ballé, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston, “Variational image compression with a scale hyperprior,” in the 6th Int. Conf. on Learning Representations, 2018.
[2] D. Minnen, J. Ballé, and G. Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” in Advances in Neural Information Processing Systems, May 2018.
[3] Y. Choi, M. El-Khamy, and J. Lee, “Variable rate deep image compression with a conditional autoencoder,”  in International Conference on Computer Vision, 2019.
[4] Cheng, Zhengxue, et al. "Learned image compression with discretized gaussian mixture likelihoods and attention modules." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.


questions: The two base models shown in the experiment share similar encoder/decoder structure and both use hyperpiror as the entropy model, the author should include more method with a greater diversity of structures in their experiments.
limitations: The proposed method may not be generalized to learned image compression with context adaptive entropy engine.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes a variable rate image compression method using selective compression and adaptive quantization. It is shown that the RD performance is similar to that of the reference models and decoding time can be reduced.
strengths_and_weaknesses: *** Strength

+ An important task, variable-rate image compression, is addressed.

+ The paper is written relatively clearly.

+ From the experiments, the method seems to work well.

*** Weakness

- The proposed method is composed of adaptive quantization and importance map-based selective compression.   First, it is said that the adaptive quantization is similar to [19,23]. It would be better to explain more concretely how similar/different the adaptive quantization in this paper is from those in [19,23]. Second, the idea of 3D importance map already exists [Li'18], although it was not used for variable rate compression. Furthermore, the added value of the selective compression seems to be limited, especially for advanced base models like the context model [7]. In fact, it will be necessary to show the effectiveness of the proposed method on more recent models like [11, 13, He'21, Gao'21, Qian'21].


[Li'18] Li et al., "Learning convolutional networks for content-weighted image compression," CVPR 2018

[He'21] He et al., "Checkerboard context model for efficient learned image compression," CVPR 2021

[Gao'21] Gao et al., "Neural image compression via attentional multi-scale back projection and frequency decomposition," ICCV 2021

[Qian'21] Qian et al., "Learning accurate entropy model with global reference for image compression," ICLR 2021

- Performance comparison with existing variable rate compression methods is missing. The proposed method needs to be compared against different ways to accomplish variable rate encoding in the previous studies.

- It is quite difficult to follow the quantitative BD-rate gain/loss results given in the text in Section 4.1. It is suggested to summarize them as a table.
questions: - Are QV_q and IQV_q trained together with the model?

- Is there any performance gain if the network for 3D importance map generation is more complex than a single 1x1 conv layer? Actually, [Li'18] used such a complex model.

- Just out of curiosity, what happens if q<1 or q>8?

- It is quite surprising that even if q=8, only 55.81% of the features are used. This means that about a half of the extracted features are useless anyway. Some discussion on this could be interesting. There are some studies on pruning of image compression models, which could be supported by this observation.

- In Fig. 6, the difference between SCR(Full) and SCR(w/o selective compression) becomes larger at lower bpp values. Some discussion on why this happens would be helpful.

- In Fig. 5, not all features for a value of q are used for larger values of q. This is unclear. Since the adjustment curves are monotonic, it has to be that increasing q only adds additional feature elements for encoding so the reuse rate is always 100%.

- (minor) Ripple [20] -> Rippel [20]

- (minor) Does "BD-rate gain of -3.21%" mean a gain by 3.21%? or a negative gain (i.e., loss) by 3.21%? I assume the former, but it's confusing.
limitations: Limitations are discussed in the paper.
review_title: null

Meta-review: metareview: Thanks for your submission to NeurIPS.

Initially, this paper was leaning reject, with three negative reviewers who had various concerns.  The rebuttal really helped a lot, and two of the negative reviewers raised their scores based on the rebuttal, leading to 3 of 4 accept scores.  The final reviewer also mentioned in a comment that they would raise their score to a borderline accept, but never updated the official score.  I also took a look at the reviews and rebuttal, and it seems that the major concerns have indeed been addressed.

Given all of this, I am happy to recommend acceptance of the paper at this point.
confidence: Certain
Decision: Accept