Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: - The authors propose Code Editing that control bitrate of neural image compression with semi-amortized inference.
- The authors solved the performance decay in low rate of Code Editing by making the quantization step size adaptive.
strengths_and_weaknesses: Strengths
- The authors shows an interesting experimental result (Fig. 1 Left) it seems to solve the performance decay in low rate of Code Editing.

Weakness
- The new part of method has not seem to be clearly explained. The basic parts of amortized inference strategy in neural image compression has already been proposed (Yang et al. [2020]). Therefore, I thought it was a bit exaggerated when section 2.2 states that they propose a new paradigm of controlling R-D trade-off by semi-amortized inference. Variable bitrate compression by changing the quantization step size has also been proposed for neural image compression (Choi et al. [2019]). It would be great if you could clarify what is new when compared to these previous studies.
- The R-D curve of Code Editing Enhanced versus conventional semi-amortized inference based neural image compression (Yang et al. [2020]) has not been compared. If quantitative comparisons with previous studies were made (e.g. overhead to support variable bitrate), it would be easier to argue the superiority of this study.
- typo: p.2 L53 bitrtae -> bitrate
questions: - When comparing the R-D curves of Code Editing Enhanced and Yang et al. [2020], what is the difference of the proposed method and is it reasonable?
- Why does combining semi-amortized inference and adaptive quantization step size improve when low-rate results?
- Why are the low-rate results in Fig. 3 approaching baseline?
limitations: - In Chapter 5, the authors told that efficiency of Code Editing could be improved with regard to limitations, but it would be good to be more specific about what cases the efficiency should be improved. For example, what are the cases in which the authors' method does not work well?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper investigates obtaining flexible-rate neural image compression (NIC) methods by adapting representations during encoding. They obtain a new representation y' by adapting the quantization width as well as the latent values themselves. Parameters are frozen so no overhead except for longer encoding time.
strengths_and_weaknesses: Overall solid idea. I'm not super familiar with the adaptive rate literature but it seems novel to me. I like the idea of adapting $\Delta$. I also appreciate that there are various interesting ablation studies in Fig 1, and that authors investigate using the formulation for perceptual quality optimization and adaptive ROI-based rates (neat!). The results in Fig. 3 are promising.

## Weaknesses

- W1) Bad formulation in Sec 2.1: To me, who is very familiar with this field, Sec 2.1, in particular L72-L84 sounded weird. What is the meaning of "connected". What is it supposed to mean that the "factorized uniform distribution is used to simulate the quantization noise" (I suppose that we add IID uniform random noise?). What about "the decoding process is connected to compute the likelihood ..., which is equivalent to distortion evaluation" (some link to VAE?). This might be a language barrier issue but it sounds to me like some badly pieced together bits from Balle et al's original paper. I would strongly suggest rewriting Sec 2.1. 
- W2) Incorrect/bad formulation in L112: "In NIC, the probability mass function (pmf) [...] over quantized y is computed via differentiating cumulative distribution function" [sic]. That is only true for the factorized prior from Balle et al, which is formulated via a CDF. For e.g. hyperprior based approaches (used in almost every compression paper since), we parameterize the density directly and calculate the PMF by integrating over boxes.
- W3) The above makes me question the soundness of the evaluation. Did the authors use range coding/arithmetic coding in the end to calculate real bitrates? Was care taken to make sure the model does not accidentally cheat?

- W4) (minor) similar to the above, the recap of distortion perception left a bad taste in my mouth. The authors cite Blau and Michaeli's work, where the perceptual loss is formulated via a _divergence_, yet authors use LPIPS, a normal pair-wise distortion, to do perceptual optimization.

## Conclusion

Overall, I will have to give a Reject rating, but if authors can convince me that they i) did a solid evaluation including range coding that implies we can trust the results and that they ii) will make sure the formulation is sound and understandable, I am willing to change my mind, since the approach itself looks interesting.

I'm giving a 4/5 confidence rating since there might be related work to adaptive-rate NIC that I'm missing

questions: Q1) Did you experiment with adapting $\Delta$ during training also? I found it surprising that you can use a model trained for lambda  = 0.015
limitations: No negative societal impact described in the main text, hidden in the supplementary material.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a method called "Code Editing" for rate control, which is an optimization of the latents at encode time. This has commonly been used in the past to boost rate distortion performance, but not as a continuous rate control mechanism. Second, this paper proposes a method of combating the rate-distortion decay of latent optimization with the use of an adaptive quantizations step size. Last, the paper allows for a per pixel spatial rate-allocation mechanism that grants even more fine-grained control of the rate.

Given the authors responses, I have raised my overall review.
strengths_and_weaknesses: Strengths:
Through "code editing", the latents can be additionally optimized for the given rate-distortion trade-off over a large range of bitrates [0.1 to 1.0] bpp. This allows one model, with additional encoding time to target a wide swatch of the rate-distortion curve. 

Per pixel rate control allows a very explicit, fine grained rate control on a per image basis. Additionally, this information is only used at encode time and isn't required to be transmitted to the decoder.

The empirical results of naive code editing only being successful in a narrow rate range (around 0.1 bpp).

Weaknesses:
No results were shown in the very high quality/high rate range. From the rate-distortion curves, it appears as if the Code Editing method starts to flatten at the high rate (see fig 3), versus the ideal range of around 0.5 bpp.

The ROI based coding examples seem very much designed to show high contrast (checkerboard, gradient, NIC). Show the result of a semantic segmentation model's ROI and showing if bit allocation on small text or faces can enhance the overall image at low bitrates would make an obvious visual difference. (Perhaps the ROI shifts between one of several faces in the same image for example).

The quantization step size is currently determined via grid search (brute force).
questions: 1. What is the rate distortion performance at very high rates (>1.0 bpp, say 2.0)? Is code editing still effective at high rates?
2. Is code editing effective with ROI across complex ROI masks? e.g. imagine you have an image with 3 faces roughly the same size, can shifting the ROI to each of the three faces increase fidelity equally? 
3. How important is the granularity of the quantization step size being searched? If the grid were two or four times as dense, would you have increased performance or just more points on the RD curve?
4. Many of the baseline models have increased capacity for the larger rates in order to keep quality high. For example, in the Balle 2018 case, was the model using the 192 or 320 throughout? Are higher capacity models needed to allow for the flexibility needed for code editing?
limitations: Author sufficiently discussed the limitations of their work and any potential negative society impact.
review_title: null

Meta-review: metareview: Thanks for your submission to NeurIPS.  The reviewers are all in agreement that the paper is ready for publication.  They in particular appreciated your rebuttals and changes to the paper, and increased their scores as a result.  The proposed method is novel, interesting, and performs well.
confidence: Certain
Decision: Accept