Flexible Neural Image Compression via Code Editing
Authors: Chenjian Gao
Tongda Xu
Dailan He
Yan Wang
Hongwei Qin
Venue: NeurIPS 2022 Accept (2022)

Abstract: Neural image compression (NIC) has outperformed traditional image codecs in rate-distortion (R-D) performance. However, it usually requires a dedicated encoder-decoder pair for each point on R-D curve, which greatly hinders its practical deployment. While some recent works have enabled bitrate control via conditional coding, they impose strong prior during training and provide limited flexibility. In this paper we propose Code Editing, a highly flexible coding method for NIC based on semi-amortized inference and adaptive quantization. Our work is a new paradigm for variable bitrate NIC, and experimental results show that our method surpasses existing variable-rate methods. Furthermore, our approach is so flexible that it can also achieves ROI coding and multi-distortion trade-off with a single decoder. Our approach is compatible to all NIC methods with differentiable decoder NIC, and it can be even directly adopted on existing pre-trained models.

Flexible Neural Image Compression via Code Editing
Chenjian Gao"l, Tongda Xu* Dailan He', Hongwei Qin', Yan Wang
TimeseTimee Research, Instittte for Al Industry Research (AIR), Tsinghua University (xutongda, http///airrssiinaa...aaaaa edu. cn, {gaochenjian, hedailan, qinhongwei BeahongveiJdsennseiimee.. @@sensettme.ccmm
Abstract
Neural image compression (NIC) has outperformed traditional image codecs in rate- distortion (R-D) performance. However, it usually requires a dedicated encoder- decoder pair for each point on R-D curve, which greatly hinders its practical deployment. While some recent works have enabled bitrate control via conditional coding, they impose strong prior during training and provide limited flexibility. In this paper we propose Code Editing, a highly flexible coding method for NIC based on semi-amortized inference and adaptive quantization. Our work a new paradigm for variable bitrate NIC, and experimental results show that our method surpasses existing variable-rate methods. Furthermore, our approach is so flexible that it can also achieves ROI coding and multi-distortion trade-off with single decoder. Our approach compatible to NIC methods with differentiable decoder NIC, and it can be even directly adopted existing pre-truc ree-trinne models.
1 Introduction
Lossy image compression is a fundamental problem of computer vision. In simplified setting, the aim of lossy compression is to minimize rate-distortion cost R + XD, with X as the Lagrange multiplier controlling R-D trade-off. Traditional compression algorithms (e.g. JPEG, JPEG2000 and BPG [Bellard 2018) ) first perform a linear transformation (e.g. DCT, wavelet transform) and then quantize the transformed coefficients for compression. Their bitrate control is achieved by controlling quantization stepsize. On the other hand, lossy neural image compression (NIC) methods have outperformed traditional codecs recent years (Xie et al.) [021] He et al., 2022] These methods implement powerful non-linear transformation with neural network, and therefore achieve better R-D performance. However, in order to realize R-D trade-offs, we need to optimize multiple networks towards different losses with different Às, which means that we cannot achieve flexible bitrate control.
To enable NIC models with bitrate control ability, several works adopt conditional networks [Choi let al. 2019 Cui et al., 2021 Sun et al. 2021] They use À as condition and feed it into the encoder and decoder. During training, they sample À from a pre-defined prior and optimize the Ex [R+ X via Monte Carlo. Due to the capacity of conditional networks, most of them use simple prior and limited sets of X, such as a uniform distribution over only 5 10 discrete As CCoi et al., 2019 Cui et al. 2021 And additional effort is required to interpolate the discrete Äs to achieve continuous
R-D curve Choi al. 2019 Cui al. 202T, [Sun al. 202T), which limits the flexibility of variable bitrate model. On the other hand, traditional image codecs achieve flexible R-D trade-off naturally by
controlling the quantization step. This control is fine grain (e.g. 100 levels for JPEG). Moreover, they support flexible spatial bits-allocation for ROI-based (region of interest) coding. And the user need only one decoder for all those flexibility.
"Equal contributions.
i Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In this work, we intend to give neural image compression methods the flexibility of traditional codecs: the continuous -ute-distortion trade-off with arbitrary spatial bits-allocation by one decoder. It is meaningful task for practical NIC considering the growing size of neural decoders and the need for fine-grain rate control. In order to achieve this, we propose Code Editing, a semi-amortized inference [Kim et al. 20188] based approach for flexible R-D trade-off. Although previous works have adopted aanttizort inference to improve R-D performance [Johnston et al., 2018 Yang et al.) 2020]. we are the first to discover its potential for flexible bitrate control. Specifically, we edit the latent code directly to change the optimization target at inference time based on desired As and ROIs. However, we find that naive Code Editing has limited bitrate control ability. We further analyze and address the problem via quantization step size adaptation, and enhance its speed- sereedd-rformmncce trade-off. Surprisingly, we find that our Code Editing can achieve not only flexible continuous bitrate control and ROI-based coding, but also multi-distortion trade-off, all with one decoder.
To wrap up, our contributions are follows:
We propose Code Editing, new paradigm for variable bitrate NIC based on semi-an amortized inference. It supports continuous bitrate control with single decoder and outperforms previous variable bitrate NIC methods. To the best of knowledge, we are the first to consider semi-amortized inference for variable bitrate NIC.
We resolve the performance decay of Code Editing by combining it with adaptive quantiza- tion step size. And we further improve the peeedd-erffrmance trade-off of Code Editing.
We demonstrate the potential of Code Editing in flexible spatial bits-allocation and multi- distortion trade-off, which is beyond the flexibility of traditional codecs.
2 Code Editing: Flexible Neural Image Compression
2.1
Background
The general optimization procedure of NIC methods can be concluded as follows: 1) given an image æ, an encoder (inference model) produces latent parameter y = (x)) 2) Then, we obtain [y] by unit scalar quantization. 3) Then, an entropy model (prior) pa py( parameterized by 0 is used to compute
the probability mass function (pmf) Po(Ly))
1+0.5 II [ 1 (dy
[I(FF(L>>)) +
0.5) (L) (0) + 0.5)), where Fg is the edf of po. 4) With that pmf, we encode [y]] with bitrate
R = log Pa (Lv]) by entropy coder. 5) For the decoding process, Ly] is feed into decoder network to obtain reconstruction image ã ge (0( (Lv7)) 1). and d(.,,) - is used to compute the distortion between reconstruction and original image. The distortion D can be interpreted as the likelihood treat distortion as energy of Gibbs distribution [Minnen et al. 2018). For
log p 1) so long
example, when d(. .) is pixel-wise MSE we can interpret d(x;, σ) = - log p( y ) constant, where p(zl yl) N 1/2AI). 6) Finally, the encoder parameter o, decoder and entropy model parameter are optimized to minimize the R-D cost R >D, where R = - log Pa (all and
D = d(ær, (o0((v )). This procedure can be described by Eq. [ and Eq. 2 As the rounding operation 1.1 is non-differentiable the majority works of NIC adopt additive uniform noise (AUN) to relax it
Ballé al. 2017
2018
Cheng al. 2020
0-, 0- arg max Co.6
020
C = - AD) = Ep(r)) log Po(Lzi) -d((x, ge (Lx1)))
(2)
rate
discoction
With the AUN relaxed latent code ğ y + L4 0.5), the encoding and decoding of NIC can be formulated as a Variational Autoencoder (VAE) (Kingma and Welling 2013) on probabilistic graphic model ÿ + x, where ÿ are continuous relaxed latent codes. The prior likelihood log po (ū) of such VAE is a continuous relaxation of log Po v1). The data likelihood log pe p=( (zlğ) is the distortion offseted by a constant. Moreover, the AUN relaxed latent code ÿ can be interpreted as the reparameterized sample through a factorized uniform posterior go (o (z) U(yy 0.5, y + 0.5).
Under such formulation, the evidence lower bound (ELBO) is directly connected to the negative R-D cost of compression (Eq. 3). Then, minimizing R + XD is relaxed into maximizing Ctrain.

---

C = Ep(a)) E (5)(r) log Po (1)
log pe Pa(a: (
log qul (ū
distortion
The above process of encoding is also known fully amortized variational inference. for each new image x', its variational posterior is fully determined by amortized paramters p. Those parameters called amortized parameters they are global parameters shared by all data points.
2.2 Code Editing Naïve
Different from previous methods in variable bitrate NIC, we propose a new paradigm of controlling R-D trade-off by semi-amortized inference, named Code Editing. Consider now we have pair of parameters xx Φxo optimized for a specific R-D trade-off parameter λo- Then, for new Lagrangian multiplier A1 encode by editing the code. To be specific, given image æ, we first initialize the unquantized latent parameters y f f0 (æx). Note that now y is exactly the same as the latent with
R-D trade-off ÅØ. Next, we iteratively optimize y to maximize the target by as Eq. [ In other words, we directly edit the code y to adapt to different R-D trade-offs. During the optimization process, the decoder and entropy parameter 02g remains the same. This process of encoding is known as semi-amortized inference [Kim et al. 2018), as it utilizes the initial value of amortized inference, and optimizes latent for each data point afterwards.
y arg max by y
(4)
-R= log Pox, (ly1) dz 90%0 (ly]))
(5)
Similar to NIC, one challenge remains is that the rounding operation L-1 is not-differentiable. not possible to adopt gradient based optimization approach directly. Learning from Yang al. [2020). we adopt stochastic gumbel anncaling (SGA) in lieu of rounding to obtain a surrogate ELBO. As shown in Sec. 4.2. adopting SGA here achieves better R-D performance than AUN. After that,
use standard gradient based optimization methods with temperature annealing to optimize latent And call this method Code Editing Naïve.
2.3 Code Editing Enhanced
Empirically, we find that Code Editing Naive can only achieve continuous R-D trade-off within narrow range (c.g. =0.1 bpp). Out of this range, the R-D performance falls rapidly (See See. 4.2
One possible reason is the train-test mismatch of the entropy model pex, (y). Naturally, (y grows sparser bitrate decreases. And this causes obvious gap between latent distribution optimized for λo and A (See Fig. 2). Thus, the entropy model fitted to latent distribution at X might perform poorly in estimating latent density of other λs. More specifically, the mismatched bitrate Ep(=) [E (m) [log Po., (1 - log Pos, (lv" [I]]- Under the assumption that the variational posterior is the true posterior, this mismatch equal to DKL(Pos, (ly" Poxo (Lv*1)), which is just the distance between distributions.
In NIC, the probability mass function (pmf) Po((y) over quantized y is computed by taking the difference of cumulative distribution function. In other words, Pos, (Ly) = [I (1 +
Fox (Ly (0) 0.5)). Following Choi et al. 2019 we augmenting it with the quantization step A, and this results in Poso (Ly/A]; A) II( ([((Fo Fo., (A ( [AA] + A/2) - Fox, (A [AA] A/2)), where AL-/A] means the dequantized result of quantization with step A. Controlling A results in significantly different pmf without changing the underlying continuous density pex, (y). Then,
can optimize the A augmented ELBO to find the optimal code y" and quantization step size A as
Eq. and Eq. 7
リ" A = arg max L.y.A
(6)
A.A
Cy. - A) log Poro (1 A) Aid(o, 90,, (Aly/A1))
(7)
3

---

In fact, similar idea of controlling A has been adopted in Choi et al. (019). It implements continuous R-D trade-off within small bitrate range (e.g. = bpp) by controlling A. However, we show that a large range of continuous R-D trade-off (e.g. 0.1 1.1 bpp) can be achieved by combining the quantization step size control and semi-amortized inference.
2.4 Speed Performaee Trade-off
Optimizing y over Eq. 7 from (2r) for each x is time-consuming, Naturally, we can accelerate Code Editing by reducing the iteration of optimization, and the R-D performance would drop accordingly. In practice, we find that reducing the number of iterations to 10% of full convergence achieves reasonable results for low bitrate regions (with 0.4 db PSNR drop). However, the R-D performance in the high bitrate range degrades severely. We propose to enhance the amortized encoder by finetuning at high bitrate R-D objective. Then the finetuned encoder can predict better initial y for high bitrate range, and thus makes computationally scalable Code Editing possible.
2.5
Extension of Code Editing
With code editing, bitrate control. By extending the optimization target for code editing, we can achieve different coding requirements. In this work, we explore the potential of code editing in ROI-based coding and multi-distortion trade-off.
empower NIC with more flexible coding capabilities out of continuous
ROI-based Coding. The need of ROI-based coding stems from the fact that different pixels in an image have different levels of importance. So that spatially different bitrate should be assigned according to regions of interest (ROI) during compression. When performing ROI-based coding, quality map m E IR H,W is provided to the encoder addition to the input image æ, which indicates bounded continuous aantinuussvvlled quality map within [0, 1],
the importance of each pixel. Here where 0 and 1 represent the least and the most important pixel respectively. To make Code Editing support
coding, extent the objective in Eq. 7 Specifically, we weight the per- per-pixel distortion with ROI map m to obtain the optimization target CROI where o represents element-wise product.
C y. = log Poro (L//A]:A) X m m d(x 96 (ALv/AT))
ROI weighted distortion
Note that different from existing ROI-based coding works for NIC (Song et al. 2021 our approach imposes no prior on the shape of ROI during training. And thus our ROI control more flexible (See results in Sec. 4.5
Multi-Distortion Trade-off. When compressing image, sometimes want to optimize the MSE, sometimes we want to optimize the perceptual loss, and sometimes we want a balanced trade-off between them. However, the different distortion metrics are in odds to each other [Blau and Michaeli 2018 2019]. For conventional NIC, we can add multiple distortion term in Eq. to optimize 8, ø for specific weights between rate, and different distortions. However, just like R-D trade-off, multi-distortion trade-off also requires multiple, or even infinite number of decoders to achieve.
Again, our Code Editing can achieve the multi-distortion trade-off by extending the objective in Eq. For example, we we want to balance the distortion and perceptual quality, we optimize towards R + DaDa + ApD where Da da 90 (A y ) is MSE, Dp dz (II 99>, (A y A1) is the LPIPS [Zhang et al. 2018 and Ad, Ap are parameters controlling the trade-off. The extended
target is as Eq. 9 Similarly, we can achieve this target by optimizing y and A to maximize LMD
MD = log PoNo (1v/A]:A) - dad go,, ( (A -Ap dp 10r0 (A/vAl)
(9)
distortion loss
perceptual loss
Moreover, it theoretically possible to achieve a trade-off between multiple loss metrics by extending the target as Eq. 10 For example, we can let di be the MSE, da be the VGG loss, dy be the style loss. Then, we can optimize our latent code to achieve the sophisticated photo-realistic effect described in Sajjadi et al. (2017) And it is flexible that do not even need to know what those loss functions

---

are during training. However, we have not conducted empirical study on Ly as it is a much less common scenario in practice.
CMUL L = log Poro (Ly/A1:4) V.a
Σ 90, g0sp (Aly/AI)
(10)
2.6 Hierarchical Latent Case
Most of the sota NIC methods [Minnen al. 2018 Minnen and Singh 2020 Cheng et al. 2020 He
et al. 2022] are based on the hierarchical latent framework proposed by Ballé et al. 2018). which
has graphic model Σ + y + æ. In this framework, y. are relaxed latent code and æ is image. To simplify notation, in this section we base our analysis upon 1 level latent framework by Ballé al. 2017] with graphic model r - æ. We note that all the formulas in this section can be easily extended to the hierarchical latent framework. We also note that for hierarchical latent case, the y quantization stepsize Ay is directly optimized by gradient descent, and the z's quantization stepsize A, is optimized by grid search (See Appendix A.2 and A.3)
3 Related Works
3.1 Variable Bitrate and ROI-based control for NIC
The pioneers of NIC (Toderici et al. 2016 2017 Johnston et al. 2018 achieve coarse rate control by
growing the number of residuals. On the other hand. [heis et al. [011]] learns the scaling vector of [and.
latent code for each bitrate to control the bitrate. However, their R-D performance is surpassed by
[2017] : Choi al. (2019), Cui et al. 2021], Sun et al. [202]] uses the one-hot encoded À as conditional input to the auto-encoder where À sampled from uniform discrete prior during training. And additional efforts are required
Ballé et al. [2017]. More recently, based the framework by Ballé al.
to interpolate the R-D curve for continuous rate control. Cui et al. [2021] improves Theis et al.
[2017) with asymmetrically scaled vectors. More recently, [Song al. [202T] inserts Spatial Feature Transform (SFT) module into the auto-encoder to modulate the intermediate feature. However, to the best of knowledge, we are the first to explore
inference based variable bitrate model.
For ROI-based control, Agustsson et al. [209]] distinguishes images into important and unimportant regions, and use GAN to generate the unimportant regions. Cai et al. by applying different losses to ROI and non-ROI regions. [Song et al. the intermediate features and recover the ROI implicitly from the hyperprior, and it claims that it support flexible ROI control. However, it requires complicated handcrafted ROI prior during training. Compared with dhove-mentioned methods, our Code Editing does require any prior of ROI, nor data contains segmentation map during training.
2020 supports bit allocation
2021 uses ROI to modulate
After the submission of this manuscript, Shi al. 2022 Fathima al. [023 also achieve pixel level variable rate coding for neural codec. On the other hand, Agustsson et al. [002)] also implement promising perception- distortion penception-distootion trade-off with conditional decoder.
3.2 Semi-amortized Inference for NIC
The semi-amortized variational inference is concurrently invented by Kim et al. [2018 and Marino al. [2018]. The basic idea is that works following Kingma and Welling [2013] lose the good tradition of learning variational posterior parameters per datapoint. Instead they learn fully amortized
parameters for the whole dataset. And this fully amortized inference might lcad to sub-optimal posterior distribution. Cremer et al. [2018] refers this phenomena as amortization gap and argues it is a major contributor to inference sub-optimality of VAE. The semi-amortized inference is to initialize the variational posterior's parameter with fully amortized parameters, and conduct stochastic optimization per data-point.
Although Kim et al. (018)) Marino et al. (2018) are alluring to apply. they require nested iterative optimization within the stochastic gradient descent loop. When adopting semi-amortized inference to NIC, Djelouah and Schroers [2019) Yang et al. (2020) simplify it into two stages: 1) training fully amortized encoder-decoder pair 2) iteratively optimize latent which is initialized by amortized encoder. Although this simplification can lead to sub-optimal generative model (recall that now
5

---

the generative model is paired with sub-optimal inference model), it indeed makes amiitimrtizee inference practical for NIC. Our method follows] Yang et al. [020] as we explore inference for R-D trade-off instead of R-D performance.
4 Experimental Results
4.1 Experimental Settings
Following He et al. 2021 we train the baseline models on subset of 8,000 images from ImageNet. For training, we use MSE as distortion. For each baseline method, we train models with 7 fixed R-D trade-off points, with À f0.0016, 0.0032, 0.0075, 0.015, 0.03, 0.045, 0.08). This setup is aligned with Cheng et al. 2020]. The baseline models' bitrate ranges from 0.1 to 1.0 bpp on Kodak dataset [Kodak| 1993]. All baseline models are trained using the Adam optimizer for 2000 epochs. Batchsize is set to and the initial learning rate is set to 10-4, For all the results reported in the R-D curve, the bitrate is measured by actual bits of range encoder, and the reconstruction is computed by the latent coded from the actual the range decoder. For all visualization involves spatial bitrate distribution, the theoretical bitrate is used. More detailed setup can be found in Appendix C
4.2 Ablation Study
We set Ballé et al. 2018 as the baseline of the ablation study. All experiments involving Code Editing are based the model trained with Å 0.015. In ablation study, all the results are tested on Kodak dataset.
Adaptive A. To reduce the prior mismatch, we propose to control the quantization step A in addition to optimizing y. To verify its effectiveness, we compare Code Editing Enhnanced (w/ adaptive A) with Code Editing Naïve (w/o adaptive A). As shown in Fig. [1 the R-D performance of Code Editing Naïve drops evidently when the bitrate changes out of a small range (+ 0.1 bpp). Further, we provide statisties of normalized dequantized results of quantized ( before and after Code Editing on the Kodak dataset in Fig. 2 It is shown that the distribution mismatch of Code Editing Naïve is more evident than Code Editing Enhanced. On the other hand, we also show the result of controlling alone. As Choi et al. (209)), we find that adjusting A can shift the bitrate in limited range (= 0.1 bpp). But out of this range, the R-D performance drops rapidly.
SGA vs. AUN. In Code Editing, we adopt SGA instead of additive uniform noise (AUN) as relaxation of discrete latent. To verify the effect of SGA, we optimize y for 2,000 iterations using SGA and AUN, respectively. As shown in Fig. [ although Code Editing with AUN also outperforms the baseline, the SGA's R-D performance significantly higher than AUN especially in high bitrate region. This is due to the fact that SGA closes the discretization gap (ang et al. 2020 via soft to hard annealing.
Encoder Fine-tuning for Fast Inference. We show that we can control the computational complexity of Code Editing by early termination. We terminate Code Editing with (50, 100, 200} iterations, which correspond to (2.5%. 5%, 10%} of full iterations. The results on the Kodak dataset are shown in Fig. In low bitrate range, the R-D performance achieves a reasonable point within 10% of iterations. However, the R-D performance in the high bitrate region degrades severely. To solve this problem, we finetune encoder with À 0.052 under A, 0.5. The latent code predicted by the finetuned encoder is used as the initial value for Code Editing in the high bitrate range. And stronger encoder achieves better R-D performance with very few iterations (marked with "Encoder FT." in the Fig.
4.3 Variable Rate Coding
First, we evaluate Code Editing Enhanced for continuous variable bitrate coding. We choose three
established works [Ballé et al. 2018 Minnen et al. 2018 Cheng et al. 2020] as baselines. For
each of the baselines, set base model's R-D trade-ofi parameter λ 0.015, which results in
mid-range bitrate (around 0.5 bpp). For each image æ to be compressed, we initialize y + Sox, (x)
and optimize for 2,000 iterations using SGA, with the learning rate of 5 x 10-3, This setting is aligned with Yang et al. 2220). A, directly optimized by gradient descent and A, is optimized by grid search. Fig. shows the R-D performance of variable bitrate via Code Editing, It can be seen

---

40
40
38
38
36
36
PSNR(dB)
34
PSNR
34
32
Boseinqjiaale
32
30
Code Batinng Entanced
5 Baadeed [T)
Code Editing Naive
Editing
30
30
53 (Toode FT)
say (radde FTD
10el
am (roode FTb
Adjustment
20
0.2
0.5 Bitrate(bits per pixel)
0.9
1.2
0.2
1.2 Bitrate(bits per pixel)
0.6 0.9
Figure 1: Results of ablation study. Left: Results of adaptive A and SGA. Right: Results of speed- performance trade-off. A adjustment is used for "O iters".
Code Editing Naive
Code Editing Enhanced
Normalized Lv/1
[AM)
1.00
1.00
Normalized [y/low]
Normalized à "i
0.98
0.98
0.96
0.96
0.94
0.94
0.04
0.04
-2
-2
Figure 2: The distribution of normalized dequantized results of quantized ( before and after Code Editing. The normalization parameter is o² from p(y z). Left: Code Editing Naïve. Right: Code Editing Enhanced. The source λu 0.015, the target À 0.0016.
that Code Editing Enhanced can effectively achieve wide range continuous R-D trade-off with fixed decoder and entropy model trained An-
Then, we compared Code Editing Enhanced with the sota variable code-rate methods Theis et al. 2017 Cui et al. 2021 Song al. 202T]. For fairness, re-implement these methods the baselines with the same setting as original papers. The results are shown in Fig. 3 Note that we only show the results SSng et al. [202] on Ballé et al. [2018) as it produces NaNs other models. As shown in Fig. 3 Our proposed Code Editing outperforms existing variable bitrate methods with
3W
38
36
36
(SNNWC (B))
14
32
32
30
Cods
Beuls dh.ir 201]
Bocla 2010
30
[io ax.
28
28
0.7 Bitratethii per pixel)
Biraesiiis per pixel)
Bitnassiia per pisel)
Figure 3: Results of Code Editing Enhanced Kodak dataset. Left: Ballé et al. 2018 baseline. Middle: Minnen et al. 2018 as baseline. Right: Cheng et al. 2020 as baseline.

---

all three baseline. Theis et al. 20117 brings larger degradation in R-D performance, especially in the high bitrate region. Cui et al. [202]] achieves similar R-D performance as the original fix-bitrate model. The R-D performance of [Song et al. [202]] on Ballé et al. (2018) exceeds the fix-bitrate model the low bitrate range but is inferior Code Editing Enhanced.
4.4 Multi-Distortion Trade-off'
Next, we explore the potential of Code Editing Enhanced in multi-distortion trade-off. For exper- iments, we choose to balance between MSE and LPIPS [Zhang al.) 2018]. MSE adopted to measure fidelity, while LPIPS is widely adopted for perceptual quality [Mentzer et al. [0200 Bhat
al., 20eT].The lower the LPIPS is, the better perceptual quality an image has. In our experiment,
we select 3 trade-off points by controlling the LPIPS weight Xp (0.1,0..5 1.0]. We use the bitrate control scheme in Mentzer al. (2220) to stabilize bitrate (See Appendix C for detail). Fig. + shows the multi-distortion trade-off results based on Ballé et al. [20T8] on the Kodak dataset. As Xp increases, both LPIPS and PSNR decrease, implying better perceptual quality and worse distortion is achieved. Further,
show qualitative results based on Ballé ct al. [2018] on the CLIC2022 dataset [CLIC] 2022] in Fig. 5 We can see that as Ap increases, the detail level of image increases, which indicates a better perceptual quality, Both quantitative and qualitative results show that our Code Editing can achieve flexible multi-distortion trade-off.
0.4
0.25
30
0.3
0.3
LIVPS
LINES
0.2
0.2
0.3 Bittte/bits per pisel)
0.2
0.2
0.3
0.4
25
30
Birntebis per pinel)
PSNRLEB)
Figure 4: The quantitative multi-distortion trade-off results. Left: Bitrate-PSNR with different Aps. Middle: Bitrate-LPIPS with different XpS. Right: PSNR-LPIPS with different X,s, the bitrate are approximately the same.
4.5 ROs-bbaed Coding
This section shows the performance of Code Editing in terms of ROI-based coding. In order to verify the effectiveness of spatially adaptive R-D trade-off, we use various quality maps. We use Ballé et al. [2018) the baseline model to verify the effectiveness of Code Editing in ROI-based coding on Kodak and CLIC2022 datasets. Fig. Gillustrates these results. And our Code Editing can allocate bitrate spatially according
urbitrary-shape ROI maps, including checkerboard and alphabet shapes. Moreover, we compared our Code Editing ROI results with Song et al.) [202T] which is trained with sophisticated prior in Fig. 7 It can be seen that [Song et al. [2021]'s result suffers from ROI map diminishing issue. We can barely sense the effect of ROI map inside the region indicated by the white rectangle. In Appendix B.2 we provide more ROI results on different types of masks, including segmentation mask.
4.6 Gap to SAVI with Multiple Models
We present the results of direct performing semi-amortized inference on baseline models on Kodak dataset. To be specific, we directly optimize using SGA based on multiple models, which is equivalent Yang et al. [2200) without bits-back coding. From Fig. g we can see that when baselines are Ballé et al. 20188) Minnen et al. [2018). our Code Editing Enhanced is comparable to SGA based on multiple models. However, when baseline Cheng et al. (2020), our Code Editing Enhanced indeed suffers from R-D performance decay in high bitrate region. We use single decoder to achieve this while Yang et al. [2020] require multiple decoder. We are neither superior nor inferior to Yang et al. [2020), as the task is very different.
8

---

(1.b). Ap = 0.1
(1.c).2g = 0.5
(1.d). 2p = 1.0
0.1 0.1955 hpp
Bierate: bpp
Bitrate: hpp
PSNR 25.29 d8
PSNR: 25.16 d8
PSNR: 24.71 d8 LPIPS:0.2448
(1a)) Origina Image
LPIPS: 0.3968
LPIPS: 0.2978
(2.b). Ap 0.1
(2.c). 0.5
(2.d). 1.0
Bitrate: 0.1575 bpp
Ritrate: 0.1600 hpp
Bitrate: 0.1586 bpp
PSNR: 25.25 dB
PSNR: 25.10 d8
24.67
(2a). Orgiiaa Image
LPIPS: 0.5364
LPIPS: 0.4358
LPIPS: 0.4054
Figure 5: The qualitative multi-distoriie trade-off results. (x.a) is the original image, and (x.b)-(x.d) are reconstruction results of three images with different multi-distortion trade-off Ap.
0.81
0.00
0.36
0.00
0:78
NIC
0.00
Original Image
b. ROI Map. black=0 (least bits), white=1 (most bits)
Reconstructed Image
A ctual bits allocation, the unit is bits per pixel (bpp)
Figure 6: Results of Code Editing Enhanced's ROI-based coding.
Note that the major difference between our work and Yang et al. 2020 is that our approach requires only one decoder for continuous bitrate control, ROI and perception-distortion trade-off. And Yang et al. [2020) require multiple decoders for them. Yang et al. [2020) adopt Kim et al. [2018] to improve the R-D performance of pair of encoder-decoder. We find SAVI can also be adopted to achieve bitrate control, ROI and multi-distortion with only a single decoder. In fact, even without the SGA of Yang al. (2020). the semi-amortized inference the simple AUN implementation can still
achieve flexible bit rate control (See Sec. 4.2

---

1.00
1,40
0.00
0.00
Figure 7: Results of Song et al.
2021 's ROI-based Coding by using the same original image and The white rectangle indicates where ROI map is not effective.
ROI map of Fig. 6
40
40
40
38
38
38
PSNR(MD)
-4
14
12
32
Rackejflell 2000)
BockngNiammi 2010
Peodeet 2000
28
Bas enal 20001550 155A
Chaeeege 2000 2000 Sex
Cale traing tobancu
Locz tdon trinncod
Cok Daang comence
0.2
0.3 0.4 0.5
0.6 0.7 03 0.9 1.0 1.1
0.1 0.2 0.5 0.4 0.5 0.6 0.7 0.8 0.9 1.0 BRineebil pinel)
1.1
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Birato(bis pisel)
Biasieibis per pisel)
Figure 8: Comparison of R-D performance between Code Editing Enhanced and Baseline + SGA per model. Left: Ballé et al. (0018) as baseline. Middle; Minnen al. [2018 as baseline. Right: Cheng et al. 2020 as baseline.
Conclusions
We propose Code Editing, paradigm for continuous variable bitrate NIC based semi- amortized inference. It outperforms current variable bitrate NIC methods. Moreover, it achieves flexible spatial bits allocation and multi-distortion trade-off, all with decoder. Limitations include that the efficiency of Code Editing could be improved, and that the empirical study on more compounded target mentioned Eq. should be addressed.
Limitation
In general, our method does not work for the cases where encoding time matters, such real-time communication. Our method is extremely useful for the cases where we encode just once but decode/view plural number of times, such as content delivery network. Another limitation of our work is that we limit the scope of discussion to scalar instead of vector A. Adopting vector increase the range of bitrate in ROI-based coding. For Code Editing Enhanced, all pixels share one A. And this means in ROI-based coding, the overall might not be suitable for all pixels with different quality map value
Acknowledgments and Disclosure of Funding
This work is supported by SenseTime Research. The content is solely the responsibility of the authors and does not necessarily represent the official views SenseTime Research.
References
E. Agustsson, M. Tschannen, F. Mentzer, R. Timofte, and I.. V. Gool. Generative adversarial networks for extreme learned image compression. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, pages 221-231. IEEE, 2019.
E. Agustsson, D. Minnen, G. Toderici, and Mentzer. Multi-realism image compression with conditional generator. arXiv preprint arXiv:2212.13882 2022.
10

---

J. Ballé, V. Laparra, and E. P. Simoncelli. End-to-end optimized image compression. In International Conference on Learning Representations, 2017.
Ballé, D. Minnen, Singh, S. J. Hwang, and N. Johnston. Variational image compression with scale hyperprior. In International Conference on Learning Representations, 2018.
F. Bellard. Bpg image formaton. https://bellard.. org bpgg 2018.
G. Bhat, M. Danelljan, and R. Timofte. Ntire 2021 challenge on burst super-resolution: Methods and results. In CVPR, pages 613 626,,,6622266 2021.
Y. Blau and Michaeli. The perception- tradeoff. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1228-6237) 2018.
Y. Blau and T. Michaeli. Rethinking lossy compression: The rate-distortion ceree--stootiooperrepppee perception tradeoff. In International Conference on Machine Learning, pages 675-685, PMLR, 2019.
B. Bross, J. Chen, J.-R. Ohm, G. Sullivan, and Y.-K. Wang. Developments in international video coding standardization after ave, with an overview of versatile video coding (vvc). Proceedings of the IEEE, 109(9):1463-1499) 2021.
C. Cai, Chen, Zhang, and Z. Gao. End-to-end optimized ROI image compression. IEEE Trans. Image Process., 19:3442-3457 2020.
Z. Cheng, H. Sun, M. Takeuchi, and J. Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2020.
Y. Choi, M. El-Khamy, and J. Lee. Variable rate deep image compression with conditional autoencoder. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, pages 3146-3154. IEEE, 2019.
CLIC. Workshop and challenge on learned image compression (clic). http://cclic.compeesss cc/ 2022.
C. Cremer, X. Li, and Duvenaud. Inference suboptimality in variational autoencoders. In International Conference on Machine Learning, pages 1078-1086. PMLR, 2018
Z. Cui, J. Wang, S. Gao, T. Guo, Y. Feng, and B. Bai. Asymmetric gained deep image compression with continuous rate adaptation. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, pages 00542.10541. Computer Vision Foundation / IEEE, 2021.
J. C. M. S. A. Djelouah and C. Schroers. Content adaptive optimization for neural image compression. In Proceedings of the CVPR, 2019.
N. Fathima, J. Petersen, G. Sautière, A. Wiggers, and R. Pourreza. A neural video codec with spatial rate-distortion control. In Proceedings of the IEEE/CVF Winter Conference on Applications of January 2023.
Computer Vision (WACV), pages
Gao, Y. Shi, T. Guo, Z. Qiu, Y. Ge, Z. Cui, Feng. Wang, and B. Bai. Perceptual learned image compression with continuous rate adaptation. In 4th Challenge on Learned Image Compression, Jun 2021.
D. He, Zheng, B. Sun, Y. Wang, and Qin. Checkerboard context model for efficient learned image compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14771-14780, 2021.
D. He, Z.. Yang, Peng, R. Ma, Qin, and Wang. Elic: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. arXiv preprint a20i 10886, 2022.
N. Johnston, Vincent, D. Minnen, M. Covell, S. Singh, T. Chinen, J. Hwang. J. Shor, and G. Toderici. Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks. 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, pages 4385-4393, Computer Vision Foundation IEEE Computer Society, 2018

---

Y. Kim, S. Wiseman, A. Miller, D. Sontag, and A. Rush. Semi-amortized variational autoencoders. In International Conference on Machine Learning, pages 2678-2687 PMLR, 2018.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.694, 2014.
D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint 2013.
E. Kodak. Kodak lossless true color image suite.
us/ http://rokkkusggrrrah..cc/ooooo 1993.
Marino, Y. Yue, and S. Mandt. Iterative amortized inference. In International Conference on Machine Learning, pages 3403-3412. PMLR, 2018.
Mentzer, G. D. Toderici, M. Tschannen, and E. Agustsson. High-fidelity generative image compression. NIPS, 33, 2020.
D. Minnen and S. Singh. Channel-wise autoregressive entropy models for learned image compression. In 2020 IEEE International Conference on Image Processing (ICIP), pages 3339-3343, IEEE, 2020.
D. Minnen, J. Ballé, and G. D. Toderici. Joint autoregressive and hierarchical priors for learned image compression. Advances in neural information processing systems, 31, 2018.
M. S. Sajjadi, B. Scholkopf, and M. Hirsch. Enhancenet: Single image super-resolution through automated texture synthesis. In Proceedings of the IEEE international conference on computer vision, pages 4491-4500, 2017.
M. Savinaud, M. Malaterre, J. Malik, M. Grizonnet, Michel, and A. Descampe. Open jpeg a free and open-source solution to access to the new jpeg2000 geospatial products. In ESA Living Planet Symposimm, volume 722, page 302, 2013.
Y. Shi, K. Zhang, J. Wang. N. Ling, and B. Yin. Variable-rate image compression based on side information compensation and r-À model rate control. IEEE Transactions Circuits and Systems for Video Technology, 2022.
M. Song, Choi, and B. Han. Variable-rate deep image compression through spatially-adaptive feature transform. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, pages 2360-2369. IEEE, 2021.
Z. Sun, Z. Tan, X. Sun, F. Zhang, Y. Qian, D. Li, and H. Li. Interpolation variable rate image compression. In MM 21: ACM Multimedia Conference, pages 5574-5582. ACM, 2021.
L. Theis, W. Shi, A. Cunningham, and F. Huszár. Lossy image compression with compressive autoencoders. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
G. Toderici, S. M. O' O'Malley, S. Hwang, D. Vincent, D. Minnen, S. Baluja, M. Covell, and R. Suk- thankar. Variable rate image compression with recurrent neural networks. In 4th International Conference on Learning Representations, ICLR 2016, 2016.
G. Toderici, D. Vincent, N. Johnston, S. J. Hwang, D. Minnen, J. Shor, and M. Covell. Full resolution image compression with recurrent neural networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, pages 5435-5443. IEEE Computer Society, 2017.
Y. Xie, K. L. Cheng, and Q. Chen. Enhanced invertible encoding for learned image compression. In Proceedings of the 29th ACM International Conference on Multimedia, pages 162-170, 2021.
Y. Yang, R. Bamler, and S. Mandt. Improving inference for neural image compression. Advances in Neural Information Processing Systems, 33:573-584, 2020.
R. Zhang, P. Isola, A. A. Efros, F. Shechtman, and O. Wang- The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're roe're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "l9VGqdAoNT7", "reviewer": "Reviewer_JZkG", "summary": "This paper proposes a method called \"Code Editing\" for rate control, which is an optimization of the latents at encode time. This has commonly been used in the past to boost rate distortion performance, but not as a continuous rate control mechanism. Second, this paper proposes a method of combating the rate-distortion decay of latent optimization with the use of an adaptive quantizations step size. Last, the paper allows for a per pixel spatial rate-allocation mechanism that grants even more fine-grained control of the rate.\n\nGiven the authors responses, I have raised my overall review.", "questions": "1. What is the rate distortion performance at very high rates (>1.0 bpp, say 2.0)? Is code editing still effective at high rates?\n2. Is code editing effective with ROI across complex ROI masks? e.g. imagine you have an image with 3 faces roughly the same size, can shifting the ROI to each of the three faces increase fidelity equally? \n3. How important is the granularity of the quantization step size being searched? If the grid were two or four times as dense, would you have increased performance or just more points on the RD curve?\n4. Many of the baseline models have increased capacity for the larger rates in order to keep quality high. For example, in the Balle 2018 case, was the model using the 192 or 320 throughout? Are higher capacity models needed to allow for the flexibility needed for code editing?", "limitations": "Author sufficiently discussed the limitations of their work and any potential negative society impact.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThrough \"code editing\", the latents can be additionally optimized for the given rate-distortion trade-off over a large range of bitrates [0.1 to 1.0] bpp. This allows one model, with additional encoding time to target a wide swatch of the rate-distortion curve. \n\nPer pixel rate control allows a very explicit, fine grained rate control on a per image basis. Additionally, this information is only used at encode time and isn't required to be transmitted to the decoder.\n\nThe empirical results of naive code editing only being successful in a narrow rate range (around 0.1 bpp).\n\nWeaknesses:\nNo results were shown in the very high quality/high rate range. From the rate-distortion curves, it appears as if the Code Editing method starts to flatten at the high rate (see fig 3), versus the ideal range of around 0.5 bpp.\n\nThe ROI based coding examples seem very much designed to show high contrast (checkerboard, gradient, NIC). Show the result of a semantic segmentation model's ROI and showing if bit allocation on small text or faces can enhance the overall image at low bitrates would make an obvious visual difference. (Perhaps the ROI shifts between one of several faces in the same image for example).\n\nThe quantization step size is currently determined via grid search (brute force).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "l4m8-spFs1z", "reviewer": "Reviewer_kNRz", "summary": "The paper investigates obtaining flexible-rate neural image compression (NIC) methods by adapting representations during encoding. They obtain a new representation y' by adapting the quantization width as well as the latent values themselves. Parameters are frozen so no overhead except for longer encoding time.", "questions": "Q1) Did you experiment with adapting $\\Delta$ during training also? I found it surprising that you can use a model trained for lambda  = 0.015", "limitations": "No negative societal impact described in the main text, hidden in the supplementary material.", "rating": 6, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall solid idea. I'm not super familiar with the adaptive rate literature but it seems novel to me. I like the idea of adapting $\\Delta$. I also appreciate that there are various interesting ablation studies in Fig 1, and that authors investigate using the formulation for perceptual quality optimization and adaptive ROI-based rates (neat!). The results in Fig. 3 are promising.\n\n## Weaknesses\n\n- W1) Bad formulation in Sec 2.1: To me, who is very familiar with this field, Sec 2.1, in particular L72-L84 sounded weird. What is the meaning of \"connected\". What is it supposed to mean that the \"factorized uniform distribution is used to simulate the quantization noise\" (I suppose that we add IID uniform random noise?). What about \"the decoding process is connected to compute the likelihood ..., which is equivalent to distortion evaluation\" (some link to VAE?). This might be a language barrier issue but it sounds to me like some badly pieced together bits from Balle et al's original paper. I would strongly suggest rewriting Sec 2.1. \n- W2) Incorrect/bad formulation in L112: \"In NIC, the probability mass function (pmf) [...] over quantized y is computed via differentiating cumulative distribution function\" [sic]. That is only true for the factorized prior from Balle et al, which is formulated via a CDF. For e.g. hyperprior based approaches (used in almost every compression paper since), we parameterize the density directly and calculate the PMF by integrating over boxes.\n- W3) The above makes me question the soundness of the evaluation. Did the authors use range coding/arithmetic coding in the end to calculate real bitrates? Was care taken to make sure the model does not accidentally cheat?\n\n- W4) (minor) similar to the above, the recap of distortion perception left a bad taste in my mouth. The authors cite Blau and Michaeli's work, where the perceptual loss is formulated via a _divergence_, yet authors use LPIPS, a normal pair-wise distortion, to do perceptual optimization.\n\n## Conclusion\n\nOverall, I will have to give a Reject rating, but if authors can convince me that they i) did a solid evaluation including range coding that implies we can trust the results and that they ii) will make sure the formulation is sound and understandable, I am willing to change my mind, since the approach itself looks interesting.\n\nI'm giving a 4/5 confidence rating since there might be related work to adaptive-rate NIC that I'm missing\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cNGrLqwt-fV", "reviewer": "Reviewer_Q7gH", "summary": "- The authors propose Code Editing that control bitrate of neural image compression with semi-amortized inference.\n- The authors solved the performance decay in low rate of Code Editing by making the quantization step size adaptive.", "questions": "- When comparing the R-D curves of Code Editing Enhanced and Yang et al. [2020], what is the difference of the proposed method and is it reasonable?\n- Why does combining semi-amortized inference and adaptive quantization step size improve when low-rate results?\n- Why are the low-rate results in Fig. 3 approaching baseline?", "limitations": "- In Chapter 5, the authors told that efficiency of Code Editing could be improved with regard to limitations, but it would be good to be more specific about what cases the efficiency should be improved. For example, what are the cases in which the authors' method does not work well?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The authors shows an interesting experimental result (Fig. 1 Left) it seems to solve the performance decay in low rate of Code Editing.\n\nWeakness\n- The new part of method has not seem to be clearly explained. The basic parts of amortized inference strategy in neural image compression has already been proposed (Yang et al. [2020]). Therefore, I thought it was a bit exaggerated when section 2.2 states that they propose a new paradigm of controlling R-D trade-off by semi-amortized inference. Variable bitrate compression by changing the quantization step size has also been proposed for neural image compression (Choi et al. [2019]). It would be great if you could clarify what is new when compared to these previous studies.\n- The R-D curve of Code Editing Enhanced versus conventional semi-amortized inference based neural image compression (Yang et al. [2020]) has not been compared. If quantitative comparisons with previous studies were made (e.g. overhead to support variable bitrate), it would be easier to argue the superiority of this study.\n- typo: p.2 L53 bitrtae -> bitrate", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept