Selective compression learning of latent representations for variable-rate image compression
Authors: Jooyoung Lee
Seyoon Jeong
Munchurl Kim
Venue: NeurIPS 2022 Accept (2022)
Keywords: NN-based image compression
variable-rate image compression
selective compression of representations

Abstract: Recently, many neural network-based image compression methods have shown promising results superior to the existing tool-based conventional codecs. However, most of them are often trained as separate models for different target bit rates, thus increasing the model complexity. Therefore, several studies have been conducted for learned compression that supports variable rates with single models, but they require additional network modules, layers, or inputs that often lead to complexity overhead, or do not provide sufficient coding efficiency. In this paper, we firstly propose a selective compression method that partially encodes the latent representations in a fully generalized manner for deep learning-based variable-rate image compression. The proposed method adaptively determines essential representation elements for compression of different target quality levels. For this, we first generate a 3D importance map as the nature of input content to represent the underlying importance of the representation elements. The 3D importance map is then adjusted for different target quality levels using importance adjustment curves. The adjusted 3D importance map is finally converted into a 3D binary mask to determine the essential representation elements for compression. The proposed method can be easily integrated with the existing compression models with a negligible amount of overhead increase. Our method can also enable continuously variable-rate compression via simple interpolation of the importance adjustment curves among different quality levels. The extensive experimental results show that the proposed method can achieve comparable compression efficiency as those of the separately trained reference compression models and can reduce decoding time owing to the selective compression.

Selective compression learning of latent representations for variable-rate image compression
Jooyoung Leel.2, Seyoon Jeong', Munchurl Kim
School of Electrical Engineering, Korea Advanced Institute of Science and Technology, Korea Electronics and Telecommunications Research Institute, Korea umpu@kai wwtekkiist... st. ac.kr, jsyβetri. .eneetri.eekr,, mk mkimee@ka itt. ac.
Abstract
Recently, many neural network- based image compression methods have shown promising results superior to the existing tool-based conventional codecs. However, most of them are often trained as separate models for different target bit rates, thus increasing the model complexity. Therefore, several studies have been con- ducted for learned compression that supports variable rates with single models, but they require additional network modules, layers, or inputs that often lead to complexity overhead, or do not provide sufficient coding efficiency. this paper, firstly propose a selective compression method that partially encodes the latent representations in a fully generalized manner for deep learning-based variable-rate image compression. The proposed method adaptively determines essential repre- sentation elements for compression of different target quality levels. For this, we first generate 3D importance map the nature of input content to represent the underlying importance the representation elements. The 3D importance map is then adjusted for different target quality levels using importance adjustment curves. The adjusted 3D importance map is finally converted into a 3D binary mask to determine the essential representation elements for compression. The proposed method can be easily integrated with the existing compression models with a neg- ligible amount of overhead increase. Our method can also enable continuously variable-rate compression via simple interpolation of the importance adjustment curves among different quality levels. The extensive experimental results show that the proposed method can achieve comparable compression efficiency as those of the separately trained reference compression models and can reduce decoding time owing to the selective compression. The sample codes are publicly available at ott.:s//ggtttoo..coooooooonnnnnr
Introduction
Recently, neural network -bNed image compression methods 2 4 & 9 10
12 have been actively studied and shown superior performance in terms of PSNR BD-rate to those of the conventional tool-based compression methods, such as BPG 04 and JPEG2000 13)
codec, called H.266 Intra-coding [16). However, since most the previous deep learning- based models
few recent methods [2] achieved comparable results with respect to the state-of-the-an
[12
are trained separately for different target compression levels, several models with a large number of parameters are required support various compression levels. To address this issue, recently several
21 22), which use conditional transform or adaptive quantization, have been proposed. However, most them require additional network modules, layers, or inputs that may cause complexity overhead. In this paper, novel 'selective compression of representations' (SCR) method presented, which performs entropy coding only for the partially selected latent
methods 1
17
18
19
20
"Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

representations. The selection of representations is determined via 3D binary mask generation process in a target quality-adaptive manner. In the 3D binary mask generation of our SCR method (see Figure Tb). (i) 3D importance map of the same size, independent of target quality levels, is generated for the 3D latent representations (multi-channel feature maps); (ii) the 3D importance map is adjusted via the channel-wise importance adjustment curves for given target quality level; and (iii) the 3D binary mask is then generated by taking the round-off the adjusted 3D importance map. Note that the srree--ualtt--nddeeneeee
3D importance map becomes karget-quality quality-deenndent after the channel-wise importance adjustment. We incorporate our method with an adaptive quantization scheme [19] into several existing deep learning- -ased reference compression models 66 7). where jointly optimize the whole elements together with our selective compression of latent representations and adaptive quantization in end-to-end manner. In the architectural aspect, our SCR method minimizes the overhead by utilizing only a single 1 x 1 convolutional layer to generate the 3D) importance map and importance adjustment curves for a finite number of target quality levels. In addition, the SCR method also supports continuously variable-rate compression through a simple non-linear interpolation of the importance adjustment curves between two discrete target quality levels. Furthermore, the SCR method significantly reduces the decoding time compared to the existing lightweight variable-rate methods [19 20 23] by skipping the entropy decoding process for a substantial amount of unselected representations. Impressively, the coding efficiency of the proposed SCR method better or comparable to those of the separately trained reference compression models 66 77, for different target quality levels, and is superior to those of the existing variable-rate compression methods
20 23 24 25]. The contributions of this study are summarized as follows:
To our best knowledge, the proposed SCR method is the first NN-based variable-rate image compression method that selectively compresses the representations in fully generalized way and target quality- manner. It provides compression efficiency comparable to those of the separately trained reference compression models.
The proposed SCR method can be applied various existing image compression models with- out modifying their architectures, thus allowing for a high applicability. We incorporate very lightweight modules for SCR, including only a single 1 convolutional layer and small number of importance-adjustmen curves, into the existing compression models. Our SCR method even reduces decoding time compared those of the existing lightweight variable-rate models and high bit rate reference compression models owing to the selective compression.
To enable continuously variable-rate compression, the proposed SCR method extends the existing interpolation- based approach by additionally incorporating the interpolation of the importance adjustment curves between discrete quality levels in which the SCR model is trained. With experiments, we verify that our extension can stably support the continuous bit rate compression.
2
Related work
22 23 26 27 have been conducted for enabling a single NN-based image compression model to support variable-rate compression. The first barning-based variable-rate image compression model [ progressively performs image compression in a low-to- high quality manner by accumulating additional binary representations as the number of compression iterations increases. For the entropy model-based approaches, Choi et al. [17] proposed a conditional convolution that adaptively operates according to different target quality levels. Scale and shift factors are derived from a one-hot vector representing a quality level, and are then applied to each output of convolutional layers. Cai et al. [18] stacks representations form multiscale structure, and then determine point-wise scales, representing the importance levels of representations to be compressed. This method also uses additional modules named a multi-scale decomposition transform layer (MSD) and an inverse multi-scale decomposition transform layer (IMSD), both of which consist of multiple convolutional layers. Cui al. 19 utilizes two types of vectors for the adaptive quantization and inverse quantization, and Chen et al. (23) similarly utilizes scaling and shifting vectors. These support variable-rate compression with negligible overhead on top of the existing
Several studies
18
19
20
21
methods 19 23
compression models (6 7 for Cui et al. [19 and B 6 10 for Chen et al. (23)). However, as
described in our experimental results and Rippel's work (0). the adaptive quantization alone does not provide sufficient coding efficiency compared to those of the separately trained models. Lu et al. (2T) use an adaptive quantization layer (AQL.) and an inverse adaptive quantization layer (IAQL.
to obtain quantization and inverse quantization factors for each representation. Here, the AQL and

---

Encoder
Hyper encodar
Penutimate
Adaptiive quantization
level
Entropy
codingldacoding
9, Representaton solection
3D mask generation
3D mask generation
Penatimatt
% Entropy codingifeccding
wipsts
lover
3D importance generation
Scaling selection
Importance map
(M].
decodar
(common quality levels)
Importance adjustmant
Reshaping
for given target quality level
9, Adaplivve inverse quant.
Adjusted importance (for a specific quality level)
Torget qualty g
Binarization (rounding-off
Decoder
3D binary mask
(a)
(b)
Figure I: (a) Overall architecture of the proposed SCR method. In this figure, the SCR method is incorporated into the Hyperprior ( model. The elements for variable-rate compression are represented as the dotted boxes, and especially those for the selective compression are highlighted in light blue. (b) The 3D binary mask generation process of the SCR method.
IAQL layers are separately trained for each target quality level. Rippel et al. (20 additionally feed the information of a target quality level into each convolutional layer in the form of "level map" where a one-hot vector of 8 quality levels is assigned for all spatial positions. The input level map can make each convolutional layer adaptively work according to a given target quality level. Lu et al. (26) firstly presented an NN-based quality scalable coding scheme, named PL,NNQ, using nested quantization and latent ordering. However, it could not achieve comparable results to its separately trained base compression model (7). Song et al. [22] supports the image compression of spatially different qualities using a spatial quality map. Although this method provides a new functionality, the additional module, named addppiv feature transform, that transforms the spatial quality map into an input for each convolutional layer may increase the overall complexity.
From the perspective of partial encoding for representations, Li et al. [24] and Mentzer et al. [25] utilize 2D importance maps to represent spatial importance of representations, which allows for spatially different bit allocation in different regions. Their models mainly focus on the fixed-rate compression, although Mentzer el al. [25] presented a few decoded images at multiple-rates using the shared en/decoder networks. It should be also noted that their 2D importance maps convey the information about how many representation elements at each spatial location should be taken forward along the channel (See Figure 15 in Appendix F). On the other hand, our component-wise 3D importance map represents the essence of individual representations, which is adaptively adjusted according to given target qualities in an R-D optimization sense. This brings good generalization with higher-coding efficiency and more stability in training, as further discussed in Appendix F.
3 Proposed method
3.1 Overall architecture
Our SCR method can be combined with adaptive quantization such as Cui el al. 19 and Chen er al. [23], on top of several compression architectures with hyper-encoder and hyper-decoder (6) such as the models in [6) 7 8 10 [2], as shown in Figure Tal In this paper, we apply our SCR method for several reference compression models, Hyperprior 16, Mean-scale (7), and Context 77]. to show its effectiveness with generality. In the architecture with hyper-encoder and hyper-decoder [6]. input image z is transformed into representation y using encoder network, and the hyper encoder and decoder are used to code the distribution parameters for the quantized representation ğ of y as a side information, with which ÿ is entropy-coded and decoded. The representation ý is then reconstructed into an image x' through a decoder network. Upon this base compression architecture, we exploit two additional elements: adaptive quantization and selective compression to enable variable-rate compression. The selection of representation elements in the encoder side is expressed as follows:
g/ =M m22 4)), with 9. = AdaQu(y),
(1)
3

---

-=1.0
R=2.0
q=3.0
q=4.0
9=5.0
9=6.0
q=7.0
q=8.0
Adjusted importance map
0.3845
0,5244
0.9897
1.8413
3.3513
7.0189
14.3548
22.6230
3D importance map
1-10
9:2.
9-3.
404.0
9-50
9.6.0
9-7.0
9-00
Figure 2: Average importance adjust-
Figure 3: Top sample masks for 8 target quality levels where the dark blue indicates the selected representation elements by 3D binary masks. Bottom the masks averaged along the channel axis. The higher the target quality is, the more the representation elements are selected, especially in more complex regions.
ment curves for each target quality level q our SCR model (on Hyperprior (
The numbers indicate the mean values of trained Y vectors (See Section 3.2).
where D1 is a set of the selected elements of Dy for a given target quality level q and AdaQq a target quality-adaptive quantization operator where AdaQ, = Roundly/VV, with quantization vector QV,- (.) is an element selection operator for Ug. and m(2.0) 9) represents generated 3D binary mask for and quantized hyperprior (Sec. (.2). The representation y is It should be noted that De is entropy-coded and entropy-decoded using an entropy model based on a target quality dependent distribution P, (Sec. 3.3). The image reconstruction z in the decoder side is given as:
the output (n) of the encoder network En(-) for input image
Figure
x' De(AdeellaL De(AAdIQ,(@g))
with Adal ( 3g IQV, and J = Re(y m)
where vy is reconstructed image for given target quality level q, the output of the decoder network De(-), ), (-)) is adaptive inverse-quantization operator that multiplies an inverse- quantization vector IQV, to input Uy. and Re(-) is a reshaping operator that converts the selected elements D in ID shape into the elements of a 3D-shaped representation in place by using the 3D binary mask m (2, q). For the unselected elements, the reshaping operator Re(-) places values the corresponding positions. It should be noted that the unselected elements are filled with 0 values for the Mean-scale 7 and Context 7 models that utilize μa estimation well for the Hyperprior 6 model. Example source codes for M M..) and Re(-) are provided in Appendix A. In Eqs. and 2 the vector dimensionalities of QV, and IQV, are both Cy, the number of channels y. so that the quantization of y and the inverse quantization of ÿy are performed channel -wise by the respective elements of QV, and I V,V, respectively, as in the previous adaptive quantization method 199
3.2 3D binary mask generation
The 3D binary mask generation process consists of the three steps: (i) 3D importance map generation, (ii) importance adjustment, and (iii) binarization, shown in Figure [b which is given as:
m(2,g) g) (iim(2)7
where im is a 3D importance map that is generated via the hyper-decoder for the hyperprior å as input, 7, = =11]- "....., ? are learned to determine the channel-wiss importance adjustment curves for given target quality q. and B(-) is binarization operator with the rounding-off.
a parameter vector of dimensionality N ((= C2)) where the parameters
3D importance map generation. The 3D importance map im(2), which has values in the range between and 1, represents the underlying importance of each element in y. Note that the 3D importance map is generated, not dependently of target quality levels but dependently of input images, thus represents the nature of z in perspective of element-wise importance. Without utilizing dedicated complex network that generates im (2). we feed the outputs of the penultimate

---

convolutional layer (after the activation) in the hyper decoder into a single 1 x 1 convolutional layer in the mask generation module, followed by a clipping function to obtain the importance values between O and 1. To be specific, along with the the single x convolutional layer above, the hyper en/decoder networks also play a role of compressing and reconstructing (generating) the 3D) importance map. However, even for the methods without hyper en/decoder networks, our method can also be applied if a dedicated network to the compression and generation of 3D importance map is adopted. In this case, we expect that the inherent compression efficiency due to the selective compression can be maintained because the bitstream for the auxiliary information is relatively very small. Since the compressed hyperprior bitstreams in the current SCR model often take about 2 ~ 3% of the total bit rates, the auxiliary bitstreams only for the 3D importance map are expected to be less than this. Further study on the dedicated en/decoder structures may lead to additional performance improvement.
Importance adjustment The actual importance of each representation element may vary according to various target quality levels. For example, some representation elements corresponding to the texture of high complexity in the images may not be necessarily required in low-quality compression. Thus, is natural to adjust the 3D importance map, which is commonly used for all quality levels, according to specific target quality level. For this, we devise scheme of adjusting the 3D importance map im (2) using importance adjustment curves for various target quality levels. The importance adjustment curves change the element values of ém(2) channel-wise where their curvatures learned as a parameter vector 7/ for < < No where No is a total number of target quality levels. Note that the target quality improves q increases. Figure 2 shows average importance adjustment curves for each target quality level q. In Figure - the horizontal and vertical axes represent the input im( value be adjusted and its adjusted result, respectively, and the numbers labeled on the importance adjustment curves indicate the average values of trained 7 vectors for No target quality levels. It is noted in Figure 2 that the importance adjustment curves for q > tend amplify the elements of input im(2) while attenuating them for < 6 in an average sense. For = 6, there is little variation with an average 7, 0.9897 before and after the importance adjustment. Consequently, (2) is more strongly amplified in an overall sense for higher target quality levels. Whereas, for the lower target quality levels, im( (2) is largely attenuated in general, so only a small number of im ( elements whose values are close to 1 can maintain their importance. It should be noted in Figure thatt although the 7, values are monotonic with the change in the target quality level q. the individual elements in the Y/ vectors are not always the cases because some representations are optimized in use only for a lower bit rate and can be ignored or de-emphasized in a higher bit rate range, as shown in Figure [ The total number of %, vectors is No. thus total of No x C2 parameters are learned for all 7 vectors. In our implementation, No is set to 8 and C2 is set to that of the original reference model.
Binarization. The 3D binary mask is finally determined by the rounding operator, denoted as B(-). Here, "1" values in the output 3D binary mask indicate that the corresponding elements in y, at the same coordinates, are selected for compression. Figure 3 shows examples of the generated masks for different target quality levels from q = 1.0 to = 8.0 when our SCR method is implemented on top of the Hyperprior 6 model and Kodim 12 image of the Kodak image set 28 used as an input sample, in which the components marked in dark blue indicate "1" values. When g is set to 1.0, the lowest quality level in our SCR method, only 3.22% of the total elements are selected, and the selection ratio gradually increases as the q value increases. For q = 8.0, 43.39% of the representation elements are selected. In addition, as shown in the averaged masks along the channel axis, the proposed method uses more representations in the high-complexity region. Over the whole Kodak image set (28), the average proportions of selected elements for the target quality levels from 1.0 to 8.0 are 6.41%. 9.66%. 14%, 19.90%, 27.00% 35.68%, 46. 20%, and 55.81%. respectively, where they are almost linearly proportional to the average bpp values, as shown in Figure 4
how many representations in a low target quality level are commonly used (or selected) for higher target quality levels. For example, the orange line indicates that 100%, 99.8%, 99.6%. 99.0%, 98.3% and 98.2% of the representation elements, which are selected for target quality level q = 2.0, are also reused for higher target quality levels from q = 3.0 to 8.0, respectively. should be noted in Figure that the case with q = 8.0 tends to highly reuse 97.6% of the selected representation elements for the case with q = 1.0. This implies that the proposed SCR method does not select the representation elements separately for different target quality levels, but actively takes large portion of representation elements as common components for various target quality levels.
5

---

Proportion of selection
40%
Proportion
30%
20%
10%
0=1
0%
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
Counterpart target quality level
BPP
9-1 q-2 q=3 --es -qu6 -q=7 qa-
Figure 4: Average proportions selected representa- tion elements versus average BPP. (test set: the Kodak image set [8]] base model: Hyperprior [6]
Figure 5: Average proportions reused representa- tion elements from low to high quality levels. (test set:
the Kodak image set 288 base model: Hyperprior (6)
3.3 Training
We train the proposed SCR model in end-to-end manner, where the base compression model, 3D importance map generation module, Yo vectors, QV, vectors, and IQV, vectors are optimized all together, using the total loss formulated follows:
C= C= R. + >9 Dy: with Rg = ( 2 (()).
(4)
where R2 and D2 represent rate and distortion terms, respectively, for the target quality level g- and Xg 0.2 29- is a parameter for adjusting the balance between the rate and distortion. Dj can be either the mean squared eтror (MSH) or MS-SSIM between the input image a and the reconstructed image = For the MS-SSIM- -asedd optimization, we use the distortion term D3 3000(1 MS (x, zz)). H-)) is calculated cross-entropy for the quantized representations of and z. In the case of y, because the quantization and mask generation processes are different for each target quality level y, we use cross entropy (-) for target quality level & as follows:
N
1
Hi 2)
log, P. (,G
(5)
with J ( U( U(- 0.5), m(E
where Na is the number of pixels in input image x, N₁ is a total number of the selected elements
(igs)em of Øg- The cross entropy, H, (ū; ž), of the selected representation elements is calculated based on an approximate probability mass function (PMF) (.(.)) to deal with the distributions of D that vary for different target quality levels. Specifically, we determine the estimated distribution parameters Hy and og of P-(-) as M (u/QV, m(2.g) 9)) and M (ar /QV, m 9)). respectively. Here, the ya and a values are obtained from the base compression models. For the Context 7 base (k.l)
model, the position-wise parameters Big (k.1) and ((..)) are obtained for each spatial coordinate (k, I) through M (x.() QQv m(2, 9) (k.1)) and M (k.(0) /QV, 9. m (0((.g))...) (R.1), respectively. Note that we disregard ftg when a zero-mean Gaussian-ber based model is used for (.). (.). As in the previous entropy minimization-bast based compression models 7). we adopt Gaussian distribution model convolved with uniform distribution as approximate PMF (.).). and use the representation with additive uniform noise U-..55, 0.5), denoted as Jü for training, rather than the rounded representation ÿ; for inference. To handle the instability in the training phase due to learning the binary representations of the mask, we use stochastically generated mask vi((-) rather than m)(-) used in the test phase. While the adjusted 3D importance map is simply rounded-off for mn-). mn(-) is constructed with randomly sampled binary representations, similarly to Raiko el al. (29]ss approach, by regarding each element value of the adjusted 3D importance map im 27 as the probability that the corresponding component of the output mask "1", which is given as follows:
m 2 g) ((im + (-0.55 0.5))
Note that discontinuity caused by the rounding-off operator (-) is handled by bypassing the gradients backward. In the actual implementation, the training can be performed without using M.)) and Re(-), because we can exclude the unselected representations using Eq. to calculate Rg, and can obtain D, via Ada (y) m 2 to compute Dy. Other training details are described in Appendix B.
1
Hg 0 2
Σ logg (T(T))) m(2,4); with J. = y/QV, U(-0.5,0.3 (7)
N₂

---

3,4 Continuously variable-rate compression
To support the continuously variable-rate compression during the test, we determine Yg by interpola- tion when q is a value between two discrete target quality levels as follows:
Y4: 74l
if E (1, No)
14-) 74
(8)
otherwise
For example, when q 3.8, 73.8 is determined by the clement-wise multiplication of 70.0 and Y4.0- Note that the QV, and IQV, vectors are also interpolated in the same manner as above, and this non-linear interpolation is inspired by Cui et al. [9]] The experimental results of the continuously variable-rate are provided in Sec.
4
Experiments
4.1 Coding efficiency measurement for various target quality levels
Experiments for discrete target quality levels. To show the effectiveness of our SCR method with selective compression and adaptive quantization, we integrate it into the following three reference compression models: Hyperprior [6]. Mean-scale 7 and Context 7 which are denoted as SHR SCRMS and SC RC2t, respectively. These three extended models with our SCR method are compared with their original models in terms of PSNR (MS-SSIM) BD-rate. The BD-rate values are measured with the target quality levels of q= = 1.0, 4.0, 6.0 and 8.0 of our SCR models and the corresponding four compression points of the original models. The experimental results of ablation study are also provided to show the effectiveness of our selective compression where the S SCRMs and SCR are compared with and without the selective compression modules. Here, the SCR variant without the selective compression is equivalent to the adaptive quantization based variable-rate compression method of Cui et al. [19], where Gain and Inverse-gain vectors are used, corresponding to the QV, and IQV, vectors in our work, respectively. Note that we additionally apply the adaptive ir and adjustment process described in Section 3.3 for each target quality level. For comparisons, we train each of the SCR SCRMS and SCRczz as two different versions, an MISE(PSNR)-optimizz model and an MMSSSIMMopptimzze and SCR SCR respectively, where represents the used reference model. We measure average bpp-PSNR and bpp-MS-SSIM values over the Kodak image set (28).
model which are denoted SC R.
As shown in Table [ our SCR models achieve comparable results to all the reference models separately trained for various target quality levels. In addition, compared with SCRR SCR and SCRear without the selective compression, our SCR (full) models obtained coding efficiency gains of -4.3%% -5.03% and 11.1 in BD-rate, respectively. This clearly demonstrates that the selective compression in the SCR method effective terms of the coding efficiency. For the MS-SSIM-optiminee models, we obtained similar results as those of MSE-opumizzed models. The rate-distortion curve for SCR and the MoStiiid- models are provided in Appendix D. As shown in Figure b the coding efficiency gains of our SCR models are more noticeable for the low bit rate, compared with the SCR variants without selective compression. This might be because the SCR (w/o selective compression) models have to encode all the representation elements regardless of target quality levels, although some representation elements may need be optimized only for certain bit rate. Whereas, the SCR (full) model can effectively exclude those unnecessary representation elements, especially at low-bpp range, thus leading to more efficient compression.
For the Context (7 reference model, the SCR and SCR models obtained the relatively smaller coding efficiency gains over their variants without the selective compression. This may come from two reasons: (i) In the autoregressive Context (7] model, the reconstructed representation elements are utilized to predict the distribution parameters of the adjacent elements. Thus, in SCRcz models, the unselected representations filled with 0 values may deteriorate the prediction accuracy; (ii) Unlike for the other two reference compression models, Hyperprior [6] and Mean-scale [7], the SCReat variants without the selective compression already achieved a fairly satisfactory coding efficiency with respect to the original Context 7 models, so there seems to not much room for performance improvement. Although there exist somewhat deviations in coding efficiency gain for several reference models, is worthwhile to note that our SCR models achieve comparable results to all the reference models over the wide range of bit rates, as shown in Figure 6 and Appendix D.
7

---

45
45
model: Hyperprior
Base model: Mean-scale [7]
- Hyperprior [6]
Mean-scale [7]
40
SCR(w/a selective compression)
40
SCR/w/o selective compression)
SCR(Full]
SCR(Fulll]
PSNR
PSNR
35
35
30
30
25
25
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
0.00
0.25
0.50
0.75
1.00 bpp
1.25
1.50
1.75
2.00
bpp
Figure 6: Rate-distortion curves in discrete target quality levels for the reference compression model, the SCR variants without the selective compression, and SCR full model. Here, MSE-optimized models used and the results
MS-SS MS-SSIM-Opee SSoM--opiti models are provided Appendix D.
45
Base model: Hyperprior [6]
Base model: Mean-scale [7]
40
P5NR
P5NR
30
Continuous target quality levels Discrete target quality levels
Continuous target quality levels target quality levels
0.00
0.25
0.50
0.75
1.00 bpp
1.25
1.50
1.75
2.00
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Figure 7: Rate-distortion curves of our SCR model in continuous target quality levels. The orange dots indicate the results for discrete target quality levels while the green dots represent the results using % vector interpolation.
In addition, we reproduce some prior variable-rate compression methods 19
25 on the
Hyperprior [6] architecture and compare them with our SCR model in of coding efficiency and decoding time. As described in Appendix F, our SCR model shows superior results compared with those models. The experimental results and further discussions are provided in Appendix F.
Experiments for fine-grain target quality levels. To verify the continuously variable-rate com- pression of the proposed SCR method, measure the rate- dite-distortion performance changing from 1.0 to 8.0 with an increment of 0.1. As shown in Figure / the proposed SCR method stably supports the continuously variable-rate compression without degrading coding efficiency. Figure shows the reconstructed samples of the proposed SCR method where the quality gradually improves q increases. More reconstruction samples are provided in Appendix G.
Table BD- rate results our SCR models compared with corresponding reference compression models (top three rows) and the SCR variants without selective compression (bottom three rows) denoted as "w/o SC". negative BD-rate value means coding efficiency gain.
MSE-optimized
MS-SSIM-optimized
Ref.
Tested
BD-rate
Ref.
Tested
BD-rate
Hyperprior 6
pipaar SC
Hyp
-3.21 %
Hyperprior
6
Rasim SC
Hyr
-0.91 %
Mean-scale
7
SC paparr
1.30%
Mean-scale
SC Dasim MS
0.96%
Context
SC SCRa
1.23%
Context
SC SCRasin pasim
-2.00%
SCR (wlo SC)
SC SCR
-4.30%
Rasim (wlo SC)
SC.assi -4.03%
SC (wlo SC)
SC Rhee
-5.03%
SCRaaim (wlo SC)
SC Rasim
SCRasin
-3.00%
SC (wlo SC)
SCRps
-1.18%
SC Rssssm
(wlo SC)
SCRssim -1.22%

---

Discrete target quality levels (1, 2, 8)
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
Conttnnuou target quality levels (2.1 -2.9)
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
2.9
Qualitative results of the proposed SCR method (best viewed in digital format). For hest viewing, we cropped the reconstruction results. The number embedded in each sample represents the target quality level q- The top row represents the reconstruction results of the discrete quality levels. The bottom row shows the reconstruction results between g 2.0 and q 3.0 with the interpolation method Sec. Sec.3.4
Figure 8:
4.2 Decoding time
We compare the decoding times for the original reference compression models (Hyperprior 6 and Mean-scale (7]) and their SCR extensions (SC Hyp SCR MS for variable bit rate image compression. To show the effect of the selective compression in our SCR method on the decoding time, the decoding times of the above models, SC Rpear and SC S are compared for the cases with and without the selective compression in the same target quality levels. For the comparison of the two reference compression models, Hyper [6] and Mean-scale 77]. their decoding times are measured for high quality compression model that corresponds to our variable-rate configuration with q = 8.0. Note that, since each of the two reference compression models has different architectures for different target quality levels, the models corresponding to the target quality level of q 8.0 are selected for comparison, which have the same en/decoder network architectures as those of SC (CR R' and SCR We perform decoding 100 times for each image in the Kodak image set (28) and measure the average decoding time of all the images. It should be noted that a GPU is used for the decoder network processing, whereas we use CPU for hyper-decoder processing to prevent the encoder and decoder discrepancy owing to the GPU characteristic that incurs tiny errors for the same inputs even on the same machine, because of its parallel algorithm that yields different numeric results (30).
As shown in Figure 9 SCRRp and SC SCR reduce the decoding time the entire bit rate range compared with their variants without the selective compression. The decoding time savings are more noticeable for the low-quality compression owing to the reduction in the number of selected representation elements, and this clearly seen when comparing the entropy decoding times (denoted in light blue color). In spite of the overheads from the reshaping (in dark blue) and 3D binary mask generation, the time saved in the entropy decoding process definitely larger than the overhead time. It should be noted that the 3D binary mask generation time is included in the hyper decoder network time (gray) because the 3D binary mask generation shares most parts of the hyper decoder. SC SCRae and SCIM reduce the average decoding time by 11.28% and 8.32%, respectively, compared with their variants without the selective compression. Furthermore, SCRaen and SCR show lower decoding times, compared with their corresponding reference compression models with q 8.0, over all target quality levels except for g = 8.0 of SC SCRMM Instead, SC Reas with = 8.0 yielded a slightly higher decoding time compared with the original Mean-scale 7 model (q 8.0).
For Context based models, the SCR reduces the average decoding time by 25.71% and 21.53% compared with the its variants without the selective compression and the reference Context 7 model (corresponding to q=8.0), respectively. These results clearly show the effectiveness selective compression, but it should be noted that we use different entropy coder for SCR from those for SC SCREer Hyp and SCRR The detailed information on the entropy coders is provided in Appendix C. Regarding the encoding time, is also significantly reduced due to selective compression as done for the decoder. The encoding time results are provided in Appendix E.

---

Base model: Hyperprior [6]
Base model: Mean-scale [7]
0.07
0.07
0.06
0.06
0.05
0.05
0.02
0.01
0.01
0.00
0.00
[Sec.)
SCR (4:1)
W/o sc (c:1)
SCR (e:2
SCR (a:3)
W/o (e:3)
SCR (a:s)
W/o Sc (q:4)
SCR (c:5)
W/o // (e:5
SCR (q:5)
SCR (s:7)
W/o Sc (q:7)
SCR (4:8)
W/o SE (q:8)
SCR (aug.))
(Sec.)
SCR (e:2)
W/o Sc (e:1)
SCR (9:2)
W/o Sc (9:2)
SCR (q:3)
W/o Sc (4:3)
SCR (q:4)
W/a Sc (q04)
SCR (e:
SCR (q:5)
SCR (e:7)
W/a sc (a:)
(a:9)
W/a Sc (e:8)
SCR (oug.)
W/o SC (aug.)
W/o
W/o Sc
W/o Sc
W/a sc
entropy decoding
entropy decoding (hyperprior)
hyper decoder network
decoder network (on GPU)
reshaping overhead
-eef compression model
(corresponding to q=8)
Figure 9: Decoding times of the proposed SCR models, the SCR variants without the selective compression,
denoted "W/o SC", and the reference compression models. We measured the average decoding time of each model over the Kodak image dataset 28 with 2 Xeon Gold 5122 CPUs and 1xRTX Titan GPU.
Table 2: Configuration of the parameters for the SC RH model
Name
Description
No. of parameters
Hyperprior
Original model
11,813,443
QV vectors
levels * 320 ch.
2,560
IQV vectors
levels * 320 ch.
2,560
gamma vectors
levels * 320 ch.
2,560
Importance map generation
192 (in) 320 (out) + 320 (out)
61,760
Total no. of parameters
11,882,883
4.3 Increasing parameters due to SCR
The increase in model parameters due to the proposed SCR method is relatively very small. For example, the Hyperprior model has 11, 813, 443 parameters, and the SCR model has 11, 882, 883. That is, the number of parameters is increased by only 0.59%. resulting 0.26MB. Similarly, the additional memory occupancy for the SCRMs and SC'cazt model is only 0.62MB. The configuration of the parameters for the SCR model is shown in Table 2
memory increase of
5
Conclusion
We firstly proposed a 'selective compression of representations' (SCR) method for -N-based variable- rate image compression, which performs entropy coding only for the partially selected latent repre- sentations. The proposed SCR method selects essential representation elements for compression in an adaptive manner according to given target quality level. For this, we first generate a 3D importance map, which represents the importance of each representation element independently of target quality levels, and then adjust in a target quality- manner using the learned importance adjustment curves to generate 3D binary mask that represents whether not to select each representation element for compression. We demonstrated through experiments that the proposed SCR method could provide comparable coding efficiency to those of the separately trained reference compression models. Furthermore, the proposed SCR method showed less decoding time than those under comparison for almost all the cases. We also showed that the proposed SCR method could enable continuously variable-rate image compression through simple non-linear interpolation of the importance adjustment curves between discrete target quality levels in which the selective compression trained.
Acknowledgments and Disclosure of Funding
This work was supported by Institute for Information & communications Technology Planning & Evaluation (IIP)) grant funded by the Korea government(MSIT) (No. 2017-0-00072, Development of Audio/Vides Coding and Light Field Media Fundamental Technologies for Ultra Realistic Tera- media).
10

---

References
[1] G. Toderici, S. M. O'Malley, S. J. Hwang. D. Vincent, D. Minnen, S. Baluja, M. Covell, and R. Sukthankar, "Variable rate image compression with recurrent neural networks," in 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. [Online] Available: http://arxiv.org/aa///t 10.008
[2] N. Johnston, D. Vincent, D. Minnen, M. Covell, S. Singh, T. Chinen, S. Jin Hwang. J. Shor, and G. Toderici, "Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks," in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[3] J. Ballé, V. Laparra, and E. P. Simoncelli, "End-to-end optimized image compression." in the 5th Int. Conf. on Learning Representations, 2017. [Online]. Available: http://rnw.oo//ahh/66110014
[4] L. Theis, W. Shi, A. Cunningham, and F. Huszár, "Lossy image compression with compressive autvencoders," in the 5th Int. Conf. Learning Representations, 2017. [Online]. Available:
http://arxix.org/ab/// 703.000355
[5] O. Rippel and L. Bourdev, "Real-time adaptive image compression," in International Conference on Machine Learning, 2017. [Online], Available: httt:://arxivoorg/abbae
[6] J. Ballé, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston, "Variational image compression with scale hyperprior." in the 6th Int. Conf. on Learning Representations, 2018. [Ouline]. Available:
[7] D. Minnen, J. Ballé, and G. Toderici, "Joint autoregressive and hierarchical priors for learned image compression," in Advances in Neural Information Processing Systems, May 2018
[8] Lee, Cho, and S.-K. Beack, "Context-adaptive entropy model for end-to-end optimized image compression," in the 7th Int. Conf. on Learning Representations, May 2019.
[9] J. Zhou, "Multi-scale and context-adaptive entropy model for image compression," in Workshop and Challenge on Learned Image Compression at CVPR, Jun. 2019.
[10] T. Chen, H. Liu, Z. Ma, Q. Shen, X. Cao, and Y. Wang. "Neural image compression via non-local attention optimization and improved context modeling," arXiv preprint urXiv:1910.06244, 2019. [Online]. Available: httpss//arxiv..rr///////11100000
[11] Z. Cheng. H. Sun, M. Takeuchi, and J. Katto, "Learned image compression with discretized gaussian mixture likelihoods and attention modules," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[12] J. I Lee, S. Cho, and M. Kim, "An end-to-end joint learning scheme of image compression and quality enhancement with improved entropy minimization." arXiv preprint arXiv:1912.1 128817, 2019.
[13] D. Minnen and S. Singh, "Channel-wise autoregressive entropy models for learned image compression," in The IEEE International Conference on Image Processing (ICIP), 2020.
[14] F. Bellard, "Bpg image format," 2014. [Online]. Available: http://bbllard.oor//bgg//
[15] D. S. Taubman and M. W. Marcellin, JPEG 2000: Image Compression Fundamentals, Standards and Practice. Norwell, MA. USA: Kluwer Academic Publishers, 2001.
[16] "Versatile video coding reference software version 7.1 (VTM-7.1)," December 2019. [Online]. Available: https://vcgithh www..hh...nooooreeeeee/oooo Seuunboferdd/yye//VVSSooove
VIM/lagsVVMM-7.1
[17] Y. Choi, M. El-Khamy, and J. Lee, "Variable rate deep image compression with a conditional autoencoder," in International Conference on Computer Vision, 2019.
[18] C. Cai, I., Chen, X. Zhang, and Z., Gao, "Efficient variable rate image compression with multi-scale decomposition network," IEEE Transactions on Circuits and Systems for Video Technology, vol. 29, no. 12, pp. 3087-7777000 2019.
[19] Z. Cui, Wang, B. Bai, T. Guo, and Y. Feng, "G-VAB: continuously variable rate deep image compression framework," " 2020. [Online]. Available: https://arxii.on///////000.3...
[20] O. Rippel, A. G. Anderson, K. Tatwawadi, S. Nair, C. Lytle, and L. Bourdev, "Elf-ve: Effficient learned flexible-rate video coding," in International Conference on Computer Vision, 2021.
[21] G. Lu, X. Zhang, W. Ouyang, L.. Chen, Z. Gao, and D. Xu, "An end-to-end learning framework for vidco compression," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 10, pp. 3292-3308, 2021.
[22] M. Song, J. Choi, and B. Han, "Variable-rate deep image compression through apatiall--adaptii feature transform." " in International Conference on Computer Vision, 2021.
[23] T. Chen and Z. Ma. "Variable bitrate image compression with quality scaling factors," in The IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), 2020.
11

---

[24] M. Li, W. Zuo, S. Gu, D. Zhao, and D. Zhang. "Learning convolutional networks for content- weiiett-wiighed image compression." " Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. 2018, pp. 3214-3223.
[25] F. Mentzer, E. Agustsson, M. Tschannen, R. Timofte, and L. Van Gool, "Conditional probability models for deep image compression." in Proceedings the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
[26] Y. Lu, Y. Zhu, Y. Yang, A. Said, and T. S. Cohen, "Progressive neural image compression with nested quantization and latent ordering," in The IEEE International Conference on Image Processing (ICIP), 2021.
[27] J. Zhou, A. Nakagawa, K. Kato, S. Wen, K. Kazui, and Z. Tan, "Variable rate image compression method with dead-zone quantizer," in The IEEE Conference Computer Vision and Pattern Recognition (CVPR) Workshops, 2020.
[28] F. Kodak, "Kodak lossless true color image suite (photocd ped0992))" 1993. [Online]. Available: http://rokkuspppphhssskoaa//////
[29] P. Raiko, M. Berglund, Alain, and Dinh, "Techniques for learning binary stochastic feedforward neural networks," in International Conference on Learning Representations (ICLR 2015). May 2015, 1-10, international Conference on Learning Representations, ICLR: Conference date: 07155-2015
Through 09-05-2015.
[30] "Floating Point and IEEE 754 Compliance for NVIDIA GPUs," 2022. [Oaline]. Available:
12


Reviews:
[{"review_id": "_msRCT48w5J", "reviewer": "Reviewer_AG9W", "summary": "This paper presents a method to improve the coding efficiency of the baseline variable bitrate model based on adaptive quantization (quantization steps controlled by quality scaling factors) while at the same time reducing entropy coding time, thus one step closer to closing the gap to the fixed bitrate models. Specifically, it proposes to generate 3D importance binary map for the latents to be selectively coded. The 3d map generation is based on side information (decoded hyper latent) and a target bitrate. Element-wise mask without much overhead is the key to bridge the gap to fixed bitrate models.", "questions": "1. please confirm that even through results in Cui 2020 (e.g. Fig 2) looks like it already closes the gap to fixed rate reference models, according to your evaluation, it is still worse at lower bitrates. As stated in line 87-88?\n2.  For your ablation, what is exactly `the SCR variant without the selective compression`?\n3.  Do you have ablation on the importance adjustment? e.g. with only adaptive quantization from Cui 2020 + 3d importance map; or quality q as input to 3d map generation besides the second to last features in hyper decoder.\n4.  line 126: just to clarify, for mean-scale hyperprior, masked out latent is just set to mean value predicted by hyper decoder, right? same for line 253.\n5.  Do you also have decoding time profiling for Context-based models? it is not shown in Sec 4.2\n6. Fig 9: \n   * the Y-axis is second?\n   * why reshape time increases as rate goes up?\n   * why hyper decoder takes comparable/ much longer than decoder network? It should be around/less than 1/10 of the decoder? \n   * 15ms for the decoder (yellow) seems too small for Kodak images. What GPU exactly are you profiling on?\n   * any details on entropy coding? any parallel impl?\n\n\n", "limitations": "- The paper does not mention the *memory* increase due to generation of 3D importance map in encoding / decoding. Should profile this in decoding time section as well.\n- I like the study in figure 5. Is it possible to extend to embedded bitstream? i.e. encode an image to a single bitstream, to get lower bitrates, just truncate to a prefix of this stream.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Almost close the gap between variable bitrate models with separately optimized fixed-bitrate models with element-wise 3D importance map. The RD improvement over the baseline method (Cui 2020) mainly over the lower bitrates is convincing.\n- The techniques induces small overhead (a single 1x1 conv, and parameterized importance adjustment curves), can be applied to many existing models, as showed in the paper: hyperprior, mean-scale, context-based.\n- The paper is well organized with detailed description of various parts of the whole compression system.\n\nWeaknesses\n- Lack of ablation on the importance adjustment, which is one of the main contribution of this paper.\n- The presentation on reducing decoding time in Sec 4.2 is a bit misleading. line 295 - \"Furthermore, SCRpsnr Hyp294 and SCRpsnr MS showed lower decoding times compared to the original reference compression models over all target quality levels except q = 8...\" Fig  9 only shows the decoding time for reference models for q=8 instead of all target quality levels.\n- Also the way to show comparison results with BD rate can be more fair when all variable bitrate models are compared to the reference model, instead of like line 248-249.\n- The third contribution claim on continuous bitrate through interpolation already exists in Cui 2020?\n- Training procedure becomes more complicated and expensive - Three stages, in total 9.4 M steps.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UVJroSX5iG7", "reviewer": "Reviewer_DRRT", "summary": "- The authors propose NN-based variable-rate image compression method that selectively compresses the representations with generated 3D binary mask.", "questions": "- Is it possible to show the encoding time and its computing overhead when the proposed method is applied?\n- Figure 6 shows that both SCR (w/o selective compression) and SCR (Full) above 0.6 bpp and below 1.2 bpp have slightly better R-D performance than baseline [6], and above 1.5 bpp, the R-D performance appears to be slightly worse than baseline [6]. Similar characteristics are exhibited for the baseline [7]. Why this characteristic?\n- As a direct comparison with previous work on variable-rate compression, is it possible to experimentally compare the R-D curve with previous work (e.g. [19][23])?", "limitations": "- In practice, I believe there are many different neural image compression network architectures. I thought it would be good to discuss the architectures to which this method cannot be applied (e.g., without using hyperprior) and future works.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Figure 6 shows that the proposed selective compression scheme appears to improve R-D performance at low rates compared to without it.\n- According to Figure 9, the proposed method seems to have an advantage in decoding time by reducing entropy decoding time.\n- Figure 3 provides an interesting visual representation of the importance of features.\n\nWeaknesses\n- The authors mention that the conventional method [1, 17, 18, 19, 20, 21, 22] requires additional network modules, layers, and inputs, which may result in complex overhead. But the proposed method may also requires additional processing, the authors present experimental results for processing time with respect to decoding, but not for encoding.\n- There have been several studies in the past to generate importance maps for representations in deep image compression [a, b]. These are not cited in the current paper, but comparison with these would be good. Also, since the supp. material in [b] mentions the application of the importance map to multiple compression rates, I think it would be good to clarify the novel point and superiority of the proposal.\n- It would be great to make direct experimental comparison of R-D curve or B-D rate with previous studies of variable-rate compression in deep image compression [1, 17, 18, 19, 23, 20, 21, 24, 22] with the authors' proposed method and demonstrate superiority.\n\n[a] M. Li, et al. \"Learning Convolutional Networks for Content-weighted Image Compression\", CVPR 2018\n\n[b] F. Mentzer, et al. \"Conditional Probability Models for Deep Image Compression\", CVPR 2018", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "D3Azdsrnciq", "reviewer": "Reviewer_mgum", "summary": "Currently used variable-rate image compression model usually adds additional modules to the base model which increases the complexity while those using adaptive quantization to achieve variable-rate compression face degradation in coding efficiency. To address this problem, this paper introduces selective compression of representations (SCR), which embeds selective compression into variable-rate models based on adaptive compression. The selective compression mechanism first generates a 3D importance map according to the output of the hyper decoder and then adjusts it using the channel-wise parameter , and finally, it is binarized to indicate whether the element is selected for compression. Also, this paper introduces a interpolation-based mothed to support continously variable-rate compression when the quality level q is a value between two discrete target quality levels.\n", "questions": "The two base models shown in the experiment share similar encoder/decoder structure and both use hyperpiror as the entropy model, the author should include more method with a greater diversity of structures in their experiments.", "limitations": "The proposed method may not be generalized to learned image compression with context adaptive entropy engine.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. According to the experiment, the proposed selective compression mechanism can distinctly increase the compression efficiency when using \"Hyperpiror [1]\" and \"Mean-scale [2]\" as the base model. Also, because the proposed method only performs entropy coding for some of the elements, the decoding time decreases, especially at low bitrate. The proposed method only uses a single 1*1 convolutional layer to generate the 3D importance map and the parameter has a limited number of parameters, so it adds few parameters to the base model.\nWeaknesses: \n1. This article argues that variable-rate compression methods like [3] may increase the compression complexity by adding additional modules to the base model, so the author chooses to implement selective compression on the adaptive-quantization-based method. Also, the author chooses the model mentioned in [1] and [2] as the base model. However, the overhead complexity of the added module may not be that high compared to the base model itself. Also, it lacks a comparison between the proposed method and other variable-rate models which add modules to the base model in compress efficiency and complexity.\n2. According to Appendix A.3, the selective compression method has little impact when implemented on \"Context [2]\". While the state-of-the-art learned image codecs usually adopt context model in their entropy engine, the proposed method may be unable to increase the compression efficiency when using the state-of-the-art codec like [4] as the base model.\n\n\n\n[1] J. Ballé, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston, “Variational image compression with a scale hyperprior,” in the 6th Int. Conf. on Learning Representations, 2018.\n[2] D. Minnen, J. Ballé, and G. Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” in Advances in Neural Information Processing Systems, May 2018.\n[3] Y. Choi, M. El-Khamy, and J. Lee, “Variable rate deep image compression with a conditional autoencoder,”  in International Conference on Computer Vision, 2019.\n[4] Cheng, Zhengxue, et al. \"Learned image compression with discretized gaussian mixture likelihoods and attention modules.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "3a85ycPfna9", "reviewer": "Reviewer_SmBv", "summary": "The paper proposes a variable rate image compression method using selective compression and adaptive quantization. It is shown that the RD performance is similar to that of the reference models and decoding time can be reduced.", "questions": "- Are QV_q and IQV_q trained together with the model?\n\n- Is there any performance gain if the network for 3D importance map generation is more complex than a single 1x1 conv layer? Actually, [Li'18] used such a complex model.\n\n- Just out of curiosity, what happens if q<1 or q>8?\n\n- It is quite surprising that even if q=8, only 55.81% of the features are used. This means that about a half of the extracted features are useless anyway. Some discussion on this could be interesting. There are some studies on pruning of image compression models, which could be supported by this observation.\n\n- In Fig. 6, the difference between SCR(Full) and SCR(w/o selective compression) becomes larger at lower bpp values. Some discussion on why this happens would be helpful.\n\n- In Fig. 5, not all features for a value of q are used for larger values of q. This is unclear. Since the adjustment curves are monotonic, it has to be that increasing q only adds additional feature elements for encoding so the reuse rate is always 100%.\n\n- (minor) Ripple [20] -> Rippel [20]\n\n- (minor) Does \"BD-rate gain of -3.21%\" mean a gain by 3.21%? or a negative gain (i.e., loss) by 3.21%? I assume the former, but it's confusing.", "limitations": "Limitations are discussed in the paper.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*** Strength\n\n+ An important task, variable-rate image compression, is addressed.\n\n+ The paper is written relatively clearly.\n\n+ From the experiments, the method seems to work well.\n\n*** Weakness\n\n- The proposed method is composed of adaptive quantization and importance map-based selective compression.   First, it is said that the adaptive quantization is similar to [19,23]. It would be better to explain more concretely how similar/different the adaptive quantization in this paper is from those in [19,23]. Second, the idea of 3D importance map already exists [Li'18], although it was not used for variable rate compression. Furthermore, the added value of the selective compression seems to be limited, especially for advanced base models like the context model [7]. In fact, it will be necessary to show the effectiveness of the proposed method on more recent models like [11, 13, He'21, Gao'21, Qian'21].\n\n\n[Li'18] Li et al., \"Learning convolutional networks for content-weighted image compression,\" CVPR 2018\n\n[He'21] He et al., \"Checkerboard context model for efficient learned image compression,\" CVPR 2021\n\n[Gao'21] Gao et al., \"Neural image compression via attentional multi-scale back projection and frequency decomposition,\" ICCV 2021\n\n[Qian'21] Qian et al., \"Learning accurate entropy model with global reference for image compression,\" ICLR 2021\n\n- Performance comparison with existing variable rate compression methods is missing. The proposed method needs to be compared against different ways to accomplish variable rate encoding in the previous studies.\n\n- It is quite difficult to follow the quantitative BD-rate gain/loss results given in the text in Section 4.1. It is suggested to summarize them as a table.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept