EVC: Towards Real-Time Neural Image Compression with Mask Decay
Authors: Wang Guo-Hua
Jiahao Li
Bin Li
Yan Lu
Venue: ICLR 2023 poster (2023)
Keywords: neural image compression
image codec
model acceleration
model compression
knowledge distillation

Abstract: Neural image compression has surpassed state-of-the-art traditional codecs (H.266/VVC) for rate-distortion (RD) performance, but suffers from large complexity and separate models for different rate-distortion trade-offs. In this paper, we propose an Efficient single-model Variable-bit-rate Codec (EVC), which is able to run at 30 FPS with 768x512 input images and still outperforms VVC for the RD performance. By further reducing both encoder and decoder complexities, our small model even achieves 30 FPS with 1920x1080 input images. To bridge the performance gap between our different capacities models, we meticulously design the mask decay, which transforms the large model's parameters into the small model automatically. And a novel sparsity regularization loss is proposed to mitigate shortcomings of $L_p$ regularization. Our algorithm significantly narrows the performance gap by 50% and 30% for our medium and small models, respectively. At last, we advocate the scalable encoder for neural image compression. The encoding complexity is dynamic to meet different latency requirements. We propose decaying the large encoder multiple times to reduce the residual representation progressively. Both mask decay and residual representation learning greatly improve the RD performance of our scalable encoder. Our code is at https://github.com/microsoft/DCVC.

Published as a conference paper at ICLR 2023
EVC: TOWARDS REAL-TIME NEURAL IMAGE COM- PRESSION WITH MASK DECAY
Guo-Hua Wang" Wa Jiahao Li, Bin Li,, Yan Lu
Sttat Key Laboratory for Novel Software Technology, Nanjing University
2 Microsoff Research Asia wangguohual lamda nju, edu. en, {1i. 1ibin, http://ww.....alllo////ooooooo..
ABSTRACT
Neural image compression has surpassed state-of-the-art traditional codecs (H.266/VVC) for rate-distortion (RD) performance, but suffers from large com- plexity and separate models for different rate-distortion trade-offs. In this paper. we propose an Efficient single-model Variable-bit-rate Codec (EVC), which is able to run at 30 FPS with 768x512 input images and still outperforms VVC for the RD performance. By further reducing both encoder and decoder complexi- ties, our small model even achieves 30 FPS with 1920x bridge the performance gap between our different capacities models, metic- ulously design the mask decay, which transforms the large model's parameters into the small model automatically. And novel sparsity regularization loss is proposed to mitigate shortcomings of Lp regularization. Our algorithm signifi- cantly narrows the performance gap by 50% and 30% for our medium and small models, respectively. At last, we advocate the scalable encoder for neural image compression. The encoding complexity is dynamic to meet different latency re- quirements. We propose decaying the large encoder multiple times to reduce the residual representation progressively. Both mask decay and residual representa- tion learning greatly improve the RD performance of our scalable encoder. Our code is at welhtps::// //gpth/gg.t..tt//// com/mi http://miccrsoo//DCC
input images. To
INTRODUCTION
The image compression based on deep learning has achieved extraordinary rate-distortion (RD) per- formance compared to traditional codecs (H.266/VVC) (Bross et al., 2021). However, two main issues limit its practicability in real-world applications. One is the large complexity. Most state- of-the-art (SOTA) neural image codecs rely on complex models, such as the large capacity back- bones (Zhu et al. 2022; Zou al., (022). the sophisticated probability model (Cheng al. 2020). and the parallelization-umee unfrienddy auto-regressive model (Minnen et al. 2018). The large complex- ity easily results unsatisfied latency in real-world applications. The second issue is the inefficient rate-control. Multiple models need be trained and stored for different rate-distortion trade-offs. For the inference, it requires loading specific model according to the target quality. For practical purposes, single model is desired to handle variable RD trade-offs.
In this paper, we try to solve both issues to make a single real-time neural image compression model, while its RD performance is still on-par with that of other SOTA models. Inspired by recent progress, we design an efficient framework that is equipped with Depth-Conv blocks (Liu et al., 2022) and the spatial prior (He al. 2021 Li al. [202aa] Qian et al., 2022). For variable RD trade-offs, we introduce an adjustable quantization step (Chen & Ma 2020 Li et al. 2022a) for the representations. All modules within our framework are highly efficient and GPU friendly, different from recent Transformer based models (Zou et al., 2022). Encoding and decoding (including the arithmetic coding) achieves FPS for the 768 512 inputs. Compared with other SOTA models, ours enjoys comparable RD performance, low latency, and single model for all RD trade-offs.
To further accelerate our model, we reduce the complexity of both the encoder and decoder. Three series models are proposed: Large, Medium, and Small (cf. Appendix Tab. 3 Note that our small
"This work done when Guo-Hua Wang was full time intern at Microsoft Research Asia.

---

Published as a conference paper at ICLR 2023
Large
Medium
0.8
EVC
0.3
BD-Rate (%)
0.2
mask
Small
12.1M
24.7M
0.7
RRL
SwinT Hyperprion
8
mask decay
+ ours
12
10.9M
EVC WwinT-Hyperprior
277
+ one by ane
14
9.1M
3.2
Scalable Encoder
separate encoders
0.1
0.2
0
0.2 MMACs per pixel (for encoding)
0.1
0.3
MMACs per pixel (for decading)
The trade-off between BD-Rate and complexities Kodak. The anchor is VTM. and (b) show the performance improvement by our mask decay and residual representation learning (RRL). We cite results from SwinT-Hyperprior (Zhu et al. 2022) for comparison.
Figure 1:
model even achieves 30 FPS for 1920 x 1080 inputs. Recent work (Zhu et al. 2022) points out that simply reducing the model's capacity results in significant performance loss. However, this crucial problem has not been solved effectively. In this paper, we advocate training small neural compression models with a teacher to mitigate this problem. particular, mask decay is proposed to transform the large model's parameters into the small model automatically. Specifically, mask layers are first inserted into a pretrained teacher model. Then we sparsify these masks until all layers become the student's structure. Finally, masks merged into the raw layers meanwhile keeping the network functionality unchanged. For the sparsity regularization, Lp-norm based losses adopted by most previous works (Li al. 2016 Ding al. 2018 Zhang al. (021), But they hardly work for neural image compression (cf. Fig. 7p. In this paper, we propose novel sparsity regularization loss to alleviate the drawbacks With the help of our large model, our medium and small models are improved significantly by 50% and 30%, respectively (cf. Fig. Taa. That demonstrates the effectiveness of our mask decay and reusing large model's parameters is helpful for training a small neural image compression model.
Lp so that optimize our masks more effectively.
In addition, considering the various device capabilities in real-world codec applications, the en- coder scalability of supporting different encoding complexities while with only one decoder is also a critical need. To achieve this, we propose compressing the cumbersome encoder multi-times to progressively bridge the performance gap. Both the residual representation learning (RRL) and mask decay treat the cumbersome encoder as a reference implicitly. and encourage the diversity of different small encoders. Therefore coders (cf. Fig. [b). And compared to SlimCAE (Yang et al. (021) which is inspired by slimmable networks Yu et al. 2019), ours enjoys simple framework and better RD performance.
achieves superior performance than training separate
Our contributions are as follows.
We propose an Efficient Variable-bit-rate Codec (EVC) for image compression. It enjoys only one model for different RD trade-offs. Our model able run 30 FPS for the 768 512 inputs, while is on-par with other SOTA models for the RD performance. Our small model even achieves 30 FPS for the 1920 1080 inputs.
We propose mask decay, an effective method to improve the student image compression model with the help of the teacher. A novel sparsity regularization loss is also introduced, which allevi- ates the shortcomings of L, regularization. Thanks to mask decay, our medium and small models are significantly improved by 50% and 30%, respectively.
We enable the encoding scalability for neural image compression. With residual representation learning and mask decay, our scalable encoder significantly narrows the performance gap from the teacher and achieves superior RD performance than previous SlimCAE.
RELATED WORKS
Neural image compression is in scene of prosperity. Ballé et al. (201)) proposes replacing the quantizer with an additive i.idd uniform noise, so that the neural image compression model en- joys the end-to-end training. Then, the hyperprior structure proposed by Ballé al. (2018)
2

---

Published as a conference paper at ICLR 2023
imput poutimag
Encoder 569.92 G
Hyper Enc 3,89
AE
alobal
ch
Spatial Prior 44.68
1920
reconstruction
Decoder 538.83
Hyper Dec
2
Step
28.06 G
Figure 2: The overall framework of EVC.
After that, numerous methods are proposed to improve the entropy model and the backbone (both encoder and decoder). Auto-regressive component (Minnen et al.] 2018) and Gaussian Mixture Model (Cheng al. 2020) proposed to improve the probability estimation in entropy model. Informer (Kim et al. 2022) proposes using an attention mechanism to exploit both global and local
information. Recently, more powerful backbones (e.g., INNs (Xie et al. 2021), Transformer (Qian et al. 2022) and Swin (Zhu et al. ))22 are introduced to replace ResNet et (He et al. 2016) in the encoder and decoder. However, recent SOTA neural image codec models suffer from high latency and separate models for each RD trade-off. In this paper, we propose an efficient neural image com- pression framework that aims at encoding and decoding in real-time. Especially, our framework can also handle variable rate- rate-distortion trade-offs by the adjustable quantization step (Chen & Ma 2020 Cui al. 202T Li al.) 2022a). The RD performance of our models on-par with other SOTA neural image compression models, and surpasses the SOTA traditional image codec (H.266/VVC),
Model compression and knowledge distillation (KD) are two prevalent techniques for accelerat- ing neural networks. Model compression, especially filter level pruning (He et al. 2017), aims at obtaining an efficient network structure from a cumbersome network. Some works (Li et al. 2016 Luo & Wu 2020; [H et al. 2019; Molchanov et al. 2019) calculate the importance of each filter, and prune the unimportant filters. Other methods directly penalize filters during training. They treat some parameters as the importance implicitly (Liu et al. [017 Ding et al. [2021] Wang & Wu 2022).
I.pnormm based sparsity losses are used in these methods (Ding et al. 2011 Wen et al. 2016 Zhang et al. 2021). For pruned network, KD) (Hinton et al. 2015) is utilized boost the performance by mimicking the teacher's 's knowledge (e.g., the logit (Zhao et al., 2022), or the feature (Wang et al,, 2.22 All these techniques are mainly studied on semantic tasks, but rarely advocated for neural image compression. This paper firstly tries to improve a small neural image compression model with the help of a large model. And we claim that, for this task, reusing the teacher's filters is an effective way to improve the student. To achieve this, we propose mask decay with a novel sparsity regularization loss. With proposed techniques, our student models are greatly improved.
METHODOLOGY
We first introduce our efficient framework briefly. Then, three models with different complexities are proposed: Large, Medium, and Small. Next, we describe how to improve our small model (student) with the help of our large model (teacher). Mask decay proposed to transform the teacher's parameters into the student automatically. At last, the scalable encoder is advocated. With residual representation learning and mask decay, several encoders are trained progressively to bridge the performance gap from the teacher.
3.1 OUR EVC FOR IMAGE COMPRESSION
The framework of EVC is shown in Fig. 2 Given an image, the encoder generates its representation y. Following Chen & Ma 2020
al. (2022)), achieve variable bit-rates within a single
model, we introduce adjustable quantization steps. global and d are two learnable parameters that quantifying y globally and channel-wisely, respectively. And different RD trade-off's can be achieved by adjusting the magnitude of global More details are introduced in Appendix Sec. A
Fig. 2 also presents MACs of each module with the 1920 x 1088 input image. Obviously, most computational costs come from the encoder and the decoder. Hence, we focus on accelerating these
3

---

Published as a conference paper at ICLR 2023
encoder
C. XXXf166 W//66 W/16.
Residual block
Depth Conv block
xx/2X x2
Ca R/4xx//4 x4/4
C x X//XX///
C #/8 86766
C x////xX
3 W
x xxxx2/2x0
Conv
imput image
Residual block
Depth Conv
Residual blockk
Depth Conv
Residual block
Conv
Conv
Depth-conv
Depth
Mask
Leaky ReLU
Conv
decoddr
C. f1/16 W/16
C. G1/111 x W 1161
Conv
Mask
xxxx222 x
x1 W//
Cg xxx//4440 XW/
G2 xx////A xW/
C, xxx H//8 W/8
Cy x 8r/8/
Leaksy ReLU
Conv
reconstruction
3x
Sub- Sub- -pieep cano
Depth Conv
Residual block
Depth Conv
Residual block
Depth Conv
Residual block
Mask
Mask
Conv 4
Depth
Mask 4
Figure 3: The architectures of our encoder and decoder. All mask layers will be merged into conv.
layers after training. For simplification, we omit Leaky ReLUs within the Depth- Conv block.
N x
Ours
12
Conv 1: N2 x N; x x
Conv M.M, M, x3x3
Mask 1: N,
N2
x
gradient
2.5
N2 X
Leaky ReLU
Leaky ReLU
Merge
N, x W
Ny x X.
Conv 2: M, N, x3x3
Conv 2: M2 x Ng x x
0.5
N₁ x W
Na x
1.5
Pruning
Pruned
Figure 4: (a) presents our pruming method. The mask layer is inserted between two conv. layers. These two cumbersome conv. layers will be transformed into efficient ones by merging the mask layer into them. (b) compares the gradient
8L-perae of different sparsity loss functions. dr
two modules, while keeping other parts' architectures unchanged. Fig. shows the detail structure of these two modules. Different from recent SOTA models (Zhu et al. 2022, Qian et al., 2022
Zou et al. 2022), we adopt residual blocks and depth-wise convolution blocks (Liu et al. 2022
rather than Transformer. Hence, ours is more GPU-friendly and enjoys lower latency. C1. C2. C3.
and C4 are pre-defined numbers for different computational budgets. Three models are proposed with channels 664, 64, 128, 192] (Small), [128, 128, 192, 192] (Medium), and [192, 192, 192, 192] (Large). Tab. 3 in the appendíx summarizes models' #Params and MACs. Note that Large is our cumbersome teacher model while Medium and Small are efficient student models.
3.2 IMPROVE THE STUDENT BY MASK DECAY
Due to the limited capacity, the student performs badly if training from scratch (Zhu et al., 2022). How improve the student is crucial but rarely studied problem for neural image compression. In this paper, we claim that reusing the teacher's 's parameters is helpful. To achieve it, we propose mask decay to transform parameters an end-to-end manner.
Compress one layer's representation by mask decay. As showed in Fig. 4a mask layer is in- serted between two layers to reduce the representation's channels (from N₂ to N,). This mask layer contains N2 parameters to prune corresponding channels. In the forward computation, the represen- tation multiplies these parameters channel-wist wisely. During training, these parameters will become sparse by optimizing Cony #1 and #2 are pruned. And the mask is eliminated by multiplying itself into Conv #2.
sparsity loss. After training, indicated by the mask, useless parameters in
To sparsify parameters in the mask layer, design novel sparsity regularization loss. Most previous works (Li et al. 2016 Wen et al.] [0166; Zhang et al.) 2021) adopt LI-norm (mml or L2- norm & m|ll)) based functions. Note that with the gradient descend, m is updated by sign(m) and
4

---

Published as a conference paper at ICLR 2023
Enc 0
Dec
Enc
Dec
Enc 0
Dec
Enc 0
Dec
Enc
Enc 2
1
Enc
1/
1
Figure 5: Illustration of compressing encoders multi-times learn residual representations progres- sively. "Enc", "Dee"", and "E" denote the encoder, the decoder, and the entropy module, respectively.
m, respectively. Miserably, both of them fail for neural image compression models (cf. Fig. 7 We argue that, for parameter with a positive value, the gradient derived by the LI-norm is a constant without considering its own magnitude. And for the 1.2-norm, when the parameter approaching zero, the gradient is too small to sparsify the mask. In addition, both of them have no stationary points when parameters are positive. To alleviate these problems, We design a novel sparsity regu- larization loss from the gradient view. As showed in Fig. 4b the gradient is defined by
d dLsparse dr
(1)
where x 2 0. For x: > 1, the parameter with a large magnitude suffers a large gradient to decay, which likes the L2-norm. And for S x < 1, a smaller parameter enjoys a larger gradient to approach zero fast. In addition, there is a stationary point at 1, 1, which means Bhaperse x==1 When it comes to 1.1 and 1.2, there always positive gradient for x > O. To make the model converge to stable, it needs negative gradient from the task loss to balance this positive gradient. Thanks to our sparsity loss, only small negative gradient is needed when æ is close 1. Leparee can be calculated by integrating Equation ! which is
Or
+ I,
if 1 x 21, i 11
Lsparse (x)
x 1,
For training, inspired by AdamW W (Loshchilov & Hutter 2018), we decouple Lsparse from the task loss LRD as weight decay. Therefore, the task loss is not affected by our sparsity loss explicitly. So we name our algorithm as mask decay. For the mask m, the updating formula is
D.sparse θLnD m, 7//m 1| -7 0m, 0m,
m++1 m. Ōm,
(3)
where y is the learning rate and r the rate for mask decay.
Mask decay for our encoder and decoder. As illustrated in Fig. to transform the teacher into the student, insert mask layers in some positions if necessary. Our principle is to insert mask layers as few as possible. In one residual block, Mask #1 and #2 are inserted to prune channels inside and outside the residual connection, respectively. In the depth-conv block, Mask #1 determines channels of the depth convolution layer, while Mask #3 prunes representations between Conv #3 and #4. Note that the output channels of these two blocks are shared, so Mask #2 and #4 in the depth-conv block share weights with Mask #2 in the residual block. And we place Mask #2 and #4 behind the residual adding to guarantee the identity gradient back paek-proppgation.
Training with mask decay. The training begins with pretrained teacher model. Then, we insert mask layers and initialize them with The whole network optimized by minimizing where À steers the trade-off between the rate It and the distortion D. Our masks are decayed before each iteration, likely weight decay (Loshchilov Hutter, 2018 During training, we stop mask decay for one mask if it is sparse enough. When all masks meet the sparsity requirements, the whole sparse network can transform into the student by merging masks, meanwhile keeping the network functionality unchanged. At last, the student will be finetuned for few epochs.
CRD =
3.3 THE SCALABLE ENCODER
At last, we advocate the scalable encoder in neural image codecs. A trivial solution for the scalable encoder is directly training separate small encoders with uniform decoder. For specific encoding
5

---

Published as a conference paper at ICLR 2023
Tecnick
Kodak
38.5
37.5
PSNRR
36.5
swint-hyperprior
SlimiHyeer
swint-dharm
MiinCAE-SimmConeettSSiiti
Entroformer
Entroformer
35.5
VTM 17.2
hyperpiop
VTM17.2
STF (CNN)
STF (Transformer)
+TST (Transformer)
swint-charm
34.5
Scalable FVC [ours]
Scalable-EVC (ours)
EVC (ours)
EVC (ours)
BPP
BPP
(a)
(b)
Figure 6: Rate Distortion curves for Tecnick and Kodak. Our models are on-par with swint (Zhu
al. 2022). Entroformer (Qian et al. 2022), STF (Zou et al. 20222, and outperforms the traditional
method VTM. Note that our models are dramatically faster than these methods (cf. Tab. T
complexity, the corresponding number of encoders are chosen encode together, and then best bitstreams will be sent to the decoder. However, these separate encoders suffer from the homoge- nization problem. We propose residual representation learning (RRL) to mitigate this problem. As illustrated in Fig. 5 we compress the cumbersome encoder several times learn the residual rep- resentations progressively. In each step, only parameters in the compressing encoder are optimized. That makes previous encoders generate decodable bitstreams. For specific encoding com- plexity, the first & encoders are applied. Our RRL encourages the encoders' diversity. Both RRL and mask decay treat the cumbersome teacher as a reference, which makes the training more effec- tive. SlimCAE (Yang et al., (021)) uses slimmable layers (Yu & Huang 2019) to obtain an image compression model with dynamic complexities. Our framework is more simple implement and achieves superior RD performance. CBANet (Guo et al. (2021) proposes multi-branch structure for dynamic complexity. It can considered as our baseline. More details are Appendix C
EXPERIMENTS
Training & Testing, The training dataset is the training part of Vimeo-90K (Xue et al. 2019) septu- plet dataset. For fair comparison, all models are trained with 200 epochs. For our method, costs 60 epochs for the teacher to transform into the student with mask decay, then the student is finetuned by 140 epochs. Testing datasets contain Kodak (Franzen 1999). HEVC test sequences (Sharman & Suchhring 2017), Tecnick (Asuni & Giachetti] (014). BD-Rate -ate [Bjontegaar]] (2001) for peak signal- to-noise ratio (PSNR) versus bits-per-pixel (BPP) is our main metric to compare different models. More details about experimental settings are introduced in Appendix Sec. D
4.1 COMPARISON WITH STATE-OF-TIEE-AR
Fig. compares our models with other SOTA models on Tecnick and Kodak. Our large model ("EVC (ours)") outperforms the SOTA traditional image codec ("VTM 17.2") and is on-par with other SOTA neural image codec models. Note that EVC handles different RD trade-offs by only one model, while other SOTA methods train separate models for each trade-off. Tab. compares latency of different models. For Entroformer (Qian et al.) 2022) and STF (Zou et al. 2202). measure the latency by running their official code. Note that the latency contains arithmetic coding, but ignores the disk 1/O time. Swin'T (Zhu et 2022 also compared Appendix Tab. 6 Our models surpass these SOTA models with a significant margin. Our large model runs at 30 FPS on A100 for the 768 x 512 inputs, while our small model even achieves 30 FPS for the 1080P inputs. For more previous but high-impact neural image codecs (c.g., Cheng2020 (Cheng et al. 2020) and Xie2021 (Xie et al. 202T)), we make comparisons Appendix

---

Published as a conference paper at ICLR 2023
Table 1: Latency (ms) comparison. Lower means better. Entroformer (Qian et al., 2022) and STF (Zu et al. 2022 are the SOTA methods. 'OM' means out of memory. Note that 2080Ti and A100 have 2GB and 80GB memory, respectively. Bold denotes running over 30 FPS.
STF
EVC Medium
Resolution
GPU
Type
Entroformer
Transformer
CNN
Large
Small
encoding decoding
OM
176.3
158.5
63.0
44.7
28.4
2080Ti
OM
202.3
210.2
41.1
32.4
24.4
768 x 512
encoding
816.8
115.9
96.4
21.1
19.8
17.7
A100
decoding
4361.9
143.2
118.0
19.1
17.1
15.6
encoding
OM
576.0
456.0
305.3
181.5
90.9
2080Ti
decoding
OM
531.7
652.0
179.2
118.1
73.2
1920 x 1080
encoding
7757.4
355.6
278.1
84.2
56.3
31.4
A100
decoding
OM
354.8
281.7
60.2
46.5
29.7
Table 2: BD-Rate (%) comparison for PSNR. L, M. and S denote Large, Medium, and Small,
respectively. The anchor is VTM. Lower BD-Rate means better. MACs is for 1920 x 1088 imputs.
HEVC
Enc
Dec
MACs (G)
Method
Kodak
C
Avg.
I.
I.
1165.39
Baseline
-5.2
-0.9
-0.9
-3.5
-2.63
-1.1
Baseline Mask Decay
-2.8
2.2
2.5
0.0
0.48
1.1
878.37
-4.5
-0.2
0.2
-3.0
-1.88 (76%))
-0.4 (68%1)
Baseline Mask Decay
0.3
7.1
7.0
2.8
4.30
4.0
688.64
-1.9
2.9
3.5
0.3
1.20 (45%))
1.6 (47%1)
Baseline Mask Decay
-2.1
3.1
3.6
1.1
1.43
2.1
M
885.12
-4.0
0.7
0.8
-1.6
-1.03 (60 (60%)) t)
0.2 (59%1)
Baseline Mask Decay
2.1
9.5
10.7
7.7
7.50
7.0
700.38
-0.4
5.6
6.9
3.3
3.85 (36% t)
4.0 (37%))
Baseline Mask Decay
-0.8
4.7
5.0
1.6
2.63
2.9
M
598.10
-3.1
1.6
1.5
-0.8
-0.20 (54 f)
0.9 (50 % t)
Baseline
6.3
14.7
15.6
10.6
11.80
10.0
S
223.63
Mask Decay
2.6
9.5
10.6
6.5
7.30
(31 1)
6.9 (28%
4.2 MASK DECAY
Previous work Zhuu et al. 2022 finds small models suffer from poor RD performance. This problem alleviated by our mask decay. We accelerate the encoder and the decoder respectively to study more cases. Note that the large model is the teacher and the decay rate is 4e 5. Fig. Ia and Tab. P summarize experimental results. In Tab, 2 we also calculate the relative improvement percentage. For example, if RD-Rate of the baseline student, our student, and the baseline teacher 1.1-(-0.4) 1.1%, -0.4%, and -..1%, respectively. The relative improvement I.1((-1.) 68%. This metric indicates how much the performance gap between the student and the teacher our method narrows. RD curves are shown in Appendix Fig. 17 From experimental results, we conclude that:
Students improved by mask decay with significant margin. For medium and small models, relative improvements are about 50% and 30%, respectively. This demonstrates the effectiveness of our method and the student benefits from reusing the teacher's parameters. Compared with SwinT-Hyperprior in Fig. Ial our EVC achieves a superior trade-off between the RD performance and the complexity. In addition, our models enjoy small sizes.
The encoder is more redundant than the decoder. Note that our encoder and decoder are symmet- ric. We surprisingly find compressing the encoder enjoys less RD performance loss, compared with the decoder. With the large decoder, compressing the encoder from Large to Small loses 2.7% BD-Rate on Kodak. On the other hand, only compressing the decoder loses 5. % BD-Rate.

---

Published as conference paper at ICLR 2023
Ours
Sparsity (8)
101
105
(a6) Agecidds
2055
50
86-5
4e-5
265
bascline
27
-1e.5
50-6
60
5
54..5
+2
50
-1.6e4
g0 sart
Rate
30
10.0
9.7
20
19.0
10.0
epoch
100
100
opach
(a)
(b)
(c)
Figure 7: Ablation studies for different sparsity losses. We adjust the decay rate 1) for each criterion. This figure is best viewed in color and zoomed in.
Appendix Fig. 20 visualizes our models' reconstructions. Reconstructions of different capacities models consistent. Mask decay does not result in extra obvious artifacts. And our encoding and decoding time are dramatically shorter than VTM's.
4.3 THE SCALABLE ENCODER
Fig. 1b presents results for the scalable encoder. First, training separate encoders suffers from poor performance, which is due to the homogenization problem. Then, with our framework, training these encoders one-by-one to bridge the residual representations greatly improves the performance. Our RRL encourages these encoders' diversity. Finally, the mask decay treats the cumbersome encoder reference and makes training more efficient. More details can be found in Appendix Sec. C
Fig. shows RD curves for our Scalable-EVC. Ours outperforms SlimCAE (Yang et al., 2021) significantly by 15.5% BD-Rate. Note that SlimCAE is powerful model equipped with hyper- prior (Ballé et al. 2018), conditional convolutions (Choi et al. 2201)) and an autoregressive context model (Minnen al. 2018). Overall, Scalable-EVC is on-par other SOTA models.
4.4 ABLATION STUDIES
Sparsity losses and decay rates. Fig. summarizes experimental results for mask decay with dif- ferent sparsity regularization losses and decay rates. For sparsity, 0% and 100% denote the teacher's and the student's 's structures, respectively. We adjust the decay rate 7) for each sparsity loss to get reasonable sparsity decay curves. BD-Rate is calculated over VTM on Kodak. For our sparsity loss, all decay rates achieve better RD performance compared with the baseline. A suitable decay rate (4e 5) achieves the best Rate. A large decay rate (8c 5) results in fast decay which destroys the model's performance in early epochs. This model suffers from the insufficient decay process, and the performance is easily saturated. Small decay rates (le 11e-5 and 2c waste too many epochs for mask decay and the model lacks finetuning. For LI, the model suffers from poor performance with both large and small decay rates. With a suitable rate (le 5), it is still worse than ours with significant margin. For L2, we find it hardly works because the gradient vanishes when the mask is close to zero.
Discussions for important channels. In the field of model pruming, many works (Li et al. 2016 He et al. 2019 Liebenwein et al. 2019 Molchanov al. 2019) study how identify unimportant
channels for pruning. However, recent works (Liu et al. 2018 Li Li al. 2022b) indicate randomly pruning channels results in comparable or even superior performance, which demonstrates there may be importance difference among all channels. Previous works study mainly semantic tasks. We want to extend this conclusion for low-level tasks, i.c., image compression.
First, we notice that our mask decay selects similar channels with different decay rates in Figure 7a We use C, to represent the set of chosen channels, where E F = {le 5, 2e - 5, 4e 5, Se 5 means the decay rate 7). We check masks in the first two blocks, and #channels of the teacher and the student are 192 and 64, respectively. Hence, about 33% channels need to be kept after pruning. However, find Ove CC and UierC, contain 23% and 43% channels, respectively. Are these
8

---

Published as a conference paper at ICLR 2023
90
Intersection
B Union
28
baseline
80
75
74
74
Raw
Percentage (%6
70
23
A1B
60
so
43
45
18
40
30
13
10
24
23
25
8
3
A1B A2B
A18 A2B F
A18 A2B
A18 A2B
0 50 100
150
200
BosBB...assk]Resee Re... m.a.k.22..k2pppp. DCB. DCB.2.c.m........... maskk
DCB.2.mask3
epoch
(a)
(b)
Figure 8: (a) presents the percentage of channels w.r.t different mask decay manners in different
layers. F, A ATB, and A2B mean all channels are free, avoiding the first block, and avoiding the first 2 blocks, respectively. Numbers of the intersection of AkB (k 1) and F are small, which means our AMD indeed avoids channels in F. (b) presents the BD-Rate over VTM on Kodak w.r. training epochs. Raw denotes the curve for 4 in Figure 7a
chosen channels important? Our answer is negative. To verify our opinion, we propose mask decay with avoiding specific channels. We divide all channels into two groups: freedom FF and avoidance A. To avoid channels in A, we set a large decay rate for them. Therefore, the loss of our Avoidable Mask Decay (AMD) is
CAMD( = ny xe (x) + ma xxxxxxxxppp Esparse
where connittion is 1 if the condition is true, O otherwise. Defaultly, we set 1he 10ny. Hence, avoidable channels enjoy large decay rate approach zero quickly. We conduct two experiments that avoid selecting channels in Uie C, for the first one two blocks, respectively. Figure Sa illustrates our AMD indeed avoid channels. Specifically, we calculate CARB nCr and AB U where l 1,2 denotes avoiding channels in the first & blocks and CF UiEFC, At most 1% channels are chosen by both AID and the first block, and A2B avoids channels of F in both two blocks. Figure 8b shows the BD-Rate for each epoch. From it, we find AMD results in comparable or a little bit superior performance compared to the raw mask decay.
Overall, if some channels are important, mance. That demonstrates there are no important channels. For future works, this conclusion guides us pay more attention the optimization process rather than the importance of each channel.
avoid these channels to achieve comparable perfor-
CONCLUSIONS AND FUTURE WORKS
We proposed EVC for image compression. With 768 x 512 inputs, EVC is able at 30 FPS, while the RD performance is on-par with SOTA neural codecs and superior to the SOTA tradi- tional codec (H.266/VVC). With adjustable quantization step, EVC handles variable RD trade-off within only one model. These three signs progress make a new milestone in the field of image codec. By reducing the complexities of both encoder and decoder, small model is proposed, which achieves 30 FPS for 1080P inputs. We proposed mask decay for training this small model with the help of the cumbersome model. And novel sparsity regularization loss designed alleviate the drawbacks of L1 and L2 losses. Thanks to our algorithm, our medium and small models are im- proved significantly by 50% and 30%, respectively. At last, the scalable encoder advocated. With only one decoder, there are several encoders for different complexities. With residual representation learning and mask decay, the scalable encoder narrows the performance gap between the teacher. And our scalable EVC outperforms previous SlimCAE by significant margin.
We believe this paper will inspire many future works. First, we point out the encoder more redundant than the decoder. Hence, improving the decoder is promising. Second, this paper mainly focuses on the pre-defined student structure. How to determine the complexity of each layer still an open problem for image compression models. Third, it is also encouraged to find more powerful pruning algorithm. But according to our AMD, we advise paying more attention to the optimization process rather than the importance criterion for each filter.

---

Published as a conference paper at ICLR 2023
REFERENCES
Nicola Asuni and Andrea Giachetti. TESTIMAGES: a large-scale archive for testing visual devices and basic image processing algorithms. In Proceedings of the Smart Tools and Apps for Graphics, pp. 63-70, 2014.
Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compression. In International Conference on Learning Representations, pp. 1-27, 2017.
Johannes Ballé, David Minnen, Saurabh Singh. Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. In International Conference Learning Represen- tations, pp. 1-23, 2018.
Johannes Ballé, Nick Johnston, and David Minnen. Integer networks for data compression with latent-variable models. In International Conference on Learning Representations, pp. 1-10, 2019.
Gisle Bjontegaard. Calculation of average PSNR differences between RD-curves. VCEG-M33,
2001.
Benjamin Bross, Ye-Kui Wang, Yan Ye, Shan Liu, Jianle Chen, Gary Sullivan, and Jens-Raines Ohm. Overview of the versatile video coding (VVC) standard and its applications. IEEE Trans- actions Circuits and Systems for Video Technology, 31110)3373-3774, 2021.
Tong Chen and Zhan Ma. Variable bitrate image compression with quality scaling factors. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2163-2167, 2020.
Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. Proceedings the the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 7939-7948, 2020.
Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Variable rate deep image compression with conditional autoencoder. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3146-3154 2019.
Ze Cui, Jing Wang, Shangyin Gao, Tiansheng Guo. Yihui Feng, and Bo Bai. Asymmetric gained deep image compression with continuous rate adaptation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10532-10541, 2021.
Xiaohan Ding, Guiguang Ding, Jungong Han, and Sheng Tang. Auto-balanced filter pruning for efficient convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, pp. 6797-6804, 2018.
Xiaohan Ding, Tianxiang Hao, Jianchao Tan, Ji Liu, Jungong Han, Yuchen Guo, and Guiguang Ding, ResRep: Lossless enn pruning via decoupling remembering and forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4510-4520, 2021.
Rich Franzen. Kodak lossless true color image suite. 1999.
source: http://hOk..uggrphhiskkkokkk 4(2),
Jinyang Guo, Dong Xu, and Guo Lu. CBANet: Towards complexity and bitrate adaptive deep image compression using single network. arXiv preprint arXiv:2105.12358 2021.
Dailan He, Yaoyan Zheng, Baocheng Sun, Yan Wang, and Hongwei Qin. Checkerboard context model for efficient learned image compression. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14771-14780, 2021.
Dailan He, Ziming Yang, Weikun Peng. Rui Ma, Hongwei Qin, and Yan Wang. ELIC: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In Proceedings of the IFFF/CVF Conference on Computer Vision and Pattern Recognition, pp-
5718-5727, 5727, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE International Conference on Computer Vision, pp. 770-778, 2016.
10

---

Published as a conference paper at ICLR 2023
Yang He, Ping Liu, Ziwei Wang. Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4340-4349 2019.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.025:: 2015.
Jun-Hyuk Kim, Byeongho Heo, and Jong- Seok Lee. Joint global and local hierarchical priors for learned image compression. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6002, 2022.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv 2016.
Jiahao Li. Bin Li, and Yan Lu. Hybrid spatial-temporal entropy modelling for neural video compres- sion. Proceedings of the 30th ACM International Conference on Multimedia, pp. 1503-1511, 2022a.
Yawei Li, Kamil Adamczewski, Wen Li, Shuhang Gu, Radu Timofte, and Luc Van Gool. Revisit- ing random channel pruning for neural network compression. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 191-201, 2022b.
Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks. arXiv preprini urXiv:I911. 1.07412, 2019.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pp. 7736-2744, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, pp. 1-21, 2018.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofeer. Trevor Darrell, and Saining Xie.
A ConvNet for the 2020s. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11966-11 1986, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, pp. 1-19, 2018.
Jian-Hao Luo and Jianxin Wu. Neural network pruning with residual-conneetion. and limited-data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 1458-1467, 2020.
David Minnen and Saurabh Singh. Channel-wise autoregressive entropy models for learned image compression. In 2020 IEEE International Conference Image Processing (ICIP), pp. 3339- 3343, 2020.
David Minnen, Johannes Ballé, and George Toderici. Joint autoregressive and hierarchical priors for learned image compression. In Advances Neural Information Processing Systems, vol- ume 31, pp. 1-10, 2018
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11264-11 1104-1127 11272, 2019,
Yichen Qian, Ming Lin, Xiuyu Sun, Zhiyu Tan, and Rong Jin. Entroformer: A transformer-basee entropy model for learned image compression. In International Conference on Learning Repre- sentations, pp. 1-14, 2022.
Karl Sharman and Karsten Suehring. Common test conditions for HM. JC7VC-71100, 2017.

---

Published as a conference paper at ICLR 2023
Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1874-1883, 2016.
Guo-Hua Wang and Jianxin Wu. Practical network acceleration with tiny sets. arXiv preprint arXiv:2202.0788. 07861. 2022.
Guo-Hua Wang, Yifan Ge, and Jianxin Wu. Distilling knowledge by mimicking features. IEEE Transactions Puttern Analysis and Machine Intelligence, 44(1 ((11):8883-819, 2022.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems, volume 29, pp- 1-9, 2016.
Yueqi Xie, Ka Leong Cheng, and Qifeng Chen. Enhanced invertible encoding for learned image compression. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 162- 170, 2021.
Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. International Journal of Computer Vision, 1276 2019.
Fei Yang, Luis Herranz, Yongmei Cheng, and Mikhail G, Mozerov. Slimmable compressive autoen- coders for practical neural image compression. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 4998-5007, 2021.
Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1803-1811, 2019.
Jiahui Yu, L. Yang, N. Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In International Conference Learning Representations, pp. 1-12, 2019.
Yulun Zhang, Huan Wang, Can Qin, and Yun Fu. Aligned structured sparsity learning for efficient image super-resolution. In Advances in Neural Information Processing Systems, volume 34, pp. 2695-2706, 2021.
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 11953-11962, 2022.
Yinhao Zhu, Yang Yang, and Taco Cohen. Transformer-based transform coding. In International Conference Learning Representations, pp. 1-35, 2022.
Renjie Zou, Chunfeng Song, and Zhaoxiang Zhang. The devil is in the details: Window-based attention for image compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17492-17501, 2022.
12

---

Published as a conference paper at ICLR 2023
EVC: Towards Real-Time Neural Image Compression with Mask Decay -Appendix-
A THE FRAMEWORK OF EVC
input image
Dual spatial prior
Encoder 549.926
Hyper Enc
Z
4.37
1920 1080
AE
AE
plobal Gptep
ch
Spatial Prior
44,68 G
AD
AD
AD
Conettucca
Decoder $38.3
Hyper
2
Qutep
28.06 G
Figure 9: The overall framework of EVC.
Our EVC for image compression is illustrated in Fig. Next, introduce each module in detail.
The encoder and the decoder. The structures of these two modules are illustrated in Fig. 3 The block output dimensionalities are predefined by C1, C22 C3, and C4, respectively. Downsampling is implemented by setting the stride as in specific conv. layers, while upsampling is achieved by the sub-pixel convolution (Shi al. 2016). In each residual block, the kernel size of both Conv #1 and #2 is 3 x 3, while Conv #3's is x I. And Conv #1 enlarges the channel number to the output dimensionality.
each depth-conv block, only the depth conv, layer uses 3 conv,, while others adopt 1 x conv. Conv #3 enlarges the channels by 4 times, while Conv #4 converts it back.
The hyper encoder and the hyper decoder. Fig. describes the structures of our hyperprior, Note that we use 3 x 3 conv. in the hyper encoder.
Hyper encoder
Hyper decoder
C. X x W /16
C4 C H/16xx/16
C. x 3/2
C. H/32 3/32
C. x H/64 W/64
SC, 1/16 w/1 W/16
2G, ///11XW/1
C, x W/16
C, x x//1xx/16
C, x H/32 x W/32
G H/64/ 1/66xx//666 /64
Depth Conv
Leaky ReLU
Depth Conv
Depth Conv
Depth Conv
Depth Conv
Depth Conv
conv
conv
z
2
Ostep
(a)
(b)
Figure 10: and present the structures of our hyper encoder and hyper decoder, respectively.
The dual spatial prior. Following He al. (002)) et al. (2022a); Qian et al. (2022). we adopt dual spatial prior to reduce the spatial redundancy. The detail structure is illustrated in Fig. [1] The representation Y split into two parts Y, and Y2 spatially. Y, first coded according to the side information from the hyperprior. Then, given the reconstruction Y, Y, is estimated more effectively. Our dual spatial prior shares similar idea of auto-regressive prior (Minnen et al. 2018)
13

---

Published as a conference paper at ICLR 2023
(P))
Spatial prror
AE
M..2a Qutep)
(1,, dga Opaap)
f
44g
Depth Conv
Stop
Qetp
Split
Spatial prior
Concat
K
Fi
Depth Conv
2C₁
x',)
Depth Conv
2G,
Figure 11: The structure of our dual spatial prior.
that predicts the target conditioning its neighbor. But ours enjoys parallel computation and fast inference speed.
Schemes of models with different capacities. Tab. summarizes three models used in this paper. We also compare the model size and MACs for 1920 x 1088 inputs.
Table 3: Architecture schemes with statistics of models' #Params (MB) and MACs (G). Note that Others include of the hyperprior model, the spatial prior model, and the entropy model. The resolu- tion of the input image 1920 x 1088.
Model
C₁, C2. C3, C4
Encoder
Decoder
Others
Total
#Params MACs
3.19
3.38
10.82
17.38
Large (L)
192, 192, 192, 192
549.92
538.83
76.64
1165.39
#Params MACs
2.08
2.33
10.82
15.23
Medium (M)
128, 128, 192, 192
262.90
258.56
76.64
598.10 10
#Params MACs
0.82
1.14
10.82
12.78
Small (S)
64, 64, 128,
73.17
73.82
76.64
223.63
B MASK DECAY
The pipeline of our model acceleration method. Figure 12 shows how accelerate cumbersome network by our mask decay. First, this cumbersome network is trained and treated as the pretrained teacher. Then, mask layers are inserted the teacher and are optimized become sparse. After training, masks are merged into neighbor layers, so that the cumbersome teacher transforms into an efficient student network.
J
Enc
Dec
Enc
Dec
Enc
Dec
Enc
Dec
A pretrained teacher
Insert masks
Mask decay
Merge masks
Figure 12: Illustration of model acceleration process. "Enc" and "Dec" denote the encoder and the decoder, respectively. And "E" represents the entropy model with the hyperprior model.
Compress one layer's representation by mask decay. In Fig, 4al our mask contains N₂ parameters which are initialized by 1. After training. N, parameters remain to positive while N₂ N. parameters become zeros. Then, conduct steps "merge" this mask. First, for N₂ N.
14

---

Published as a conference paper at ICLR 2023
zero channels, corresponding filters in Conv #1 and #2 are pruned safely. Second, for N, positive parameters, they are multiplied into Conv #2. After these two steps, the mask can be removed. Claim [ guarantees that these two steps will not change the outputs. Hence, sparse mask is able to be eliminated safely.
Claim 1. Our mask can be merged into neighbor convolution layers while keeping the output unchanged.
Proof. As showed in Fig. 4a assume W and by parameters in pruning Conv #1, and W2 and b2 are parameters in pruning Conv #2. The LeakyRel.U applies the element-wise function:
I
if 2 0,
f(x)
(5)
negativee slope x x, otherwise.
Then given the input X E RNxxHXW the forward computation of the pruning model (the left part Fig. 4a) is
Y1 = W1 & X
(Y, E RNXXHXW W1 E RN2xM1x333 b, E 1²²),
(6)
Y2 Y1 O m,
(Y2 E RN m E RN!),
(7)
Y3 f(Y2).
(Y3 E RNSxHXW),
(8)
Y, W. Y: b2.
(Y4 E RNAXHXW W2 RRMXXNX3X33, b2 E RNo),
where denotes the convolution operator, and Oe means the channel-wise product. Note that N. elements of m are positive, while other N2 N, entries are zeros. In the first step, these N2 N. zero channels are pruned safely, and the forward computation becomes
Y = W
X b
(V' E RN.XHXW W' E N.N.X33. bi EER
(10)
Y2 = Y O. m',
(Y:2 E RNNxHXW m'' E RN,),
(11)
Y'S f(Y2),
(Y's E RNNxHXWW),
(12)
Y4- W Y b2
(Y4 E RNXxHXW W/ RNNxN. b2 RNo..
(13)
Then, in the second step, N, positive entries in m' will be multiplied into W2- We have
Y W f(Y) O. m') + b2
(14)
W (S(Y)) C. m b2
(15)
(W2 O m') @ f(Y)) + b2.
(16)
The last equation holds for the linearity property of the convolution operator. Notice that Eq. 16
denotes the computation process in the pruned model (the right part in Fig. 4ap. Hence, the pruned model also outputs Y4 with the input X. That means the outputs are unchanged. By merging the mask m. parameters within Conv #1 and # become (W1, b/) and (Wo..m' b2), respectively. O
Mask decay for our encoder and decoder. How prune the channels outside the residual con- nection is also crucial problem. Fig. 13 shows two manners. Note that Mask #1, #2, and #3 share weights. If we insert Mask #2 and #3 after the residual adding (as in Fig. [3b), the identity output will be degraded by these masks, because most masks are in [0, 1]. Inserting masks as in Fig. 13a mitigates this problem and guarantees the identity gradient baok-prop..
C THE SCALABLE ENCODER
In this part, we introduce more details for different methods to achieve a scalable encoder.
End-to-end. CBANet (Guo et al. 2021 proposed multi-branch structure for scalable network. The computation complexity is dynamic by using different numbers of branches in testing. This method can be considered the baseline in our paper. That is directly training our framework end-to-end (cf. Fig. [4). During training. all encoders generate bitstreams for decoding, and the cor- responding loss is calculated. However, this baseline requires large GPU memory for training and
15

---

Published as a conference paper at ICLR 2023
Residual block
Residual block
Mask 1
Mask 1
Depth Conv
Depth Conv
Mask 2
Mask 2
Depth Conv 2
Depth Conv
3
Mask 3
(a)
(b)
Figure 13: aa and (b) present two manners to insert masks for the block-level dimensionality. better because guarantees the identity gradient back-propagation.
is
Enc 0
Yo
4o
Enc
Yo +
4
Dec
Enc 2
Yo + +
L2
Enc
Yo 43 1/2 ya
4g
Figure 14: naive baseline for the scalable encoder trained an end-to-end manner. Dash lines means detaching gradients.
Enc
Yo
Dec
Enc 1
Dec
Enc
Enc 1
Ensemble
Ensemble
Dec
Enc
Ya
Dec
Enc 3
Y'
Dec
Enc
Enc Ya3
Figure 15: Illustration of training separate small encoders achieve scalable encoder.
16

---

Published as a conference paper at ICLR 2023
Enc 0
%
Dec
Enc 0
@
Dec
Enc 0
Yo
Dec
Enc 0
Yn
- @
Dec
Enc 1
Xi
Enc
y1 Enc
Enc 22
Enc y2
Enc 3
Figure 16: Illustration of training small encoders one-by-one to achieve scalable encoder. In each time, the new small encoder encouraged to learn residual representations.
Table 4: BD-Rate (%) comparison for PSNR on Kodak. The encoder is scalable and the decoder is Large. MACs in the table are for encoding with 1920 1088 resolution, while MACs for other modules is 615.47 G in total. The anchor is VTM.
So
So.1
S,,
S0,1,2,3
L
MACs (G)
73.17
146.34
219.51
292.68
549.92
End-to-end
3.1
2.2
1.8
1.6
Separate
2.8
1.9
1.7
1.5
-1.1
One-by One-by-one
2.8
1.4
0.9
0.6
Ours
1.4
0.4
-0.1
-0.2
performs badly demonstrated by our experiments. End-to-end training requires optimizing all pa- rameters of the whole network. And does not utilize the knowledge in the pretrained cumbersome model. Hence, the training is inefficient.
Separate. Another baseline method is directly training separate small encoders with uniform de- coder. As shown in Fig. [5] four small encoders are trained with the frozen decoder and entropy model. For testing, different numbers of encoders can be chosen for the specific encoding complex- ity. In the right part of Fig. 15 all four encoders are chosen. The ensemble module will receive all bitstreams from encoders and choose the best bitstream to send. Note that the raw image is avail- able in encoding. The ensemble module is able reconstruct each bitstream and compute the RD score according to the raw image. However, these separate encoders suffer from the homogenization problem. With the uniform decoder, these encoders generate similar bitstreams. In Fig. Tb find that adding more encoders improves the RD performance marginally.
One-by-one (RRL). We propose residual representation learning (RRL) to mitigate the homoge- nization problem. As illustrated in Fig. [6 we train small encoders progressively. In each step, a small encoder is added and trained from scratch. Note that other parts frozen. The new encoder is encouraged to learn the residual representations which effectively alleviates the homog- enization problem and enlarges the diversity of these small encoders. Fig. lb shows "one-by-one" outperforms "separate encoders" by a significant margin.
Ours (RRL + mask decay). At last, mask decay further improves the RD performance. As illus- trated in Fig. 5 mask decay directly utilizes knowledge of the cumbersome model. In each step, the small encoder is obtained by pruning from the cumbersome teacher. Here the teacher means the large encoder in Tab. 3 whereas the student means the small encoder. In Fig. 5) the encoder with dash boundaries denotes the teacher encoder equipped with masks. During training, these masks will become sparse so that the network is able to transform into the student structure (i.c., the student encoder). Both RRL and mask decay treat the cumbersome teacher as a reference, which makes the training more effective. Fig. Ib shows "ours" (RRL + mask decay) achieves superior RD performance and significantly narrows the performance gap from the teacher.
Tab. summarizes results for scalable encoders with the large decoder. We also conduct exper- iments for the small decoder, and Tab, 5) summarizes results. In these tables, So..... denotes using i small encoders, while I. means the large encoder.
17

---

Published as a conference paper at ICLR 2023
Table 5: BD-Rate (%) comparison for PSNR on Kodak. The encoder is scalable and the decoder is Small. MACs in the table are for encoding with 1920 1088 resolution, while MACs for other modules is 150.46 G in total. The anchor is VTM.
So
So.1
S0,1,
S0,1,2,3
I.
MACs (G)
73.17
146.34
219.51
292.68
549.92
End-to-end
9.1
8.6
8.2
One-by-one
7.3
6.3
5.8
5.5
4.0
Ours
6.3
5.6
5.0
4.8
D DETAILS FOR EXPERIMENTS
Training. The training dataset is the training part Vimeo-90K (Xue al. 2019)
septuplet dataset.
During training, the raw image is randomly cropped into 256 x 256 patches. For a fair comparison, all models are trained with 200 epochs. For our method, it cost 60 epochs for the teacher to transform into the student with mask decay, then the student is finetuned by 140 epochs. Adam W is used as the optimizer with batch size 16. The initial learning rate is 2e 4, and decays by 0.5 at 50, 90, 130, 170 epochs. The default decay rate y for our mask decay is 4e 5. All models are trained on a computer with 8 V100 GPUs. Our large and small model cost about 33 and 16 hours for training. respectively.
Testing. BD-Rate [Bjontegaard 2001) for peak hignal-to-noise ratio (PSNR) is our main metric to compare different models. We use VTM 17.2 as the anchor, while negative numbers indicate bitrate saving and positive numbers indicate bitrate increase. Testing datasets contain Kodak (Franzen, 1999), HEVC test sequences (Sharman & Suehring. (017), Tecnick (Asuni & Giachetti 2014)) Kodak contains 24 images with the 768 x 512 resolution. And Tecnick consists of 100 images with the 1200 x 1200 resolution. HEVC is video dataset that consists of 4 classes: B (1080P), C (480P), D (240P), E (720P). For testing image codecs on HEVC, only the first frame of each video sequence used. We report BD-Rate on each HEVC class and average all for convenient comparison. For one testing image, we pad zeros on the boundary to make the resolution as multiples of 64. We test the latency on two computers. One is equipped with one 2080Ti GPU and two Intel(R) Xeon(R) E5-2630 v3 CPUs. Another is equipped with one A100 GPU and one AMD Epyc 7V13 CPU. Both encoding and decoding times are gathered. Note that the latency contains arithmetic coding, but ignores the disk I/O time.
E IMPLEMENT DETAILS FOR OTHER IMAGE CODECS
VTM-17.2 is the reference software for the state-of-the-arr traditional codec (H.266/VVC), and it released on Jul 17, 2022. We use to gather evaluation results. We install these two softwares according to the official instructions. The default configuration of VTM is used. All results are gathered by running the following command:
python -m compressai utils. bench vtm [path image folder] -c [path to VTM folder)/cfg/ encoder inntr. wtt.iinttttttttttttt vtm.cfg -b [path to VTM folder]/bin -j 8 -q 24 26 28 30 32 34 36 38 40 42
Entroformer is proposed in Qian et al. 2022). Two series models are released: parallel, non- parallel. We evaluate the RD performance of the parallel model on Tecnick and Kodak, respectively. But when we measure its latency, we find it requires a large GPU memory for generating bitstreams.
VTM 17.2
18

---

Published as a conference paper at ICLR 2023
Table 6: Latency (ms) comparison with SwinT. Lower means better. To compare with SwinT, we test ours on 2080Ti with 768 x 512 inputs, and directly cite their numbers. Note that Swin'T does reconstruct images for encoding, whereas ours reconstructs images.
Swin'T'
EVC Medium
Latency
SwinT-ChARM SwinT-Hyperprior Conv-ChARM Conv-Hyperprior
Large
Small
encoding
135.3
100.5
95.7 264.0
62.0
63.0
44.7
28.4
dccoding 167.3
114.0
219.3
41.1
32.4
24.4
STF is proposed in Zou al. 2022) The authors release two architectures: CNN- model and Transformer based model. But, only two pretrained models for two qualities are available. We evaluate model Tecnick and cite numbers in their paper for Kodak.
SwinT-ChARM is proposed in Zhu et al. (2022). Because the authors did not release their code, cite numbers in their paper and the webpage The latency is reported in their paper, and Tab. compares ours with theirs. Our models achieve lower latency.
ELIC is proposed in He al. (2022). Uneven channel-conditional adaptive coding is proposed in this work. [e et al. 2022) designed two models ELIC and ELIC-sm, where ELIC-sm is adapted from ELIC with a slim architecture. Because the authors did not release their code, we make com- parisons with numbers in their paper.
Cheng2020 is proposed in Cheng et al. (2020). Attention modules and Gaussian Mixture Model are proposed to improve the RD performance. Due to its high impact, it is always considered as the baseline method for neural image codecs. However, the autoregressive module is adopted in their raw model. And inference on GPU is not recommended for the autoregressive models. When testing on CPU, it takes more than one second process one image. He et al. (2021) proposed the checkerboard module to mitigate this problem. We compare ours with this model in Tab.
Xie2021 is proposed in Xie et al. (202T). The invertible neural networks are proposed and used the stronger backbone. But the same as Cheng2020, Xie2021 I is also equipped with the autoregressive module. It suffers from high latency due to this module. We compare ours with this model in Tab.
Minnen2018 and Minnen2020 are proposed in Minnen et al. (018) and Minnen & Singh 2020). respectively. The autoregressive context model was first introduced in Minnen et al. (2018) to boost the RD performance of neural image codecs. Minnen & Singh (2020) further improved it by con- sidering channel-wise redundancy. Both two works have a significant impact in the neural image compression and are always considered as the baseline method for comparisons. We also compare ours with them in Tab. 7
Ballé2018 is proposed in Ballé et al. (2018). The hyperprior structure was introduced, which is a milestone in the field of image codec. Most latter models adopt this structure, including ours. Without context module, it achieves a high inference speed. But without using recent proposed techniques, it suffers from poor RD performance compared with recent SOTA models.
F MORE EXPERIMENTAL RESULTS
Comparison to more methods. He et al. (2022) proposed two models for efficient learned image compression, namely ELIC and ELIC-sm, respectively. However, they did not release their codes and models and it is difficult for to make fair comparison. We follow their testing manner to compare numbers in their paper. First, in their supplementary material, they mentioned the de- terministic inference mode not enabled when testing the model speeds. We test latencies in modes, respectively. We find that non-determinism reduces the la- tency by average 16% but easily results in catastrophic failures for decoding (Ballé et al. 2019). Second, different from testing end-to-end latency with arithmetic coding in our main paper, here we test each module's latency as that in He et al. 2022). Hence, without considering arithmetic coding,
determiniiticcnoo-dterrmmmi
htttos://aittubb.ccm//ooooooss httess///geenrrriee.eett///
DDN66jHHKK
19

---

Published as a conference paper at ICLR 2023
Table BD-Rate (%) and inference latency (ms) comparisons. BD-Rate computed from PSNR- BPP curve of Kodak over the anchor VTM. Lower BD-Rate means better. Our models were tested in deterministicmoon-deeemee
modes, respectively. Other methods were tested by He et al. (2022) with the non-deterministic mode.
Inference Latency
GPU
Model
BD-Rate
Tot.
YEne.
YDec. 7Enc. ZDec. Param.
Ballé2018 BBallé et al.
2018
40.85
30.3
11.9
16.1
1.2
Minnen2018 (Hc ct al
2021
20.00
37.6
11.6
17.3
1,5
5.7
Cheng2020 (Hc ct al. 2021
3.89
90.8
36.8
44.2
0.9
1.2
7.7
TITAN Xp
Minnen2020 (Minnen & Singh
2020
1.11
77.7
10.6
15.4
1.4
1.5
48.8
Xie2021 Xie et al.
2021
-0.54
10a
58.0
162.2
1.3
10
ELIC-sm [e el 2022
-1.07
41.5
12.6
17.4
1.1
1.4
9.0
ELIC [ee al. 2022)
-7.88
82.4
31.2
38.3
1.2
1.6
10.1
EVC-SS (deterministic)
6.9
22.1
4.2
7.4
1.4
5.4
3.7
EVC-SS (non-determiniiticc)
6.9
17.9
3.8
5.7
0.9
3.7
3.8
EVC-MM (deterministic)
0.9
31.4
9.1
11.9
1.4
5.3
3.7
2080Ti
EVC-MM (non-deterministic)
0,9
26.5
8.3
10.2
0.9
3.6
EVC-LL (deterministic)
-1.1
44.5
15.5
18.6
1.4
5.3
3.7
EVC-LL (non-deterministic)
-1.1
38.4
14.1
16.2
0.9
3.6
3.6
Table 8: Speed (megapixel/s) comparison. Higher means better. Entroformer (Qian et al. STF (Zou et al. 2022) are the SOTA methods. 'OM' means out of memory. Note that 208011 and A100 have 12GB and 80GB memory, respectively.
2022 and
STF
EVC Medium
Resolution
GPU
Type
Entroformer
Transformer
CNN
Large
Small
encoding decoding
OM
2.23
2.48
6.24
8.80
13.85
2080Ti
OM
1.94
1.87
9.57
12.14
16.12
768
encoding
0.48
3.39
4.08
18.64
19.86
22.22
A100
decoding
0.09
2.75
3.33
20.59
23.00
25.21
encoding decoding
OM
3.60
4.55
6.79
11.42
22.81
2080Ti
OM
3.90
3.18
11.57
17.56
28.33
1920 1080
encoding
0.27
5.83
7.46
24.63
36.83
66.04
A100
decoding
OM
5.84
7.36
34.45
44.59
69.82
the latency lower than that in our main paper. Third, unfortunately, we do not have TITAN X GPUs, and we test our models on 2080T
Tab. summarizes results. For our models, "S", "M", and "I." denote small, medium, large, respec- tively. We use two letters to represent the encoder and the decoder, respectively. Compared with previous SOTA methods (Ballé2018, Minnen2018, Cheng2020, Minnen2020, Xie2021), our large model ("EVC-LL") achieves better RD performance and inference latency. And it is also compa- rable to ELIC-sm. Note that ours enjoys a single model to handle all RD trade-offs, while ELIC trained different models for each trade-off.
Speed (megapixel/s) comparison. Tab, 8, we also compute models' speeds as megapixel/s ac- cording Tab. Our models outperform previous models for this metric, too.
Comparison for our different models. Fig. 17 presents RD curves on Kodak. Our mask decay "MD" improves our baseline models significantly. In these figures, "L", "M", and "S" denote Large,
http::::/wwwwwiiiii/.m com wtw.://wwwwwoooooooo//o/oooaaeaa
compare-20
20

---

Published as a conference paper at ICLR 2023
Accelerate Encoder
Accelerate Decoder
Accelerate Both
38
37
37
-EXC-4I [bookinn)
-BC Ibaseine)
EVC 55 Ibase inc)
-EVC S1 (MD)
-Evt-as (MB)
33
-FeCAi (saselrn)
EVC (M4))
33
-ENCUM (M0))
BVC 1M [seectinc]
33
-EVCARR (MC)
(baselino)
-EVC-LL
92
DPP
D.7
BPP
(a)
Figure 17: RD curves on Kodak for our models with different complexities. "MD" denotes mask decay. @a) and (b) show curves for accelerating only encoder or decoder, respectively. (c) presents curves for accelerating both encoder and decoder.
Table 9: BD-Rate (%) comparison for PSNR CLIC2021 and Tecnick. L,, M, and S denote Large, Medium, and Small, respectively. The anchor VTM. Lower BD-Rate means better. MACs for 1920 1088 inputs.
Enc
Dec
MACs (G)
Method
CLIC2021
Tecnick
L
1165.39
Baseline
-0.7
-2.0
Baseline
M
L
878.37
Mask Decay
0.6 (24%1)
-0.3 (15%1)
Baseline
1.9
S
L
688.64
Mask Decay
1.9 (48%11
0.2 (44%))
Baseline
1.0
L
M
885.12
Mask Decay
0.9 (48%1)
-0.3 (43%1)
Baseline
5.4
L
S
700.38
Mask Decay
4.3 (36%))
2.7 (36%))
Baseline Mask Decay
4.6
2.6
M
M
598.10
1.6 (57%)1
0.7 (11%)
Baseline
11.6
9.9
S
S
223.63
Mask Decay
8.5 (25%1)
7.6 (19%))
Medium, and Small, respectively. And "EVC-SL" means the encoder is small and the decoder is large.
Tab. presents these model's results on CLIC202 and Tecnick. Mask decay outperforms the baselines by significant margins.
Sparsity losses and decay rates, For easier to read, we replicate Fig. for larger sizes. Figs.
and 19 are these replications.
Disseusions for important channels. Tab. 10 presents the specific numbers in Fig. 8a There are four masks in the first two blocks. We analyze the first mask (ResB. 1mmakk1) for example. Mask decay is required to choose 64 channels from the teacher's 192 channels. With four different decay rates, only 83 channels are considered and they choose 44 channels together. It seems like these channels are important for superior performance. We treat these 83 channels as CF and let our algorithm avoid choosing these channels. A1B (avoid specific channels in the first block) only selects 2 channels in Cr. while A2B selects 3. First, it demonstrates our AMD indeed avoids specific channels. Next, we compare the RD performance of these methods. Surprisingly. AMD
https::/storraee htt.://eee...aaaaa.o..o.ooooooa.
Satoanall_ttttttt_eeeeeee_ee 20011_
zip
21

---

Published as a conference paper at ICLR 2023
Ours
100
Sparsity (96
1e-5
50
2e-5
4e-5
8e-5
0
baseline
27
1e-5
2e-5
DD-Rate (%6)
4e-5
8e-5
17
12
10.0
9.5
9.0
8.3
6.9
150
epoch
Figure 18: Mask decay with sparsity loss (Eq. 2 Different decay rates 7) were tried. This figure is the replication of Fig. 7a
L1
L2
100
100
Sparsity (%)
5e-6
1e-5
Sparsity
50
4e-5
8e-5
2e-5
1.6e-4
baseline
70
baseline
27
5e-6
60
4e-5
1e-5
8e-5
R-Ratte (%6)
22
2e-5
50
+1.e--4
17
Rate
40
36.5
10.8
30
.12
10.8
10.0
20
19.0
10
10.0
200
50
150
epoch
epoch
(a)
(b)
Figure 19: Mask decay with L1 and 1.2 losses. We adjusted the decay v for each criterion.
These two figures are replications of Fig. 19a and 19b
22

---

Published as a conference paper at ICLR 2023
Table 10: The relationship for channels chosen by different settings. "ResB. and "DCB.2" means the Residual block and the Depth Conv block, respectively. Note that they are the first two blocks in our model. denotes the set of chosen channels, and {le - 5, 2e 5, 4e 5, 8e - 5} which A1B and A2B means avoiding specific channels in the first 1 and 2 blocks, respectively. We report the cardinality of each set.
indicates different decay rates in Figure 7a
Mask Layer
Teacher Student
CeeeC UgrCC Cais nCp Caia UC CazB O Or Caze U CF
ResB.I.mask
192
64
44
83
2
145
144
ResB.I.mask
192
64
47
83
0
147
145
DCB.2.maskl
192
64
44
56
143
DCB.2.mask)
768
256
179
328
191
393
12
572
results in comparable performance. That indicates that there are no important channels for superior performance.
G VISUALIZATIONS
Fig. 20 visualizes our models' reconstructions. Comparing our small model (EVC-SS) and large model (EVC-LL), the reconstructions are almost the same. That demonstrates our mask decay does result extra artifacts. Compared with VTM, ours enjoy fast encoding and decoding.
EVC-SS
EVC-MM
EVC-LL
VTM 17.2
The raw image
[0.3829/31.11/883324555
(0.3752/31.98/46.0335
[0.3766/32.02/771144..6 03379//9196/7777./222..11
Figure 20: Visualization of our models' reconstruction. EVC-SS denotes our model equipped with the small encoder and the small decoder, while M and L means medium and large, respectively. Numbers the tuple are BPP, PSNR, the encoding time (ms), and the decoding time (ms), re- spectively. Note that the latency is measured on a computer with 2080Ti GPU. Our models are dramatically faster than VTM.
23


Reviews:
[{"review_id": "s8Yk0ZiJXJ", "reviewer": "Reviewer_7uFq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a real-time neural image compression solution. This approach utilizes the knowledge distillation approach to obtain low-complexity codecs through the mask decay design. Furthermore, the paper also introduces the scalable encoder for neural image compression and achieves the dynamic complexity for different latency requirements. Experimental results on several benchmark datasets are provided to demonstrate the superiority of the proposed approach. In summary, the paper focuses on practical neural image compression based on knowledge distillation and slimmable design.", "summary_of_the_review": "The paper focuses on practical neural image compression based on the KD technique and dynamic complexity. The selection of baseline methods is not convincing and several strong baseline methods are missing. ", "strength_and_weaknesses": "**Strengths**\n\n1. The proposed method is useful for practical neural image compression. The Mask decay solution seems to be reasonable and interesting. \n2. The inference speed reported by this paper is encouraging.  And dynamic complexity is an interesting design in practical applications. \n\n**Weaknesses**\n\n1. Several strong baseline methods[1,2,3] should be discussed and compared. In particular, ELIC[1] method is also proposed for practical image compression and the authors are suggested to make further comparisons. \n\n     [1] He, Dailan, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang. “ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding.” arXiv, March 29, 2022. http://arxiv.org/abs/2203.10886.\n\n      [2] Xie, Yueqi, Ka Leong Cheng, and Qifeng Chen. “Enhanced Invertible Encoding for Learned Image Compression.” ArXiv:2108.03690 [Cs, Eess], August 8, 2021. http://arxiv.org/abs/2108.03690.\n\n      [3] Kim, Jun-Hyuk, Byeongho Heo, and Jong-Seok Lee. “Joint Global and Local Hierarchical Priors for Learned Image Compression.” In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5982–91. New Orleans, LA, USA: IEEE, 2022. https://doi.org/10.1109/CVPR52688.2022.00590.\n\n2. Using the transformer based NIC methods as baseline methods is not a good choice. Most transformer based approaches suffer from the high complexity cost. \n\n3. For the proposed mask decay approach, the improvements on the kodak dataset are not significant. It brings 1-3% bitrate savings. \n\n4. The comparison between the proposed method and SlimCAE is not fair since they use different backbone networks. \n\n5. The novelty of dynamic complexity is not significant as several approaches like SlimCAE and CBA-NET[4] have been proposed. The authors should make further explanations. \n\n     [4] Cbanet: Towards complexity and bitrate adaptive deep image compression using a single network. \n\n6. The training strategy seems a little tricky. For example, how to decide the termination of mask decay. The authors are suggested to provide more details. \n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "h6k9KDH3qPN", "reviewer": "Reviewer_RU83", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores neural image compression and focuses on improving existing methods in several ways:\n\n1. A single model that allows variable bit rate encodings, variable encode compute, and faster run time. Although previous research has explored these items in isolation, I do not know of a model that incorporates all of them.\n\n2. Better model pruning through a novel sparsity loss (\"mask decay\"). This loss (applied via weight decay) leads to better rate-distortion (RD) loss than training models with identical architectures from scratch or by using simpler sparsity losses like L1 and L2.\n\n3. Scalable encoding, in terms of computation, is achieved through residual representation learning (RRL), where the encoder is iteratively refined by learning a mapping from the input image to a residual in latent space (Fig 5). At inference time, the encoder can decide how many of the residual encoders to run to trade off runtime for RD performance.\n\n", "summary_of_the_review": "I'm recommending that this paper be accepted based on promising results from the novel sparsity loss and mask decay. This approach can be applied to other models and appears to give much better results than previous pruning methods applied to compression models.\n\nAs mentioned above, a comparison to ELIC should be added, and a favorable comparison (runtime and RD) would strengthen my recommendation based on the empirical results.", "strength_and_weaknesses": "The paper is well-motivated, and the claims are empirically evaluated and compared against strong/appropriate baselines.\n\nThe mask decay idea, which combines a novel sparsity loss applied via weight decay, addresses a significant problem for neural image compression. Other papers have looked at methods for speeding up neural codecs by reducing the channel depth of conv layers (the primary runtime bottleneck), but the RD penalty is typically quite high. The authors show that mask decay has a much smaller penalty (Fig 1b) and thus provides a new approach to close the runtime gap between learning-based and hand-engineered codecs.\n\nThe main weakness of the paper is empirical. The authors should compare against this paper from CVPR 2022:\n\nELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding\nDailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, Yan Wang\nhttps://arxiv.org/abs/2203.10886\n\nThis paper explores a combination of channel-wise and spatial decomposition for the entropy model that's different than what's used in the paper under review (primarily, it uses five channel-wise groups with uneven depths). The authors show that this allows them to reduce the computational complexity of the analysis and synthesis transforms and achieve a net benefit in terms of RD performance vs. runtime. In particular, I think the results is both faster and gives a slightly better compression rate than what's presented here (it's hard to be 100% sure without overlaying the RD graphs, and runtime comparisons are difficult since different hardware was used (2080Ti vs Titan Xp)).\n\nRegardless, I think there are enough novel ideas with promising impact on the neural image compression subfield that this paper should still be accepted.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VsXDMGbwgzc", "reviewer": "Reviewer_5xdh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an efficient CNN-based neural image compression algorithm. Their efficient CNN framework achieves a high speed while maintaining a high RD performance on par with SoTA works, and also can handle variable RD trade-offs through a single model. They also propose to effectively compress the large teacher model through knowledge distillation using a sparse channel mask technique along with a decaying sparsity regularization. During the mask decay based knowledge distillation, they further apply residual representation learning on encoder for higher scalability.", "summary_of_the_review": "This paper proposes a pure CNN-based efficient neural image compression model and a knowledge distillation algorithm based on mask decay. The network is more efficient and GPU friendly while maintaining an on-par performance compared to recent Transformer based models. The evaluation and analysis have been made thoroughly enough, proving the effectiveness of the proposed algorithm properly. In conclusion, the proposed work successfully solved a hard challenge on efficiency-performance trade-offs in image compression.", "strength_and_weaknesses": "* Strength\n\nTheir CNN only framework is more efficient and GPU friendly, compared to recent Transformer based models, while the performance is on par with more complex SoTA neural image compression models.\n\nWith an adjustable quantization step, their model handles variable RD trade-offs through a single model unlike previous works.\n\nThey successfully enabled knowledge distillation on an image compression model through mask decay with a new sparsity loss that regularizes more effectively than Lp losses.\n\nTheir residual representation learning technique enables scalable encoder design with various complexities while maintaining an overall performance. \n\n\n* Weakness\n\nMore visual results would better present their results, currently none in the main paper.\n\nThere are several points they insist as their technical contributions, which are rather scattered thus hard to get a clear picture. It would be better to clarify each more clearly in an overview.\n\nMask merging can be better clarified within the main paper. Also, does the Fig.4(a)-Pruned and the derivation in Eqs.(5-12) in Appendix B match?\n\nFigure 7 seems to be an important study, but hard to read due to small size.\n\nThe intention of discussions on channel importance at p8 is rather confusing, considering their knowledge distillation mainly targets channel reduction, which was successful according to their experiments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LUz2Ddv7ewp", "reviewer": "Reviewer_Y5zE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an efficient single-model variable-bit-rate network, which is able to run at 30 FPS with 768x512 input images and still outperforms VVC for RD performance. By further reducing both encoder and decoder complexities, the small model even achieves 30 FPS with 1920x1080 input images. To bridge the performance gap between different capacities models, the authors meticulously design the mask decay, which transforms the large model’s parameters into the small model automatically. And a novel sparsity regularization loss is proposed to mitigate the shortcomings of Lp regularization.", "summary_of_the_review": "The proposed method is novel and effective. The reviewer recommends to accept the paper, however, the authors should address the questions regarding the experiments.", "strength_and_weaknesses": "Strength: The proposed method is novel and effective.\n\nWeaknesses: The reviewer has some concerns about the experiments:\n\n(a) The authors should compare the RD performance, speed, etc. of the proposed method with the SOTA neural image compression methods, e.g., Xie et al., Enhanced Invertible Encoding for Learned Image Compression. ACM MM 2021; Cheng et al., Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules. CVPR 2020, etc. \n\n(b) The result of Tecnick should be added to Table 2.\n\n(c) HEVC dataset is a video dataset, which is rarely used for image compression. Did the authors test their image compression methods on all frames of each video in the HEVC dataset? Instead, CLIC is a popular dataset for image compression.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster