Transformer-based Transform Coding
Authors: Yinhao Zhu
Yang Yang
Taco Cohen
Venue: ICLR 2022 Poster (2022)
Keywords: transformer
transform coding
image compression
video compression

Abstract: Neural data compression based on nonlinear transform coding has made great progress over the last few years, mainly due to improvements in prior models, quantization methods and nonlinear transforms. A general trend in many recent works pushing the limit of rate-distortion performance is to use ever more expensive prior models that can lead to prohibitively slow decoding. Instead, we focus on more expressive transforms that result in a better rate-distortion-computation trade-off. Specifically, we show that nonlinear transforms built on Swin-transformers can achieve better compression efficiency than transforms built on convolutional neural networks (ConvNets), while requiring fewer parameters and shorter decoding time. Paired with a compute-efficient Channel-wise Auto-Regressive Model prior, our SwinT-ChARM model outperforms VTM-12.1 by $3.68\%$ in BD-rate on Kodak with comparable decoding speed. In P-frame video compression setting, we are able to outperform the popular ConvNet-based scale-space-flow model by $12.35\%$ in BD-rate on UVG. We provide model scaling studies to verify the computational efficiency of the proposed solutions and conduct several analyses to reveal the source of coding gain of transformers over ConvNets, including better spatial decorrelation, flexible effective receptive field, and more localized response of latent pixels during progressive decoding.


Published as a conference paper at ICLR 2022
TRANSFORMER-BASER TRANSFORM CODING
Yinhao Zhu Yang Yang" Qualcomm Al Research* {yinhaoz, yyangy, tacos]eqti. cacos)@@ta.uailcc...cmmm com
Taco Cohen
ABSTRACT
Neural data compression based nonlinear transform coding has made great progress over the last few years, mainly due to improvements in prior models, quantization methods and nonlinear transforms. A general trend in many re- cent works pushing the limit of rate-distortion performance to use ever more expensive prior models that can lead to prohibitively slow decoding. Instead, we focus on more expressive transforms that result in a better rate-distortion- computation trade-off. Specifically, we show that nonlinear transforms built on Swin-trangn wwin-transformers can achieve better compression efficiency than transforms built on convolutional neural networks (ConvNets)) while requiring fewer parameters and shorter decoding time. Paired with compute-effciiett Channel-wise Auto- Regressive Model prior, our SwinT-ChARM model outperforms VTM-12.1 by 3.68% in BD-rate on Kodak with comparable decoding speed. In P-frame video compression setting, we are able to outperform the popular ConyNet-based scale- space-flow model by 12.35% in BD-rate on UVG. We provide model scaling stud- ies to verify the computational efficiency of the proposed solutions and conduct several analyses reveal the source of coding gain of transformers over Conv- Nets, including better spatial decorrelation, flexible effective receptive field, and more localized response of latent pixels during progressive decoding.
INTRODUCTION
Transform coding Goyal 2001 is the dominant paradigm for compression of multi-media signals, and serves as the technical foundation for many successful coding standards such JPEG, AAC, and HEVC/VVC. Codecs based on transform coding divide the task of lossy compression into three modularized components: transform, quantization, and entropy coding. All three components can be enhanced by deep neural networks: autoencoder networks are adopted as flexible nonlinear trans- forms, deep generative models are used as powerful learnable entropy models, and various differen- tiable quantization schemes
proposed to aid end-to-uu training. Thanks to these advancements, we have seen rapid progress in the domain of image and video compression. Particularly, the hyper- prior line of work (Ballé et al. 2018 Minnen al. 2018 Lee al. 2019 Aguusssoo et al.) 2020) Minnen & Singh, 2020) has led to steady progress of neural compression performance over the past two years, reaching
even surpassing state-of-the-arr traditional codecs. For example, in image compression, BPG444 was surpassed by neural codec in 2018 (Minnen et al. 2018), and (Cheng al. 2020 Xie et al. 2021 Ma et al. 2021 Guo et 2021 Wu al. 2020 have claimed on-par or better performance than VTM (a test model of the of art non- leamndd VVC standard).
One general trend in the advancement of neural image compression schemes to develop ever more expressive yet expensive prior models based on spatial context. However, the rate-distortion improvement from context based prior modeling often comes with a hefty price tag in terms of decoding complexity. Noteably, all existing works that claimed on-par or better performance than
VTM (Cheng et al.) 2020 Xie et al. 2021: Ma al. 202T Guo et al. 2021 on slow and expensive spatial context based prior models.
Wu et 2020) rely
"qqual contribution.
Qualcomm AI Research
initiative of Qualcomm Technologies, Inc.
In the extreme case when a latent-pixee -ient-pixel-eevel spatial autoregressive prior is used, decoding of single 51 568 image requires no less than 1536 interleaved executions of prior model inference and entropy decoding (assuming the latent downsampled by factor of 16x16).

---

Published as a conference paper at ICLR 2022
The development of nonlinear transforms, the other hand, largely overlooked. This leads us to the following questions: can we achieve the same performance as that of expensive prior models expressive transform together with simple prior models? And if so, how much more complexity in the transform required?
by designing
Interestingly, show that by leveraging and adapting the recent development of vision transformers, not only can we build neural codecs with simple prior models that can out- perform ones built on expensive spatial auto- regressive priors, but do so with smaller trans- form complexity compared its convolutional counterparts, attaining a strictly better rate- iustortion-complees trade-off. As can be seen in Figure our proposed neural image codec SwinT-ChARM can outperform VTM-12.1 at comparable decoding time, which, the best of our knowledge, a first in the neural com- pression literature.
-5
BD-ate BD-aat ever VIM-12.1 Kodak
*
SwinT-ChARM
VTM-112.1
Wu et 2020
Guo
Conv Conv-ChARM ChARM
Cheng at al. 2020
10
SwinT-tyyprrprirr (different sizes) Coperiryr (different sizes)
20
Lee el al. 2019
100
150
200
250
300
202
30r
Decoding time (ms)
As main contributions, we 1) extend Swin- Transformer (Liu et al. 2021) to a decoder set- ting and build Swin-transformer based neural image codecs that attain better rate-distortion performance with lower complexity compared with existing solutions, verify its effective- ness in video compression by enhancing scale- space-flow, popular neural P-frame codec, and 3) conduct extensive analysis and ablation study to explore differences between convolution and transformers, and investigate potential source coding gain.
Figure BD-rate (smaller is better) vs decod- ing time. Our Swin transformer based image compression models land in favorable region of rate rate-distortion rere-diitottioncomppaaaee
trade-off that has never been achieved before. See Section for more results and evaluation setup.
2 BACKGROUND & RELATED WORK
The seminal hyperprior architecture (Ballé et al. 2018 Minnen et al. 2018) is a two-level hierarchical variational autoencoder, consisting pair of encoder/decoder ga. Us. and pair of byyer-enccoeerlyyppr-eeoooo
Conv-Hyperprior
h., ft. Given an input image x, pair of latent y = ga (x) and hyper-latent ha (y) is computed. The quantized hyper-latent = Q(z) modeled and entropy- coded with a learned factorized prior. The latent y is modeled with factorized Gaussian distribution p(y 2) = N(u, diag(o) (or) whose parameter is given by the hyper-decoder (μ, or) = (2). The
quantized version of the latent Q(y (1) + fa is then entropy coded and passed through decoder
g. to derive reconstructed image & 9x ( The tranforms Ja Is, ha, hg are all parameterized as
ConvNets (for details, see Appendix A.I
Conv-ChARM MMinnen & Singh 2020) extends the baseline hyperprior architecture with channel-wise auto-regressive model (ChARM? in which latent y is split along channel dimension into S groups (denoted YI...... : ss.. and the Gaussian prior ply, &<s) is made autoregressive across groups where the mean/scale of y, depends on quantized latent in the previous groups ÿ<-- In practice, S 10 provides good balance of performance and complexity and is adopted here.
Spatial AR models Most of recent performance advancements of neural image compression is driven by the use of spatial auto-regressive/contee models. Variants include causal global predic- tion (Guo al. 2021), 3D context (Ma et al. 202T). block-level context (Wu al. 2020), nonlocal context (Li et al. 2020 Qian et al. 2021). One common issue with these designs is that decoding cannot be parallelized along spatial dimensions, leading impractical decoding latency, especially for large resolution images.
2 For details refer to Figure and 12 the Appendix.
It is reported in (Wu et al. 2020 Table I) that decoding time of spatial autoregressive models a 512x768 image range from 2.6s than half minute, depending the specific designs. Also see Figure
2

---

Published as a conference paper at ICLR 2022
ConvNet-based transforms While the design space of prior models is extensively explored, non- linear transforms, as an important component, have received less attention. standard convolution encoder-decoder with GDN et 2016 (017) as activation is widely adopted in the literature. Later works introduce new transform designs, such as residual blocks with smaller kernels (Cheng et al.. 2020), nonlocal (sigmoid gating) layers (Zhou et al., 2019) Chen et al. (202T), invertible neural networks (Xie et al. 2021) , and PReLU as an efficient replacement of GDN (Egilmez et al. 2021
Vision transformers Although many transform networks are proposed, they are still mainly based on ConvNets. Recently transformers (Vaswani et al. 2017) have been introduced the vision do- main and have shown performance competitive with ConvNets in many tasks, e.g. object detec- tion (Carion et al. 2020), classification (Dosovitskiy et al., 202T), image enhancement (Chen al.) 2020), and semantic segmentation (Zheng et al.) (021). Inspired by their success, in this work we explore how vision transformers work as nonlinear transforms for image and video compression.
3 SWIN-TRANSFORMER BASED TRANSFORM CODING
Among the large number of vision transformer variants, choose Swin Transformer (Liu et al, 2021] (hereafter referred to as Swin'T) to build the nonlinear transforms, mainly because of 1) its lin- complexity w.r.t. input resolution due to local window attention, and 2) its flexibility in handling varying input resolutions at test time, enabled by relative position bias and hierarchical architecture.
Swin Transformer
Swin Transformer
Swin Transfrmer
Transormer
Swin Transformer Block
Input Image
Swin Transfo
Patch Merge
Patch Merge
Patch Merge
Patch Merge
Patch Merge
Block
Block
Block
Block
Patch
Swin
x 61
x 6g
wboke My
15 C5
Chanial-vilse -vilse
AutoReg Ragieeeesvve
<bake
Factorized
Model
Reconstructio
Trareformer Block
Patch Spit
Patch Spit
Swin Traneformel
Patch Spit
Swin Transformer
AD
Patch Spitt
Swin Trarffomme
Patch Spiit
Swin fareformmr
80
Swin Transfor
Block
Block
Block
Block
Riee
3c
98
Xdg
xd4
Xde
Figure 2: Network architecture of our proposed SwinT-ChARM mode In the SwinT--yperprior
model, the ChARM component is removed and instead μ and a are directly output by the hyper- decoder h2 (For details see Figure 13 in Appendix A.
3.1 SWINT ENCODER AND DECODER
The original Swin'Ti is proposed as vision backbone, i.e. encoder transform with downsampling. As shown in Figure 2 the SwinT encoder 9a contains SwinT blocks interleaved with Patch Merge blocks. The Patch Merge block contains Space-to-Depp De-Deppth (for downsampling), LayerNorm, and Linear layers sequentially. Swin'T' block performs local self-attention within each non-overlapping window of the feature maps and preserves feature size. Consecutive SwinTT blocks the same feature size shift the window partitioning with respect to the previous block to promote information propagation across nearby windows the previous block.
We adopt SwinT encoder as the encoder transform 9a in our model, and extend it to Swin'T decoder Is by reversing the order of blocks Ja. and replacing the Patch Merge block with Patch Split block, which contains Linear, LayerNorm, Depth-to- So-Spaee (for upsampling) layers in sequence. The architectures for hyper transforms h2 h2 are similar Ja Is with different configurations.
"The ChARM architecture Minnen & Singh 2020 is detailed in Figure 12 of Appendix A.1
3

---

Published as a conference paper at ICLR 2022
With these four Swin'T transforms, propose two image compression models, Syypr-Hyerrrr and SwinT-ChARM, whose prior and hyper prior models are respectively the same as in Conv- The full model architectures are shown in Hyperprior and Conv-ChARM introduced in Section 2 Figure Segure@and and Figure 13
3.2 EXTENSION TO P-FRAME COMPRESSION
To investigate the effectiveness of Swin'T transforms for video compression, we study one popular P-frame compression model called Scale-Space Flow (SSF) (Agustsson al. (2200. There are three instances of Cony-Hyperprior in SSF, which are respectively for compressing I-frames, scale-space flow and residual. We propose a SwinTT variant, referred as SwinT-SSF. which is obtained by replacing Conv transforms 9a- in the flow codec and residual codec of SSF with Swin'T tranforms. To stabilize training of flow codec in SwinT-SSF, we need to remove all LayerNorm layers and reduce the window size (e.g. from 8 to 4). The baseline SSF model will be referred as Conv-SSF. Even though we build our solution on top of SSF, we believe this general extension can be applied to other ConyNet-based video compression models (Rippel et al. 2021 Hu et al. 2021 as well.
4 EXPERIMENTS AND ANALYSIS
4.1 EXPERIMENT SETUP
Training All image compression models are trained on the CLIC2020 training set. Conv- Hyperprior and SwinT-Hyperprior are trained with 2M batches. Conv-ChARM and SwinT-ChARM are trained with 3.5M and 3.1M steps. Each batch contains 8 random 256 x 256 crops from training images. Training loss = D + 3R a weighted combination of distortion D and bitrate R, with being the Lagrangian multiplier steering rate-distortion trade-off. Distortion D is MSE in RGB color space. To cover a wide range of rate and distortion, for each solution, we train 5 models with B E (0.003, 0.001, 0.0003, 0.0001, 0.00003). The detailed training schedule is in Appendix B.1
For P-frame compression models, we follow the training setup of SSF Both Conv-SSF and SwinTT- SSF are trained Vimeo-90k Dataset (Xue al. 2 for 1M steps with learning rate 10-4, batch size of 8, crop size of 256 x 256, followed by 50K steps of training with learning rate 10-5 and crop size 384 x 256. The models are trained with 8 values 2 x 10 10- y E {0, 1, ..., ..... 7). We adopt one critical trick to stablize the training from (Jaegle et al. 2021 Meister et al.) 2018), i.e. to forward each video sequence twice during one optimization step amini-batch), once in the original frame order, once in the reversed frame order. Finally we add flow los: Jnnyy
only between 0 and 200K steps, which we found not critical for stable training but improves the RD.
Evaluation We evaluate image compression models on 4 datasets: Kodak (Kodak 1999)
CLIC2021 testset (CLIC) 2021), Tecnick testset (Asuni & Giachetti] (014),, and JPEG-AI test- set (JEEG-AI 2020). We use BPG and VTM-12.1 to code the images in YUV444 mode, and then calculate PSNR in RGB. For fair comparison all images are cropped to multiples of 256 to avoid padding for neural codecs.
We evaluated P-frame models on UVG (Mercat al. 2020 and MCL-JCV Waan et al. 2016). and compare them with the test model implementation of HEVC, referred to as HEVC (HM), and open source library implementation of HEVC, refered to as HEVC (x265). To align configuration, all video codecs are evaluated in low-delay-P model with fixed GOP size of 12.
Besides rate-distortion curves, we also evaluate different models using BD-rate (Tan et al. 2016), which represents the average bitrate savings for the same reconstruction quality. For image codecs, BD-rate is computed for each image and then averaged across all images; for vidco codecs, BD-rate is computed for each video and then averaged across all videos. More details on testset preprocess- ing, and traditional codecs configurations can be found in Appendix B.2 B.2
'We did not observe RD improvement when applying flow loss to Conv-SSF training. WW use the original UVG sequences that are commonly used in other works (Agustsson et al. 2020
4

---

Published as a conference paper at ICLR 2022
Kodak
SwinT-ChARM Conv-tlypepproor
SwinT-Hyperprior
VTM-12.1
44
Conv-ChARM
BPG444
Kodak
Rate saving (%) VTM-12.1
PSNR (RGB)
(larger better)
-5
-10
36
SwinT-ChARM
SwinT-Hyperprior
-15
34
Conv
Conv-ChARM ChARM
Conv-Hyperprior
32
VTM-12.1
-5
BPG444
-30
0.5
1.0
1.5
2.0
2.5
32
34
36
38
40
42
44
Rate (bits per pixel)
PSNR (RGB)
(a) Rate distortion comparison on Kodak.
(b) Rate saving over VTM-12.1 (larger is better)
Figure 3: Comparison of compression efficiency Kodak 8
Note that the encoding time of VTM-
12.1 is much longer than all neural codecs, as shown in Table Table 7 in the Appendix.
4.2 RESULTS
RD and BD-rate for image codecs The RD curves for all compared image codecs evaluated on Kodak are shown in Figure and the relative rate reduction of each codec compared VTM-12.1 at range of PSNR levels is shown in Figure 3b
As can be seen from Figure 3 SwinT transform consistently outperforms its convolutional counter- part; the RD-performance of Swin'-Hyperppria is on-par with Cony-ChARM, despite the simpler prior; SwinT-ChARM outperforms VT'M-2.1 across wide PSNR range. In the Appendix (Fig-
ure 28 and Figure 30), further incorporate the results from existing literature known to us for a complete comparison. Particularly, our Conv-Hyperprior much better than the results reported
in (Minnen et al. 2018
(no context), and Conv-ChARM is par with (Minenn & Singh 2020
Image Codec
Kodak
CLIC2021
Tecnick
JPEG-AI
BPG444
20.87%
28.45%
27.74%
27.14%
Conv-Hyperprio
11.65%
12.23%
12.49%
20.98%
Conv-ChARM
3.44%
4.14%
3.50%
9.59%
SwinT-Hyperprior
1.69%
0.83%
-0.15%
6.86%
SwinT-ChARM -3.68%
-5.46%
-7.10%
0.69%
Table 1: BD-rate of image codecs relative to VTM-12.1 (smaller is better).
In Table 1 we summarize the BD-rate of image codecs across all four dataset with VTM-12.1 as anchor. On average SwinT-ChARM able achieve 3.8% rate reduction compared VTM- 12.1. The relative gain from Conv-Hyperprior to SwinT-Hyperprior is on-average 12% and that from Conv-ChARM to SwinT-ChARM is on-average 9%. Further gain over VTM-12.1 can be obtained by test-time latent optimization (Campos et al. 2019) or full model instance adaptation (van
Rozendaal et al. 2021). which are out the scope of this work.
Thhe relative rate-saving curves Figure 3b generated by first interpolating the discrete RD points (av- eraged across the testset) with cubic spline, and then compare bitrate of different models at fixed PSNR. RRD plot for the other three datasets be found in Appendix (Figure Figure 15 and Figure
5

---

Published as a conference paper at ICLR 2022
UVG
MCL-/CV
40
38
37
PSNR
36
PSNR
SwinT-SSF
SwinT-S5F
(repraduced) Conv-SSF (Agustsson al.) (X265)
(reproduced)
34
ConN-SSF (Agustsson al))
(X265)
AVC (X264)
AVC (X264)
HEVC (HM)
HEVC (HM)
0.0
0.2
0.4
0.6
0.0
0.2
0.4 Rate (bits per pixel)
0.6
Rate (bits pixel)
Figure 4: PSNR vs bitrate curves of P-frame codecs on UVG and MCL-JCV datasets.
RD and BD-rate for video codecs For P-frame compression, we evaluated SwinT-SSF on UVG and MCL-JCV, with RD comparison shown in Fig- ure Again, Swin'T' transform leads to consis- tently better RD. Table Table/2]summarize our reproduced Conv-SSF model as anchor. We can that SwinT-SSF achieves average of 11% rate saving over Conv-SSF, Additionally, we show that if SwinTT transform is only applied to residual-autee autoencoder (labeled as SwinT-SSF-Res), it only get about 4.6% gain, which indicates that both flow and residual compression benefit from Swin'T as encoder and decoder transforms. Note that SwinT-SSF still lags behind HM, suggesting lots room for improvement neural video compression. For per-video breakdown of BD-rate, see Figure [8 and Figure [7 in the Appendix.
Table 2: BD-rate of video codecs, with Conv- SSF (reproduced) as anchor.
Video Codec
UVG
MCL-JCV
BD-rate with
HEVC (x265)
25.97%
25.83%
HEVC (HM)
-15.80%
-24.96%
SwinT-SSF SwinT-SSF-RR
-12.35%
-10.03%
-5.08%
-4.16%
Table 3: Decoding complexity. All models are trained with = 0.001, evaluated on 768 512 images (average 0.7 bpp). Decoding time is broken down into inference time of hyper-decoder ht and decoder gs, entropy decoding time of hyper-code å and code ý (including inference time of the prior model if it is ChARM).
Time (ms)
Codec
GMACs
Peak memory
Model params
h
y
Js
total
Conv- Hypeepriprr
5.5
4.0
38,2
168.9
219.3
350
0.50GB
21.4M
Conv-ChARM
5.3
4.1
82.9
168.5
264.0
362
0.53GB
29.3M
SwinT-Hyperprias
6.0
4.8
38.1
59.6
114.0
1.44GB
24.7M
SwinT-ChARM 5.9
4.8
90.7
60.1
167.3
111
1.47GB
32.6M
Decoding complexity We evaluate the decoding complexity of 4 image codecs on 100 images of size 768 512 and show the metrics in Table including decoding time, GMACs and GPU peak memory during decoding and total model parameters. The models run with PyTorch 1.9.0 on workstation with one RTX 2080 Ti GPU. From the table, the inference time of SwinTT decoder is less than that of Conv decoder. The entropy decoding time of ChARM prior is about twice than the factorized prior. The total decoding time of SwinT-based models less than Conv-based models. In ablation study A5, we show a smaller Swin SwinT-Hyperprior with 20.6M parameters has almost the same RD as the Hypperrrrrrrr profiled here. For details on encoding complexity, profiling setup, scaling to image resolution, please refer to Table Section D. in the Appendix.

---

Published as a conference paper at ICLR 2022
Scaling behavior To how the BD-rate varies with model size, we scale SwinT- Hyperprior and Cony-Hyperprion to be twice or half of the size of the base models (i.e. medium size y The result is shown in Figure [ For both types of models, as we reduce the base model size, there a sharp drop in perfor- mance, while doubling model size only leads to marginal gain. Noticeably, SwinT-Hyperpiig small is on-par with Con-HHyperprior-mmediu even with half of the parameters, and Swin'T transforms in general incur fewer MACs per pa- rameter.
41.MM
BD-rate over VTM-1123 (%) Kodak
0
SwinT-Hypeer Honn-HHyperprirr
medium large
41 41.0M
21
10
small
medium
large
20
Conv Hyperprior
25
small
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
In Figure we further consolidate the decoding latency and scaling behavior study into sin- gle plot and show that SwinT-ChARM runs at comparable speed as VTM- 12.1 while achiev- ing better performance prior that decodes orders of magnitude slower.
MMACs per pixel (for decoding)
Figure 5: Model size scaling.
opposed to state-of-the-art neural codecs with spatial autoregressive
4.3 ANALYSIS
Latent correlation One of the motivating principles of transform coding is that simple coding can be made more effective in the transform domain than in the original signal space (Goyal) 2001 Ballé al. T02T). A desirable transform would decorrelate the source signal that simple scalar quantization and factorized entropy model can be applied without constraining coding performance. In most mature neural compression solutions, uniform scalar quantization adopted together with learned factorized or conditionally factorized Gaussian prior distribution. It is critical, then, to effec- tively factorize and Gaussianize the source distribution so that coding overhead can be minimized.
0.5073
0.0040 0.0335 0.0100
0.0030
Hyperpriori 1tta=0.00011. Kodak: 86d8, 8PP-0.52
SwinT SeiiiT-tyyerrriottttta..oo 8PP-0.55
of Figure 6: Spatial correlation of (y achieves uniformly smaller correlation than Con--Hyperprior (left).
a with models trained at / 0.001. SwinT-Hyperprior (right)
Specifically, in hyperprior based models (Ballé et al. 2018 A (y p1//ar is modeled stan- dard spherical normal vector. The effectiveness the analysis transform Ia can then be evaluated by measuring how much correlation there is among different elements in y. We are particularly in- terested measuring the correlation between nearby spatial positions, which are heavily correlated in the source domain for natural images. In Figure O we visualize the normalized spatial correlation of y averaged over all latent channels, and compare Conv-Hyperprior with Swin T-yyperrrioo at 0.001. It can be observed that while both lead to small coss-correlations, wrin--Tranffomer does a much better job with uniformly smaller correlation values, and the observation is consistent
'detailed model configurations are provided in Appendix A.3
Note that it is difficult fairly compare the decoding time of VTM and neural codees since they on different hardware. For more discussion please refer to Appendix The value with index (1,2) corresponds to the normalized cross-coreelaioon of latents spatial location (ur, and (in + h j), averaged across all latent elements all images Kodak.

---

Published as a conference paper at ICLR 2022
with other 3 values, which are provided in Figure 20 in the Appendix. This suggests that trans- former based transforms incur less redundancy across different spatial latent locations compared with convolutional ones, leading to an overall better rate-distortion trade-off. The larger spatial correlation (and thus redundancy) in Conv- Hoyy-Hyyrrrr also explains why compute-heavy spatial auto-regressive model is often needed to improve RD with convolutional based transforms (Minnen itt al., 2018) Lee et al. 2019 Ma et al.) [021 Guo et al. [202]] Wu et al.) 2020). FigureGlalse reveals that most of the correlation of a latent comes from the four elements surrounding it. This suggests that a checkerboard- based conditional prior model (He et al. 2021) may yield further coding gain.
(a) Conv- Conv-Hyperprior
(b) Cony-ChARM
(c) Conv-SSF-Res
(d) Conv-SSF-Floo
-32
-28
-10
-22
6
100
432
14
53
32
20
222
-208
-50
3.00
(c) Swin Swin'--Hypen H-yperrrrrr
(f) SwinT-ChARM
(g) SwinT-SSF-Res
(h) SwinT-SSF Flow
-20 2307
-23
-12
com -10-
20-
20-
38
1.00
22
Figure 7: Comparison of effective receptive field (ERF) of the encoders go, which is visualized as absolution gradients of the center pixel in the latent (i.e. dy /dx) with respect to the input image. The plot shows the close up of the gradient maps averaged over channels each input of test images/videos. Check Figure for the ERF of the composed encoding transform ha o ga.
Effective receptive field Intra prediction in HEVC or AVI only rely on left and top boarders of the current coding block (Sullivan et al. 2012, Chen et al.) 2018), except for intra block copy for screen content (Xu et al." 2016). We would like to see how large the effective receptive field (ERF) (Luo et al. 2017) of SwinT encoders compared to Conv encoders. The theoretical receptive field of the encoders (ga, ha o 9g))) in SwinT-based codecs is much larger than that of Conv-based codecs. However comparing Figure 77 with 7 and Figure 7b with 77 the ERF of SwinT encoders after training is even smaller than Conv encoders. When we examine the ERF of the released Swin transformers for classification, detection and segmentation tasks, they are all spanning the whole input image. This contrast suggests that (natural) image compression with rate-distortion objective is local task, even with transformer- bassdd nonlinear transforms. We further look into P-frame compression models, particularly the ERF of two types of transforms in flow codec and residual codec, as shown in Figure 7 & 7h and Figure 7c & 7g Clearly for flow codec, SwinT transform has much larger ERF than the convolution counterpart. For residual codec, the ERF of Swin'T transforms similar to image (I-frame) compression case. This shows of flexibility of Swin'T encoders attend to longer or shorter range depending on the tasks. To get better picture of the behavior of attention layers in SwinT transforms, we also show the attention distance each layer in Figure 22
Progressive decoding The ERF in the previous section shows the behavior of the encoder trans- forms, here we further investigate the decoder transforms through the lens of progressive decod- ing (Rippel al. 2014 Minnen & Singh 2020: Lu al., 2021). Initialized with the prior mean, the imput to the decoder Is progressively updated with the dequantized latent ÿ in terms of coding units, leading to gradually improved reconstruction quality. For the latent with shape (C, H, W), we consider three types of coding units, i.e. per channel (1, H. W), per pixel (C, 1, 1), per element (1, 1, 1). The coding units are ordered by the sum of prior std of all elements within each unit. The RD curves of progressive decoding for SwinT-Hyperprior and HoyyrHyprrrr are shown in Fig- ure 8al which closely follow each other when ordered by channel or element, but significantly apart when ordered by pixel (spatial dim). Particularly, we show an extreme case when the half pixels in the latent (masked by checkerboard pattern) are updated with dequantized values, corresponding to the two scatter points in Figure Sal One visual example (CLIC2021 test) is shown in Figure 8b under
8

---

Published as a conference paper at ICLR 2022
this setup, where clearly see Swin'T decoder achieves better reconstruction quality than the Conv decoder, mainly terms of more localized response to a single latent pixel. This is potentially useful for region- interet decompression, More visual examples are shown in Figure 26
--- SwinT, channel Conv,
40.0
37.5
SwinT, pixe!
Conv, pixel
SwinT, element
PSNR (RGB)
Conv, element
32.5
SwinT- checkerboard
Conv.chetkerbeard
30.0
25.0
22.5
0.25
0.50
0.75
1.00
1.25
1.50
(bits per pixel)
(a) Progressive decoding.
(b) Recon with checkerboard masked latent.
Figure 8: (Left) Progressive decoding according to the order of the sum of prior std of all elements within each latent coding unit, which be latent channel, pixel element. SwinT-Hyperpriir archives about 2dB better reconstruction the same bitrate than Conv-Hyperprior The reconstructions with Swin'T-Hi WiinT--yyerprior (top) and Conv-Hyperprior (bottom) for the latent masked with checkerboard pattern corresponding to the two plus markers on the left subfigure. The SwinTT decoder has more localized reconstruction to single latent pixel.
per-pixel progressive decoding. (Right)
4.4
ABLATION STUDY
Swinr-Hyyerpriar
Relative position bias There are two sources of positional information in Swin'T transforms, namely the Space-to-Depth modules and the additive relative position bias (RPB). Even when the RPB is removed, Swirrr-Hyprrprior still outperforms Conv-Hypen nonv-Hyperppior across all bi-
[A1] No relative positional bias
[A2] window shift
[A3] Deeper
[A4] Replace Attention with
[AS] SwinT--dppths=(22225]]
Hoperryprrrrior
Conv-Hyperprior
trates, which indicates image compression may require accurate relative position.
12.5
Rate saving (%) over Conv-
(larger is better)
10.0
Shifted window The original motivation of shifted window design is to promotes the inter-layer feature propagation across non- overlapping windows. Image compression performance drops slightly when there is no shifted window at all. This further suggests im- age compression requires local information.
7.5
5.0
0.0
-2.5
32
34
36
The details of ablations A3-A5 in Figure Can be found in Section of the appendix.
PSNR (RGB)
Figure 9: Ablation study
5 CONCLUSION
In this work we propose Swin transformer based transforms for image and video compression. In the image compression setting, Swin'T transform consistently outperforms its convolutional counterpart. Particularly, the proposed SwinT-ChARM model outperforms VTM-12.1 at comparable decoding speed, which, to the best of our knowledge, is the first in learning- based methods. We also show the effectiveness SwinT transforms when extended to the P-frame compression setting. Compared with convolution transforms, Swin'T transforms can spatially decorrelate the latent better, have more flexible receptive field to adapt to tasks that requires either short-range (image) and long-range (motion) information, and better progressive decoding of latent pixels. While pushing the neural trade-off, we believe is
image compression to a new level in terms of rate- distortionncompuutaai only the starting point for developing more efficient transformer-hasee image and video codecs.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We would like to thank Amir Said for developing entropy coding and great advice on data compres- sion in general. We would also appreciate the helpful discussions from Reza Pourreza and Hoang Le, and draft reviews from Auke Wiggers and Johann Brehmer.
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George Toderici. Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8503-8512, 2020.
Nicola Asuni and Andrea Giachetti. Testimages: large-scale archive for testing visual devices and basic image processing algorithms. In STAG, pp. 63-70, 2014.
Johannes Ballé, Valero Laparra, and Eero Simoncelli. Density modeling of images using a gener- alized normalization transformation. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. UtLhttp:// wwtp/aaxivvv..../.o abs /anes/1511 1511. 0015511.8881
Johannes Ballé. Valero Laparra, and Eero P. Simoncelli. End-to-end optimized image compres- sion. 5th International Conference on Learning Representations, ICIR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: //openeeviww.. wwwp:awwwiwww...e.o...eeeeee (
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational Im- age Compression with Scale Hyperprior. In 6th International Conference on Learning Represen- fations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceed- ings. OpenReview.net, 2018. URL/ttts:: /pppn//iee./peeeeee..ee. net www.uuiidd-...........
Johannes Ballé, Philip A. Chou, David Minnen, Saurabh Singh, Nick Johnston, Eirikur Agusts- son, Sung Jin Hwang, and George Toderici. Nonlinear transform coding. IEEE J. Sel. Top.
Signal Process., 15(2)3339 353. 2021. doi: 10.1009USSTSP:222000445555 //doi. 1109 JSTSP 2020 3034501
URL hetps:
Joaquim Campos, Simon Meierhans, Abdelaziz Djelouah, and Christopher Schroers. Con- tent Adaptive Optimization for Neural Image Compression. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 0. Computer Vision Foundation IEEE, 2019.
URL http:/ http///ppppcccccece/o//osoccscsa twectfeeeeeec www.///ooettcoooooooooo CVWRW 201 /CCI/ 201 Compression CVPRW 2019 paper. haper.html
fortiiionnoor_ooee Neurrreeeraa___. Image___________________________
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th Eu- ropean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 1, volume 12346 of Lec- ture Notes in Computer Science, pp. 213 229. Springer. 2020. doi: 10.1007/978- 030-58452--8 113. WRRlhttps://doo www.doi..orggl/ 10. 1000000077700 3- 030 5-852-8558811
Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. CoRR, abs/2012.00364, 2020. URL httph:
org hbtp/bb//22110 20b.b///0/022106
Tong Chen, Haojic Liu, Zhan Ma, Qiu Shen, Xun Cao, and Yao Wang. End-to-end learnt image compression via non-local attention optimization and improved context modeling. IEEE Trans. Image Process., 30:3179-3111 2021. doi: 10. 10.11009711:2221.33555 3058615
URL htt.::ttp///d...
3xg/10.1109//1
Yue Chen, Debargha Murherjee, Jingning Han, Adrian Grange, Yaowu Xu, Zoe Liu, Sarah Parker, Cheng Chen, Hui Su, Urvang Joshi, Ching-Han Chiang, Yunqing Wang. Paul Wilkins, Jim Bankoski, Luc Trudeau, Nathan Egge, Jean-Mare Valin, Thomas Davies, Steinar Midtskogen, Andrey Norkin, and Peter de Rivaz. An overview of core coding tools in the avl video codec. In 2018 Picture Coding Symposium (PCS), pp. 41-45 2018. doi: 10.1109/PCS.211 84566449.
Zhengxue Cheng. Heming Sun, Masaru Takeuchi, and Jiro Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition. CVPR 2020, Seattle, WA. USA, June 13-19, 2020, pp. 7936 7936-7945. IEEE, 2020, doi: 000100902222220002200000 org/10. 111.:0002210////221102 2020. 00796
URL https: //doi.

---

Published as a conference paper at ICLR 2022
CLIC. Clic2021 challenge on learned image compression, 2021. URL httl:ttt:://otsssstttee// be/tasks/|
Yihe Dong, lean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: pure attention loses rank doubly exponentially with depth. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learming Research, pp. 2793-2803. PMLR, 2021. URL http:::pppppeeeeeeeeeeeeeee ngs mir. aress/v119/donngeee
html
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. 1d=YicbF
net forum?
Hilmi E. Egilmez, Ankitesh Singh, Muhammed Coban, Marta Karezewicz, Yinhao Zhu, Yang Yang, Amir Said, and Taco S. Cohen. Transform network architectures for deep learning based end-to-end image/video coding in subsampled color spaces. CoRR, abs/2103.0176.. 2021. URL.
https://arxits .rgg aww./////2200 210...1166600
Alaacldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski. Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. Xeit: Cross-covariance image transformers. CoRR, abs/2106.09681, 2021. URL UtRphttp:://art
org/abs/210e 09681
V.K. Goyal. Theoretical foundations of transform coding. IEEE Signal Processing Magazine, 18 (5):9-21 21, 2001. doi: 10. 0009799952802.
Zongyu Guo, Zhizheng Zhang, Runsen Feng, and Zhibo Chen. Causal contextual prediction for learned image compression. IEEE Transactions on Circuits and Systems for Video Technology, pp. 1-1, 2021. doi: 10.1 10010999TSST.2.20.00999999999999
Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming-Ming Cheng, Jiaying Liu, and Jingdong Wang. Demysti- fying local vision transformer: Sparse connectivity, weight sharing, and dynamic weight. CoRR, abs/2100.04226, 2021. URL
abs/ 2011..60000 04263
Dailan He, Yaoyan Zheng, Baocheng Sun, Yan Wang, and Hongwei Qin. Checkerboard context model for efficient learned image compression. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14771-14780, 2021.
Zhihao Hu, Guo Lu, and Dong Xu. FVC: A New Framework towards Deep Video Compression in Feature Space. arXiv preprint arXiv:2105.09600 2021.
Andrew Jacgle, Sebastian Borgeaud, -ean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier J. Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver IO: A General Architecture for Structured Inputs & Outputs. CoRR, abs/2107.14795, 2021. URL
httts::/arxxvv.oom http:////aaaa..oog///////1.1111
JPEG-AI. JPEG-AI Test Images. https: nttps:///pegai: http:ppaaaa.ggpguubbbppp wwthubtthhh.ggg. wwthub.io/tes........ooeeeee _atrstt__mmaee///ee 2020.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ica: A unifying framework. International Conference on Artificial Intel- ligence and Statistics, pp. 2207-22722 2207-2217, PMLR, 2020.
Kodak. Kodak Test Images. httt::/rr00 us/ http:///wk..aggggap.ccsc//o//ooe htt/ 1999.
Simon Korbblith, Mohammad Norouzi, Honglak Lee. and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR, 2019.
12

---

Published as a conference paper at ICLR 2022
Jooyoung Lee, Seunghyun Cho, and Seung- Kwon Beack. Context-adaptive entropy model for end- to-end optimized image compression. In International Conference on Learning Representations, 2019. URL/Lttss//ppeeeeeeeee...ct http://wwww....ttooooo.orooaaoaa
Mu Li, Kai Zhang, Wangmeng Zuo, Radu Timofte, and David Zhang. Learning context-basee non-
local entropy modeling for image compression. CoRR, abs/2005.0466h 2020. URL https:
/ //arxiv.org/abel 2200aaa//222.... 2005. 04661
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. CoRR, Abs/2103.14030 2021. URL pttp:ttt:://// /ttp:tttp////iix.orrggr///
14030
Yadong Lu, Yinhao Zhu, Yang Yang, Amir Said, and Taco S Cohen. Progressive neural image compression with nested quantization and latent ordering. In 202/ IEEE International Conference on Image Processing (ICIP), pp. 539-543, 2021. doi: 10. 10.1109//C1P42222..000 195060206
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the Effective Receptive Field in Deep Convolutional Neural Networks. CoRR, abs/1701.041 2017. URL http:
Changyue Ma. Zhao Wang, Ru-Ling Liao, and Yan Ye. A cross channel context model for latents in deep image compression. CoRR, (bb//203300882) 2021. URL https://aaxii.......// / 2103. 02884
Simon Meister, Junhwa Hur, and Stefan Roth. Unflow: Unsupervised learning of optical flow with bidirectional census loss. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018,
pp. 7251-7259. AAAI Press, 2018. URL https: /www. aaai owg. iinex.ndxx..////. php/
AAAI
Alexandre Mercat, Marko Viitanen, and Jarno Vanne. Uvg dataset: 50/120fps 4k sequences for video codec analysis and development. In Proceedings of the 11th ACM Multimedia Systems Conference, MMSys 220, pp. 297-302, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 4781450368452.
doi: 10.1145/3399825.394997
URL https://doi.
oxg/10.1145/333982 33949377
David Minnen and Saurabh Singh. Channel-Wiss Autoregressive Entropy Models for Learned Image Compression. In IEEE International Conference on Image Processing, ICIP 2020, Abu Dhabi, United Arab Emirates, October 25-28, 2020, pp. 3339 3343. IEEE, 2020. doi: 10. 1100/IC144077..200..99999
URLlattpss//doi..ooo http:///................./C/CCC. 11t.::111111111111111111 2020.
91 9190935
David Minnen, Johannes Ballé, and George Toderici. Joint autoregressive and hierarchi- cal priors for learned image compression. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Informa- tion Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, newrips. wwt.cc//paaa poc/ppppe//201/ Abst hhmll
pp. 00794-10803 2018. URL
Yichen Qian, Zhiyu Tan, Xiuyu Sun, Ming Lin, Dongyang Li, Zhenhong Sun, Hao Li, and Rong Jin. Learning accurate entropy model with global reference for image compression. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL eTbIjyrUVw
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the pressive power of deep neural networks. In international conference on machine learning, pp. 2847-2854. PMLR, 2017.
13

---

Published as a conference paper at ICLR 2022
Oren Rippel, Michael A. Gelbart, and Ryan P. Adams. Learning ordered representations with nested dropout. In Proceedings of the 31th International Conference on Machine Learning. ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceed- ings, pp. 1746-1754. JMLR.org, 2014. URL ri rippe html]
/ press/u321
Oren Rippel, Alexander G. Anderson, Kedar Tatwawadi, Sanjay Nair, Craig Lytle, and Lubomir D. Bourdev. ELF-VC: efficient learned flexible-rate video coding. CoRR, abs/2104, 143355, 2021.
URL https::/aaxxi...ottt
abs abtpaaas..//.../..///..//a.....a 14335
Gary J Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (hevc) standard. IEEE Transactions on circuits and systems for video technology, 22412)1149-11668 2012.
Thiow Keng Tan, Rajitha Weerakkody, Marta Mrak, Naeem Ramzan, Vittorio Baroncini, Jens- Rainer Ohm, and Gary J. Sullivan. Video quality evaluation methodology and verification testing of heve compression performance. IEEE Transactions on Circuits and Systems for Video Tech- nology, 26(1):76-90 60, 2016. doi: 10. 10.10099
Ties van Rozendaal, Iris A. M. Huijben, and Taco Cohen. Overfitting for Fun and Profit: Instance- Adaptive Data Compression. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: //openreview.
wwworrmmmddoopppooo 1d=oFp8Mxxxxxooo V5FL
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 30: Anmual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL https: https: http::///cppppeceoeeecece neur ips. CC/ htte/appr//aee 2ateer///1//hhhhh... 3f 7dee fbdd c 5aa Abst html
Haiqiang Wang, Weihao Gan, Sudeng Hu, Joe Yuchieh Lin, Lina Jin, Longguang Song, Ping Wang, Ioannis Katsavounidis, Anne Aaron, and C-C Jay Kuo. MCL-JCV: a JND-based H. 264/AVC video quality assessment dataset. In 2016 IEEE International Conference on Image Processing (ICIP), pp. 1509-1513. IEBE, 2016.
Yaojun Wu, Xin Li, Zhizheng Zhang, Xin Jin, and Zhibo Chen. Learned block-based hybrid image compression. arXiv preprint arXi::2012.09550 2020.
Yueqi Xie, Ka Leong Cheng, and Qifeng Chen. Enhanced invertible encoding for learned image compression. August 2021.
Xiaozhong Xu, Shan Liu, Tzu-Der Chuang, Yu-Wen Huang, Shaw-Min Lei, Krishnakanth Rapaka, Chao Pang, Vadim Seregin, Ye-Kui Wang, and Marta Karczewicz. Intra block copy in heve screen content coding extensions. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 6(4):409 2016. doi: 10. 109//0TTCSS 201.....111.5555555
Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video Enhancement with Task laie-triented Flow. International Journal of Computer Vision (IJC),, 127(8): 106 1025-125, 2019.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H. S. Torr, and Li Zhang. Rethink- ing semantic segmentation from sequence-to-sequence perspective with transformers. In IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2021, vir- tual, June 19-25, 2021, pp. 6881-6890. Computer Vision Foundation / IEEE, 2021. URL https: httppp//ppeenceeesseeeeeeee
2heng
Brechinkinn_Semanticce Stw.aatttte.eeeeeetttttiiio.oooo htt.a/aaceeeeaaoooaoeeeeeeeeeeee wewpueeeeneetttoeuuueeeee-eee
Berrpeettive_Mich_taaee rr Bttpsss__c________ 20___22022_ppp_ttmmmmmmm
Lei Zhou, Zhenhong Sun, Xiangji Wu, and Junmin Wu. End-to-end optimized image compression
with attention mechanism. In CVPR workshops, pp. 0, 2019.
14

---

Published as a conference paper at ICLR 2022
Appendix
Table of Contents
Models
15
A. Convolution baselines
15
A.2 Swin- Transformer based compression models
15
A.3 Model configurations for model size scaling study
16
B
Training and Evaluation
17
Training
17
B.2 Traditional codee evaluation
19
rate computation
BD rate image codec
20
BD rate for video codec
20
D More Results
21
D. Image compression
21
D.2 Video Compression
22
D.3 Coding complexity
22
E
Analysis
24
E.1
Spatial correlation of latent
24
E.2
Effective Receptive Field
................................
25
Rate distribution across latent channels
25
Centered kernel alignment
25
Progressive decoding
25
F
More ablation studies
26
A MODELS
A. CONVOLUTION BASELINES
Conv-Hyperprior and Conv-ChARM The architecture of Cony-Hyperprior and Conv-
ChARM are shown in Figure 10 and Figure [ For both architecture, our base model (.ee medium size) has the following hyperparameters: (C1, C3, C4, Co Co, C7) (320,32) 320, 320, 192, 192, 192).
A.2 SWIN-TRANSFORMER BASED COMPRESSION MODELS
SwinT-Hyperprior, SwinT-ChARM For both Syyne--yperrriorr and SwinT-ChARM we use the same configurations: (Wg, wa. (8, 4). (Ci, C22 Ca, Ca, Cs, Co) (128, 192, 320, 192, 192), (di, d2, dg, d4, ds, da) = 2, 6, 2, 5, 1) where C, d, and w are defined in Figure 13 and Figure The head dim is 32 for all attention layers in SwinT-based models.
SwinT-SSF For SwinT transforms used in SSF variant, the first Patch Merge block is with down- sample rate of 4 and two other Patch Merge blocks with downsampling rate of 2. Thus the down- sampling rate for the encoder is still 16, the same the image compression models. There are only 3 transformer stages with depths 2, 4, 2. The embedding dim is 96. The number latent and hyper latent channels all 192. The window size is 4 for flow codec and for residual codec.
15

---

Published as a conference paper at ICLR 2022
Input image
Cbnv Sx5
Stride-
GDN
Conv Sx5
GDN
Conv SxS
Stride=2
Stride=2
ReU.
Conv Sx5
Stride-
RaUU
Chiiv Sw5
Go
hag
x x
x x
W C
1 x GG
x
1
Bit
BRts
Factorizes
AD
Conv 3x3 striel
Reconst
Convi Sx5
Canvi 5x5
Convi 5x5 Stride=2
Stride=2
Convi 5x5
Stride=2
Convi 5x5
Stride
IGDN
1GDN
Rell
Rell
1
AD
h
Figure 10: Conv-Hyperpe
Input image
stride
Conv SxS
Stride-2
GDN
Conv SX5
GDN
Conv Sx5
Stride=2
GDN
Stride=2
Rei.J
Conv SXS
Stride-2
ReUU
Chniv Sx5
h
AE
4 x xxx
5xxx0 K5
16 Co
Charnel wwise
AutoReg eenoeegressivee
esssive
(CMARM)
Factortbed
Bits
Reconstructio
Conv 3x3 stridl
%
ConvT 5x5
Stride
1GDN
ConvT 515
JGDN
ConvT 5x5
Stride=2
1GDN
ConvT
Stride=2
RelU
Convi 5x5
Stride=2
ReLU
ConvT 5x5
Stride-2
AD
0<c
de 24
9.
hg
Figure 11: Conv-ChARM
SwinT-SSF-Res This is variant where only residual autoencoder Swin'T transforms, Same architecture as the residual autoencoder in TwiSF-
A.3 MODEL CONFIGURATIONS FOR MODEL SIZE SCALING STUDY
A.3.1 SWINT-HYPERPRIOR
Set of model hyperparameters that are common to all experiments: (d1, d2, do d1, ds. do) = (2 6, 2, (Wg, wh) (8.4) 4)
SwinT-Hyperprior (small) (C,, CC,,C,,C C3 C4, Cs. C.) = (96, 128, 160, 192, 128)
SwinT-HYyerprroo (medium)
(C1, C2 C3. C4, Cz Cc) = (128, 192, 256, 320, 192, 192)
SwinT-Hyperprior (large)
C2,, Cg. C4 Co Co) (160, 256 352, 448, 192, 256)
A.3.2 CONV-HYPERPPIOR
Con--Hyperprior (small) (C1,C,, C4, Cr. Co. C7) (192, 192, 192, 192, 128, 128, 128)
Cony-Hyperprior (medium)
(C1, C2, Ca, Ca, Ca. Ca, C7) (320, 320, 320, 320, 192, 192, 192)
16

---

Published as a conference paper at ICLR 2022
h
Channel-vise AutoRegressive Model (ChARM)
tei
Channel Spii
ChARM-Block
y<2
B
Channel Concat
Channel Concat
Channel Concat
Channel Concat
Conv 3x3 stridel
Conv stride1
ReLU
ReLU
ChARM- Block
ChARM- Block
ChARM Block
ChARM
stridel
Conv stridel
Block
ReLU
ReLU
Conv stridel
Conv stridel
Cos
Cou
Figure 12: ChARM architecture. Since can only be decoded after 14; and o; is obtained, the S ChARM-blocks are executed sequentially. We use S 10 all our experiments, which is consistent
with (Minnen & Singh 2020).
Swin Transformer
Transformer Block
Swin Transformer
Swin Transformri
Swin Transformer
Swin Transformer
Input Image
Patch Merge
Potch Merge
Patch Merge
Patch Merge
Patch Merge
Pattch Merge
Block
Block
Block
Q
9a
AE
x x
wabie
Bits
35
Bitt
Factorized need Model
Reconstruction
Swin Transformed Block
Patch Split
Transformel Block
Patch Split
Swin rransformer Block
Patch Spllt
Swin Transformei
AD
Patch Split
Swin Traina Cormer
Patch Split
Swin Traneformer
Block
Block
Patch
am
9s
xdd
xds
xde
Figure 13: SwinT--yperprior
Conv-Hyperprior (large) (C1, C2, C3, C4, Cs. Co. C7) (448, 448, 448, 448, 256, 256, 256)
B
TRAINING AND EVALUATION
B. TRAINING
All image compression models are trained on CLIC2020 training set, which contains both profes- sional and mobile training sets, in total 1,633 high resolution natural images. Cony-Hyperprior and SwinT-Hyperprior are trained with 2M batches. Each batch contains 8 patches of size 256 256 randomly cropped from the training images. Learning rate starts at 10-4 and is reduced 10-5 1.8M step.
For Conv-ChARM, we first train a model Conv-ChARM at 8 0.0001 from scratch for 2M steps, with it as the starting point, we continue train other beta values Conv-ChARM-66 8 E B for 1.5M steps. For SwinT-ChARM--, we load the transform weights from the checkpoint at 2M step of the pretrained SwinT- HiiiTT--prrprio---- then finetune the transforms together with the random initialized ChARM prior for 1.1M steps. Learning rate starts at 10 and is reduced to 10 for the last 100K steps.
17

---

Published as a conference paper at ICLR 2022
Training loss L D + 3R is weighted combination of distortion D and bitrate R. with B being the Lagrangian multiplier steering rate-distotiion trade-off. Distortion is MSE RGB color space. To cover wide range of rate and distortion, for each solution, we train 5 models with B E B = {0.003, 0.001, 0.0003, 0.0001, 0.00003).
Usually need to train longer for the model with larger bitrates (i.e. smaller ( converge. Particularly for the results presented in this paper, We train Shin1--Hyprprior-0.000000 for 2.5M steps instead of 2M steps for the other 4 lower bitrates.
For P-frame compression models, we follow the training setup of SSF. Both Conv-SSF and Swin'-- SSF are trained Vimeo-90k Dataset (Xue et al. 2019) for 1M steps with learning rate 10-1 batch size of 8, стоp size of 256 x 256, followed by 50K steps of training with learning rate 10-6 and crop size 12 384 256. The models are trained with 3 values 2 10
{0, 1, ..... 7}. We adopt one critical trick to stablize the training from (Jaegle et al. 2021 Meister et al. 2018). i.e. to forward each video sequence twice during one optimization step (mimi-batch), once the original frame order, once in the reversed frame order. When this trick is used, we set the batch size to be instead of 8. Finally we add flow loss only between and 200K steps, which we found not critical for stable training but helps improve the RD.
For all model training, Adam optimizer used without weighted decay. Training for 2M steps takes about 10 days and 14 days respectively for Conv-Hyperprion and SwinT-Hyperprior Nvidia V100 GPU. Total training time is about 7.5 days on a single Nvidia V100 GPU.
single
For all models, we use mixed quantization during training (Minnen & Singh 2020), i.e. adding uniform noise to the continuous latent before passing to the prior model, subtracting prior mean from the continuous latent followed by rounding before passing the decoder transform.
We did not use the crop size 384 x 384 during the second stage in the original paper because the resolution of Vimeo dataset is 448 x 256. We found in our case increasing crop size from 256 256 to 384 256 in the second stage does improve RD.
18

---

Published as a conference paper at ICLR 2022
B.2 TRADITIONAL CODEC EVALUATION
In this section, we provide evaluation script used to generate results for traditional codecs.
B.2.1
IMAGE CODECS
VTM-12.1: VTM-12.1 software built from https://vccgit.hin fraunhofer.co de/
sAI (tttts::/gitthiil com/ Int VIM-://MMM122222 12.1 and the script from Compres- dataset evaluation. Specifically, the following command is issued to gather VTM-12.1 image com- pression evaluation results:
jvet wwet/WWSofttaae VIM/-/tags http////ag//VXX-12/
for
python -m compressai. utw.eutttllleee........ vtm [path to image folder] -c [path to WCSoftware__TT folder]/cfo http:///enoerrnitrr__rieeeee [path to WCCsoftware VTM folder] /bin
16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40
BPG: BPG software is obtained from ht:ps: mands are used for encoding and decoding.
and the following com-
bpgenc -e x265 -q [O to 51] -f 444 [encoded heic file] [original png file] bpgdec -o [decoded png file] [encoded heic file]
B.2.2
VIDEO CODECS
HEVC (x265)
ffmpeg -y -pix fmt yuv420p [resclution] [frame-rate] -i [input yuv420 raw video] -c:v 1ibx265 -preset medium -erf [9, 12, 15, 18, 21, 24, 27, 30] -tune zerolatency -x265-params [output mkv file path]
AVC (x264)
ffmpeg -y -pix_fmt yuv420p -s [resolution] -r [frame-rate] -i [input yuv420 raw video] -c:v 1ibx264 -preset medium -erf [9, 12, 15, 18, 21, 24, 27, 30] -tune zerolatency -x264-params [output mkv file path]
[Path to HM foldee://hhh//ppghhcooessatt
-c [Path to HM folder] /tctee///cct//rrrodee_oooeeeee.p Baaas___maannffii
-i [input yuv raw video] --IrputBitDepthheeeeet wwdt [width]
-het [height] -fr [frame-rate] -f [number of frames]
-o [output yuv video] -b [encoded bitstream bin file]
-ip 12 -q [12, 17, 22, 27, 32, 37, 42]
C BD RATE COMPUTATION
def Bjontegaarc_Delta__eee
# rate and psnr in ascending order rate_ref, psnr_ref, reference rate_new, psnr_new, new result
):
min_psnr max (psnr_ref [C], panr_new [0], 30)
19

---

Published as a conference paper at ICLR 2022
max psnr min (psnr_reff] [-1],
psnr new (enr_new[-1] 44)
log_ rat_rae__raa ref log (rate_ret
log_rate_new log (rate_new)
spline_red
www..ittepolate.uubic...
panr_ref, log_rate_ref, be_type-' not pott--knoe a-knot', extrapolate=True,
spline_new cripy.interpolate. inm
psnr_new, log_rate_new, be_type=' not-a-tt-enn -ot-aa-not', extrapolate=True,
delta delta_log_rate =
spline_nes new inttegratee (min_psme max_psnr)
spline_ref. integrate (min_psnt, max_psnr)
delta_rate exp _eelta_log_rate _sax_ssnr min psnz))
return 100 (delta_rate 1)
C.I BD RATE FOR IMAGE CODEC
Evaluate BD-rate on an image dataset bd_rates list () for image in
dataset:
# evaluate rate and panr reference and codec
# for this image with different qualities rate_ref, panr_ref ReferenceCodec (image, ep=(...)) rate_new, panr_new = NewImageCodec (image, beta=(...] bd_rates append $jontegaard__elle
rate_ref, psnr_ref,
rate_new, panr_new,
BD is computed per image and then averaged bd_rate rate bd_rrates.maat
C.2 BD RATE FOR VIDEO CODEC
Evaluate BD-rate on a video dataset bd_ bd_rates list ()
for video in video_datased
# evaluate rate and panr on reference and new codec # for this video with different qualities rate_ref, psnr_res rate_new psnr_ new = NewVideoCodec (video, beta=[...]] bd_rates
ReferenceCodec (video, [p=(...])
@jjontegarr_Deelt__RRat rate_ref, psnr_ref, rate_new, psnr_new,
BD is computed per video and then averaged bd_rate bd_rates.meat
20

---

Published as a conference paper at ICLR 2022
D MORE RESULTS
D.1 IMAGE COMPRESSION
Additional rate-distortion results on CLIC2021, Tecnick, and JPEG-AI are provided Figure 14 Figure 13 and Figure T6
For a complete comparison results from existing literatures, we provide a summary RD plot of all neural image codec solutions known to in Figure 28 Kodak. In Figure 29 and Figure 30 plot the percentage rate saving with BPG444 and M1M-12.1 as reference, respectively.
CLIC Test 2021
46
SwinT-ChARM SwinT-Htperprrirr
Conv-typerprior VTM-12.1
Conv.ChARM
44
CLIC Test 2021
42
Rate saving 195) over VTM-12.11 flarger better)
PSNR (RGB)
40
38
SwinT-ChARM
SwinT-Hyperprie
-20
36
Conv-ChARM
Conv-Hyperprior VTM-12.1
-30
34
BPG444
0.25
0.50
0.75
1.00
1.25
1.50
1.75 2.00
PSNR (RGB)
(bits per pixel)
(b) BD-rate reduction over VTM-12.1.
(a) Rate distortion comparison.
Figure 14: Comparison of compression efficiency CLIC Test 2021.
Tecnick Test
SwinT-ChARM ChARM
Conv Hyperrirrr
44
SwinT-Hyperprior
VTM-12.1
Conv-ChARM
42
Tecnick Test
10
40
Rate saving (85) over over VTM-12.1
PSNR (RGB)
Harger better)
38
-10
36
SwinT-Hii Hyperrrrpr
-20
Conv-ChARD
Conv-Hyperprior
34
-30
VTM-12.1
BPG444
32
0.5
1.0
1.5
2.0
2.5
PSNR (RGB)
Rate (bits per pixel)
(b) BD-rate reduction over VTM-12.1.
(a) Rate distortion comparison.
Figure 15: Comparison of compression efficiency on Tecnick Test.
21

---

Published as a conference paper at ICLR 2022
JPEG Al Test
SwinT-ChARM ChARM
Conv Hyperrrrerrr
SwinT-Hyperprior
VTM-12.1
42
40
JPEG AI Test
Rate saving (%) over over VTM-32.1
PSNR (RGB)
(larger better)
36
-10
SwinT-ChARM
SwinT-Hyperprior
-20
Conv-ChARM
32
Conv-Hyperprior VTM-12.1
-30
BPG444
30
0.5
1.0
1.5
2.0
2.5
PSNR (RGB)
Rate (bits per pixel)
(b) BD-rate reduction over VTM-12..
(a) Rate distortion comparison.
Figure 16: Comparison of compression efficiency JPEG Al Test.
D.2 VIDEO COMPRESSION
In Figure 17 and Figure 18,
we provide performance comparison of Conv-SSF, SwinT-SSF, HEVC (x265), and AVC (x264) with per-video breakdown.
MCL-CCV- File Relative HEVC (x264)
HEVC (X265)
File size rabo 055
Cams-SSe (repraduced) SwinT-SSF
100
75
50
29
21
08
01
17
07
14
26
02
22
05
19
00
18
15
30
20
Figure 17: Rate savings for each video the MCL-JCV dataset. Values represent the file size
relative to H.264 as estimated by BD rate. [18, 20, 24, 25] are animated sequences.
UVG File Size Relative to HEVC (x264)
Conv-S5F (reproduced) SwinT-555 55F
File size rablo
Beauty
ShakeWDry
Beaphorus
HoneyBep
YachtRidde
Jackey Ready SetGo
Figure 18: Rate savings for each video in the UVG dataset. Values represent the file size relative H.264 as estimated by BD rate.
D.3 CODING COMPLEXITY
We evaluate the decoding complexity of all neural image codecs in terms of the time for network inference and entropy coding, peak GPU memory, model size, etc. We select 100 high resolution images from CL.CC testset and center crop them to three resolutions (768 512, 1280 x 768, 1792 x
22

---

Published as a conference paper at ICLR 2022
1024) to how those metrics scale with image size. Batch size is one for all model inference. We run the experiment on local workstation with one RTX 2080 Ti GPU, with PyTorch 1.9.0 and Cuda toolkit 11.1. For bit-exact entropy coding, need to deterministi convolution. The neural networks run on the single GPU and entropy coding runs on CPU with 8 threads. We follow the standard protocols to measure the inference time and peak memory of neural nets, such GPU warm up and synchronization. File open/close is excluded from coding time measurement. MACs, GPU peak memory, and model parameter count profiled using vet_modell. prodel.profile function of deepspeed profile 14
We show more details on the coding complexity of neural image codecs in Figure 19 particularly the linear scaling to image resolution of both SwinT-based and Conv-based models. The break-down of encoding complexity is shown in Table
Transform inference time (ms)
SwinT encoder
450
SwinT-Hyperprior
SwinT decoder
400
Conv Hyperpriar
Conv encoder
decoder
Encoding time (ms)
350
Conv-ChARM
400
300
250
300
200
200
150
100
100
0.4
1.6
1.8
Mega pixels
Mega pixels
800
Conv-Hyperprior
SwinT-Hyperprior
Decoding peak memory (GB)
SwinT-typerprior
Conv-Hyperprior
Decoding time (ms)
700
SwinT-ChARM
Conv-ChARM
Conv-ChARM
4:
500
300
100
0.4
1.0
Mega pixels
Mega pixels
Figure 19; Detailed profiling single RTX 2080 Ti with deterministic Conv encoder and Conv decoder are of symmetric architectures, Conv decoder takes more time than Conv encoder mainly because Transposed Conv layers in the decoder take much longer than Conv layers in the encoder.
Note that while
For completeness, we also report the profiling for CPU coding time in Table and Table 6 The evaluation setup is the same as the GPU profiling case, except models are run on the CPU instead (same host machine with Intel(R) Xeon(R) W-2123 CPU @ 3.60GHz).
Table / reports encoding and decoding time of VTM-12.1 under different quantization parameters (QPs), evaluated on Intel Core 19-9940 CPU @ 3.30GHz, averaged over Kodak images. As can be seen from the table, decoding time of 12. is function of reconstruction quality, where longer decoding time observed for higher quality reconstruction. In Figure] the reported VTM-- 12.1 decoding speed corresponds to QP value of 28, where the bpp value is similar to that obtained by models trained with 0.001. is worth pointing out that VTM-12.1 encoding process much slower, ranging anywhere from to minutes per image, whereas neural codec runs much faster.
algori
Fusage cete-ccttideeeeeeee tegg
23

---

Published as a conference paper at ICLR 2022
Table 4: GPU encoding complexity. All models are trained with 3 0.001, evaluated on the resolution 768 x 512 (the average bitrate is around 0.7 bpp). For the encoding time, we show the network inference time for the encoder Ias the hyper encoder ha the hyper decoder hg the entropy encoding time for the hyper latent 2 and the latent (including network inference time for the prior model it is ChARM). GMACs and GPU peak memory during encoding and the parameter count of the entire model are also listed.
Time (ms)
Codec
GMACs
Memory (GB)
Params (M)
9n
ho.
2
h
y
total
Conv-Hyperprior
22.7
2.3
2.8
4.0
29.8
62.0
102
0.62
21.4
Swin'-Hyyperprio
57.5
1.2
2.4
4.8
34.3
100.5
100
1.47
24.7
Conv-ChARM 22.9
2.4
2.6
4.1
63.4
95.7
114
0.66
29.3
SwinT-ChARM 57.8
1.3
2.4
4.8
68.7
135.3
112
1.51
32.6
Table 5: CPU decoding complexity. All models are trained with 3 = 0.001, evaluated on 768 x 512 images (average 0.7 bpp). Decoding time broken down into inference time of dyper-decoder h, and decoder Is. entropy decoding time of hyper-code ž and code ý (including inference time of the prior model it ChARM).
CPU decoding Time (ms)
Codec
GMACs
h
y
9s
total
Conv- Conv-Hyperprion
5.1
20.9
30.5
1636.2
1703.3
350
SwinT-Hyperpric
5.7
18.3
30.3
1912.8
1979.5
99
Conv-ChARM,
4.7
20.9
154.6
1634.3
1825.6
362
SwinT-ChARM
5.8
18.2
141.1
1877.3
2054.7
111
Table 6: CPU encoding complexity. All models trained with 3 0.001, evaluated the resolution 768 x 512 (the average bitrate is around 0.7 bpp). For the encoding time, we show the network inference time for the encoder Ja the hyper encoder ha the hyper decoder h2, the entropy encoding time for the hyper latent & and the latent ý (including network inference time for the prior model it ChARM).
CPU encoding Time (ms)
Codec
GMACs
Ja
ha
à
h
y
total
Con--Hyperprior 896.0
8.9
2.7
20.9
31.1
960.9
102
SwinT-Hyperprior
1781.6
14.5
2.6
18.3
29.7
1847.9
100
Conv-ChARM, 895.7
8.9
2.5
20.9
146.3
1075.5
114
SwinT-ChARM 1697.1
14.5
2.6
18.2
133.5
1867.1
112
Table 7: Decoding and encoding time of VTM-12.1 averaged over 24 Kodak images.
QP
bits per pixel
PSNR (dB)
Decoding time(s)
Encoding time (s)
16
2.5441
44.09
0.284
300.47
28
0.7844
36.76
0.249
118.85
40
0.1557
29.51
0.149
71.10
E ANALYSIS
E.
SPATIAL CORRELATION OF LATENT
We visualize the spatial correlation map for Conv-Hyperpric and SwinT-Hyperprior at different
in Figure 20
24

---

Published as a conference paper at ICLR 2022
E.2 EFFECTIVE RECEPTIVE FIELD
See Figure 21 for the effective receptive field for the composed encoding tranforms ha Iu and Figure for the mean attention distance visualization of each head within each transformer layer.
E.3 RATE DISTRIBUTION ACROSS LATENT CHANNELS
generally believed that ConvNets learn to extract various features and store them in each chan- nel of the activations. Here we look into the features in the latent channels which are to be quantized and entropy coded to bitstreams. Particularly we order the total bitrate cach channel averaged over Kodak dataset (24 768 256 images). The result is shown in Figure 23 We find interesting phenomenon across models under different bitrates: there is a cutoff point of the bitrate-vs-chamee curve where the bitrate suddenly drops to zero, which manifest the rate constraint in the loss func- tion. As expected, the cutoff index decreases for the model trained for smaller bitrate (larger 3).
E.4 CENTERED KERNEL ALIGNMENT
To investigate the difference or similarity between latent features of Conv-based and SwinT-based models, we resort commonly used tool in representation learning called centered kernel align- ment (CKA) (Kornblith et al. 2019). We evaluate CKA between each of the Conv latent channel and Swin'T latent channel (both models are trained under the same 3) over the 24 Kodak images. There are 320 channels for both Conv latent and SwinT latent, resulting 320 x 320 CKA matrix. The result is shown in Figure 24 The latent channel is ordered by the averaged bitrate of each channel over Kodak images (same in Section E.3 The CKA matrix has clear block structure, where high similarity region corresponds to the latent channels before the bitrate cutoff in the rate distribution curve (Figure 23).
Identification of Swin'T and Conv latent channels with CKA Within the block of high similarity (from the CKA matrix), we identify the 'less' similar Swin'T latent channels with lowest CKA values between this SwinT channel and all other Conv latent channels. For each of the identified SwinT channel, we find the Conv latent channel with the largest CKA value between the two. This way, we are able to identify latent channels of two different models with high similarity. We show the identified top 8 channels in Figure 25 The channels are indeed highly similar, up to a sign flip, even through the two model architectures are quite different. This empirical result is relevant the literature on the identifiability of generative models (Khemakhem et al. 2020).
E.5 PROGRESSIVE DECODING
More visual examples of reconstructions of checkerboard (spatially) masked latent are provided in
Figure 26
Channel-wise progressive decoding Here we visualize the behavior of thannel-wise progressive decoding of Conv and SwinT models. For both models, we order the latent channels with a heuristic importance metric: the reduction of distortion over the increase rate if one channel is included for decoding. We start with the order of bitrate per channel, we pass the leading channels of bitrate order (zero out all rest channels) to the decoder to obtain reconstruction and calculate distortion. We plot the top 8 channels following this importance order. For each channel, show maps from top to bottom: latent values, mean prediction from the hyper decoder, standard deviation prediction from the hyper decoder, the bitmap, the reconstruction with only current channel, the reconstruction with up to current channel (all leading channels). The result is shown in Figure 27 For Conv models, usually the top 3 important channels are responsible for lightness, and two color components (blue- yellow, red-green), similar to the LAB colorspace. The rest of the latent channels are responsible for adding details like texture and edges. For the Swint models, at low bitrate, there is one significantly different channel (the first column in Figure 27b). which is in coarse scale (with smooth blocks) and responsible for the reconstruction a nearly constant image with value close to 120 (the mean value of natural image dataset). This latent channel costs extremely small bitrate but reaches PSNR of 13dB. We tried remove this first channel, the progressive reconstruction with the rest leading channels only leads to PSNR around 16dB, instead of 26dB shown in the figure.
25

---

Published as a conference paper at ICLR 2022
F MORE ABLATION STUDIES
Local self-attention To see local self-attention is the most important component in transform- ers, We replace it by depthwise separable convolution block (Han al. 2021 El-Nouby al, 2021), which performs similar spatial feature aggregation as sell-attention, while keeping all other components the same as the SwinT. We found this change only leads minor degradation in RD. This suggests other components in transformers such as MLPs and skip connections may also play big role, other than just self-attention, for the leading performance our work and many other tasks (Dong et al. 2021).
Small depths Upon investigating the mean attention distance as shown in Figure 22 we find the last block in each of the last two encoder stages has about half of its attention heads degenerate to attending to fixed nearly pixels. This suggests redundant transformer blocks at that stage, so we remove those two blocks, i.e. from depths 2, 2] to [2, 1j. The resulting SwinT-Hyperprior has even less parameters (20.6M) than Conv-Hyperprios (21.4M) while with almost no RD loss compared the larger model. We expect more hyperparameter search will identify models with better RD-complexity trade-off than we currently show in this work.
Deeper Conv encoder Deeper models are usually more expressive (Raghu et al. 2017) and the state-of-the-art Conv-based compression models typically use much deeper layers than the encoder in the original Hyperprior model (Ballé et al., 2018). As a sanity check on whether deeper con- volutional transforms
outperform T-bsed encoder transforms with 12 blocks, we take an existing design (Chen et al. 2021) with residual blocks and attention (sigmoid gating) layers, which has over 50 conv layers in either encoder decoder, and more parameters than baseline. It indeed improves the RD in lower bitrate, but still worse than Swinr-Hyyerpprior, and gets much worse in higher bitrate. This is probably the reason that compression models based on this type of transforms did not report results at higher bitrates.
1"Convl http:h/wwaaaaaaaooooooooo////oee
26

---

Published as a conference paper at ICLR 2022
0.340
0.0033
0.0060
0.0071
0,2027
0.0026
D.0017
0.0218
0.2064
0,000M
0,0010
0.335
0.330
0.0033
0.0016
0. 0.925
0.0078
0.0096
1.3
0.0098
5.0078
0.005
6,0090
3.0
6,0090
0.205
0.020
0.015
0.000
0. 0.0023
0.0027 0.0 6.0070
0.0039
0.0032
2 0.0007
6.0006
0.0041
6.0034
0.0017
0.205
0.200
wwyprpproriBettaoooooo 0.000))
wwwpni/tytrrriitttttt--aa 0.000))
Kodalc PSNR= 80B, BPP=1.60
Kooak: PSNR= 83dB. BPP=1.36
-1
0.040
0.0051 0.0071 8.0 8.0131
0.0061
0.0036
-2
0.0026
6,0022 0.0050
6.0016 0.0016
0.015
0.0069
0.0083
0.0231
0.0084
0.0059
0.0027
0.0210
0.0090
0.0021
0.0018
0-
0.0066
0. 0.0062
0.0087
0.0212
0.0085
5.0088
0.0022
6.0321
0.0091
0.0012 0.0029
www.-HyeerrroorBeeta---o 0,000000003))
SwinT--yppeprrorrBeeta 0.0003)
Kodal: PSMR 37 62dB, BPP- 1000
Kodak: PSNR- A545, BPP 0..9
1 0.0151 0.0210
0.0514
0.0184
0.0125
-1 0.0052
0,0047
0.0140
0.0047
0.0033
0.025
0.0307
0.0505
0.0132
0.0171
0.0170
0.0133
0.000
0.0129
1
0.0947
0.0050
0.010
0.0076
0.118
6.0255
0.0124
0.0106
0.0227
6.0032
0.0096
6.0031
0.2041
0.205
0.000
Conv-Hyyesrpiorreeetaoooo PSNR=
SwinT Wotitt/tyyrrrrioBBtaaao00000
34.458B, UPP=0.52
0.240
0.0361
0.088
8.0388
0.0181
0.0150
0.0062
0,0062
0.0146
0.0070
0.0053
0.235
0.230
0.025
0-0.0459
0.0824
1.0
0.0819
0.0455
0- 0.0188
6.0272
1.0
0.0269
0.0181
0.020
0.015
0.0196
0.0239
0.0701
0.0241
0.0205
0.0071
0.8281
0.0208
0.0083
0.0074
0.010
0.0150
0.0393
2
0.0064
0.0058
0.005
Coru-HyyerpriorrBetaa Beta-O 0000)
SwinT HoiiT--HpprrriorBBetaa 003)
Kodak: PSNR-31. 31,09d8, BPP-Q.28
Kooak: PSNR- 31,6ddB,
.666, 8PP-0.79
Figure 20: Spatial correlation (y - (0())((()/))) averaged across all latent elements of all images on Kodak. The value with index (i,j) corresponds to the normalized cross-correlation of latents at spatial location (w, h) and (w + é, + f). Left column corresponds to Conv- non--Hyerrprior and right columns corresponds to Swin SwinT-Hyperprioor where the 3 values from top to bottom are 0.0001, 0.0003, 0.001, 0.003. consistent observation across all these models that Swinr-Hyperpriorr achieves uniformly smaller correlation than its convolutional counterpart. As 3 gets smaller (rate becomes lower), the correlation in both models increase, with Swin H-yerrror increasing much slower than Hyperpror...
Each row shows pair of models trained the same 3,
27

---

Published as a conference paper at ICLR 2022
(a) Conv-Hyperprior
(b) Cony-ChARM
(c) Conv-SSF-Ro
(d) Conv-SSF-Flee
sew
coos
50
300
238
c001
(c) Swin Sewin--Hyperpriin
(f) SwinT-ChARM
(g) T-SSI--SSS-RR
(h) SwinT-SSF-Flow
-200
238
cocs
-120
-35
-50
-5
co01
3000
Co03
-300 300 100
Figure 21: Comparison of effective receptive field (ERF) of the composed encoders ha Ja- The ERF is visualized as the absolution gradients of the center pixel in the hyper latent (i.e. dz with respect to the input images/videos, specifically 24 Kodak images cropped to 512 x 512 for image codecs (the left four models, all with 0.003), and 24 randomly selected batches from UVG video dataset cropped to 768 x 768 for P-frame codecs (the right four models, all with B 0.0008).
Mean attention distance
Mean attention
2.5-
2.0
2.5-
40
60
Sorted attention head
(a) SwinT-Hyperprioo encoder, 8 0.0001
(b) SwinT--Hyerrpiior decoder, / 0.0001
Mean distance
Mesn sttenton distance w
Sorted attention head
Sorted attention head
(c) Swin S-ipe-Hyyrrrrior encoder, _ = 0.001
(d) Swin decoder, 0.001
Mean attention distance of SwinT-Hyperprior models evaluated on Kodak. It calculated as the average relative distance between each query and key weighted by the attention weight for each query, in each head each layer. Each vertical color bar in the figure shows the mean and 1-std for the mean attention distance of one head, dashed black bar separates heads from different transformer stages (feature resolutions). The order of input to output is from left to right within each figure. Lower bitrate models have smaller attention distance.
Figure 22:
28

---

Published as a conference paper at ICLR 2022
Meea BPP per channel over Kodak
Comv-Hlyperprior
Mean BPP per channe! over Kodak
10
0.007
SwinT-typerprior
10-3
0.006
0.005
0.004
10
Conv-Hyperp
SwinT Hyperprior
10
0.002
10-7
0.001
0.000
10.0
100
150
200
250
300
250
300
Channel Index (sorted by BPP)
Channel Index (sorted by BPP)
(a) Hyperprior (rate linear scale)
(b) Hyperprior (rate in log scale)
10
Mean per channe over Kodak
0.006
Conv-ChARM SwinT-ChARM
Mean 3PP per channel over Kodak
10
0.005
10
0.004
0.003
0.002
10
0.001
10-7
Canv ChARM SwinT-ChARM
0.000
10
0
50
100
150
200
250
300
50
100
150
200
250
300
Channel index (sorted BPP)
Channel Index (sorted by BPP)
(c) ChARM prior (rate in linear scale)
(d) ChARM prior (rate log scale)
Figure 23: Rate distribution of latent channels. Five lines with the same colormap from bright to dark corresponds to the model trained under five 3 values from small to large (i.e. overall bitrate from large to small). The cutoff index decreases the total rate goes down.
(a) Hyperprior 0.003
(b) Hyperprior 0.001
(c) Hyperprior 3c-4
(d) Hyperprior lc-4
(c) Hyperprior 3c-5
ChARM 0.003
(g) ChARM 0.001
(h) ChAMR 3e-4
(i) ChARM le-4
(i ChARM
Figure 24: Centered Kerne Alignment between 320 SwinTT latent channels and 320 Conv latent
channels. Blocks with high similarity correspond to latent channels before the cutoff Figure 23
29

---

Published as a conference paper at ICLR 2022
33 Esssso
Hyperprior, & 0.003
(b) Hyperprior 3 0.00003
(c) ChARM, 8 = 0.003
(d) ChARM 3 = 0.00003
Figure 25: Identification of latent features from Swin'T and Conv models with CKA (on 23rd image in Kodak). For each subfigure, top row shows 8 channels of SwinT latents with the smallest mean CKA values between this channel and all Conv latent channels (before the rate cutoff in Figure 23 The corresponding image on the bottom shows the Conv latent channel with the largest CKA with the top Swin'T channel. The identified channels are highly similar, up sign flip.
30

---

Published as a conference paper at ICLR 2022
matthaen-heoSSAAAvv.uuuuphhh
(a) unsplash
sitnrrsuu-nnngh--uaaa---S001 Qx w-
Figure 26: More examples of reconstructions with SwinT-Hyperprior (top row) and Conv- Hyperprior (bottom row) for the latent masked with checkerboard pattern where half of the latent dequantized values are replaced with the corresponding prior mean before passing to the decoder Is- The example image shown in TigureBblis andy-kelly-0E Www..0.....WV....g.. unssplash from CLIC2021 test, same as the two images shown here.
31

---

Published as a conference paper at ICLR 2022
(a) Conv- Hony-Hyperprior, 0.003
(b) Swin Hyyperrrrrrroo 8 = 0.003
Figure 27: Channel-wise progressive decoding. The 3 leading latent channels following an impor- tance order are responsible for grayscale, and two color components of the reconstruction (second to the last row). For Swin'T models, we find a coarse-scale latent channel (the left most channel) which costs negligible bits but important for reconstruction (without it, the other leading chan- nels can only decode to reconstruction with PSNR around 16dB), For each subfigure. from top bottom: latent, prior mean, prior std, bitmap, reconstruction with current channel, reconstruction with leading channels (up to current channel).
32

---

Published as a conference paper at ICLR 2022
Kodak
42
40
PSNR (RGB)
38
SwinT-Hyperprior
Canv-ChARM
Conv-Hyperprior
VTM-12.1
BPG444
Cheng 2020
Xie et al. 2021
Ma al. 2021
Guo et al. 2021
Balle et al. 2018
Minnen al. 2020
32
2019
Minnen 2018 (no context)
Minnen al. 2018 (with context)
0.5
1.0
1.5
2.0
2.5
Rate (bits per pixel)
Figure 28:
Rate-distortion performance Kodak, comparing with existing works (Cheng et al.
2020 Xie et al.
2021 Ma et al. 2021 Guo et al. 2021 Ballé et al. 2018 Minnen & Singh 2020
Lee al. 2019
Minnen al. 2018). Dashed line-style indicates the of spatial autoregressive model (block-wise or pixel-wise) as prior model.
Wu al. 2020
33

---

Published as a conference paper at ICLR 2022
Kodak
30
Rate saving (%) over BPG444 (larger is better)
10
SwinT-ChARV CATA-
SwinT-Hyperprior
Conv-ChARM
Conv-Hyperprior
VTM-12.1
BPG444
Cheng et al. 2020
Xie et al. 2021
Ma et al. 2021
Guo et al. 2021
Balle et
2020
-10
Lee 2019
et al. 2020
Minnen al. 2018
(na context) x
Minnen 2018 (with context)
30
32
34
36
38
40
44
PSNR (RGB)
Figure 29:
Percentage of rate-saving over BPG444 evaluated on Kodak (extended version of Fig-
ure
3b
comparing with existing works (Cheng et al. 2020; Xie et al. 2021 Ma et al.) 2021 Guo
al.
2021
Ballé et al. 2018 Minnen & Singh 2020 Lee al.) 2019 Wu et al. 2020 Minnen
al.
2018).
Dashed line-style indicates the use of spatial autoregressive model (block-wise or
pixel wise) as prior model.
34

---

Published as a conference paper at ICLR 2022
Kodak
10
Rate saving (%6 over VTM-12.1 (larger is better)
-10-
-20
-30
SwinT-ChARM
SwinT-H Hype-ryperrrioor
Conv-ChARM
Conv-Hyperprior
VTM-12.1
BPG444
Cheng et al. 2020
Xie et al. 2021
400
Ma et al. 2021
Guo et al. 2021
Balle et
Lee 2019
Wu et al. 2020
Minnen al. 2018
(na context! xt)
Minnen 2018
-50
(with context)
30
32
34
36
38
42
44
PSNR (RGB)
Figure 30: Perrenttaee of rate-saving over VTM- 12.1 evaluated on Kodak (extended version of Fig-
ure
3b
comparing with existing works (Cheng et al. 2020; Xie et al.) 2021 Ma et al. 2021 Guo
al.
2021
Ballé et al. 2018 Minnen & Singh 2020 Lee al.) 2019 Wu et al. 2020 Minnen
al.
2018). Dashed line-style indicates the use of spatial autoregressive model (block-wise or
pixel wise) as prior model.
35


Reviews:
[{"review_id": "w2DWDoD0j_y", "reviewer": "Reviewer_aEdf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed transformer-based framework shows good performance, proving that the transformer is a good replacement for typical convolution for compression tasks. \n\nThe experiments are rich, and enough evidences from many different aspects show that the transformer is just better than custom convolution. \n\nHowever, a more important question is why transformer is better for image compression network, there is a lack of good organization and dig-in analysis over these many experiment results.\n", "summary_of_the_paper": "This paper proposed to replace the typical cnn-based transform in image and video compression networks by Swin-transform. Experiments show its better performance and lower computational complexity over cnn-basd methods. Some extensive analysis are conducted to explore the differences between convolution and transformers.", "summary_of_the_review": "The contribution of this paper is simple and direct, and the results also show good performance. But since transformer itself is not new, I wish to see deeper analyze over the experiments. So my recommendation is marginally below the borderline.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MMfQfZpR9D3", "reviewer": "Reviewer_U2rZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe authors show strong results for SwinT networks across two entropy modeling networks (channel-wise autogressive and hyper prior) for image compression and scale-space architectures for video compression at much faster decoding times.\n\nSwinT's high relative performance also appears to be more resilient to scaling the network to smaller capacities, as compared to a Conv model. The small SwinT is half the BD-rate loss of the Conv equivalent at under half the speed.\n\nLatent analysis for SwinT shows a uniformly smaller correlation than the Conv models. It is a noteworthy result and leads to future research questions on if imposing that form leads to better image compression or if it is a result of a naturally better method. This question would also hold for the receptive field analysis, where SwinT's receptive fields are more rectilinear than the Conv models, which may be showing that Conv models are getting mostly unneeded latents influencing the reconstructions.\n\nOverall, this paper includes an abundance of strong results, across multiple architectures and datasets for image and video compression. The appendices have a plethora of detail about training, architecture design and results that should make this paper very easy to reproduce and verify.\n\n\nWeaknesses:\nIn Figure 19, SwinT models are shown to be much slower at encoding than Conv models while much faster at decoding. Is there a possible shift of compute going on in the SwinT networks that aren't being taken advantage of in the Conv models?\n\nThe gap at low bitrates is amazing, the SwinT models heavily out perform the Conv models, but that is gain disappears at higher quality / rates, any intuition on why that is?\n\nThe image and video codec baselines were used in medium and with low/zero latency optimizations enabled. This will generally put classic codecs at a lower R/D in comparisons. This would generally be appropriate if you're are comparing with a method that has a very fast encode, and in Figure 19, SwinT models are strictly slower at encoding than Conv models.\n\nIn Table 3 the SwinT entropy transform models h_s are slower than their Conv counterparts, is there an explanation for that? Is there performance to gain by using the Conv variants there and using SwinT exclusively for the image synthesis layers g_s?\n\nSuggestions / Comments:\n4.1 Training = \"3.5M and 3.1M batches\" => \"3.5M and 3.1M steps\" is a clearer description for describing how many training steps the network sees.\n\nFootnote 11 in the Appendix is repeated three paragraphs above, feel free to remove one of them:\n\"We did not use the crop size 384 × 384 during the second stage as in the original paper because the\nresolution of Vimeo dataset is 448×256. We found in our case increasing crop size from 256×256\nto 384 × 256 in the second stage does not improve RD.\"\n\nDo you have any timing for the SwinT vs. Conv models without a GPU? Do SwinT models have a runtime advantage with using a larger working set (max GPU memory) of very fast GPU memory vs the Conv models that don't have as high of a max memory usage?\n\n", "summary_of_the_paper": "In this paper, the authors show that applying the Swin Transformer to both neural and image video compression show significant rate distortion gains over Conv Transformer methods and provide an analysis of latents and ablasion studies to understand the difference in the two methods.", "summary_of_the_review": "This paper shows that Transformer based transforms can replace Conv based transforms in image and video compression, and simultaneously achieve better RD performance at much faster decode times. The results are strong across datasets for image and video compression and multiple entropy modeling techniques. The abundance of evidence and details, along with the state of the art results in performance make this a very strong paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "I0SvSigTNft", "reviewer": "Reviewer_7pae", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper is well organized and the authors provide very comprehensive results and analysis for SwinT in compression. For example, the analysis of ERF is quite interesting and provides some insights for the following works.  \n2. The experimental results are convincing. Although some latest works like Guo et al. achieve slightly better compression performance, the proposed approach is much neater. \n3. Basically, this is the first transformed-based coding solution, although performance is somewhat expected considering the success in vision transformer. The overall architecture and implementation are meaningful.  \n\nWeaknesses:\nSome related works or experiments are suggested.  \n(1) It would be better to provide results if SwinT based compression also uses more expensive entropy coding approaches. Could we further boost the compression performance?\n(2) There are also some approaches for low-complexity entropy coding approaches like He et al., Checkerboard Context Model for Efficient Learned Image Compression.  If we can use this approach to replace the expensive entropy coding method, could we achieve a better performance-complexity trade-off even we use Conv based model?  Please provide some discussion. \n(3) There are some related works in video compression, it is suggested for discussion or comparison.  \n(a) FVC: A New Framework Towards Deep Video Compression in Feature Space. Hu et al.\n(b) ELF-VC: Efficient Learned Flexible-Rate Video Coding. Rippel et al. ", "summary_of_the_paper": "This paper introduces the swin-transformer for the learned image and video coding. Instead of developing more advanced and expensive entropy coding approaches, the authors focus on the transform network(i.e., networks of encoder/decoder) and improve the compression performance significantly.  The experimental results are impressive. Although introducing SwinT is not very new, the authors provide a neat and practical solution, which should be encouraged. The authors also have a good balance between the coding gain and complexity. ", "summary_of_the_review": "I think the paper could be accepted considering its effective design for transformed-based compression as well as the comprehensive analysis. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "GXaBJU58pcA", "reviewer": "Reviewer_xjay", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Nice exploration in extending Swin-Transformer to a decoder setting and build Swin-transformer based neural image codecs. The experiments show that Swin achieved better rate-distortion performance with lower complexity than existing solutions.\n\n2. The authors further deomonstrated the effectiveness of Swin-transformer in video compression by enhancing scale-space-flow, a popular neural P-frame codec.\n\n3. For the detailed experiments, they show that the inference time of SwinT decoder is less than that of Conv decoder, also Swin has effective receptive field, incur less redundancy across different spatial latent locations, and progressive decoding", "summary_of_the_paper": "The authors extended Swin-Transformer to neural image codecs and demonstrated that Swin-transformers - SwinT-ChARM (proposed) can achieve better compression efficiency than convolutional neural networks (ConvNets), while requiring fewer parameters and shorter decoding time.", "summary_of_the_review": "The authors did a novel exploration in using Swin-Transformer for transform coding and proved that SwinT-ChARM (the authors proposed) demonstrated better compression efficiency as opposed to CNN counterparts. It is a nice exploration and provides new perspective in image codecs.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Fq7uHksQ3nc", "reviewer": "Reviewer_EyC9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The primary strength of the paper is reaching the empirical milestone of outperforming VTM in terms of both rate-distortion and decode runtime. This is achieved by merging existing models: swin transformers (SwinT) for transforms and a channel-wise autoregressive model (ChARM) for the entropy model.\n\nAnother strength is the thorough evaluation in terms of transform variants (SwinT vs. convolution-based), entropy models (ChARM vs. a hyperprior), architecture size (number of layers and channel depths), and application to p-frame encoding for video compression. The authors also include a thorough range of ablation experiments, look at the ability for different transforms to generate a spatially independent latent representation, and explore the effective receptive field.\n\nOne weakness of the paper is in the runtime comparison between VTM and the various learned methods, though this is generally a difficult comparison to make. The difficulty arises because VTM runs on CPU (the authors state that an i9-9940X was used), while neural networks are typically run on a GPU or TPU (the authors say that an RTX 2080 Ti was used). This leads to an apples-to-oranges comparison that should be stated more explicitly. Decode times on the CPU would also strengthen the paper.\n\nIn addition, the runtime of VTM varies with encoding quality. At least based on our tests (caveat: on a different CPU and with a different version of VTM), VTM can decode a 768x512 image in anywhere from 20ms (at the lowest quality level) up to 230ms for the highest quality (note that these numbers include disk I/O). Based on this, a more accurate claim is that SwinT-Charm decodes faster than VTM at some quality levels (very high ones), but can be much slower for high compression rates.\n\nAlthough the authors don't mention it, VTM can actually be very slow to *encode* an image at a very high quality level (on the order of dozens of seconds). I agree that decode speed is most important for typical applications (since images are often encoded once and decoded millions of times), the relatively fast encode speed of the proposed model can be seen as a strength.", "summary_of_the_paper": "This paper addresses the problem of improving rate-distortion (RD) performance for learned image and video compression without ignoring runtime. Much of the literature on learned compression improves RD performance with more entropy models. Especially for autoregressive models, these models can have excessively slow decode times. This paper uses a less expensive (and generally less powerful) entropy model and explores the use of a transformer in the encoder and decoder transforms instead.\n\nThe result is a much faster model that still achieves near-SOTA rate-distortion performance. In particular, the SwinT-ChARM model described in this paper outperforms VVC (via VTM 12.1 a recent version of the reference implementation for H.266) with faster decode times. As far as I know, this is a milestone for learned image compression (other methods have outperformed in terms of RD but only with much slower decoders).\n", "summary_of_the_review": "Despite some reservations about the technical novelty and the runtime comparison, I still think this paper deserves an \"accept, good paper\" recommendation.\n\nThis conclusion is based on the strong empirical results and the thorough evaluation. The results are also somewhat surprising to me, which is an indicator of important research. I would have expected transformers to improve RD performance compared to conv-nets, but I expected a smaller difference and for runtime to *increase*, not *decrease*.\n\nI also think that if I were to start working on learned image/video compression, I would use the model in this paper as a baseline.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper aims to improve image and video compression keeping in mind computation cost. In this regard, authors propose variants of Swin-Transformer for image and video coding. The experimental results shows that Transformer based transforms can replace Conv based transforms in image and video compression, and simultaneously achieving better rate-distortion performance at much faster decode times, i.e. resulting in a better rate-distortion-computation trade-off. We thank the reviewers and authors for engaging in an active discussion. The reviewers found some results to be surprising (in a good way) and are in a consensus that the empirical results are strong across datasets for image and video compression. For completeness, the authors should provide FLOPs or CPU runtimes in the final version so that one can compare to methods like VTM even if CPU is not the desired hardware for proposed method.
Decision: Accept (Poster)