Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors extended Swin-Transformer to neural image codecs and demonstrated that Swin-transformers - SwinT-ChARM (proposed) can achieve better compression efficiency than convolutional neural networks (ConvNets), while requiring fewer parameters and shorter decoding time.
main_review: 1. Nice exploration in extending Swin-Transformer to a decoder setting and build Swin-transformer based neural image codecs. The experiments show that Swin achieved better rate-distortion performance with lower complexity than existing solutions.

2. The authors further deomonstrated the effectiveness of Swin-transformer in video compression by enhancing scale-space-flow, a popular neural P-frame codec.

3. For the detailed experiments, they show that the inference time of SwinT decoder is less than that of Conv decoder, also Swin has effective receptive field, incur less redundancy across different spatial latent locations, and progressive decoding
summary_of_the_review: The authors did a novel exploration in using Swin-Transformer for transform coding and proved that SwinT-ChARM (the authors proposed) demonstrated better compression efficiency as opposed to CNN counterparts. It is a nice exploration and provides new perspective in image codecs.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper addresses the problem of improving rate-distortion (RD) performance for learned image and video compression without ignoring runtime. Much of the literature on learned compression improves RD performance with more entropy models. Especially for autoregressive models, these models can have excessively slow decode times. This paper uses a less expensive (and generally less powerful) entropy model and explores the use of a transformer in the encoder and decoder transforms instead.

The result is a much faster model that still achieves near-SOTA rate-distortion performance. In particular, the SwinT-ChARM model described in this paper outperforms VVC (via VTM 12.1 a recent version of the reference implementation for H.266) with faster decode times. As far as I know, this is a milestone for learned image compression (other methods have outperformed in terms of RD but only with much slower decoders).

main_review: The primary strength of the paper is reaching the empirical milestone of outperforming VTM in terms of both rate-distortion and decode runtime. This is achieved by merging existing models: swin transformers (SwinT) for transforms and a channel-wise autoregressive model (ChARM) for the entropy model.

Another strength is the thorough evaluation in terms of transform variants (SwinT vs. convolution-based), entropy models (ChARM vs. a hyperprior), architecture size (number of layers and channel depths), and application to p-frame encoding for video compression. The authors also include a thorough range of ablation experiments, look at the ability for different transforms to generate a spatially independent latent representation, and explore the effective receptive field.

One weakness of the paper is in the runtime comparison between VTM and the various learned methods, though this is generally a difficult comparison to make. The difficulty arises because VTM runs on CPU (the authors state that an i9-9940X was used), while neural networks are typically run on a GPU or TPU (the authors say that an RTX 2080 Ti was used). This leads to an apples-to-oranges comparison that should be stated more explicitly. Decode times on the CPU would also strengthen the paper.

In addition, the runtime of VTM varies with encoding quality. At least based on our tests (caveat: on a different CPU and with a different version of VTM), VTM can decode a 768x512 image in anywhere from 20ms (at the lowest quality level) up to 230ms for the highest quality (note that these numbers include disk I/O). Based on this, a more accurate claim is that SwinT-Charm decodes faster than VTM at some quality levels (very high ones), but can be much slower for high compression rates.

Although the authors don't mention it, VTM can actually be very slow to *encode* an image at a very high quality level (on the order of dozens of seconds). I agree that decode speed is most important for typical applications (since images are often encoded once and decoded millions of times), the relatively fast encode speed of the proposed model can be seen as a strength.
summary_of_the_review: Despite some reservations about the technical novelty and the runtime comparison, I still think this paper deserves an "accept, good paper" recommendation.

This conclusion is based on the strong empirical results and the thorough evaluation. The results are also somewhat surprising to me, which is an indicator of important research. I would have expected transformers to improve RD performance compared to conv-nets, but I expected a smaller difference and for runtime to *increase*, not *decrease*.

I also think that if I were to start working on learned image/video compression, I would use the model in this paper as a baseline.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces the swin-transformer for the learned image and video coding. Instead of developing more advanced and expensive entropy coding approaches, the authors focus on the transform network(i.e., networks of encoder/decoder) and improve the compression performance significantly.  The experimental results are impressive. Although introducing SwinT is not very new, the authors provide a neat and practical solution, which should be encouraged. The authors also have a good balance between the coding gain and complexity. 
main_review: Strengths:
1. The paper is well organized and the authors provide very comprehensive results and analysis for SwinT in compression. For example, the analysis of ERF is quite interesting and provides some insights for the following works.  
2. The experimental results are convincing. Although some latest works like Guo et al. achieve slightly better compression performance, the proposed approach is much neater. 
3. Basically, this is the first transformed-based coding solution, although performance is somewhat expected considering the success in vision transformer. The overall architecture and implementation are meaningful.  

Weaknesses:
Some related works or experiments are suggested.  
(1) It would be better to provide results if SwinT based compression also uses more expensive entropy coding approaches. Could we further boost the compression performance?
(2) There are also some approaches for low-complexity entropy coding approaches like He et al., Checkerboard Context Model for Efficient Learned Image Compression.  If we can use this approach to replace the expensive entropy coding method, could we achieve a better performance-complexity trade-off even we use Conv based model?  Please provide some discussion. 
(3) There are some related works in video compression, it is suggested for discussion or comparison.  
(a) FVC: A New Framework Towards Deep Video Compression in Feature Space. Hu et al.
(b) ELF-VC: Efficient Learned Flexible-Rate Video Coding. Rippel et al. 
summary_of_the_review: I think the paper could be accepted considering its effective design for transformed-based compression as well as the comprehensive analysis.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed to replace the typical cnn-based transform in image and video compression networks by Swin-transform. Experiments show its better performance and lower computational complexity over cnn-basd methods. Some extensive analysis are conducted to explore the differences between convolution and transformers.
main_review: The proposed transformer-based framework shows good performance, proving that the transformer is a good replacement for typical convolution for compression tasks. 

The experiments are rich, and enough evidences from many different aspects show that the transformer is just better than custom convolution. 

However, a more important question is why transformer is better for image compression network, there is a lack of good organization and dig-in analysis over these many experiment results.

summary_of_the_review: The contribution of this paper is simple and direct, and the results also show good performance. But since transformer itself is not new, I wish to see deeper analyze over the experiments. So my recommendation is marginally below the borderline.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors show that applying the Swin Transformer to both neural and image video compression show significant rate distortion gains over Conv Transformer methods and provide an analysis of latents and ablasion studies to understand the difference in the two methods.
main_review: Strengths:
The authors show strong results for SwinT networks across two entropy modeling networks (channel-wise autogressive and hyper prior) for image compression and scale-space architectures for video compression at much faster decoding times.

SwinT's high relative performance also appears to be more resilient to scaling the network to smaller capacities, as compared to a Conv model. The small SwinT is half the BD-rate loss of the Conv equivalent at under half the speed.

Latent analysis for SwinT shows a uniformly smaller correlation than the Conv models. It is a noteworthy result and leads to future research questions on if imposing that form leads to better image compression or if it is a result of a naturally better method. This question would also hold for the receptive field analysis, where SwinT's receptive fields are more rectilinear than the Conv models, which may be showing that Conv models are getting mostly unneeded latents influencing the reconstructions.

Overall, this paper includes an abundance of strong results, across multiple architectures and datasets for image and video compression. The appendices have a plethora of detail about training, architecture design and results that should make this paper very easy to reproduce and verify.


Weaknesses:
In Figure 19, SwinT models are shown to be much slower at encoding than Conv models while much faster at decoding. Is there a possible shift of compute going on in the SwinT networks that aren't being taken advantage of in the Conv models?

The gap at low bitrates is amazing, the SwinT models heavily out perform the Conv models, but that is gain disappears at higher quality / rates, any intuition on why that is?

The image and video codec baselines were used in medium and with low/zero latency optimizations enabled. This will generally put classic codecs at a lower R/D in comparisons. This would generally be appropriate if you're are comparing with a method that has a very fast encode, and in Figure 19, SwinT models are strictly slower at encoding than Conv models.

In Table 3 the SwinT entropy transform models h_s are slower than their Conv counterparts, is there an explanation for that? Is there performance to gain by using the Conv variants there and using SwinT exclusively for the image synthesis layers g_s?

Suggestions / Comments:
4.1 Training = "3.5M and 3.1M batches" => "3.5M and 3.1M steps" is a clearer description for describing how many training steps the network sees.

Footnote 11 in the Appendix is repeated three paragraphs above, feel free to remove one of them:
"We did not use the crop size 384 × 384 during the second stage as in the original paper because the
resolution of Vimeo dataset is 448×256. We found in our case increasing crop size from 256×256
to 384 × 256 in the second stage does not improve RD."

Do you have any timing for the SwinT vs. Conv models without a GPU? Do SwinT models have a runtime advantage with using a larger working set (max GPU memory) of very fast GPU memory vs the Conv models that don't have as high of a max memory usage?


summary_of_the_review: This paper shows that Transformer based transforms can replace Conv based transforms in image and video compression, and simultaneously achieve better RD performance at much faster decode times. The results are strong across datasets for image and video compression and multiple entropy modeling techniques. The abundance of evidence and details, along with the state of the art results in performance make this a very strong paper.
review_title: null

Meta-review: The paper aims to improve image and video compression keeping in mind computation cost. In this regard, authors propose variants of Swin-Transformer for image and video coding. The experimental results shows that Transformer based transforms can replace Conv based transforms in image and video compression, and simultaneously achieving better rate-distortion performance at much faster decode times, i.e. resulting in a better rate-distortion-computation trade-off. We thank the reviewers and authors for engaging in an active discussion. The reviewers found some results to be surprising (in a good way) and are in a consensus that the empirical results are strong across datasets for image and video compression. For completeness, the authors should provide FLOPs or CPU runtimes in the final version so that one can compare to methods like VTM even if CPU is not the desired hardware for proposed method.
Decision: Accept (Poster)