Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work introduces a transformer-based entropy coding model for learned image compression. The backbones of the commonly used hyperprior encoder and decoder are replaced with transformer encoder layers, and the context model is replaced with the transformer decoder. The striking features of the proposed method include (1) a diamond-shaped relative position encoding with clipping, (2) a top-k selection scheme, and (3) a parallel bi-directional context model. The parallel bi-direction context model is a derivative work from He et al. 2021. 
main_review: ** Strengths **

(1) The improvements over the CNN-based hyperprior and context models are promising. 

(2) The ablation study justifies well the design choices. 

** Weaknesses **

(1) Transformers are costly in terms of complexity and runtimes as shown in Table 2 (Transformer Serial). Some prior works introduce the non-local attention module (NLAM) to the hyperprior codec without using layers of transformer encoders. One example is "End-to-End Learnt Image Compression via Non-Local Attention Optimization and Improved Context Modeling", TIP2021. It is unclear how the proposed transformer-based hyperprior compares with such NLAM-based hyperprior design. 

(2)  It is expected that transformer-based design may not scale up easily from the complexity perspective when it comes to processing higher-resolution images. Essentially, its complexity may grow exponentially with the spatial resolution of the image latents. 

(3) In some ablation studies, the performance differences are expressed in BPP or delta BPP. It is unclear at what PSNR (or quality) level these numbers are reported. Why not report BD-rate savings and present a comparison of complete RD curves? The same argument applies to Figs. 6 & 7. 

(4)  The number of network weights spent on the entropy coder should be compared.  
summary_of_the_review: Overall, the idea is novel and interesting. The results look promising. However, there are some critical details missing. Further clarifications are needed to understand better the pros and cons of the transformer-based scheme as compared to the prior work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the problem of learned image compression using a transformer as the entropy model. The authors introduce a diamond-shaped relative position encoding scheme that makes sense for image modeling. They also adopt a two-step, bidirectional context model based on a checkerboard-style spatial decomposition of the latent tensor.

The modeling choices are well-supported by an ablation study. In particular, the authors show that the checkerboard model is almost as good as a much slower spatial context model that is often used for learned image compression. They also show that the transformer-based entropy model ("entroformer") leads to a model with better rate-distortion performance compared to earlier models.
main_review: I have not seen transformers used as entropy models for image compression before this paper (though there appears to be other submissions ICLR 2022 exploring the same idea). Transformers seem like a natural fit for this problem so an empirical evaluation is useful for the community. The diamond-shaped relative position encoding scheme introduced in this paper is also novel (as far as I know) and makes sense given the falloff in spatial correlation typical to natural images.

The use of a checkerboard decomposition for the latent tensor to create a "parallel bidirectional context model" is not novel (the authors appropriately cite He et al. 2021). The authors do provide runtime information (Table 2) showing that "serial context" models can be quite slow, while the parallel approach is much faster (170ms vs. thousands of milliseconds, though simpler models allow for decoding in under 22ms with the trade-off being worse compression rates).

My main concern with this paper is that the empirical evaluation does not show better results compared to the best published models. For example, Figure 5 compares again Qian (2021) but not He (2021), Minnen (2020), or Cheng (2020), all of which are cited elsewhere in the paper. It's hard to compare curves by eye across papers, but it appears that all of these methods have similar rate-distortion (RD) results. Furthermore, other models yield better RD results, for example:

Zongyu Guo, Zhizheng Zhang, Runsen Feng, Zhibo Chen. Causal Contextual Prediction for Learned Image Compression. IEEE Transactions on Circuits and Systems on Video Technology. 2021.

Changyue Ma, Zhao Wang, Ruling Liao, Yan Ye. A Cross Channel Context Model for Latents in Deep Image Compression. 	arXiv:2103.02884. 2021.

It appears (again, I'm eyeballing but you can trace the graph or contact the authors for more accurate numbers) that the Entroformer model hits 33.1 dB at 0.4 bpp, while Guo (2021) is closer to 33.5 dB and Ma (2021) is around 33.3 dB.

I do think that the entroformer model has an advantage since it allows for much faster decoding speeds (both Guo and Ma use a serial context model), but this speed benefit comes from the parallel decoding introduced by He (2021), not by the transformer introduced in this paper.


summary_of_the_review: I think this paper is "marginally above the acceptance threshold" since the use of transformers as entropy models for image compression is novel and interesting. The paper needs a more thorough empirical evaluation to warrant a higher score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a transformer-based entropy modeling order to capture long-range dependencies in probability distribution estimation. This model is optimized for image compression. The authors extend this architecture with a parallel bidirectional context
model to speed up the decoding process.
main_review: Strenghts:
1) First approach using Transformer based method to image compression task.
2)  The proposed method outperforms previous methods based on CNNs.
3) An efficient parallel architecture is proposed and is more time-efficient than the serialized one on modern GPU device.

Weaknesses
Experiments could be more extensive.
summary_of_the_review: The paper present a novel contribution by using first Transformer based method to image compression task. Experiments validate the performance of the proposed approach named Entroformer compared to CNNs based methods.
review_title: null

Meta-review: The three reviewers all felt the paper was above threshold for acceptance to ICLR. To improve the final version, they suggest some additional discussion and experiments may help the paper.
Decision: Accept (Poster)