Title: Towards Efficient Image Compression Without Autoregressive Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Image Compression

Correlation
Abstract: Recently, learned image compression (LIC) has garnered increasing interest with its rapidly improving performance surpassing conventional codecs. A key ingredient of LIC is a hyperprior-based entropy model, where the underlying joint probability of the latent image features is modeled as a product of Gaussian distributions from each latent element. Since latents from the actual images are not spatially independent, autoregressive (AR) context based entropy models were proposed to handle the discrepancy between the assumed distribution and the actual distribution. Though the AR-based models have proven effective, the computational complexity is significantly increased due to the inherent sequential nature of the algorithm. 
In this paper, we present a novel alternative to the AR-based approach that can provide a significantly better trade-off between performance and complexity. To minimize the discrepancy, we introduce a correlation loss that forces the latents to be spatially decorrelated and better fitted to the independent probability model. Our correlation loss is proved to act as a general plug-in for the hyperprior (HP) based learned image compression methods. The performance gain from our correlation loss is ‘free’ in terms of computation complexity for both inference time and decoding time. To our knowledge, our method  gives the best trade-off between the complexity and performance: combined with the Checkerboard-CM, it attains **90%** and when combined with ChARM-CM, it attains **98%** of the AR-based BD-Rate gains yet is around **50 times** and **30 times** faster than AR-based methods respectively

Towards Efficient Image Compression Without Autoregressive Models

Abstract

Recently, learned image compression (LIC) has garnered increasing interest with its rapidly improving performance surpassing conventional codecs. A key ingredient of LIC is a hyperprior-based entropy model, where the underlying joint probability of the latent image features is modeled as a product of Gaussian distributions from each latent element. Since latents from the actual images are not spatially independent, autoregressive (AR) context based entropy models were proposed to handle the discrepancy between the assumed distribution and the actual distribution. Though the AR-based models have proven effective, the computational complexity is significantly increased due to the inherent sequential nature of the algorithm. In this paper, we present a novel alternative to the AR-based approach that can provide a significantly better trade-off between performance and complexity. To minimize the discrepancy, we introduce a correlation loss that forces the latents to be spatially decorrelated and better fitted to the independent probability model. Our correlation loss is proved to act as a general plug-in for the hyperprior (HP) based learned image compression methods. The performance gain from our correlation loss is 'free' in terms of computation complexity for both inference time and decoding time. To our knowledge, our method gives the best trade-off between the complexity and performance: combined with the Checkerboard-CM, it attains 90% and when combined with ChARM-CM, it attains 98% of the AR-based BD-Rate gains yet is around 50 times and 30 times faster than AR-based methods respectively.

1 Introduction

In the digital world, visual media has proven to be a mixed blessing. On the one hand, it provides a mechanism to disseminate a plethora of data, information, and knowledge in a form easily consumable by humans Silk et al. [2021], while on the other, improved quality and increased content is very quickly pushing the boundaries of data storage, communication, and processing technologies Kim et al. [2021]. A software-driven solution to skew the balance of this state has been extensively presented in the form of lossy compression techniques Wallace [1991], Rabbani and Joshi [2002], Ballé et al. [2017]. In the case of images, these techniques aim to reduce the data size while maintaining its context. Traditional image codecs like JPEG Wallace [1991] and JPEG2000 Rabbani and Joshi [2002] use mathematically well-defined transformations such as discrete cosine transform (DCT) and discrete wavelet transform (DWT). These context-less transformations lead to reduced efficiency in terms of maintaining an optimal balance between the image quality and bitrate of the encoded image. In complex and varied images, this problem is especially pronounced, leading to the inclusion of various artifacts such as ringing and blocking Ballé et al. [2017]. In recent years,

deep learning has significantly advanced several computer vision tasks Hu et al. [2022], especially image compression, by incorporating structure and learned behavior. In particular, learned image compression (LIC) has gained popularity as a field of research that has the potential to go beyond traditional compression methods by exploring the inherent nature of images Toderici et al. [2016], Lee et al. [2022], Johnston et al. [2018], Qian et al. [2022], Cheng et al. [2019].

State-of-the-art learned image compression techniques employ transform coding strategies for lossy image compression, which maps image pixels into a quantized latent space Cheng et al. [2020], Zhong et al. [2020]. This map is then losslessly compressed using entropy coding methods such as Huffman or arithmetic coding to create a bitstream for transmission over a channel Shannon [1948]. The entropy coding algorithm requires an entropy model, which is a prior probability model on the quantized latent representation. In this framework, deep neural networks are used for both transform coding and entropy modeling. By learning both modules end-to-end, the ability of deep neural networks is fully utilized to improve the compression performance Yang et al. [2023]. The bitstream size depends on the accuracy of the entropy model, making it essential to design an accurate model for efficient compression Kim et al. [2022]. The objective of the entropy model is to estimate a joint probability distribution over the elements of the quantized latent representation. A simplistic approach is to assume complete independence among the elements, but this approach results in subpar compression performance Ballé et al. [2017, 2018]. Thus, modeling the quantized latent representation interdependencies in learned image compression is a significant challenge Lee et al. [2019], Minnen et al. [2018], Qian et al. [2021], Theis et al. [2017].

Acommon approach is to use convolutional neural networks (CNNs) to extract additional features, known as "hyperprior," which captures the

0

2

4

6

8

10

Inference Time (sec)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

BD-Rate Gains (%)

9.5%

6.13%

3.3%

0.48%

1

30

sec

250

251

252

253

1.97%

1

50

sec

Cheng's Hyperprior (CH)

CH + L

Corr

(Proposed)

CH + Ckbd

CH + Ckbd +  L

Corr

(Proposed)

CH ChARM

CH ChARM  +  L

Corr

(Proposed)

CH + AR Context

context from the quantized latent representation to improve the efficiency of the entropy model Ballé et al. [2018]. The underlying assumption of hyperprior-based entropy models is the probabilistic independence of the latent features, which contradicts the highly correlated nature of latent space, resulting in a discrepancy between the actual distribution and the assumed probability model Kim et al. [2022]. To minimize this discrepancy, Autoregressive context model (AR-CM) based LIC methods have been proposed Minnen et al. [2018], where the probability of each latent is updated based on the previously coded neighboring latents. While this increases the model's overall performance, it introduces sequential dependencies, making it roughly 50 × slower than simple hyperprior-based models He et al. [2021] and hinders its practical implementation of neural image compression in real-world scenarios

To mitigate the above issue, a line of works have attempted to discard AR method for improving efficiency. A notable advancement in this domain is the channel-wise autoregressive entropy model (ChARM) introduced by Minnen et al. Minnen and Singh [2020]. ChARM focuses on reducing the element-level serial processing within the context model presented in Minnen et al. [2018], and it demonstrates a reasonably good RD performance compared to a full autoregressive context model. Li et al. introduced CCN Li et al. [2020], which enables faster context calculation with moderate parallelizability. However, the effectiveness of CCN is still constrained by the image size, limiting its overall efficiency. He et al. He et al. [2021] proposed a parallelizable checkerboard context model

along with a two-pass decoding method, aiming to achieve a better trade-off between rate distortion (RD) performance and computational efficiency. The efficiency of the above methods is increased at a cost of performance drop compared to AR-based models.

To this end, we introduce a correlation loss to minimize the discrepancy without incurring any sequential dependency. The correlation loss forces latents to be decorrelated spatially, resulting in a better fit with the spatially independent probability model. Our findings indicate that decorrelating the latent aids in enhancing the general effectiveness of the hyperprior entropy model. Figure 1 illustrates the effectiveness of the proposed approach. Applying correlation loss to Cheng's hyperprior model yields a comparable performance improvement to that achieved by integrating efficient context modeling approaches. When our approach is integrated with the Checkerboard method, it leads to substantial BD rate improvements of approximately 90%, and when coupled with ChARM, it exhibits BD rate gains of about 98%, akin to the gains observed in a full AR-CM. Notably, our method achieves these remarkable performance gains while requiring only 1/50th and 1/30th of the inference time required by the AR model, respectively. Our key contributions can be summarized as follows:

Our work takes the path of decorrelating latents that have never been tried in the field. The previous approaches primarily focus on CMs to enhance their performance, whereas, in our work, we address the fundamental need for employing computationally heavy CMs. With the introduction of our correlation loss, the correlation among spatially-neighbored elements in the latent space is decreased. This would minimize the discrepancy between the hyperprior entropy model's assumed probability distribution and the actual distribution of the latents, which eventually leads to an overall improvement of the rate-distortion (RD) performance.

Since our method modifies only the loss function, i.e. adding a correlation loss to the original one, it does not require any additional memory or computational complexity beyond the base model. In contrast to the AR-based method which has severe sequential dependency among neighboring latent elements and thus could increase the inference time or decoding time up to 100 × the base model, our method does not create any sequential dependency and thus can maintain the inference or decoding time of the base model.

Our proposed correlation loss acts as a plug-in for the existing learned image compression methods and can achieve BD rate gains of up to 9.5% by combining with the base hyperpriorbased models. Remarkably, the gains are comparable to the performance improvements achieved by combining the hyperprior module with efficient context-based LIC models, like the SOTA efficiency-oriented Checkerboard's context model (CM). When combined with Checkerboard method, our proposed approach acheives 90% and when combined with ChARM our proposed approach acheives 98% of the AR-CM's BD-Rate gains while incurring only 1/50th and 1/30th of the inference time of the AR-CM. Our method achieves the best trade-off between the complexity and performance.

2 Related Work

The autoregressive context based entropy model, inspired by the concept of context used in traditional codecs, is used to predict the probability of unknown codes based on latents that have already been decoded Bellard [2015]. These models use hyper latent and context to predict both the location (mean value) and scale parameters (standard deviation) of the entropy model Minnen et al. [2018]. By combining elements of differentiable entropy modeling, hyper latent, and context models, it is possible to outperform BPG Bellard [2015] in terms of PSNR and MS-SSIM Minnen et al. [2018], He et al. [2021], Qian et al. [2022]. These context models have also been extended to become more powerful but also more computationally expensive Cheng et al. [2020]. In particular, AR context-based models, essential for achieving the state of the art performance, have low computational efficiency due to the sequential dependency of decoding the latents. As AR context models are sequentially dependent, they cannot be deployed on edge devices or take advantage of parallel hardware processing. As a result, models focusing on the real-world deployment of learned image compression often exclude AR context models due to their inefficiency and high inference costs He et al. [2021].

Parallel decoding approaches like Checkerboard He et al. [2021] and ChARM Minnen and Singh [2020] offer a better performance tradeoff over computation complexity. Checkerboard introduces a parallel decoding approach by encoding and decoding only half of the latent variables using a checkerboard-shaped context and hyperprior. The remaining latents, referred to as anchors, are coded solely based on the hyperprior. On the other hand, ChARM utilizes channel conditioning instead of

𝒚

Analysis

Transform

Hyper Analysis

Transform

Hyper

Synthesis

Transform

Synthesis

Transform

Mean-Scale

Entropy

Model

Factorized

Entropy

Model

𝒚 "

Q

𝒛 $

Q

𝝁, 𝝈

𝐿

!

𝐿

"

𝐿

#

𝒙

𝒙 "

𝒚

Analysis

Transform

Synthesis

Transform

𝒚 "

Q

𝐿

!

𝐿

"

𝒙

𝒙 "

Hyper

Analysis

Transform

Hyper

Synthesis

Transform

Factorized

Entropy

Model

𝒛 &

Q

𝝁

#

, 𝝈

#

𝐿

$

CM

𝝁

%

, 𝝈

%

EP

Mean-Scale

Entropy

Model

𝝁, 𝝈

element-wise conditioning, along with latent residual prediction, to enhance the rate-distortion (RD) performance. The recent efficient SOTA methods are all built on top of the ChARM and checkerboard context model as an alternative to a fully AR-based context model Zhu et al. [2022], Qian et al. [2022], Kim et al. [2022].

Recent research in the field of entropy coding has primarily concentrated on improving the efficiency of context-based methods. These methods aim to exploit the interdependencies within the latent space more effectively, utilizing the inherent dependencies present Lei et al. [2022], Lee et al. [2019]. However, there has been a lack of attention towards enhancing the effectiveness of the hyperprior module, an integral component of prominent learned image compression (LIC) techniques. Notably, recent studies utilizing Transformer Vaswani et al. [2017] models have demonstrated that achieving a higher level of decorrelation in the latent space leads to improved rate-distortion (RD) performance Zhu et al. [2022]. This improvement can be attributed to the reduction in the discrepancy between the actual distribution of the latent space and the assumed probability model Kim et al. [2022]. Our work aims to address this research gap by introducing a novel approach that builds upon the existing findings. Specifically, we incorporate correlation loss during the training phase of our model, which promotes better decorrelation among the latent variables. By doing so, we enhance the performance of the hyperprior module, thereby improving the overall efficacy of LIC models.

3 Methodology

3.1 Formulation of Learned Image Compression Models

Image compression can be formulated through the transform coding approach by (as Figure 2 and 3)

where x and ˆ x are input and reconstructed image, y is a latent representation, and ˆ y is the quantized latent representation. ϕ and θ are the parameters of analysis and synthesis transforms g a and g s , respectively, Q represents the quantization. During training, the quantization is approximated using uniform noise. While for inference, a round function is applied for quantization to generate ˆ y , and then entropy encoded to generate the bitstream. Given a probability model for the quantized representation, entropy encoding techniques such as arithmetic coding Rissanen and Jr. [1981] can losslessly compress the quantized codes. Arithmetic coders are optimal entropy coders which can be used to estimate the entropy of ˆ y during training.

A prior work Ballé et al. [2017] has employed a non-adaptive density model which was shared between the encoder and decoder also called factorized prior. It proposed a hyperprior in the followup work as a mechanism to introduce side information z to capture the spatial dependencies in the latent space y Ballé et al. [2018]. The side information predicted the mean and scale information for the latent space as:

Quantized Latent (

𝑦 "

)

Mean (

µ)

Scale (

σ)

Normalized (

! " #

µ

σ

)

Cheng's Hyperprior (HP)

𝐿

$%&&

bpp_y = 0.059

bpp_z (

µ

+

σ

): 0.003

bpp (y + z): 0.062

PSNR: 24.37

bpp_y = 0.057

(Δ=0.002↓)

bpp_z (

µ

+

σ

): 0.004

(Δ=0.001↑)

bpp (y + z): 0.061

(Δ=0.001↓)

PSNR: 24.75

Avg Correlation: 0.0454

Avg Correlation: 0.0107

(Δ=0.0347↓)

Correlation Map

L

stto

9.100

14

0.50

RES

5.075

.25

20

L

17

0.125

0.100

051

L9G

8.04

3.929

0.010

0011

1231

sroo-

54'9-

where h a and h s denote the analysis and synthesis transform for the hyperprior auto-encoder. p ˆ y | ˆ z (ˆ y | ˆ z ) is the estimated probability distribution for the quantized latent space ˆ y , given by:

where the mean µ i and scale σ i parameter is the i th element of h s (ˆ z ) for each spatial location i in the latent space ˆ y . With the premise that individual latent components are mutually independent, each spatial element in latent ˆ y is modeled as a Gaussian with its own mean and standard deviation. We may therefore estimate entropy using eq. 4. We modeled the probability of the hyper-latent ˆ z using a non-parametric fully factorized density model as:

where z i denotes the i th element of z , and i specifies the position of each element, and the vectors ψ ( i ) encapsulate the parameters of each univariate distribution ( p z i | ψ ( ψ )) .

A context model can be added to boost the RD performance in the mean-scale hyperprior framework. Additionally, a parameter inference network ( g ep ) is employed to estimate the mean and scale parameters Φ = ( µ, σ ) by leveraging the outputs of the hyperprior module h s (ˆ z ) and the context model g cm (ˆ y <i ) :

where ˆ y <i means the causal context of the nearby visible latents to the latent ˆ y i . The context model can be calculated as:

where ⊙ refers to the Hadamard product, b is a bias term. The variable M represents a mask commonly used to establish a top-left reference, necessitating strict Z-ordered serial decoding.

The Checkerboard context model He et al. [2021] introduces a parallel decoding approach where only half of the latent variables are encoded/decoded using a checkerboard-shaped context and hyperprior as shown in the Figure 3. The coding of the remaining half of the latents, referred to as anchors, relies solely on the hyperprior. To implement this approach, the eq. 6 above is modified as:

The masked convolution, denoted as g cm , is applied conditioned on a checkerboard-shaped mask M , as described in eq. 7. The input for g cm , denoted as ˆ y anchor , corresponds to the set of anchors, and i represents the index for the i-th element ˆ y i in the latent variable ˆ y . For methods that utilize

𝒚

…

𝑦

!

𝑦

"

𝝁

…

𝜇

!

𝜇

"

𝝈

…

𝜎

!

𝜎

"

Point Mask

Short Cross Mask

Long Cross Mask

mean-scale Gaussian entropy models, the entropy parameter Φ is represented as ( µ, σ ) , where µ and σ correspond to the mean and standard deviation, respectively. In the presence of visible anchors, the context features of all non-anchors can be calculated simultaneously using masked convolution in parallel. The decoding of anchors also occurs in parallel. As a result, the calculation of entropy parameters in eq. 8 for decoding can be performed in two passes, which significantly improves efficiency compared to a serial context model.

LIC is a langrangian multiplier based rate-distortion optmization problem. The loss function is then defined as

where λ controls the rate-distortion tradeoff and D ( x, ˆ x ) is the distortion term usually using mean square error.

3.2 Our Proposed Correlation Loss

As shown in the eq. 4, the joint probability of ˆ y in the hyperprior-based entropy is calculated based on the assumption that the latent features are mutually independent. However, as shown in the upper portion of last column of Figure 4, the latent space is highly correlated, resulting in a discrepancy between the actual and the calculated probability distribution. To address this discrepancy, we introduce correlation loss to force the latent to be decorrelated spatially. We can therefore model the correlation loss by calculating the correlation in the latent space. We start by computing the correlation map as:

where k × k is the window size of the correlation map, µ and σ refer to the hyper-latent's corresponding i th and m th elements, m refers to the central point of the window of size k × k as shown in Figure 5. We slide the window with the stride of 1 across the entire latent space to calculate the correlation of the central point m with all the other points in the window of size k × k . We then take the mean of all the individual correlation maps to get the resultant correlation map of the latent y of size k × k .

We then apply the Mask on the correlation map. As shown in Figure 6, the point mask only masks the central location of latent space as it corresponds to self-correlation of 1 (when i = m ). We propose several mask designs (discussed in supplementary material) to investigate the effect of the spatially relaxed decorrelation constraint. Mask can then be applied as:

The correlation loss is formulated by applying the L 2 norm on the Masked _ Map k × k .

By incorporating the correlation loss L corr , the RD loss can now be given as:

where α is the scaling factor for the correlation loss.

Given that the entropy estimate presumes independent latent elements, our proposed loss function increases compression efficiency by reducing the correlation between latent features, hence reducing the discrepancy between the assumed probability distribution of the entropy model and the actual latent probability distribution.

0.1

0.2

0.3

0.4

0.5

0.6

BPP

26

28

30

32

34

PSNR

Minnen's Hyperprior + Correlation Loss

Minnen's Hyperprior

0.1

0.2

0.3

0.4

0.5

0.6

BPP

24

26

28

30

32

34

PSNR

Cheng's Hyperprior + Correlation Loss

Cheng's Hyperprior

0.1

0.2

0.3

0.4

0.5

BPP

24

26

28

30

32

34

PSNR

SwinT with Correlation Loss

SwinT with Hyperprior

4 Experiments and Results

4.1 Experimental Setup

We train all our models on the Vimeo-90k dataset Xue et al. [2019], with the images cropped to the resolution of 256 × 256 for training. The models were optimized using the Adam optimizer Kingma and Ba [2015] with a batch size of 16 and trained for 1.5 million iterations with a learning rate of 1 × 10 -4 for the first million iterations and then halved every 50,000 iterations till 1.25 million iterations.

RD loss given in eq. 9 is used to train the baseline models, whereas the modified loss in eq. 13 is used to train models with correlation loss. The rate-distortion tradeoff is guided by λ , whose value is contained in the set [0.0009, 0.0018, 0.0035, 0.0067, 0.0130, 0.0250].

We tested our model on a commonly used Kodak lossless images dataset Kodak [1993], with 24 uncompressed images of 768 × 512 or 512 × 768 resolution. Bits per pixel (bpp) provide the rate, while PSNR indicates the quality of the reconstructed image when evaluating rate-distortion performance. In order to demonstrate the coding efficiency, RD curves are drawn.

4.2 Rate-Distortion Performance

We employ Minnen's Minnen et al. [2018], Cheng's Cheng et al. [2020], and SwinT Liu et al. [2021] based mean scale hyperprior models as well as Cheng's with Checkerboard He et al. [2021] for our experiments. We perform all our experiments on the Pytorch framework Paszke et al. [2017] and use the CompressAI library Bégaint et al. [2020]. Minnen's and Cheng's models were trained using an NVIDIA 2080Ti, whereas the SwinT model was trained on an NVIDIA 3070Ti due to the transformers' high memory requirement. Note that including correlation loss has a negligible effect on the hardware requirement of these base models.

Minnens' Hyperior: Figure 7a depicts the RD curve for Minnen's model Minnen et al. [2018] with and without correlation loss; for Minnen's model, correlation loss achieves BD rate gains of 3.20%.

Cheng's Hyperprior: Figure 7b illustrates the RD curve for Cheng's model Cheng et al. [2020] with and without correlation loss. Cheng's model yields a BD rate gain of 9.5% with correlation loss. Cheng's model consists of attention modules along with CNN layers that significantly increase the overall efficiency of the correlation loss. Figure 9 depicts the spatial correlation of latent y averaged across all latent channels and compares the baseline Cheng's model with the one trained on correlation loss. Figure 9 demonstrates that the model trained with our proposed loss function exhibits significantly less correlation than the baseline model. This observation holds for the models trained with other λ values. The qualitative comparison in Figure 8 between Cheng's model with and without correlation loss demonstrates the superior performance achieved by incorporating correlation loss. It implies that models trained on correlation loss incur less redundancy across different spatial latent locations, resulting in a better rate-distortion trade-off overall.

Transformer Hyperprior: Wealso compared the effectiveness of correlation loss on the Transformerbased LIC model. We implemented our SwinT with a hyperprior model inspired by Zhu's work and employed a similar architectural setting Zhu et al. [2022]. Transformers tend to have a lesser correlation in their latent space than the convolutional-based LIC models Zhu et al. [2022]. The

0.1

0.2

0.3

0.4

0.5

BPP

24

26

28

30

32

34

PSNR

Cheng's with Checkerboard + Correlation Loss

Cheng's with Checkerboard

Cheng's Hyperprior

(

λ

=0.0067)

Kodak: PSNR=30.58dB, BPP=0.27

Cheng's with Correlation Loss

(

λ

=0.0067)

Kodak: PSNR=30.84dB, BPP=0.26

0

3

5

6

0

5

6

0.040

0

0.008

0.010

0.013

0.014

0.011

0.010

0.008

F 0

0.001

0.004

0.006

0.007

0.002

0.003

0.001

S000

1

0.011

0.012

0.014

0.020

0.015

0.010

0.011

1 -

0.004

0.006

0.007

0.014

0.007

0.003

0.003

0000

2 -

0.013

0.015

0.015

0.032

0.018

0.014

0.011

2

0.005

0.007

0.008

0.028

0.014

0.007

0.004

0.025

Correlation Map

3

0.019

0.028

0.037

0.037

0.027

0.020

3 -

0.014

0.023

0.032

0.032

0.023

0.013

0.020

4 -

0.011

0.014

0.017

0.032

0.015

0.015

0.013

4 -

0.004

0.008

0.014

0.028

8000

0.007

0.005

-0.015

5 -

0.010

0.010

0.014

0.021

0.013

0.012

0.011

5 -

0.003

0.003

0.007

0.013

0.006

0.006

0.004

-0.010

6 -

0.009

0.011

0.012

0.015

0.014

0.011

0.009

F 9

0.002

0.004

0.003

0.008

0.006

0.004

0.002

- 0.005

0.000

0.1

0.2

0.3

0.4

0.5

BPP

26

28

30

32

34

PSNR

Cheng's with ChARM

Cheng's with ChARM + Correlation Loss

Figure 7c illustrates the performance of the SwinT model with and without correlation loss. We found that with the correlation loss, the SwinT model achieves a BD-rate gain of about 4.8%. The effectiveness of correlation loss for the SwinT model further indicates the efficacy of the correlation loss, which in turn improves the mean scale based entropy model.

Cheng's with Checkerboard Context Model: Figure 10 illustrates the RD curve for Cheng's model with Checkerboard He et al. [2021] with and without correlation loss. Cheng's with Checkerboard yields a BD rate gain of 6.13% with correlation loss. Checkerboard uses a parallel decoding approach by encoding and decoding only half of the latent variables using a checkerboard-shaped context. The remaining latents, referred to as anchors, are coded solely based on the hyperprior. Since half of the latents are encoded by the hyperprior module. Incorporating correlation loss into the Checkerboard context model significantly improves the performance of the hyperprior module, resulting in an overall enhancement of the model's performance.

Cheng's with ChARM: Figure 11 illustrates the RD curve for Cheng's model with ChARM Minnen and Singh [2020], both with and without the integration of correlation loss. Notably, when correlation loss is incorporated into Cheng's with ChARM, there is a substantial BD rate gain of 3.3%. ChARM effectively employs channel context and latent residual prediction for the encoding and decoding of the latents. The combination of correlation loss with ChARM leads to a significant enhancement in its overall performance, marking a performance difference of approximately 0.5% when compared to a full AR model.

4.3 Complexity Tradeoff

The primary advantage of the correlation loss lies in its capacity to minimize the discrepancy between the actual and presumed probability distribution within the entropy model. Figure 1 illustrates that for

0.1

0.2

0.3

0.4

0.5

0.6

BPP

24

26

28

30

32

34

PSNR

Cheng's Hyperprior with Correlation Loss

Cheng's Hyperprior with AR Context

Cheng's with Checkerboard + Correlation Loss

Cheng's with ChARM + Correlation Loss

Cheng's Hyperprior

0.1

0.2

0.3

0.4

0.5

0.6

0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Correlation

+1.22

+0.66

+0.21

+0.14

+0.07

Cheng's Hyperprior (CH)

PSNR Gain

0.1

0.2

0.3

0.4

0.5

0.2

0.1

0.0

0.1

0.2

0.3

0.4

0.5

Correlation

+1.24

+0.5

+0.12

+0.01

+0

+0.05

Cheng's HP + CKBD

PSNR Gain

0.1

0.2

0.3

0.4

0.5

0.10

0.05

0.00

0.05

0.10

0.15

Correlation

+0.37

+0.48

+0.02

+0.02

+0.01

SwinT HP

PSNR Gain

0.1

0.2

0.3

0.4

0.5

0.6

0.10

0.05

0.00

0.05

0.10

0.15

0.20

0.25

Correlation

+0.35

+0.12

+0.1

+0.08

+0.07

+0.07

Minnen's HP

PSNR Gain

a given encoder and decoder architecture, which is Cheng's model in this case, the choice of entropy model determines the performance of the resulting model. Note that Cheng's hyperprior signifies the lower bound with the given encoder-decoder architecture, whereas Cheng's AR defines the upper limit with the same encoder-decoder architecture. Using Cheng's hyperprior model as the baseline, we compared the effectiveness of including an AR module vs. the correlation loss. The Figure 12 shows the RD curve comparison, while the Table 1 compares the inference time for processing the entire Kodak test dataset. Incorporating correlation loss into Cheng's hyperprior model results in a performance improvement roughly half that of the AR-context-based model, all while reducing inference costs to just 1/55th of the original. Similarly, when utilizing Cheng's Checkerboard with correlation loss, a substantial performance enhancement of approximately 90% is achieved compared to the AR-context-based model, with an inference cost as low as 1/50th of the latter. Additionally, Cheng's ChARM with correlation loss attains an impressive performance gain of about 98% when compared to the AR-context-based model, while incurring an inference cost of merely 1/30th of the AR-context-based model as shown in Figure 1.

It should be noted that when correlation loss is applied to Cheng's hyperprior, significant improvements can be achieved due to the large performance gap between Cheng's hyperprior baseline and the upper limit. On the other hand, since the gaps are diminished in models like Cheng's with Checkerboard and Cheng's with ChARM, the potential for improvement by applying correlation loss is also limited when compared to Cheng's hyperprior baseline. In summary, the introduction of correlation loss to these models would result in a closer approximation to the full AR performance, albeit with gains that are comparatively smaller.

5 Analysis

5.1 Visual Illustration of Effectiveness of the Correlation Loss

Figure 4 visualizes some of the internal workings of our models. We compare Cheng's hyperprior model with and without correlation loss. By observing the first column of Figure 4, it is clear that our correlation loss makes the latent feature preserve more detailed texture information yielding higher PSNR compared with the base model. The mean µ can also extract more distinctive and meaningful information from the latent space. Since the hyperpriors (mean µ and scale σ ) contain more information, they require slightly more bits than the base model. Nevertheless, the rise in hyperprior rate is adequately offset by the relatively greater decrease in latent bitrate, as demonstrated in the final column of Figure 4, where the normalized latents exhibit improved decorrelation compared to the base model. Overall, our proposed method surpasses the base model by enhancing PSNR through more decorrelated latents while either maintaining or reducing bpp with better decorrelated latents.

5.2 Effect of Correlation score on the Model's performance

Our proposed loss function consistently demonstrates significant BD-rate gains at lower bit rates while maintaining comparable performance at high bit rates. This behavior is primarily attributed to learned image compression (LIC) models exhibiting higher correlation among latent variables at lower bpp values compared to higher bpp values as reported by Zhu et al. [2022] and also evident from the correlation maps in Figure 6 of the supplementary material. We conducted a comprehensive analysis and presented the findings in Figure 13. Figure 13 showcases the relationship between PSNR gains, bpp, and correlation for different models, including Cheng's hyperprior , Cheng's with Checkerboard, SwinT hyperprior, and Minnen's hyperprior. From the graphs in the Figure 13, a clear pattern can be observed: as the bpp decreases, the correlation of the latents increases, resulting in a higher gain in PSNR. However, as we move towards higher bpps, the correlation becomes notably reduced, resulting in decreased PSNR gain. This trend explains the reason why the efficacy of the correlation loss is larger in low bpp range and diminish at high bpp range.

6 Ablation Studies

Comprehensive ablation studies regarding various mask types, mask sizes, and α values are presented in the supplementary material.

7 Conclusion

The hyperprior-based entropy models assume probabilistic independence, which leads to a discrepancy between the actual and the assumed probability distribution. We proposed correlation loss, which reduces the correlation among spatially neighbored elements in the latent space resulting in a better fit with the spatially independent probability model. Our proposed loss function does not require any model structure or capacity changes and acts as a plug-in method for the existing neural compression methods. The proposed approach, without modifying the entropy models and increasing the inference time, significantly improves the RD performance with BD rate gains up to 17.99%.

Acknowledgments and Disclosure of Funding

Authors are thankful to Muhammad Awais for his constructive and helpful feedback. Authors are also thankful to Taegoo Kang for managing and providing the required compute resources. This work was supported in part by the Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korea Government [Ministry of Science and Information Communication Technology (MSIT)] (Development of Audio/Video Coding and Light Field Media Fundamental Technologies for Ultra Realistic Tera-media) under Grant 2017-0-00072; in part by the IITP grant funded by the Korea Government (MSIT) (Artificial Intelligence Convergence Innovation Human Resources Development, Kyung Hee University) under Grant RS-2022-00155911; in part by the IITP grant funded by the Korea Government (MSIT) (Artificial Intelligence Innovation Hub) under Grant 2021-0-02068.

References

Johannes Ballé, Valero Laparra, and Eero P. Simoncelli. End-to-end optimized image compression. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 2426, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openreview. net/forum?id=rJxdQ3jeg .

Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018. URL https://openreview.net/forum?id=rkcQFMZRb .

Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja. Compressai: a pytorch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:2011.03029 , 2020.

Fabrice Bellard. Bpg image format. URL https://bellard. org/bpg , 2015.

Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Deep residual learning for image compression. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019, Long Beach, CA, USA, June 16-20, 2019 , page 0. Computer Vision Foundation / IEEE, 2019. URL http://openaccess.thecvf.com/ content_CVPRW_2019/html/CLIC_2019/Cheng_Deep_Residual_Learning_ for_Image_Compression_CVPRW_2019_paper.html .

Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020 , pages 7936-7945. Computer Vision Foundation / IEEE, 2020. doi: 10.1109/CVPR42600.2020.00796. URL https://openaccess.thecvf.com/content_CVPR_2020/html/Cheng_ Learned_Image_Compression_With_Discretized_Gaussian_Mixture_ Likelihoods_and_Attention_CVPR_2020_paper.html .

Dailan He, Yaoyan Zheng, Baocheng Sun, Yan Wang, and Hongwei Qin. Checkerboard context model for efficient learned image compression. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021 , pages 14771-14780. Computer Vision Foundation / IEEE, 2021. doi: 10.1109/CVPR46437.2021.01453. URL https://openaccess. thecvf.com/content/CVPR2021/html/He_Checkerboard_Context_Model_ for_Efficient_Learned_Image_Compression_CVPR_2021_paper.html .

Yueyu Hu, Wenhan Yang, Zhan Ma, and Jiaying Liu. Learning end-to-end lossy image compression: A benchmark. IEEE Trans. Pattern Anal. Mach. Intell. , 44(8):4194-4211, 2022. doi: 10.1109/ TPAMI.2021.3065339. URL https://doi.org/10.1109/TPAMI.2021.3065339 .

Nick Johnston, Damien Vincent, David Minnen, Michele Covell, Saurabh Singh, Troy Chinen, Sung Jin Hwang, Joel Shor, and George Toderici. Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4385-4393, 2018.

Jun-Hyuk Kim, Byeongho Heo, and Jong-Seok Lee. Joint global and local hierarchical priors for learned image compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5992-6001, 2022.

Youmin Kim, Jinbae Park, Younho Jang, Muhammad Salman Ali, Tae-Hyun Oh, and Sung-Ho Bae. Distilling global and local logits with densely connected relations. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pages 6270-6280. IEEE, 2021. doi: 10.1109/ICCV48922.2021.00623. URL https: //doi.org/10.1109/ICCV48922.2021.00623 .

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1412.6980 .

Eastman Kodak. Kodak lossless true color image suite (photocd pcd0992), 1993. URL http: //r0k.us/graphics/kodak/ .

Jae-Han Lee, Seungmin Jeon, Kwang Pyo Choi, Youngo Park, and Chang-Su Kim. Dpict: Deep progressive image compression using trit-planes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16113-16122, 2022.

Jooyoung Lee, Seunghyun Cho, and Seung-Kwon Beack. Context-adaptive entropy model for end-toend optimized image compression. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https: //openreview.net/forum?id=HyxKIiAqYQ .

Jianjun Lei, Xiangrui Liu, Bo Peng, Dengchao Jin, Wanqing Li, and Jingxiao Gu. Deep stereo image compression via bi-directional coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19669-19678, 2022.

Mu Li, Kede Ma, Jane You, David Zhang, and Wangmeng Zuo. Efficient and effective contextbased convolutional entropy modeling for image compression. IEEE Trans. Image Process. , 29: 5900-5911, 2020. doi: 10.1109/TIP.2020.2985225. URL https://doi.org/10.1109/ TIP.2020.2985225 .

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pages 9992-10002. IEEE, 2021. doi: 10.1109/ICCV48922.2021.00986. URL https://doi.org/10.1109/ICCV48922.2021.00986 .

David Minnen and Saurabh Singh. Channel-wise autoregressive entropy models for learned image compression. In 2020 IEEE International Conference on Image Processing (ICIP) , pages 33393343. IEEE, 2020.

David Minnen, Johannes Ballé, and George Toderici. Joint autoregressive and hierarchical priors for learned image compression. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 10794-10803, 2018. URL https://proceedings.neurips.cc/paper/2018/hash/ 53edebc543333dfbf7c5933af792c9c4-Abstract.html .

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.

Yichen Qian, Zhiyu Tan, Xiuyu Sun, Ming Lin, Dongyang Li, Zhenhong Sun, Hao Li, and Rong Jin. Learning accurate entropy model with global reference for image compression. International Conference on Learning Representations (ICLR) , 2021.

Yichen Qian, Ming Lin, Xiuyu Sun, Zhiyu Tan, and Rong Jin. Entroformer: A transformer-based entropy model for learned image compression. International Conference on Learning Representations (ICLR) , 2022.

Majid Rabbani and Rajan Joshi. An overview of the jpeg 2000 still image compression standard. Signal processing: Image communication , 17(1):3-48, 2002.

Jorma Rissanen and Glen G. Langdon Jr. Universal modeling and coding. IEEE Trans. Inf. Theory , 27(1):12-22, 1981. doi: 10.1109/TIT.1981.1056282. URL https://doi.org/10.1109/ TIT.1981.1056282 .

Claude E. Shannon. A mathematical theory of communication. Bell Syst. Tech. J. , 27(3):379-423, 1948. doi: 10.1002/J.1538-7305.1948.TB01338.X. URL https://doi.org/10.1002/j. 1538-7305.1948.tb01338.x .

Matthew Silk, Ricardo Correia, Diogo Veríssimo, Audrey Verma, and Sarah L Crowley. The implications of digital visual media for human-nature relationships, 2021.

Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compression with compressive autoencoders. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openreview.net/forum?id=rJiNwv9gg .

George Toderici, Sean M O'Malley, Sung Jin Hwang, Damien Vincent, David Minnen, Shumeet Baluja, Michele Covell, and Rahul Sukthankar. Variable rate image compression with recurrent neural networks. International Conference on Learning Representations (ICLR) , 2016.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Gregory K Wallace. The jpeg still picture compression standard. Communications of the ACM , 34(4): 30-44, 1991.

Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T. Freeman. Video enhancement with task-oriented flow. Int. J. Comput. Vis. , 127(8):1106-1125, 2019. doi: 10.1007/ S11263-018-01144-2. URL https://doi.org/10.1007/s11263-018-01144-2 .

Yibo Yang, Stephan Mandt, Lucas Theis, et al. An introduction to neural data compression. Foundations and Trends® in Computer Graphics and Vision , 15(2):113-200, 2023.

Zhisheng Zhong, Hiroaki Akutsu, and Kiyoharu Aizawa. Channel-level variable quantization network for deep image compression. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20 , pages 467-473. International Joint Conferences on Artificial Intelligence Organization, 7 2020. doi: 10.24963/ijcai.2020/65. URL https://doi.org/10.24963/ijcai.2020/65 . Main track.

Yinhao Zhu, Yang Yang, and Taco Cohen. Transformer-based transform coding. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id=IDwN6xjHnK8 .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims at providing a efficient and effective entropy model to achieve better trade-off between performance and complexity for learned image compression. They introduce a correlation loss to force the latent to be spatially decorrelated so that it can fit the independent probability model better.\n\nSTRENGTHS:\n1. While most existing methods primarily focus on the context model, this paper makes a try at decorrelating latents with the proposed correlation loss. The correlation loss can act as a general plug-in for the hyperprior-based methods, which is flexible.\n\n2. It is proved that when the proposed method is combined with the checkerboard method,  the performance gain is about 85% compared with the auto-regressive model with only 1/50th inference time, which is efficient and effective.\n\nWEAKNESSES:\n1.  As shown in the RD curves, the BD-rate gains are obvious in the low bit-rate regime, however, there are less or even no gains in the high bit-rate regime. I think these results require further analysis, such as why it works better in the low bit-rate regime. It is one of the core problems that should be answered to help readers better understand the method. \n\n2.  It is just a little problem. There are many references of the figures and equations in the paper, but they are somewhat orderless. It is better to put the figures not so far away from their references.\n\n3.  I am somewhat confused about the analysis about Figure 4. It is better to provide the input image, which can help readers know the correspondence between the visualization and the original input.\n\n4.  I am not sure about the claim in line 307 \"The fourth column of Figure 4 shows that for our method, latent space has significantly reduced correlation compared to the baseline indicating the correlation loss’s efficacy\". I can only see that compared with Cheng's Hyperprior, the normalized latent of the proposed method has lower energy. I do not think it can provide evidence to show the proposed approach will reduce the correlation. Maybe it is because the proposed method just gives more accurate mean and variance. I think you can also provide the visualization of using auto-regressive context model but without the proposed approach. It may provide similar visualization results.\n\nQUESTIONS:\nNo\n\nLIMITATIONS:\nsee the weakness part", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-30"}

comments: SUMMARY:
This paper proposed a correlation loss to decrease the correlation among spatially-neighbored elements in the latent space. By only modifying the loss function, this method acts as a plug-in method for the existing neural compression methods with no complexity increasing. Experiments show improvement in the compression performance to several baseline models.

STRENGTHS:
The main contribution of the paper is the correlation loss proposed to decrease the correlation among spatially in latent features. 

Experiments show improvement in the compression performance to several baseline models which provide some insights about designing better neural image compression networks from the prospective of feature map decorrelation.

WEAKNESSES:
1) For SwinT and Cheng with checkerboard, the correlation loss seems to only work for lower bitrates, the author may provide some explanation. Besides, The current tested bitrate range is relatively low. RD performance at higher bitrates (>1bpp) should also be provided. 

2) The paper writing can be improved. The citation format in the article looks messy. For equations, Some are ‘eq’, but some are ‘Equation.’

3) Experiments on factorized Ball\'e method [1] and HP+AR+correlation loss can be provided to makes the experiments and evaluation more complete.

[1] Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compression.

QUESTIONS:
Minor issues:

1) Some experiments in the supplementary materials are kind of important and can be put into the main paper.

2) Some citations are still preprint version (e.g. [1][2][3] in reference). Their officially published version should be cited.

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new loss to improve neural image compression models. The authors identify a potential issue with typical neural image compression models where the hyperprior, which predicts entropy parameters over a quantized latent representation, assumes conditional independence between latents, but that may not be true in practice. Previous methods improved the entropy model through context modeling (e.g., using a spatially autoregressive model or a \"checkerboard\" decomposition) but this leads to higher compute and slower runtimes.\n\nInstead, the authors add a local correlation loss averaged spatially over the latents. This loss directly encourages the encoder to generate decorrelated latents, which is a better match for the conditional independence assumption built in to the hyperprior. In particular, the loss affects training (and isn't very expensive) and adds no additional computation at inference time, which leads to significant runtime improvements.\n\nThe paper shows that the new correlation loss improves rate-distortion (RD) performance when applied to several popular models (see Fig. 1 and Fig. 7).\n\nSTRENGTHS:\nThe primary strength of the paper is the simplicity and effectiveness of the main idea.\n\nMost papers on neural compression boost RD performance by making the model more complex, which typically leads to slower decode times. The authors discuss this problem and present a well-motivated loss that leads to significant RD gains without affecting runtime. Their correlation loss is also quite general and can be applied to many different compression models.\n\nAs far as I know, the correlation loss has not been proposed elsewhere in the neural compression literature. There are some similarities to a diversity loss in VQ or clustering, though I don't have a specific reference for this. So I think the originality, especially for the neural compression subfield, is high.\n\nThe quality and clarity of the writing is also high.\n\nWEAKNESSES:\nIdeally, the correlation loss presented in the paper would be applied to a SOTA comperssion model leading to a new SOTA. For instance, the paper cites (He 2021), which introduced the checkerboard decomposition for entropy modeling, but they don't build on top of (He 2022), which presents a more powerful model that combines the checkerboard with CHARM.\n\nELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding\nDailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, Yan Wang\nhttps://arxiv.org/abs/2203.10886\n\nFigure 1 implies that the benefit of the correlation loss shrinks as models become more powerful so it may be that the benefit for ELIC is negligible?\n\nThe visualization in Fig. 4 is great (and is common for compression papers) but it's not obvious to me what I should be looking at to see the impact of the correlation loss. Is it a sharper scale image? The goal is an i.i.d. Gaussian normalized image (far right column) but it's not visually obvious to me that \"our approach\" is closer to i.i.d. Gaussian than the baseline.\n\nMask patterns are mentioned and shown in Fig. 6 but results are only in the supplemental material. Presumably that's because the mask shape did not have a large impact. That's fine, but maybe cut Fig. 6 or at least add a sentence to the main paper summarizing the findings.\n\nQUESTIONS:\nAnalysis exploring why the correlation loss boosts performance would strengthen the paper. Specifically, why is it needed when the existing rate-distortion loss is minimized by decorrelated latents? The generic answer is \"the network is stuck in a (bad) local minimum, and the correlation loss changes the loss landscape such that the optimizer doesn't get stuck\".\n\nLIMITATIONS:\nadequately addressed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focus on efficient learned image compression. Different from existing method, which generally aims to parallelize the autoregressive operations, this paper propose to speed up the framework by removing the whole autoregressive model by introducing the correlation loss, aiming to decorrelate the latent features. The introduced correlation loss can act as a plug-in for the existing learned image compression methods to achieve superior RD performance and at the same time reduce the inference time without autoregressive entropy model.\n\nSTRENGTHS:\n1. The paper is well-motivated and easy to follow.\n2. The introduced correction loss is statistically sound and empirically proved to be effective. The proposed loss is novel to me and can act a plug-in for the existing learned image compression to achieve consistent improvement, especially at low-bit ranges.\n3. By removing the correlation, the propose method can ease the requirement of autoregressive entropy model, so as to speed up the whole compression framework.\n\nWEAKNESSES:\n1. The citation in Figure 3 is wrong.\n2. The caption of Figure 4 states that the correlation loss can provide more flexible parameterized distribution models with significant spatial redundancy reduction. However, from my perspective, the plots show in Figure 4 cannot showcase both \"more flexible parameterized distribution models\" and \"significant\" spatial redundancy reduction. I hope the author can have more comments on this, otherwise such claims are unsupported or inaccurate.\n3. The introduction of section 3.1 should be reduced, which are general, well-known concepts in LIC, the main focus with more words should be put into the contribution introduced section 3.2.\n4. The paper should also report the performance of the proposed method on top of some more recent efficient LIC methods like [1].\n\n[1] Dailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang. ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding. CVPR 2022\n\nQUESTIONS:\n1. Is the hyperprior shown in Figure 4 a GMM-based (K=3) or a simple mean-scale hyperprior? If it is GMM, the plots shown in Figure 4 is only one latent feature of the K=3? I think this should be specified to be clearer to readers.\n2. Will the code be released to the general public?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-12"}