Towards Efficient Image Compression Without Autoregressive Models
Authors: Muhammad Salman Ali
Yeongwoong Kim
Maryam Qamar
Sung-Chang Lim
Donghyun Kim
Chaoning Zhang
Sung-Ho Bae
Hui Yong Kim
Venue: NeurIPS 2023 poster (2023)
Keywords: Image Compression
Correlation
TL;DR: Towards Efficient Image Compression Without Autoregressive Models

Abstract: Recently, learned image compression (LIC) has garnered increasing interest with its rapidly improving performance surpassing conventional codecs. A key ingredient of LIC is a hyperprior-based entropy model, where the underlying joint probability of the latent image features is modeled as a product of Gaussian distributions from each latent element. Since latents from the actual images are not spatially independent, autoregressive (AR) context based entropy models were proposed to handle the discrepancy between the assumed distribution and the actual distribution. Though the AR-based models have proven effective, the computational complexity is significantly increased due to the inherent sequential nature of the algorithm. 
In this paper, we present a novel alternative to the AR-based approach that can provide a significantly better trade-off between performance and complexity. To minimize the discrepancy, we introduce a correlation loss that forces the latents to be spatially decorrelated and better fitted to the independent probability model. Our correlation loss is proved to act as a general plug-in for the hyperprior (HP) based learned image compression methods. The performance gain from our correlation loss is ‘free’ in terms of computation complexity for both inference time and decoding time. To our knowledge, our method  gives the best trade-off between the complexity and performance: combined with the Checkerboard-CM, it attains **90%** and when combined with ChARM-CM, it attains **98%** of the AR-based BD-Rate gains yet is around **50 times** and **30 times** faster than AR-based methods respectively

Towards Efficient Image Compression Without Autoregressive Models
Muhammad Salman Ali Yeongwoong Kim Maryam Qamar 1
Sung-Chame Lim², Donghyun Kim², Chaoning Zhang', Saong",Sunge Sung-Ho Ho Bac"), Hui Yong Kim
1 Kyung Hee University, Republic of Korea
Electronics and Telecommunications Research Institute (ETRI), Republic of Korea (sa Imanal i duddnd7575, ma ryamqamar) @' ac. kr,
(sclim, kimddng @etriee.........e... re. kr, (ttpniingzhanggg900mmmm... com
{shbae, hykim. v 10kukkueee ac
Abstract
Recently, learned image compression (LICC) has gamered increasing interest with its rapidly improving performance surpassing conventional codecs. A key ingredient of LIC a hyperprior-based entropy model, where the underlying joint probability of the latent image features is modeled as product of Gaussian distributions from each latent element. Since latents from the actual images are not spatially independent, autoregressive (AR) context based entropy models were proposed to handle the discrepancy between the assumed distribution and the actual distribution. Though the AR- AR-based models have proven effective, the computational complexity is significantly increased due the inherent sequential nature of the algorithm. In this paper, we present novel alternative to the AR-based approach that can provide significantly better trade-off between performance and complexity. To minimize the discrepancy, we introduce correlation loss that forces the latents to spatially decorrelated and better fitted to the independent probability model. Our correlation loss is proved to act a general plug-in for the hyperprior (HP) based learned image compression methods. The performance gain from our correlation loss is 'free' in terms of computation complexity for both inference time and decoding time. To our knowledge, our method gives the best trade-off between the complexity and performance: combined with the Checkerboard- C- it attains 90% % and when combined with ChARM-CM, it attains 98% of the AR-based BD-Rate gains yet is around 50 times and 30 times faster than AR-based methods respectively.
1 Introduction
In the digital world, visual media has proven to be mixed blessing, On the one hand, it provides mechanism to disseminate a plethora of data, information, and knowledge a form easily consumable by humans Silk et al. [2 while the other, improved quality and increased content is very quickly pushing the boundaries data storage, communication, and processing technologies Kim al. [2021). A software-driven solution to skew the balance of this state has been extensively presented in the form of lossy compression techniques Wallace [191]]. Rabbani and Joshi (2002). Ballé et al. [2017). In the case of images, these techniques aim reduce the data size while maintaining its context. Traditional image codecs like JPEG Wallace [191]] and JPEG2000 and Joshi 22022 use mathematically well-defined transformations such as discrete cosine transform (DCT) and discrete wavelet transform (DWT). These context-less transformations lead to reduced efficiency in terms of maintaining an optimal balance between the image quality and bitrate of the encoded image. In complex and varied images, this problem is especially pronounced, leading to the inclusion of various artifacts such as ringing and blocking Ballé al. [2017]. In recent years,
"Corresponing Authors
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

deep learning has significantly advanced several computer vision tasks Hu et al. [202], especially image compression, by incorporating structure and learned behavior. In particular, learned image compression (LIC) has gained popularity as field of research that has the potential to go beyond traditional compression methods by exploring the inherent nature of images Toderici et al. (2016).
Lee et al. 2022]. Johnston et al. (2018). Qian et al. [0022. Cheng et al. [2019].
State-of-the-art learned image compression tech- niques employ transform coding strategies for lossy image compression, which maps image pixels into quantized latent space Cheng et al. [2020). Zhong et al. [2020). This map then losslessly compressed using entropy coding methods such as Huffman or arithmetic cod- ing to create bitstream for transmission over a channel (Shannon 1948). The entropy coding algorithm requires an entropy model, which is prior probability model on the quantized latent representation. In this framework, deep neural networks are used for both transform coding and entropy modeling. By learning both modules end-to-end, the ability of deep neural networks is fully utilized to improve the compression per- formance Yang et al. 2023). The bitstream size depends the accuracy of the entropy model, making it essential to design an accurate model for efficient compression Kim al. [2022]. The objective of the entropy model is to estimate joint probability distribution over the elements of the quantized latent representation. A sim- plistic approach to assume complete indepen- dence among the elements, but this approach re- sults in subpar compression performance [alle et al. 2017 2018). Thus, modeling the quan- tized latent representation interdependencies in learned image compression is a significant chal- lenge Lee et al. [2019), Minnen et al. (2018).
20.0
sec
17.5
0.40%
97%
3.,%%
57 sec
15.0
BD Ra-R Gains (%)
6.3%%
12.5
10.0
7.5
Cheng's Hyperprior (CH)
CH + Low (Propased)
CH + Ckbd
5.0
CH Ckbd LLar (Proposed)
CH ChARM
2.5
CH ChARM + Loor (Proposed)
CH + AR Context
0.0
10
250
251
253
Inference Time (sec)
tradeoff using vari- entropy models on the top of Cheng's s architecture. Incorporating correlation loss to Cheng's hyperprior yields a BD rate gain of 9.5%, comparable to the gains achieved by the Checkerboard (CKBD) method. Apply- ing correlation loss to the CKBD model results in a BD rate gain of 6.13% in comparison to the CKBD haseline, signifying approximately 90% of the gain seen in AR- based models. Notably, this improvement is achieved with a computational expense merely 1/50th that of the full AR-based model. Similarly. incorporating corre- lation loss into ChARM lcads to BD rate gain of 3.3% over the ChARM bascline, akin 98% of the gain achieved by full AR model, while utilizing only 1/30th of the computational resources.
Figure 1: Performance-compleet
Qian et al. 2021 Theis et al. [2017].
common approach is use convolutional neu-
ral networks (CNNs) to extract additional fea- tures, known as "hyperprior." " which captures the context from the quantized latent representation to improve the efficiency of the entropy model Ballé al. [018). The underlying assumption of hyperprior- based entropy models is the probabilistic independence of the latent features, which contradicts the highly correlated nature of latent space, resulting a discrepancy between the actual distribution and the assumed probability model /iim et al.. [2022). To minimize this discrepancy, Autoregressive context model (AR-CM) based LIC methods have been proposed Minnen et al. [2018), where the probability of each latent is updated based on the previously coded neighboring latents. While this increases the model's overall performance, it introduces sequential dependencies. making roughly 50x slower than simple hyperprior-ber based models He et al. [2021] and hinders its practical implementation of neural image compression in real-world scenarios
To mitigate the above issue, a line of works have attempted to discard AR method for improving efficiency. A notable advancement in this domain is the channel-wise autoregressive entropy model (ChARM) introduced by Minnen et al. Minnen and Singh 2020]. ChARM focuses on reducing the element-level serial processing within the context model presented in Minnen et al. [2018), and demonstrates reasonably good RD performance compared to full autoregressive context model. Li et al. introduced CCN Li et al. (2020), which enables faster context calculation with moderate parallelizability. However, the effectiveness of CCN is still constrained by the image size, limiting its overall efficiency. He et al. He et al. [202 proposed a parallelizable checkerboard context model
2

---

along with two-pass decoding method, aiming to achieve better trade-off between rate distortion (RD) performance and computational efficiency. The efficiency of the above methods is increased at a cost of performance drop compared to AR-based models.
To this end, introduce correlation loss minimize the discrepancy without incurring any sequential dependency. The correlation loss forces latents to be decorrelated spatially, resulting in better fit with the spatially independent probability model. Our findings indicate that decorrelating the latent aids in enhancing the general effectiveness of the hyperprior entropy model. Figure I illustrates the effectiivnesss of the proposed approach. Applying correlation loss to Cheng's hyperprior model yields a comparable performance improvement to that achieved by integrating efficient context modeling approaches. When our approach is integrated with the Checkerboard method, it leads to substantial BD rate improvements of approximately 90%, and when coupled with ChARM, it exhibits BD rate gains of about 98%, akin to the gains observed in a full AR-CM. Notably, our method achieves these remarkable performance gains while requiring only 1/50th and 1/30th of the inference time required by the AR model, respectively. Our key contributions can be summarized as follows:
Our work takes the path of decorrelating latents that have never been tried in the field. The previous approaches primarily focus on CMs to enhance their performance, whereas, in our work, we address the fundamental need for employing computationally heavy CMs. With the introduction of our correlation loss, the correlation among neighbbe elements in the latent space is decreased. This would minimize the discrepancy between the hyperprior entropy model's s assumed probability distribution and the actual distribution of the latents, which eventually leads to an overall improvement of the rate-distortion (RD) performance.
Since our method modifies only the loss function, i.c. adding correlation loss to the original one, does not require any additional memory or computational complexity beyond the base model. In contrast the AR-based method which has severe sequential dependency among neighboring latent elements and thus could increase the inference time or decoding time up to 100x the base model, our method does not create any sequential dependency and thus can maintain the inference or decoding time of the base model.
Our proposed correlation loss acts as a plug-in in for the existing learned image compression methods and can achieve BD rate gains of up to 9.5% by combining with the base hyperprior- based models. Remarkably, the gains are comparable to the performance improvements achieved by combining the hyperprior module with efficient context-basedl LIC models, like the SOTA efficiency-oriented Checkerboard's context model (CM). When combined with Checkerboard method, our proposed approach acheives 90% and when combined with ChARM our proposed approach acheives 98% of the AR-CM's RD-Rate gains while incurring only 1/50th and 1/30th of the inference time of the AR-CM. Our method achieves the best trade-off between the complexity and performance.
2 Related Work
The autoregressive context based entropy model, inspired by the concept of context used in traditional codecs, is used to predict the probability of unknown codes based on latents that have already been decoded Bellard 2015). These models use hyper latent and context to predict both the location (mean value) and scale parameters (standard deviation) of the entropy model Minnen et al.) [2018). By combining elements of differentiable entropy modeling, hyper latent, and context models, is possible to outperform BPG Bellardd [2015) in terms of PSNR and MS-SSIM Minnen et al. [2018 1 He et al. M (222T), Qian et al. [002). These context models have also been extended to become more powerful but also more computationally expensive Cheng et al. (2020. In particular, AR context- based models, essential for achieving the state of the art performance, have low computational efficiency due to the sequential dependency of decoding the latents. As AR context models are sequentially dependent, they cannot be deployed on edge devices or take advantage of parallel hardware processing. As a result, models focusing on the real-world deployment of learned image compression often exclude
AR context models due to their inefficiency and high inference costs He et al. (021).
Parallel decoding approaches like Checkerboard He et al. [202]] and ChARM Minnen and Singh 20200) offer a better performance tradeoff over computation complexity. Checkerboard introduces parallel decoding approach by encoding and decoding only half of the latent variables using checkerboard-shaper context and hyperprior. The remaining latents, referred to as anchors, are coded solely based on the hyperprior. On the other hand, ChARM utilizes channel conditioning instead of
3

---

Anatyda Transfom
Analysk Tirrsfoom
Arsayda
Checkettard Contect Model
Tape Transform
Faddonsoed Entreey Model
S Moxiel
Moan Soak
Erewpy Modd
Fatrey
Synthaca Toneforn
Pypper Transform
Syninoss Transfam
Mun
Syttheii
Figure Operational diagram combining Hyper- prior Ballē et al. (2118) with our proposed correla- tion loss
Figure 3: Operational diagram combing Checkerboard method He et al. 2021] with our proposed correlation loss
element-wise conditioning, along with latent residual prediction, to enhance the rate-distortion (RD) performance. The recent efficient SOTA methods are all built top of the ChARM and checkerboard context model as an alternative to a fully AR-based context model Zhu et al. 2022 Qian et al.
[2022] Kim al. [2022].
Recent research the field of entropy coding has primarily concentrated on improving the efficiency of context-based methods. These methods aim to exploit the interdependencies within the latent space more effectively, utilizing the inherent dependencies present [ei al. [2022], Lee et al. [201)] However, there has been lack of attention towards enhancing the effectiveness of the hyperprior module, an integral component of prominent learned image compression (LIC) techniques. Notably, recent studies utilizing Transformer Vaswani et al. (0177] models have demonstrated that achieving higher level of decorrelation in the latent space leads to improved rate-distortion (RD) performance be attributed to the reduction in the discrepancy between
Zhu et al. [2022] This improvement the actual distribution of the latent space and the assumed probability model Kim et al.) [2022]. Our work aims to address this research gap by introducing novel approach that builds upon the existing findings. Specifically, we incorporate correlation loss during the training phase of our model, which promotes better decorrelation among the latent variables. By doing so, enhance the performance of the hyperprior module, thereby improving the overall efficacy of LIC models.
3 Methodology
3.1 Formulation of Learned Image Compression Models Image compression can be formulated through the transform coding approach by (as Figure 2 and
9a (z; Φ)
9 (v)
è = 9. (ú: 0).
where r and è are input and reconstructed image, is a latent representation, and 9 is the quantized latent representation. & and 0 are the parameters of analysis and synthesis transforms Ia and ga respectively, Q represents the quantization. During training, the quantization is approximated using uniform noise. While for inference, a round function is applied for quantization generate ÿ. and then entropy encoded to generate the bitstream. Given a probability model for the quantized representation, entropy encoding techniques such as arithmetic coding Rissanen and Jr. [1981] can losslessly compress the quantized codes. Arithmetic coders are optimal entropy coders which can be used estimate the entropy of ÿ during training.
A prior work Ballé et al. [0017) has employed a non-adaptive density model which was shared between the encoder and decoder also called factorized prior. It proposed hyperprior in the follow- up work as a mechanism to introduce side information z to capture the spatial dependencies in the latent space y Ballé al. 2018). The side information predicted the mean and scale information for the latent space as:
z = ha (y: Φn.) 8 = Q(z)
(2)
( 2) + ha (2: 0k),

---

Cheng's Hmperperr
Quantized Latent (
Mean lul
Scale (d)
Normalized 1
Cortelation Map
bpp (y +2) 0.052
bpp.: x at: 0.003
2op_y=
Aug Correlation: O 0.0454
PSNR: 24.37
Chang's Lea
Input Image
bpp ly zi: 061 (6:0 0010000
bpp_z o}: 0.004 (a=0.001
bep_y 0.057 (A-0.002 61)
Aug Correlation: 0.0107 (6-0.03474)
PSNR: 24, (b-0.38
Figure Visualization quantized latent ÿ. hyperprior (j and σ), normalized latent, and correlation map using Cheng's 's Hyperprior and Cheng's 's Hyperprior with correlation loss using an image from the Kodak dataset. The application of correlation loss enhances the details of the hyperprior and significantly reduces the correlation of latent variable g. leading to improved compression efficiency and reconstruction quality.
where ha and ht denote the analysis and synthesis transform for the hyperprior auto-encoder, Pala (Ú 2) is the estimated probability distribution for the quantized latent space ý, given by:
Pols (9: 2)
N (ui,o,) of) +u
- 1
2 2'
(0:)
(3)
Po 2)
II Paie (Ua 8),
where the mean μ4 and scale o, parameter is the ith element of h2) for each spatial location in the latent space j. With the premise that individual latent components are mutually independent, each spatial element in latent ý is modeled as a Gaussian with its own mean and standard deviation. We
may therefore estimate entropy using eq. 4 We modeled the probability of the hyper-latent 2 using non-parann non-parametric fully factorized density model as:
( vb) = II (PZIO(W) * ( 2 (-2 2) (a))
where denotes the th element of and i specifies the position of each element, and the vectors (i) encapsulate the parameters of each univariate distribution (P2.je (v))
context model can be added to boost the RD performance the mean-scale hyperprior framework. Additionally, parameter inference network (gep) is employed estimate the mean and scale parameters & = (μi o)) by leveraging the outputs of the hyperprior module (z) and the context model Dem ()::):
Φ = (yu, oi) = Jep (h, (2), Jem (f<r))
(6)
where dey means the causal context of the nearby visible latents to the latent Ú:. The context model can be calculated as:
9cm (x)) (M W)x+
where @ refers to the Hadamard product, b is a bias term. The variable M represents mask commonly used to establish a top-left reference, necessitating strict Z-ordered serial decoding.
The Checkerboard context model He et al. [2021] introduces a parallel decoding approach where only half of the latent variables are encoded/decoded using checkerboard-sharg -shpped context and hyperprior as shown in the Figure 3 The coding of the remaining half of the latents, referred to as anchors, relies solely on the hyperprior. implement this approach, the eq. 6 above is modified as:
+ = {
Gep (h2 (2), 0),,
gi E Janchor otherwise
Jep (ha (2), Fem (fanchor M @ W)),
The masked convolution, denoted as Dem. is applied conditioned on a theeckebbarddssappee mask M, as described in eq. [7] The input for gem, denoted as Manchors corresponds to the set of anchors, and represents the index for the i-th clement g in the latent variable ÿ. For methods that utilize
5

---

Point Mask
Short Cross Mask
Box Mask
Long Cross Mask
Figure 6: Our proposed mask designs for the corre- lation loss. We have used Point Mask as standard for our experiments. Please refer to the supple- mentary material for the detailed ablation studies.
After normalizing the latent _= the correla- tion will be calculated for all latents in the active x window, shown in pink, with the central point mn.
Figure 5:
mean- Gaussian entropy models, the entropy parameter & is represented as (μ, J),, where μ and correspond to the mean and standard deviation, respectively. In the presence of visible anchors, the context features of all non-anchors can be calculated simultaneously using masked convolution in parallel. The decoding of anchors also occurs in parallel. As result, the calculation of entropy parameters in eq. ( for decoding can be performed in two passes, which significantly improves efficiency compared serial context model.
LIC a langrangian multiplier based rate-distortion optmization problem. The loss function is then defined as
R + R(2) ((2, .)
=B log, (p a)) + E [ (Daje [))] À D(x,
where Ả controls the rate-distortion tradeoff and (xx; it) is the distortion term usually using mean square error.
3.2 Our Proposed Correlation Loss
As shown in the eq. 4 the joint probability of ú in the hyperprior-basee entropy is calculated based on the assumption that the latent features are mutually independent. However, as shown the upper portion of last column of Figure 4 the latent space is highly correlated, resulting in a discrepancy between the actual and the calculated probability distribution. To address this discrepancy, we introduce correlation loss to force the latent to be decorrelated spatially. We can therefore model the correlation loss by calculating the correlation in the latent space. We start by computing the correlation map as:
Corr_Mapkxie
Exwp(>)
[
/4i Vi (i-111 σ;
Um
1mm ]
,0s ,osi<k
(10)
Om
where x k is the window size of the correlation map, Jr and refer to the hyper-latent's corresponding l: as shown Figure We slide the window with the stride of across the entire latent space to calculate the correlation of the central point m with all the other points in the window of size k x k. We then take the mean of all the individual correlation maps get the resultant correlation map of the latent y of size k x k.
7th and mh elements, m refers to the central point of the window of size
We then apply the Mask on the correlation map. shown in Figure 6 the point mask only masks the central location of latent space as it corresponds to self-correlation of 1 (when i = m). We propose several mask designs (discussed in supplementary material) to investigate the effect of the spatially relaxed decorrelation constraint. Mask can then be applied as:
Masked Mopsxxlli = Morr_Mapexaee @ Mask.
(11)
The correlation loss is formulated by applying the L2 norm on the Masked Masked_MMpixxe
Leorr [|MasseddMMapxxxii]]] [i]lxx
(12)
By incorporating the correlation loss Tyoor. the RD loss can now be given as:
RDioss = E |- logg (0 2) - log, p2)) + A Enppie [d(x,))] + a [Loor]s (13)
where a is the scaling factor for the correlation loss.
Given that the entropy estimate presumes independent latent elements, our proposed loss function increases compression efficiency by reducing the correlation between latent features, hence reducing the discrepancy between the assumed probability distribution of the entropy model and the actual latent probability distribution.

---

PS.RR
FOWR
Meners thperzrer
0.1
0.2
0.3
0.4
0.5
9.6
0.1
0.2
0.3
0.5
0.6
0.3
BPP
BPP
(a) Minnen's 's mean scale hyperprior (b) Cheng's 's mean scale hyperprior
(c) Swin'T mean scale hyperprior
Figure 7: RD rate comparison of (a) Minnen's (b) Cheng's and (c) Swin'T scale hyperprior models with the inclusion correlation loss.
4 Experiments and Results
4.1 Experimental Setup
We train all our models on the Vimeo-90k dataset al. [2019), with the images cropped to the resolution of 256 x 256 for training. The models were optimized using the Adam optimizer Kingma and Ba [2015] with batch size 16 and trained for 1.5 million iterations with learning rate of 1 x 10 for the first million iterations and then halved every 50,000 iterations till 1.25 million iterations.
RD loss given in eq. 9 is used to train the baseline models, whereas the modified loss in eq. 13 is used train models with correlation loss. The rate-distortion tradeoff is guided by X, whose value contained in the set [0.0009, 0.0018, 0.0035, 0.0067, 0,0130, 0.0250J.
We tested model uncompressed images of 768 x 512 or 512 x 768 resolution. Bits per pixel (bpp) provide the rate, while PSNR indicates the quality of the reconstructed image when evaluating rate-distortion performance. In order to demonstrate the coding efficiency, RD curves are drawn.
commonly used Kodak lossless images dataset Kodak [993] with 24
4.2 Rate-Distortion Performance
We employ Minnen's Minnen et al. [018). Cheng's Cheng et al. 2020 and Swin'T Liu et al. 2021 based mean scale hyperprior models well as Cheng's with Checkerboard He al.) [021] for our experiments. We perform all our experiments on the Pytorch framework Paszke et al. [017] and use the CompressAl library [Bégaint et al. [2020] Minnen's 's and Cheng's models were trained using an NVIDIA 2080Ti, whereas the Swin'T model was trained on an NVIDIA 3070Ti due to the transformers' high memory requirement. Note that including correlation loss has a negligible effect
on the hardware requirement of these base models.
Minnens' Hyperior: Figure 7a depicts the RD curve for Minnen's model Minnen al. 2018 with and without correlation loss; for Minnen's 's model, correlation loss achieves BD rate gains of 3.20%.
Cheng's Hyperprior: Figure 7b illustrates the RD curve for Cheng's model Cheng et al. 2020 with and without correlation loss. Cheng's 's model yields BD rate gain of 9.5% with correlation loss. Cheng's 's model consists of attention modules along with CNN layers that significantly increase the overall efficiency of the correlation loss. Figure 9 depicts the spatial correlation of latent averaged across all latent channels and compares the baseline Cheng's model with the one trained on correlation loss. exhibits significantly less correlation than the baseline model. This observation holds for the models trained with other À values. The qualitative comparison in Figure g between Cheng's 's model with and without correlation loss demonstrates the superior performance achieved by incorporating correlation loss. It implies that models trained on correlation loss incur less redundancy across different spatial latent locations, resulting in better rate inte-distortion trade-off overall.
that the model trained with our proposed loss function
Transformer Hyperprior:
We also compared the effectiveness of correlation loss on the Transformer-
based LIC model. We implemented our Swin'T with a hyperprior model inspired by Zhu's work and employed similar architectrall setting Zhu et al. (2022). Transformers tend have lesser correlation in their latent space than the convolutional-baae LIC models Zhu et al. 2022 The

---

Coct
0900
000
0001
2006
8.000
1.00
Cheng's Hyperprior
Cheng's with Correlation
SPP: 27.7448 gPP:
(A=0.0067]
(A =0.00057)
Kodal: PSNR=30 58d8, 8PP-0.27
Kodak: PSNR- BPP-0
Figure 8: Comparison of reconstruction quality of the original image from Kodak dataset using Cheng's hyperprior and Cheng's hyperprior with correlation loss.
Figure 9: Spatial correlation map comparison of Cheng's hyperprior (left) with Cheng's hyperprior with correlation loss (right)
34
34
32
PSNR
30
PSNR
30
28
28
26
26
- Cheng's with Checkerb + Correlation Lozs Cheng's with Checkerbzant
Cheng's with CMAR Creng's with CHARK Correationn Loss
24
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3 BPP
0.4
0.5
BPP
RD rate comparison of Cheng's Checkerboard with the inclusion of correlation loss.
Figure 10:
Figure 11:
RD rate comparison of Cheng's 's + ChARM with the inclusion of correlation loss.
Figure 7c illustrates the performance of the SwinT model with and without correlation loss. We found that with the correlation loss, the SwinT model achieves BD-rate gain of about 4.8%. The effectiveness of correlation loss for the SwinT model further indicates the efficacy of the correlation loss, which in turn improves the mean scale based entropy model.
Figure 10 illustrates the RD curve for Cheng's 's model with Checkerboard He et al. [021]] with and without correlation loss. Cheng's with Checkerboard yields BD rate gain of 6. 13%% with correlation loss. Checkerboard uses a parallel decoding approach context. The remaining latents, referred to as anchors, are coded solely based on the hyperprior. Since half of the latents are encoded by the hyperprior module. Incorporating correlation loss into the Checkerboard context model significantly improves the performance of the hyperprior module, resulting in an overall enhancement of the model's performance.
Cheng's with Checkerboard Context Model:
by encoding and decoding only half of the latent variables using checkerboard-sheee
Cheng's with ChARM: Figure II illustrates the RD curve for Cheng's 's model with ChARM Minnen and Singh 2020). both with and without the integration of correlation loss. Notably, when correlatio loss is incorporated into Cheng's with ChARM, there is a substantial BD rate gain of 3.3%. ChARM effectively employs channel context and latent residual prediction for the encoding and decoding of the latents. The combination correlation loss with ChARM leads to a significant enhancement in its overall performance, marking performance difference of approximately 0.5% when compared to a full AR model.
4.3
Complexity Tradeoff
The primary advantage the correlation loss lies its capacity to minimize the discrepancy between the actual and presumed probability distribution within the entropy model. Figure I illustrates that for

---

Table 1: Average encoding and decoding time the Kodak dataset for different entropy madels applied
Cheng's Hyperprior. BD-Rate gains are evaluated using Cheng Hyperprior as baseline.
Architecture
BD Rate Gains ( (%)
Inference Time (sec)
Cheng's Hyperprior (CH)
4.66
CH Correlation Loss (Proposed)
9.5
4.66
CH + Checkerboard
10.37
5.33
CH Checkerboard + Correlation Loss (Proposed)
16.50
5,33
CH + ChARM
14.69
8.14
CH ChARM + Correlation Loss (Proposed)
17.99
8.14
CH + AR Context
18.47
251.65
Change taperater (011 MAR man
Olang CI00 Park dar.
34
+0.66
H
32
+8.21
+0.4
PSNR
30
83
28
6.18
7dkl Gan
6.18
+822
4.33
Cheng's ttyperption Correaaioon
Conration
chang's with Checkerboard Coreeation
+8.42
405
Cheng's with ChARM Cormtation
Cheng's Hyperppior
-825
0.1
0.2
0.3 BPP
0.4
0.5
0.6
65 -1.0
0.1
3.2
9.
82 0.3
Figure 12: Comparison between Cheng's 's AR con- text, Cheng's 's hyperprior, Cheng's 's hyperprior with correlation loss, Cheng's Checkerboard with corre- lation loss, and Cheng's ' ChARM with correlation loss
Figure 13: Visual representation depicting the relationship between bits per pixel (bpp) and correlation, and influ- ence on peak signal-to-noiss ratio (PSNR) gains. For all the four models, both the average correlation and the gains decreases as the bpp value incrcases.
a given encoder and decoder architecture, which is Cheng's model in this case, the choice of entropy model determines the performance of the resulting model. Note that Cheng's 's hyperprior signifies the lower bound with the given encoder-decoder architecture, whereas Cheng's ' AR defines the upper limit with the same decoder architecture. Using Cheng's hyperprior model the baseline, we compared the effectiveness of including an AR module vs. the correlation loss. The Figure [2] shows the RD curve comparison, while the Table compares the inference time for processing the entire Kodak test dataset. Incorporating correlation loss into Cheng's 's hyperprior model results in performance improvement roughly half that of the AR-context-based model, all while reducing inference costs to just 1/55th of the original. Similarly, when utilizing Cheng's 's Checkerboard with correlation loss, a substantial performance enhancement approximately 90% is achieved compared to the AR-context--eee -based model, with an inference cost as low as 1/50th of the latter. Additionally, Cheng's ChARM with correlation loss attains an impressive performance gain of about 98% when compared to the AR-context based model, while incurring an inference cost of merely 1/30th of the AR-context-baae based model as shown in Figure [
It should be noted that when correlation loss is applied to Cheng's hyperprior, significant improve- ments can be achieved due to the large performance gap between Cheng's 's hyperprior baseline and the upper limit. On the other hand, since the gaps are diminished in models like Cheng's 's with Checkerboard and Cheng's with ChARM, the potential for improvement by applying correlation loss is also limited when compared to Cheng's hyperprior baseline. In summary, the introduction of correlation loss to these models would result in a closer approximation to the full AR performance, albeit with gains that are comparatively smaller.

---

5 Analysis
5.1
Visual Illustatioon of Effectiveness of the Correlation Loss
Figure 4 visualizes some of the internal workings of our models. We compare Cheng's s hyperprior model with and without correlation loss. By observing the first column of Figure 4 is clear that our correlation loss makes the latent feature preserve more detailed texture information yiclding higher PSNR compared with the base model. The mean can also extract more distinctive and meaningful information from the latent space. Since the hyperpriors (mean ju and scale σ)) contain more information, they require slightly more bits than the base model. Nevertheless, the rise in hyperprior rate is adequately offset by the relatively greater decrease in latent bitrate, as demonstrated in the final column of Figure where the normalized latents exhibit improved decorrelation compared to the base model. Overall, our proposed method surpasses the base model by enhancing PSNR through more decorrelated latents while either maintaining or reducing bpp with better decorrelated latents.
5.2 Effect of Correlation score on the Model's performance
Our proposed loss function consistently demonstrates significant BD-rate gains at lower bit rates while maintaining comparable performance at high bit rates. This behavior is primarily attributed to learned image compression (LIC) models exhibiting higher correlation among latent variables at lower bpp values compared to higher bpp values as reported by Zhhu et al. [2022] and also evident from the correlation maps in Figure 6 of the supplementary material. We conducted a comprehensive analysis and presented the findings in Figure 13 Figure 13 showcases the relationship between PSNR gains, bpp, and correlation for different models, including Cheng's 's hyperprior, Cheng's 's with Checkerboard, SwinT hyperprior, and Minnen's 's hyperprior. From the graphs in the Figure 13 a clear pattern can be observed: as the bpp decreases, the correlation of the latents increases, resulting in a higher gain in PSNR. However, as we move towards higher bpps, the correlation becomes notably reduced, resulting in decreased PSNR gain. This trend explains the reason why the efficacy of the correlation loss is larger in low bpp range and diminish high bpp range.
6 Ablation Studies
Comprehensive ablation studies regarding various mask types, mask sizes, and a values are presented in the supplementary material.
7 Conclusion
The hyperprior-busee entropy models assume probabilistic independence, which leads to a discrepancy between the actual and the assumed probability distribution. We proposed correlation loss, which reduces the correlation among spatially neighbored elements in the latent space resulting in a better fit with the spatially independent probability model. Our proposed loss function does not require any model structure or capacity changes and acts as a plug-in method for the existing neural compression methods. The proposed approach, without modifying the entropy models and increasing the inference time, significantly improves the RD performance with BD rate gains up to 17.99%.
Acknowledgments and Disclosure of Funding
Authors are thankful Muhammad Awais for his constructive and helpful feedback. Authors are also thankful to Taegoo Kang for managing and providing the required compute resources. This work was supported in part by the Institute of Information and Communications Technology Planning and Evaluation (ITTP) grant funded by the Korea Government [Ministry of Science and Information Communication Technology (MSIT)] (Development of Audio/Video Coding and Light Field Media Fundamental Technologies for Ultra Realistic Tera-media) under Grant 201 7-0-0007;;; in part by the IITP grant funded by the Korea Govermment (MSIT) (Artificial Intelligence Convergence Innovation Human Resources Development, Kyung Hee University) under Grant RSS2002-00015591 in part by the IITP grant funded by the Korea Government (MSIT) (Artificial Intelligence Innovation Hub) under Grant 2021 -0-00002
10

---

References
Johannes Ballé, Valero Laparra, and Eero P. Simoncelli. End-to-end optimized image compression. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24- 26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL htttps:://ppprreeeeeee jeg
net /
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. In 6th International Conference on Learning Represen- tations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja. Compressai: pytorch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:2011.03029 2020.
Fabrice Bellard. Bpg image format. URL https://bellard. org/bpg, 2015.
Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Deep residual learning for image compression. IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019, Long Beach, CA. USA, June 16-20, 2019, page O. com/ content CVPRW 2019 CLIC 2019/ htw./////ggg...eee Deep Residual Learning paper.htm.
Computer Vision Foundation IEEE, 2019. URL
For Image Compression ccvpss
CVPRN 2019
Zhengxue Cheng, Heming Sun. Masaru Takeuchi, and Jiro Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In 2020 IEEE/CVF Conference
Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages
7936-7945. Computer Vision Foundation / IEFE, 2020. doi: 10.1109/CVRR42600.200 20020000000000076666 URL. htt:s:: http:s://ppencccesss.cheeee wtw.:/cwwe...e..cc......... com cwnt
/Cheng Likelihooe and Awteention_eeeeROOOOOOO 1a1o020ppppeehttre Mttuaaussssaaa_iuuree____
Learned Image_ wtt.apppreessaoooata_iiaaaae...a
Dailan He, Yaoyan Zheng, Baocheng Sun, Yan Wang, and Hongwei Qin. Checkerboard context model for efficient learned image compression. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 14771 17771-1470. Computer Vision Foun-
dation IEEE, 2021. doi: 10. 20.110//VPR44437..000 2ww.:P/4444477222200222 011453.. URL https: https//essspppp//e/////eeee
thecvf. ccm/ for wefficiiet__Learneeeeee http:///22//aaaeeeeeeeceeeeerrrr
htw.://///tmmmeeeeeeeeee Checkerboard Caataa_oooeettaae Moootext_MMdell11 Model Image Compression 200.pppr...
Yueyu Hu, Wenhan Yang, Zhan Ma, and Jiaying Liu. Learning end-to-end lossy image compression: A benchmark. IEEE Trans. Pattern Anal. Mach. Intell, 4441),419442111, 2022. doi: 10.1 TPAML.2021.3065339, URL htLp:tts::0 /oi. wtt.:......././......... 2021 30652339333399
Nick Johnston, Damien Vincent, David Minnen, Michele Covell, Saurabh Singh, Troy Chinen, Sung Jin Hwang. Joel Shor, and George Toderici. Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4385 4393, 2018
Jun-Hyuk Kim, Byeongho Heo, and Jong-Seok Lee. Joint global and local hierarchical priors for learned image compression. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5992-6001, 2022.
Youmin Kim, Jinbae Park, Younho Jang, Muhammad Salman Ali, Tae-Hyun Oh, and Sung-Ho Bae. Distilling global and local logits with densely connected relations. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC. Canada, October 10-17, 2021, pages 6270-6280, IEEE, 2021. doi: 10. 10.1009/CCCV489222220..000 //doi wrw/r0...00............o 1 109
URL https
10t.0.1199/0/44300 Tttp//CCV4422222 2021 60623
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL.
http: / /arxiv wwwp/bs./////ooooooo//t..ot... abs 1412 6980

---

Eastman Kodak. Kodak lossless true color image suite (photocd pcd0992), 1993. URL http:
7/rCk https//a/ahc.///oaak//////
Jac-Han Lee, Seungmin Jeon, Kwang Pyo Choi, Youngo Park, and Chang-Su Kim. Dpict: Deep progressive image compression using trit-planes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16113-1612,, 2022.
Jooyoung Lee, Seunghyun Cho, and Seung-Kwon Beack. Context-adaptive entropy model for end-to- end optimized image compression. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https
7/openreview wwt.eeewweeeeeeeeeeeeeeeeeee.e.e / for um didmmmididdiiaaa.
Jianjun Lei, Xiangrui Liu, Bo Peng. Dengchao Jin, Wanqing Li, and Jingxiao Gu. Deep stereo image compression via bi-directional coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19669-19678, 2022.
Mu Li, Kede Ma, Jane You, David Zhang, and Wangmeng Zuo. Efficient and effective context- based convolutional entropy modeling for image compression. IEEE Trans. Image Process. 29: 5900-5911, 2020. doi: (0.1109/TI2220222299522 URL https:
o.. http:////oooooogo.....o/ 1109/
TIP 2020. 2985225
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEE//CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC. Canada, October 10-17, 2021, pages 9992-10002, IEEE, 2021. doi: 10. 10.1109//0O 10.109//CCY4899222210000999
URL
http:///............1111////122/ 2021. 00986
David Minnen and Saurabh Singh. Channel-wise autoregressive entropy models for learned image compression. In 2020 IEEE International Conference on Image Processing (ICIP), pages 3339- 3343. IEEE, 2020.
David Minnen, Johannes Ballé, and George Toderici. Joint autoregressive and hierarchi- cal priors for learned image compression. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 10794-10803, 2018. [SccaboesSScccff2s599333333eeee3
www.riiiinnnnnnussssp....s neurips hocppaper/201/hhasl .tml
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Yichen Qian, Zhiyu Tan, Xiuyu Sun, Ming Lin, Dongyang Li, Zhenhong Sun, Hao Li, and Rong Jin. Learning accurate entropy model with global reference for image compression. International Conference on Learning Representations (ICLR), 2021.
Yichen Qian, Ming Lin, Xiuyu Sun, Zhiyu Tan, and Rong Jin. Entroformer: A transformer-based en- tropy model for learned image compression. International Conference on Learning Representations (ICLR), 2022.
Majid Rabbani and Rajan Joshi. An overview of the jpeg 2000 still image compression standard. Signal processing: Image communication, 17(1):3 48, 2002.
Jorma Rissanen and Glen G. Langdon Jr. Universal modeling and coding, IEEE Trans. Inf. Theory, 27(1):12 22, 1981. doi: 11110/0T..981.11066666... TIT.1981 1056282
Claude E. Shannon. A mathematical theory of communication. Bell Syst. Tech. J., 27(3):379-423 1948. TB0133888...
1948. doi: 10 1002/J 1538- 7305. 1 1948.14. Eb01338.
Matthew Silk, Ricardo Correia, Diogo Veríssimo, Audrey Verma, and Sarah I. Crowley. The implications of digital visual media for human- relationships, 2021.
12

---

Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compression with compressive autoencoders. In Sth International Conference on Learning Representations. ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL htt.pss://ppeeeevieeee
net/forum 1d=gg.Nwy9
George Toderici, Sean M O' O'Malley, Sung Jin Hwang. Damien Vincent, David Minnen, Shumeet Baluja, Michele Covell, and Rahul Sukthankar. Variable rate image compression with recurrent neural networks. International Conference on Learning Representations (ICLR), 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Gregory K Wallace. The jpeg still picture compression standard. Communications of the ACM, 34(4): 30 44, 1991.
Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T. Freeman. Video enhance- ment with task-oriented flow. Int. J. Comput. Vis., 12708):110--1155, 2019. doi: 10.1007/ 1tt.:/..00//.1110//11111111
$11263-018-0114422 URL
018 018-01144-2
Yibo Yang, Stephan Mandt, Lucas Theis, et al. An introduction to neural data compression. Founda- fions and Trends in Computer Graphics and Vision, 15(2):113-200, 2023.
Zhisheng Zhong, Hiroaki Akutsu, and Kiyoharu Aizawa. Channel-level variable quantization network for deep image compression. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 467-473. International Joint Conferences on Artificial Intelligence Organization, 7 2020. doi: 10.24963/jcaai20200666 URL httpss://coi.oogg org 10. ott.://oo.....20//////.......... 2020//5 Main track.
Yinhao Zhu, Yang Yang, and Taco Cohen. Transformer- transform coding. In The Tenth Inter- national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https:: /ttp:///opeeeeeeeeeeeeooe net. ormommmmmi de jHnk8
13


Reviews:
[{"review_id": "JNWzXHiMx1", "reviewer": "Reviewer_xepj", "summary": "This paper focus on efficient learned image compression. Different from existing method, which generally aims to parallelize the autoregressive operations, this paper propose to speed up the framework by removing the whole autoregressive model by introducing the correlation loss, aiming to decorrelate the latent features. The introduced correlation loss can act as a plug-in for the existing learned image compression methods to achieve superior RD performance and at the same time reduce the inference time without autoregressive entropy model.", "questions": "1. Is the hyperprior shown in Figure 4 a GMM-based (K=3) or a simple mean-scale hyperprior? If it is GMM, the plots shown in Figure 4 is only one latent feature of the K=3? I think this should be specified to be clearer to readers.\n2. Will the code be released to the general public?", "limitations": "NA", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well-motivated and easy to follow.\n2. The introduced correction loss is statistically sound and empirically proved to be effective. The proposed loss is novel to me and can act a plug-in for the existing learned image compression to achieve consistent improvement, especially at low-bit ranges.\n3. By removing the correlation, the propose method can ease the requirement of autoregressive entropy model, so as to speed up the whole compression framework.", "weaknesses": "1. The citation in Figure 3 is wrong.\n2. The caption of Figure 4 states that the correlation loss can provide more flexible parameterized distribution models with significant spatial redundancy reduction. However, from my perspective, the plots show in Figure 4 cannot showcase both \"more flexible parameterized distribution models\" and \"significant\" spatial redundancy reduction. I hope the author can have more comments on this, otherwise such claims are unsupported or inaccurate.\n3. The introduction of section 3.1 should be reduced, which are general, well-known concepts in LIC, the main focus with more words should be put into the contribution introduced section 3.2.\n4. The paper should also report the performance of the proposed method on top of some more recent efficient LIC methods like [1].\n\n[1] Dailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang. ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding. CVPR 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and suggestions.\n\n> The caption of Figure 4 states that the correlation loss can provide more flexible parameterized distribution models with significant spatial redundancy reduction. However, from my perspective, the plots show in Figure 4 cannot showcase both \"more flexible parameterized distribution models\" and \"significant\" spatial redundancy reduction. I hope the author can have more comments on this, otherwise such claims are unsupported or inaccurate.\n\nWe acknowledge the reviewer's observation regarding the clarity of the caption of Figure 4 and the accompanying text, and we appreciate the feedback. To rectify this concern and provide a clearer presentation of our intended message, we have undertaken significant revisions to the Figure itself, which can be referred to in the attached PDF document (Figure 1). Moreover, we have taken comprehensive steps in our general response to thoroughly address all the concerns raised by the reviewers concerning Figure 4.\n\n> The paper should also report the performance of the proposed method on top of some more recent efficient LIC methods like [1].\n\nPlease refer to the “Performance on ELIC: Efficient Learned Image Compression” in General Response for Common Comments Section\n\n> Is the hyperprior shown in Figure 4 a GMM-based (K=3) or a simple mean-scale hyperprior? If it is GMM, the plots shown in Figure 4 is only one latent feature of the K=3? I think this should be specified to be clearer to readers.\n\n The hyperprior shown in Figure 4 is a simple mean-scale hyperprior.\n\n> Will the code be released to the general public?\n\nWe shall release the code with the camera-ready version of the paper.\n\n> The citation in Figure 3 is wrong.\n \n>The introduction of section 3.1 should be reduced, which are general, well-known concepts in LIC, the main focus with more words should be put into the contribution introduced section 3.2.\n \nWe express our gratitude to the reviewer for their valuable insights, comments, and suggestions, which have greatly contributed to the improvement of our paper. We are committed to incorporating these recommended changes and enhancements in the final version of the paper to ensure its quality and accuracy.\n\n\n\n"}, {"review_id": "6LjIubcAq1", "reviewer": "Reviewer_7bvo", "summary": "This paper presents a new loss to improve neural image compression models. The authors identify a potential issue with typical neural image compression models where the hyperprior, which predicts entropy parameters over a quantized latent representation, assumes conditional independence between latents, but that may not be true in practice. Previous methods improved the entropy model through context modeling (e.g., using a spatially autoregressive model or a \"checkerboard\" decomposition) but this leads to higher compute and slower runtimes.\n\nInstead, the authors add a local correlation loss averaged spatially over the latents. This loss directly encourages the encoder to generate decorrelated latents, which is a better match for the conditional independence assumption built in to the hyperprior. In particular, the loss affects training (and isn't very expensive) and adds no additional computation at inference time, which leads to significant runtime improvements.\n\nThe paper shows that the new correlation loss improves rate-distortion (RD) performance when applied to several popular models (see Fig. 1 and Fig. 7).", "questions": "Analysis exploring why the correlation loss boosts performance would strengthen the paper. Specifically, why is it needed when the existing rate-distortion loss is minimized by decorrelated latents? The generic answer is \"the network is stuck in a (bad) local minimum, and the correlation loss changes the loss landscape such that the optimizer doesn't get stuck\". ", "limitations": "adequately addressed", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The primary strength of the paper is the simplicity and effectiveness of the main idea.\n\nMost papers on neural compression boost RD performance by making the model more complex, which typically leads to slower decode times. The authors discuss this problem and present a well-motivated loss that leads to significant RD gains without affecting runtime. Their correlation loss is also quite general and can be applied to many different compression models.\n\nAs far as I know, the correlation loss has not been proposed elsewhere in the neural compression literature. There are some similarities to a diversity loss in VQ or clustering, though I don't have a specific reference for this. So I think the originality, especially for the neural compression subfield, is high.\n\nThe quality and clarity of the writing is also high.\n\n", "weaknesses": "Ideally, the correlation loss presented in the paper would be applied to a SOTA comperssion model leading to a new SOTA. For instance, the paper cites (He 2021), which introduced the checkerboard decomposition for entropy modeling, but they don't build on top of (He 2022), which presents a more powerful model that combines the checkerboard with CHARM.\n\nELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding\nDailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, Yan Wang\nhttps://arxiv.org/abs/2203.10886\n\nFigure 1 implies that the benefit of the correlation loss shrinks as models become more powerful so it may be that the benefit for ELIC is negligible?\n\nThe visualization in Fig. 4 is great (and is common for compression papers) but it's not obvious to me what I should be looking at to see the impact of the correlation loss. Is it a sharper scale image? The goal is an i.i.d. Gaussian normalized image (far right column) but it's not visually obvious to me that \"our approach\" is closer to i.i.d. Gaussian than the baseline.\n\nMask patterns are mentioned and shown in Fig. 6 but results are only in the supplemental material. Presumably that's because the mask shape did not have a large impact. That's fine, but maybe cut Fig. 6 or at least add a sentence to the main paper summarizing the findings.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and suggestions.\n\n\n> Ideally, the correlation loss presented in the paper would be applied to a SOTA compression model leading to a new SOTA. For instance, the paper cites (He 2021), which introduced the checkerboard decomposition for entropy modeling, but they don't build on top of (He 2022), which presents a more powerful model that combines the checkerboard with CHARM.\nELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding Dailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, Yan Wang https://arxiv.org/abs/2203.10886\n\nPlease refer to the “Performance on ELIC: Efficient Learned Image Compression” in General Response for Common Comments Section\n\n> Figure 1 implies that the benefit of the correlation loss shrinks as models become more powerful so it may be that the benefit for ELIC is negligible?\n\nThe primary advantage of the correlation loss lies in its capacity to minimize the discrepancy between the actual and presumed probability distribution within the entropy model. Figure 2 in the attached PDF illustrates that for a given encoder and decoder architecture, which is the Cheng's Hyperprior (CH) in this case, the choice of entropy model determines performance of the resulting model. Note that CH signifies the lower bound with the given encoder-decoder architecture, whereas Cheng’s AR defines the upper limit with the same encoder-decoder architecture. When correlation loss is applied to CH, significant improvements can be achieved due to the large performance gap between the CH baseline and the upper limit. On the other hand, since the gaps are diminished in the models like CH + CKBD and CH + ChARM, the pothential for improvement by applying correlation loss are also limited when compared to the CH baseline. In summary, the introduction of correlation loss to these models would result in a closer approximation to the full AR performance, albeit with gains that are comparatively smaller. \n\nWe expect the similar pattern might be observed if we change the encoder-decoder architecture from CH's to ELIC's. Note that ELIC not only impove the entropy model by the proposed space-channel context model (SCCTX), which combines CKBD and ChARM, but also introduced architectural modification of the encoder and decoder, which contribute an additional BD rate gain of approximately 8-12% [1]. While we expect that the performance of Cheng's with SCCTX might be lower than ChARM with correlation loss, the inclusion of correlation loss has the potential to elevate it to a level comparable to, or akin to, the full AR performance, all while costing only about 1/15th of the computation expense (2x of ChARM).\n\n[1] He, D., Yang, Z., Peng, W., Ma, R., Qin, H., & Wang, Y. (2022). Elic: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5718-5727).\n\n> The visualization in Fig. 4 is great (and is common for compression papers) but it's not obvious to me what I should be looking at to see the impact of the correlation loss. \n\nWe acknowledge the reviewer's observation that the interpretation of Figure 4 lacks clarity, particularly in relation to the depiction of the reduced correlation of the latent variable y in the last column of our Figures (main manuscript [Figure 4], supplementary material [Figure 5]). We have given careful consideration to this concern and have taken steps to enhance the clarity of the Figure (please see Figure 1 in the attached PDF). Our goal is to provide a more straightforward and comprehensive illustration of the impact of correlation loss. For a more concise and coherent explanation, we kindly refer the reviewer to the general response section.\n\n> Mask patterns are mentioned and shown in Fig. 6 but results are only in the supplemental material.\n\nWe highly value the constructive feedback provided by the reviewer and are committed to incorporating these suggested changes into the camera-ready version of our paper.\n\n> Analysis exploring why the correlation loss boosts performance would strengthen the paper. \n>  Specifically, why is it needed when the existing rate-distortion loss is minimized by decorrelated latents? The generic answer is \"the network is stuck in a (bad) local minimum, and the correlation loss changes the loss landscape such that the optimizer doesn't get stuck\".\n\nThe primary focus of our correlation loss is directed at diminishing the correlation present among neighboring elements within the latent space, as illustrated in Figure 1 of the attached PDF. This reduction in correlation assumes a critical role in mitigating the disparities between the assumed probability distribution of the hyperprior entropy model and the actual distribution of latent variables. The effect of decreased correlation is illustrated in the figure and the detailed analysis can be found in the text of the our main rebuttal section.  \n\nWe agree there is high possibility that the correlation loss introduces alterations to the loss landscape. However, it remains uncertain due to the current lack of concrete theoretical evidence about exact interplay between the correlation loss, rate loss, and the distortion loss. Thus, in the current circumstances, we cannot definitively assert whether this is indeed the case.\n\n\n\n"}, {"review_id": "L4kMD5yg0m", "reviewer": "Reviewer_vwAa", "summary": "This paper proposed a correlation loss to decrease the correlation among spatially-neighbored elements in the latent space. By only modifying the loss function, this method acts as a plug-in method for the existing neural compression methods with no complexity increasing. Experiments show improvement in the compression performance to several baseline models.", "questions": "Minor issues:\n\n1) Some experiments in the supplementary materials are kind of important and can be put into the main paper.\n\n2) Some citations are still preprint version (e.g. [1][2][3] in reference). Their officially published version should be cited.\n\n", "limitations": "None", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The main contribution of the paper is the correlation loss proposed to decrease the correlation among spatially in latent features. \n\nExperiments show improvement in the compression performance to several baseline models which provide some insights about designing better neural image compression networks from the prospective of feature map decorrelation.", "weaknesses": "1) For SwinT and Cheng with checkerboard, the correlation loss seems to only work for lower bitrates, the author may provide some explanation. Besides, The current tested bitrate range is relatively low. RD performance at higher bitrates (>1bpp) should also be provided. \n\n2) The paper writing can be improved. The citation format in the article looks messy. For equations, Some are ‘eq’, but some are ‘Equation.’\n\n3) Experiments on factorized Ball\\'e method [1] and HP+AR+correlation loss can be provided to makes the experiments and evaluation more complete.\n\n[1] Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compression.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and suggestions.\n\n> For SwinT and Cheng with checkerboard, the correlation loss seems to only work for lower bitrates, the author may provide some explanation\n\nPlease refer to our general response for a detailed explanation.\n\n> Besides, The current tested bitrate range is relatively low. RD performance at higher bitrates (>1bpp) should also be provided.\n\nOur experimentation focused on a bits per pixel (bpp) range spanning from 0.05 to 0.6, corresponding approximately to a peak signal-to-noise ratio (PSNR) range of 25 to 35. In alignment with this approach, Cheng's and Checkerboard methods also adhere to a comparable range, operating within 0.1 to 0.8 bpp, which roughly corresponds to a PSNR range of 27 to 37.\n We also observed from the recent works that as the compression rate increases, lossy compression approaches the realm of lossless compression, where there is limited potential for coding gain through improved predictions, as the codec must encode the inherent noise (unpredictable) within the data [5]. This notion aligns with the modest enhancements observed in generative modeling, such as the slight improvement in negative log-likelihood as depicted in Figure 1 (a) of [3], or the minor bit-per-dim discrepancy in recent learned lossless image compression, as shown in Table 1 of [4].\nTo align with the trends observed in recent research, we concentrated our experiments within widely adopted ranges where significant improvements have been reported.\nHowever, we are committed to incorporating more bpp points for our experiments, particularly for SwinT and Minnen's hyperprior-based methods, in the camera-ready version of the paper. This will allow us to provide a comprehensive evaluation of these methods across a broader range of bpp values.\n\n\n[1] Cheng, Z., Sun, H., Takeuchi, M., & Katto, J. (2020). Learned image compression with discretized gaussian mixture likelihoods and attention modules. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 7939-7948).\n\n[2] He, D., Zheng, Y., Sun, B., Wang, Y., & Qin, H. (2021). Checkerboard context model for efficient learned image compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14771-14780).\n\n[3] Kingma, D., Salimans, T., Poole, B., & Ho, J. (2021). Variational diffusion models. Advances in neural information processing systems, 34, 21696-21707.\n\n[4] Berg, R. V. D., Gritsenko, A. A., Dehghani, M., Sønderby, C. K., & Salimans, T. (2020). Idf++: Analyzing and improving integer discrete flows for lossless compression. arXiv preprint arXiv:2006.12459.\n\n[5] Zhu, Y., Yang, Y., & Cohen, T. (2021, October). Transformer-based transform coding. In International Conference on Learning Representations.\n\n> Experiments on factorized Ball'e method [1] and HP+AR+correlation loss can be provided to makes the experiments and evaluation more complete.\n\nAs recommended by the reviewer, we will incorporate Balle's Factorized prior [1] and Balle's Hyperprior (HP) [2], along with HP + Correlation Loss and HP + AR, in the camera-ready version of our paper. This addition will provide a comprehensive analysis and further insights into our proposed approach.\n\n[1] Ballé, Johannes, Valero Laparra, and Eero P. Simoncelli. \"End-to-end optimized image compression.\" arXiv preprint arXiv:1611.01704 (2016).\n\n[2] Ballé, Johannes, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. \"Variational image compression with a scale hyperprior.\" arXiv preprint arXiv:1802.01436 (2018).\n\n> Some experiments in the supplementary materials are kind of important and can be put into the main paper.\n\n> Some citations are still preprint version (e.g. [1][2][3] in reference). Their officially published version should be cited.\n\n> The paper writing can be improved. The citation format in the article looks messy. For equations, Some are ‘eq’, but some are ‘Equation.’\n\nWe thank the reviewer for the valuable comments, we shall fix these errors in the camera-ready version of the paper.\n\n"}, {"review_id": "o3L8DgERoC", "reviewer": "Reviewer_qpdX", "summary": "This paper aims at providing a efficient and effective entropy model to achieve better trade-off between performance and complexity for learned image compression. They introduce a correlation loss to force the latent to be spatially decorrelated so that it can fit the independent probability model better.", "questions": "No", "limitations": "see the weakness part", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. While most existing methods primarily focus on the context model, this paper makes a try at decorrelating latents with the proposed correlation loss. The correlation loss can act as a general plug-in for the hyperprior-based methods, which is flexible.\n\n2. It is proved that when the proposed method is combined with the checkerboard method,  the performance gain is about 85% compared with the auto-regressive model with only 1/50th inference time, which is efficient and effective.", "weaknesses": "1.  As shown in the RD curves, the BD-rate gains are obvious in the low bit-rate regime, however, there are less or even no gains in the high bit-rate regime. I think these results require further analysis, such as why it works better in the low bit-rate regime. It is one of the core problems that should be answered to help readers better understand the method. \n\n2.  It is just a little problem. There are many references of the figures and equations in the paper, but they are somewhat orderless. It is better to put the figures not so far away from their references.\n\n3.  I am somewhat confused about the analysis about Figure 4. It is better to provide the input image, which can help readers know the correspondence between the visualization and the original input.\n\n4.  I am not sure about the claim in line 307 \"The fourth column of Figure 4 shows that for our method, latent space has significantly reduced correlation compared to the baseline indicating the correlation loss’s efficacy\". I can only see that compared with Cheng's Hyperprior, the normalized latent of the proposed method has lower energy. I do not think it can provide evidence to show the proposed approach will reduce the correlation. Maybe it is because the proposed method just gives more accurate mean and variance. I think you can also provide the visualization of using auto-regressive context model but without the proposed approach. It may provide similar visualization results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and suggestions.\n\n> As shown in the RD curves, the BD-rate gains are obvious in the low bit-rate regime, however, there are less or even no gains in the high bit-rate regime. I think these results require further analysis, such as why it works better in the low bit-rate regime. It is one of the core problems that should be answered to help readers better understand the method.\n\nPlease refer to the general response for a detailed answer to this question\n\n> It is just a little problem. There are many references of the figures and equations in the paper, but they are somewhat orderless. It is better to put the figures not so far away from their references.\n\nWe appreciate the valuable feedback provided by the reviewer. We will address and correct these errors in the camera-ready version of the paper. Thank you for bringing these to our attention.\n\n> I am somewhat confused about the analysis about Figure 4. It is better to provide the input image, which can help readers know the correspondence between the visualization and the original input.\n\nPlease refer to the Figure 1 in the attached PDF, where we have included the input image for a better understanding of the correspondence between the visualization and the original input. We shall update all the Figures, including Figure 4 in the main manuscript and Figure 5 in the Supplementary, in the camera-ready version of our paper to ensure clarity and accuracy. Your feedback is greatly appreciated.\n\n> I am not sure about the claim in line 307 \"The fourth column of Figure 4 shows that for our method, latent space has significantly reduced correlation compared to the baseline indicating the correlation loss’s efficacy\". I can only see that compared with Cheng's Hyperprior, the normalized latent of the proposed method has lower energy. I do not think it can provide evidence to show the proposed approach will reduce the correlation. Maybe it is because the proposed method just gives more accurate mean and variance. I think you can also provide the visualization of using auto-regressive context model but without the proposed approach. It may provide similar visualization results.\n\nWe appreciate the reviewer's feedback regarding the interpretation of Figure 4 and the need for improved clarity, particularly concerning the representation of reduced correlation of the latent variable y in the last column of our figures. We have given thoughtful attention to this matter and have taken measures to enhance the clarity of the Figure, as demonstrated in the Figure 1 of the attached PDF. We aim to offer a more straightforward and comprehensive visualization of the effects of correlation loss. For a concise and coherent explanation, we kindly direct the reviewer to the general response section. \n"}]

Author rebuttal:
We thank all the reviewers for their insightful comments and suggestions.

# Additional Experiment Results

During the rebuttal period, we have performed additional experiments as shown in Figure 2 of the attached PDF, which will be the updated version of the Figure 1 of the main manuscript. The updated results include enhancements to the performance of the Checkerboard (CKBD) model with the integration of correlation loss, as well as the application of correlation loss to the ChARM model.
With the incorporation of correlation loss, the CKBD model now exhibits a BD rate gain of 16.5% when compared to Cheng’s Hyperprior (CH), representing around 90% of the performance achieved by a full AR model, all at approximately 1/50th of the computational cost.
Furthermore, the utilization of correlation loss with the ChARM model results in a significant BD rate gain of 18% over the baseline Cheng’s Hyperprior (CH). This gain corresponds to approximately 98% of the improvement obtained through the use of a full AR model. Notably, this progress is achieved with a considerably reduced computational cost, amounting to roughly 1/30th of the full AR cost.
Figure 2 in the attached PDF also serves to visually underline the implications of these findings. It demonstrates that a fixed encoder and decoder architecture lays down performance boundaries defined by the chosen entropy model. At one end, Cheng's Hyperprior (CH) signifies the lower performance limit, while at the other, Cheng's AR defines the upper threshold for given the CH's encoder-decoder architecture. With correlation loss incorporated into CH, significant improvements are attained. Note that CH + CKBD and CH + ChARM offer performance gains over CH closer to the upper limit. Consequently, the room for enhancement by adding correlation loss in these cases is relatively constrained when compared to the CH model. Nonetheless, applying correlation loss to these models leads to better approximations of full AR performance.

# General Response for Common Comments:

## Analysis on less gains in the high bit-rate regime is required. {Reviewers: 7bvo, qpdX}

One limitation of our proposed method, as mentioned in Section 5 of the main manuscript, is that the efficacy of the proposed loss function tends to be less pronounced at higher bit rates, despite the considerable performance gains at lower bitrates. This behavior can be attributed to the characteristics of learned image compression (LIC) models, which exhibit higher correlation among latent variables at lower bit-per-pixel (bpp) values compared to higher bpp values, as reported by Zhu et al. [1] and also evident from the correlation maps in Figure 4 of the supplementary material. 
We conducted a comprehensive analysis and presented the findings in Figure 3 of the attached PDF. This Figure showcases the relationship between PSNR gains, bpp, and correlation for different models, including Cheng's Hyperprior (CH), CH + CKBD, SwinT Hyperprior, and Minnen's Hyperprior. From the graphs in the Figure 3, a clear pattern can be observed: as the bpp decreases, the correlation of the latents increases, resulting in a higher gain in PSNR. However, as we move towards higher bpps, the correlation becomes notably reduced, resulting in decreased PSNR gain. This trend explains the reason why the efficacy of the correlation loss is larger in low bpp range and diminish at high bpp range.

[1] Transformer-based transform coding, ICLR. 2021.


## The visualization of Figure 4 does not explain the impact of Correlation loss. {Reviewers: xepj,7bvo, qpdX}

The reviewers have pointed out that the last column in all our Figures (Figure 4 in the main manuscript and Figure 5 in the supplementary material) may not clearly illustrate the reduced correlation of the latent variable y. While we acknowledge this limitation, the bpp values and correlation maps in Figure 4 (Supplementary material) provide compelling evidence of the reduced correlation in the latent space achieved through the application of correlation loss. To address this concern and provide a better visual demonstration, we have included the correlation map and the corresponding total correlation value of the latent space in the Figure 1 of attached PDF.
The Figure 1 illustrates that the improved mean μ and scale σ effectively capture image structures with notable superiority, incurring a minimal extra cost of 0.001 bpp. Similarly, the latent variable y undergoes a reduction of around 0.002 bpp, stemming from a correlation decrease of roughly 3.5 times. These combined enhancements manifest in a PSNR gain of approximately 0.38 dB when compared to the baseline, all while ensuring a total bpp that remains lower than the baseline.

## Performance on ELIC: Efficient Learned Image Compression: {Reviewer: xepj, 7bvo}

We conducted experiments on ELIC (using the unofficial implementation available on Github), investigating its performance both with and without the integration of the correlation loss. A comparison between ELIC and Cheng's Hyperprior revealed a significant BD rate gain of 28.85% for ELIC. Intriguingly, when the correlation loss was introduced to ELIC, the BD rate gain was further elevated to approximately 30.36%.
It is worth highlighting that the complete inference process for the ELIC on Kodak dataset took approximately 17.77 seconds which is about 1/15th of the total inference time of full AR. These findings strongly underscore the effectiveness of our proposed approach, showcasing its potential to achieve enhanced image compression performance.


### If given the opportunity to revise our original manuscript, we will update the Figures and text to clearly convey our new results and findings in this rebuttal and improve the overall clarity of our message. We appreciate the reviewers' feedback and are committed to presenting our research in the best possible way in the final version of the paper.

Meta-review: The paper proposes a new loss to improve neural image compression models. The paper shows that the loss improves rate-distortion performance for several popular models. 

After the rebuttal, all reviewers recommend to accept the paper. The reviewers agree that the paper provides a simple and effective idea for improving image compression models. The reviewers also agree that the idea is sufficiently validated and shown to be effective for improving image compression models. 

The reviewers also identified minor issues that were for the most part addressed by the rebuttals of the authors.
Decision: Accept (poster)