Diversify, Contextualize, and Adapt: Efficient Entropy Modeling for Neural Image Codec
Authors: Jun-Hyuk Kim
Seungeon Kim
Won-Hee Lee
Dokwan Oh
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural codec
entropy model
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: We propose a fast and effective entropy model for neural image codec by efficiently leveraging sufficient contextual information.

Abstract: Designing a fast and effective entropy model is challenging but essential for practical application of neural codecs. Beyond spatial autoregressive entropy models, more efficient backward adaptation-based entropy models have been recently developed. They not only reduce decoding time by using smaller number of modeling steps but also maintain or even improve rate--distortion performance by leveraging more diverse contexts for backward adaptation. Despite their significant progress, we argue that their performance has been limited by the simple adoption of the design convention for forward adaptation: using only a single type of hyper latent representation, which does not provide sufficient contextual information, especially in the first modeling step. In this paper, we propose a simple yet effective entropy modeling framework that leverages sufficient contexts for forward adaptation without compromising on bit-rate. Specifically, we introduce a strategy of diversifying hyper latent representations for forward adaptation, i.e., using two additional types of contexts along with the existing single type of context. In addition, we present a method to effectively use the diverse contexts for contextualizing the current elements to be encoded/decoded. By addressing the limitation of the previous approach, our proposed framework leads to significant performance improvements. Experimental results on popular datasets show that our proposed framework consistently improves rate-distortion performance across various bit-rate regions, e.g., 3.73\% BD-rate gain over the state-of-the-art baseline on the Kodak dataset.

Diversify, Contextualize, and Adapt: Efficient Entropy Modeling for Neural Image Codec
Jun-Hyuk Kim" Seungeon Kim Won-Hee Lee Dokwan Oh
Samsung Advanced Institute of Technology (jh131.king se2. kim, why_wh lee, dokwan. oh) ahamsusmunngcc com
Abstract
Designing fast and effective entropy model is challenging but essential for prac- tical application of neural codees. Beyond spatial autoregressive entropy models, more efficient backward adaptation-based baaed entropy models have been recently de- veloped. They not only reduce decoding time by using smaller number of modeling steps but also maintain even improve rate distortiio performance by leveraging more diverse contexts for backward adaptation. Despite their significant progress, we argue that their performance has been limited by the simple adoption of the design convention for forward adaptation: using only a single type of hyper latent representation, which does not provide sufficient contextual information, especially in the first modeling step. In this paper, we propose simple yet effective entropy modeling framework that leverages sufficient contexts for forward adaptation with- out compromising on bit-rate. Specifically, we introduce strategy of diversifying hyper latent representations for forward adaptation, i.e., using two additional types of contexts along with the existing single type of context. In addition, we present method effectively use the diverse contexts for contextualizing the current elements to be encoded/decoded. By addressing the limitation of the previous approach, our proposed framework leads to signíficant performance improvements. Experimental results on popular datasets show that our proposed framework con- sistently improves rate-de disto-rtoooonn performance across various bit-rate regions, e.g., 3.73% BD-rate gain over the state-of-ther baseline the Kodak dataset.
Introduction
Most neural image codecs [8, 9, 11, 15, 17, 18] first transform an image into quantized latent representation. It is then encoded into a bitstream via an entropy coding algorithm, which relies on leamee probability model known as the entropy model. According to the Shannon's 's source coding theorem, the minimum expected length of bitstream equal the entropy of the source. Thus, accurately modeling entropy of the quantized latent representation is crucial.
Entropy models estimate joint probability distribution over the elements of the quantized latent representation. Generally, it is assumed that all elements follow conditionally independent probability distributions. To satisfy this, the probability distributions are modeled in context-adaptive manners, which is key to accurate entropy modeling [18]. Recent methods are based on the joint backward and forward adaptation where the probability distributions adapt by leveraging contexts two different ways: directly using previously encoded/decoded elements (i.c., backward adaptation), and extracting and utilizing an additional hyper latent representation (i.c., forward adaptation). Here, the type of contexts leveraged can be diverse depending on the spatial range they cover. First, each element has dependencies with other elements in the same spatial location along the channel dimension. Since the whsn dependencies correspond to the local image area (e.g., a 16 x 16 patch), we denote
"ooresponning author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

them as the "local" context. Second, dependencies exist among spatially adjacent elements, and we refer them as the "regional" context. Lastly, long-range spatial dependencies span the entire image area, referred to as the "global" context.
For the backward adaptation, the modeling order, i.e., which elements modeled first, is an important factor, and the key lies in how effectively we can utilize diverse contexts in the modeling process. Early studies employ spatial autoregressive (AR) models that access regional context including the most spatially adjacent elements. However, they suffer from significantly slow decoding times due to the inevitably large number of modeling steps, which is equal to the spatial dimensions [18]. To enhance efficiency in entropy modeling, several attempts reduce the number of modeling steps while leveraging diverse contexts: a 10-step channel-wiss AR model [17]. 2-step spatial non-AR model with checkerboard pattern [8], and a 4-step non-AR model that operates across spatial and channel dimensions using quadtree partition [14].
Entropy models based on the efficient backward adaptation methods have led significant im- provements, However, they are still limited in fully leveraging contexts for forward adaptation. Since they use multiple neural layers with down- sampling and upsampling for modeling hyper latent representation, they can only access the regional context. This limits the performance improvement due to the insufficient contexts (Figure la). particular, this limitation is exac- erbated at the first step where only forward adap- tation is utilized due to the absence of previous elements (Figure 6). Therefore, it is necessary to develop effective forward adaptation in synergy with the efficient backward adaptation.
(a) Previous approach
(Rogiend)
Adauiie devebeatic
Parriczas elemenes
Araal dosilation
Contextualize
(b) Diversify, Contextualize, and Adapt (DCA)
Diversify
ttrp lacst reprrscaaiion
(Rogival, Giived Toal)
Atagtion
Acual
In this paper, we propose a simple yet effec- tive entropy modeling framework, called DCA (Diversify, Çontextualize, and Adapt), leverag- ing sufficient contexts for forward adaptation without compromising on bit-rate (Figure lb). Building the quadtree partition-based back- ward adaptation [14], we introduce a strategy of diversification, i.e., extracting local, regional, and global hyper latent representations unlike only single regional one in the previous ap- proach. Note that simply using more contexts for forward adaptation does not guarantee performance improvements because forward adaptation requires additional bit allocation unlike backward adap- tation. Then, we propose how to effectively utilize the diverse contexts along with the previously modeled elements for contextualizing the current elements be encoded/decoded. To consider step- wise different situations, e.g., increased number of previous elements over steps, our contextualization method is designed utilize each hyper latent representation separately in a step-adaptive manner. Additionally, our contextualization method proceeds in the sequence of regional, global, and local hyper latent representations. Similarly to backward adaptation, empirically observe that modeling order also matters in forward adaptation.
disibottie
devibatio
Contextualize
Figure 1: DCA diversifies the hyper latent repre- sentations and contextualizes the current elements by leveraging the diverse hyper latent representa- tions along with the previous elements. As a result, the probability distributions adapt effectively, lead- ing to accurate entropy modeling.
Our main contributions are summarized as follows:
We propose a strategy of diversifying contexts for forward adaptation by extracting three different hyper latent representations, i.e., local, regional, and global ones. This strategy can provide sufficient contexts for forward adaptation without compromising on bit-rate.
We introduce how effectively leverage the diverse contexts, i.e., previously modeled elements and the three hyper latent representations. We empirically show that the modeling order of three types of contexts affects the performance.
Through the diversification and contextualization methods, our DCA effectively adapts, resulting in significant performance improvements. For example, DCA achieves 3.73% BD-rate gain over the state-of-the the method [14] on the Kodak dataset.

---

2 Related work
Joint backward and forward adaptation. Ballé al. [3] propose scale hyperprior for forward adaptation. A hyper latent representation is extracted and utilized for inferring local scale parameters of the parameterized entropy model. Minnen et al. [18] extend the hyperprior model by using an additional mean hyperprior, and introduce joint backward and forward adaptation by combining the extended hyperprior model with a spatial autoregressive (AR) model. A patch matching based non-local referring model [20] and multi- Meali-head inteention-basen global hyperprior [11] are proposed to enrich contexts for backward and forward adaptation, respectively.
Efficient backward adaptation. To address the slow decoding times of spatial AR-based entropy models, several studies have proposed group-wise backward adaptation methods. They first divide the quantized latent representation into multiple groups and then process them in group- -wse manner, resulting in improved efficiency. He et al. [8] propose dividing the quantized latent representation into two groups using the checkerboard pattern, which is further improved by incorporating Transformer- based modules [21]. While they apply a group-wise modeling in spatial dimension, Minnen and Singh [17] introduce a channel-wise AR model that divides the quantized latent representation into ten groups along channel dimension. Some studies [22, 23] improve this model by applying Swin Transformer [16]. Based on the channel-wise AR model, He et al. [9] optimize the channel division and combine it with the checkerboard- partition- backward adaptation that divides the quantized latent representation into four groups considering both channel and spatial dimensions.
model. Recently, Li et al. [14] propose quadtree
In this paper, we propose a novel fast and effective entropy model that achieves better rate distortion performance by diversifying not only the quantized latent representation but also the hyper latent representations for backward and forward adaptation, respectively.
3 Methods
We provide an overview of the proposed methods in Figure 2. The analysis transform fa(-) and the synthesis transform f.(-) (-) (the gray blocks in Figure 2) are learned to find an effective mapping between an input image and quantized latent representation ÿ. i.e., ÿ = [ax]] and a (U). where [-1 is a round operation and à is the decoded image. For the analysis and synthesis transforms, we adopt the same model structure as the ELIC-sm model [9] due to its efficiency.
All other components are learned to model a prior probability distribution on the quantized latent representation ÿ. i.e., the entropy model Pý- The learned entropy model is utilized in the process of entropy coding, for which we employ the asymmetric numeral systems [6]. Here, our goal is to design fast and effective learned entropy model.
Quadtree partition. We build our entropy model on the joint forward and backward adapta- tion where the quadtree partition is used, which is formulated as follows [14]]
Chee Paeettonn
W
p(0) - -())= - p(())
II p p
F
g
Figure 3: Example of the quadtree partition-based backward adaptation for ý E p4x4x320 For sim- plicity, channel dimensions are represented via different colors. ÿ' means the elements to be en- coded/decoded at the i-th step. For modeling the current elements ÿ² all the previous modeled el- ements ý are used. For example, the elements corresponding to the red arrow leverage diverse contexts including elements across different chan- nels at the same spatial location (local context de- noted as L) and spatially adjacent four elements of the same channel (regional context denoted as R).
where ý is the quantized latent representation, is the quantized regional hyper latent repre- sentation, ğ' is the elements to be modeled at the i-th step, and ğxt is all the previous mod- eled elements before the i-th step. At each step, one-fourth of the total elements are modeled. The method partitions the quantized latent rep- resentation into four groups along the channel dimension, and then divides each group into non-overlapping 2x2 patches along the spatial dimension. The entropy modeling proceeds over
3

---

Rc
Paich spli
Mane
Swint Block
(x
Panch Merze
Marge
#x
#Ic x2
Purch Sptit
Swart Block
Swint Blook
Swinl Block
Loared
Quantiration
Mix W
Putch Syli
5k
Swant Block (x.5)
Eomeotartiion
i's of xac
k.
Putch Syli
Thare Traoxfomec foens:
33t Coay
Leaxttal
3xt Cony
Leipyte
Blods
Blok
C
[x] Come
11 TC
HM 26
Mxx
Diversify (Sectioa 3.0
Contextualize (Section
Figure 2: Overview the neural image codec with the proposed entropy model, referred to as DCA.
DCA can be employed by any analysis and synthesis transforms f.(-) and fs(-). DCA is adaptive entropy model consisting two main stages: diversify (Section 3.1) and contextualize (Section 3.2). First, given the latent representation y, DCA extracts diverse hyper latent representations 21, &,, and Eg. and then encodes them into the bitstreams using learned factorized entropy models, which are omitted in this figure for simplicity. Second, contextualization proceeds over four steps. By using the three features Φ), Φ,, and Φ (from the three hyper latent representations, respectively) and all the previously encoded/decoded elements before the i-th step, i.e., 6e DCA contextualizes the current elements be encoded/decodedd i.e., y', and finally obtains adaptive distribution parameters μr' and for probability modeling. Using the learned adaptive probability model, the quantized latent representation encoded into bitstream.
four steps, with each step modeling different elements as shown in Figure This quadtree partition- based method uses diverse contexts for backward adaptation, capturing dependencies from both spatial and channel dimensions.
Motivation. Recent studies to efficient modeling of backward adaptation have made significant advancements in terms of optimizing the rate pntc-distotion-comppte gap between their assumptions the probability modeling and actual data, leaving room for further performance enhancement. The assumptions are as follows: 1) All elements of & are independent; 2) All elements ý' are conditionally independent given ÿ and Here, the more the actual data deviates from the assumptions, the lower the accuracy of the entropy modeling. At the first modeling step, the elements are modeled conditioned only on the quantized hyper latent representation, i.e.. p()")2) resulting in a hyperprior model known for deviating from to the second assumption [18]. This can more problematic because the state-of-the-as methods process relatively large number of elements at the first step in order to complete the overall modeling with minimal number of steps. We also empirically show that this problem actually occurs in Figure 6.
trade-off: however, there is still
One straightforward solution is to increase the number of steps so that fewer elements are modeled in the first step. However, this leads slower modeling speeds, which conflict with the goal of our paper, i.e., developing fast and effective entropy model. Another simple approach is to provide more quantized regional hyper latent representation when modeling the quantized latent representation ÿ' However, paradoxically, this approach can introduce another issue due to the first assumption. Since elements of hyper latent representation are the same type of information (i.e., regional context), there is relatively high likelihood of dependencies among the elements. Therefore, to meet both assumptions, the newly added hyper latent representation is required be independent from the existing regional hyper latent representation. This is why proposed diversification method using

---

local, regional, and global hyper latent representations is needed. In this paper, we propose a fast and effective entropy model, called DCA, which consists of three main stages: diversifying the hyper latent representations, contextualizing the elements targeted for probability modeling, and ultimately adapting the probability distribution of the elements to the given contexts.
3.1 Diversify
The proposed DCA aims to diversify the information that the hyper latent representations contain. Specifically, given the latent representation y E R xd"xxxC xxC, where H. W, and C are the height, width, and the number of channels, respectively, DCA extracts three different types of hyper latent representations depending the range they cover: a local hyper latent representation &ạ R #xx xC; a regional hyper latent representation ž, E IR xC-, and a global hyper latent representation The whole process is illustrated in the orange blocks of Figure 2.
2g E R Nx &
Local context. To model remaining dependencies along channel dimension at each spatial location, which correspond to a 16 x 16 local patch in the image domain, we introduce local hyper analysis and synthesis transforms, Le (+) and (-), based on Swin Transformer (Swin'T) [16]. The local hyper analysis transform (-) analyzes local information in the latent representation, followed by the quantization operation to obtain a local hyper latent representation ε.. The local synthesis transform (+) synthesizes the local features Φb E &W x2C for contextualization from the local hyper latent representation 21.
Each transform proceeds in the order of Patch Split block, SwinT block, and Patch Merge block. The Patch Split block serves the function of shifting all channel- elements at each spatial location to 2 2 spatial resolution, consisting of the depth-to-space layer normalization, and linear layers in sequence. The SwinT block then captures dependencies between elements within each non-overlapping window of the input, producing an output of the same size as the input. By setting the window size to 2 x 2 in conjunction with the use of the Split block, we enforce the local hyper transforms to focus only on the local image area. The Patch Merge block performs the opposite function of the Patch Split block, containing the layer normalization, linear, and space-to-depth layers in sequence.
Regional context. While the receptive field of the local hyper transforms is limited to the local image area (i.e., 16 x 16 patches), regional hyper analysis transform r-(-) and regional hyper synthesis transform r, (-) model remaining dependencies between elements distributed across a relatively wide image area. The regional hyper analysis transform (-)-) analyzes regional information in the latent representation and yields a regional hyper latent representation 2, after quantization. From the extracted regional hyper latent representation Σ,, the regional synthesis transform r-(() generates the for contextualization.
regional features Φr R XXXWx2C
To do this, we stack multiple layers with the downsampling and upsampling operations for the regional hyper analysis and synthesis transforms, respectively. We adopt the same structure the previous work [22], which is based on SwinT. Specifically, the regional hyper analysis transform r.(:) conducts a Patch Merge block, five SwinT blocks, Patch Merge block, and SwinT block. The regional hyper synthesis transform (.) is constructed in the opposite order of the hyper analysis transform ra (-), using the Patch Split block instead of the Patch Merge block.
Global context. Lastly, to capture remaining dependencies between elements across the whole image area, we construct global hyper analysis and synthesis transforms ga (.) and g-(-) by adopting model structure of the global hyperprior model of Informer [11]. The global hyper analysis transform (.(+) extracts globally abstracted information from the latent representation using Transformer block with cross-attentin and a X 1 convolutional layer. After quantization, it obtains global hyper latent representation ky. Using 1 x convolutional layer, the global synthesis transform go (-) infers the global features O, E - x2C for contextualization.
3.2 Contextualize
Diverse contexts, i.e., previously modeled elements (i.e., أوو and hyper latent representations (i.e., é,, é,, and 2g) can be used for adapting probability distributions. Here, an important research question
5

---

emerges: How can we effectively leverage the diverse contexts? First, to consider step-wise varied situations, e.g., increased previously encoded/decoded elements over modeling steps, we propose a step-adaptive utilization of the three hyper latent representations. In other words, instead of applying a combined set of the three hyper latent representations, each hyper latent representation is adaptively leveraged at each step. In addition, we use regional, global, and local information sequentially. We argue that modeling order is crucial for forward adaptation, which is already known to be a key factor for backward adaptation.
The green part of Figure 2 illlustrates our proposed contextualization model c(-) at the i-th step. First, the previously modeled ÿs and the regional feature Φ, are combined based on the same structure as in the previous approach [14], consisting of concatenation, a 1 X convolutional layer, and three ConthConv Blocks. The DepthConv Block employs depth-wise wise separable convolutional layers for more efficient implementation. Second, the global feature Øg combined with the output of the last DepthCony Block using the Transformer block with cross-attention [11]. Finally, we combine the output of the Transformer block with the local feature Φi using concatenation followed by three x convolutional layers. yielding the distribution parameters Je and o', To make our contextualization model e(-) more efficient in terms of the number of parameters, all layers share weights across the four steps except for the initial x I convolutional layer.
Discussion on modeling order. According to the study on theoretical understanding of masked autoencoder via hierarchical latent variable models, the semantic level of the learned representation varies with the masking ratio [12]. Specifically, extremely large or small masking ratios lead to low-level detailed information such as texture, while non-extreme masking ratios result in high- level semantic information. Inspired by this, we can infer that the local and global hyper latent representations correspond to relatively low-level information because they extracted via limited utilization of the latent representation. The receptive field of the local hyper latent representation is limited by 1 1 convolutional layers. While the receptive field of the global hyper latent representation is whole image area, its attention mechanism selectively use the latent representation. Through the same reasoning, we can infer that regional hyper latent representation corresponds to relatively high-level information. Since different type of contexts has different characteristics, we argue that the modeling order is important for effective entropy modeling. In Figure 11, we empirically confirm that modeling higher-level information (i.e., regional context) first and lower-level information (i.e., local and global contexts) later is more effective than the opposite. In addition, the order between global and local is shown to be not influential.
3.3 Adapt
We design an adaptive entropy model on the quantized latent representation ý where each element is assumed to follow the Gaussian distribution, and each distribution parameters are obtained from the previous diversification and contextualization stages. Following the previous works [3, 18], we formulate our entropy model as follows:
Ps (0) = (V + +u
+ 2) (0:),
where b4 and a; are the mean and scale of the Gaussian distribution for each element Di. respectively.
The transforms and entropy model are jointly trained in an end-to-end manner by minimizing the expected length of the bitstream (rate) and the expected distortion between the original image and the decoded image, d(-,-). When a learned entropy model precisely matches the actual probability distribution, the entropy coding algorithm achieves the minimum rate. Therefore, we minimize the cross-entropy between the two distributions. We use mean squared error (MSE) for measuring image distortion. The objective function for our method is as follows:
(2) where Pr is the distribution of the training dataset, the entropy models Pε., Ps.,, and p2, are the non-parametric fully factorized entropy models [2], and À is the Lagrange multiplier that determines weighting between rate and distortion. As the value increases, a model is trained in a direction that reduces information loss, and consequently leads to higher bit-rate.
C = Exrps [- logz Py (0) logz Ps, (2)) - -log Ps, (2.) - log, Ps, (2g) + A ((xx)))]

---

38-
36
PSNR (4.)
PENR (d.)
32
(Minnco
30
(ChARMI (Minnen Singh, 2028
(ux) (MA) Ikmm
(ChARM0 + 1R0 (Mimnen Singh, 2028)
(ut) Isum et 2002
(Quadtrec) F 180 et 2023)
(Quadtree) F (RI 2025)
28
(Quddrre)) P USGLI (Durs, PCA)
(qudtreee + LSGLI (Durs, BCAI
0.2
0.4
0.6
0.8
0.1
0.2
0.3
0.4 rattrate (bpp)
0.5
0.6
Bit-rate (bpp?
(a)
(b)
Figure Performance comparison with latest entropy models on the two benchmark datasets: (a) Kodak and (b) Tecnick. For clear comparisons, we denote each method follows. B and F mean backward and forward adaptation, respectively, and the corresponding methods are written in parentheses. For backward adaptation, AR, ChARM, and Quadtree represent spatial autoregressive model, channel-wis -wise autoregressive model, and qaudtree partition-based model, respectively. For forward adaptation, L, R, G mean local, regional, and global hyper latent representations, respectively.
4 Experiments
We use a PyTTrch [19] based open-source library and evaluation platform, CompressAl [4], which has been widely used for developing and evaluating neural image codecs.
Training. We set model parameters as follows: C = 320, C = 10, C, = 192, and N = 8. We train our models corresponding six different bit-rates. We use 300,000 images randomly sampled from the OpenImages [13] dataset. We construct a batch size of 16 with 256 256 patches randomly cropped from different training images. All models are trained for 100 epochs using the Adam optimizer. The learning rate is set to 10T up to 90 epoch, and then decreases to 10-5, We use PyTorch v1.9.0, CUDA v11.1, CuDNN v8.0.5, and all experiments are conducted using a single NVIDIA A100 GPU.
Evaluation. We evaluate our method on the two popular datasets: Kodak [7] and Tecnick [1]. The Kodak dataset consists of 24 images with resolution either 768x512 512x pixels. The Tecnick dataset is composed of 100 images with a resolution of x200 1200 pixels. We evaluate our method in terms of rate-distortion performance. For this, we calculate the bits per pixel (bpp) after the encoding phase, and measure distortion between the decoded image and the original image using the peak signal-to-noise ratio (PSNR).
4.1 Comparison with slate-of-thh-are methods
We compare the proposed entropy model, DCA, with state-of-the-art entropy models. DCA can be combined with any transforms; in this paper, DCA is implemented with transforms that have the same structure as ELIC-sm [9]. For the comparison, we further train image compression methods with four different entropy models [11, 14, 17, 18]. For a fair comparison, they are also implemented with transforms that have the same structure in ELIC-sm [9].
Figures 4a and 4b show the rate -ite-distottion performance the Kodak and Tecnick datasets, respectively. The proposed DCA consistently achieves the best rate dist--ritoiionn performance across all bit-rate regions and two benchmark datasets. Specifically, DCA achieves 11 1.96% average rate savings over VTM-12.1 on the Kodak dataset, while the second (Lie et al. 2023) [14] and third best (Minnen & Singh, 2020) [17] methods obtain 8.55% and 4.86%, respectively.
Rate-Distortion.
Complexity. We also evaluate DCA in terms of efficiency. To this end, provide the de- coding time, the number of model parameters, and Bjøntegaard delta rate (BD-rate) [5] in Fig- ure 5. Decoding time is measured the Kodak dataset using single NVIDIA V100 GPU.
7

---

Table I: Performance comparison with state-of-the-an art entropy models on Kodak.
Methods
BD-rate (%) 4
Decoding time (ms) +
# Parameters (M) 4
Baseline (CVPR'23) [14]
0.00
67.05
32.64
LIC-TCM (CVPR'23 23) [15]
-0.72
139. 04
55.19
MLIC+ (ICMLW"23) [10]
5.76
242.61
107.80
DCA (Ours)
-3.73
82.05
37.89
BD-rate means the average bit-rate savings com- pared to baseline while maintaining the same quality of decoded images. We set VTM--2.1 as the baseline, calculate BD-rate for each im- age in the Kodak dataset, and average them. As shown in Figure 5, our DCA achieves better rale-distotiion-com computaiion trade-off than the AR models [11, 18] and the ChARM model [17]. Even compared to the quadtree- based entropy model [14], DCA improves performance signif- icantly, i.e., 3.73% BD-rate gain, without com- promising efficiency as much as possible.
(AR)) F (R) (Minnen et al, 2018)
(AR) (CL) (Kim 2022)
(ChARM) + F (R) (Mimen & Singh. 2020:
32.6
(Quadtree)
al. 20231
(Quadtree)
(Ours, DCA)
-0-
BD-cute (K)
-6
53:3 M
-4
0-
26,0 M
0.00
0.05
0.10
0.15
Decoding
Using the same structure of transforms, we ad- ditionally compare DCA with two state-of-the- art entropy models (Table 1), which shows that DCA improves performance most efficiently. It worth noting that performance improvements are significantly difficult to achieve when there constraints compute and memory usage, and this is the achievement our DCA.
Figure 5: Performance comparison with latest en- tropy models on the Kodak dataset in terms of decoding time, BD-rate, and model size. Decoding time is measured on NVIDIA V100 GPU. BD- rate means average rate savings over VTM-12.1. The size of the circle is determined proportionally the number of model parameters, and the spe- cific numbers are written to the left of the circles.
e (64.77
E 6.00.820
Ee[ 6.31
5
02
02
02
02
Xodim23
eLi al.
-00
400
000
00
-02
-02
5'e(-6.5
5 [- 6.19. 7.20)
[-
6.77. 7.57)
f' [e([4445,4.0
DCA (Oun)
0.
000
-02
-02
-02
-02
Figure 6: Illustration of normalized latent representations y' across four steps using both the baseline and proposed DCA. Each sub-figure includes the minimum and maximum values normalized latent representations. Notably, the baseline exhibits broader range of values the first modeling step, resulting in a higher bit-rate. More examples are provided in the appendix.
Probability modeling. To further show the role of the proposed DCA, we measure the normalized latent representation for each step, i.c., y² = (yy =²'//"' provides standardized measure of how far the latent representation deviates from the predicted mean in terms of estimated standard deviations. Smaller values indicate that learned entropy model estimates the true probability distribution more accurately. In Figure 6, we compare the result with that of the baseline model [14], which does not leverage diverse contexts for forward adaptation. Here, we have an interesting observation: While the existing work shows significantly high values at the first modeling step, DCA demonstrates consistent modeling performance across four steps. At the first step, since only forward adaptation possible, the previous approach can utilize only limited amount of context. On the other hand, DCA enables sufficient contextualization through diverse hyper latent representations, addressing the limitation of previous approach, which has difficulty effectively adapting various situations.

---

Original
DCA
VTM-12.1
(Bit-rate, PSNR)
(0.062 bpp. 32.552 dB)
(0.062 bpp. 31,470 dB)
Figure 7: Qualitative comparison of the decoded images by the proposed DCA and VTM- 12.1.
R+
+R0C+L
Large
G+
B (U90)
(URG) FLRRG)
12.40
37.28
7500
52.3
23.375
32.30
PSNN (60)
20.77
32.850
37.00
22.2
32.87
0.30
0.51
5.06
0.302
0.06 0.000
0.000
8i:/oto Bgp))
Dilisko Oggo
BA Bppt
Figure 8: Analysis of forward context diversity. R. G. L denote the regional, global, and local forward contexts, respectively. "Large R" means using larger amount of the regional context.
Figure 9: Analysis of backward context diversity. B and F mean backward and forward contexts, respectively. L, R, G denote the local, regional, and global contexts, respectively.
Qualitative results. We provide visual results in Figure 7, showing the decoded image from DCA has better visual quality and a higher PSNR value than that VTM 1--12,,, under the same bit-rate.
4.2
Model analysis
We conduct detailed analyses of DCA. To do this, we train different variants of DCA depending on various aspects for analysis. All results are shown in two different bit-rate regions (Figures 8 to 12).
Analysis of diversification. To validate the effectiveness of diversifying contexts for forward adaptation, we compare three different methods depending on the context diversity (Figure 8): one using regional context ("R"), one using both regional and global contexts ("R + G"), and one using regional, global, and local contexts altogether ("R + G L"). We use two additional models: one using a larger regional context ("Large R"") and the other using both global and local contexts ("G + L")). The comparison among the first three methods show diversifying forward contexts effective in both bit-rate regions. Through the results showing that the "Large R" method does not contribute to performance improvement, we once again demonstrate the effectiveness of our diversification. The last one is decomposing regional context into global and local ones rather than diversifying, which is equal to simply adopting the forward adaptation of Informer [11]. The result shows the decomposing approach significantly decreases compression efficiency, and diversifying is more effective.
In addition, while our focus lies on diversifying forward context, someone might be curious about whether the context utilized for the quadtree-based backward adaptation is diverse enough. To verify this, we conduct experiments additionally extracting global information from the backward context. Figure 9 demonstrates that there is no distinguishable advantage between two: one simply using the quadtree-bar based method [14] for backward adaptation ("B (LR) F (LRG)") and the other using additional global information for backward adaptation ("B (LRG) + F (LRG)"). This implies that backward adaptation is favorable when focusing on local and regional contexts.
show the effectiveness our contextualization approach, we first compare two different methods in Figure 10. "R G (Step-independdnnt) combines regional and global information advance and utilizes the combined one regardless of the step. The other adaptively utilizes regional and global information separately for each step, "R G (Step-adaptive)".
Analysis contextualization.

---

(Stpp edaplasc
G-1--1 1-+5-8
35,050
35,860
22.38
32.925
po0n cm
32.39
23.54
12 32,875
35,000
12.54
35,900
52.22
32.30
36,873
32,050
0,310
0666
Biocato Bpg!
88.cata Upp?
Figure 10: Analysis of how contextualize. R and G denote the regional and global contexts, respectively. "Step-independent" first combines and and then utilizes them for all steps, while "Step-adaptive" does not pre-combine them and utilizes each separately for all steps.
Figure 11: Analysis of contextualization order. R, G, L denote the regional, global, and local for- ward contexts, respectively. + means the order. For example. the "R+G~>" method utilizes R, G, and sequentially.
As reference, we use the model utilizing only regional context, i.e., "R". The result shows that both effective and step-adaptive approach is beneficial for boosting performance.
In addition, we analyze the effectiveness of our modeling order for contextualization in Figure 11. Four different ordering methods and one reference method are used for the comparison. Our ordering approach ("R->G+>")) achieves the best rate- distorriooioon performance, and the methods are catego- rized into two groups based on the performance in both bit-rate regions. Models that prioritize the regional context ("R-GGsi and "R->-+'") show better performance compared to those that do prioritize it ("G-+L-R" and "L"+,G+RR).. We observe that the method using the opposite mod- eling order of the proposed sequence ("I.-G even exhibits performance decline compared to the baseline ("R") in lower bit-rate region.
Analysis of architecture. Applying attention mechanisms on various tasks is one of most actively researched topics. We analyze the effect of the architecture combination for forward and backward adaptation in DCA. Figure 12 compares three different methods: one without attention ("B (CNN) F (CNN)"), another applying attention only to forward adaptation ("B (CNN)+ + (Attnen- tion)"), and the last applying attention for both ("B (Attention) + F (Attention)"). The results show that CNN and attention are effective for forward and backward adaptation, respectively. We infer that focusing on local and regional information is preferable in backward adaptation: thus, a CNN with locality inductive bias may be more effective.
32:38
35,925
32.52
32.30
35,050
0.303
0.205
0.213
0.215
0.662
0:645
6:500 0.653 0.663
Sticate Bgg?
38-catx Bgp!
Figure 12: Analysis of model architecture for backward and forward adaptation. B and F mean backward and forward adaptation, respectively.
5 Conclusion
In this paper, proposed fast and effective entropy modeling framework, DCA, which diversifies forward contexts by extracting local, regional, and global information, and contextualizes current elements with the diverse forward and backward contexts. We demonstrated that our DCA improves rate-distortion performance significantly compared to previous approach without compromising efficiency as much possible. Furthermore, provided diverse insights into entropy modeling by conducting a comprehensive and in-depth analysis the design aspects of DCA.
Limitation and future works. To address the limitation of the state-of of-the-att art entropy models, we focused on paving a novel framework with diverse contexts rather than designing neural architec- tures. Therefore, DCA can be limited by architectural designs that are inspired by the existing works [11, 14, 22]. In the future, we expect that it would be further improved by neural architectures especially designed for the diverse contexts. In addition, is worth exploring alternative criteria for diversification beyond the spatial range the contexts covers (i.e., local, regional, and global contexts).
10

---

References
[1] N. Asuni and A. Giachetti. TESTIMAGES: large-scale archive for testing visual devices and basic image processing algorithms. Proceedings of the Smart Tools and Apps for Graphics (STAG), 2014.
[2] J. Ballé, V. Laparra, and E. P. Simoncelli. End-to-end optimized image compression. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
[3] J. Ballé, D. Minnen, S. Singh, J. Hwang, and N. Johnston. Variational image compression with scale hyperprior. In Proceedings of the International Conference on Learning Representutions (ICLR), 2018.
[4] J. Bégaint, Racapé, S. Feltman, and A. Pushparaja. CompressAI: a PyTorch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:201 1.03029, 2020.
[5] G. Bjontegaard. Calculation of average PSNR differences between RD-curves. VCEG-M33, 2001.
[6] J. Duda. Asymmetric numeral systems: entropy coding combining speed of Huffman coding with compres- sion rate of arithmetic coding, arXiv preprint arXiv:1311.2500, 2013.
[7] R. Franzen. Kodak lossless color image suite. http://fOk..a/grraahhsssktdaaad 1999.
[8] D. He, Y. Zheng, B. Sun, Y. Wang, and H. Qin. Checkerboard context model for efficient learned image compression. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[9] D. He. Yang. Peng, Ma. Qin, and Wang. ELIC: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[10] W. Jiang and R. Wang. MLIC++: Linear complexity multi reference entropy modeling for learned image compression. In Proceedings of the International Conference on Machine Learning (ICML) Workshop, 2023.
[11] J.HH. Kim, B. Heo, and J.SS. Lee. Joint global and local hierarchical priors for learned image compression. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[12] L. Kong, M. Q. Ma, G. Chen, E. P. Xing. Y. Chi, L.-P. Morency, and K. Zhang. Understanding masked autoencoders via hierarchical latent variable models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[13] I. Krasin, T. Duerig, N. Alldrin, V. Ferrari, S. Abu-El-Hlaja. A. Kuznetsova, H. Rom, Uijlings, S. Popov, A. Veit, S. Belongie, V. Gomes, A. Gupta, C. Sun, G. Chechik, D. Cai, Z. Feng, D. Narayanan, and K. Murphy, Openlmages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://githubb.. http://ghhhhb.ooooppnnnnnn////// com/openimages, 2017.
[14] J. Li, B. Li, and Y. Lu. Neural video compression with diverse contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[15] J. Liu, H. Sun, and J. Katto. Learned image compression with mixed Transformer-CNN architectures. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[16] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin Transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[17] D. Minnen and S. Singh. Channel-wise autoregressive entropy models for learned image compression. In Proceedings of the IEEE International Conference Image Processing (ICIP), 2020.
[18] D. Minnen, Ballé, and G. Toderici. Joint autoregressive and hicrarchical priors for learned image compression. Advances Neural Information Processing Systems (NeurIPS), 2018.
[19] A. Paszke, S. Gross, F. Massa, A. Lerer. J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein. L. Antiga, A. Desmaison, A. Kopf, E. Yang, DeVito, M. Raison, A. Tejani. Chilamkurthy, B. Steiner. L. Fang, J. Bai, and S. Chintala. PyTorch: An imperative style, high-performane deep learning library. In Advances in Neural Information Processing Systems (NeurIPS). 2019.
[20] Y. Qian, Z. Tan, X. Sun, M. Lin, D. Li, Z. Sun. L. Hao, and R. Jin. Learning accurate entropy model with global reference for image compression. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.

---

[21] Y. Qian, X. Sun. M. Lin, Z. Tan, and R. Jin. Entroformer: A transformer-based entropy model for learned image compression. In Proceedings of the International Conference Learning Representations (ICLR), 2022.
[22] Y. Zhu, Y. Yang, and T. Cohen. bassd transform coding. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
[23] R. Zou, C. Song, and Z. Zhang. The devil is in the details: Window-basee attention for image compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
12

---

A
Additional results
We provide the normalized latent representations of images from the Kodak dataset in Figure 13.
595.19
(-11007 821
[-5.00
-614 530
(Li 2023)
-02
e'c
(-6.37,6.5))
DCA (Oun)
5 [-5.8000..
(Li al.
-02
y'E|-511,630
DICA (Ours)
A PE2--20000 5.17
[-4.28, 2,53]
y'E|-B12,08
(1A 2023)
y'El 2/E1-9.25,00 (-9.2566744)
ccc 6.30
E 7.00
y'El 1-|-30 7.04,
DICA (Omns)
LL diatll 2023)
1-276,00, 27.88
2 7.00.0.6
5 (-6,22, L.0
5 8.20 4.57
52
02
82
7.80
c 6.05
9.00.4.91
DCA (Ous)
Figure 13: Illustrations of normalized latent representations D across four steps for Kodak images using both the baseline [14] and proposed DCA. Each sub-figure includes the minimum and maximum values of normalized latent representations. Notably, the baseline exhibits broader range of values at the first modeling step, resulting in a higher bit-rate.
provide an in-depth runtime analysis by sub-systems in Table
Table 2: Runtime (ms) of DCA. Total encoding/decoding time includes the ANS entropy coding.
Transforms
Entropy model (DCA)
Total
Sa
1,
Fa
9a
e
Encoding
Decoding
3.46
1.54
0.98
0.96
5.32
4.78
0.63
0.08
14.75
117.49
82.05
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We summarized our contributions in Section 1 and illustrated our method in Figure I in a compact manner.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
Our method is evaluated based on empirical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Figure 2 and Sections 3 and
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer:
[No]
Justification:
The code is proprietary.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. wwtppcccpp/aaeeecccccccccccceeee http//blccguuuds/////eececc
httaa//ggiieeeeeeeeuiiiieeeeee /Co////oddmssssseeeeeeeee soiiiissioooccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specifyy all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: See Section 4.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We evaluate method across six different bit-rate regions and method consistently achieves performance improvements.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: See Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the ines?
NeurIPS Code of Ethics https: httpp://///uui/s./t/// http::/twrp.sc.ppplii//e////////
Answer: [Yes]
Justification: Our work conforms the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We think our work is foundational research.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yss]
Justification: See Section 4.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19


Reviews:
[{"review_id": "DKgFquJMTQ", "reviewer": "Reviewer_hsbA", "summary": "This paper presents a method for learned image compression called DCA, which stands for \"Diversify, Contextualize, and Adapt\". DCA is a more sophisticated approach for building the entropy model compared to previous methods. The entropy model is a key component in compression models and is optimized to predict the distribution of quantized latents that represents an encoded image. This is crucial for compression since a more accurate entropy model directly leads to higher compression rates when used with an entropy coding algorithm.\n\nThe core insight of DCA is to use multiple (\"diverse\") contexts for forward adaption. More specifically, the authors define a local, regional, and global context corresponding to a single spatial location in the latent representation (local), a spatial patch in the latent tensor (regional) and a compact summary of the full latent (and thus the full image) learned via cross-attention. The authors show that the three contexts are complementary (Fig. 7) and that a serial applications (architecture details in Fig. 2) outperforms a merged approach (Fig. 9 and 10).\n\nThe result is a SOTA model in terms of compression rate with a moderate model size and relatively fast decoding time (see Fig. 5).", "questions": "No questions", "limitations": "Adequately addressed.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The primary strength of this paper is achieving a non-trivial rate savings over very strong existing image codecs: 11.96% average rate savings over VTM, the current best standard codec for lossy image and video compression, and 3.73% rate savings over a strong neural method [12]. This is achieved with a well-motivated, intuitive improvement over previous entropy models along with extensive experimentation and evaluation.\n\nFurthermore, the rate savings are achieved with a moderate-size model (37.9M) and with a relatively fast decode speed. This is import because ultimately neural compression models compete with standard codecs with extremely fast decode speeds, and because it's already clear that incremental improvements to compression rates can be easily achieved at the expense of massively more compute and memory usage.\n\nFinally, the evaluation in the paper is quite thorough in terms of comparisons with appropriate baselines, runtime analysis, and investigation of importance of the different contexts used, their order, and the architecture (CNNs vs. attention). I also appreciated that the authors showed *where* DCA helped (Fig. 6) by showing the difference in the scale of the normalized latents with and without DCA. Specifically, DCA (with a global context) shows much better prediction for the first step (\\bar{y}^1) where there is no opportunity for backward adaptation and the baseline only utilizes regional context.", "weaknesses": "The primary weakness of the paper is modest gains (at least in the context of NeurIPS) on a version of the learned compression problem that is no longer the focus of the compression subfield.\n\nIn terms of the rate gains, 3.73% over the previous SOTA method is non-trivial (for reference, papers at conferences focused on improvements to standard codecs often show a gain of less than 0.5%), but it's still incremental progress. It's also not a large enough change to garner more interest from practitioners developing new standard codecs.\n\nIn terms of the problem itself, there is now more attention on modeling video and boosting subjective (perceptual) quality. Optimization for MSE can still be important since it allows clear, objective evaluation, but then we're back to the first point where I think a larger improvement is now needed to have high impact at NeurIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1] Performance Improvements**\n\nTo explain the significant performance improvements, we clarify our goal $-$ To develop an **efficient** entropy model. As already mentioned by you, performance improvements can be easily achieved at the expenses of massively more compute and memory usage. However, for the image compression task, which is highly required to be efficient, it is very important to efficiently work. Therefore, we focused on studying how we can efficiently modeling entropy. In this respect, we would like to say that 3.73% BD-rate gain over the previous SOTA method is significant because it is achieved with only a small amount of additional computation and memory. \n\nFor a more in-depth analysis, we compared our method with two additional SOTA methods (Tab. 1 of the uploaded PDF). You can confirm that our method improves performance most efficiently $-$ DCA achives the best improvement over basesline (-3.73% BD-rate gain) with the most efficiency in terms of the decoding time and the number of parameters.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3Kruj9YT6B", "reviewer": "Reviewer_295Z", "summary": "**Disclaimer:** My research direction is not related to image compression. I will try my best to review this work but could be biased. \nThe work studies neural image codec. The work proposes an entropy modeling framework that uses contents for forward adaptation without comprising the bit rate. The authors build upon the quadtree partition-based backward adaptation method and introduce additional forward context to address limitations in existing approaches, particularly in the first modeling step.", "questions": "1. While the proposed method enjoys better performance, it seems the proposed method is largely derived from B (Quadtree) + F (R) (Li et al., 2023), while uses a lager network, and therefore, slightly longer decoding time. \n2. I think it will further improve the paper if the authors put some additional qualitative results say in the appendix for the readers.", "limitations": "NA", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work conducts in-depth analysis of the different components in the proposed method。 \n2. The proposed method enjoys very efficient decoding while having a better BD-rate.", "weaknesses": "- There are some presentation issue, e.g. line 256-257, the links to citation are missing.\n\nPlease see the questions for methodological discussion", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1] Presentation Issue** \n\nWe will revise it. Thank you for commenting. \n\n**[Q1] Difference from Baseline** \n\nFirst of all, we would like to say that the proposed entropy model, DCA, addresses the limitation of insufficient context for forward adaptation in the baseline (Li et al., 2023). We would like to emphasize that simply using more contexts for forward adpatation does not guarantee performance improvements because forward adaptation requires additional bit allocation unlike backward adaptation. For example, Fig. 9 of the previous work [10] shows using both regional and global contexts does not always helpful. In addition, in our paper, Fig. 7 shows just using more regional contexts (denoted as \"Large R\") does not lead to perfromance improvemnts compared to the baseline (denoted as \"R\"). Please note that our proposed method is very strong, consistently leading to significant performance improvements with a simple and intuitve idea. \n\n**[Q2] Additional Qualitative Results** \n\nWe provide additional results in Fig. 1 of the uploaded PDF. We will add it. Thank you for commenting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QSGQ0TzFxZ", "reviewer": "Reviewer_TpTZ", "summary": "This paper proposes an entropy model that achieves higher compression rates than previously proposed models in the context of learned image compression. This model combines global, regional, and local information to make better (i.e., more accurate, and therefore requiring fewer bits) predictions with respect to the latent values (quantized) that the autoencoder produces. The authors discuss the method in detail and provide a visualization. The results show an improvement over the existing methods..", "questions": "- Do you plan to open source your code? I am asking this because (a) it would allow researchers to use it (as it's clearly a useful component of any learned image compression pipeline); and (b) determine how well it works in practice when submitting to competitions such as CLIC which will almost entirely guarantee that the decoding will happen on different hardware than the encoding. \n- I appreciate the experiments with the ordering of contexts, but I am very puzzled by the final order. Regional -> Global -> Local sound quite counterintuitive. I wonder why Global is not actually first, as it would \"seem\" like it should be the most general, and a decrease in granularity with respect to ordering would make sense.\n- Since the goal of this paper was to achieve the absolute best RD, have you considered backpropping into the latents once an encoding has been found (after a forward pass) in order to see whether jointly optimizing the RD loss for the particular image would yield even larger improvements?", "limitations": "They seem addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper ran well designed experiments\n- The authors do discuss runtime, which is great. I wish more paper did this.\n- If I am reading the numbers correctly, the entropy model seems to provide a pretty good speedup over the existing models.", "weaknesses": "- This paper does not seem to be too theoretical, but it's clearly results-focused. I wonder whether a more appropriate venue would afford this a) more visibility to the correct (i.e., compression) audience; and b) be much more well received.\n\nI don't really think that this will be of broad enough interest at NeurIPS.\n\n- When discussing runtime performance, it would be great if the authors could break it down by sub-system. For example, it's useful to know how long does the autoencoder take to encode the image/decode the image. It would be great to know how much time does the entropy model take to predict all values needed to encode/decode. It would also be interesting to know (this is highly optional, but would make this paper much more interesting from a deployment standpoint) do discuss the runtime when ANS is also interleaved in the decoding / encoding process of the image.\n- The proposed entropy model shows some slight improvement, but it's nothing exciting. I was hoping to see a 30% improvement given the added complexity, but we got a much smaller number than that.\n\nThe biggest issue:\n- Since this paper is focused on entropy modeling, which I believe is a great topic, I would have thought that a primary concern would have been reproducibility. Entropy models are famously flaky, meaning that it's quite possible that an image encoded on a V100 GPU may not be decoded by any other device (including CPUs.). As such, I would have liked the authors to go into detail on what mitigations were put in place to ensure that their model can be used on variety of hardware. I truly believe, that this is the difference between a paper that slightly improves theoretical compression rates, and one which makes it possible to apply in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1] Venue**\n\nIt is well known that entropy models in neural image codecs are highly related to generative models. We sincerely believe that our work will be helpful not only to compression experts but also to many researchers in the NeurIPS conference.\n\n**[W2] Runtime Analysis**\n\nFollowing your advices, we added an in-depth runtime analysis in Tab. 2 of the uploaded PDF. You can see that runtime by sub-systems and total encoding/decoding time including the ANS entropy coding. \n \n\n**[W3] Performance Improvements**\n\nTo explain the significant performance improvements, we clarify our goal $-$ To develop an **efficient** entropy model. As already mentioned by you, performance improvements can be easily achieved at the expenses of massively more compute and memory usage. However, for the image compression task, which is highly required to be efficient, it is very important to efficiently work. Therefore, we focused on studying how we can efficiently modeling entropy. In this respect, we would like to say that 3.73% BD-rate gain over the previous SOTA method is significant because it is achieved with only a small amount of additional computation and memory. \n\nFor a more in-depth analysis, we compared our method with two additional SOTA methods (Tab. 1 of the uploaded PDF). You can confirm that our method improves performance most efficiently $-$ DCA achives the best improvement over basesline (-3.73% BD-rate gain) with the most efficiency in terms of the decoding time and the number of parameters. \n\n**[W4] Reproducibility**\n\nAlthough we deeply agree that the cross-platfrom robustness is cirtical for real-world applications, we kindly argue that the topic is beyond the scope of this paper focusing on efficient entropy modeling. It would be one of the interesting future works. Thanks for your thoughtful advice.\n\n\n**[Q1] Source Code**\n\nWe completely agree with you and plan to release the source code for the advancement of the NeurIPS community.\n\n**[Q2] Ordering of Context**\n\nAccording to the study on theoretical understanding of masked autoencoder via hierarchical latent variable models, the semantic level of the learned representation varies with the masking ratio [1]. Specifically, extremely large or small masking ratios lead to low-level detailed information such as texture, while non-extreme masking ratios result in high-level semantic information.\n\nInspired by this, we can infer that the local and global hyper latent representations correspond to relatively low-level information because they are extracted via limited utilization of the latent representation. The receptive field of the local hyper latent representation is limited by 1x1 convolutional layers. While the receptive field of the global hyper latent representation is whole image area, its attention mechanism selectively use the latent representation. Through the same reasoning, we would like to argue that regional hyper latent representation corresponds to relatively high-level information.\n\nIt sounds reasonable that modeling higher-level informaton first and lower-level information later is effective, because higher-level information can be helpful to lower-level information, while the opposite seems to be relatively difficult. The proposed modeling ordering is also intended to take that approach. We set the order to be regional (higher-level information), global (lower-level information), and local (lower-level information) contexts in sequence. Here, the order between global and local is empirically determined, which is shown to be not influetial in Fig. 10.\n\nThanks for your questions and we will add the above discussion.\n\n[1] Kong et al., \"Understanding masked autoencoders via hierarchical latent variable models,\" in CVPR 2023.\n\n**[Q3] Further Optimization**\n\nOur goal is to achieve not only RD performance improvements but also efficienty entropy modeling in terms of computation and memory usage. So, we haven't tried the test-time further optimization for best RD performance, but we believe it would be an interesting future topic.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kn3VHgZb3F", "reviewer": "Reviewer_YQWe", "summary": "The authors proposed a novel method to improve the entropy encoding in deep image compression. The key idea is to introduce three contexts, local, region, and global, to capture contextual information at different scales while balancing the compute cost. The authors show that the proposed method achieves SOTA compression rate while being computationally cheap.", "questions": "* Is the proposed model mobile-friendly?", "limitations": "N/A", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* While introducing global context in entropy model has been explored in previous, how to efficiently capture these context information with minimal compute cost is still an open challenge. This work proposed a novel solution to fill the gap.\n\n* Experiments on real-world high resolution images show significant improvements over baseline methods.", "weaknesses": "* There are quite a few deep image compression methods proposed recently. The authors only compared 4 baseline methods. It would be necessary to compare more STOA methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1] Additional SOTA Methods**\n\nFollowing your advices, we compare our method with six additonal SOTA methods. Two methods are employed in Tab. 1 of the uploaded PDF. For the experiment, we use the same structure of transforms (i.e., ELIC-sm) and different entropy models for fair comparison. The results clearly show that the DCA's superiority in terms of efficiency. Furthermore, four methods are compared in Fig. 2 of the uploaded PDF. \n\nWe believe this will improve the completion of our paper. Thank you. \n\n**[Q1] Mobile-friendly**\n\nThere can be two orthogonal research directions for efficient entropy modeling: 1) Developing an efficient model by novel algorithm and 2) Making existing entropy models more efficient by conducting QAT or PTQ as in the recent work [1].\n\nWe focused on the former direction, but the latter direction can be further studied in the future.  \n\n[1] \"MobileNVC: Real-time 1080p neural video compression on a mobile device,\"  in WACV 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their valuable comments.

**Strengths.**

The proposed entropy model was appreciated for the performance improvement over state-of-the-art baseline methods (all reviewers: hsbA, 295Z, TpTZ, YQWe) with the well-motivated and intuitive method (hsbA). The reviewers recognized that our experiemnts were well-designed (TpTZ), extensive (hsbA), and quite thorough with in-depth analysis (hsbA, 295Z). Efficient entropy modeling, the problem we address, was recognized as important and necessary (hsbA, TpTZ, YQWe), and we were appreciated for providing a novel solution (YQWe).

**Summary of Rebuttal.**

We address reviewers' concerns and questions by providing experiments and clarifications, which are summarized as follows:
- **Significant improvements** are explained by additional comparison with SOTA in terms of BD-rate and efficiency (Tab. 1 of the below PDF). $-$ hsbA, TpTZ
- **In-depth runtime analysis** is conducted (Tab. 2 of the uploaded PDF). $-$ TpTZ
- **Additional qualitative results** are provided (Fig. 1 of the uploaded PDF). $-$ 295Z
- **Additional SOTA methods** are employed for evaluation (Tab. 1 and Fig. 2 of the uploaded PDF). $-$ YQWe
- **Differences from baseline** are discussed. $-$ 295Z
- **Discussion on the ordering of context** is provided. $-$ TpTZ
- **Source code** will be released. $-$ TpTZ

Details are provided in each rebuttal below.

Meta-review: Author rebuttal addressed most of the reviewer concerns and all the reviewers unanimously voted for acceptance in their final rating. AC agrees with the reviewer recommendations and supports acceptance of the paper.
Decision: Accept (poster)