Once-for-All: Controllable Generative Image Compression with Dynamic Granularity Adaption

## Abstract

Abstract Although recent generative image compression methods have demonstrated impressive potential in optimizing the rate-distortion-perception trade-off, they still face the critical challenge of flexible rate adaption to diverse compression necessities and scenarios. To overcome this challenge, this paper proposes a Control lable G enerative I mage C ompression framework, Control-GIC , the first capable of fine-grained bitrate adaption across a broad spectrum while ensuring high-fidelity and generality compression. We base Control-GIC on a VQGAN framework representing an image as a sequence of variable-length codes ( i.e. VQ-indices), which can be losslessly compressed and exhibits a direct positive correlation with the bitrates. Therefore, drawing inspiration from the classical coding principle, we naturally correlate the information density of local image patches with their granular representations, to achieve dynamic adjustment of the code quantity following different granularity decisions. This implies we can flexibly determine a proper allocation of granularity for the patches to acquire desirable compression rates. We further develop a probabilistic conditional decoder that can trace back to historic encoded multi-granularity representations according to transmitted codes, and then reconstruct hierarchical granular features in the formalization of conditional probability, enabling more informative aggregation to improve reconstruction realism. Our experiments show that Control-GIC allows highly flexible and controllable bitrate adaption and even once compression on an entire dataset to fulfill constrained bitrate conditions. Experimental results demonstrate its superior performance over recent state-of-the-art methods.

## 1 Introduction

Figure: Figure 1: Our Control-GIC achieves controllable target bit rates in a single inference, making it easier and more practical to adapt to different bandwidth requirements than other variable bit rate methods. The right figure shows that our average LPIPS is much lower (see Ours-mean), and each image is well-controlled near the target rate than the MSE-based variable method SCR [].
Refer to caption: x1.png

Lossy image compression complies with the rate-distortion criterion in Shannon’s theorem [], which aims to pursue minimal storage of images without quality sacrifice. Traditional standardized codecs [, , ] adhere to a typical hand-crafted “transforming-quantization-entropy coding” rule, showing substantial performance on generic images. Learnable compression algorithms [, , , ] follow a similar pipeline that parameterizes it as convolutional neural networks (CNNs) operating on latent variables with end-to-end R-D optimization. Recent works [, , , ] leverage the generation capability of generative adversarial networks (GANs) [] to deal with the compression task, known as generative image compression, which minimizes the distribution divergence between original and reconstructed images, resulting in perfect perceptual quality. However, these methods train the models separately for specific R-D points with Lagrange multiplier ($\lambda$)-weighted R-D loss, each corresponding to an individual $\lambda$. In this way, multiple fixed-rate models are necessitated to vary bitrates, resulting in dramatic computational costs and inefficient deployment to cater to diverse bitrates and devices. Some CNN-based models propose to learn scalable bitreams [, , , , , ] or truncated quantization parameters to control the bitrates [, , , ]. On the one hand, these models typically support a restricted range of bitrates with substantial variance distribution, thereby constraining their adaptability to finer bitrate adjustments. On the other hand, they tend to quantify the distortion using the pixel-wise mean square error (MSE), which is inconsistent with human perception and often yields implausible reconstruction, particularly at low bitrates [, ]. There are several methods that introduce the scalable [] or variable-rate [] designs into generative models. While they have achieved remarkable improvements in perceptual quality, the models are still constrained by finite compression rates.

In the light of the preceding discussion, in this work, we propose an innovative generative image compression paradigm, dubbed Control-GIC, which accommodates highly flexible and fine-grained controls on a broad range of bitrates and perceptually realistic reconstruction with solely one set of optimized weights. Motivated by that VQ-based models [, ] enable to encode images into discrete codes that represent the local visual patterns, Control-GIC hybridizes the classical coding principle in the architecture with VQGAN to relax the typical R-D optimization and provide a controllable unified generative model. Specifically, Control-GIC first characterizes the inherent information density and context complexity of local image patches as the information entropy. We devise the granularity-informed encoder that determines the granularity of these patches based on their entropy values, which are represented by sequential variable-length codes (*i.e.*, VQ-indices) based on the learned codebook prior (see Figure 1). One can flexibly control the statistics of the granularities to adjust the VQ-indices of patches dynamically adapting to diverse desirable bitrates.
As correlated to the regional information of images, the VQ-indices are spatially variant to adapt to the local contents. We then develop a statistical entropy coding module, which is no-parametric and captures the code distribution in the learned codebook prior across a large-scale natural dataset to approximate a generalized probability distribution, encoding the VQ-indices losslessly and more compactly during inference, improving the compression efficiency. On top of entropy coding, a probabilistic conditional decoder is further developed, which formalizes the reconstruction of granular features in a conditional probability manner leveraging the historic encoded multi-granularity representations given entropy-decoded indices. Our comprehensive experimental results demonstrate the outstanding adaption capability of the proposed Control-GIC by achieving superior performance from perceptual quality, distortion quality, and non-reference quality perspectives compared with three types of recent state-of-the-art methods including generative, progressive, and variable-rate compression methods using only a single unified model.

The main contributions of this work are three-fold:

- •
We propose the Control-GIC, a unified generative compression model capable of variable bitrate adaption across a broad spectrum while preserving high-perceptual fidelity reconstruction. To our best knowledge, this is the first that allows highly flexible and controllable bitrate adaption and even once compression on an entire dataset to fulfill constrained bitrate conditions.
- •
We propose a granularity-inform encoder that represents the image patches of sequential spatially variant VQ-indices to support fine-grained variable rate controlling and adaption. Besides, a non-parametric statistical entropy coding is introduced to encode the VQ-indices losslessly based on the approximation of the code distribution in the learned codebook prior across a large-scale natural dataset.
- •
We design a probabilistic conditional decoder, which aggregates historic encoded multi-granularity representations to reconstruct hierarchical granular features in a conditional probability manner, achieving realism improvements.

## 2 Related work

#### Neural Image Compression

Transformation, quantization, and entropy coding are three key components in neural image compression (NIC). Since Ballé et al. [] propose the pioneering learnable NIC method using convolutional neural network (CNN), later methods make improvements in transformation to learn a more compact and exact representation with efficient architecture designs [, , ]. With the help of hyperprior and context models [, , ], the entropy model captures more precise spatial dependencies in the latent, helping probability distribution estimation.

Inspired by the GANs [] and Diffusion Model (DM) [] in various image processing tasks, some algorithms [, , , , , , ] incorporate generative models into NIC frameworks.
For instance, Agustsson et al. [] uses GAN loss along with R-D loss to achieve end-to-end full-resolution image compression while giving dramatic bitrate savings. Mentzer et al. [] incorporates GAN with compression architecture systematically and generates robust perceptual evaluation.
Hoogeboom et al. [] explore autoregressive diffusion model to achieve competitive lossless compression evaluated on a small image dataset.
Yang et al. [] propose an end-to-end DM-based compression framework and reconstruct images through the reverse diffusion process conditioned with context information, achieving competitive performance compared to some GAN-based compression methods.
Superisely, VQGAN []-based generative techniques have shown impressive performance for learning strong codebook priors to discretely represent the visual features in image synthesis, which provides new inspiration to image compression.
Mao et al. [] introduce VQ-indices compression based on the VQGAN model to achieve simple but efficient compression, significantly improving the compression ratio. Based on these observations, we further unleash the potential of VQ and build custom designs around VQGAN to achieve controllable generative compression accommodating general bitrates via a unified single model.

#### Rate-Adaption NIC

The aforementioned methods often face the challenge of deployment in resource-limited devices they are trained as separate models for specific bitrates, which increases the complexity overhead to support multiple bitrates. Current research solving such a rate adaption problem can be roughly divided into two categories: variable-rate compression [, , , ] and progressive compression [, , , , ]. For the former, Theis et al. [] propose to learn rate-specific scalar parameters to scale the transformed features before quantization. Choi et al. [] present a conditional
convolution that adaptively learns the scale and shift factors from a one-shot vector to compress the image for different target quality levels. Cai et al. [] stack multi-scale representations to perform content-adaptive rate allocation. Yang et al. [] leverage slimmable neural networks to perform low- and high-rate compression using partial and all parameters respectively. In [], Cui et al. incorporate gain units that guide the network to allocate bits on specific channels as well as control rates. Lee et al. [] present a selective compression approach that performs entropy coding only for the partially selected latent representations via 3D binary masks.

Progressive compression aims to learn a scalable bitstream that can be progressively exploited to compress images from low to high bitrates. Torderici et al. [] develop a full-resolution recurrent network to transmit bits progressively for high-resolution image compression. Johnston et al. [] introduce an effective initialization scheme for the hidden states of RNN and provide a spatially adaptive rate controller. Zhang et al. [] propose to explore the receptive field with uncertainty guidance for both quality and bitrate scalable compression. Lee et al. [] propose to encode the latent representations into a compressed bitstream trip-plane to support fine-granular progressive compression. Jeon et al. [] further improve this method with context-based trit-plane coding, increasing the R-D performance.

In contrast to these methods, in this work, we base our framework on VQGAN and combine it with the classical coding principle, to design a controllable generative compression framework that allows highly flexible and controllable bitrate adaption while generating plausible compressed images with solely one set of optimized weights.

## 3 Method

Our goal is to learn a unified generative compression model capable of compressing an image $x$ for flexible and continuous bitrates while ensuring high perceptual fidelity. To this end, we propose Control-GIC, where the overview architecture is illustrated in Figure 3. Control-GIC contains three components: 1) granularity-informed encoder $E$ to encode the image into variable-length codes; 2) statistical entropy coding module $H$ for bitrate reduction; and 3) probabilistic conditional decoder $D$ to reconstruct perceptually plausible results.

### 3.1 Correlation between Entropy and Information Density

Inspired by [],
we measure the information density of image regions based on a non-parametric spatial entropy algorithm.
Unlike general feature-level entropy models [, ], our VQGAN-based model does not have the neural entropy model and is not optimized for entropy during training.
We choose the non-parametric algorithm to reduce computational overhead while achieving good granularity selection performance.
The mathematical representation process is as follows:

Assume $x\in\Omega$ is a pixel with a value $p_{x}$ in [-1,1], where $\Omega$ is a patch of the image. We definite the pixel value bin $i=-1+\frac{2k}{n-1}$, where $k=0,1,2,...,n-1$, and the number of bins $n$ is set to 32.
We compute the Gaussian distance $f_{x,i}$ of $x$ to each $i$, where $\sigma$ is the standard deviation, see Eq. (1). Thus $f_{x,i}$ exhibits an unnormalized truncated discrete Gaussian distribution along $i$. It implies that we hypothesize a probability $f_{x,i}$, indicating the potential diffusion of $p_{x}$ to $i$.

$$ $f_{x,i}=\exp\left(-\frac{(p_{x}-i)^{2}}{2\sigma^{2}}\right),\\$ (1) $$

We average $f_{x,i}$ in the patch $\Omega$ to get the probability distribution $f_{\Omega,i}$ of this region, and normalize it to get $\overline{f_{\Omega,i}}$. We denote the average operation as “ $\text{mean}_{\begin{subarray}{c}x\in\Omega\end{subarray}}$”. The formulation is as follows:

$$ $\overline{f_{\Omega,i}}=\frac{f_{\Omega,i}}{\sum_{j}f_{\Omega,j}},\quad\text{ where}\,f_{\Omega,i}=\text{mean}_{\begin{subarray}{c}x\in\Omega\end{subarray}} f_{x,i}.\\$ (2) $$

Finally, we use the following formulation to compute the spatial entropy $H(\Omega)$ of the patch $\Omega$.

$$ $H(\Omega)=-\sum_{i}\overline{f_{\Omega,i}}\log\overline{f_{\Omega,i}}.$ (3) $$

Control-GIC considers the entropy of local patches as the basis of the information density distribution of the image,
and divides it into multiple non-overlapped patches
sorted by their entropy value from low to high, which are then transformed into multi-grained features.
The corresponding patch sizes required for entropy calculation are $\{4,8,16\}$, as detailed in Figure 2.

Figure: (a) Original
Refer to caption: extracted/5646692/figs/method/kodim20.png

### 3.2 Granularity-Informed Encoder

The granularity-informed encoder $E$ distills hierarchical features according to the determined entropy-based granularity assignments. These features are then integrated to form a multi-grained hybrid feature.
Specifically, as visualized in Figure 3, given an input image $x\in\mathbb{R}^{H\times W\times 3}$, $E$ distills $x$ into features of three granularities: fine-grained $z_{1}\in\mathbb{R}^{\frac{H}{4}\times\frac{W}{4}\times d}$, medium-grained $z_{2}\in\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}\times d}$, and coarse-grained $z_{3}\in\mathbb{R}^{\frac{H}{16}\times\frac{W}{16}\times d}$.
Based on the user-given ratios $\left(r_{1},r_{2},r_{3}\right)\in[0,1]$ for three granularties, $E$ extracts the elements with lowest $r_{i}$ proportion of the entropy from $z_{i}$ and generates binray masks $m_{i}\in\{0,1\}$ at the same scale as $z_{i}$ to indicate the position of extracted elements, where $i=1,2,3$. This process is executed from coarse to fine, assigning the multi-grained representations progressively and finely.
Subsequently, the continuous values in each $z_{i}$ in $\{z_{i}\}_{i=1}^{3}$ are quantized against $C$ to produce the quantized counterparts $\hat{z}_{i}$ and a set of discrete indices $Ind_{i}$ that represent the closest matches in $C$ based on the Euclidean distance. This quantization step $\mathbf{q}(\cdot)$ is mathematically formalized as:

$$ $\begin{cases}\hat{z}_{i}=\mathbf{q}(z_{i})=\mathop{\arg\min}\limits_{c_{k}\in C }\|z_{i}-c_{k}\|,\\ Ind_{i}=k.\end{cases}$ (4) $$

The hybrid multi-grained representation $\hat{z}$ is constructed to match the spatial scale of the finest granularity, detailed in Eq. (5), where $(\cdot)\uparrow_{4}$ and $(\cdot)\uparrow_{2}$ signify upsampling operations that amplify the spatial dimensions by factors of 4 and 2, respectively. $\odot$ is element-wise multiplication along the spatial dimension. The upsampling operation is the nearest neighbor interpolation that employs replicates values of feature points along both the width and height, ensuring that the original local structure integrity of each feature point is maintained.

$$ $\hat{z}=(\hat{z}_{1}\odot m_{1})+(\hat{z}_{2}\odot m_{2})_{\uparrow_{2}}+(\hat {z}_{3}\odot m_{3})_{\uparrow_{4}}.$ (5) $$

Figure: Figure 3: The overall framework of our Control-GIC. In the figure, all components cooperate for efficient compression with end-to-end training, and dashed lines represent the unparameterized entropy coding module. The symbols in the diagram are defined as: $m$: the binary mask; $(\cdot)_{\downarrow}$: average pooling operation; $\odot$: element-wise multiplication.

### 3.3 Probabilistic Conditional Decoder

In the context of the decoder within an autoencoder architecture, the latent representation $y_{i}$ at layer $i$ is reconstructed by integrating information from all preceding layers, starting from the initial input $y_{1}$ up to the output of the previous layer $y_{i-1}$.
This reconstruction is formalized through the conditional probability, where $y_{i}$ is distributed according to $p(y_{i}|y_{i-1},\ldots,y_{2},y_{1})$, capturing the likelihood of $y_{i}$ given the history of latent representations.
The more accurate the previous latent representations are, the more accurate $y_{i}$ is, which further influence the overall quality of the image reconstruction.

According to Eq. (4), given the indices and corresponding masks indicating the codes of three granularities, we can losslessly reconstruct the multi-gained representation $\hat{z}$ in the decoder end.
While the upsampled components $\hat{z}_{2}$ and $\hat{z}_{3}$ maintain their local structure through direct value duplication as shown in Eq. (5), their global structure differs. Feeding $\hat{z}$ directly into the decoder layers results in further loss of information.

Based on these considerations, we introduce a probabilistic conditional decoder specifically tailored for the initial layers of the decoder. This decoder employs $(\cdot)\downarrow_{4}$ and $(\cdot)\downarrow_{2}$ consisting of a $4\times$ and a $2\times$ average pooling layer to downscale $\hat{z}$ back to the $\{\hat{z_{i}}\odot m_{i}\}_{i=1}^{3}$ losslessly.
We provide $(\hat{z})_{\downarrow_{4}}$ as decoder input $y_{1}$ which contains the same $\hat{z_{3}}\odot m_{3}$ as the encoder output to ensure the accuracy of the input. $\hat{z_{1}}\odot m_{1}$ and $\hat{z_{2}}\odot m_{2}$ are provided as conditions to $y_{2}$ and $y_{3}$, respectively. These conditions serve as additional guidance for the reconstruction process which is formalized as:

$$ $\begin{split}y_{2}&\sim p(y_{2}\ |\ y_{1},\bf{(\hat{z})_{\downarrow_{2}}\odot m _{2}})\\ y_{3}&\sim p(y_{3}\ |\ y_{2},y_{1},\bf{\hat{z}\odot m_{1}})\end{split}$ (6) $$

Specifically, the decoding process is detailed in Eq. (7), $D$ begin with the $(\cdot)\downarrow_{4}$ operation on $\hat{z}$ to produce $y_{1}$. The first decoder layer $D_{1}$ processes $y_{1}$ and generates $y_{2}$ in which $y_{2}\odot m_{2}$ are deliberately replaced with the medium-grained representation $(\hat{z})_{\downarrow_{2}}\odot m_{2}$ (equal to $\hat{z}\odot m_{2}$). After that, $D_{3}$ condition with the $\hat{z}\odot m_{1}$ and replace the unexact $y_{3}\odot(1-m_{1})$, ensuring the precision of features in deep layers. This systematic replacement of representations at varying granularities with increasingly precise conditions progressively refines the latent space representation, helping $D$ diminish information loss and substantially elevate the accuracy of the reconstructed images.

$$ $\begin{split}\begin{cases}y_{1}&=(\hat{z})_{\downarrow_{4}}\\ y_{2}&=D_{1}(y_{1})\odot(1-m_{2})+\bf{(\hat{z})_{\downarrow_{2}}\odot m_{2}}\\ y_{3}&=D_{2}(y_{2})\odot(1-m_{1})+\bf{\hat{z}\odot m_{1}}\\ \hat{x}&=D_{3}(y_{3})\end{cases}\end{split}$ (7) $$

### 3.4 Statistical Entropy Coding Strategy

The decoding process requires these pieces of bitstreams: the three granularity indices $\{Ind_{i}\}_{i=1}^{3}$, along with their corresponding masks $\{m_{1},m_{2}\}$.
These elements are encoded using lossless coding algorithms. The mask consisting of 0 and 1, is encoded directly into a binary stream.
The indices, which point to codebook entries, exhibit an uneven frequency distribution, with a minority of codes being used for quantization [].
To optimize indices’ encoding, we introduce a statistical entropy coding strategy that captures the frequency distribution of indices usage across a natural dataset during training.
Each index starts with a frequency count of 0, while the frequency is updated each time it is matched for vector quantization.
To simulate the data distribution in the inference process, we utilize the last epoch’s frequency statistics to construct a frequency table tailored for Huffman coding.
We denote the bitrate after coding as $\mathcal{R}(\cdot)$, then the total bitrate of the entire image can be formulated as:

$$ $\mathcal{R}=\sum\limits_{i=1}^{3}\mathcal{R}(Ind_{i})+\mathcal{R}(m_{1})+ \mathcal{R}(m_{2}),$ (8) $$

Note that our model does not optimize network parameters specifically for bitrate. During inference, we control the bitrate using different multi-granularity allocation ratios. Specifically, we set a group of hyperparameters $\{r_{1},r_{2},r_{3}\}$ to represent the allocation ratios of the masks $\{m_{1},m_{2},m_{3}\}$ at various granularities.
Since the bitrate is only related to the granular representations of local patches, we can flexibly determine the statistics of the granularities based on the entropy values to achieve dynamic adaption, achieving dynamic adaption in a target of the quality-bitrate adaptive manner in a unified model without any post-training.
See Appendix A.3 for the bit-saving experiment of the statistical entropy coding.

### 3.5 Loss Function

The loss function $\mathcal{L}$ for our approach contains the loss associated with the VQVAE architecture and GAN component. The optimization objective of $\mathcal{L}_{\text{VQVAE}}(E,G,C)$ is twofold: to minimize the distortion between the original inputs $x$ and their reconstructions $\hat{x}$, and to reduce the divergence between the continuous representations $z=E(x)$ and their quantized versions $\hat{z}$, as shown in Eq. (9). The distortion of $x$ is measured by MSE ($d_{M}$) and LPIPS ($d_{P}$).
Due to the non-differentiable nature of quantization, the stop-gradient operation $sg[\cdot]$ is widely utilized in VQ-based models which enables the quantized representations $\hat{z}$ to propagate gradients directly for optimizing the codebook $C$, and allows the continuous representations $z$ to receive gradients for the refinement of the encoder $E$.
We fine-tune the balance between these objectives using a hyperparameter $\beta$.

Our Control-GIC diverges from the conventional R-D optimization paradigm with no-parametric indices compression, enabling the model to adapt to different types of data and desired quality levels, rather than being constrained by a fixed R-D trade-off.

$$ $\displaystyle\mathcal{L}_{VQVAE}(E,G,C)$ $\displaystyle=\beta d(x,\hat{x})+d(z,\hat{z})$ $\displaystyle=\beta(d_{M}+d_{P})(x,\hat{x})+\|sg[z]-\hat{z}\|_{2}^{2}+\|sg[ \hat{z}]-z\|_{2}^{2}$ (9) $$

We use a hyperparameter $\beta$ to control the trade-off between VQVAE and GAN. The GAN loss and the the total loss $\mathcal{L}$ is as follows:

$$ $\displaystyle\mathcal{L}_{GAN}(\{E,G,C\},D)$ $\displaystyle=\mathbb{E}_{x\sim p(x)}[\log D(x)+\log(1-D(G(\hat{z})))]$ (10) $\displaystyle\mathcal{L}$ $\displaystyle=\mathcal{L}_{VQVAE}+\lambda\mathcal{L}_{GAN}$ (11) $$

## 4 Experiment

### 4.1 Experimental Setup

Our method is based on MoVQ [] which improves the VQGAN model by adding spatial variants into the decoder’s representation maps, avoiding the repeat artifact in neighboring patches. We leverage MoVQ’s pre-trained codebook and redesign the architecture to suit the compression objective.

#### Training & Inference

Our training dataset consists of random 300k images from the OpenImages [] dataset. We randomly crop images to a uniform $256\times 256$ resolution. Within our model, we take three representation granularities: $4\times 4$, $8\times 8$, and $16\times 16$. The codebook $C\in\mathbb{R}^{k\times d}$ comprises $k=16384$ code vectors, each with a dimension of $d=4$.
We train the model for 165k iterations with the learning rate of $5\times 10^{-5}$ on 4 NVIDIA RTX 3090 GPUs.
Throughout the training, we maintain the ratio setting of $(0.5,0.4,0.1)$ for the fine, medium, and coarse granularity, respectively.
During inference, our controllable multi-granularity approach accepts any-resolution images and enables fine bitrate adjustment with a unified model.

#### Evaluation

We evaluate our method on Kodak [] dataset, a standard in compression assessment, which contains 24 high-quality images at $768\times 512$ resolution. We carry out multi-dimensional evaluation and utilize a comprehensive set of evaluation metrics including perceptual metrics: LPIPS [], DISTS [], distortion metric: PSNR, generative metrics: FID [], IS [], as well as the no-reference measurement: NIQE [] to thoroughly assess the performance of our method. More details of metrics are in the Appendix A.2.

### 4.2 Performance Comparison

Figure: Figure 4: Compression performance with compared methods. In this figure, the lines with forks represent GIC methods, and the lines with rhombus represent progressive and variable-rate methods.
Refer to caption: x3.png

We evaluate our approach with recent state-of-the-art (SOTA) NIC methods. In terms of generative compression approaches, 1) HiFiC [] is an influential approach leveraging conditional GAN and taking rate-distortion-perception trade-off.
2) CDC [] is a representative diffusion-based lossy compression approach providing a compression performance that rivals GAN-based methods.
Besides, we compare our Control-GIC with SOTA variable-rate and progressive neural compression methods in which images under various bitrates are generated by a unified network. 3) SCR [] proposes a 3D important map adjusted by quality level to decide the selected representation elements leading to variable-rate NIC. 4) CTC [] progressively decodes the bit stream which can be truncated at any point to finely regulate the bitrate. We utilize BPG (YUV444, x265 HEVC, 8-bit depth) [] and VVC (VTM10.0) [] as compared traditional codecs and M&S (Hyperprior) [] as the classical NIC method for reference.
For a fair comparison, we utilize a CDC version that the lpips weight $p=0.9$ which is the endpoints close to the perception quality. We employ our unified model, getting the results of all bitrates with a single trained model, the ability to finely control the bit rate is reflected in the Appendix A.4.

Figure 4 demonstrates that our Control-GIC surpasses the majority of SOTA NIC methods and all traditional codecs across six distinct metrics. Our methodology exhibits a significant improvement in LPIPS, NIQE, and IS performance, approaching the SOTA performance of the CDC in terms of the FID with our unified model. Appendix A.1 presents more comparison results on high-resolution images.

While PSNR is pivotal for pixel-level evaluation, we find that our methods excel in generating realistic images. This performance is notable even when compared to methods that are specifically optimized for pixel-level accuracy. Thus, we explore the quantitative comparison between our Control-GIC and the compared methods. As depicted in Figure 5, VVC, M&S (Hyperprior) as well as variable-rate SCR and progressive CTC exhibit noticeable issues with blurred artifacts and detail loss, and generative HiFiC and CDC generate artifacts that were not originally present in the images. Compared to them, our method demonstrates superior performance in preserving texture integrity and image sharpness for a great visual experience.

Figure: (a) Original
Refer to caption: x4.png

### 4.3 Efficiency of Probabilistic Conditional Decoder

Figure: Figure 6: The ablation experiments on multi-grained conditions for the probabilistic conditional decoder.

Based on the theoretical analysis of the probabilistic conditional decoder, we explore the contribution of the conditions: the medium-grained $(\hat{z})_{\downarrow_{2}}\odot m_{2}$ (denoted as $med$) and fine-grained $\hat{z}\odot m_{1}$ (denoted as $fin$) to the decoder in Eq. (7). We train all the models for the same iterations and evaluate the performance on both perceptual and generative quality measurements. Figure 6 demonstrates that the model without the $med$ and $fin$ exhibits the worst results in three metrics. Both $med$ and $fin$ have a significant improvement in model performance, and conditioning on both of them presents the best results, especially on DISTS. It can be observed that adding the fine-grained $fin$ to the model brings more benefits than adding $med$, which is because the fine-grained $fin$ corrects the features in deeper decoder layers, ensuring the accuracy of features after multiple nonlinear transformations of the decoder.

## 5 Conclusion

Our proposed Control-GIC framework addresses the critical challenge of flexible rate adaption in generative image compression. By leveraging a VQGAN foundation and correlating local image information density with granular representations, Control-GIC achieves fine-grained bitrate control across a wide range while maintaining high-fidelity compression performance. The innovative probabilistic conditional decoder enhances reconstruction realism by aggregating hierarchical features informed by conditional probability. Our experiments validate the superior adaptability and performance of Control-GIC, showcasing its capability for efficient compression and bit control, outperforming recent state-of-the-art methods.

## 6 Limitations

Our Control-GIC unleashes the capabilities of the VQGAN model, delivering high-quality reconstruction across a broad range of bitrates, and outperforming traditional MSE-optimized methods in visual performance. However, it is important to recognize that at higher bitrates, specifically above 1.0 bpp, our method encounters limitations. MSE-optimized algorithms excel in preserving fine details and textures at these bitrates due to their focus more on subtle distortions. Thus GIC methods, including ours, may not match the detail precision achievable by MSE-optimized techniques.

## Appendix A Supplemental Material - Once-for-All: Controllable Generative Image Compression with Dynamic Granularity Adaption

### A.1 Performance Comparison on High-Resolution Images

In this section, we present a performance comparison on the DIV2K [] validation dataset containing 100 high-resolution images. As depicted in Figure 7, we compare with traditional codecs, variable-rate NIC method as well as generative method (Note that the high-resolution image compression is not supported in CDC []), and employ six key metrics to assess the performance of the compression algorithms, including the perceptual metrics (LPIPS, DISTS), distortion metric (PSNR), generative metrics (FID, IS), and no-reference NIQE. These metrics collectively provide a comprehensive evaluation of the image compression quality and our Control-GIC achieves the best performance in a wide spectrum of bitrates.

Figure: Figure 7: Compression performance in high-resolution images with compared methods. In this figure, the lines with forks represent GIC methods, and the lines with rhombus represent variable-rate methods.
Refer to caption: extracted/5646692/figs/exp/compare_div2k/lpips.png

### A.2 Evaluation Metrics

we adopt a comprehensive set of evaluation metrics to thoroughly assess the performance of our image compression and reconstruction techniques. Our selection encompasses perceptual metrics, distortion metrics, generative metrics, and a no-reference metric, ensuring a multifaceted evaluation.
The perceptual metrics include LPIPS [], which measures the perceptual difference between images, and DISTS [], which evaluates the structural dissimilarity. These metrics are crucial for understanding how closely compressed and reconstructed images resemble their original counterparts in terms of human visual perception.
We also include the widely recognized distortion metric PSNR, which quantifies the pixel-level differences between the original and reconstructed images. PSNR is a standard in the field, providing a straightforward measure of image fidelity.
For generative metrics, we employ FID (Fréchet Inception Distance) [] to offer statistical assessments of the similarity between the distributions of original images and those of reconstructed images, which is particularly valuable in the context of generative models.
Additionally, we incorporate IS (Inception Score) [], which is another way to gauge the quality and diversity of generated images.
NIQE (Natural Image Quality Evaluator) [] stands out as a no-reference metric, capable of evaluating image quality without requiring an original reference image. This feature renders NIQE exceptionally beneficial in applications such as super-resolution where an original high-resolution image may not be available.
For comparing with other methods on FID and IS, we split images on Kodak into 192 patches and DIV2K into 6 573 patches with a patch size of 256.

### A.3 Bit Saving of Statistical Entropy Coding Strategy

In the Control-GIC method, the accurate reconstruction of images from compressed data is highly dependent on the lossless compression of indices, which determine the codes for decoding. To achieve this, we employ Huffman coding, an algorithm that efficiently compresses data by assigning shorter binary codes to elements that occur more frequently. However, using frequency statistics tailored to each individual image would require transmitting both the encoded bitstream and the mapping of elements to binary codes, incurring significant bit costs.

As described in Table  1, we compare our statistical entropy coding strategy with the classical ZIP compression algorithm and Huffman coding with uniform frequencies for each index.
Our statistical entropy coding strategy brings 68.4%, 65.7%, and 59.0% bit savings compared to ZIP in the three granularity ratios, as well as 6.4%, 6.6%, and 8.2% bit savings compared to Huffman coding with uniform frequency for each code.

**Table 1: Bit cost comparison of ZIP, Huffman coding with uniform frequency for each index, and our entropy coding strategy on the Kodak dataset. Our strategy counts the frequency of all indexes being used in the training process as a generalized and shared probability in the encoding and decoding process.**
| Granularity Ratio | ZIP | Huffman | Huffman (Ours) |
| --- | --- | --- | --- |
| $w.$ uniform frequency | $w.$ statistical frequency |  |  |
| (0, 0.5, 0.5) | 0.304 | 0.141 | 0.132 |
| (0.1, 0.8, 0.1) | 0.784 | 0.288 | 0.269 |
| (0.5, 0.4, 0.1) | 1.232 | 0.550 | 0.505 |

### A.4 Fine Control of Bitrate

Our experiment with variable-rate control demonstrates the adaptive bitrate adjustment capabilities of our method as well as the corresponding quality changes across the bitrates, as measured by four different metrics, as detailed in Table 2. By fine-tuning the granularity ratio, we are able to make incremental adjustments to the bitrate. This results in a continuous range of bitrates that offer higher efficiency and flexibility with minimal impact on computational resources and processing time than single-rate models.

**Table 2: The elastic and fine bitrate control presentation over the Kodak dataset at bitrates ranging from 0.389 to 0.391. All results are shown in five valid digits to represent differences more clearly. The Granularity Ratio refers to the proportion of fine granularity, medium granularity, and coarse granularity from left to right.**
| Granularity Ratio | Bpp$\downarrow$ | LPIPS$\downarrow$ | DISTS$\downarrow$ | FID$\downarrow$ | NIQE$\downarrow$ |
| --- | --- | --- | --- | --- | --- |
| (0.301, 0.599, 0.1) | 0.38925 | 0.03027 | 0.05984 | 33.327 | 2.9975 |
| (0.302, 0.598, 0.1) | 0.38983 | 0.03025 | 0.05961 | 33.338 | 2.9971 |
| (0.303, 0.597, 0.1) | 0.39050 | 0.03020 | 0.05973 | 33.233 | 2.9967 |
| (0.304, 0.596, 0.1) | 0.39108 | 0.03014 | 0.05968 | 33.215 | 2.9942 |

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: See weakness.
soundness: 3
strengths: 1. Control-GIC combines classical coding principles with VQGAN to achieve controllable generative compression across various bitrates with a unified model.
2. The framework allows for highly flexible and controllable bitrate adaption, which is a significant advancement over existing methods.
3. Unlike other methods that require training multiple models for different bitrates, Control-GIC can adapt to various bitrates with a single model, reducing computational costs.
summary: The paper titled "ONCE-FOR-ALL: CONTROLLABLE GENERATIVE IMAGE COMPRESSION WITH DYNAMIC GRANULARITY ADAPTION" introduces Control-GIC, a framework for controllable generative image compression. It addresses the challenge of flexible rate adaption in image compression by leveraging a VQGAN framework that encodes images as variable-length codes . The framework correlates local image patch information density with granular representations, allowing for fine-grained bitrate control. It includes a granularity-informed encoder, a statistical entropy coding module, and a probabilistic conditional decoder. The experiments demonstrate that Control-GIC outperforms state-of-the-art methods in terms of flexibility, perceptual quality, and compression efficiency.
weaknesses: 1. More comparion with other GIC methods need to be provided.
2. The novelty is limited compared to other VQGan based GIC method

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Why is the mask ratio set to 50%, 40%, 10% for fine, medium, and coarse features in the training setting? Please provide further justification for this choice.
2. In Mao et al.'s papers [2, 3], the bitrate used in VQGAN is significantly lower, ranging from <0.05 bpp to ≤0.03 bpp, while in GIC, the bpp ranges from 0.1 to 0.6 bpp. Could you explain why the bitrate range cannot be lower in Control-GIC, and if it is possible to extend the model to support lower bitrates?



References:
[1] Qi Mao, et al, Extreme image compression using fine-tuned vqgan models.DCC, 2024.
[2] Naifu Xue, Qi Mao, et al, Unifying Generation and Compression: Ultra-low bitrate Image Coding Via Multi-stage Transformer.ICME.2024.
soundness: 3
strengths: 1. The proposed Control-GIC introduces a unified model that allows dynamic bitrate adjustment, which effectively solves the inefficiencies faced by existing models that need multiple fixed-rate versions.
2. The granularity-informed encoder and probabilistic conditional decoder are well-designed to achieve efficient encoding and high perceptual fidelity.
3. Experimental results show superior performance over state-of-the-art methods, demonstrating both flexibility and effectiveness in compression.
summary: This paper introduces Control-GIC, a controllable generative image compression framework aimed at addressing the limitations of existing generative image compression methods in achieving flexible bitrate adjustment. Built upon the VQGAN framework, Control-GIC incorporates multi-granularity encoding mechanisms and a probabilistic conditional decoder to achieve flexible bitrate control and high-quality image reconstruction. Both qualitative and quantitative results demonstrate the effectiveness of the proposed method.
weaknesses: 1. The paper lacks sufficient details on how the features are divided into different granularities in Section 3.1.
2. The DIV2K comparisons in Figure 4 do not include evaluations against important baselines like VVC, M&S, and other methods (presented in Figure 3), which limits the completeness of the analysis.
3. The paper does not compare Control-GIC with other VQ-based methods, such as GLC [1], Mao et al. [2], and UIGC [3], which would provide a better context for understanding the model's relative performance.
4. It is suggested that the authors add more comparisons with results from other datasets featuring different image sizes to enhance the robustness, such as CLIC-dataset.


References:
[1] Jia Z, Li J, Li B, et al. Generative Latent Coding for Ultra-Low Bitrate Image Compression[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 26088-26098.
[2] Qi Mao, et al, Extreme image compression using fine-tuned vqgan models. DCC, 2024.
[3] Naifu Xue, Qi Mao, et al, Unifying Generation and Compression: Ultra-low bitrate Image Coding Via Multi-stage Transformer.ICME.2024.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 1
questions: see the weakness part
soundness: 1
strengths: The paper proposes to flexibly determine a proper allocation of granularity for patches, supporting dynamic adjustment for VQ-indices and make the framework capable of fine-grained bitrate adaptation.
summary: This paper targets at addressing the challenge of rate adaption proplem for generative image compression and proposes a controllable generative image compression framework. The paper represents the image patches of sequential spatially variant VQ-indices to support precise variable rate control and adaption. A non-parametric statistical entropy coding is devised to encode the VQ-indices losslessly.
A probabilistic conditional decoder is proposed to aggregate historic encoded multi-granularity representations, achieving realism improvements.
weaknesses: - In Figure 3, the proposed method shows worse performance than CDC for DISTS, and is also worse than CDC at the high bitrate range. Could you give some analysis? 

- In Figure 4, why do you compare only with BPG, CTC, and HiFiC, instead of aligning with the methods used in Figure 3?

- In Figure 3 and Figure 4, most metrics are nearly reaching saturation when the bpp increases. Is there obvious difference for the visualization quality when the bpp increases at the high bitrate range? For example, in Figure 6, when the bpp increases, I do not see obvious enhancement between r2=40%(bpp=0.3864) and r2=32.9%(0.4171).

- Visualization analysis about the influence of the granularity of image patches. Will increasing the image patch size lead to block artifacts in the reconstructed images? And can making the image patch size smaller solve some artifacts in VQ based methods, such as the unsatisfactory artifacts for small faces?

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. How are the masks $m_1$, $m_2$, and $m_3$ obtained? The main text does not seem to provide a clear calculation method for these.
2. If compression is required at a specific bitrate, how should $r_1$, $r_2$, and $r_3$ be determined?
soundness: 3
strengths: 1. The authors’ granularity-informed encoder effectively leverages the information density distribution to distill image patches into a hierarchical structure with three levels of granularity. This approach enables compression with a controllable encoding rate, providing a novel solution for adaptable image compression.

2. The paper’s codec framework is lightweight and demonstrates relatively fast encoding and decoding speeds, making it more suitable for practical applications where computational efficiency is a priority.
summary: The paper Once-for-All: Controllable Generative Image Compression with Dynamic Granularity Adaption introduces Control-GIC, a flexible framework for high-quality image compression that allows fine-grained bitrate control without needing multiple models. Built on a VQGAN foundation, Control-GIC encodes images as variable-length sequences, adapting bitrate based on local image information density, which allows for effective compression adjustments to meet different content complexities. The model includes a granularity-aware encoder that assigns varying levels of detail across image patches, and a probabilistic conditional decoder that reconstructs images with high perceptual quality by aggregating multi-scale features. Experimental results show that Control-GIC outperforms recent state-of-the-art methods in perceptual quality, flexibility, and efficiency, achieving this with a single unified model across a wide bitrate range.
weaknesses: 1. The experimental results show mixed outcomes compared to existing baselines like HiFiC, without demonstrating a clear advantage.
2. The paper should consider comparing with MS-ILLM [1], which uses a similar VQ-VAE structure, to better position its contributions.
3. Tests on the CLIC 2020 dataset are missing, which would help validate the model's robustness across diverse datasets.
4. Section 3.2 is overly detailed and could be streamlined for clarity and conciseness.
5. Section 3.3 appears to be a direct application of existing methods, which could benefit from additional elaboration or innovation.

[1] Muckley M J, El-Nouby A, Ullrich K, et al. Improving statistical fidelity for neural image compression with implicit local likelihood models. International Conference on Machine Learning. PMLR, 2023: 25426-25443.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the above weakness.
soundness: 3
strengths: 1. A comprehensive explanation on adapting VQ-GAN into a rate-adaptive perceptual codec.

2. Highly flexible and controllable bitrate adaptation.

3. Well-conducted ablation study.

4. Open-sourced code! This is fantastic, and I want to thank the authors for their commitment to sharing this.
summary: This paper presents a novel framework for controllable generative image compression (Control-GIC). The proposed Control-GIC integrates a granularity-aware encoder to enable precise variable rate control and adaptation, a non-parametric statistical entropy coding method for lossless encoding of VQ-indices, and a probabilistic conditional decoder to reconstruct hierarchical granular features. Experimental results on the Kodak and DIV2K datasets demonstrate that Control-GIC not only delivers strong perceptual compression performance but also provides highly flexible and controllable bitrate adaptation.
weaknesses: 1. Lack of comparison with the latest state-of-the-art (SOTA) perceptual codec, MS-ILLM [1]. It appears that MS-ILLM outperforms both HiFiC and Control-GIC in terms of compression performance.

2. Control-GIC utilizes a mask for multi-scale encoding, but the method for generating the masks $m_1, m_2$ and $m_3$ during training and inference is not clearly explained. Please provide a more detailed description of this process.

3. Misuse of capitalization: 'The Encoding and decoding' on line 432 should be corrected to 'The encoding and decoding.'"

[1] MJ Muckley, A El-Nouby, K Ullrich, H Jegou, J Verbeek. Improving Statistical Fidelity for Neural Image Compression with Implicit Local Likelihood Models. In ICML, 2023.

Meta-review: The paper received mixed reviews from five experts.

The authors' provided responses trying to address the reviewers' concerns.

Particularly, Reviewer P96B decided to lower significantly her/his ratings after receiving the authors' responses without giving any explanation to the authors/the fellow reviewers.  
Therefore, Reviewer P96B's ratings are to be ignored. 

The other reviewers are satisfied with the responses, either keeping or elevating their initial ratings.

After rebuttal, Reviewer qmzM champions acceptance (8: accept, good paper) while Reviewers o1HZ, WUkA, and bPMe are borderline, generally appreciating the contributions and not having remaining important concerns.

AC finds merits to the paper (Reviewer qmzM : "a strong paper that introduces an innovative approach to adjusting bitrate within a unified framework using mask-ratio strategies") and recommends acceptance and invites the authors to refine the paper contents so that most of the concerns are addressed in the paper.

Decision: Accept (Poster)