Image Compression with Product Quantized Masked Image Modeling
Authors: Alaaeldin El-Nouby
Matthew J. Muckley
Karen Ullrich
Ivan Laptev
Jakob Verbeek
Herve Jegou
Venue: Accepted by TMLR ()

Abstract: Recent neural compression methods have been based on the popular hyperprior framework. It relies on Scalar Quantization and offers a very strong compression performance. This contrasts from recent advances in image generation and representation learning, where Vector Quantization is more commonly employed.
In this work, we attempt to bring these lines of research closer by revisiting vector quantization for image compression. We build upon the VQ-VAE framework and introduce several modifications. First, we replace the vanilla vector quantizer by a product quantizer. This intermediate solution between vector and scalar quantization allows for a much wider set of rate-distortion points: It implicitly defines high-quality quantizers that would otherwise require intractably large codebooks. Second, inspired by the success of Masked Image Modeling (MIM) in the context of self-supervised learning and generative image models, we propose a novel conditional entropy model which improves entropy coding by modelling the co-dependencies of the quantized latent codes. The resulting PQ-MIM model is surprisingly effective: its compression performance on par with recent hyperprior methods. It also outperforms HiFiC in terms of FID and KID metrics when optimized with perceptual losses (e.g. adversarial). Finally, since PQ-MIM is compatible with image generation frameworks, we show qualitatively that it can operate under a hybrid mode between compression and generation, with no further training or finetuning. As a result, we explore the extreme compression regime where an image is compressed into 200 bytes, i.e., less than a tweet.

Published in Transactions on Machine Learning Research (03/23)
Image Compression with Product Quantized Masked Image Modeling
Alaacldin El-Nouby 1.8.1
aelnouby@meta. mamuckley@ment com karenu emera com ioun, lopteve@iriii Siverbeek@mett com rujumeta. com
Matthew Muckley
Karen Ullrich
Ivan Laptev S.f
Jakob Verbeek
Hervé Jégou
o Mela AI, PAIR Team, Paris, SENS, PSL University, VINRIA, Paris
Reviewed on OpenReview: http:://woemeeeen.n////mmttttt/
Abstract
Recent neural compression methods have been based the popular hyperprior framework. It relies Scalar Quantization and offers very strong compression performance. This contrasts from recent advances in image generation and reprosentation learning, where Vector Quantization is more commonly employed.
In this work, we allempt to bring these lines of research closer by revisiting veclor quantization for image compression. We build upon the VQ-VAF. framework and introduce several modifications. First, we replace the vanilla vector quantizer by a product quantizer. This intermediate solution between vector and scalar quantization allows for a much wider set of rate-distortion points: It implicitly defines high-quality quantizers that would otherwise require intractably large codebooks. Second, inspired by the success of Masked Image Modeling (MIM) in the context of self-supervised learning and generative image models, we propose a novel conditional entropy model which improves entropy coding by modelling the co-dependencies of the quantized latent codes. The resulting PQ-MIM model is surprisingly effective: its compression performance is on par with recent hyperprior methods. also out perforosss HiFiC in terms of FID and KID metrics when optimized with perceptual losses (e.g. adversarial). Finally, since PQ-MIM compatible with image generation frameworks, show qualitatively that it can operate under hybrid mode between compression and generation, with no further training or finetuning. As a result, explore the extreme compression regime where an image is compressed into 200 bytes, i.e., less than a tweet.
1 Introduction
Efficient image codecs have accelerated the rapid growth of the internet by enabling the transmission of images in few dozens kilobytes, thanks to the emergence of effective lossy methods. This democratization was accompanied by standardization efforts facilitate the interoperability, which led to the emergence of standards such as the Joint Photographic Experts Groups (JPEG). Subscquent formats have leveraged scientific advances on all components of source coding, ranging from transforms (Antonini et al., 1992), and quantization (Gray & Neuhoff, 1998), to entropy coding (Witten et al., 1987; Taubman, 2000), eventually leading to modern video compression codecs enabling streaming and vidco-conferencing applications.
Neural methods have recently become increasingly popular for image compression as well as other image processing tasks, such as denoising (Tian et al., 2020), super-resolution (Bruna et al., 2016; Dong et al., 2015; Ledig et al., 2017; Wang ct al., 2021) or image reconstruction (Wang et al., 2020; Knoll et al., 2020). In typical scenarios, neural image compression not necessarily mature enough to take over standard tochniques like the BPG format. inherited from the High-Ffficiency Video Coding standard (Sullivan et al., 2012). This

---

Published in Transactions on Machine Learning Research (03/23)
is because they do not offer significant quantitative advantage over prior works that would justify the higher complexity, which depends on the context and operational constraints. A key advantage of neural compression methods is their enhanced qualitative reconstruction when incorporating an adversarial loss or likewise psycho-visual objectives favoring visually appealing reconstructions (Agustsson et al., 2019; Mentzer et al., 2020). From this perspective, neural compression is related to image generation. The two subfields, however, are currently dominated by different approaches, noticcably they employ different discretization procedures. Indeed, while earlier neural compression methods utilized vector quantization (Agustsson et al., 2017, VQ), recent methods mostly employ scalar quantization (SQ). In contrast, the recent literature on image generation (Chang et al., 2022; Yu et al., 2021; Esser et al., 2021; Rombach et al., 2022) relies on Vector Quantization jointly with distortion criterion akin to those used in compression.
In this work we aim to reduce the methodological gap and to make a step towards unification of neural image compression and image generation, and allowing image compression to more directly benefit from the rapid advances in image generation methods. Patch- basod masking mothods for self-supervisod learning (Bao et al., 2022; He et. al., 2021b; El-Nouby el. al., 2021a) have recently demonstrated their potential for image generation (Chang et al., 2022). Inspired by this work we propose compression approach built upon Vector Quantized Variational Auto-Encoder (Oord et al., 2017: Razavi et al., 2019). In this context, we focus on two intertwined questions: (1) How to define vector quantizer offering a range of rate-distortion operating points? (2) How to define an entropy model minimizing the cost of storing the quantization indexes, while avoiding the prohibitive complexity of an auto-regressive model?
To address the challenges above, we revisit vector quantization in image compression, and investigate product quantization Jégou et al. (2010) (PQ) in compression system derived from VQ-VAE Oord et al. (2016b). We show that PQ offers strong and scalable rate-distortion trade-off. We then focus on the spatial entropy modeling and coding of the quantization indexes in the VQ or PQ latent layer, hence we name our Masked Image Modeling (VQ-MLM and PQ-MIM). To this end,
method as (Vector/1 we introduce a multi-stage vector- sector-quantized image model: we gradually reduce the conditional entropy of the patch latent codes by increasing the number of observed patches we condition on for each slage. The conditional distribution over patches is estimated by a transformer model and provided to an entropy coder, symmetrically on the emitter and receiver sides.
In summary, we make the following contributions:
We introduce a novel Masked Image Modeling conditional entropy model that significantly reduces the rates by leveraging the spatial inter- anter-dppendeccier between latent codes.
We introduce product quantization for VQ-VAE. This simple PQ-VAE variant offers a strong and scalable rate-distortion trade-off.
When trained with adversarial and perpetual losses, PQ-MIM exhibits a strong performance in terms of perceptual metrics like FID and KID, outperforming HiFIC (Mentzer et al., 2020).
We qualitatively show that PQ-MIM is capable of operating in a hybrid mode, between generative and compression, without requiring further training and finetuning. This allows for higher resilience to corrupted missing signal where our model can fill-in the missing information.
2
Related work
Neural image compression Early approaches to neural image compression reach back to the late 1980s (Sonehara, 1989; Sicuranza et al., 1990; Bottou et al., 1998). Recent rapid advances in explicit and implicit density modelling (Goodfellow et al., 2014; Kingma & Welling, 2014; Larochelle & Murray, 2011; Oord et al., 201 Rezende & Mohamed, 2015; Salimans et al., 2017) have renewed interest in posing image compression as a learning problem. Due to the connection to variational learning (Gregor et al., 2016; Frey, 1998; Alemi et al., 2018), variational auto-encoders have been the primary choice for lossy image compression. In contrast to standard variational models, the evaluation of neural compression models focuses on achievable bitrates, multi-scale applicability and computational complexity. The initial works in the field used fully
2

---

Published in Transactions on Machine Learning Research (03/23)
Original
PQ-MIM
HFC Lo -0.005 bpp
PO D. APE
Figure 1: Qualitative example of PQ-MIM compression. PQ-MIM provides a strong compression performance. We retain many of the details present original image with minimal blurring effect even with comrpession rate as low as 0.196 hpp. Moreover, compared to HiFiC we achieve lower rate for the same image. PQ-MIM provides colors that are more faithful to the original image, while HiFiC has a darkening effect and some high frequency artifacts. On the other hand, PQ-MIM can have some smoothing effect that cause loss of detail for particular regions (e.g. some of the Arabic letters in the example above). More qualitative examples are provided in Appendix D.
(Ballé et al., 2017; Theis et al., 2017; Mentzer et al., 2018). The resulting encoded latent image representations are quantized and compressed via an entropy coder with learned explicit density model or "entropy bottleneck". Initial variational approaches directly modeled a single level of code densities. Ballé extended these models by introducing a second "hyperprior" that yielded improved performance (Ballé et al., 2018). Hyperprior models have been the basis for several subsequent advances with further improvements for density modeling, such joint autoregressive models (Minnen et al., 2018), Gaussian mixture/ attention (Cheng et al., 2020), and channel-wise auto-regressive models (Minnen & Singh, 2020). He et al. (2021a) improve the efficiency of spatial autoregressive context models by utilizing checkerboard contexts which allows for more parallelization friendly decoding. This approach can be considered a special case when we use only two steps of quincunx pattern encoding/ prcodiin/decoddngg decoding. Another line of work has proposed to use vector quantization with histogram-baser based probabilities for image compression (Agustsson et al., 2017; Lu et al., 2019). Contrary to VQ-VAE models, these models lypically optimize the rate (or a surrogate of the rate) directly and may include a spatial component for the quantized vectors. Yang et al. (Yang al., 2020) showed multiple ways improve the encoding process, including the fino-tuning of the discrelization process and employing bils-back coding in the entropy bottleneck. Finally, (Santurkar et al., 2018; Mentzer et al., 2020; Rippel & Bourdev, 2017; Agustsson et al., 2019) showed that altering the distortion metric to include an additional adversarial loss can make large difference for compression rate. Another interesting line of work considers image compression by training image-specific networks, or network adapters, that map image coordinates to RGB values, and compressing the image-specific parameters (Dupont. et al., 2021; 2022; Strüpler al., 2022).
convolutional architectures for
VQ-models for image generation. There has been significant interest in generative models based in discrete image representations, as introduced by VQ-VAE (Oord et al., 2017; Razavi et al., 2019). A discrete representation of reduced spatial resolution is learned by mcans of an autocncoder which quantizes the latent representations. This discrete representation is coupled with a strong prior, for example implemented as an autoregressive pixel-CNI model. VQ-GAN (Esser et al., 2021) replaces the prior architecture with a transformer model (Vaswani et al., 2017), and introduces an adversarial loss term to learn an autoencoder with more visually pleasing reconstructions and improved sample quality. This approach has been extended to text- based generative image models by extending the prior to model a longer sequence that combines the discrete
3

---

Published in Transactions on Machine Learning Research (03/23)
X
PQ
Encoder
Decoder
PQ
b
Step
Step =
q
Symbols
qi
Arithmetic Encoder
b
q
bi
Arithmetic
qi
LAAXCIVOLL 11100011
Decoder
OLL-3100CUUMIOD 1 00000 11101001
CDF
CDF
pqq 9s<i)
60
44
10
65
67
p(q: 9x<i)
26
82
20
Quincunx
MIM
Quincunx
MIM Entropy Model
Entropy Model
qi 9s<i
9s<i
9s<i
9s<i
Figure 2: PQ-MIM overview. Our model consists of (i) a transformer based encoder and decoder, (ii) a masked image model (MIM) for conditional entropy modeling, and (iii) an entropy coder, e-g an arithmetic coder (AE/ AD). The imput image x is projected to a set of latent features, followed by product quantization to yiekd quantization indices q. The arithmetic coder encodes (and decodes) into a bitstream b in a lossless manner. The elements in q are spatially split into groups, as detailed Figure Conditional Entropy Modeling. Our model estimates the conditional probabilities of the discrete indices in S steps. Every step, a subset of the tokens & is selected using the quincunx pattern. Our MIM transformer estimates M(gilessi) and passes it to the Arithmetic encoder as CDF, effectively reducing the lossless compression cost.
image representation with a prefix that encodes the conditioning text. This has yielded impressive results by scaling the model capacity and training data to tens or hundreds of million text-image pairs (Ding et al., 2021; Gafni et al., 2022; Ramesh et al., 2021). A fundamental limitation of autoregressive generative models is that they sample data sequentially, requiring separate non-parallel evaluation of the predictive (transformer) model to sample cach token. To alleviate this, several data items can be sampled independently in parallel, conditioning on all previously sampled tokens. This has been leveraged to speed-up pixel-CNNs for small images and video by two to three orders of magnitude (Reed et al., 2017). More recently, MaskGIT (Chang et al., 2022) and follow-up work (Lezama et al., 2022) explored this for generative models of VQ-VAE representations, and find that similar or better sample quality is obtained by parallel sampling of image patch subsets in few steps, reducing the generation time siguificantly. Despite their success for image synthesis we are not aware of the use of such models for image compression in earlier work.
3 Product Quantized Masked Image Modeling
This work takes a step towards to closing the gap between neural image compression and image generation methodologies. We revisit vector quantization for image compression and propose entropy model inspired by masked image modelling. Our compression pipeline, depicted in Figure 2, relies on three neural networks:
1. The Encoder network ::X' - Z maps input images x E X to quantized representation E Z.
The Mask Image Model (MIM) compresses the quantized representations without loss of informa- tion. This network is involved both on the compression and decompression side.
3. The Decoder network G: X' produces an estimate x G(ā(x)) of the original image x.
We now detail the architecture, in particular our PQ proposal, of the image model that we employ in the statistical lossless coding, as well the training scheme.
3.1 High-level architecture: PQ-VAE
High-level architecture. We follow recent work on discrete generative image models for the design of image encoder and decoder (Chang al., 2022; Esser al., 2021; Oord et al., 2017; Razavi et al., 2019; Yu
4

---

Published in Transactions on Machine Learning Research (03/23)
et al., 2021). The encoder E : X' - R wxhxd takes an RGB image x of resolution WxH as input and maps it to latent representation [ with d feature channels and a reduced spatial resolution wxh, downsampling the input resolution by a factor f = W /W/w- = H /h. The T' = wxh elements of the latent representation Exx) are quantized with a vector quantizer Q(-) to produce the quantized latent representation z = Q(E(x)) = É(x), where each element in E(x) is replaced with its nearest cluster center. The decoder G uses the quantized latents z to reconstruct the image.
Product Quantization. In VQ-VAE, the quantizer Q is simply an online k-means quantizer that produces quantization indices from veal-valued vectors. We denote by q E {I...., V)" the map of quantization indices indicating for each element of z which of the V centroids is selected. The higher V the more precise is the approximation z, leading to higher bit rates. For instance, assuming that indices are coded with naive coding scheme (see next section), the bit rate is doubled when moving to V = 256 to V = 65536 rontsoids.
However, scaling the number of centroids K is possible up to thousands of centroids, but beyond that it is com- putationally prohibitive. Additionally, it is challenging to train large codebooks where each centroid has very low probability of being updated. To address this problem, we replace the online k-means quantizer by a product quantizer (Jégou et al., 2010) (PQ): the latent vector z of
Bits per pixe (bpp)
PQ-MIM
VQ-MIM
31
PSNR
30
is split into M subvectors as z = [x],..., v²....,,"! dimension /dd. Each subvector is quantized by a distinet quantizer having V, quantization value. The set of quantiz- ers implicitly defines a vector quantizer in the latent space with V =V distinct centroids. Hence, we can easily de- fine very large codebooks without the computational and optimization problems mentioned above, because both the assignment and learning are marginalized over the different subspaces. Empirically, we observe in Figure 3 that PQ provides a better scaling behaviour for higher rates compared to VQ whose codebook size needs to grow exponentially to achieve the same rates.
29
28
0.015
0.05
0.1
0.15
Figure 3: PQ and VQ comparison. While VQ pro- vides a comparable performance to PQ for extremely low rates where the codebook size is small, PQ ex- hibits better scaling behaviour for higher rates. For VQ we only vary the codebook size, while for PQ we fix the codebook size to 256 and vary the number of sub-vectors between 2 and
Neural network. Without loss of generalily, we choose all neural nelwork models to be identical. This is not requirement but this offers the property that the encoder and decoder have identical complexities, and that the memory and compute peaks are identical. More specifically we choose a cross-variance transformer (EI- Nouby et al., 2021b) (XCIT), whose complexity is linear with respect to image resolution. In contrast, standard vision transformers (Dosovitskiy et al., 2021) (VTT) are quadratic in the image surface, which is prohibitive for high resolution images that can typically require strong compression. We point out that recent work has shown that Swin-Transformest (Zhu et al., 2021) could be a compelling choice as well in the context of image compression. Formally they also have a quadratic complexity, but this is amortized by the hierarchical structure of this architecture.
3.2 Image entropy model
In this section we present PQ-MIM. The objective is to compress the discrete representations q without loss of information, producing a bitstream of the compressed representation that can be transmitted or stored. During the decoding stage, we invert the aforementioned lossless compression. This model can be regarded as the VQ/PQ- VAE counterpart of adaptive contextual arithmetic coders, like EBCOT (Taubman, 2000) or CABAC (Richardson, 2004), proposed in early compression standards, in that it couples a conditional probabilistic model with an arithmetic coder.
Lossless compression. A naive manner for lossless compression of the discrete image codes == {g))T= is to use fixed-length codes. In that case, each code word is assigned to a unique binary representation of equal length, resulting in flogz v1 bits per element q. This approach is computationally very efficient as fixed-length codes are not model-based, and as such do not require likelihood estimation, and because all
5

---

Published in Transactions on Machine Learning Research (03/23)
p(q1)
p(42441)
p493992 91)
p(q 9s. 92. 91)
pl9s44 43 92 q1)
R
R
Figure 4: Illustration of PQ-MIM with a quincunx pattern. We employe the quincuux pattern both on the encoder and decoder side. Each panel represents one of five stages in which en/de-code a of tokens in parallel using probability model Ps implemented by a transformer with parameters 8,. The transformer predicts the tokens in 9s displayed grayscale and marked by "?", and takes imput the preceding groups of tokens I.....:9-- that are displayed in color. The distribution provided by this neural network is fed to an arithmetic en en/de-coder.
codes are of equal length by construction, the computation is perfectly parallelizable. However, theoretically this coding scheme could be Shannon optimal only if codewords uniformly and independently distributed. Those assumptions are not met in practice due to the architecture choices we have made previously: k-means does not produce uniformly distributed indices except in singular cases (Gray & Neuhoff, 1998). More details about losseless compression are covered in Appendix A.
Entropy model. To improve the bitrate, we hence rely on an entropy coder, which provides an inverse pair of functions, enc, and decp, achicving near optimal compression rates on sequences of symbols for any distribution p. The better p matches the (unknown) underlying data distribution, the belter the compression rate (Cover, 1991). Generally, more powerful generative models will ensure better compression performance.
Fully autoregressive generative models p(q) = I p(u (xx) are powerful (Ding el al., 2021; Esser et al., 2021; Gafni et al., 2022; Ramesh et al., 2021; Yu et al., 2021), however, they are inconvenient in that the likelihood estimation for this type of model not trivially parallclizable: cach patch index must be processod sequentially as is used to condition subsequent patch indices. Thus, similar to prior works (Chang el al., 2022; Reed et al., 2017) we propose masked image model, which we to predict the image patch indices in several stages. Specifically, partition into S subsets qı, 9....... qs of patch indices, that we refer to as tokens by analogy to language modelling:
q q.
s=1
We model the elements in each subset conditionally independent given all preceding groups:
p(q)= p p (q.lacci
(2)
pq.11ew:0.)) =
p p 92 9.-10.))))
(3)
9.
Since p(qı) is not conditioned on any previous elements, it fully factorizes over di E qr, and we model it the marginal distribution over the vocabulary observed on the training data. The non-trivial conditional distributions p pqqll<ai (,,) for 2 2 are modeled using transformer networks, which have few inductive biases and have been successful across many tasks, including image generation. An overview of the MIM entropy model illustrated in Figure? 2.
Amortized encoding and decoding. From a computational point of view, our proposal allows for the compression (resp. decompression) to procoed in S stages. In cach stage we ancod//ocood the set q. of tokens conditioned on the groups qI..... q--1 accoded/decoder in preceding stages, but independently among the tokens in the set q.- This allows for paralellization among the elements in each subset qa and

---

Published in Transactions on Machine Learning Research (03/23)
(a) TECNICK PSNR
(b) TECNICK AIS-SSIM
per pixel (bop)
Bits per pixel (bpp)
38
PQ MIM
22
PQ MIM
Scale Hyperprior
Mean Scale Hyperprior
36
Scale flyperprior
Scale Hyperprior
34
JPEG 2000 (Open]PEG)
18
JPEG 2000 (Open)PEG)
Webp
Webp
32
16
12
0.03
0.05
0.20
0.60
0.03
0.05
0.20
0.60
(c)
KODAK PSNR
(d) KODAK MS-SSTM
Bits pixel (bop)
Bits pixel (bpp)
PQ-MIM
PQ-MIM
34
Cheng et. al
Cheng al
Hyperprior
Mean Hyperprior
18
32
Scale Hyperprior
Scale Hyperprior
BPG
16
BPG 444
JPEG 2000 (Open)PEG)
2000 (OpenjPEG)
PSNR
Webp
Webp
12
0.03
0.60
0.05
0.60
Figure 5: Rate-Distortion performance for Tecnick and Kodak datasets. We report PQ-MIM PSNR and MS-SSIM performance for various operaling points. PQ-MIM provides competitive performance, particularly for MS-SSIM, compared to standard codecs such as JPEG 2000 (Taubman & Marcellin, 2012) and BPG (Bellard) as well as recent neural methods (Cheng et al., 2020; Minnen et al., 2018; Ballé et al., 2018).
requires strictly S forward passes through the model independent of the image size, rather than T sequential forwards passes for fully autoregressive models.
Quincunx partitioning. In practice, we typically S stages. We have explored different patterns to partition the T tokens over the S stages. In particular we consider the "quincunx" regular grid pattern, where in each stage we double the number of tokens to predict, see Figure 4 for an illustration. This multi-level refinement was previously explored for image compression in the context of lifting schemes designed with oriented wavelots (Chappelier & Guillemot, 2006). our experiments we contrast this partitioning with alternative ones with other patterns and subset cardinalities (Figure 9).
3.3 Training the PQ-MIM
Reconstruction objective and training. The goal of lossy image compression is to match the image and its reconstruction as closely as possible according to some distortion metric. In this paper, we train our model to reduce the image distortion and the quantization loss using the following objective:
I. TRec n TPQ
(4)
7

---

Published in Transactions on Machine Learning Research (03/23)
(a) FID
(b) KID
(c) MS-SSIM
per pixe! (bpp)
Bits per (bpp)
Bits per pixel (bpp)
PQ-MIM
PQ-MIM
PQ-MIM
17
1.0
HIFIC
3.5
3.0
0.8
MS- DSS-SSIM
16
0.6
15
2.0
0.4
1.5
0.15
0.20
0.25
0.30
0.35
0.2
0.15
0.20
0.25
0.30
0.35
0.15
0.20
0.25
0.30
0.35
Figure 6: Perceptual training and evaluation using CLIC 2020 test-set. Performance of our adversarially
trained PQ-MIM w.r.t perepetual metries compared to HiFIC Mentzer al. (2020). PQ-MIM provides stronger performance on FID, KID and MS-SSIM across all reported operaling poinls.
Following VQ-VAE, our quantization objective Lpq consists of embedding and commitment losses, averaged over the M different PQ sub- vectors For the distortion loss Lrees present two setups where use different types of distortion measures:
MSE & MS-SSIM. Typical distortion measures used in the majority of the neural compression literature such mean squared error (MSF) or multi-scale structural similarity (Wang et al., 2003) (MS-SSIM). For this setup, the model is trained solely using one distortion measure at a time.
Tooo (x, R. G) = (x x)
(5)
Perceptual measures. Alternatively, we report a setup where we utilize perceptual objectives such as LPIPS (Zhang et al., 2018) and adversarial training (Goodfellow et al., 2014) to enhance psycho-visual image quality. The distortion loss defined as:
Roo (x, G) LMSE(x, x) a (x, x) + y LAd (É,
(6)
where a and y are weighing coefficients and the adversarial loss Ladv is defined as:
Laav (,, G, D) Ex[In D(x)] Eg[In(1 D()))],
where D--) is the discriminator and Eg denotes the expectation over sampled uniformly from the training set. Similarly Eg denotes the expectation over reconstructed training images. Note that, unlike the MSF and perceptual losses, the adversarial loss does not compare individual images and their reconstructions, but aims to match the distributions of original images and their reconstructions.
Training the entropy model. Our MIM module is an XCIT transformer that accepts T tokens as input representing the image patches. During training, we randomly mask a set of tokens by sampling from a uniform distribution (O0 1). The masked tokens are replaced with a mask embedding vector, while the observed token indices are mapped to their corresponding continuous representation using an embedding look-up table. The MIM module outputs a context vector for every masked token which in turn is passed M linear heads, representing the different. PQ sub-vector indices, followed by softmax yield a distribution (gg.les) The module is trained using a standard cross-entropy objective. While we train the autoencoder and the MIM modules simultancously, we do not backpropagate gradients from the MIM module to the encoder E or the quantization parameters, so the two components can be trained separately in sequence.
4 Experiments
We first present our experimental setup in Section 4.1 and the present our results in Section 4.2. We provide ablation studies in Section 4.3. We will share our code and models.
8

---

Published in Transactions on Machine Learning Research (03/23)
Original
LOVE rou
PQ-MIM
LOVE roU
201 bytes
193 bytes
203 bytes
168 bytes
202 bytes
Figure Extreme Image Compression. PQ-MIM exhibits non-trivial compression performance at the extreme
compression regime (e.g. 0.03 bpp), leading to compressed image codes that can fit in short tweet (280 characters).
4.1 Experimental setup
Rate-distortion control. For all our experiments we fix the codebook size = 256 and only vary the number of sub-vectors M (2, 4, 6) for two different down-sampling factors {8. 16).
PQ-VAE implementation details. Our PQ-VAE training uses the straight-through estimator (Bengio et al., 2013) to propagate gradients through the quantization bottleneck. As for the quantization, the elements of the latent representation z are first linearly projected to low dimensional look-up vector (dim=8 per sub-vector) followed by & normalization, following Yu et al. (2021). We train our model using ImageNet (Deng et al., 2009) for 50 epochs with a batch size 256. We use an AdamW (Loshchilov & Hutter, 2019) optimizer with a peak learning rate of 1.10"a, weight decay of 0.02 and B3 0.95. We apply linear warmup for the first 5 epochs of training followed by a cosine decay schedule for the remaining 45 epochs to a minimum learning rate of 5.10 Unless mentioned otherwise, for all experiments, the encoder and decoder use an XCTT-L6 with 6 layers and hidden dimension of 768. We use sinusoidal positional embedding (Vaswani et al., 2017) such that our model can ficxibly operate on variable sized images.
For the results reported in Figure 5, the models are trained using solely MSE (17 0.5) or MS-SSIM (1= 10.0) distortion losses for their corresponding plots. As for the models trained with perceptual objectives (Figure 6 and Table 1), they are trained with weighted sum of MSE, LPIPS (a 1) and adversarial loss (? = 0.1). We use a ProjectedGAN Discriminator (Sauer et al., 2021) architecture. The perceptual training is initialized with an MSE only trained checkpoint and trained for 50 epochs using ImageNet with a learning rate of 10-4 and weight decay of 5.10 Similar HiFiC (Mentzer et al., 2020), we freeze the encoder during the perceptual training. Additionally, we find that clipping the gradient norm to maximum value of 4.0 improves the training stability. Our discriminator takes only the decoded image as an input and does not. rely on any other conditional signal.
MIM implementation details. Our MIM module is an XCIT-L12 with 12 layers and embedding dimension of 768. MIM is trained simultaneously with the PA-VAF, but the gradients are not backpropagated to the encoder or the quantizer parameters. The PQ indicios q are split into inputs and targets for the MIM model as defined by the quincunx partitioning pattern. By default we use S = 5 stages. All stages are processed with the same MIM model. The masked patches are replaced by learnable "mask" token embedding. The loss is computed only for the masked patches. Since every token is assigned M PQ indices, the output of the MIM transformer is passed to M separate linear heads to predict M softmax normalized distributions over their corresponding codebooks. The marginal distribution of the codebook is computed as a normalized histogram over the ImageNet training set. For entropy coding, we use the implementation of the torchac' arithmetic coder.
"httpa://githin http://gggttbbh.///te. con/fab- jul/torchaaee
9

---

Published in Transactions on Machine Learning Research (03/23)
Table 1: Discriminator Architecture. We investi- gate multiple discriminator including StyleGAN (Karras et al., 2019), ProjectedGAN (Sauer et al., 2021) and UNet (Schönfeld al., 2020).
Table 2: Different MIM masking policies. PQ-MIM with quincunx pattern with 5 steps reduces the bpp signifi- cantly (27%). Additionally, PQ-MIM orders of magni- tude cheaper in terms of FLOPs compared to an autore-
Discriminator FID 4
KID 4
MS-SSIM f
gressive raster order masking pattern.
Masking policy
#steps
bpp
MACs/Pixel (M)
None
26.1
1.2x-2
15.3
StyleGAN
3.57
4.8x10-4
13.3
Marginal baseline
1
0.512
0.69
ProjectedGAN
3.69
6.6x10-4
14.1
Raster order
T
ODM
24.4x10*
UNet
3.87
6.7x10-4
13.9
Quincunx
5
0.373
6.66
Datascts. We train our models using ImageNet (Deng et al., 2009). For data augmentation, we apply random resized cropping to 256 x 256 images and horizontal flipping. For evaluation and comparison to prior work, we use KODAK (Kodak, 1993) and TECNICK (Asuni & Giachetti, 2014) datasets for PSNR and MS-SSIM. Moreover, we compute the perceptual metries (FID (Heusel et al., 2017), KID (Binkowski et al., 2018)) for perceptually trained models using the CLIC 2020 test-set (Toderici et al., 2020) (428 images) using the same patch cropping scheme detailed by Mentzer al. (2020).
Baselines. We compare several existing neural compression baselines: the scale hyperprior model (Ballé et al., 2018), mean & scale hyperprior (Minnen et al., 2018), GMM hyperprior (Cheng et al,, 2020), and HiFIC (Mentzer et al., 2020). Among the non-neural codecs, we compare to the popular BPG (Bellard), WebP, and JPEG2000.
4.2 Main experimental results
Comparison to existing (neural) codecs. We compare PQ-MIM to other approaches across a wide range of bitrates in Figure 5². Note that in our evaluation we consider bitrates that are an order of magnitude lower than what is typically studied in the literature: most previous studics were limited to 0.1 bpp and above, see e.g (Ballé et al., 2018; Cheng et al., 2020; Minnen et al., 2018). The extremely low bitrates we consider make it possible to transmit a 256x256 image in an SMS or a tweet (280 characters) as shown in Figure 7. PQ-MIM achieves a strong and competitive performance for both KODAK and TECNICK datasets, outperforming all prior neural and standard codecs with the exception of GMM hyperprior (Cheng et al., 2020). We observe that PQ-MIM is particularly strong for low rates, making a good fit for extreme compression scenarios. Moreover, PQ-MIM exhibits a partieularly strong MS-SSIM performance which was designed to model the human visual contrast perception (Wang et al., 2004; 2003).
Perceptual metrics comparison. In Figure 6, we compare PQ-MIM to HiFiC (Mentzer et al., 2020) in percept quality measures like FID and KID as well as MS-SSIM. HiFiC is based on the mean & scale hyperprior model (Minnen et al., 2018), but adds adversarially trained discriminator model to improve the perceptual quality of the image reconstructions. PQ-MIM, with perceptual training, achieves a strong performance for all reported metrics, outperforming HiFiC for all operating points.
4.3 Analysis and Ablations
Model size and architecture. In Figure 8, we analyze the effect of using XCIT of different capacities for the autocncoder with respect to rate-distortion tradc-off. We observe that the performance improves with higher capacity autoencoders, but their is a diminishing return with further increase in capacily. For all our experiments we use an XCTT-L6 since it achieves the best performance.
Masking patterns. In Table 2, we compare to predicting tokens one-by-one autoregressively in a raster- scan order, the same pattern used in VQ-VAE based generative image models such as DALL-E (Ramesh et al., 2021) and VQ-GAN (Esser al., 2021). In contrast to PQ-MIM, raster-scan models require causal attention, which makes XCTT not a good fit. We use standard ViT model instead. However, due to the quadratic complexity of VIT and the high resolution of images typically used for evaluation of compression
2 All results for the baselines are reported using the authors official repositories
For example, a bitrate of 0.03 yields 256 256 0.03/8 246 bytes for 256x256 image.
10

---

Published in Transactions on Machine Learning Research (03/23)
Bits per pixel (bpp)
XCIT-T6
Quincunx
XCIT-56
0.43
Confidence Double Schedule
XCTT-L6
Bits per pixe! (bpp)
0.42
Confidence Linear Schedule
0.41
PSNR
0.40
30
0.39
28
0.38
0.05
0.20
0.60
Steps
Figure 8: Autoencoder capacity. The RD performance of different pocoder/decoder capacities. We use the same model trunk for (i) the encoder; (fi) the PQ-MIM entropy model; ; and (iii) the decoder. Increasing the model size from XCTT-16 model (3.5M params) to XCTT-L6 (47M params) increases the performance by typically +0.8dB.
Figure 9: MIM number of steps. In addition to quineuux, we explore using the prediction confidence as the patch masking policy following MaskGiT (Chang et al., 2022). We test a linear and doubling schedules. Quincuux provides higher rate saving, even compared to confidence policy with a longer schedule.
method (e.g. TECNICK), our autoregressive variant consistently exceeded the memory limits, even when using A100 GPUs with 40GB memory. Moreover, raster scan fully-autoregressive models results in extremely expensive FLOP count since it needs T separate forward passes per image. On the other hand, our stage-wise MIM with quincunx pattern requires evaluations, and does not scale with the image resolution as is the case for raster, making it a more practical solution.
Number of prediction stages. We compare the quineunx masking pattern with masking based on confidence score following MaskGiT (Chang et al., 2022) patch selection procedure for image generation. In the latter case, at given step, pick the patches to transmit dynamically based their confidence score. The confidence is defined as the maximum across the probabilities assigned over the vocabulary by the model." For the quincunx and confidence based masking policy we use the same 5-stage scheme, in which the number of tokens in subsequent groups doubles in size. We ablate the number of prediction stages for the quincunx and sefidenco-based sampling. For the latter we consider two options: (i) a lincar schome where each group of tokens contains (approxinately) the same number of tokens; and (ii) a doubling scheme where each subsequent group of tokens is double the size of the previous group, as is also used for the quincunx pattern. Every point on the curves in Figure 9 corresponds to the bitrate when encoding, gooodinggdeoodiinn with a given number of steps. For example, S = 2 steps means we have only two steps each encoding/ decoding 50% of the patches. For the 3-steps doubling schedule the groups have sizes of 1/4, 1/4, 1/2, and so on. For all three patterns the bitrate monotonically decreases with the number of steps, as more tokens can be predicted from larger contexts.
On the one hand, using confidence-baeee based masking patterns, doubling and linear schemes lead to mostly comparable bitrates for the same number of steps, with further improvement for linear with more steps however with diminishing return. On the other hand, quincunx provides a stronger reduction ratio with strictly 5 steps, even when compared with confidence- based masking with higher mumber of steps.
Predicting missing patches. When encoding/decon decoding the discrete image representation q, we reduce the bitrate with our models by their ability to predict the remaining tokens given those of preceding stages. To illustrate the predictive ability of our model, we consider an experiment where we remove a subset of the tokens (sampled randomly), and use our model to fill in the missing patches by conditioning on the observed ones. We then use the PQ-VAE decoder to decode the discrete latent codes including the filled in ones. In Figure 10 we show the results obtained when removing 10%, 20%, 30% and 50% of the patches. Our MIM is able to model redundancies among the patches, which corroborates our findings of its compression ability.
Note that for decoding the same confidence score can be used to identify the group tokens decode.
11

---

Published in Transactions on Machine Learning Research (03/23)
No Patch Dropping
10% Drop Rate
20% Drop Rate
30% Drop Rate
50% Drop Rate
ALC
STLRBEGE
Figure 10: PQ-MIM can operate on a partial set of transmitted tokens. Since PQ-MIM is compatible with generation, the conditional entropy model can be repurposed to predict the PQ codes for missing parts in image. We show results for different dropping rates transmitted tokens. PQ-MIM exhibits strong inpainting abilities. Even for the extreme case where half of the image patches are dropped, PQ-MIM can still retain large percentage of the original image structure and details.
4.4 Limitations
Image compression, and compression visual data in general, is an important technology to scale the distribution of visual data. This is ever more important to cope with the growing quantity of visual data that is streamed in the form of video and for augmented and virtual reality applications. Compression is also critical to allow users with low-bandwidth connections to benefit from applications relying sharing of image, video, or virtual reality data.
Caveats of learned neural compression models: biases and performance. As with any machine learning model, potential biases in the training data may be transferred into the model via training. In our setting, this can alfect the autoencoding reconstruction abilities for content. under-represented in the training data, well as the compression abilities of the model for such data. Such biases should be assessed before deployment of the model. Beyond rate-distortion tradc-offs, important evaluation dimensions include the energy and latency performance of compression models. Current neural compression methods, including ours, need to be further optimized to be competitive with existing codecs on these criteria.
Specific limitations of our approach. In our work, we specifically focus on a high compression regime, with compression rate lower than 0.6 bit per pixel. This a favorable case for our approach: it benefits from the generative model capability inherited from the VQ-VAE. Compared to VQ. PQ allows for higher rates, but becomes comparatively less effective when increasing the number of subquantizers. As one can deduce from the slopes of the rale-distortion curves in Fig. 5, SQ methods like scale hyperprior exhibits relatively stronger performance for the high end of bit-rates.
5 Conclusion
In this paper, we have revisited vector quantization for neural image compression. We introduced product- quantization variants of VAEVAE and shown that it has a better scaling properties in terms of bit-rate. Additionally, we introduced novel conditional entropy model based on masked image modeling. We have shown that combined with the qunicunx partitioning pattern, PQ-MIM provides strong reduction in bit-rate. PQ-MIM exhibits a compelitive performance for PSNR and MS-SSIM metric compared to strong neural compression baselines. Furthermore, when we train PQ-MIM using perceptual losses, provides a strong performance on multiple metrics (e.g. FID, KID and MS-SSTM) compared the strong bascline HiFiC. Finally, we have shown that PQ-MIM can operate in hybrid compression/geneeeee /geneaation mode where can fill the gaps for non-transmitted patches.
12

---

Published in Transactions on Machine Learning Research (03/23)
References
Eirikur Agustsson, Fabian Mentzer, Michael Tschannen, Lukas Cavigelli, Radu Timofte; Luca Benini, and Luc van Gool. Soft-to-hard vector quantization for end-to-end learning compressible representations. In NewrIPS, 2017. 2,
Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, and Luc Gool. Generative adversarial networks for extreme learned image compression. In ICCV, 2019. 2, 3
Alexander Alemi, Ben Poole, lan Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing a broken ELBO. In ICLR, 2018. 2
Mare Antonini, Michel Barlaud, Pierre Mathieu, and Ingrid Daubechies. Image coding using wavelet transform.
IEEE Transactions Image Processing, 1992. 1
N. Asuni and A. Giachetti. TESTIMAGES: large-scale archive for testing visual devices and basic image processing algorithms (SAMPLING 1200 RGB sct). In STAG: Smart Tools and Apps for Graphics, 2014. URL htttps://sourceffos zip. 10
Johannes Ballé, Valero Laparra, and Eero P. Simoncelli. End-to-end optimized image compression. In ICLR, 2017. 3
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. In IOLR, 2018. 3, 7, 10
Hangbo Bao, Li Dong, and Furu Wei. BETT: BERT pre-training of image transformers. In ICLR, 2022. 2 Fabrice Bellard. BPG image format. URL https::/bellarr.orgg///g
7, 10
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint, arXiv:1308.3332 2013. 9
Mikolaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Denystifying MMD GANs. ICLR, 2018. 10
Leon Bottou, Patrick Haffner, Paul G Howard, Patrice Simard, Yoshua Bengio, and Yann Le Cun. High
quality document image compression with "djvu". Journal of Flectronic Imaging, 1998. 2
Joan Bruna, Pablo Sprechmann, and Yann LeCun. Super-resolution with deep convolutional sufficient.
statisties. In ICLR, 2016. 1
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. MaskGIT: Masked generative image
transformer. In CVPR, 2022. 2, 4, 6, 11
Vivien Chappelier and Christine Guillemot. Oriented wavelet transform for image compression and denoising. IEEE Transactions on Image Processing, 2006. 7
Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learned image compression with discretizod
Gaussian mixture likelihoods and atlention modules. In CVPR, 2020. 3, 7, 10
Thomas M Cover. Elements of information theory. John Wiley Sons, 1991. 6, 17
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, ImageNet: A large-scale hierarchical image database. In CVPR, 2009. 9, 10
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. CogView: Mastering text-to-image generation via transformers. NeurIPS, 2021. 4, 6
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. PAM1, 2015. 1
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
13

---

Published in Transactions on Machine Learning Research (03/23)
Emilien Dupont, Adam Goliñskki Milad Alizadeh, Yee Whye Tch, and Arnaud Doucet. COIN: COmpression
with Implicit Neural representations. In ICLR Neumal Compression Workshop, 2021. 3
Emillen Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Goliñkii Yee Whye Teh, and Arnaud Doucet. COIN++: Data agnostic neural compression. TMLR, 2022. 3
Alaacldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale dalasets necessary for self-supervised pre-training? arviv preprinl arXim:21122 107700, 2021a. 2
Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Tvan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeck, et al. XCIT: Cross-covariance image transformers. In NeurIPS, 2021b. 5
Patrick Esser, Robin Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021. 2, 3, 4, 6, 10
Brendan J Frey. Bayesion nelworks for pallern classificalion, data compression, and channel coding. PhD thesis, University of Toronto, 1998. 2
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In ECCV, 2022. 4, 6
I. Goodfellow, J. Pougel-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NewrIPS, 2014. 2, 8
Robert M. Gray and David L. Neuhoff. Quantization. IEEE Trans. Information Theory, 1998. 1, 6
Karol Gregor, Frederic Besse, Danilo Jimenez Rezende, Ivo Danihelka, and Daan Wierstra. Towards conceptual
compression. In NearIPS, 2016. 2
Dailan He, Yaoyan Zheng, Baocheng Sun, Yan Wang, and Hongwei Qin. Checkerboard context model for efficient learned image compression. In CVPR, 2021a. 3
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick. Masked autoencoders scalable vision learners. In CVPR, 2021b. 2
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to local Nash equilibrium. In NewrIPS, 2017. 10
H. Jégou, C. Schmid, H. Harzallah, and J. Verbeek. Accurate image search using the contextual dissimilarity measure. PAMI, 2010. 2, 5
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019. 10
D. Kingma and M. Welling, Auto-encoding variational Bayes. In ICLR, 2014. 2
Florian Knoll, Kerstin Hammernik, Chi Zhang, Steen Moeller, Thomas Pock, Daniel K Sodickson, and Mehmet Akcakaya. Deep-learning methods for parallel magnetic resonance imaging reconstruction: A survey of the current approaches, trends, and issues. IEEE Signal Processing Magazine, 2020.
Eastman Kodak. Kodak lossless true color image suite (photoed ped0992), 1993. URL http://rOk.uu
praphics/kodak. 10
H. Larochelle and I. Murray, The neural auloregressive distribution estimator. In AISTATS, 2011. 2
Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image super-resolution using generative adversarial network. In CVPR, 2017. 1
José Lezama, Huiwen Chang, Lu Jiang, and Irfan Essa. Improved masked image generation with token-critic. In ECCV, 2022. 4
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 9
Xiaotong Lu, Heng Wang, Weisheng Dong, Fangfang Wu, Zhonglong Zheng, and Guangming Shi. Learning a deep vector quantization network for image compression. IEEE Access, 2019. 3
14

---

Published in Transactions on Machine Learning Research (03/23)
David J. C. MacKay. Information Theory, Inference, and Learning Algorithms. Cambridge University Press, 2003. 17
Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, and Lue van Gool. Conditional probability models for deep image compression. In CVPR, 2018. 3
Fabian Mentzer, George D. Toderici, Michael Tschannen, and Eirikur Agustsson. High-fidelity generative image compression. In NewrIPS, 2020. 2, 3, 8, 9, 10
David Minnen and Saurabh Singh. Channel-wist autoregressive entropy models for learned image compression. In ICIP, 2020. 3
David Minnen, Johannos Ballé, and George D. Toderici. Joint autoregrossive and hicrarchical priors for learned image compression. In NewrIPS, 2018. 3, 7, 10
A. den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu. Wavenet: generalive model for raw audio. In ISCA Speech Symlesis Workshop, 2016a. 2
A. van den Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, and K. Kavukenoglu. Conditional image generation with PixeICNN decoders. In NewrlPS, 2016b. 2
A. van den Oord, O. Vinyals, and K. Kavukeuoglu. Neural discrete representation learning. In NewrIPS, 2017. 2, 3, 4
Yichen Qian, Ming Lin, Xiuyu Sun, Zhiyu Tan, and Rong Jin. Entroformer: A transformer-basel entropy model for learned image compression. ICLR, 2022. 18
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Tlya Sutskever. Zero-shot text-to-image generation. In ICML, 2021. 4, 6, 10
Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. In NewrIPS, 2019. 2, 3, 4
S. Reed, A. den Oord, N. Kalehbrenner, S. Gómez Colmenarejo, 7. Wang, D. Belov, and N. de Freitas. Parallel multiscale autoregressive density estimation. In ICML, 2017. 4, 6
D. Rezende and S. Mohamed. Variational inference with normalizing flows. In ICML, 2015. 2
Iain Richardson. H. 264 and MPEG-4 video compression: video coding for next- gentegennrotion multimedia. John Wiley & Sons, 2004. 5
Oren Rippel and Lubomir Bourdev. Real-time adaptive image compression. In ICML, 2017. 3 Robin Rombach, Andreas Blatimann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
T. Salimans, A. Karpathy, X. Chen, and D. Kingma. PixeICNN++: Improving the PixeICNN with discretized logistic mixture likelihood and other modifications. In ICLR, 2017. 2
Shibani Santurkar, David Budden, and Nir Shavit. Generative compression. In 2018 Picture Coding Symposium (PCS), 2018. 3
Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected GANs converge faster. In NewrIPS, 2021. 9, 10
Edgar Schönfeld, Bernt Schiele, and Anna Khoreva. A U-Net based discriminator for generative adversarial networks. In CVPR, 2020.
GL Sicuranza, G Romponi, and Stefano Marsi. Artificial neural network for image compression. Electronics letters, 1990. 2
N Sonehara. Image data compression using a neural network model. In Newral Networks, 1989.
Yannick Strüpler, Janis Postels, Ren Yang, Luc van Gool, and Federico Tombari. Implicit neural representations for image compression. In ECCV, 2022.
Gary J Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (HEVC) standard. IEEE Transactions on circuits and systems for video technology, 2012.
15

---

Published in Transactions on Machine Learning Research (03/23)
David Taubman. High performance scalable image compression with EBCOT. IEEE Transactions on Image Processing, 2000. 1,
David Taubman and Michael Marcellin. Jpeg2000 image compression fundamentals, standards and practice: Image compression fundamentals, standards and practice. In Springer Science and Business Media, 2012. 7
Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferene Huszár. Lossy image compression with compressive autoencoders. In ICLR, 2017. 3
Chunwei Tian, Lunke Fei, Wenxian Zheng, Yong Xu, Wangmeng Zuo, and Chia- Lin. Deep learning image denoising: An overview. Neural Networks, 2020.
George Toderici, Wenzhe Shi, Radu Timofte, Johannes Balle Lucas Theis, Eirikur Agustsson, Nick Johnston, and Fabian Mentzer. Workshop and challenge learned image compression (clic2020), 2020. URL
/twp:////wwwwwee/ http::/www.compresion.co cc. 10
A. Vaswani, N. Shazeer, N. Parmar, J. Usukoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In NewrlPS, 2017. 3, 9
Ge Wang, Jong Chul Ye, and Bruno De Man. Deep learning for tomographic image reconstruction. Nature Machine Intelligence, 2020.
Zhihao Wang, Jian Chen, and Steven C. H. Hoi. Deep learning for image super-resolution A survey. PAMI, 2021.
Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In Asilomar Conference on Signals, Systems & Computers, 2003. 8, 10
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004. 10
lan H Witten, Radford M Neal, and John G Cleary, Arithmetic coding for data compression. Communications
of the ACM, 1987. 1
Yueqi Xie, Ka Leong Cheng, and Qifeng Chen. Enhanced invertible encoding for learned image compression.
In ACM MM, 2021. 18
Yibo Yang, Robert Bamler, and Stephan Mandt. Improving inference for neural image compression. In NewrIPS, 2020. 3
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, and Yonghui Wn Jason Baldridge. Vector-quantized image modeling with improved VQGAN. In ICLR, 2021.
2, 4, 9
R. Zhang, P. Isola, A. Efros, E. Shechtman, and O. Wang. The unreasonable elfectiveness of deep features a perceptual metric. In CVPR, 2018. 8
Yinhao Zhu, Yang Yang, and Taco Cohen. Transformer-based transform coding. In ICLR, 2021. 5
16


Reviews:
[{"review_id": "GXDOxrYTe2", "reviewer": "Reviewer_VDgp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\nThe paper introduces several new ideas for improving the entropy modeling in non-linear transform coding, and does a fairly comprehensive job evaluating different architectural choices of the proposed entropy model. The resulting method offers competitive compression performance compared to SOTA methods, as well as visually pleasing progressive reconstructions. \n\n**Weaknesses**:\n\nIt's a bit difficult to understand the effect of the various aspects of the proposed approach and how they are justified.\nSpecifically, there is a lack of clear motivation for preferring vector/product quantization instead of the baseline scalar quantization approach -- other than the appeal to the success of VQ-VAE on image modeling. Isn't it possible to still apply the transformer-based entropy model with scalar-quantized latents (e.g., as in Mentzer et al., 2022)? It'll be helpful to understand the effect of VQ/PQ v.s. SQ separately from the newly introduced entropy model. \nSimilarly, the proposed Quincunx-patterned entropy model appears similar to checkerboard-patterned entropy model (He et. al., 2021), with largely the same motivation; therefore a discussion/comparison seems appropriate. \nFinally, I'm not sure if the rate-distortion-realism comparison to HiFiC in Fig. 6 is done 100% correctly. My understanding is that for comparing R-D-P tradeoffs, the appropriate procedure is to fix one of the axes and compare the remaining two; e.g., training a couple of models for low/mid/high rate regimes for the proposed method, and compare the D-P tradeoff with HiFiC at *matching bit-rates* (as done in the HiFiC paper).\n\nReferences:\n\nMentzer et al., 2022. \"VCT: A Video Compression Transformer\". NeurIPS 2022.\n\nHe et al., 2021. \"Checkerboard Context Model for Efficient Learned Image Compression\". CVPR 2021.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new quantization scheme and entropy model for neural image compression, drawing inspiration from recent work in image generation models. Specifically, the idea of vector quantization (from VQ-VAE) was adapted into product quantization for better scalability, and an autoregressive entropy model based on a quincunx pattern was proposed. The combined architecture yields competitive compression performance with existing neural image compression methods.\n\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Main changes** (also described in the \"weaknesses\" above):\n1. A better discussion / motivation for VQ/PQ v.s. scalar quantization, even if the result is negative. Classical results in quantization/compression indicate that at high bit-rates, uniform scalar quantization is close to optimal (see Ziv, 1985), which seems to speak against VQ/PQ. Examining this result and understanding to what extent this holds in non-linear transform coding would add to the scientific value of this paper.\n2. A discussion on the related work of checkerboard-patterned entropy model (He et. al., 2021), which would help better place this work in the literature. \n3. A more rigorous R-D-P comparison to HiFiC, in order to justify the claim that the method outperforms HiFiC. One valid procedure could be fixing one of the three axes and comparing the remaining two. \n\nZiv 1985. \"On Universal Quantization\". IEEE TRANSACTIONS ON INFORMATION THEORY.\n\n\nMinor typo:\np. 3, \"with a prefix that encoding the conditioning text\" -> \"with a prefix that *encodes* the conditioning text\"", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3oYyCGOmPX", "reviewer": "Reviewer_GQkS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is over all well-writen and easy to follow. High-level intuitions are well explained with figures. Numerical experiments are provided with sufficient details. Experiments are well organized.\n\nThe multi-stage encoding / decoding in an interesting idea. From the numerical experiments, more stages give better compression rate.\n\nThe numerical experiments show that the proposed method outperforms conventional JPEG and BPG as well as some deep learning baselines.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a novel deep image compression model that combines several novel techniques proposed in recent years. The proposed method groups input pixels (tokens) into a quincunx pattern. Pixels in the same group are encoded / decoded parallelly. A progressive decoder is then applied on top of groups instead of pixels, which accelerates decoding speed. The model is trained to minimize the MSE or perceptual loss. The authors demonstrate the performance improvement of the proposed method against popular codecs and deep learning based image compression baseline methods.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The deep image compression baseline models compared in the experiments are not update-to-date. Only methods proposed in 2018-2020 are compared. Please consider to compare to more recent works, especially 2021-2022 works.\n\nOne of the key arguments of this work is to accelerate the encoding /decoding speed via parallalization with token groups. However, there is no speed comparison in the experiments. Please append speed comparison experiment.\n\nThe multi-stage approach significantly improves bit rate but might incur with slower decoding speed. Would be better to compare the speed v.s. number of stages v.s. bit rate together.\n\nIt seems that, more stages always improve bit rate. Let us image that the number of stages is being pushed into infinite. What would we expect? Is there a break point that the performance becomes worse agaist more steps?\n\nIt is not very clear to me that at the last step, how many tokens will be used in encoding / decoding. From Eq. (3), it seems all decoded tokens will be used as input. If this is true, wow to deal with increasing number of tokens in multi-step decoding?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ebXTFC5e2J", "reviewer": "Reviewer_yFdz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\nThe paper is well-written and easy to follow, with only a few minor mistakes that can be easily fixed.\n\nThe authors study a relevant problem, and their proposed solution is sound. Furthermore, I appreciated that the authors put a good amount of thought into the time complexity of the method to ensure its scalability, e.g. via their choice of transformer architecture.\n\nThe authors conduct a reasonable amount of ablation studies to confirm the validity of their modelling choices. And they show that their method with their proposed architecture performs at least on par and sometimes clearly better than the state-of-the-art on the standard set of distortion and perceptual metrics. The authors also demonstrate that their method scales gracefully to the very small bitrate regime and that their learnt prior model is strong enough to impute a large percentage of missing latents.\n\n# Weaknesses\nI have a couple of prominent points to clarify, alongside a few more minor ones.\n\nFirst, the relation of PQ-MIM to hierarchical VAEs is unclear. The way Eqs (2) and (3) define the prior structure look identical to the generative model of a particular hierarchical VAE with $S$ stochastic layers where there are an equal number of latents on each layer. Hence, the power of PQ-MIM, in terms of coding efficiency and generative / imputation ability, could be explained by the power of hierarchical VAEs. It also makes the distinction the authors make between PQ-MIM and hierarchical VAEs seem artificial. Could the authors comment on this?\n\nGiven the above, it is unclear to me why any partition of the latent space into equal-sized sets should be special so long as each set in the partition reasonably covers the whole latent space (as opposed to each set in the partition being a contiguous \"patch\", like a set of squares or rectangles). In particular, I don't see what causes the discrepancy reported in Figure 9 between different partitioning techniques. Uniformly randomly partitioning the latent space into $S$ sets would also be a good baseline.\n\nFurther, it is unclear how the two proposed improvements of PQ and the prior model perform independently. As I understand, the number of subvectors $M$ and the number of prior partitions $S$ are independent. For example, it would be important to know how much flexibility is lost using PQ instead of full VQ. For example, how does the performance of a VQ-VAE with a codebook of size 16 x 16 x 16 for each latent compare to a PQ-VAE with $M = 3$ subvectors per latent and a codebook of size 16?\n\nSome more minor points are:\n - In the introduction, the authors pose the following question: \"How to define a vector quantizer offering a range of rate-distortion operating points?\" - I am not sure what the authors meant here. This sentence reads as if the authors were proposing a variable-rate compression method. While dropping some of the transmitted tokens, as done in Section 4.3, could be conceived as doing this, if the authors want to claim variable rate compression as a contribution, they need to examine the performance of their method more thoroughly.\n - Roughly, how do the computational costs of the model the authors used to generate results reported in Figure 5 compare with the cost of other methods?\n - In the contributions, the authors claim: \"This simple PQ-VAE variant offers a strong and scalable rate-distortion trade-off\" - I get that a strong here means that the method gives a good performance, but what is meant by a \"scalable rate-distortion trade-off\"?\n - \"In particular we consider the \"quincunx\" regular grid pattern, where in each stage we double the number of tokens to predict, see Figure 4 for an illustration\" - This is not what Figure 4 shows. Figure 4 shows that at each stage the same number of new tokens are revealed.\n - \"Without loss of generality, we choose all neural network models to be identical\" - What generality is not lost?\nIn section 3.1: How was Figure 3 generated? At least provide a pointer to the description of how it was generated. Perhaps include some description in the caption.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "First, the authors propose a generative model for image compression, representing a middle ground between standard variational autoencoders (VAE) and vector quantized VAEs (VQ-VAEs). In particular, the authors propose to use product quantization (PQ). PQ partitions the embedding space into a set of subspaces and trains a separate vector quantizer for each subspace. Thus, one can use much smaller codebooks for each vector quantizer while retaining good flexibility of the model.\n\nSecond, the authors propose a hierarchical prior model for the product-quantized representations using a masked image model (MIM). Hence they call their model PQ-MIM.\n\nThe authors conduct usual image compression experiments and report results on the standard datasets Tecnick and Kodak. They also perform ablation studies on the model size, the masking patterns and the masking process, and imputing latents using their predictive model.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- [Necessary] If there is a connection with hierarchical VAEs, the authors need to discuss this connection.\n- [Necessary] Some empirical studies on how PQ performs on its own relative to VQ.\n- [Optional] The first paragraph of the introduction is a bit too grandiose and doesn't contribute to the paper's content. Hence it should be heavily shortened or removed altogether.", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers acknowledged the contribution of the paper and recommended its acceptance. I agree with the reviewers to recommend accepting.

In the official recommendation, one reviewer suggested that the authors polish their references; "for instance, some arXiv papers have been published in conferences or journals, and some abbreviations should be written in full format." While it is common to cite arXiv papers, it would be good to cite the conference or journal versions if they have been accepted. Additionally, the authors could consider unifying the style of the references, as currently, both abbreviated and full formats are used.
Decision: Accept (poster)