LVQ-VAE:End-to-end Hyperprior-based Variational Image Compression with Lattice Vector Quantization
Authors: Shinobu Kudo
Yukihiro Bandoh
Seishi Takamura
Masaki Kitahara
Venue: Submitted to ICLR 2023 (2023)
Keywords: Image Compression
Variational Autoencoder
Vector Quantization
Lattice

Abstract: Image compression technology has become more important research topic. In recent years, learning-based methods have been extensively studied and variational autoencoder (VAE)-based methods using hyperprior-based context-adaptive entropy model have been reported to be comparable to the latest video coding standard H.266/VVC in terms of RD performance.
   We think there is room for improvement in quantization process of latent features by adopting vector quantization (VQ). Many VAE-based methods use scalar quantization for latent features and do not exploit correlation between the features. Although there are methods that incorporate VQ into learning-based methods, to the best our knowledge, there are no studies that utilizes the hyperprior-based VAE with VQ because incorporating VQ into a hyperprior-based VAE makes it difficult to estimate the likelihood.
   In this paper, we propose a new VAE-based image compression method using VQ based latent representation for hyperprior-based context-adaptive entropy model to improve the coding efficiency. The proposed method resolves problem faced by conventional VQ-based methods due to codebook size bloat by adopting Lattice VQ as the basis quantization method and achieves end-to-end optimization with hyperprior-based context-adaptive entropy model by approximating the likelihood calculation of latent feature vectors with high accuracy using Monte Carlo integration. Furthermore, in likelihood estimation, we model each latent feature vector with multivariate normal distribution including covariance matrix parameters, which improves the likelihood estimation accuracy and RD performance.
   Experimental results show that the proposed method achieves a state-of-the-art RD performance exceeding existing learning-based methods and the latest video coding standard H.266/VVC by 18.0%.

Under review as a conference paper at ICLR 2023
HYPERPRIOR--BAERR VARIA- TIONAL IMAGE COMPRESSION WITH LATTICE VEC- TOR QUANTIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Image compression technology has become more important research topic. In re- cent years, learning-based methods have been extensively studied and variational autoencoder (VAE)-based methods using hyperprior-beet based context-adaptive en- tropy model have been reported to be comparable the latest video coding stan- dard H.266/VVC in terms of RD performance. We think there is room for im- provement in quantization process of latent features by adopting vector quantiza- tion (VQ). Many VAE-based methods use scalar quantization for latent features and do not exploit correlation between the features. Although there are methods that incorporate VQ into learning- based methods, to the best our knowledge, there are no studies that utilizes the hyperprior-bai based VAE with VQ because incorporat- ing VQ into a hyperprior-based VAE makes it difficult to estimate the likelihood. this paper, we propose new VAE-based image compression method using VQ based latent representation for hyperprior-bait based context-adaptive entropy model to improve the coding efficiency. The proposed method resolves problem faced by conventional VQ-based methods due to codebook size bloat by adopting Lat- tice VQ as the basis quantization method and achieves end-to-end optimization with hyperprior-basee context-adaptive entropy model by approximating the like- lihood calculation of latent feature vectors with high accuracy using Monte Carlo integration. Furthermore, likelihood estimation, we model each latent feature vector with multivariate normal distribution including covariance matrix param- eters, which improves the likelihood estimation accuracy and RD performance. Experimental results show that the proposed method achieves a state-of-the-arr RD performance exceeding existing learning- based methods and the latest video coding standard H.266/VVC by 18.0 % for Kodak, 21.9 % for CLIC2022 and 39.2 for Tecnick.
INTRODUCTION
Image compression technology has become important than achieve efficient data trans- mission and storage due to the demand for of high-quality contents and the increase the popularity of video services. Various conventional image compression technologies have been standardized so far (JPEG (Wallace, 1991; ITU, 1993), JPEG2000 (Taubman & Marcellin, 2002; ISO/TEC, 2004), WebP (Google), H.264/AVC (Marpe et al., 2006; ISO/IEC. 2003), H.265/HEVC (Sullivan al., 2012: ISO/IEC 2013), H.266/VVC (Bross et al., 2021; ISO/IEC, 2020), etc.). These technologies consist of combination of transform, quantization and entropy coding. Transform major part of JPEG, H.265/HEVC and H.266/VVC which use DCT or DST, while JPEG2000 uses wavelet transform, all of which are based on handcrafted linear transforms. These hand-crafted design are limited in their ability to capture features for variety of images.
In recent years, deep learning has made remarkable progress, and learning- methods are be- ing actively explored in the field of image compression. Most recent learning-based methods are based on transform coding (Goyal, 2001). Many of these methods use convolutional neural network (CNN)-based autoencoders in which the encoder transforms the imput image into latent repre- sentation and then performs quantization and entropy coding, while the decoder reconstructs the restored image. This approach achieves flexible nonlinear transforms that have higher potential to

---

Under review as a conference paper at ICLR 2023
map pixels into more compressible latent representation than the linear transforms used by clas- sical image compression approaches. It can be divided into two types according to the metric used for encoder optimization. One is the generative approach that directly maximizes subjective im- age quality (Rippel & Bourdev, 2017; Santurkar et al., 2018; Agustsson et al., 2019; Mentzer et al., 2020; Kudo et al., 2021). This approach aims optimize the distribution of reconstructed images to approach that of natural images by using generative adversarial networks. The other maximizes an objective metric such as peak signal-to-noise ratio (PSNR). This approach solves the rate-distortion (RD) optimization problem in the same way as the classical image compression described above. This paper focuses on the latter approach as it applicable to a wider range of applications. The lat- ter approach is found in various proposals. Toderici et al. (2016; 2017) introduced recurrent neural networks for feature extraction and Johnston et al. (2017) enhanced these networks to improve the coding performance. Cai & Zhang (2018); Cai et al. (2018) directly trained the quantization. These methods quantize the latent features as fixed-length codes.
By contrast, variational autoencoder (VAE)-based methods have been proposed that formulate the optimization as the problem of minimizing the entropy of the quantized latent features as well as the expected distortion of the reconstructed image with respect to the original. The first image compression method using VAE was proposed by Theis et al. (2017) and Ballé et al. (2017). They studied entropy models to approximate the actual distributions of the quantized latent features. To improve the accuracy of the entropy model, a hyperprior-basee context-adaptive entropy model was proposed by Ballé et al. (2018); it has been the baseline in most subsequent research. Whereas the actual distributions of each latent feature are fixed in (Theis et al., 2017: Ballé et al., 2017), Ballé et al. (2018) approximated the entropy model zero-mean Gaussian distribution with scale parameter for each latent feature to remove the spatial redundancy, where contexts are encoded as side information. Based on this hyperprior-baan based context -daattiv entropy model, various methods have been proposed to estimate the entropy model with higher accuracy. The autoregressive con- text model is one of the technologies that has experienced significant performance improvements. Minnen al. (2018) and Lee al. (2019) proposed to jointly utilize an autoregressive context model and the mean and scale hyperpriors. Mentzer et al. (2018) and Chen et al. (2021) extended an au- toregressive context model that utilizes channel neighbors with 3D Masked Convolution module. In (Minnen& & Singh, 2020) and (Zhu et al., 2022b), the channel- applied to reduce the computational complexity of the spatial-directed autoregressive model and He et al. (2022) was further improved by dividing the model non-evenly into channel directions. To further improve the entropy model, Liu et al. (2020) and Cheng et al. (2020) proposed a Gaus- sian mixture model and developed a network architecture by adopting an attention module. As an- other improvement perspective, Hu et al. (2020) proposed coarse-to-fine hyperprior modeling while Yang et al. (2020) improved the performance by devising an inference process without changing the training process. Ho et al. (2021) and Xie al. (2021) focused on improving the network architec- ture by adopting normalizing flow module. Some methods have been reported to better the RD performance of H.266/VVC, the latest video coding standard, which is not learning based, in terms of the MS-SSIM metric, but are still comparable in the PSNR metric.
autoregressive model was
Vector quantization (VQ) is incorporated into learning-based methods to leverage perfor- mance. Since VQ potentially offers better performance than scalar quantization in terms of RD (Gray& & Neuhoff, 1998; Gray, 1984; Chou et al., 1989). various studies have exam- ined it (Shin & Lu, 1991; Antonini et al., 1992; Tatsaki et al., 1995; Shnaider & Paplinski, 2001; Voinson et al., 2002; Salleh & Soraghan, 2007; Chiranjeevi & Jena, 2018; Nag, 2019). The chal- compression methods is how to incorporate the likelihood estimation of latent feature vectors into the optimization process. van den Oord et al. (2017) pro- posed VQEV which avoids likelihood calculations by assuming uniformity of the prior distribu- tion of latent features and separately designing encoder/decoder and codebooks to perform gradi- ent optimization. Razavi et al. (2019) and Fauw et al. (2019) extended VQ-VAE to hierarchical network structure. Williams et al. (2020) revised the quantization process while Xue et al. (2019) combined optimization with supervised learning. However, all of the above methods do not per- form well because the learning parameters underlying the encoder/decoder and codebook are de- signed separately to achieve gradient optimization and/or the probability distribution is assumed to be uniform. Agustsson et al. (2017) attempts end-to-end optimization by performing VQ using soft-to-hard annealing strategy. However, its learning suffers from unstable convergence, because it approximates the prior distribution of the codebook with histogram taken from the training process. Zhu et al. (2022a) proposed a cascaded vector quantization with multi-codebooks to keep memory
lenge in applying VQ to learning-basee
2

---

Under review as a conference paper at ICLR 2023
Q
2
0
2
1
Rg
hs
6m
POPP)-X
P(TR)-MIK ()
919
y
01
U/
91
8
x|
2|
(a) Baseline
(b) Autoregressive
(c) Proposed (key features are noted by red)
Figure 1: Operational diagrams of VAE-based image compression. U UQQ and UA QA represents either quantization during training (producing variables with tilde) quantization during inference (producing variables with a hat).
capacity and performance at high bitrates. However, due to the limitation on the vector dimension, this approach does not fully utilize the potential performance of vector quantization. In addition, to the best knowledge the authors, no study has utilized the above-mentioned hyperprior-band context-adaptive entropy model with VQ. Incorporating VQ into the hyperprior-basee baaeed method and optimizing them in an end-to-end manner has not been achieved, because incorporating VQ into hyperprior-baid based -daative entropy model makes difficult estimate the likelihood.
In this paper, propose new VAE-based image compression method using VQ- VQ-based latent repre- sentation for a hyperprior- based context-adaptive entropy model (named LVQ-VAE) with improved coding efficiency. The proposed method has two main features. First we introduce Lattice VQ for quantization, which resolves the codebook size bloat problem faced by conventional VQ-based methods without coding efficiency degradation. At the same time, the proposed method achieves end-to-end optimization with the hyperprior Carlo integration to approximate with high accuracy the likelihood calculation of latent feature vec- tors. Second, likelihood estimation, we model each latent feature vector as multivariate normal distribution including covariance matrix parameters, which improves the likelihood estimation curacy and RD performance. Zhu et al. (2022a) also models latent feature vectors as multivariate normal distribution, but this method models the features contained in each representative point as single distribution, whereas the proposed method models each feature vector, which eliminates correlation among feature vectors and is expected improve coding performance.
contexl-ades aadaptiv entropy model by using Monte
The contributions of this paper are summarized as follows:
We incorporate VQ into VAE-based image compression method using hyperprior-based context-adaptie entropy model while optimizing them for end-to-end manner, which leads to obtain better RD performance that exceeds the limits possible with scalar quantization without codebook size bloat.
We improve the entropy model by modeling each latent feature vector as multivariate nor- mal distribution including covariance matrix parameters, which improves RD performance.
We show that the proposed method achieves state-of-the-an RD performance compared existing learning-based methods and the latest video coding standard H.266/VVC with regard to the PSNR metric.
VAE-BASED IMAGE COMPRESSION
Most recent VAE-based methods are based on the hyperprior model developed by Ballé et al. (2018)
shown in Fig. (a). This model consists of four parametric transforms:
Ia (æ; (g)): feature encoder that transforms input image æ into latent feature y, which is quantized to form ÿ = Q(y),
gs (x(;; 0g): feature decoder that reconstructs image á ())

---

Under review as a conference paper at ICLR 2023
(yy: Φx)): hyper-encoder that extracts latent representation for context information, which is quantized to form & Q(z),
(,; 0s): a hyper-decoder that generates the context information for estimating the en- tropy model Pole (ū 2).
Øg. gg. Φr, On are optimized parameters of each transform, which are generally composed of neural networks such as CNNs. U FiQ denotes a quantizer in training phase (U) and one in an infer- ence phase (Q). In the training phase, quantizer U is approximated using additive uniform noise as y U ( = [y) + WI...., MN + uN], where is sampled from univariate uniform distribu- tion 1(- 3. ). This is because end-to-end learning requires the quantization realize gradient- based optimization. In addition to the above approximation, several other approximation techniques have been studied, such as stochastic binarization (Toderici al., 2016), universal quantization (Choi et al., 2019), straight-through estimator (van den Oord et al., 2017), and soft-to-hard quan- tization (Agustsson et al., 2017). the inference phase, quantizer Q is actual quantization such as rounding operator. In this manuscript, we represent approximated data as a variable with tilde and quantized data as one with a hat as shown in Fig.
The encoder compresses latent representation D and & by using entropy coding such as arithmetic coding (Rissanen & Langdon, 1979) and transmits it as a bitstream. The entropy coding estimates probability Pøj= (ý 2) as zero-mean Gaussian (O, ²)) its context information is scale parameter h2; 0.). To further improve this estimation, Minnen et al. (2018) and Lee et al. (2019) jointly utilized the autoregressive model and the mean and scale parameters of Gaussian distribution (Fig. (b)), where Cm denotes a context prediction model conditioned on decoded features that are composed of an autoregressive module such as masked convolutions (van den Oord et al., 2016). As for ($), the non-adaptive fixed density model called factorized prior is trained and shared between the encoder and the decoder.
Finally, the learning process is formulated RD optimization which minimizes the following loss function.
- -
E.geee [- log (ū) ž) [=))] + A Eggep. |
where pv: is the marginal distribution of natural images, and D represents the expected distortion between the reconstructed image and the original, and R represents the entropy of ÿ and z that approximates the expected code length of the bitstream. the Lagrange multiplier that controls the RD trade-off.
PROPOSED METHOD
The proposed method has two key features:
1. We introduce Lattice VQ for quantization of the latent features y, as this achieves better RD performance than scalar quantization without codebook size bloat.
2. We model each latent feature vector with multivariate normal distribution including co- variance matrix parameters and incorporate it into the autoregressive model to achieve highly accurate likelihood estimation.
Figure 1 (c) shows the operational diagram of the proposed method. In the following, we will describe a quantizer design based on Lattice VQ and then explain how to incorporate into hyperprior-basee based context-adaptive entropy model.
3.1 QUANTIZER DESIGN BASED ON LATTICE VECTOR QUANTIZATION
We use Lattice VQ for quantizing latent feature y. Given N elements of features denoted by ./...., yN. the features are split into n-dimensional vectors V1,..., U/n//] as follows.
8: [Uni-n+1. Uni-n+2, .... Vnel iis [N/n]
(2)
4

---

Under review as a conference paper at ICLR 2023
OPT WRO
CLB
CLO
Figure 2: A2 lattice
Then these ne-dimensional vectors are divided into clusters, and each cluster is represented by its centroid point.
Lattice VQ a VQ that places representative points at the location to form a lattice. Lattice A E IR"" is formed as linear combination of basis vectors (Conway & Sloane, 1988):
T Σ
b;b; h; E Z
where b1,..., b, basis vectors for R", and matrix with these basis vectors column vectors is also called the generating matrix which is uniquely defined for the structure of lattice, and & is an integer coefficient. For example, the 2-dimensional hexagonal lattice Ag shown in Fig. 2. The lattice points с E A are the representative points of Lattice VQ. Let QA (v) be the nearest neighbor of v in terms of Euclidean norm in the lattice:
Qx(v) = {c :1 cill s l|v cjll, ci, E A for all j +
(4)
The Voronoi region of lattice point c; the set of all vectors mapped into this point defined by
V(c,) = {v 1| - all - a| e e. E Afr for all #
(5)
Key features of Lattice VQ are follows:
1. A fast quantization method has been developed. Several lattices (A2, Dd (4-dimensional checkerboard root lattice), Es (8-dimensional Gosset's root lattice), A24 (24-dimensional Leech lattice), etc.) have fast quantization methods (Conway & Sloane, 1982: 1986; Vardy & Be' ery,,, 1993), and there are also fast iterative methods such as (Agrell et al., 2002) for arbitrary lattices.
2. Lattice VQ with particular generator matrix such as A2, D4. Eg, A2s is an optimal quan- tizer minimizing the distortion for uniform distribution source.
In terms of effectively utilizing the potential of Lattice VQ, channel direction. There are two reasons for this. One is that the autoregressive model based on pixel-by-pixel update cannot be applied when the vectorization axis
the vectorization axis in the
in the spatial direction. The other is that most conventional VAE-based methods do not take account of correlations in the channel direction. Although Minnen Singh (2020); Zhu al. (2022b); He et (2022) apply au- toregressive models in the channel direction, the distribution of latent features is indirectly predicted using features obtained by AR models. By contrast, the proposed method predicts the distribution of latent features directly, which is expected to improve performance.
To perform end-to-end optimization with the gradient method, we employ the approxima- tion method proposed by Lee et al. (2019), that is, we use straight-through estimator (STE) (Courbariaux & Bengio, 2016) for a decoder and the additive noise (Ballé et al., 2018) for the en- tropy model. Unlike when using additive noise for both is common in most conventional meth- ods, this method can eliminate train-test mismatch. Let Uv, be a n--iimensional random variable uniformly distributed over the basic cell of the lattice Vo (0). the Voronoi region of the lattice
5

---

Under review as a conference paper at ICLR 2023
point O. We generate u, Uv, and perform the following process.
(A:((;)) (o;) 02 + uz
for decoder
8; UA(e:)
(in training)
(6)
for an entropy model
0 = QA(v.)
(in inference)
(7)
0 where A is the same as QA but its gradient is ie; d2 1.
3.2 PROBABILITY MODEL
The probability entropy model for e is represented as
[N/n]
(
II
Pa.l= (0; dch
where dec = [01, 02..... -1 Each element of vectorized features ú; is modeled multivariate Gaussian with its own mean μa; and covariance matrix C, where the mean and covariance matrix are jointly predicted by the hyper-decoder hg (ž; 0,,) and autoregressive neural networks f consisting of masked convolution:
Po. L (0: Uci 2) = V.+Vo
N( C.)(a) da
where μ4; C₁ are estimated using f (ūcú, (2; 0n)). As for the probability entropy model of 2, we use the factorized density model that is the same method as used the previous work (Ballé al., 2018).
3.2.1 COVARIANCE MATRIX ESTIMATION
necessary to estimate covariance matrix C; to satisfy the condition of a positive definite sym- metric matrix. It is known that the positive definite symmetric matrix can be generated by matrix operation on an arbitrary matrix and its transpose. Here, network S ( (2 0x)) outputs mean vector μa; IR" and matrix A; E Rexa for each vectorized feature:
(e(; =
(10)
A
f4:
[AL,
(11)
[.1,1:
01,2
01,7
02,1:
02
02,r
A;
(12)
On,1,
On,2
On,a_
Then, covariance matrix C, is calculated as
C A A +el,
(13)
n
where the second term stabilizes the estimation and e is a small value. We set e to 10-2 our experiment.
3.2.2 PROBABILITY CALCULATION
It is numerically difficult to compute the integration of Eq. (9) because the integration region 6;+ Vo complex polytope in most lattices. Furthermore, with increasing dimensionality, the integral com- putations become impractical in terms of computational complexity. To resolve this, introduce Monte Carlo (MC) method (MacKay, 1998), which approximates integration as
M. N(p, C.)(2,) p(z,)
f Je N(u, (.) dx R
(14)
M
==1

---

Under review as a conference paper at ICLR 2023
where z, is an n-dimensional point sampled from arbitrary probability density function p. The method also has the advantage of being applicable to more complex probability distributions, since end-to-end optimization is possible as long as the probability density function and its derivative can be computed.
To further improve estimation accuracy, (Radovic et al., 1996), which uses low-discrepancy sequences such as Halton, Sobol', and Faure sequences as sampling points. The advantage of the QMC method is a faster rate of convergence than the naive MC method. The QMC method has rate of convergence close to O11/M), whereas the rate for the MC method is O()/VM). Thus Eq. (14) is re-written as
adopt Quasi-Monte Carlo (QMC) method
M V(u,,,))a))
V M
Je N(p (.) do
(15)
p(ay) represents the volume of the Voronoi region. In the experiment section, we employ Sobol's sequence as sampling points and set the number of sampling points, M, to 256 in training and 8192 in inference.
where V =
Since the MC/QMC method contains estimation error, the following two processes are introduced during inference to prevent decoding failure. First, when calculating the cumulative probabilities in both the encoder and the decoder, the cumulative probabilities are accumulated order of the centroid closest to each estimated mean vector, as long as the cumulative value does not exceed 1. The feature vector is quantized to the nearest centroid with the smallest distance among calculated centroids. Second, the encoder and the decoder use the fixed samples, namely, they use the same (fixed) random seed when calculating the probability with the QMC method in order to match the probability values of the encoder and the decoder.
EXPERIMENTS
We conducted simulations to verify the efficiency of the proposed method.
4.1 NETWORK ARCHITECTURE
Our network architecture is based on (Cheng et al., 2020) (See Appendix A.1 for details). N is channel size corresponding to the network capacity, which is set according to the bitrate as described below. As we use multivariate Gaussian model, the output of fo requires (n + 1) channels.
4.2 EXPERIMENT CONDITIONS
We used A24 lattice (n = 24). The reason is that A24 gives the optimal representation for uniform distribution. Hence it is expected that the proposed method brings its latent features towards uniform distribution through a optimization process (Ag, D1, Eg in Sec. 4.3.1 are used for the same reason). We adopted a fast quantization method for LVQ, which does not need to store representative points and offers a low complexity as described in Appendix A.2. We used a subset of ImageNet dataset (Russakovsky et al., 2015), and randomly scaled and cropped them to the size of 192 192 during training. We used Adam optimizer(Kingnit & Ba, 2015) with a batch size of 16. The learning rate was scheduled 10-4 for the first 20 epochs, and reduced to 10-5 for the last 10 epochs. was set to (256, 1024, 2048, 4096) and network channel size N was set to 144 for the lowest rate model and 192 for three higher rate models. For evaluation, we tested the Kodak image dataset (Kodak, 1993) consisting of 24 images (See Appendix B for other datasets). To evaluate the RD performance, the bitrate measured by bits per pixel (bpp), and the quality is measured by PSNR, where bitrate and PSNR are calculated by averaging the encoding results for all 24 images. We plot the RD curves and calculate the Bjontegaard delta bitrate (BDBR) (Bjontegaard, 2001) to compare their coding efficiency. We compared the proposed method with VTM 15.0 (JVET), the reference software of the latest video coding standard H.266/VVC, with intra profile and H.265/HEVC-baar encoder BPG (Bellard) and several of the learning- based methods. For VTM and BPG, the input RGB images were converted into YUV444 format and encoded, then the reconstructed images were converted into RGB and PSNR were calculated by averaging the MSE of each RGB value. For the

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison of BDBR for Kodak dataset for different numbers of dimensions and the use or non-use of covariance matrix in the proposed method. A2, Da. Es, A24 are the optimal lattice quantizers for 2-, 4-, 8- and 24-dimension as mentioned Sec. 3.1.
96.
37
353
36
(68
25
15000
Papeoud (A2)
34
Papeoed (26)
Dimension n
C
BDBR
Papead (BS)
Poppout (134)
1 (Z, scalar)
0.0 % (anchor)
Pripoued (iCCorDD))
0.0 %
31
-te- (PSC
2 (A2)
/
-2.3 %
0.3
-1.8 %
BB rule (1yo)
4 (Ds)
/
-6.8 %
Figure 3: Comparison of RD performance for Kodak dataset for different number of dimensions and the use or non-use of covariance matrix in the proposed method
-6.3 %
8 (Es)
/ -00.0 %
-9.2 %
24 (A24)
-14.7 %
Table 2: Comparison of BDBR for Kodak dataset
27
Method
BDBR
16
BPG (4:4:4)
23.8 % 0.0 (anchor)
1000
VTM 15.0 (4:4:4)
PSNN
- (P0/4444)
VTM (5-44--)
Ballé et al. (2018)
29.8 %
Halk (2885)
Minnan (201))
Minnen et al. (2018)
11.4 %
12
Cheng et al. (2020)
0.5 %
(2021)
Xie al. (2021)
1.4 %
Zuu di (20136)
Proposed/A30
He et al. (2022)
-6.4 %
Zhu et al. (2022a)
-2.7 %
0.1
0.5
Bi Capp)
Proposed (A24)
-18.0
Figure RD performance for Kodak dataset
learning-based methods, including the proposed method, we used RGB format as the input. For the proposed method, the bitrate is calculated by the estimated values (- 1olog2
4.3 RESULTS AND DISCUSSIONS
4.3.1 ABLATION STUDY
In order to clarify the effect of the number of dimensions Lattic VQ and the use of the estimated covariance matrix in the proposed method, we additionally simulate in other dimensions (n (Z, scalar quantization), n 2(A2), n 4(Da), n 8(Es)) and compared the performance with and without the estimation of the covariance matrix. Without estimation of the covariance matrix, fe output 2N channels (n means and n scales for each feature vector) and squared scales were aligned the diagonal components of each matrix C,; the other elements were set O.
The results of RD curves and BDBR are shown in Fig. 3 and Tab. 1. As for the number of di- mensions, can be shown that RD performance increases with the number of dimensions. This is consistent with the property of VQ that the expected error decreases as the number of dimen- sions increases, and there is the potential for further increases in gain with dimensionality increases. Regarding the of covariance matrix. the results shows better performance when covariance matrix is used. This suggests the presence of correlation within the feature vector, which could be eliminated by using covariance matrix as mentioned Sec. 3.1.
8

---

Under review as a conference paper at ICLR 2023
mm
mm
Original
Chang 0.100ppp. 31.050a0, 0.937
Proposed (Aud) 0.08 258d0, 0.955
100bpp, 30028ED, 0,916
0.097bpp. 30.727dD, 0,926
Figure 5: Reconstructed images of kodim04 (bpp. PSNR, MS-SSIM)
4.3.2 RD PERFORMANCE
Fig. and Tab. show the results of comparison between the proposed method and existing methods in terms of RD performance. The results show that the proposed method gave better RD performance than all previous learning-based methods. Furthermore, the proposed method outperforms VTM 15.0 by 18.0 %.
4.3.3 QUALITATIVE EVALUATION
To evaluate the qualitative performance, we visualized the reconstructed images. Fig.5 shows re- constructed images at the level of approximately 0.1 bpp. In Fig. 5, the proposed method maintains more detail, such as the woman's hair and the contours of teeth, than the other methods. Moreover, it is observed that the other methods suffer from some artifacts and degradation.
CONCLUSION
This paper proposed new VAE-based image compression method characterized by Lattice VQ for improving the hyperprior-based context-adaptive entropy model approach. The proposed method achieves end-to-end optimization with hyperppror-basee context-adaptive entropy model by ap- proximating the likelihood calculation of latent feature vectors with high accuracy by using Monte Carlo integration. Furthermore, the proposed method provides highly accurate likelihood estimation by modeling the distribution parameters of latent feature vectors.
Experiments public data sets showed that the proposed method achieves state-of-the-art RD performance compared to existing learning baaning-bassed methods and outperforms VTM 15.0, the ref- erence software of the latest video coding standard H.266/VVC, by 18.0 % for Kodak, 21.9 % for CLIC2022 and 39.2 % for Tecnick in the PSNR metric.
As future work, we will address reduce the complexity. This paper pursue maximizing coding efficiency rather than reducing complexity. Therefore, compared to the latest methods, the process- ing time due to the use of autoregressive models, etc. is large. In addition, in entropy coding, the amount of calculation of the cumulative probability table increases exponentially with the number of dimensions. These are prospective solutions the following ways; the spatial autoregressive mod- ule, which is mainly dominant in network processing, could be solved by introducing the parallel computation mechanism such as (He et al., 2021) and entropy coding by restricting the number of representative points and introducing cascade estimation such as (Zhu al., 2022a).

---

Under review as a conference paper at ICLR 2023
REFERENCES
E. Agrell, T. Eriksson, A. Vardy, and K. Zeger. Closest point search in lattices. IEEE Transactions on Information Theory, 4888))2201-221111 2002. doi: 10.1109/TIT.2002.80000 800449
Eirikur Agustsson, Fabian Mentzer, Michael Tschannen, Lukas Cavigelli, Radu Timofte, Luca Benini, and Luc Van Gool. Soft-to-hard vector quantization for end-to-end learning compress- ible representations. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 1141-1151, 2017.
Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, and Luc Van Gool. Gener- ative adversarial networks for extreme learned image compression. In 2019 IEEE/CVF Interna- tional Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 221-231. IEEE, 2019. doi: 10.1000/CC.2200900000
Mare Antonini, Michel Barlaud, and Thierry Gaidon. Adaptive
lattice vector quantization for multiresolution image coding. Petros Maragos (ed.). Visual Communications and Image Processing '92, volume 1818, pp. 441 457. International Society for Optics and Photonics, SPIE, 1992. doi: 10.1117/12.131662
Nicola Asuni and Andrea Giachetti. TESTIMAGES; a large-scale archive for testing visual devices and basic image processing algorithms. In Andrea Giachetti (ed.), Italian Chapter Conference 2014 Smart Tools and Apps in computer Graphics, STAG 2014, Cagliari, Italy, September 22- 23, 2014, pp. 63-70. org/ / 2312 stag. 20141242.
70. Eurographics, 2014. doi: (0.2312/sta. URL https: /oi.
Johannes Ballé, Valero Laparra, and Ecro P. Simoncelli. End-to-end optimized image compression. In Sth International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with scale hyperprior. In 6th International Conference on Learning Rep- resentations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018
F. Bellard. BPG image format. https: httpp//////elaa.orr/ooooooo/o///
G. Bjontegaard. Calculation of average PSNR differences between RD-curves. VCEG-M33, 2001.
Benjamin Bross, Ye-Kui Wang, Yan Ye, Shan Liu, Jianle Chen, Gary J. Sullivan, and Jens- Rainer Ohm. Overview of the versatile video coding (VVC) standard and its applications. IEEE Trans. Circuits Syst. Video Technol., (1(10))373633666 2021. doi: 10.1099/CCSVT22011110005..
Jianrui Cai and Lei Zhang. Deep image compression with iterative non-uniform quantization. In 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 451-4554 2018. doi: 10. 10.1100//..../118.. 84511111.
Jianrui Cai. Zisheng Cao, and Lei Zhang. Learning single tucker decomposition network for lossy image compression with multiple bits-per-pixel rates, 2018
Tong Chen, Haojie Liu, Zhan Ma, Qiu Shen, Xun Cao, and Yao Wang. End-to-end learnt image compression via non-local attention optimization and improved context modeling. IEEE Trans- actions on Image Processing,
2021. doi: 10.1 1..1109/11222211355556
Zhengxue Cheng. Heming Sun, Masaru Takeuchi, and Jiro Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition. CVPR 2020, Seattle WA. USA, June 13-19, 2020, pp. 7936-7945. Computer Vision Foundation IEEE, 2020. doi: 10. 00.10999VPR42200.220000 00796.
10

---

Under review as a conference paper at ICLR 2023
Karri Chiranjeevi and Uma Ranjan Jena. Image compression based vector quantization using cuckoo search optimization technique. Ain Shams Engineering Journal, 9(4):1417-1431, 2018.
ISSN 2090-4479. doi: https://oi.orgg/ http:///oooor////// 101.111....ccj201...00
Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Variable rate deep image compression with conditional autoencoder. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 20/9, Seoul, Korea (South), October 27 November 2, 2019, pp. 3146-3154. IEEE, 2019. doi: 10.1109//CCV220190000333
P.A. Chou, T. Lookabaugh, and R.M. Gray. Entropy-constrained vector quantization. IEEE Trans- actions on Acoustics, Speech, and Signal Processing, 37(1):31-44) 1989. doi: 0.11009/99174990
CLIC. Workshop and challenge on learned image compression. http::/comprett 2022.
cc/,
John Conway and N. Sloane. Sphere Puckings, Lattices and Groups, volume 290. 01 1988. ISBN 978-1-4757-2018-11 doi:
John H. Conway and Neil J. A. Sloane. Fast quantizing and decoding and algorithms for lattice quantizers and codes. JEEE Trans. Inf. Theory, 28(2):227-231, 1982. doi: 10.1109/TIT.198. 1056484.
John H. Conway and Neil J. Sloane. Soft decoding techniques for codes and lattices, including the golay code and the leech lattice. IEEE Trans. Inf. Theory, 32(1):41-50, 1986. doi: 10.1 109/ TIT. 1986. 1057135.
Matthieu Courbariaux and Yoshua Bengio. Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1. CoRR, abs/1602.02880, 2016.
Jeffrey De Fauw, Sander Dieleman, and Karen Simonyan. Hierarchical autoregressive image models with auxiliary decoders. CoRR, /et///933.0995 2019.
Google. WebP image format. https: http:::/ddveellerrss google com/ wtt.c//c///pp.eecc wpwe///ebd///b.....//////
Vivek K. Goyal. Theoretical foundations of transform coding. IEEE Signal Process. Mag., 18(5): 9-21,2001 doi: 1911109999999800
R. Gray. Vector quantization. IEEE ASSP Magazine, 1(2):429, 1984. doi: 10. 19.100//ASSP1184. 1162229.
Robert M. Gray and David L. Neuhoff. Quantization. IEEE Trans. Inf. Theory, 44(6):2325-23888 1998. doi: 10. 109/18.720544
Dailan He, Yaoyan Zheng, Baocheng Sun, Yan Wang, and Hongwei Qin. Checkerboard context model for efficient learned image compression. JEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021. virtual, June 19.25, 2021, pp. 14771-14780. Computer Vision Foundation IEEE, 2021. doi: 10.1109/CVPR44437, 202.4444437.22.01155.. URL https:// openaccess. thecvf.... com/ wwwwtooottttttennooooooo /html httphh/mmeeeeeeeeeeeeeeeeeeeeeee /he_Chhceebbbarrdd Context MaceettMoodeee_e_eeeeeeee for Efficiffiinii Sefficciett_Larnneee Learned Image wwtps//ooooepeeeeeeeeeeeee CVPR 2021 paper.html.
Dailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang. ELIC: efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 3708-5717. IEEE, 2022. doi: 10.1109/VVPR2228822200 201.226668.2222056633 URL https::t////..c /oiiooiii.oo/ org 10. 1109
httpg//....1.9./522eeeee Ctt.// 2022 60563.
Yung-Har Mang-Han ue Ho, Chih-Chun Chan, Wen-Hsiao Peng, and Hsueh-Ming Hang. End-to-end learned image compression with augmented normalizing flows. In IEEE Conference Computer Viston and Pattern Recognition Workshops, CVPR Workshops 2021, virtual, June 19-25, 2021, pp. 1931- 1935. Computer Vision Foundation IEEE, 2021. doi: 10.1 R0.1109/CVPPW55500 2021 2020000000.22000000...

---

Under review as a conference paper at ICLR 2023
Yueyu Hu, Wenhan Yang. and Jiaying Liu. Coarse-to to-fine hyper-prioe prior modeling for learned image compression. In The Thirty- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 11013-11020. AAAI Press, 2020.
ISO/IEC ISO/IE 14496-10 2003.
Advanced Video Coding for Generic Audio- Services, May
ISO/E ISO/I 23008-2: High Efficiency Video Coding, 2013.
ISO/EЕ ISO/EE 23090-3: Versatile Video Coding, 2020.
ISO/TEC. Information Technology JPEG 2000 Image Coding System Part 1: Core Coding System. ISO/EE 15444-1:2004, October 2004.
ITU.
ISOЛEC 10918-1
1993(E)
CCIT
Recommendation T.81.
http://ww.w3..om www.://orgGrGgrhiiiiee htggGGraphii///EG//ittt....
1993.
Nick Johnston, Damien Vincent, David Minnen, Michele Covell, Saurabh Singh, Troy Chinen, Sung Jin Hwang, Joel Shor, and George Toderici. Improved lossy image compression with prim- ing and spatially adaptive bit rates for recurrent networks, 2017.
JVET. H.266/VVC Official Test Model VTM. https: /vogit hhi Fraunhofer... de/ juet/ /wW.softwwre_//we
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceedings, 2015.
Eastman Kodak. Kodak lossless true color image suite (photocd pcd0992). http://rOk... graphics /oodak/ 1993.
Shinobu Kudo, Shota Orihashi, Ryuichi Tanida, Seishi Takamura, and Hideaki Kimata. Gan-based image compression using mutual information for optimizing subjective image similarity. IEICE Trans. Inf. Syst., 104-D(3):450 460, 2021.
Jooyoung Lee, Seunghyun Cho, and Seung- Kwon Beack. Context-adaptive entropy model for end- to-end optimized image compression. In 7th International Conference on Learning Representa- tions, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Jiaheng Liu, Guo Lu, Zhihao Hu, and Dong Xu. A unified end-to-end framework for efficient deep image compression, 2020.
David John Cameron MacKay. Introduction to monte carlo methods. In Michael I. Jordan (ed.), Learning in Graphical Models, volume 89 of NATO ASI Series, pp. 175-204. Springer Nether- lands, 1998. doi: 10. 00077978-94-001-5001
Detlev Marpe, Thomas Wiegand, and Gary J. Sullivan. The H.264/MPEG4 advanced video coding standard and its applications. IEEE Commun. Mag., 44(8):134-143, 2006. doi: 10. 109/MCOM. 2006.1678121.
Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, and Luc Van Gool. Condi- tional probability models for deep image compression. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 4394 4402. Computer Vision Foundation / IEEE Computer Society, 2018. doi: 10.1 109//P.. 2018.00462. URL http: /ttett:::/pppreee wptp/cccscccccccppppppppc thecvf. com http://oonnetttct___010 /html/ Ment www_eeeerrttiiiiee Baaaallllrraaaaaaaeaeeee www_aMMeelssccpee hhtm..........ll
Fabian Mentzer, George Toderici, Michael Tschannen, and Eirikur Agustsson. High-fidelity genera- tive image compression. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020.
12

---

Under review as a conference paper at ICLR 2023
David Minnen and Saurabh Singh. Channel-wise autoregressive entropy models for learned im- age compression. In IEEE International Conference on Image Processing, ICIP 2020, Abu Dhabi, United Arab Emirates, October 25-28, 2020, pp. 1339-33443 IEEE, 2020. doi: 10. 2020.
1109/CC144078..20009999999 URL https::: /ttss:/ddoo 91 9190935.
David Minnen, Johannes Ballé, and George Toderici. Joint autoregressive and hierarchi- cal priors for learned image compression. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Informa- tion Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, wwciiis.ccc//// ptppp//pper/..////// Abstrratttt......e .hmll.
pp. 10794-10803, 2018. URL
Sayan Nag. Vector quantization using the improved differential evolution algorithm for im- age compression. Genet. Program. Evolvable Mach., 20(2):187-211) 2019. doi: 10.1007/
s10710-019-09342--8
Igor Radovic, Ilya M. Sobol, and Robert F. Tichy. Quasi-monte carlo methods for numerical inte- gration: Comparison of different low discrepancy sequences. Monte Carlo Methods Appl., 2(1): 1-14, 1996. doi: 10.1515/menmn 1996.2.1.1.
Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQE-2. Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 14837- 2019.
Oren Rippel and Lubomir D. Bourdev. Real-time adaptive image compression. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 2922-2930 PMLR, 2017.
Rissanen and G. Langdon. Arithmetic coding. IBM Journal of Research and Development, 23 (2):199-166 1979. doi: 10.11447rd.232..0499
Olga Russakovsky. Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. Int. J. Compus. Vis., 11522)2 2015. doi: 10.1007/s 1263-015-08166.
Mohd Fadzli Mohd Salleh and John Soraghan. A new multistage lattice vector quantization with adaptive subband thresholding for image compression. EURASIP J. Adv. Signal Process., 2007, 2007. doi: 10. 155/2007/999288.
Shibani Santurkar, David M. Budden, and Nir Shavit. Generative compression. In 2018 Picture Coding Symposium, PCS 2018, San Francisco, CA, USA, June 24-27, 2018, pp. 258-262. IEEE, 2018. doi: 10.1109/PCS.200 8.556298..
Y.H. Shin and C.-C. Lu. Image compression using vector quantization and artificial neural net- works. In Conference Proceedings 1991 IEEE International Conference on Systems, Man, and Cybernetics, pp. 1487-1491 vol.3, 1991. doi:
1991
Mikhail Shnaider and Andrew Peter Paplinski. Still image compression with lattice quantization in wavelet domain, pp. 56 - 119. Academic Press, United States of America, edition, 2001. ISBN 0120147610.
Gary J. Sullivan, lens-Rainer Ohm, Woojin Han, and Thomas Wiegand. Overview of the high efficiency video coding (HEVC) standard. IEEE Trans. Circuits Syst. Video Technol., 22(12): 1649-1668, 2012. doi: 10.1 1..109/CCSVT...222211999
13

---

Under review as a conference paper at ICLR 2023
Anna Tatsaki, Thanos Stouraitis, and Costas E. Goutis. Progressive image compression algorithm based on lattice vector quantization. In Naohisa Ohta, Heinz U. Lemke, and Jean Claude Lehureau (eds.), Advanced Image and Video Communications and Storage Technologies, volume 2451, pp- 228 236. International Society for Optics and Photonics, SPIE, 1995. doi: 10.1117/12.20122
David S. Taubman and Michael W. Marcellin, JPEG2000 image compression Cumprrssion.uundameree fundamentals, stan- dards and practice, volume 642 of The Kluwer international series in engineering and computer science. Kluwer, 2002. ISBN 978-0-7923-751997, doi: 10.1007/978- 1-1------66119-999-
Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferene Huszár. Lossy image compression with compressive autoencoders. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
George Toderici, Sean M. O' MaMlllyy, Sung Jin Hwang, Damien Vincent, David Minnen, Shumeet Baluja, Michele Covell, and Rahul Sukthankar. Variable rate image compression with recurrent neural networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
George Toderici, Damien Vincent, Nick Johnston, Sung Jin Hwang, David Minnen, Joel Shor, and Michele Covell. Full resolution image compression with recurrent neural networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolula, HI, USA, July 21-26, 2017, pp. 5435-5443, 2017. doi: 10.1 0099/CVPR.001..5777
Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with pixelcnn decoders. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 4790-4798, 2016
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Prucessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6306-6315, 2017.
Alexander Vardy and Yair Be'ery. Maximum likelihood decoding of the leech lattice. IEEE Trans. Inf. Theory, 19354):435-1444 1993. doi: 10.1109/18 1.24466...
Teddy Voinson, Ludovic Guillemot, and Jean-Marie Moureaux. Image compression using lattice vector quantization with code book shape adapted thresholding. In Proceedings of the 2002 International Conference on Image Processing, ICIP 2002, Rochester. New York, USA, September 22-25, 2002, pp. 641-644. IEEE, 2002. doi: 10. 10.1109// 1040032.
Gregory K. Wallace. The JPEG still picture compression standard. Commun. ACM, 34(4):30 44, 1991. doi: 10. 00.114//00855.0033333
Will Williams, Sam Ringer, Tom Ash, David MacLeod, Jamie Dougherty, and John Hughes. Hi- erarchical quantized autoencoders. Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Yueqi Xie, Ka Leong Cheng, and Qifeng Chen. Enhanced invertible encoding for learned image compression. In Heng Tao Shen, Yueting Zhuang, John R. Smith, Yang Yang, Pablo Cesar, Florian Metze, and Balakrishnan Prabhakaran (eds.), MM '21: ACM Multimedia Conference, Virtual Event, China, October 20 24, 2021, pp. 162-170. ACM, 2021. doi: 10. 00.1145//474085.447521.
Yifan Xue, Michael Q Ding, and Xinghua Lu. Supervised vector quantized variational autoencoder for learning interpretable global representations. CoRR, abs/1909. 111119999111122 1114, 2019.
Yibo Yang, Robert Bamler, and Stephan Mandt. Improving inference for neural image compression. In Hugo Larochelle. Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florin Balcan, and Hsuan-Tie Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
14

---

Under review as a conference paper at ICLR 2023
Xiaosu Zhu, Jingkuan Song. Lianli Gao, Feng Zheng, and Heng Tao Shen. Unified multivariate gaussian mixture for efficient neural image compression. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp- 17591-17600. IEFE, 2022a. doi: 10.1 11100//YP2228882222200099999 URL https:: / /doi. org/10.1109/CVPPS50 202..017022. 019
Yinhao Zhu, Yang Yang, and Taco Cohen. Transformer-base based transform coding. The Tenth Inter- national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022b. URL https:: http///www/w/eee//////ppp./..eep net/ forum? D=WN6
15

---

Under review as a conference paper at ICLR 2023
Figure 6: Network architecture
A IMPLEMENTATION DETAILS
A.1 NETWORK ARCHITECTURE
Our network architecture is based on (Cheng et al., 2020) and is illustrated in Fig. 6. We use residual blocks and attention module for both the feature encoder/decoder.
A.2 Low COMPLEXITY LATTICE VECTOR QUANTIZATION
We used fast quantization method (Conway & Sloane, 1982) for A2, D4, Es and (Conway & Sloane, 1986) for A24. These methods calculate the Fuclidean norm for several candidate points (ex. 256 points for A21) for each vector and selects the one with the smallest norm among them. These methods have two advantages: one is that it does not need to keep representative points in memory, and the other is that the quantization process be performed high speed without calculating the distance to all quantized representative points as the conventional VQ method.
B ADDITIONAL EXPERIMENTS
We also tested on the CLIC2022 test set (CLIC, 2022) consisting of 30 high resolution images, and Tecnick image dataset (Asuni & Giachetti, 2014) consisting of 40 images with 1200 x 1200 resolutions.
RD performance results for CLIC2022 are shown in Fig. 7 and Tab. and for Tecnick are shown in Fig. and Tab.
For both CLIC2022 and Tecnick, the proposed method also gave the state-of-the-an performance. Especially for Tecnick, it showed relatively large performance gains compared to Kodak and CLIC2022. This may be due the fact that Tecnick has lower texture component compared to Kodak and CLIC2022. The proposed method tends to produce higher gains at lower rates than at higher rates, which may have contributed Tecnick's 's higher performance.
16

---

Under review as a conference paper at ICLR 2023
Table 3: Comparison of BDBR for CLIC2022 dataset
30
Method
BDBR
PSNR (48)
BPG (4:4:4)
40.9 % 0.0 (anchor)
VTM 15.0 (4:4:4)
Ballé et al. (2018)
38.5 %
(ooa))
Minnen et al. (2018)
13.5 %
31
Cheng et al. (2020)
-0.8 %
Chong (3009)
30
Progosed(AD0
Proposed (A24)
-21.9 %
0.2
0.5
0.4
0.5
0.7
0.9
(hpp)
Figure 7: RD performance for CLIC2022 dataset
Table 4: Comparison of BDBR for Tecnick
dataset
Method
BDBR
PSNRR (088
BPG (4:4:4)
45.8 % 0.0 (anchor)
37
VTM 15.0 (4:4:4)
(6-444))
Ballé al. (2018)
57.3 %
Minnen et al. (2018)
13.1 %
Cheng et al. (2020)
-1.5 %
Chinge (2100)
Propood((30g
Proposed (A24)
-39.2 %
Figure 8: RD performance for Tecnick dataset
17


Reviews:
[{"review_id": "oOF242Pltb", "reviewer": "Reviewer_9ZNU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces Lattice Vector Quantization (LVQ) to hyperprior-based learned image compression. The LVQ is applied to co-located feature samples along the channel dimension. Moreover, it applies QMC, a traditional technique, to arrive at differentiable rate estimates. The LVQ is incorporated into Cheng’s learned codec (CVPR2020) to validate its coding performance. ", "summary_of_the_review": "The idea of introducing Lattice Vector Quantization (LVQ) to learned image compression is interesting and novel. Several well known LVQ techniques, including the design of the lattice generating matrix and the rate estimation, are extended to address learned compression. While it is intuitively agreeable that the coding performance may be improved further by introducing LVQ, the increased complexity is another issue that must be addressed. It is unfortunate that this aspect is currently missing. Moreover, one key reference is missing too.  ", "strength_and_weaknesses": "Strengths:\n\n(1) The idea of introducing Lattice Vector Quantization to learned image compression is interesting and novel. \n\n(2) The RD performance of the proposed method looks impressive, showing up to 18% BD-rate saving over VTM. \n\nWeaknesses:\n\n(1) A similar work in CVPR2022 [1] that addresses Vector Quantization for learned image compression should be included for comparison and mentioned in related work. The novelty over [1] should be highlighted too. \n[1] Zhu, Xiaosu, et al. \"Unified Multivariate Gaussian Mixture for Efficient Neural Image Compression.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n(2) More ablation experiments should be conducted to analyze the complexity-performance trade-off among different choices of the lattice generating matrix. The complexity should include the model size, MAC, peak memory requirements, and encoding/decoding runtimes. It is unfortunate that these experiments are currently missing. \n\n(3) In Eq. (8), the LVQ is modeled by additive uniform noise during training. This additive noise model does not appear to consider the generating matrix chosen for LVQ.\n\n(4) The choice of the generating matrix is empirical and not well justified. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "klAhycf8G0H", "reviewer": "Reviewer_SVp9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a neural image compression models based on lattice vector quantization (LVQ).\n\nThe authors note two deficiencies in existing work: (1) most models use scalar quantization (SQ), which is typically less powerful than vector quantization (VQ) if a large enough codebook is used, and (2) models that do use VQ do not use a hyperprior or end-to-end optimization. The authors address these problems by using LVQ, which has a large, though constrained, codebook, and they show how to jointly optimize a distribution over the lattice (the entropy model) with a hyperprior and spatially autoregressive model.\n\nTwo other important contributions are the evaluation of predicting a full covariance matrix for each LVQ group, rather than a diagonal covariance matrix, and using Monte Carlo integration to more quickly calculate (an approximation of) the likelihood of each feature vector.\n\nThe result is, as far as I know, state of the art rate-distortion (RD) performance for lossy image compression. The evaluation shows an 18% rate savings over VTM, one of the best standard (hand-engineered) image codecs, as well as gains over other learning-based (neural) codecs (Fig 5 and Table 2). The paper also compares different lattice dimensionalities and the effect of predicting a full covariance matrix (Fig 4 and Table 1).\n", "summary_of_the_review": "As far as I know, this paper presents state-of-the-art rate-distortion results for lossy image compression. That's a significant achievement given the existing work on this problem.\n\nThat said, the improvement in rate-savings is only moderate compared to the best existing methods (see references above), and the other methods appear to support much faster decode speeds, which is crucial for deployment.\n\nSo my recommendation is \"5: marginally below the acceptance threshold\" since it's not entirely clear how much of the performance gain comes from LVQ vs. predicting a full covariance matrix within each VQ group vs. the spatial autoregressive model.", "strength_and_weaknesses": "The primary strength of this paper is the achieving SOTA rate-distortion performance. As far as I know, now other papers or standard codecs achieve as good a compression rate when using PSNR as the quality metric.\n\nThe primary weakness of the paper is a lack of runtime data. I suspect this method is extremely slow since spatially AR context models are known to be slow due to the difficulty of parallelizing the decoding process, and the prediction of a full covariance matrix leads to a large number of output parameters (if I understand the setup correctly, 192 channels using \\Lambda_24 leads to 8 groups of 24D vectors and thus 8 * (24^2 + 24) = 4800 parameters per spatial location in the latent image representation). I suspect this leads to decode times of multiple seconds, even on fast hardware, when a deployable model needs to decode kodak-size images in a handful of milliseconds. To the best of my knowledge, no neural methods achieve decode speeds on par with standard codecs like VTM (or BPG or JPEG, etc.), but other models are probably two orders of magnitude faster than the method in this paper while only losing 5-10% in terms of rate.\n\nThe other weakness is a lack of direct comparison against stronger neural methods. Balle 2018, Minnen 2018, and Cheng 2020 are all relatively old methods. In particular, part of the motivation of this paper is the lack of modeling the dependencies between channels (SQ doesn't do this, VQ does). But existing papers adopt other methods for (partially) modeling these dependencies, for example:\n\nChannel-wise Autoregressive Entropy Models for Learned Image Compression\nDavid Minnen, Saurabh Singh\n(ICIP 2020) https://arxiv.org/abs/2007.08739\n\nELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding\nDailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, Yan Wang\n(CVPR 2022) https://arxiv.org/abs/2203.10886\n\nTransformer-based Transform Coding\nYinhao Zhu, Yang Yang, Taco Cohen\n(ICLR 2022) https://openreview.net/forum?id=IDwN6xjHnK8\n\nThe channel-wise AR approach used by these papers should be mentioned and compared to the LVQ approach adopted here since they both address the same underlying deficiency. I believe the RD performance of the LVQ approach is better than what is reported in these papers. However, the runtime is vastly different (ELIC in particular is focused on balancing RD performance with runtime), and the preferable trade-off probably falls to ELIC over the proposed LVQ method, i.e. I wouldn't accept 10-100x slower decoding to save 5-10% on file sizes.\n\nFinally, I'd like to better understand how much of the RD performance is due to the LVQ vs. the full covariance matrix. Fig 4 and Table 1 partially answer the question by showing that for several lattice dimensionalities, predicting a full covariance matrix helps (as expected). Another interesting comparison would be to use a trivial lattice (hyper-cubes) but still group the channels into 24D vectors and predict a full covariance matrix. I expect \\Lambda_24 is better (the underlying theory predicts this for mse) but the size of the gap would make it clear whether or not the added complexity of \\Lambda_24 is warranted.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hJMHawZmA8", "reviewer": "Reviewer_fFaA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to incorporate lattice vector quantization into recent hyperprior-based image compression framework. Due to the nature of lattice VQ, the proposed method achieves better R-D performance without optimizing and storing a large VQ codebook. Furthermore, for end-to-end optimization, the codeword probabilities are approximated by the Monte Carlo integration of multivariate normal distribution. On Kodak dataset, the proposed method achieves 18% BD-rate saving over the latest video coding standard H.266/VVC.", "summary_of_the_review": "Even though I think the idea is interesting and deserving of discussion (especially the entropy modeling), I still have concerns about the complexity and practicability of the proposed method, which play a crucial role for the task of image compression. And the experimental results are not enough to support the authors’ claims. I will reconsider based on the authors' response.", "strength_and_weaknesses": "Strengths: \n1.\tThe proposed entropy model is interesting and inspiring, i.e., approximating the discrete codeword probabilities by using the integration of continuous probability density.\n2.\tThe overall performance on Kodak dataset is significant.\n\nWeaknesses:\n1.\tComplexity and practicability. First of all, the authors didn’t provide any complexity evaluation, such as model size and encoding/decoding time. Despite the authors claimed that the use of lattice VQ solves the problem of codebook size bloating and provides fast quantization process, there is no experimental result to support this claim. More importantly, for lattice VQ, the complexity of practical entropy coding still exponentially increases with VQ dimensions. As mentioned in Section 3.2.2, the model calculates the cumulative probability table of a huge amount of sorted codewords, where the complexity is even larger than that of conventional vector quantization process. The entropy model needs to first produce and store the density integration of each codeword in the lattice VQ codebook, and it then obtains the cumulative probability table at both the encoder and decoder sides. In this paper, the VQ dimensions are set to 24, which is quite large for practical entropy coding. It is important to provide a detailed complexity evaluation for each part of the proposed method. Besides, are the claimed R-D results in Figure 4 & Figure 5 the practical coding results or the estimated results (by using -log(P))?\n2.\tMore results on more datasets. The proposed method is only evaluated on Kodak with PSNR metric. What about the performance on high-resolution datasets such as CLIC-2021 and Tecnick? And it would be better to provide the results in terms of MS-SSIM.\n\nMinor concerns:\n1.\tA related work [1] for VQ-based image compression is missing. Please provide the performance comparison and analysis.\n[1] Zhu, Xiaosu, et al. \"Unified Multivariate Gaussian Mixture for Efficient Neural Image Compression. CVPR2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "B8pltmesup", "reviewer": "Reviewer_nBnd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduced Lattice Vector Quantization into VAE-based image compression framework. Besides, a multivariate normal distribution including covariance matrix parameters are proposed. Experiments demonstratedd the effectiveness of these two features with a 18% BD-rate save compared with VTM15.0.\n", "summary_of_the_review": "This paper demonstrated that Lattice Vector Quantization can further improve the coding performance of VAE-based image compression framework. Experiments show the effectiveness but the conclusion can be stronger if more experiments can be performed. If the authors can provide more information about the implementation details of lvq, we can give a more solid recommendation.", "strength_and_weaknesses": "Experiments including the ablation study are convincing. The performance is also good.\nA more detailed introduction about some aspects of lattice vector quantization is preferred. For example, how the vector quantization matrix generation is performed during training.\nFor the experiment part, the effect of this quantization method on the computational complexity and memory consumption during training and inference is also expected. It will also be better if more experiments on different anchors (except Cheng) and different test datasets are performed.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject