Multi-Sample Training for Neural Image Compression
Authors: Tongda Xu
Yan Wang
Dailan He
Chenjian Gao
Han Gao
Kunzan Liu
Hongwei Qin
Venue: NeurIPS 2022 Accept (2022)

Abstract: This paper considers the problem of lossy neural image compression (NIC). Current state-of-the-art (SOTA) methods adopt uniform posterior to approximate quantization noise, and single-sample pathwise estimator to approximate the gradient of evidence lower bound (ELBO). In this paper, we propose to train NIC with multiple-sample importance weighted autoencoder (IWAE) target, which is tighter than ELBO and converges to log likelihood as sample size increases. First, we identify that the uniform posterior of NIC has special properties, which affect the variance and bias of pathwise and score function estimators of the IWAE target. Moreover, we provide insights on a commonly adopted trick in NIC from gradient variance perspective. Based on those analysis, we further propose multiple-sample NIC (MS-NIC), an enhanced IWAE target for NIC. Experimental results demonstrate that it improves SOTA NIC methods. Our MS-NIC is plug-and-play, and can be easily extended to neural video compression.


Multi-Sample Training for Neural Image Compression
Tongda Xul Yan Wang!2 Dailan He', Chenjian Gao! Han Gao 1:4
Kunzan Liu Hongwei Qin' SenseTimme Research, PInstitute for Al Industry Research (AIR), Tsinghua University, Beihang University, "University of Electronic Science and Technology of China, *Department of Electronic Engineering, Tsinghua University (xutongda, wangyan/@air. www.gnua..neee... edu. cn,
(hedailan, gaochenjian, gaohan1, liukunzan, qinhongwei wwweeettimeeeemmmmmeeee.
Abstract
This paper considers the problem of lossy neural image compression (NIC). Current state-of-the-art (sota) methods adopt uniform posterior to approximate quantiza- tion noise, and single-sample pathwise estimator to approximate the gradient evidence lower bound (ELBO). In this paper, we propose to train NIC with multiple-sampi importance weighted autoencoder (IWAE) target, which is tighter than ELBO and converges to log likelihood as sample size increases. First, we identify that the uniform posterior of NIC has special properties, which affect the variance and bias of pathwise and score function estimators of the IWAF target. Moreover, provide insights variance perspective. Based on those analysis, we further propose multiple-sample NIC (MS-NIC), an enhanced IWAE target for NIC. Experimental results demon- strate that it improves sota NIC methods. Our MS-NIC is plug-and-play, and can be easily extended to other neural compression tasks.
commonly adopted trick in NIC from gradient
1 Introduction
Latent variable-basee based lossy neural image compression (NIC) has witnessed significant success. The majority of NIC follows the framework proposed by Ballé image æ is transformed into by the encoder. Then scalar-quantized into integer Y, estimated with an entropy model p(ğ) and coded. For decoding, 3 is transformed back by the decoder obtain reconstructed ã. The optimization target of NIC is R-D cost: R + XD. R denotes the bitrate of J. D denotes the distortion between æ and æ. and À denotes the hyper-paramete controlling their trade-off. During training, the quantization noise. And e is fully factorized uniform noise e p(E) = 1[ +
[2017]: For encoding, the original scalar
[y] is relaxed with ú =
to simulate the quantization
Ballé et al. 2017] further recognises that such training framework is closely related to variational inference. Indeed, the above process be formulated is transformed into variational parameter y by inference model (encoder), and ÿ is sampled from variational posterior q æ)), which a unit unifrom distribution centered in y. The prior likelihood p(ğ) is computed, and is transformed back by the generative model (decoder) to compute the likelihood p(== Under such formulation, the prior connected to the bitrate, the likelihood connected to the distortion, and the posterior likelihood is connected to the bits-back bitrate (See Appendix. 2.3 which 0 in NIC. Finally, the evidence lower bound (ELBO) is the negative R + XD target (Eq. Denote the transform function ğ(E; Φ) y e, and sampling
graphic model ÿ. During encoding,
q(ā]z) (e) is equivalent to transforming through ő(E; Φ). Then the gradient of ELBO is estimated via pathwise estimator with single-sample Monte Carlo (Eq. 2h. This is the same SGVB-1 Kingma and Welling 2013)
Yan Wang is the corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

L= - AD) Butgla poggpaa
log p(ū)
log ( (x)
(1)
back
VoL = Ep(e) V (log pla. [[[)):) )]
log
((c,)) {(]))))]
(2)
Ballé et al. 2018 further extends this framework into a two-level hierarchical structure, with graphic model + y 2. The variational posterior is fully factorized uniform distribution uly - 311 + [(( 2 2 + To simulate the quantization noise. And y, 2 denote outputs of their inference networks.
C R plgg(a))) + log ž) + log p()) log (g) log q(
distertion
rate
bits back rate:
The majority of later NIC follows this hierarchical latent framework [Minnen et al.
2018 Cheng
al. 2020). Some focus on more expressive network architectures (Zhu al. 202 Xie
2021
some stress better context models [Minnen and Singh, 20200 He et al. 2021 Guo et al. some emphasize semi-amon amortizaotin inference (Yang al.] 2020]. However, there little research on multiple-sample methods, or other techniques for tighter ELBO.
202 la and
On the other hand, IWAE [Burda al., 2016 has been successful in density estimation. Specifically, IWAE considers a multiple-sample lowerbound Lı (Eq. which is tighter than its single-sample counterpart. The benefit of such bound is that the implicit distribution defined by IWAE approaches true posterior as k increases [Cremer et al. (017]. This suggests that its variational posterior is less single mode of true posterior, and the learned representation richer. The gradient of L; is computed via pathwise estimator. Denote the exponential ELBO sample as u,, its reparameterization as w(;;; Φ), and its weight w; = wyy Then VoLc has the form of importance weighted sum (Eq, 5).
likely to collapse
p(x. He)
k p(a, T 0)) pp(e (log x
Cx E flog 1
g(o(E;; Φi (z))
(4)
w((;0)
V VoL. Ep(c
1 te V log ((;)()))) ~
> Uz V log ((ei
(5)
In this paper, consider the problem of training NIC with multiple-sample IWAE target (Eq. 4. which allows us to learn a richer latent space. First, we recognise that NIC's factorized uniform variational posterior has impacts we find NIC's pathwise gradient estimator equivalent to an improved STL estimator [Roeder et al. 2017]. which is unbiased even for the IWAE target. However, NIC's IWAE-DReG estimator [Tucker et al. 2018) has extra bias, which causes performance decay. Moreover, we provide insights on commonly adopted but little explained trick of training NIC from gradient variance perspective. Based on those analysis and observations, we further propose MS-NIC, a novel improvement of multiple- sample IWAE target for NIC. Experimental results show that improves sota NIC methods [Ballè et al. 2018 Cheng et al. 2020 and learns richer latent representation. Our method is plug-and-play and can be extended into neural video compression.
variance and bias properties of gradient estimators. Specifically,
To wrap up, our contributions are as follows:
We provide insights on the impact of the uniform variational posterior upon gradient estimators, bits-back coding and commonly adopted but little discussed trick of NIC training from gradient variance perspective.
We propose sample neural image compression (MS-NIC). It a novel enhancement of hierarchical IWAE [Burda et al. 2016) for neural image compression. To the best of our knowledge, we are the first to consider a tighter ELBO for training neural image compression.

---

We demonstrate the efficiency of MS-NIC through experimental results on sota NIC methods. Our method is plug-and-plaz for neural image compression and can be easily applied to neural video compression.
2 Gradient Estimation for Neural Image Compression
The common NIC framework (Eq.
1
Eq
adopts fully factorized uniform distribution g(.. =|x) = to simulate the quantization noise. Such formulation has
I[u(y² _14 y 5) [I 2
the following special properties:
Property I: q( ()) and ( x) is support depends the parameter.
Property II: log ( log ( a)) on their support.
The impacts of these two properties are frequently neglected in previous works, which does not influence the results for single-sample pathwise gradient estimators (a.k.a. reparameterization trick in Kingma and Welling [2013]). this section, we discuss the impacts of these two properties upon the variance and biasness of gradient estimators. Our analysis is based on single level latent (Eq. instead of hierarchical latent (Eq. to simplify notations.
2.1 Impact on Pathwise Gradient Estimators
First, let's consider the single-sample case. We can expand the pathwise gradient of ELBO in Eq. into Eq. [6 As indicated in the equation, & contributes to L in ways. The first way is through the reparametrized ((E: [: Φ) (pathwise term), and the other way is through the parameter of log 9 2)) (parameter score term). Generally, the parameter score term has higher variance than the pathwise term. The STL [Roeder et al. 2017) reduces the gradient by dropping the score. It is unbiased since the dropped term's is expectation (x())=)) a log 9 (g) a=) is
2
p(a ()
VoL [Epce V (log
VV (E) - V log go (ūlz)
parameter score term
pathwise
Now let's consider the STL estimator of multiple-sample IWAE bound (Eq. As shown in Tucker et al. 20188. the STL estimation of IWAE bound gradient is biased. To reveal the reason, consider expanding the gradient Eq. into partial derivatives we expand Eq. 2 into Eq. 6 Unlike single- sample case, the dropped parameter score term Ep(cya) [Σ ū. (-V & log g y 5 is no longer 0 due the importance weight प. This means that STL loses its unbiasness general IWAE cases.
Regarding NIC, however, the direct pathwise gradient for IWAE bound is automatically an unbiased STL estimator. Property II means that variational posterior has constant entropy, which further means that the parameter score gradient is O. So, NIC's pathwise gradient of IWAE bound is equvailent to an extended, unbiased STL estimator.
2.2 Impact on Score Function Gradient Estimators
In previous section, we show the bless NIC's special properties on pathwise gradient estimators. In this section, show their curse score function gradient estimators. Sepcifically, Property I implics that (²)))) and ())(z) are not absolute continuous, and hence the score function gradient estimators of those distributions are biased.
For example, consider a univariate random variable x po(x) u( 1 0 1). Our task is
to estimate the gradient of a differentiable function f(x). And consider the à è-independent random variable ~ p(+) = ud( (1 +5), the transform æfe; 0) = 0 + e. Under such conditions, the Monte Carlo estimated pathwise gradient and score function gradient are:
pathwise gradient: Volpotz) Vo Vol ~ > VoJ +
(7)
score function gradient: Vel Velpo(> = (o = 0
(8)
3

---

Eq. 7 does not equal to Eq. 8 and Eq is wrong. The score function gradient is only unbiased when the distribution satisfies the absolute continuity condition of MMhamedd al., 2020]. This reflects that under the formulation of NIC, the equivalence between the score function gradient (a.k.a.
REINFORCE [Williams] [992]]) and pathwise gradient (a.k.a reparameterization trick in [Kingma
and Welling 2013 no longer holds.
Table 1: Effect of DReG gradient estimator in NIC.
Sample Size
bpp
MSE
PSNR (db)
R-D cost
Single-sample
Baseline [Ballé al. 2018
0.5273
32.61
33.28
1.017
Multiple-samppi
MS-NIC-MIX (pathwiss gradient) MS-NIC-MIX(DReG gradient)
5
0.5259
31.84
33.38
1.003
0.5316
35.09
32.90
1.058
Such equivalence is the cornerstone of many gradient estimators, and IWAE-DReG [uuckrr et al. 2018) is one of them. IWAE-DReG is a popular gradient estimator for IWAE target (Eq. as resolves the vanish of inference network gradient SNR (signal to noise ratio). However, the correctness WWE-DRed depends on the equivalence between the score function gradient and pathwise gradient, which does not hold for NIC. Specifically, IWAB-DReG expand the total derivative of IWAE target as Eq. 9 and perform another round of reparameterization on the score function term Eq. To further reduce the gradient variance. However, Eq. requurres the equivalence of pathwise gradient and score function gradient.
Vall (B L2) llog A
wi} Ep Ep(en
Wi
log 4s U Φ)
2 log w(en 6) @ğ(e: 0)
0ÿ:
OΦ
1=1
Wj
scure function termi
(9)
W2 ô log qs (ū: (ū.
0 Wt 05 (ū(C;:)) o)
E (m)
E Eptccl
(10)
Σ}=1 W
0g:
Σ Eド=10 Wy
0ø:
As we show empirically in Tab. I blindly adopting IWAE-DReG estimator for multiple-sample NIC brings evident performance decay. Other than IWAE-DReG, many other graident estimators such as NVIL MMni and Gregor 2014), VIMCO [Mnih and Rezende 2016) and GDReG [Bauer and Mnih do not apply to NIC. They either bring some extra bias or are totally wrong.
2021
2.3
Impact on
Coding
the minus overall bitrate for bits-back coding in compression [Hinton and Van Camp 1993 Hinton et al. 1995 Chen et al., 2017 and the entropy of variational
is well known that the ELBO
posterior exactly the bits-back back rate itself. For this reason, earlier works [Townsend et al. 2018 Yang
et al. 2020 point out that [Ballé et al.) 2018 Minnen et al., 2018) waste bits for not using bits back coding on 2. However, during training the differential entropy [oover, 1999 E [log ( [g) is constant. And this means that this term does not have impact on the optimization procedure. And due to the deterministic inference, the log y 2 7) is O, which means that the bitrate saved by bits-back coding is 0. In this sense, [Ballé et al.) 2018 Minnen et al. 2018 is also optimal in bits-hack coding perspective, although no actual bits-back coding performed. In fact, there no space for bits-back coding long encoder is deterministic. Since we can view deterministic encoder posterior distribution with mass on a single point. And then the posterior's s entropy is always
2.4 The direct-y Trick in Training NIC
In NIC, we feed deterministic parameter into Z inference model instead of noisy samples ý. This implies that ž is sampled from (((lg) instead q(īl)). This trick is initially adopted in Ballé et al. 2018 and followed by most of the subsequent works. However, it is little discussed. In this

---

paper, we refer it to direct-y trick. Yang et al. [2020] observes that feeding ÿ instead of y causes severe performance decay. We confirm this result Tab. Thus, direct-y trick is essential train hierarchical NIC.
Table 2: Effects of direct-y R-D performance. 2-1evel VAE is equivalent to Ballé al. 2018 without direct-y.
bpp
MSE
PSNR
R-D cost
2-Level VAE
0.9968
33.08
33.22
1.493
Ballé al. 2018
0.5273
32.61
33.28
1.017
Table Effects of direct-y on gradient SNR of different parts of the model. 2-1evel VAE equivalent to Ballé et al. [2018] without direct-y. "early" is 5 10 iterations, "mid" is 5 x 10 iterations and "late" x 10 iterations. "infer" the abbreviation for "inference model", and "gen" the abbreviation for "generative model".
gradient SNR of #
Iteration
Method
infer
y gen
7 infer
Z. gen
7. prior
early
2-level VAE
2.287
0.5343
0.3419
0.4099 0.3813
0.9991
Ballé et al. [2018)
2.174
0.5179
0.5341
1.069
mid
2-level VAB
1.350
0.4793
0.2414
0.3583
0.8861
Ballé et al. [2018]
1.334
0.4813
0.4879
0.3761
0.9693
late
2-level VAE
1.217
0.4746
0.2863
0.3439
0.8691
Ballé al. [0018]
1.206
0.4763
0.5506
0.3707
0.9339
One explanation is to view ((āly) as ((ālz), and q(. 2 ž)z) factorized as q( x (( (((z)) (See Fig ! (a)-(c)), A similar trick of feeding mean parameter can be traced back to the Helmholtz machine [Dayan et al.) 1993]. However, this provides rationale why direct-y is fine to be adopted but does not explain why samping z from q( y) fails. We provide an alternative explanation from the gradient variance perspective. Specifically, q( z)) has two stochastic arguments that could cause high variance in the gradient of inference model. and make its convergence difficult. To verifyy this, we follow Rainforth et al. (0088) to compare the gradient SNR, which is the absolute value of the empirical mean divided by standard deviation. We trace the gradient SNR during different training stages as model converges (See Sec. 5.1 for detailed setups).
As demonstrated in Tab. 3 the gradient SNR of z inference model of standard 2-level VAE (without direct y) is indeed significantly lower than Ballé et al. (2018) (with direct y) during all 3 stage of training. This result reveals that the z inference model is more difficult to train without direct-y, And such difficulty could be the source of the failure of NIC without direct-y trick.
3 Multiple-sample Neural Image Compression
In this section, we consider the multiple-sample approach based on the 2-level hierarchical framework by [Ballé et al. [2018), which is the de facto NIC architecture adopted by many sota methods. To simplify notations, log ( (5) and log ( directly applying 2-1evel IWAE to NIC without direct-y trick (See Fig. (d)). Regarding A sample IWAE, we first compute parameter y of q x)) and sample Li.k from it. Then, we compute parameter Zish of q( & (1:k) and samples Σ from it. Afterward, F1.k and ă are fed into the generative model and compute wi-k- Finally, we follow Eq to compute the gradient and update parameters. In fact, this is the standard 2-level IWAE in the original IWAE paper.
x) in ELBO are omitted as they are 0. First, let's consider
However, the vanilla 2-level IWAE becomes problem for NIC with direct-y trick. Concerning a sample IWAE, we sample &I. from x))x.). Due to the direct-y trick, we feed y instead of F1. into z inference network, and our ((āl)) has only one parameter other than parameter 21:k- If we follow the 2-level IWAE approach, only one sample ž is obtained, and wiek can not be computed.
5

---

2
2
y
(a). generative model (b)-(c)
(b). inference model of VAE
(e). inference model of Balll al., 2018]
(d), inference model IWAE
(c). inference model of MS-NIC-MIX
(0) inference model MS-NIC-DMS
Figure 1: The plate notation of different NIC methods. æ is the observed image, y and are latent. The inference models show how sample from variational posterior duing training. is the number of data points in dataset, k, I is the sample size multiple-sampp approaches. The generative model of (b), (c) is (a). The generative model of (d)-(f) is shown in Appendix. A.I For clarity, omit the parameters.
One method is limit the multiple-sampl part to y related term only and optimize other parts via single-sample SGVB-1, which produces our MS-NIC-MIX (See Fig (e)). Another method is to sample another 1 samples of Σ, from (( and nest with MS-NIC-MIX, which generates our MS-NIC-DMS (See Fig (f).
3.1 MS-NIC-MIX: Multiple-sample NIC with Mixture
One way to optimize multiple-sample IWAE target of NIC with direct-y trick is sample ğ k times to obtain V1:k and & only 1 time. Then we perform i sample log mean of p( ( ) ()) to obtain muultipe-samplee estimated log p( ž), add with single-sample log ()). This brings C CMIX with the form of mixture of 1-level VAE and 1-level IWAE ELBO:
CMIX E ( E ( [log Ep(r) (() + logp($))
(11)
Moreover, L. is reasonably preferable target over ELBO as satisfies the following properties (See Appendix. A.2 for proof):
1. log p(x) 2 LMIX
2. LMIX; 2 CMIX for k 2
Although L does not converge to true log p(x) grows, it is still lower bound of log p(æ) and tighter than ELBO (as C ELBO), Its gradient can be computed via pathwise estimator. Denote the per-sample integrand p( ÿ. ( E) as W.X.X and its relative weight as W.M then the gradient V VoL. LIX can be estimated as Eq. 13
C = =E Eptey, [lg :E (
tog
&
(12)
~ log + p(olū(e" o) (( ()E o)) +og
w MIX (0))
V CMIX DupIIx log i MIX (6) 60:0) + V logp(((();)))
(13)
Another way understand MS-NIC-MIX is to view the y inference/generative model as a single level IWAE, and the inference/gemerrtiie model as large prior of ÿ which is optimized via SGVB-1.

---

and This perspective is often taken by works in NIC context model [Minnen et al. 2018 He et al. 2021 as the context model of NIC often limited to y.
3.2 MS-NIC-DMS: Multiple-sample NIC with Double Multiple Sampling
An intuitive improvement over MS-NIC-MIX is add another round of multiple-sample over Specifically, we sample ž times, nest it with L to obtain kx LPMS
LDMS = R (E log E exp (Bna(tna Dog E ( (11
(14)
And name it MS-NIC-DMS it adopts multiple sampling twice. Moreover, LPA is reasonably better target for optimizaion over ELBO and L as satisfies the following properties (See proof in Appendix. :
1. log (ogp(x)= 2
CDMS
2. K. 2 CDMS for 2 m,l n
3. CDMS; 2 C MIX
4. CDMS + p(z) as k + oo, under the assumption that log pp kRl"
((9.2))
log p(a) (p(z) are bounded.
((E](=))
In other words, the target LDMS is lowerbound of log p(x). converging to log p(z)( as k, - oc, tighter than L and tighter than ELBO (as C ELBO). However, its Monte Carlo estimation biased due the nested transformation and expectation. Empirically, we find that directly adopting biased pathwise estimator works fine. And its gradient can be estimated by pathwise estimator similar original IWAE target (See Eq. 5).
CDS = Eg(et,) ) flog E exp (B flog & p(( (le.
log t 1 1 E (0)
(15)
Another interpretation of MS-NIC-DMS is view multiple level IWAE with repeated local samples. The LDMS Monte Carlo pathwise estimator has the form of IWAE with / samples. However, there are multiple repeated samples that contain the same Y and žz. For example, the DMS samples TW AE of 2 level IWAE with sample size 6 look like Eq. 16 While the samples w12,1:3 of MS-NIC-DMS with
3 samples look like Eq. 17 We independently sampled ÿ and ž, while in MS-NIC-DMS, we have 2 independent ÿ and independent ž, they paired to generate 6 samples in total. Note that this is only applicable to NIC as ÿ and are conditionally independent given due to direct-y trick.
that in IWAE,
have 6 pairs of
w
TWAR (p(=))1)p(ūn 2 1:6
DMS 1:2,1:3
{p(x )
p(
U2) (2 (2)
p( Ø1 |r 52) (8)) 8g),
p(a (s)p((p) 2. Eg).
p(m in ( Es)p
(16)
(17)
p(a: 94)p (4)
p x
p(a 2s)
p( a (2)
p(
p x (2)

---

Related Work
4.1 Lossy Neural Image and Video Compression
Ballé et al. [2017] and Ballé et al. (2018) formulate lossy neural image compression variational inference problem, by interpreting the additive uniform noise (AUN) relaxed scalar quantization as a factorized uniform variational posterior. After that, the majority of sota lossy neural image
compression methods adopt this formulation [Minnen et al. 20188 Minnen and Singh 2020 Cheng
et al.
2020 Guo et al., 22221a Gao et al., | He et al. [022]. And Yang et al. [2020), Guo et al.
202 also require a AUN trained NIC as base. Moreover, the majority of neural video compression
also adopts this formulation [Lu et al.
2019 2020 Agustsson et al. 2020. Hu et al.) 2021 Li et al.)
2021
implying that MS-NIC can be extended to video compression without much pain.
Other approaches to train NIC include random rounding (Toderici et al., 2015 2017] and straight through estimator (STE) Theis et al., 2017]. Another promising approach is the VQ- VAE [Van Den Oord et al. 2017]. By the submission of this manuscript, one unarchived work (Zhu al.) 2022] has shown the potential VQ-VAB in practical NIC. Our MS-NIC does not apply to the approaches mentioned in this paragraph, the formulation of variational posterior is different.
4.2 Tighter Lower Bound for VAE
IWAE [Burda et al., 2016] stirs up the discussion of adopting tighter lower bound for training VAEs. However, the first glance it is not straightforward why it might works. Cremer et al. 2018 decomposes the inference suboptimality of VAE into two parts: 1) The limited expressiveness of interence model. 2) The gap between ELBO and log likelihood. However, this gap refers to inference not training. The original IWAE paper empirically shows that IWAE can learn a richer latent representation. Cremer al. [2017] shows that the IWAE target converges to ELBO under the expectation of true posterior. And thus the posterior collapse is avoided.
From the information preference [Chen al. 2017] perspective, VAE prefers to distribute information in generative distribution than autoencoding information in the latent. This preference formulates another view of posterior collapse. And it stems from the gap between ELBO and true log likelihood. There are various approaches alleviating it, including soft free bits (Theis et al.] 2017]] and KL annealing [Serban et al., 2017). In our opinion, IWAE also belongs to those methods, and it is asymptotically optimal. However, we have not found many works comparing IWAE with those methods. Moreover, those approaches are rarely adopted in NIC community.
Many follow-ups of IWAE stress gradient variance reduction Roeder et al. 2017 Tucker et al. 2018 Rainforth et al., 2018), discrete latent (Mnih and Rezende [0166] and debiasing IWAE target Nowozin 2018). Although the idea of tighter low bound training has been applied to the field of neural joint source channel coding [Choi et al. 2018 Song al. 2020 to the best of our knowledge, no work in NIC consider it yet.
4.3 Multi-Sample Inference for Neural Image Compression
Theis and Ho| 2021] considers the similar topic of importance weighted NIC. However, does not consider traming of NIC. Instead, it focuses on achieving IWAE target with an entropy coding technique named softmin, just like BB-ANS [Townsend et al., 2018 achieving ELBO. alluring to apply softmin to MS-NIC, as it closes the muluple-sample training and inference gap. However, it requires large number of samples (e.g. 4096) to achieve slight improvement for 64x64 images. The potential sample size required for practical NIC is forbidding. Moreover, we believe the stochastic lossy encoding scheme [Agustsson and Theis. 2020 that Theis and Ho (202 not yet ready to be applied (See Appendix. A.7 for details).
5
Experimental Results
5.1 Experimental Settings
Following He et al. 2022], we train all the models on the largest 8000 images of ImageNet [Deng
al. 2009 followed by downsampling according to Ballé al. (2018) And we use Kodak [Kddkk
8

---

1993 for evaluation. For the experiments based on Ballé et al. [2018] (include Tab. Tab. 2 we follows the setting of the original paper except for the selection Äs. For the selection Äs, we set À E {0.0016, 0.0032, 0.0075, 0.015. 0.03. 0.045, 0.08) as suggested in Cheng et al. [000). And for the experiments based on Cheng al. 2020 we follows the setting of original paper. More detailed experimental settings can be found in Appendix. AA.4
And when comparing the R-D performance of models trained on multiple As, use Bjontegaard metric (BD-Metric) and Bjontegaard bitrate (BD-BR) [Bjontegaard] 2001]. which is widely applied when comparing codecs. More detailed experimental results can found in Appendix, A.5
Table 4: Results based on Ballé et al. 2018
PSNR
MS-SSIM
BD-BR (%)
BD-Metric
BD-BR (%)
BD-Metric
Single-sample
Baseline [Ballé et al.
2018
0,000
0.000
0.000
0.0000
Multiple-sampa sample IWAE [Burda et al.
2016
64.23
-2.318
68.67
-0.01648
-I--II-MMIX
-3.847
0.1877 0.2405
-4.743
0.001618
MS- MSS-NIC-MMS
-4929
-5.617
0.001976
Table 5: Results based on Cheng et al. R-D is not monotonously increasing.
2020
The BD Metrics of IWAE can not be computed as its
PSNR
MS-SSIM
BD-BR
BD-Metric
BD-BR (%)
BD-Metric
Single-sample
Baseline [Cheng al.
2020
0.0000
0.0000
0.0000
0.0000
Mulriple-sample
IWAE [Burda et al.
2016
MS-NIC-MIX
-1.852
0.0805
2.238 1.998
-0.0006764
MS-NIC-DMS
-2.378
0.1046
-0.0006054
5.2 R-D Performance
We evaluate the performance of MS-NIC-MIX and MS-NIC-DMS based NIC methods [Ballé jet al. 2018 Cheng et al. 2020]. Empirically, we find that MS-NIC-MIX works best with sample size and MS-NIC-DMS with sample size 16. The experimental results on sample size selection can be found in Appendix. AA.3 Without special mention, we set the sample size of MS-NIC-MIX to 8 and
MS-NIC-DMS 16.
For Ballé et al. 2018 MS-NIC-MIX saves around 4% bitrate compared with single-sample
And MS-NIC-DMS saves around 5% of bitrate. On the other hand, the original IWAE suffers per performance decay as it is not compatible with direct-y trick. For Cheng et al. 2020 find that both MS-NIC-MIX and NS-NIC-DMS suppress baseline in PSNR. However, is as evident as Ballé et al. 2018). Moreover, the MS-SSIM is slightly lower than the baseline. This probably due to the auto-regressive context model. Besides, the original IWAE without direct-y trick suffers from severe performance decay in both cases. The BD metric of IWAE on Cheng et al. (2020) can not be computed as its R-D is not monotonous increasing, we refer interested readers to Appendix. A.5 for details.
baseline (See Tab.
5.3
Latent Space Representation of MS-NIC
To better understand the latent learned by MS-NIC, evaluate the variance and coefficient of variation (Cov) of per-dimension latent distribution mean parameter (1), 2(1, with regard to input

---

distribution p(z). As we are also interested in the discrete representation, we provide statistics of rounded mean y(, z(() These metries show how much do latents vary when imput changes, and a large variation latents means that there are useful information encoded. A really small variation indicates that the latent is "dead" in that dimension.
As shown in Tab. of Appendix. A.6 the latent of multiple-sample approaches has higher variance than those of single-sample approach. Moreover, the Cov(y) of multiple-sample approaches is around 4 5 times higher than single-sample approach. Although the Cov(z) of multiple-sample approaches is around 2 times lower, the main contributor of image reconstruction is y. and 2 only serves predict y's s distribution. Similar trend can be concluded from quantized latents y, ž. From the variance and Cov perspective, the latent learned by MS-NIC is richer than single-sample approach. It is also noteworthy that although the variance and Cov of y. y of MS-NIC is significantly higher than dinglessapll approach, the bpp only varies slightly.
Table The average of per-dimension latent variance and Cov across Kodak test images. The model is trained with À = 0.015.
Var(#)
Cov(#)
bpp of #
Method
y
z
y
y
Single-sample
Ballé et al. [0188)
1.499
0.3255
19.70
9.944
0.5136
0.01342
Mulriple-sample
MS-NIC ------I-IX
1.906
0.7594
111.1
7.425
0.5108
0,01521
-IS-NICDMMS
1.919
0.7648
95.51
7.243
0.5092
0.01634
6 Limitation & Discussion
A major limitation of our method is that the improvement in R-D performance is marginal, especially when based on Cheng al.) [2200. Moreover, evaluations on more recent sota methods are also helpful to strengthen the claims of this paper. In general, we think that the performance improvement of our approach bounded by how severe the posterior collapse is in neural image compression. We measure the variance in latent dimension according to data in Fig. A.6] And from that figure it might be observed that the major divergence of IWAE and VAE happens when the variance is very small. And for the area where variance is reasonably large, the gain of IWAE is not that large. This probably indicates that the posterior collapse in neural image compression is only alleviated to limited extend.
See more discussion in why the result on Cheng et al. [2020 negative in Appendix. A.9
7 Conclusion
In this paper we propose MS-NIC, multiple-sample importance weighted target for training NIC. It improves sota NIC methods and learns richer latent representation. A known limitation is that its R-D performance improvement is limited when applied to models with spatial context models (c.g. Cheng et al. )020).. Despite the somewhat negative result, this paper provides insights to the training of NIC models from VAE perspective. Further work could consider improving the performance and extend it into neural video compression.
Acknowledgments and Disclosure of Funding
This work is supported by SenseTime Research. The content solely the responsibility of the authors and does not necessarily represent the official views of SenseTime Research.
10

---

References
E. Agustsson and L. Theis. Universally quantized neural compression. Advances in neural information processing systems, 33: 33:12367-12376, 2020.
E. Agustsson, D. Minnen, Johnston, Balle, Hwang, and G. Toderici. Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 8503-8512, 2020.
J. Ballé, V. Laparra, and E. P. Simoncelli. End-to-end optimized image compression. In International Conference on Learning Representations, 2017.
Ballé, D. Minnen, Singh, S. J. Hwang, and N. Johnston. Variational image compression with a scale hyperprior. In International Conference Learning Representations, 2018.
M. Bauer and A. Mnih. Generalized doubly reparameterized gradient estimators. In International Conference on Machine Learning, pages 738-747. PMLR, 2021.
G. Bjontegaard. Calculation of average psnr differences between rd-curves. VCEG-M33, 2001.
B. Bross, J. Chen, J.-R. Ohm, G. Sullivan, and Y.-K. Wang. Developments in international video coding standardization after ave, with overview of versatile video coding (vvc). Proceedings of the IEEE, 109(9):1433-1933 2021.
Y. Burda, R. B. Grosse, and R. Salakhutdinov. Importance weighted autoencoders. In ICLR (Poster), 2016.
X. Chen, D. P. Kingma, T. Salimans, Duan, P. Dhariwal, Schulman, I. Sutskever, and P. Abbeel. Variational lossy autoencoder. In International Conference Learning Representations, 2017.
Z. Cheng, Sun, M. Takeuchi, and Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7939-7948, 2020.
K. Choi, K. Tatwawadi, T. Weissman, and S. Ermon. Necst: neural joint source-channel coding. 2018.
M. Cover. Elements of information theory. John Wiley Sons, 1999.
C. Cremer, Q. Morris, and D. Duvenaud. Reinterpreting ayportancc-weightie autoencoders, 2017.
C. Cremer, X. Li, and D. Duvenaud. Inference suboptimality in variational autoencoders. In International Conference on Machine Learning, pages 1078-1086. PMLR, 2018
P. Dayan, G. E. Hinton, R. M. Neal, and R. S. Zemel. The helmholtz machine. Neural computation, 7(5):889 9009,99044410 1995.
J. Deng, W. Dong, R. Socher, L.-J. K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Ieee, 2009.
G. Flamich, M. Havasi, and J. M. Hermández-Lobato. Compressing images by encoding their latent representations with relative entropy coding. Advances in Neural Information Processing Systems, 33:16131-16144 2020.
G. Gao, You, R. Pan. S. Han, Zhang, Dai, and H. Lee. Neural image compression via attentional multi-scale back projection and frequency decomposition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14677-14686, 2021.
Z. Guo, 7. Zhang, R. Feng, and Z. Chen. Causal contextual prediction for learned image compression. IEEE Transactions Circuits and Systems for Video Technology, 2021a.
Guo, 7.. Zhang, Feng, and Chen. Soft then hard: Rethinking the quantization in neural image compression. In International Conference on Machine Learning, pages 3920-3929, PMLR, lb.

---

D. He, Y. Zheng. B. Sun, Y. Wang. and H. Qin. Checkerboard context model for efficient learned image compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14771-14780, 2021.
D. He, Z. Yang, W. Peng, R. Ma, H. Qin, and Y. Wang. Elic: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. arXiv preprint arXiv:2203.10886, 2022.
G. E. Hinton and D. Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Compulational learning theory, pages 5-13, 1993.
G. E. Hinton, P. Dayan, B. J. Frey, and R. M. Neal. The" wake-sleep" algorithm for unsupervised neural networks. Science, 168 1995.
Z. Hu, G. Lu, and D. Xu. Fve: A new framework towards deep video compression in feature space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1502-1511, 2021.
D. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. 2013.
E. Kodak. Kodak lossless true color image suite. http:///wk.//grpppcca//////aaaaa 1993.
J. Li, B. Li, and Y. Lu. Deep contextual video compression. Advances in Neural Information Processing Systems, 34, 2021.
I. Loshchilov and Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.
G. Lu, W. Ouyang, D. Xu, X. Zhang, Cai, and Z. Gao. Dve: An end-to-end deep video compres- sion framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1006-11015, 2019.
G. Lu, X. Zhang, W. Ouyang, L. Chen, Z. Gao, and D. Xu. An end-to-end learning framework for video compression. IEEE transactions on pattern analysis and machine intelligence, 43(10): 2020.
D. Minnen and S. Singh. Channel-wise autoregressive entropy models for learned image compression. In 2020 IEEE International Conference on Image Processing (ICIP), pages 3339-3343, IEEE, 2020.
D. Minnen, J. Ballé, and G. D. Toderici. Joint autoregressive and hierarchical priors for learned image compression. Advances in neural information processing systems, 31, 2018.
A. Mnih and K. Gregor. Neural variational inference and learning in belief networks. In International Conference Machine Learning, pages 1791-1799. PMLR, 2014.
A. Mnih and D. Rezende. Variational inference for monte carlo objectives. In International Conference on Machine Learning, pages 2188-2198 PMLR, 2016.
S. Mohamed, M. Rosca, M. Figurnov, and A. Mnih. Monte carlo gradient estimation in machine learning. J. Mach. Learn. Res., 21(132):1-62, 2020.
S. Nowozin. Debiasing evidence approximations: On suportance-weightere autoencoders and jackknife variational inference. In International conference on learning representations, 2018.
M. B. Paulus, C. J. Maddison, and A. Krause. Rao-blackwellizinn the traight-throughe gumbel- softmmx gradient estimator. arXiv preprint arXiv:2010.04833 2020.
T. Rainforth, A. Kosiorek, T. A. Le, C. Maddison, M. Igl, F. Wood, and Y. W. Teh. Tighter variational bounds are not necessarily better. International Conference on Machine Learning, pages 4277-4285. PMLR, 2018.
12

---

G. Roeder, Y. Wu, and D. K. Duvenaud. Sticking the landing: Simple, lower-varianee gradient estimators for variational inference. Advances in Neural Information Processing Systems, 30, 2017.
Ryder, C. Zhang. N. Kang, and S. Zhang. Split hierarchical variational compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 386-395, 2022.
I. Serban, A. Sordoni. R. Lowe, L. Charlin, J. Pineau, A. Courville, and Y. Bengio. A hierarchical latent variable encoder-decoder model for generating dialogues. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Y. Song, M. Xu, L. Yu, H. Zhou, S. Shao, and Y. Yu. Infomax neural joint source-channel coding via adversarial bit flip. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages
2020.
L. Theis and E. Agustsson. On the advantages of stochastic encoders. arXiv:2102.094 092007 2021.
ar preprint
L. Theis and Ho. Importance weighted compression. In Neural Compression: From Information Theory to -otirttiionnoooooopppeee ICLR 2021, 2021.
L. Theis, W. Shi, A. Cunningham, and F. Huszár. Lossy image compression with compressive autoencoders. arXiv preprint arXiv:1703.0039) 2017.
G. Toderici, S. M. O'Malley, J. Hwang, Vincent, D. Minnen, Baluja, M. Covell, and R. Sukthankar, Variable rate image compression with recurrent neural networks. arXiv preprint arXiv:1511.06085 2015.
G. Toderici, D. Vincent, N. Johnston, S. Jin Hwang, D. Minnen, Shor, and M. Covell. Full resolution image compression with recurrent neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1300-5314, 2017.
Townsend, T. Bird, and D. Barber. Practical lossless compression with latent variables using bits back coding. In International Conference on Learning Representations, 2018.
G. Tucker, D. Lawson, S. Gu, and C. J. Maddison. Doubly reparameterized gradient estimators for monte carlo objectives. 2018.
A. Van Den Oord, O. Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-266, 1992.
Y. Xie, K. L. Cheng, and Chen. Enhanced invertible encoding for learned image compression. In Proceedings of the 29th ACM International Conference on Multimedia, pages 162-170, 2021.
Y. Yang, R. Bamler, and S. Mandt, Improving inference for neural image compression. Advances in Neural Information Processing Systems, 33:573-584, 2020.
X. Zhu, J. Song, L. Gao, F. Zheng, and H. Shen. Unified multivariate gaussian mixture for efficient neural image compression. arXiv preprint arXiv:2203.108/) 2022.
Y. Zhu, Y. Yang, and T. Cohen. Transformer-based transform coding. International Conference on Learning Representations, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you deseribe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix
13

---

(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes) In Appendix
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi-
ments multiple times)? [No] It really expensive to train a NIC model. The error bar
rarely reported out of toy size model and dataset.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "rZtzMELI-_7", "reviewer": "Reviewer_XrNr", "summary": "This paper proposes to train neural compression VAEs with multi-sample IWAE-style objectives, and analyzes the effect of the choice of variational distribution (constant width uniform posterior; the choice between q(y|x) v.s. q(y|z)) on various gradient estimators. They theoretically showed that the uniform posterior in neural compression automatically implements a form of the STL estimator [Roeder et al., 2017], and that naively applying the score function gradient estimator would give incorrect results. They propose two IWAE-style objectives that are potentially tighter than the standard NELBO objective, and show empirically that training on them yield small improvement in R-D performance (5% rate savings). ", "questions": "There are a few claims in this paper that I'm not unsure about. \n\n1. I'm not following the discussions around line 153, regarding why the direct-y trick poses an issue for IWAE. Here we simply have a factorized posterior q(z, y|x) = q(z|x) q(y|x), so why can't we just use k samples of (z, y) from this joint posterior to form an IWAE estimator?\n\n2. The authors' statements about IWAE estimator converging to log p(x) (e.g., line 187-189) can be misleading, as they may not hold for the neural compression VAEs. Again this has to do with the posterior having bounded support, whereas the prior and likelihood models having full support. \n\n3. Is there an obvious explanation for why the mult-sample IWAE [Burda et al., 2016] baseline is so much worse in Table 5? I could not find it. \n\n3. This is a nitpick: the statement  \"q(z˜|y˜) and q(y˜|x˜) are not absolute continuous\" is technically incorrect (line 103), although I understand the authors meant to say they don't satisfy the \"absolute continuity condition of [Mohamed 2020]\". The uniform posterior distributions *are* absolutely continuous (w.r.t. Lebesgue measure) since they have densities.  To be fair, the language used in [Mohamed 2020] is also confusing in this regard.", "limitations": "It would be helpful to also discuss the computation/implementation complexity of the multi-sample approach. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is mostly clear and easy to follow; the proposed method is somewhat original (see Theis and Ho [2021]) and conceptually straightforward. Even though the experimental results are somewhat negative, I believe a better understanding between variational inference and compression would be of significant interest  to researchers in both fields. \n\n\nWeaknesses:\n\n1. The authors made several observations regarding different gradient estimators for IWAE, specialized to the VI setup of neural image compression (NIC), but it's unclear what the impact/significance they have.  The first observation (sec 2.1; \"STL is unbiased because in NIC the posterior entropy is constant\") seems trivial, and the second observation (sec 2.2; score function gradient doesn't work because the NIC posterior has bounded support) also seems straightforward after checking the conditions for when the score function gradient apply ([Mohamed 2020]). As for impact, the first observation appears to be a sanity check for the proposed multi-sample bound, and the second observation appears to caution against an approach (score function gradient estimator) which nobody seems to be using in neural compression ...\n\n2. The experimental results are not very informative/satisfying.  It's perhaps unsurprising that training a NIC model with a tighter bound does not yield significantly improved compression performance, unless an accompanying encoding/inference scheme is used to realize such a bound. It would be interesting to see if softmin coding (Theis and Ho [2021]) can fulfill the potential of the proposed multi-sample training scheme, even on a small scale. Similarly, it'd be helpful to provide some insight into why increasing the number of IWAE samples has a limited impact on performance. \n\n3. Also see below about potentially questionable claims.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Zi7t_8odtLC", "reviewer": "Reviewer_5bm9", "summary": "Neural Image Compression (NIC) is typically formulated with a VAE objective for optimization, with the rate-distortion objective corresponding to the ELBO, but with some pecularities: the posterior of the inference network is assumed to be uniformly distributed to simulate quantization noise. With this in mind, the paper carefully studies a) the impact of this on various gradient estimators and b) how to properly adapt multi-sample objectives such as the IWAE target.\nThe proposed MS-NIC objective results in 2-5% BD-rate savings in terms of PSNR for two well known approaches (Balle et al, 2018) and (Cheng et al, 2020).", "questions": "See above.", "limitations": "Yes.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Note: While very familiar with neural image compression, I am not an expert on multi-sample VAEs and gradient estimators.\n\nI found the paper well written. It gives a very detailed analysis on how the NIC objective connects to VAEs, going into the details such as the direct-y trick of the hyperprior, how the unform posterior is discontinuous and how that affects various gradient estimators.\n\nThe resulting MS-NIC objective seems well motivated to me and the empirical results are promising, given that only the training objective has been changed. This means we get better systems with the exactly same architecture.\n\nHowever, this also brings me to point of confusion: as far as I understand, the multi sample objective is training-only, while at inference time we use the standard single-forward-pass encoder and quantization.\nSo do we have any guarantees that the \"tighter ELBO\" that we get from the multi-sample NIC objective  actually improves the inference objective?\n\nThe other concern is how the quantization vs noise mismatch interacts with the whole analysis, as indeed the NIC objective is only a proxy for quantization (while it can be implemented at test time with Universal Quantization).\n\nA third concern is how the proposed objective affects training time. Because another way to improve results is to train longer or with a larger batch size, so it would be interesting to understand the trade-offs there.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LacQHX8kFGh", "reviewer": "Reviewer_8LQV", "summary": "Most work on lossy neural image compression (NIC) us a \"single-sample pathwise estimator\" to estimate the ELBO gradients to optimize the core rate-distortion loss function. This paper observes that multiple-sample methods provide a tighter bound than ELBO and thus may lead to better compression results (a lower rate-distortion loss). The paper explores different formulations (IWAE, MS-NIC-MIX, and MS-NIC-DMS) that incorporate multiple samples, offers an explanation for why IWAE does not work here (\\tilde{z} depends on \\tilde{y} which increases gradient variance compared to models where \\tilde{z} depends on y), and shows empirically that both multi-sample methods (MIX and DMS) boost compression performance in most cases (the exception is an autoregressive entropy model optimized for MS-SSIM).\n\nGlossary:\nELBO = evidence lower bound\nIWAE = importance-weighted autoencoder\nMS = multi-sample\nMIX = combine multi and single-sampling\nDMS = double multi-sampling\nMS-SSIM = multiscale structural similarity\n", "questions": "Presumably, the training time for MS-NIC-DMS will be higher per step compared to the baseline single-sample approach. If so, does multi-sample optimization make sense with a fixed training budget? In other words, if I have a compute (or time) budget for training, should I prefer fewer MS steps or more SS steps?", "limitations": "I don't see any potential negative societal impacts from this work. The authors did discuss limitations (the poor results on MS-SSIM with the (Cheng 2020) model is the primary limitation).", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper looks at a fundamental aspect of neural image compression (tighter bounds than ELBO when uniform noise is used for latents) that, as far as I know, has not be previously discussed. The discussion of multi-sample methods is thorough.\n\n2. The approach is quite general in the sense that it is applicable to most neural image compression methods (models that use VQ are an exception) and the empirical benefits are non-trivial (2.38% - 4.93% rate savings for MS-NIC-DMS when optimizing for PSNR). For reference, a 10% rate gain is considered huge, less than 1% is not very interesting since it's on par with the random variation across training runs, and a new generation of a standard codec typically provides a 30% rate savings).\n\nWeaknesses:\n1. It's not obvious to me why multi-sample methods would perform worse on MS-SSIM and the (Cheng 2020). Additional exploration is warranted (as the authors say in the conclusion).\n\n2. Confidence intervals for the empirical results would strengthen the paper. The authors do address this in the \"Checklist\" section, and I agree that training models can be expensive and \"error bars\" are not commonly seen in NIC papers (though they should be). Perhaps a reasonable compromise would be to estimate the confdience interval in just a single case (e.g. PSNR for (Balle 2018).\n\n3. Either actual code or more guidance on how to implement MS-NIC-DMS would strengthen the paper. I think I can figure this out by reviewing the IWAE and SGVB algorithms.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept