Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to train neural compression VAEs with multi-sample IWAE-style objectives, and analyzes the effect of the choice of variational distribution (constant width uniform posterior; the choice between q(y|x) v.s. q(y|z)) on various gradient estimators. They theoretically showed that the uniform posterior in neural compression automatically implements a form of the STL estimator [Roeder et al., 2017], and that naively applying the score function gradient estimator would give incorrect results. They propose two IWAE-style objectives that are potentially tighter than the standard NELBO objective, and show empirically that training on them yield small improvement in R-D performance (5% rate savings). 
strengths_and_weaknesses: Strengths:

The paper is mostly clear and easy to follow; the proposed method is somewhat original (see Theis and Ho [2021]) and conceptually straightforward. Even though the experimental results are somewhat negative, I believe a better understanding between variational inference and compression would be of significant interest  to researchers in both fields. 


Weaknesses:

1. The authors made several observations regarding different gradient estimators for IWAE, specialized to the VI setup of neural image compression (NIC), but it's unclear what the impact/significance they have.  The first observation (sec 2.1; "STL is unbiased because in NIC the posterior entropy is constant") seems trivial, and the second observation (sec 2.2; score function gradient doesn't work because the NIC posterior has bounded support) also seems straightforward after checking the conditions for when the score function gradient apply ([Mohamed 2020]). As for impact, the first observation appears to be a sanity check for the proposed multi-sample bound, and the second observation appears to caution against an approach (score function gradient estimator) which nobody seems to be using in neural compression ...

2. The experimental results are not very informative/satisfying.  It's perhaps unsurprising that training a NIC model with a tighter bound does not yield significantly improved compression performance, unless an accompanying encoding/inference scheme is used to realize such a bound. It would be interesting to see if softmin coding (Theis and Ho [2021]) can fulfill the potential of the proposed multi-sample training scheme, even on a small scale. Similarly, it'd be helpful to provide some insight into why increasing the number of IWAE samples has a limited impact on performance. 

3. Also see below about potentially questionable claims.
questions: There are a few claims in this paper that I'm not unsure about. 

1. I'm not following the discussions around line 153, regarding why the direct-y trick poses an issue for IWAE. Here we simply have a factorized posterior q(z, y|x) = q(z|x) q(y|x), so why can't we just use k samples of (z, y) from this joint posterior to form an IWAE estimator?

2. The authors' statements about IWAE estimator converging to log p(x) (e.g., line 187-189) can be misleading, as they may not hold for the neural compression VAEs. Again this has to do with the posterior having bounded support, whereas the prior and likelihood models having full support. 

3. Is there an obvious explanation for why the mult-sample IWAE [Burda et al., 2016] baseline is so much worse in Table 5? I could not find it. 

3. This is a nitpick: the statement  "q(z˜|y˜) and q(y˜|x˜) are not absolute continuous" is technically incorrect (line 103), although I understand the authors meant to say they don't satisfy the "absolute continuity condition of [Mohamed 2020]". The uniform posterior distributions *are* absolutely continuous (w.r.t. Lebesgue measure) since they have densities.  To be fair, the language used in [Mohamed 2020] is also confusing in this regard.
limitations: It would be helpful to also discuss the computation/implementation complexity of the multi-sample approach.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Most work on lossy neural image compression (NIC) us a "single-sample pathwise estimator" to estimate the ELBO gradients to optimize the core rate-distortion loss function. This paper observes that multiple-sample methods provide a tighter bound than ELBO and thus may lead to better compression results (a lower rate-distortion loss). The paper explores different formulations (IWAE, MS-NIC-MIX, and MS-NIC-DMS) that incorporate multiple samples, offers an explanation for why IWAE does not work here (\tilde{z} depends on \tilde{y} which increases gradient variance compared to models where \tilde{z} depends on y), and shows empirically that both multi-sample methods (MIX and DMS) boost compression performance in most cases (the exception is an autoregressive entropy model optimized for MS-SSIM).

Glossary:
ELBO = evidence lower bound
IWAE = importance-weighted autoencoder
MS = multi-sample
MIX = combine multi and single-sampling
DMS = double multi-sampling
MS-SSIM = multiscale structural similarity

strengths_and_weaknesses: Strengths:
1. This paper looks at a fundamental aspect of neural image compression (tighter bounds than ELBO when uniform noise is used for latents) that, as far as I know, has not be previously discussed. The discussion of multi-sample methods is thorough.

2. The approach is quite general in the sense that it is applicable to most neural image compression methods (models that use VQ are an exception) and the empirical benefits are non-trivial (2.38% - 4.93% rate savings for MS-NIC-DMS when optimizing for PSNR). For reference, a 10% rate gain is considered huge, less than 1% is not very interesting since it's on par with the random variation across training runs, and a new generation of a standard codec typically provides a 30% rate savings).

Weaknesses:
1. It's not obvious to me why multi-sample methods would perform worse on MS-SSIM and the (Cheng 2020). Additional exploration is warranted (as the authors say in the conclusion).

2. Confidence intervals for the empirical results would strengthen the paper. The authors do address this in the "Checklist" section, and I agree that training models can be expensive and "error bars" are not commonly seen in NIC papers (though they should be). Perhaps a reasonable compromise would be to estimate the confdience interval in just a single case (e.g. PSNR for (Balle 2018).

3. Either actual code or more guidance on how to implement MS-NIC-DMS would strengthen the paper. I think I can figure this out by reviewing the IWAE and SGVB algorithms.


questions: Presumably, the training time for MS-NIC-DMS will be higher per step compared to the baseline single-sample approach. If so, does multi-sample optimization make sense with a fixed training budget? In other words, if I have a compute (or time) budget for training, should I prefer fewer MS steps or more SS steps?
limitations: I don't see any potential negative societal impacts from this work. The authors did discuss limitations (the poor results on MS-SSIM with the (Cheng 2020) model is the primary limitation).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Neural Image Compression (NIC) is typically formulated with a VAE objective for optimization, with the rate-distortion objective corresponding to the ELBO, but with some pecularities: the posterior of the inference network is assumed to be uniformly distributed to simulate quantization noise. With this in mind, the paper carefully studies a) the impact of this on various gradient estimators and b) how to properly adapt multi-sample objectives such as the IWAE target.
The proposed MS-NIC objective results in 2-5% BD-rate savings in terms of PSNR for two well known approaches (Balle et al, 2018) and (Cheng et al, 2020).
strengths_and_weaknesses: Note: While very familiar with neural image compression, I am not an expert on multi-sample VAEs and gradient estimators.

I found the paper well written. It gives a very detailed analysis on how the NIC objective connects to VAEs, going into the details such as the direct-y trick of the hyperprior, how the unform posterior is discontinuous and how that affects various gradient estimators.

The resulting MS-NIC objective seems well motivated to me and the empirical results are promising, given that only the training objective has been changed. This means we get better systems with the exactly same architecture.

However, this also brings me to point of confusion: as far as I understand, the multi sample objective is training-only, while at inference time we use the standard single-forward-pass encoder and quantization.
So do we have any guarantees that the "tighter ELBO" that we get from the multi-sample NIC objective  actually improves the inference objective?

The other concern is how the quantization vs noise mismatch interacts with the whole analysis, as indeed the NIC objective is only a proxy for quantization (while it can be implemented at test time with Universal Quantization).

A third concern is how the proposed objective affects training time. Because another way to improve results is to train longer or with a larger batch size, so it would be interesting to understand the trade-offs there.
questions: See above.
limitations: Yes.
review_title: null

Meta-review: metareview: This paper studies the problem of neural image compression (NIC). Standard methods for NIC use a "single-sample pathwise estimator" to estimate the ELBO gradients to optimize the rate-distortion loss function. This paper improves the estimation by using multiple samples, leading to better compression results. Experimental results show that multi-sample methods improve compression performance in many cases.

The reviewers' comments are appropriately addressed, and all reviewers appreciate the contribution of this paper on neural image compression, so I recommend accept. 
confidence: Less certain
Decision: Accept