Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: 
Authors propose a transform coding solution by extending the work in Balle 2016. They define an hyperprior for the entropy coder to model the spatial relation between the transformed coefficients.    

The paper is well written, although I had trouble following some parts. The results of the proposal are state-of-the-art, and there is an extremely exhaustive comparison with many methods.

In my opinion the work has a good quality to be presented at the ICLR. However, I think it could be excellent if some parts are improved. Below I detail some parts that I think could be improved.


*** MAIN ISSUES

I have two main concerns about motivation that are related. The first refers to hyperprior motivation. It is not clear why, if GDN was proposed to eliminate statistical dependencies between pixels in the image, the main motivation is that GDN coefficients are not independent. Perhaps this confusion could be resolved by broadening the explanation in Figure 2. My second concern is that it is not clear why it is better to modify the probability distribution for the entropy encoder than to improve the GDN model. I think this is a very interesting issue, although it may be outside the scope of this work. As far as I know, there is no theoretical solution to find the right balance between the complexity of transformation and the entropy encoder. However, it would be interesting to discuss this as it is the main novelty of the work compared to other methods of image compression based on deep learning. 

*** OTHER ISSUES 

INTRODUCTION

-"...because our models are optimized end-to-end, they minimize the total expected code length by learning to balance the amount of side information with the expected improvement of the entropy model." 
I think this point is very interesting, it would be good to see some numbers of how this happens for the results presented, and also during the training procedure. For example, a simple comparison of the number of bits in the signal and side information depending on the compression rate or the number of iterations during model training.  


COMPRESSION WITH VARIATIONAL MODELS

- There is something missing in the sentence: "...such as arithmetic coding () and transmitted..."

- Fig1. To me it is not clear how to read the left hand schemes. Could it be possible to include the distributions specifically? Also it is strange that there is a \tiled{y} in both schemes but with different conditional dependencies. Another thing is that the symbol ψ appears in this figure and is not used in section 2. 

- It would be easier to follow if change the symbols of the functions parameters by something like \theta_a and \theta_s.

- "Distortion is the expected difference between..." Why is the "expected" word used here? 

- "...and substituting additive uniform noise..." is this phrase correct? Are authors is Balle 2016 substituting additive uniform noise?   

- In equation (1), is the first term zero or constant? when talking about equation (7) authors say "Again, the first term is constant,...".

- The sentence "Most previous work assumes..." sounds strange.

- The example in Fig. 2 is extremely important to understand the motivation behind the hyperprior but I think it needs a little more explanation. This example is so important that it may need to be explained at the beginning of the work. Is this a real example, of a model trained without and with normalization? If so specify it please. Why is GDN not able to eliminate these spatial dependencies? Would these dependencies be eliminated if normalization were applied between spatial coefficients? Could you remove dependencies with more layers or different parameters in the GDN?

INTRODUCTION OF A SCALE HYPERPRIOR

- TYPO "...from the center pane of..."

- "...and propose the following extension of the model (figure 3):" there is nothing after the colon. Maybe there is something missing, or maybe it should be a dot instead of a colon. However to me there is a lack of explanation about the model. 

 
RESULTS

- "...,the probability mass functions P_ŷi need to be constructed “on the fly”..."
How computationally costly is this? 

- "...batch normalization or learning rate decay were found to have no beneficial effect (this may be due to the local normalization properties of GDN, which contain global normalization as a special case)."

This is extremely interesting. I see the connection for batch normalization, but not for decay of the learning rate. Please, clarify it. Does this mean that when using GDN instead of regular nonlinearity we no longer need to use batch normalization? Or in other words, do you think that batch normalization is useful only because it is special case of GSN? It would be useful for the community to assess what are the benefits of local normalization versus global normalization.

- "...each of these combinations with 8 different values of λ in order to cover a range of rate–distortion tradeoffs." 

Would it be possible with your methods including \lambda as an input and the model parameters as side information?

- I guess you included the side information when computing the total entropy (or number of bits), was there a different way of compressing the image and the side information?

- Using the same metrics to train and to evaluate is a little bit misleading. Evaluation plots using a different perceptual metric would be helpful. 

-"Since MS-SSIM yields values between 0 (worst) and 1 (best), and most of the compared methods achieve values well above 0.9, we converted the quantity to decibels in order to improve legibility." 
Are differences of MS-SSIM with this conversion significant? Is this transformation necessary, I lose the intuition. Besides, probably is my fault but I have not being able to "unconvert" the dB to MS-SSIM units, for instance 20*log10(1)= 20 but most curves surpass this value.  

- "..., results differ substantially depending on which distortion metric is used in the loss function during training."   
It would be informative to understand how the parameters change depending on the metric employed, or at least get an intuition about which set of parameters adapt more g_a, g_s, h_a and h_s.

- Figs 5, 8 and 9. How are the curves aggregated for different images? Is it the mean for each rate value? Note that depending on how this is done it could be totally misleading.
 	
- It would be nice to include results from other methods (like the BPG and Rippel 2017) to compare with visually.

RELATED WORK

Balle et al. already published a work including a perceptual metric in the end-to-end training procedure, which I think is one of the main contributions of this work. Please include it in related work:

"End-to-end optimization of nonlinear transform codes for perceptual quality." J. Ballé, V. Laparra, and E.P. Simoncelli. PCS: Picture Coding Symposium, (2016) 

DISCUSSION

First paragraphs of discussion section look more like a second section of "related work". 
I think it is more interesting if the authors discuss the relevance of putting effort into modelling hyperprior or the distribution of images (or transformation). Are these things equivalent? Or is there some reason why we can't include hyperprior modeling in the g_a transformation? For me it is not clear why we should model the distribution of outputs as, in principle, the g_a transformation has to enforce (using the training procedure) that the transformed data follow the imposed distribution. Is it because the GDN is not powerful enough to make the outputs independent? or is it because it is beneficial in compression to divide the problem into two parts?   

REFERENCES

- Balle 2016 and Theis 2017 seem to be published in the same conference the same year. Using different years for the references is confusing.

- There is something strange with these references

Ballé, J, V Laparra, and E P Simoncelli (2016). “Density Modeling of Images using a Generalized
Normalization Transformation”. In: Int’l. Conf. on Learning Representations (ICLR2016). URL :
https://arxiv.org/abs/1511.06281.
Ballé, Valero Laparra, and Eero P. Simoncelli (2015). “Density Modeling of Images Using a Gen-
eralized Normalization Transformation”. In: arXiv e-prints. Published as a conference paper at
the 4th International Conference for Learning Representations, San Juan, 2016. arXiv: 1511.
06281.
– (2016). “End-to-end Optimized Image Compression”. In: arXiv e-prints. 5th Int. Conf. for Learn-
ing Representations.


review_title: In my opinion the work has a good quality to be presented at the ICLR. However, I think it could be excellent if some parts are improved.

final_score: rating: 7: Good paper, accept
review_content: Summary:

This paper extends the work of Balle et al. (2016, 2017) on using certain types of variational autoencoders for image compression. After encoding pixels with a convolutional net with GDN nonlinearities, the quantized coefficients are entropy encoded. Where before the coefficients were independently encoded, the coefficients are now jointly modeled using a latent variable model. In particular, the model exploits dependencies in the scale of neighboring coefficients. The additional latent variables are used to efficiently represent these scales. Both the coefficients and the representation of the scales are quantized and encoded in the binary image representation.

Review:

Lossy image compression using neural networks is a rapidly advancing field and of considerable interest to the ICLR community. I like the approach of using a hierarchical entropy model, which may inspire further work in this direction. It is nice to see that the variational approach may be able to outperform the more complicated state-of-the-art approach of Rippel and Bourdev (2017). That said, the evaluation is in terms of MS-SSIM and only the network directly optimized for MS-SSIM outperformed the adversarial approach of R&B. Since the reconstructions generated by a network optimized for MSE tend to look better than those of the MS-SSIM network (Figure 6), I am wondering if the proposed approach is indeed outperforming R&B or just exploiting a weakness of MS-SSIM. It would have been great if the authors included a comparison based on human judgments or at least a side-by-side comparison of reconstructions generated by the two approaches.

It might be interesting to relate the entropy model used here to other work involving scale mixtures, e.g. the field of Gaussian scale mixtures (Lyu & Simoncelli, 2007).

Another interesting comparison might be to other compression approaches where scale mixtures were used and pixels were encoded together with scales (e.g., van den Oord & Schrauwen, 2017).

The authors combine their approach using MS-SSIM as distortion. Is this technically still a VAE? Might be worth discussing.

I did not quite follow the motivation for convolving the prior distributions with a uniform distribution.

The paper is mostly well written and clear. Minor suggestions:

– On page 3 the paper talks about “the true posterior” of a model which hasn’t been defined yet. Although most readers will not stumble here as they will be familiar with VAEs, perhaps mention first that the generative model is defined over both $x$ and $\tilde y$.

– Below Equation 2 it sounds like the authors claim that the entropy of the uniform distribution is zero independent of its width.

– Equation 7 is missing some $\tilde z$.

– The operational diagram in Figure 3 is missing a “|”.
review_title: Hierarchical entropy model a good idea; value of MS-SSIM improvement not clear

final_score: rating: 7: Good paper, accept
review_content: The paper is a step forward for image deep compression, at least when departing from the (Balle et al., 2017) scheme.
The proposed hyperpriors are especially useful for medium to high bpp and optimized for L2/ PSNR evaluation.

I find the description of the maths too laconic and hard to follow. For example, what’s the U(.|.) operator in (5)?

What’s the motivation of using GDN as non linearity instead of e.g. ReLU?

I am not getting the need of MSSSIM (dB).  How exactly was it defined/computed?

Importance of training data? The proposed models are trained on 1million images while others like (Theis et al, 2017) and [Ref1,Ref2] use smaller datasets for training.

I am missing a discussion about Runtime / complexity vs. other approaches?

Why MSSSIM is a relevant measure? The Fig. 6 seem to show better visual results for L2 loss (PSNR) than when optimized for MSSSIM, at least in my opinion.

What's the reason to use 4:4:4 for BPG and 4:2:0 for JPEG?

What is the relation between hyperprior and importance maps / content-weights [Ref1] ?

What about reproducibility of the results? Will be the codes/models made publicly available?

Relevant literature:
[Ref1] Learning Convolutional Networks for Content-weighted Image Compression (https://arxiv.org/abs/1703.10553)
[Ref2] Soft-to-Hard Vector Quantization for End-to-End Learned Compression of Images and Neural Networks  (https://arxiv.org/abs/1704.00648)

review_title: Variational image compression with a scale hyperprior

Meta-review: Thank you for submitting you paper to ICLR. The reviewers and authors have engaged well and the revision has improved the paper. The reviewers are all in agreement that the paper substantially expands the prior work in this area,  e.g. by Balle et al. (2016, 2017), and is therefore suitable for publication. Although I understand that the authors have not optimised their compression method for runtime yet, a comment about this prospect in the main text would be a sensible addition.
Decision: Accept (Poster)