Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: There's a certain degree of complexity introduced by the choice of putting an expression for rate/entropy in the training objective. If I understand correctly, you still train different encoders-decoders for a set of different weighted combinations of rate and distortion. 

I wonder then, if it wouldn't have been easier to minimize distortion for a set of different rates. Fixing the rate could've been achieved by restricting y to be in [0,1] (for e.g., by using a sigmoid), and then adding different levels of random noise (like you already do) to simulate different number of quantization levels. 

Presumably, for each choice of number of quantization levels, the encoder-decoder pair would learn to minimize distortion, and would achieve a rate no worse than that corresponding to a uniform distribution over the quantization levels.
review_title: Optimizing different tradeoff vs distortion for different rates

final_score: rating: null
review_content: 1) I think a relevant paper to cite is one by Van der Oord and Schrauwen:
http://jmlr.org/papers/volume15/vandenoord14a/vandenoord14a.pdf

2) It would be great to get a visualization of some of the filters the network learns - can they be compared to other known transforms like the steerable pyramid etc.?

3) Given the learned representation is capable of modeling long range dependencies and subtle image structure, can it be used to other image restoration tasks such as denoising?

review_title: Some minor questions

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This is the most convincing paper on image compression with deep neural networks that I have read so far. The paper is very well written, the use of the rate-distortion theory in the objective fits smoothly in the framework. The paper is compared to a reasonable baseline (JPEG2000, as opposed to previous papers only considering JPEG). I would expect this paper to have a very good impact. 

Yes, please include results on Lena/Barbara/Baboon (sorry, not Gibbons), along with state-of-the-art references with more classical methods such as the one I mentioned in my questions. I think it is important to clearly state how NN compare to best previous methods. From the submitted version, I still don't know how both categories of methods are positioned. 
review_title: Very good paper

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper extends an approach to rate-distortion optimization to deep encoders and decoders, and from a simple entropy encoding scheme to adaptive entropy coding. In addition, the paper discusses the approach’s relationship to variational autoencoders.

Given that the approach to rate-distortion optimization has already been published, the novelty of this submission is arguably not very high (correct me if I missed a new trick). In some ways, this paper even represents a step backward, since earlier work optimized for a perceptual metric where here MSE is used. However, the results are a visible improvement over JPEG 2000, and I don’t know of any other learned encoding which has been shown to achieve this level of performance. The paper is very well written.

Equation 10 appears to be wrong and I believe the partition function should depend on g_s(y; theta). This would mean that the approach is not equivalent to a VAE for non-Euclidean metrics.

What was the reason for optimizing MSE rather than a perceptual metric as in previous work? Given the author’s backgrounds, it is surprising that even the evaluation was only performed in terms of PSNR.

What is the contribution of adaptive entropy coding versus the effect of deeper encoders and decoders? This seems like an important piece of information, so it would be interesting to see the performance without adaptation as in the previous paper. More detail on the adaptive coder and its effects should be provided, and I will be happy to give a higher score when the authors do.
review_title: Great progress performance-wise but missing details

final_score: rating: 7: Good paper, accept
review_content: Two things I'd like to see.

1) Specifics about the JPEG and JPEG2000 implementations used, and how they were configured.  One major weakness I see in many papers is they do not include specific encoders and configuration used in comparisons.  Without knowing this, it's hard to know if the comparison was done with a suitably strong JPEG implementation that was properly configured, for example.

2) The comparison to JPEG2000 is unfortunately not that interesting, since that codec does not have widespread usage and likely never will.  A better comparison would be with WebP performance.  Or, even better, both.

Very nice results.  Is a software implementation of this available to play with?
review_title: Very interesting results

final_score: rating: null
review_content: 1) In my opinion, the way the CABAC is fed with quantization values lacks of details. Could you please elaborate a bit?

2) Have you measured achievable rate-distortion compromises on Lena, Barbara, Gibbons, for which the reader may have more comparison points with the most recent compression techniques than the Kodak dataset? 

3) Similarly, for the sake of completness, could you please include a comparison with methods developed after JPEG2000, like the oriented Wavelet Transform (http://vivien.chappelier.free.fr/owavelets/). 

review_title: Comparison and details

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This is a nice paper that demonstrates an end-to-end trained image compression and decompression system, which achieves better bit-rate vs quality trade-offs than established image compression algorithms (like JPEG-2000). In addition to showing the efficacy of 'deep learning' for a new application, a key contribution of the paper is the introduction of a differentiable version of "rate" function, which the authors show can be used for effective training with different rate-distortion trade-offs. I expect this will have impact beyond the compression application itself---for other tasks that might benefit from differentiable approximations to similar functions.

The authors provided a thoughtful response to my pre-review question. I would still argue that to minimize distortion under a fixed range and quantization, a sufficiently complex network would learn automatically produce  codes within a fixed range with the highest-possible entropy (i.e., it would meet the upper bound). But the second argument is convincing---doing so forces a specific "form" on how the compressor output is used, which to match the effective compression of the current system, would require a more complex network that is able to carry out the computations currently being done by a separate variable rate encoder used to store q.

review_title: Official review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This nicely written paper presents an end-to-end learning method for image compression. By optimizing for rate-distortion performance and a clever relaxation the method is able to learn an efficient image compression method by optimizing over a database of natural images.

As the method is interesting, results are interesting and analysis is quite thorough it's easy for me to recommend acceptance.
review_title: A good paper with an interesting premise, some novel methods and good results

Meta-review: This is one of the two top papers in my stack and I recommend it for oral presentation. The reviewers were particularly careful and knowledgable of the topic.
Decision: Accept (Oral)