Title: Lossy Image Compression with Compressive Autoencoders
Authors: Lucas Theis
Wenzhe Shi
Andrew Cunningham
Ferenc Huszár
Venue: ICLR 2017
Keywords: Computer vision
Deep learning
Applications

Abstract: We propose a new approach to the problem of optimizing autoencoders for lossy image compression. New media formats, changing hardware technology, as well as diverse requirements and content types create a need for compression algorithms which are more flexible than existing codecs. Autoencoders have the potential to address this need, but are difficult to optimize directly due to the inherent non-differentiabilty of the compression loss. We here show that minimal changes to the loss are sufficient to train deep autoencoders competitive with JPEG 2000 and outperforming recently proposed approaches based on RNNs. Our network is furthermore computationally efficient thanks to a sub-pixel architecture, which makes it suitable for high-resolution images. This is in contrast to previous work on autoencoders for compression using coarser approximations, shallower architectures, computationally expensive methods, or focusing on small images.

TL;DR: A simple approach to train autoencoders to compress images as well or better than JPEG 2000.

Review discussion:
* What is the motivation for using mirror-padding and valid convolutions in the encoder but zero-padded convolutions in the decoder?
* In figure 3B there are large jumps in the loss even for non-incremental training. Do you have any insight into what the cause might be?
* The masked incremental training is somewhat reminiscent of the common practice for VAEs of layering in the KL term. Did you try the analogous approach of multiplying the log Q term of Eq. 2 by a scalar that transitions from 0 to 1 over the course of training?
* In the human evaluation, is there a particular reason for conducting a 1-5 rating experiment as opposed to a more direct experiment such as pairwise comparison or ranking of the images?
Dear Reviewer 3, thank you for taking the time to review our paper.
The motivation for mirror-padding in the pixel space is that this leads to an image with more natural statistics than zero-padding.
Similarly, zero-padding in the space of coefficients is natural, since the coefficients are sparsely distributed. In our paper we used zero-padded convolutions in each layer of the decoder.
However, a better motivated choice would have been to (analogously to the encoder) zero-pad the coefficients once and then use “valid convolutions”. We find that both options work equally well, but might update the paper with this slightly more elegant although slightly slower architecture.
We considered annealing but didn’t experiment enough with it to be able to say anything about it’s effects on training performance.
We agree that there would be better ways to evaluate perceptual quality. The motivation for using mean opinion scores with a 1-5 rating is that this is (unfortunately) the industry standard. We hope to replace it in future work after gaining experience with other psychophysical paradigms.
In the last para on page 7, it should read Figure 4 instead of 9 right ?
The curves in Fig 4 alone do not seem to back up the claim that CAE does better than JPEG2000---the only noticeable difference appears to be in SSIM at higher bit-rates, and I feel the difference may be too small to be meaningful (especially for values > 0.9).
Therefore, I think the paper should back up the claim with more visual examples (like the last two rows of Fig 5). It wouldn't be a bad idea to add error bars to Fig 4 as well.
Moreover, I feel a fair comparison would be to the outputs of neural network trained to reconstruct images from their JPEG2000 encoded versions. You want to do this to show that the network's actually learning a better coding scheme, and not just applying better "denoising" during reconstruction.
Finally, what's the running time of compression and decompression vs JPEG2000 ?
Dear Reviewer 1, thanks for taking the time to review our paper.
Yes, it should be Figure 4 instead of Figure 9 on page 7, thanks for catching that.
Regarding your first comment, first note that we do not claim that CAE is always better than JPEG 2000. We claim that we are able to “train deep autoencoders competitive with JPEG 2000” (abstract), and that “we achieve performance similar to or better than JPEG 2000 when evaluated for perceptual quality” (introduction). We also point out that “in terms of PSNR, our method performs […] slightly worse at low and medium bit rates” (Section 3.3).
Secondly, note that the ultimate test for image compression has to be perceptual quality, and PSNR and SSIM should only be viewed as an approximation to it.
In terms of perceptual quality as measured by a MOS test, we found a statistically significant albeit small improvement over JPEG 2000.
That said, further improvements could easily be gained by using adaptive entropy encoding (as done by JPEG 2000) instead of simple range coding as we do now.
Regarding your second point, we agree that it would be interesting to evaluate the effect of artefact removal on the rate-distortion curve and we will try to include such a comparison in a later version of our manuscript.
However, note that the achievement described in this paper is not a new state-of-the-art encoding scheme (better algorithms than JPEG 2000 exist), but a simple solution to the unsolved problem of learning a useful encoder at all. This is a non-trivial contribution in itself, since it is not clear how to best deal with the non-differentiability in compression. It enables the use of deep learning where before a lot of expertise in signal processing was necessary to come up with a new encoder.
We will add encoding and decoding times to the paper.
However, fair comparisons of any two algorithms are tricky, since the running time depends so heavily on the implementation. Some things to consider are that highly optimized and specialized (but not necessarily widely used) implementations of JPEG 2000 exist, while we used a general purpose framework designed for training but not for deployment.
Finally, while our main contribution is conceptual, we are confident that our work will lead to state-of-the-art compression algorithms. We are currently exploring architectures and training better tuned for the low bit rate regime as well as introducing more sophisticated entropy encoding schemes (as used by other compression algorithms such as JPEG 2000). Speedups may be possible at little performance loss, for example through training separable filters.
You say "Empirically, we found the identity r(y)=y to work as well as more sophisticated choices" -- would it be possible to explain which approaches did you actually try, and how well did they perform under your architecture?
When talking in 3.2 about the number of updates, do you assume a batch size of 1? If not, it may be worth mentioning what was the batch.
Also in 3.2, you mention "incremental bit enabling" and very briefly say that it works "better". Do you have any numbers we could compare? It's not clear "how much better" does this trick achieve, otherwise.
Regarding the MOS setup:
1. What motivated using a discrete scoring scale with no reference?
2. Have you considered using triplets?
3. What conditions did the raters have? Was there a strict protocol in place to ensure a constant distance from the monitor?
4. Was the monitor and lighting conditions the same for all raters?
Dear reviewer 2, thank you for taking the time to review our paper.
Instead of the identity, we tried using a particular smooth approximation to the rounding function, and increasing the steepness/closeness of the approximation over the course of the training. The performance was the same as the performance achieved using the identity.
Regarding batch size, not that the first sentence of Section 3.2 says that “All models were trained using […] batches of 32 images”.
Regarding incremental training, note that we already included a comparison of incremental training and non-incremental training Figure 3B.
Regarding MOS scores, the main motivation for using mean opinion scores with a rating from 1-5 is that this is a widely used industry standard. We agree that there should be better ways to do perceptual evaluations (e.g. 2AFC tasks, ranking), and we hope to explore these and replace MOS testing in the future. We will explain our motivation for using MOS tests in a revised manuscript.
The raters were instructed not to zoom into the pictures, but for natural viewing conditions were otherwise free to view the images for however long or from whatever distance they liked.
The paper proposes a neural approach to learning an image compression-decompression scheme as an auto-encoder.
While the idea is certainly interesting and well-motivated, in practice, it turns out to achieve effectively identical rates to JPEG-2000.
Now, as the authors argue, there is some value to the fact that this scheme was learned automatically rather than by expert design---which means it has benefits beyond the compression of natural images (e.g., it could be used to automatically learning a compression scheme for signals for which we don't have as much domain knowledge).
However, I still believe that this makes the paper unsuitable for publication in its current form because of the following reasons---
1.
Firstly, the fact that the learned encoder is competitive---and not clearly better---than JPEG 2000 means that the focus of the paper should more be about the aspects in which the encoder is similar to, and the aspects in which it differs, from JPEG 2000. Is it learning similar filters or completely different ones ? For what kinds of textures does it do better and for what kinds does it do worse (the paper could show the best and worst 10 patches at different bit-rates) ?
2.
Secondly, I think it's crucial that the paper demonstrate that the benefits come from a better coding scheme (as opposed to just a better decoder), as suggested in my initial pre-review question. How would a decoder trained on JPEG-2000 codes (and perhaps also on encoded random projections) do worse or better ?
3.
Finally, I think the fact that it does as well/worse than JPEG-2000 significantly diminishes the case for using a 'deep' auto-encoder. JPEG-2000 essentially uses a wavelet transform, which is a basis that past studies have shown could be recovered using a simple sparse dictionary algorithm like K-SVD. This is why I feel that the method needs to clearly outperform JPEG-2000, or show comparisons to (or atleast discuss) a well-crafted traditional/generative model-based baseline.
Dear Reviewer 1, thanks again for reviewing our paper. We would be grateful if you could reconsider your score after taking into account the following points addressing some of your concerns:
1.
While the encoder is non-linear, we will include estimates of the filters of a linear approximation to the encoder so that they can be compared with JPEG 2000. (Preliminary noisy estimates of some of the filters: https://ndownloader.figshare.com/files/7260815 – note that the estimates are noisy, not the filters themselves).
3. In a sense we are giving JPEG 2000 an unfair advantage, since its representations are encoded using adaptive entropy coding, and CAE’s are encoded using range coding with simple i.i.d. assumption. Additional gains can be achieved by adding adaptive entropy coding, but unfortunately no easy-to-use implementation exists (we are working on it).
Ballé et al. submitted closely related work to ICLR (https://openreview.net/forum?id=rJxdQ3jeg). The main difference lies in their treatment of quantization. Together with adaptive entropy encoding, they achieve clear gains over JPEG 2000. We find that when everything else is the same, our treatment of quantization works better (https://ndownloader.figshare.com/files/7260857). Together, these results show that it is possible to significantly outperform JPEG 2000 using our approach.
We will add this comparison to the paper. Since Ballé’s approach is equivalent to a particular VAE, this should also qualify as a comparison to a well-crafted generative model. Ballé’s submission also shows that this topic is of strong interest to the ICLR community.
A critical difference of this work with that of Balle et al.'s is that the latter is actually trained with respect to a rate-distortion loss that is meant to optimize performance for adaptive coding. Given that their work clearly outperforms JPEG2000 while this one doesn't, it would appear that being able to handle, and optimize for, adaptive coding is critical in achieving effective compression.
What is more, doing this is non-trivial---the main contribution of the Balle et al. paper is the way they manage to back-propagate with respect to "rate" (or an approximation thereof).
I also don't see how Balle's approach can be considered a proxy for a well-crafted generative model, since it's discriminatively trained.
Moreover, clearly the Balle approach outperforms the current one, so I'm not sure whether a comparison would help make the case for the approach used by this paper.
As I stated in my original review, I agree that this paper considers an interesting problem, but I just feel that it doesn't make enough progress for publication at ICLR.
The reviewer is mistaken. Just as in our work, Balle et al. optimize for an independent code (bottom at page 3; Equation 5). During optimization, only the marginals are modeled and adaptive entropy coding, which exploits dependencies, is not taken into account. We can apply adaptive entropy encoding posthoc, just as in their work.
The reviewer describes Balle et al.’s main contribution as their ability to back-propagate [through] the rate. I am confused by this statement since the main contribution of our paper is exactly that.
Furthermore, Balle’s approach was published in an earlier paper, so this is in fact not even a contribution of their current submission. Our approach is novel and works better (again, this is for two deep models using the same training procedure, only swapping the way rates are back-propagated: https://ndownloader.figshare.com/files/7260857).
Both our paper and Balle’s paper discuss under which cases their approach is equivalent to a variational autoencoder. What does the reviewer mean by “discriminatively trained”? For certain choices, optimizing the lower bound of a variational autoencoder is exactly optimizing a certain rate-distortion trade-off as done by Balle. We can expect any deviation from those choices to only make the generative model worse, since it wouldn’t be optimizing a rate-distortion trade-off.
- By 'backpropagate through the rate', I mean the actual approximation used as the training objective. As I understand, Balle's work doesn't just differ from this work in the treatment of quantization, but also in the way  the distribution (and therefore the entropy) is computed. This work uses uni-variate GMM's to approximate the probability distribution of coefficients, while Balle's work uses a finely-quantized piecewise linear approximation for p(x).
Therefore, the actual objective that their paper optimizes is different from this one (although both are trying to approximate the same rate-distortion loss).
I'm not sure what the plots in the figure in the previous message corresponds to. Is the "uniform noise" plot essentially the same method as CAE, but with the explicit forward-pass quantization replaced by the addition of uniform noise ? If so, that still would be different from what Balle's work is doing.
I guess most of this speculation can be laid to rest if authors simply use the same regime as Balle's, i.e., apply a variable entropy coder post-hoc, and show that they get noticeably better results than JPEG-2000. (Beating Balle's work isn't necessary, since it's a concurrent submission).
- The 'generative' comparison was a separate suggestion in my original review for a comparison if the paper is *not* able to show a clear benefit over JPEG2000. I realize that something that is not 'discriminatively trained' with respect to rate-distortion would give worse results.
But that is the point---showing that at least the proposed approach offers an advantage over a naive generative model (as opposed to an expertly hand-crafted one like JPEG-2000).
Here is the argument again: the paper, as it stands now, doesn't provide a system for natural image compression that would be preferable to use over the traditional standard (JPEG2000). If it is able to show better performance, then that's that.
But if it's not, the paper can still claim a contribution in that the proposed compression method was automatically learned from data while JPEG2000 is hand-crafted, and can therefore be used in arbitrary domains.
But to make this claim, it should be able to at least do better than other simple "generative" models (doing better than Balle's work, which is a concurrent submission, and also an end-to-end compression strategy is not necessary).
Again, if the authors feel they can do better than JPEG-2000 using a variable coder, they should go for it.
Yes, we use Gaussian scale mixtures while Balle et al. use a piece-wise linear density. This is in no way means that Balle et al. optimized the encoder for adaptive encoding, as claimed earlier by the reviewer.
I would appreciate it if the reviewer could take another look at Balle et al.'s paper, as their approach is in fact exactly to replace quantization with additive noise (Equation 5, $\delta y$ is i.i.d. uniform noise).
Thus, our plot is a comparison of Balle's approach (uniform noise) and the CAE.
Instead of a comparison with generative models, we provide a much more relevant comparison with Toderici et al. (2016b), for which code and results on the Kodak dataset are available. We went to great lengths and unlike Balle et al. or Toderici et al. even performed a psychophysical evaluation.
I took another look at the Balle et al. paper. In eq 5, quantization is modeled through the addition of uniform noise in the quantization interval, as you say.
But the probability $p(\tilde{y})$ is also modeled differently from this paper. Since the objective in eq 5 in Balle's work includes **both** the quantization approximation and density approximation, I don't see how just swapping out one of them (quantization) makes the plot a comparison to Balle's approach (in their concurrent ICLR submission). Balle's approach isn't just "quantization = uniform noise", it is a combination of that and their density approximation.
You are right in that their objective isn't better optimized for adaptive encoding per-se, but it may be better optimized for entropy encoding in general---i.e., their approximation of rate (combination of approximations for quantization and density) may lead to better codes than the one in this paper. I'd argue that that is likely, since in fact Balle et al. report a 20% improvement in bit-rate over JPEG-2000 (it's possible, but unlikely, that _all_ of that is from using a variable coder).
Again, since Balle's work is a concurrent submission, this paper doesn't need to outperform it.
But it should at least outperform the 16 year old hand-crafted shallow JPEG-2000 standard.
When I wrote my initial review, I didn't notice that you weren't using an adaptive coder (like both JPEG200 and Balle's work do). It seems to me that that should be a simple experiment to run (perhaps use CABAC like Balle et al did). This can only improve CAC's performance beyond what's currently in the paper.
Dear Lucas & AnonReviewer1,
Since our ICLR submission (https://openreview.net/forum?id=rJxdQ3jeg) has been discussed in this thread, we felt we should comment on a few of the claims that have been made.  Note that none of us are reviewers, official or unofficial, for this paper.  In our view, both papers offer interesting contributions, and we think the ICLR meeting would benefit from having both presented!
[Theis, 23dec]: Ballé et al. submitted closely related work to ICLR (https://openreview.net/forum?id=rJxdQ3jeg). The main difference lies in their treatment of quantization. Together with adaptive entropy encoding, they achieve clear gains over JPEG 2000. We find that when everything else is the same, our treatment of quantization works better (https://ndownloader.figshare.com/files/7260857). Together, these results show that it is possible to significantly outperform JPEG 2000 using our approach.
Our implementation differs in a number of ways from that of the current paper: we use a 3-stage generalized divisive normalization transform to map the image data into a space suitable for (uniform) quantization, an objective function based on continuous relaxation of the quantizer to an additive additive noise process, and an adaptive entropy coder. This coder is adaptive to the statistics of individual images, but only weakly spatially adaptive within images (unlike JPEG 2000, which uses a much more sophisticated form of spatially adaptive coder). In response to one of the reviews of our paper, we computed the entropy of the encoded test images from the Kodak set, assuming the probability model learned from the training set.
As we expected, these entropies, which should closely approximate the performance of a *non-adaptive* entropy coder, are only slightly worse (on average) than the bit rates obtained with our adaptive entropy coder (see http://www. cns. nyu. edu/~balle/kodak-entropy2-psnr. pdf). We conclude that most of the rate-distortion performance of our system arises from the 3-stage GDN transform.
Despite this, we want to caution against over-interpreting averaged/summary rate-distortion results, and emphasize that compression results are best evaluated on individual images.  In our case, the average JPEG2000 R-D curve was computed over images encoded at the same *rate*, whereas the average R-D curves for our coder were computed over images encoded at the same *lambda* (R-D slope).
Although this makes no difference for results computed on individual images, it does affect the summary plot.  To illustrate this, we re-computed the average performance of JPEG2000 over images encoded at the same *distortion* level (the implementation allows specification of either distortion or the rate), and found that it gave a different curve, albeit still significantly worse than that of our coder (this plot is included in the appendix of the revised paper).  This discrepancy serves to illustrate the difficulty of providing a “correct” method of summarizing coder performance.  We suggest looking at the results for individual images (for our coder, these were made available at http://www.cns.nyu.edu/~lcv/iclr2017).
[Theis, 25dec]: I would appreciate it if the reviewer could take another look at Balle et al.'s paper, as their approach is in fact exactly to replace quantization with additive noise (Equation 5, $\delta y$ is i.i.d. uniform noise).
Thus, our plot is a comparison of Balle's approach (uniform noise) and the CAE.
In our paper, the substitution of additive uniform noise in place of quantization is done *only* for optimization (i.e., to create a continuously differentiable objective function).  All compression results (the rate-distortion curves and decoded images, both in the paper and online) are computed using actual quantization, followed by entropy coding.  This was explicitly stated in the paper, but might easily have been missed (we’ve emphasized it more in the revision).  It appears that the rate-distortion plot linked in the response paragraph above, and also the images presented in figure 1 of this paper, may have been computed using additive uniform noise, and if so, are not a correct characterization of our coding method.
In the end, what matters is rate-distortion performance given actual quantization, not the rate-distortion performance under the proxy objective.
Hi Eero,
I appreciate your feedback!
Regarding “In our paper, the substitution of additive uniform noise in place of quantization is done *only* for optimization”. This is true for the figure I posted: For evaluation, actual quantization was used and no noise added. I am sorry if I didn’t make this clear, it didn’t occur to me that there was an alternative way of doing it.
Regarding differences between your implementation and ours: Note that I mentioned two of the three differences you described. I didn’t go into the neural network architecture, since I was mainly concerned with differences affecting qualitative performance, and you state in your paper that similar compression can be achieved using an architecture similar to ours, albeit less efficiently [1]. In preliminary experiments we found 5-10% reduction in bit rate using very simple context-free adaptive entropy encoding, so I assumed your use of CABAC would play a bigger role than the updated version of your paper shows.
Where I am not sure about your implementation is whether you compute a separate encoder/decoder pair for each lambda. If that is the case, a further difference would be the number of transformations used.
Regarding summary plots, I agree that it’s easy to get these wrong, although I am less pessimistic about their value if the same averaging is applied to all methods. If all curves are averaged at the same rate and also plotted against rate, the curves are easy to interpret as each point along the curve simply corresponds to the average distortion at that rate. To be able to measure distortion at arbitrary rates, we interpolated the curves for each image first. Interpolating and averaging has its own intricacies, which can be taken care of by measuring the actual rate-distortion at enough points for each method (we used 20 points for JPEG and JPEG 2000).
[1] “Preliminary results indicate that qualitatively similar results are achievable with other activation functions we tested, but that rectified linear units generally require a substantially larger number of model parameters to achieve the same rate–distortion performance as the GDN/IGDN nonlinearities”
Hi Lucas,
to answer your question: yes, we optimize a separate set of transforms for each lambda and always pick one of them ahead of encoding, whereas you appear to optimize three different sets of transforms and then test and select one of them during runtime, which is a form of signal adaptiveness (similar to our use of adaptive entropy coding).
Re figure 10: I'm still unsure how to interpret the figure. I assume that you trained a set of transforms for each method, and then generated the curves in the same way you did before for your method.
When you trained the models, you certainly substituted our method of relaxing the distortion term.
But did you also use our method of approximating the rate (i.e., additive uniform noise *without* quantization; piecewise linear densities)? If you substitute only the distortion term, you're not comparing to our method, but to some hybrid method, which might be worse than both of our proposals.
We also didn't mention that the optimization method is somewhat different (mostly due to your "incremental" scheme). What if our method just requires slower training, etc.? In general, due to all these differences, it seems to me it's too early to call out a "winner", and I'm not quite following your claim that your method is better. (I do think both papers have some unique aspects to contribute to the field and would be happy if the reviewers decide to accept both of them.)
With respect to the rate-distortion plots: We found that averaging over identical rates versus identical distortions can distort the appearance of the RD-plot in a way that it overemphasizes or underemphasizes high bit rates vs. low bit rates (see appendix of our paper).
Thus, even if you take care that you generate the plot in exactly the same way for all methods, it might mislead you to think that some method is particularly well suited for the high- or low-bitrate regime, etc. Have you taken that into account?
Also, note that starting with the very first version of our paper, we documented the way we generated the plot in the appendix, but some other references haven't done that.
So there's a chance people can come to the wrong conclusion comparing plots across papers. We decided to enable a comparison based on individual images and stay away from averaged plots for now, until there is an agreed-upon solution.
Finally, regarding that last point: Why don't you show images with somewhat lower bitrates (and more images in general)? I find that it is very hard to see any differences at all in the images you show in the paper.
Best,
Johannes.
Hi Johannes,
> But did you also use our method of approximating the rate (i.e., additive uniform noise *without* quantization; piecewise linear densities)?
Yes, in this figure I was only concerned with the different ways of dealing with the quantization during training, and I used uniform noise without quantization as in your paper. I did not use piecewise linear densities, as I consider that a separate design decision.
> In general, due to all these differences, it seems to me it's too early to call out a "winner", and I'm not quite following your claim that your method is better.
I only meant to claim that redefining derivatives works better than uniform noise when everything else is kept the same as in our paper (training procedure, network architecture, etc), which is of course the only thing the figure shows. It may well be that adding uniform noise works better in other contexts, e.g., when the architecture or the entropy model are different (GSMs vs piecewise linear). Sorry if I did not make that clear somewhere. Your overall algorithm clearly performs better, and it will be interesting to figure out the deciding differences.
Regarding RD plots, I am not doubting that different ways of visualizing the same results can influence the perception of these results (this seems true even for the simplest results, e.g. classification accuracy vs classification error), and I agree that having this discussion is important.
But it does not mean that the summary plots are categorically invalid or uninformative, only that readers should be made aware of exactly what they are looking at. Only providing results for individual images has its own obvious downsides, namely being harder to digest by not being concise. I will post results for individual images later (or provide them via email for anyone who's interested).
> Why don't you show images with somewhat lower bitrates (and more images in general)? I find that it is very hard to see any differences at all in the images you show in the paper.
We were focusing on quantitative analyses and realized too late that we need to bring down the bit rates much more to get into the regime of highly visible artefacts (which arguably is not that relevant from a production point of view, but certainly nicer to show in a paper).
That said, the images do look visibly different, as confirmed by our MOS test. Note that the differences may be more or less visible depending on the PDF viewer and screen you use. I did not include more images in the paper because the PDF was already approaching 20 MB, but I am happy to provide a full set of compressed images on request.
Thanks for your feedback!
This paper proposes an autoencoder approach to lossy image compression by minimizing the weighted sum of reconstruction error and code length. The architecture consists of a convolutional encoder and a sub-pixel convolutional decoder. Experiments compare PSNR, SSIM, and MS-SSIM performance against JPEG, JPEG-2000, and a recent RNN-based compression approach. A mean opinion score test was also conducted.
Pros:
+ The paper is clear and well-written.
+ The decoder architecture takes advantage of recent advances in convolutional approaches to image super-resolution.
+ The proposed approaches to quantization and rate estimation are sensible and well-justified.
Cons:
- The experimental baselines do not appear to be entirely complete.
The task of using autoencoders to perform compression is important and has a large practical impact.
Though directly optimizing the rate-distortion tradeoff is not an entirely novel enterprise, there are enough differences (e.g. the quantization approach and sub-pixel convolutional decoder) to sufficiently distinguish this from earlier work. I am not an image compression expert but the approach and results both seem compelling. The main shortcoming is that the implementation of Toderici et al. 2016b appears to be incomplete, and there is no comparison to Balle et al. 2016.
Overall, I feel that the fact that this architecture achieves competitive performance with JPEG-2000 while simultaneously setting the stage for future work that varies the encoder/decoder size and data domain means the community will find this work to be of significant interest.
I have no further specific comments at this time as they were answered sufficiently in the pre-review questions.
This work proposes a new approach for image compression using auto encoders. The results are impressive, besting the state of the art in this field.
Pros:
+ Very clear paper. It should be possible to replicate these results should one be inclined to do so.
+ The results, when compared to other work in this field are very promising. I need to emphasize, and I think the authors should have emphasized this fact as well: this is very new technology and it should not be surprising it's not better than the state of the art in image compression. It's definitely better than other neural network approaches to compression, though.
Cons:
- The training procedure seems clunky. It requires multiple training stages, freezing weights, etc.
- The motivation behind Figure 1 is a bit strange, as it's not clear what it's trying to illustrate, and may confuse readers (it talks about effects on JPEG, but the paper discusses a neural network architecture, not DCT quantization)
Dear reviewers, we made the following changes to our paper:
– added direct comparison with VAE/quantization approximation of Balle et al. (Figure 10)
– added another control to Figure 3 (incremental training vs fixed small learning rate)
– added a motivation and reference for MOS tests
– added more detail to appendix to make reimplementation easier
– improved caption of Figure 3A
Minor:
– added number of scales used in Gaussian scale mixtures
– fixed reference to Figure 4 (which pointed to Figure 9 before)
Toderici et al. kindly provided us with their results which include entropy encoding*. We will rerun MOS experiments and intend to update quantitative comparison figures before the meeting.
* https://drive.google.com/file/d/0B-n7IsUzdeWOb0dNSm42aEp5aE0/view

Meta-review:
role: PC
data: comment: This paper optimizes autoencoders for lossy image compression. Minimal adaptation of the loss makes autoencoders competitive with JPEG2000 and computationally efficient, while the generalizability of trainable autoencoders offers the added promise of adaptation to new domains without domain knowledge.
 The paper is very clear and the authors have tried to give additional results to facilitate replication. The results are impressive. While another ICLR submission that is in the same space does outperform JPEG2000, this contributions nevertheless also offers state-of-the-art performance and should be of interest to many ICLR attendees.

Decision: Accept (Poster)