Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: In the last para on page 7, it should read Figure 4 instead of 9 right ?

The curves in Fig 4 alone do not seem to back up the claim that CAE does better than JPEG2000---the only noticeable difference appears to be in SSIM at higher bit-rates, and I feel the difference may be too small to be meaningful (especially for values > 0.9). Therefore, I think the paper should back up the claim with more visual examples (like the last two rows of Fig 5). It wouldn't be a bad idea to add error bars to Fig 4 as well.

Moreover, I feel a fair comparison would be to the outputs of neural network trained to reconstruct images from their JPEG2000 encoded versions. You want to do this to show that the network's actually learning a better coding scheme, and not just applying better "denoising" during reconstruction.

Finally, what's the running time of compression and decompression vs JPEG2000 ?
review_title: Vs JPEG2000

final_score: rating: 7: Good paper, accept
review_content: This paper proposes an autoencoder approach to lossy image compression by minimizing the weighted sum of reconstruction error and code length. The architecture consists of a convolutional encoder and a sub-pixel convolutional decoder. Experiments compare PSNR, SSIM, and MS-SSIM performance against JPEG, JPEG-2000, and a recent RNN-based compression approach. A mean opinion score test was also conducted.

Pros:
+ The paper is clear and well-written.
+ The decoder architecture takes advantage of recent advances in convolutional approaches to image super-resolution.
+ The proposed approaches to quantization and rate estimation are sensible and well-justified.

Cons:
- The experimental baselines do not appear to be entirely complete.

The task of using autoencoders to perform compression is important and has a large practical impact. Though directly optimizing the rate-distortion tradeoff is not an entirely novel enterprise, there are enough differences (e.g. the quantization approach and sub-pixel convolutional decoder) to sufficiently distinguish this from earlier work. I am not an image compression expert but the approach and results both seem compelling. The main shortcoming is that the implementation of Toderici et al. 2016b appears to be incomplete, and there is no comparison to Balle et al. 2016. Overall, I feel that the fact that this architecture achieves competitive performance with JPEG-2000 while simultaneously setting the stage for future work that varies the encoder/decoder size and data domain means the community will find this work to be of significant interest.

I have no further specific comments at this time as they were answered sufficiently in the pre-review questions.
review_title: Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This work proposes a new approach for image compression using auto encoders. The results are impressive, besting the state of the art in this field.

Pros:
+ Very clear paper. It should be possible to replicate these results should one be inclined to do so.
+ The results, when compared to other work in this field are very promising. I need to emphasize, and I think the authors should have emphasized this fact as well: this is very new technology and it should not be surprising it's not better than the state of the art in image compression. It's definitely better than other neural network approaches to compression, though.

Cons:
- The training procedure seems clunky. It requires multiple training stages, freezing weights, etc.
- The motivation behind Figure 1 is a bit strange, as it's not clear what it's trying to illustrate, and may confuse readers (it talks about effects on JPEG, but the paper discusses a neural network architecture, not DCT quantization)
review_title: Final Review

final_score: rating: null
review_content: * What is the motivation for using mirror-padding and valid convolutions in the encoder but zero-padded convolutions in the decoder?
* In figure 3B there are large jumps in the loss even for non-incremental training. Do you have any insight into what the cause might be?
* The masked incremental training is somewhat reminiscent of the common practice for VAEs of layering in the KL term. Did you try the analogous approach of multiplying the log Q term of Eq. 2 by a scalar that transitions from 0 to 1 over the course of training?
* In the human evaluation, is there a particular reason for conducting a 1-5 rating experiment as opposed to a more direct experiment such as pairwise comparison or ranking of the images?
review_title: Pre-review questions

final_score: rating: null
review_content: You say "Empirically, we found the identity r(y)=y to work as well as more sophisticated choices" -- would it be possible to explain which approaches did you actually try, and how well did they perform under your architecture?

When talking in 3.2 about the number of updates, do you assume a batch size of 1? If not, it may be worth mentioning what was the batch.

Also in 3.2, you mention "incremental bit enabling" and very briefly say that it works "better". Do you have any numbers we could compare? It's not clear "how much better" does this trick achieve, otherwise.

Regarding the MOS setup:
1. What motivated using a discrete scoring scale with no reference? 
2. Have you considered using triplets?
3. What conditions did the raters have? Was there a strict protocol in place to ensure a constant distance from the monitor? 
4. Was the monitor and lighting conditions the same for all raters?
review_title: Some more questions

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a neural approach to learning an image compression-decompression scheme as an auto-encoder. While the idea is certainly interesting and well-motivated, in practice, it turns out to achieve effectively identical rates to JPEG-2000.

Now, as the authors argue, there is some value to the fact that this scheme was learned automatically rather than by expert design---which means it has benefits beyond the compression of natural images (e.g., it could be used to automatically learning a compression scheme for signals for which we don't have as much domain knowledge). However, I still believe that this makes the paper unsuitable for publication in its current form because of the following reasons---

1. Firstly, the fact that the learned encoder is competitive---and not clearly better---than JPEG 2000 means that the focus of the paper should more be about the aspects in which the encoder is similar to, and the aspects in which it differs, from JPEG 2000. Is it learning similar filters or completely different ones ? For what kinds of textures does it do better and for what kinds does it do worse (the paper could show the best and worst 10 patches at different bit-rates) ?

2. Secondly, I think it's crucial that the paper demonstrate that the benefits come from a better coding scheme (as opposed to just a better decoder), as suggested in my initial pre-review question. How would a decoder trained on JPEG-2000 codes (and perhaps also on encoded random projections) do worse or better ?

3. Finally, I think the fact that it does as well/worse than JPEG-2000 significantly diminishes the case for using a 'deep' auto-encoder. JPEG-2000 essentially uses a wavelet transform, which is a basis that past studies have shown could be recovered using a simple sparse dictionary algorithm like K-SVD. This is why I feel that the method needs to clearly outperform JPEG-2000, or show comparisons to (or atleast discuss) a well-crafted traditional/generative model-based baseline.
review_title: Official Review

Meta-review: This paper optimizes autoencoders for lossy image compression. Minimal adaptation of the loss makes autoencoders competitive with JPEG2000 and computationally efficient, while the generalizability of trainable autoencoders offers the added promise of adaptation to new domains without domain knowledge.
 The paper is very clear and the authors have tried to give additional results to facilitate replication. The results are impressive. While another ICLR submission that is in the same space does outperform JPEG2000, this contributions nevertheless also offers state-of-the-art performance and should be of interest to many ICLR attendees.
Decision: Accept (Poster)