Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper investigates convolutional autoencoder (CAE) bottleneck. The research problem is important, as CAE is widespread adopted. It cam be interesting to shed insight into how the bottleneck works. 

In particular, two observations are made: (1) By measuring the performance of the latent codes in downstream transfer learning tasks, the authors show that  increased height/width of the bottleneck drastically improves generalization; The number of channels in the bottleneck is secondary in importance.  (2) CAEs do not learn to copy their input, even when the bottleneck has the same number of neurons as there are pixels in the input.

It would make this submission more convincing if the authors show the some important applications of their findings. For example, 

A. How does (1) above help architecture search when designing CAEs in a new application? 
B. How does (2) above help style transfer if ``"CAEs do not learn to copy their input" 

Concerns: If (2) is true, Could the authors explain that almost perfect reconstruction results are shown in [*] ?

[*] Generating Diverse High-Fidelity Images with VQ-VAE-2
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The authors evaluate convolutional autoencoders (CAE) by varying the size (width & height) and depth of the bottleneck layer on three datasets and compare test and training performance. They furthermore evaluate the quality of the bottleneck activations for linear classification. The authors also investigate the belief that a bottleneck layer of size equal to the input image will copy the image. 

I am not an expert in the field of (C)AEs. As such, I cannot approriately judge the relevance of the questions which are answered here. In the following, I will therefore make the assumption that those questions are relevant.
If so, I (weakly) recommend accepting the paper. 

While it does not propose any novel algorithms, it does ask a clear question and provides a compelling experimental answer, which (assuming the question is relevant), should be interesting for the community. 
On the other hand, further experiments to provide initial insights into the further questions raised by the authors would improve the 'novelty' aspect of the paper. 

Minor comment: 
I believe the paper "Reconciling modern machine learning practice and the bias-variance trade-off" by Belkin et. al is relevant. 

Edit:
Thank you for your response.

I wasn't referring to any specific experiments, but the questions you are raising in the paper, for example in the conclusion section. 

Regarding Belkin et al: Sorry for the misunderstanding, I didn't mean to imply that it's highly relevant, but I thought it might be a good addition to the related work section as it also relates to the generalization of neural networks.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Summary:
This paper studies some of the properties of fully convolutional autoencoders (CAE) as a function of the shape and total size of the bottleneck. They train and test CAEs with bottlenecks consisting of different ratios of spatial resolution versus number of channels, as well as different total number of neurons. The authors investigate which type of change in the bottleneck is most influential on training behavior, generalization to test set, and linear separability for classification/regression. Their first main finding is that the spatial resolution of the bottleneck is a stronger influencer of generalization to the test set than the number of channels and the total number of neurons in the bottleneck. The second main finding is that even when the total number of neurons in the bottleneck is equal to the data input size, the neural network does not appear to simply learn to copy the input image into the bottleneck. 


Decision: 
Weak reject: It is always refreshing to see papers that address/challenge/investigate common assumptions in deep learning. However, I find the experimental findings and discussion of borderline quality for a full conference paper. It might be more suitable as a good workshop paper.

Supporting arguments for decision:
It is unclear to me why the authors have chosen to only take a subset of the CelebA and STL-10 datasets for training and testing. It seems like dataset size is also an important factor that increases the complexity for training a model, and it certainly affects how a model can generalize. When auto-encoders are studied in other literature it is uncommon practice to restrict the dataset sizes like this, so this makes me question the applicability of this paper’s results to the literature. 

It seems that the experimental validation is based on one run per CAE model with one single seed. This is on the low side of things, especially when quite extensive claims are made. An example of such a claim is on page 6 when discussing a sudden jump in training and test scores for 8x8x48 model trained on the Pokemon dataset. Because the same behavior appeared when the authors repeated the experiment with the same seed, the authors conclude “This outlier suggests, that the loss landscape might not always be as smooth towards the end of training, as some publications (Goodfellow et al., 2014) claim and that ‘cliffs” (i. e., sudden changes in loss) can occur even late in training.” Making this claim based on something that occurs with one single model for a single seed is not convincing and overstating this finding. 
Another example is on page 7 under bullet point 4, where the authors discuss the obtained evidence against copying behaviour when the bottleneck is of the same size as the input. The authors state “We believe this finding to have far-reaching consequences as it directly contradicts the popular hypothesis about copying CAEs.” The paper definitely shows some empirical evidence that supports the claim that copying does not occur, but these findings are all done with a single seed and by considering small subsets of datasets (celebA and stl-10). In my opinion, it is therefore too much to state that the current findings have far reaching consequences. It has potential, but I wouldn’t go much further than that. 

On page 7 in the second to last paragraph the influence of dataset complexity is discussed. The authors state “the loss curves and reconstruction samples do not appear to reflect the notion of dataset difficulty we defined in Section 2.3” and “This lack of correspondence implies that the intuitive and neural network definitions of difficulty do not align. Nevertheless, a more detailed study is required to answer this question definitively as curriculum learning research that suggests the opposite (Bengio et al., 2009) also exists.” It is unclear to me what the authors expected to find here. Moreover, the absence of major differences across the chosen datasets does not immediately make me doubt or question results from curriculum learning. My skepticism is again enhanced by the fact that the authors have taken a subset of the data for the more complex celebA and STL-10 datasets. Dataset size seems like a crucial part of dataset complexity.

An interesting smaller finding is that linear separability of the latent codes for classification is better on the test set for the pokemon dataset, even though the training and test reconstruction losses showed signs of overfitting. The authors hypothesize that overfitting might occur more in the decoder than in the encoder.

Additional feedback to improve the paper (not part of decision assessment)
- Section 2.3.1: what was the original resolution of the pokemon dataset?
- Section 3.1, c_j is used as the compression level, but in eq 3 and 4 c^i is also used to indicate the number of channels. - For clarity please use n_ci in eq 2 and 3 or choose a different all together for the compression level.
- Please increase the font size of the plots in figures 1, 3 and 4.

review_title: Official Blind Review #3

Meta-review: The paper investigates the effect of convolutional information bottlenecks to generalization. The paper concludes that the width and height of the bottleneck can greatly influence generalization, whereas the number of channels has smaller effect. The paper also shows evidence against a common belief that CAEs with sufficiently large bottleneck will learn an identity map. 

During the rebuttal period, there was a long discussion mainly about the sufficiency of the experimental setup and the trustworthiness of the claims made in the paper. A paper that empirically investigates an exiting method or belief should include extensive experiments of high quality in to enable general conclusions. I’m thus recommending rejection, but encourage the authors to improve the experiments and resubmitting.
Decision: Reject