Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a simple task (learning the manifold of all the images of disks) to study some properties of Autoencoders. They show that Autoencoders don't generalize to disks of radius not in the training set and propose several regularization to improve generalisation.

The task proposed in the paper is interesting but the study made is somewhat limited:

- They only studied one choice of Autoencoder architecture, and the results shown depends heavily on the choice of the activation, in particular sigmoid should not suffer from the same problem. 

- It would be interesting to study the generalization in terms of the size of the gap.

- The regularization proposed is quite simple and already known, and other regularization have been proposed (e.g. dropout, ...). A more detailed comparison with all previous regularization scheme would be much needed. 

- The choice of regularization at the end seems quite arbitrary, it works better on this example but it's not clear at all why, and if this choice would work for other tasks.

Also Denoising Autoencoders (Pascal et al.) should probably be mentioned in the previous work section, as they propose a solution to the regularization of Autoencoder.

Overall nothing really new was discovered or proposed, the lack of generalization of those kind of architecture is a well known problem and the regularization proposed was already known. 
review_title: Interesting toy task but without real new insights or contribution to Autoencoders

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper considers a toy problem: the space of images of discs of variable radius - a one dimensional manifold.

An autoencoder based on convolutional layers with ReLU is experimented with, with a 1D embedding.

It is shown that 
1) if the bias is not included, the resulting function is homogeneous (meaning f(ax)=af(x)), and so it fails because the 1D representation should be the radius, and the relationship from radius to image is more complex than a homogeneous function.
- if we include the bias and L2 regularise only the encoder weights, it works better in terms of interpolation for a limited data sample.

The thing is that 1) is trivial (the composition of homogeneous functions is homogeneous... so their proof is overly messy btw). Then, they continue by further analysing (see proposition 2) the solution for this case. Such analysis does not seem to shed much light on anything relevant, given that we know the autoencoder fails in this case due to the trivial proposition 1.

Another point: since the homogeneous function problem will not arise for other non-linearities (such as the sigmoid), the focus on the bias as the culprit seems arbitrary.

Then, the story about interpolation and regularisation is kind of orthogonal, and then is solved by an arbitrary regularisation scheme. The lesson learned from this case is basically the second last paragraph of section 3.2. In other words, it just works.

Since it's a toy problem anyway, the insights seem somewhat trivial.

On the plus side, such a toy problem seems like it might lead somewhere interesting. I'd like to see a similar setup but with a suite of toy problems. e.g. vary the aspect ratio of an oval (rather than a disc), vary the position, intensity, etc etc.
review_title: a bit trivial and lacking in justification/insight as to the regularisation method

final_score: rating: 4: Ok but not good enough - rejection
review_content: 1. The idea is interesting, but the study is not comprehensive yet
2. need to visualize the input data space, with the training data, test data, the 'gaps' in training data [see a recent related paper - Stoecklein et al. Deep Learning for Flow Sculpting: Insights into Efficient Learning using Scientific Simulation Data. Scientific Reports 7, Article number: 46368 (2017).]. 
3. What's the effect of training data size? 
4. How do the intermediate feature maps look like? 
5. Is there an effect of number of layers? Maybe the network architecture is too deep for the simple data characteristics and size of training set. 
6. Other shapes are said to be part of future work, but I am not convinced that serious conclusions can be drawn from this study only? 
7. What about the possible effects of Batch normalization and dropout?  
8. size of 'd' is critical for autoencoders, only one example in appendix does not do justice, also it seems other color channels show up in the results (fig 10), wasn't it binary input?
review_title: This is an interesting study to understand the innerworkings of an autoencoder, however, the study is not quite convncing, yet.

Meta-review:  + interesting approach for a detailed analysis of the limitations of autoencoders in solving a simple toy problem
 - resulting insights somewhat trivial, not really novel, nor practically useful => lacks demonstration of a gain on non-toy task
 - regularization study too limited in scope: lacking theoretical grounding, and more exhaustive comparison of regularization schemes.
Decision: Reject