Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper describes a generalization of autoencoders that are trained to reconstruct a close neighbor of its input, instead of merely the input itself. Experiments on 3 datasets show that this yields better representations in terms of post hoc classification with a linear classifier or clustering, compared to a regular autoencoder.

As the authors recognize, there is a long history of research on variants of autoencoders. Unfortunately this paper compares with none of them. While the authors suggest that, since these variations can be combined with the proposed neighbor reconstruction variant, it's not necessary to compare with these other variations, I disagree. It could very well be that this neighbor trick makes other methods worse for instance. 

At the very least, I would expect a comparison with denoising autoencoders, since they are similar if one thinks of the use of neighbors as a structured form of noise added to the input. It could very well be in fact that simply adding noise to the input is sufficient to force the autoencoder to learn a valuable representation, and that the neighbor reconstruction approach is simply an overly complicated approach of achieving the same results. This is an open question right now that I'd expect this paper to answer.

Finally, I think results would be more impressive and likely to have impact if the authors used datasets that are more commonly used for representation learning, so that a direct performance comparison can be made with previously published results. CIFAR 10 and SVHN would be good alternatives.

Overall, I'm afraid I must recommend that this paper be rejected.

review_title: Lack of comparison with other autoencoders

final_score: rating: 6: Marginally above acceptance threshold
review_content: A representation learning framework from unsupervised data, based not on auto-encoding (x in, x out), but on neighbor-encoding (x in, N(x) out, where N(.) denotes the neighbor(s) of x) is introduced. 

The underlying idea is interesting, as such, each and every degree of freedom do not synthesize itself similar to the auto-encoder setting, but rather synthesize a neighbor, or k-neighbors. The authors argue that this form of unsupervised learning is more powerful compared to the standard auto-encoder setting, and some preliminary experimental proof is also provided. 

However, I would argue that this is not a completely abstract - unsupervised representation learning setting since defining what is "a neighbor" and what is "not a neighbor" requires quite a bit of domain knowledge. As we all know, the euclidian distance, or any other comparable norm, suffers from the "Curse of Dimensionality" as the #-of-Dimensions increase. 

For instance, in section 4.3, the 40-dimensional feature vector space is used to define neighbors in. It would be great how the neighborhood topology in that space looks like.

All in all, I do like the idea as a concept but I am wary about its applicability to real data where defining a good neighborhood metric might be a major challenge of its own. 
review_title: Nice Idea but what about "Curse of Dimensionality"?

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents a variant of auto-encoder that relaxes the decoder targets to be neighbors of a data point. Different from original auto-encoder, where data point x and the decoder output \hat{x} are forced to be close, the neighbor-encoder encourage the decoder output to be similar to the neighbors of the input data point. By considering the neighbor information, the decoder targets would have smaller intra-class distances, thus larger inter-class distances, which helps to learn better separated latent representation of data in terms of data clusters. The authors conduct experiments on several real but relative small-scale data sets, and demonstrate the improvements of learned latent representations by using neighbors as targets. 

The method of neighbor prediction is a simple and small modification of the original auto-encoder, but seems to provide a way to augment the targets such that intra-class distance of decoder targets can be tightened. Improvements in the conducted experiments seem significant compared to the most basic auto-encoder.

Major issues:

There are some unaddressed theoretical questions. The optimal solution to predict the set of neighbor points in mean-squared metric is to predict the average of those points, which is not well justified as the averaged image can easily fall off the data manifold. This may lead to a more blurry reconstruction when k increases, despite the intra-class targets are tight. It can also in turn harm the latent representation when euclidean neighbors are not actually similar (e.g. images in cifar10/imagenet that are not as simple as 10 digits). This seems to be a defect of the neighbor-encoder method and is not discussed in the paper.

The data sets used in the experiments  are relatively small and simple, larger-scale experiments should be conducted. The fluctuations in Figure 9 and 10 suggest the significant variances in the results. Also, more complicated data/images can decrease the actual similarities of euclidean neighbors, thus affecting the results.

The baselines are weak. Only the most basic auto-encoder is compared, no additional variants or other data augmentation techniques are compared. It is possible other variants improve the basic auto-encoder in similar ways. 

Some results are not very well explained. It seems the performance increases monotonically as the number of neighbors increases (Figure 5, 9, 10). Will this continue or when will the performance decrease? I would expect it to decrease as the far away neighbors will be dissimilar. The authors can either attach the nearest neighbors figures or their statistics, and provide explanations on when and why the performance decrease is expected.

Some notations are confusing and need to be improved. For example, X and Y are actually the same set of images, the separation is a bit confusing; y_i \in y in last paragraph of page 4 is incorrect, should use something like y_i in N(y).
review_title: Review: neighbor-encoder -> neighbor encoder

Meta-review: The paper proposes a form of autoencoder that learns to predict the neighbors of a given input vector rather than the input itself.  The idea is nice but there are some reviewer concerns about insufficient evaluation and the effect of the curse of dimensionality.  The revised paper does address some questions and includes additional helpful experiments with different types of autoencoders.  However, the work is still a bit preliminary.  The area of auto-encoder variants, and corresponding experiments on CIFAR-10 and the like, is crowded.  In order to convince the reader that a new approach makes a real contribution, it should have very thorough experiments.  Suggestions:  try to improve the CIFAR-10 numbers (they need not be state-of-the-art but should be more credible), adding more data sets (especially high-dimensional ones), and analyzing the effects of factors that are likely to be important (e.g. dimensionality, choice of distance function for neighbor search).
Decision: Reject