Image Representation Learning Using Graph Regularized Auto-Encoders
Authors: Yiyi Liao, Yue Wang, Yong Liu
ICLR 2014

Abstract: We consider the problem of image representation for the tasks of unsupervised learning and semi-supervised learning. In those learning tasks, the raw image vectors may not provide enough representation for their intrinsic structures due to their highly dense feature space. To overcome this problem, the raw image vectors should be mapped to a proper representation space which can capture the latent structure of the original data and represent the data explicitly for further learning tasks such as clustering. Inspired by the recent research works on deep neural network and representation learning, in this paper, we introduce the multiple-layer auto-encoder into image representation, we also apply the locally invariant ideal to our image representation with auto-encoders and propose a novel method, called Graph regularized Auto-Encoder (GAE). GAE can provide a compact representation which uncovers the hidden semantics and simultaneously respects the intrinsic geometric structure. Extensive experiments on image clustering show encouraging results of the proposed algorithm in comparison to the state-of-the-art algorithms on real-word cases.


Peer review:
[Review]:
A new auto-encoder method is proposed. Features are learned by minimizing the sum of the square reconstruction error and a penalty on feature dissimilarity weighted according to a variety of criteria. The basic idea is to weight more strongly the features that are in the neighborhood of the training sample or based on the class labels.
A multi-layer version of the autoencoder is proposed by following the layer-wise training protocol.
The method is tested using several metric on a few small datasets.

Strengths
The problem is relevant and interesting.
The method is technically sound.

Weaknesses
The paper lacks clarity. It does not read well and the language is often vague (what does it mean “represent well” or “positive impact” or “hurt numerical optimization”?).
The paper is also rather incremental. The idea is similar to [6] but also related to methods for dimensionality reduction like deep parametric t-sne (see R. Min, L.J.P. van der Maaten, Z. Yuan, A. Bonner, and Z. Zhang. Deep Supervised t-Distributed Embedding. In Proceedings of the International Conference on Machine Learning (ICML), pages 791-798, 2010 ) and methods like H. Mobahi, R. Collobert, J. Weston. Deep Learning from Temporal Coherence in Video. ICML 2009.
In this light, it would be valuable to add a discussion of the advantages of this method versus [6], for instance. The major difference is that the reconstruction error replaces the “pull apart” term in the loss function. What’s the advantage of having an explicit decoder? Doesn’t it introduce even more parameters in the model?
Finally, the empirical validation could be more convincing if the authors used larger datasets where many other authors already benchmarked (e.g., cifar, mnist, timit, svhn, to mention a few).