Image Representation Learning Using Graph Regularized Auto-Encoders
Authors: Yiyi Liao, Yue Wang, Yong Liu
ICLR 2014

Abstract: We consider the problem of image representation for the tasks of unsupervised learning and semi-supervised learning. In those learning tasks, the raw image vectors may not provide enough representation for their intrinsic structures due to their highly dense feature space. To overcome this problem, the raw image vectors should be mapped to a proper representation space which can capture the latent structure of the original data and represent the data explicitly for further learning tasks such as clustering. Inspired by the recent research works on deep neural network and representation learning, in this paper, we introduce the multiple-layer auto-encoder into image representation, we also apply the locally invariant ideal to our image representation with auto-encoders and propose a novel method, called Graph regularized Auto-Encoder (GAE). GAE can provide a compact representation which uncovers the hidden semantics and simultaneously respects the intrinsic geometric structure. Extensive experiments on image clustering show encouraging results of the proposed algorithm in comparison to the state-of-the-art algorithms on real-word cases.


Peer review:
[Review]:
Summary of contributions:
	Proposes to regularize auto encoders so that the encoded dataset has a similar nearest neighbor graph structure to the raw pixels. This method is advocated specifically for images. 

Novelty: moderate (note: the authors seem to believe they are introducing the use of multi-layer auto encoders on images, but they are not)
Quality of results: low - moderate
Quality of presentation: low

Pros:
	Demonstrates improvements in clustering and semi-supervised learning performance
Cons:
	Presentation is confusing and in many cases factually incorrect
	Presentation lacks motivation and reasoning about why the method should work
	Quantitative results are on small and obscure datasets, and improvements are relative to baselines of unclear value

Detailed comments:

Abstract:
	The abstract is ramble and not very focused. This is a conference on representation learning, we don't need you to explain representation learning and deep nets in the abstract. Focus on how you've changed the auto encoder.
	
	'we introduce the multiple- layer auto-encoder into image representation': definitely not true, you even cite papers from 5 years ago that use multi-layer auto encoders on images.

	The abstract should say something about what your new method actually is / does and why you think it is a good idea. I can't tell from the abstract what your method is except that you've changed auto-encoders in some way.

	'Extensive experiments on image clustering show encouraging results of the proposed algorithm in comparison to the state-of-the-art algorithms on real-word cases.'
	Be up-front about what your results are. What does encouraging mean? 

Introduction
	Par 1
	f(H)? H hasn't been introduced yet. Do you mean f(X)?
	What do you mean by well approximate X? If H is meant to be similar to X, what's the point of switching to it? Do you mean it should preserve info about X?

	Par 2
	You cite a purely supervised method (Krivhevsky et al's ImageNet classifier) and then say 'Those methods normally need to use the auto-encoders to pre-train…' Not true. 
	'It has been generally accepted as the consensus that the pre-trained network does provide a better representation for the original data.' Definitely not true! See for example Charle's Cadieu's work presented at ICLR last year.

	Par 3
	Typo: Locally Linear Embedding is LLE, not LEE
	
	Par 4
	What does 'weighted connected' mean?
	The wording of this paragraph is not especially clear, but I take it to mean you want f(x_1) to be near f(x_2) if x_2 is a nearest neighbor of x_1. Why do you think this is a desirable property? It's well known that Euclidean distances in images are not very meaningful. That's the whole reason we want to use representation learning on them.

Section 3 Graph Regularized Auto-Encoder
	Par 1
	This paragraph seems incredibly dismissive of the body of work that develops our understanding of auto encoders as learning manifolds, and how these manifolds relate to classification problems. I would say previous work such as the manifold tangent classifier definitely explores ideas related to the 'geometrical and discriminating structure of the data'

	Par 2
	This paragraph consists of nothing but the letter 'f'

Section 3.1

	Equations 4 and 5: 'sigmoid' should not be in all caps, that makes it looks like the product between variables s, i, g, etc.    (This comment applies throughout the paper, not just these equations)
	
	Equation 7: when you say V is 'the weight matrix' do you mean it is a weighted adjacency matrix describing which examples should be near each other? Usually in auto encoder literature people use 'the weight matrix' to refer to W_H or W_Q. If V is indeed this adjacency matrix you should describe how it is computed and what the weights mean. Even just putting in a forward reference to section 3.3 can help the reader be less confused.

Section 3.2

	You really do not need to spend so much space describing the extremely well-known concept of greedy layer wise pre training

Section 3.3
	OK, so V is the graph encoding matrix.

	3.3.1: Could you please explain the motivation for each of these? i.e., what effect you are hoping their use will have on the learning algorithm?

Section 4
	What is ORL? You should cite the publication that introduced it. Is this ORL? http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html It looks like the name has changed.

	You might want to get the datasets you work on added here: http://rodrigob.github.io/are_we_there_yet/build/
	This will make it easier for reviewers to understand your work in context.

	In general I do not find these experiments very compelling because they are mostly done on small and obscure datasets. It's also not clear to me which of the baselines you ran yourself and which if any are taken from the literature. Baselines that you ran yourself are less compelling because you may not have the same familiarity with pre-existing methods as the inventors of those methods, and you certainly have less incentive to make them perform well.

	At a minimum, I'd like to see some more explanation of why the baselines you're improving upon are impressive. What would be better is to demonstrate good results on datasets that are used more frequently by people in the deep learning community. You are introducing a new kind of auto encoder so you should compare it to pre-existing auto encoders on datasets where auto encoders are frequently used, such as MNIST or Cover Type.

Section 4.1
	I notice you specify the hyper parameters for GAE and GNMF are optimized by grid search, but you have no mention of this for the SAE. Did you also optimize the SAE hyper parameters by grid search?

	You say that the GAE has 2 coefficients to be set by grid search, k and lambda, but it seems like there must be a whole lot of other values to set, such as the dimensionality of H. What did you do about these?

Section 4.2
	Footnote 3: why is a single sample per class 'meaningless'? I agree it's really hard to do well in this case, but why is 1 sample totally worthless and 2 acceptable? If you've added more labels, isn't your work no longer comparable to previous work on the same data set?