Image Representation Learning Using Graph Regularized Auto-Encoders
Authors: Yiyi Liao, Yue Wang, Yong Liu
ICLR 2014

Abstract: We consider the problem of image representation for the tasks of unsupervised learning and semi-supervised learning. In those learning tasks, the raw image vectors may not provide enough representation for their intrinsic structures due to their highly dense feature space. To overcome this problem, the raw image vectors should be mapped to a proper representation space which can capture the latent structure of the original data and represent the data explicitly for further learning tasks such as clustering. Inspired by the recent research works on deep neural network and representation learning, in this paper, we introduce the multiple-layer auto-encoder into image representation, we also apply the locally invariant ideal to our image representation with auto-encoders and propose a novel method, called Graph regularized Auto-Encoder (GAE). GAE can provide a compact representation which uncovers the hidden semantics and simultaneously respects the intrinsic geometric structure. Extensive experiments on image clustering show encouraging results of the proposed algorithm in comparison to the state-of-the-art algorithms on real-word cases.


Peer review:
[Review]:
Thanks for the comments. We have revised the paper and here are some responses below.
-------------------------------------------
“I found the whole paper hard work to follow.” 
We have revised the paper in many places and submit a new version.
-------------------------------------------
“Results tables don't give column headings. It wasn't clear what the variable was - the dimension? ”
It would be clearer to break out the 'error measure' into a separate column. 
Each column is the class we used in the clustering or semi-supervised tasks. It is also equivalent to the dimension of hidden layer or representation. The ‘error measure’ is given in the first column, each row indicates a method in a error measure with respect to different number of classes we use.
-------------------------------------------
“The two metrics used weren't very clear to me – 
the mapping from unsupervised k-means clusters to class labels ”
It follows the evaluation method in the GNMF paper which is on the dimension reduction too.
http://www.cad.zju.edu.cn/home/dengcai/Publication/Journal/TPAMI-GNMF.pdf
-------------------------------------------
“How does it compare to a simple 1-NN hinge-loss technique similar to those used by Weston e.g. for training joint embeddings? http://www.thespermwhale.com/jaseweston/papers/wsabie-ijcai.pdf”
We have read the paper. The technique in this paper gives an idea to preserve the local property. However, it takes much time to implement this algorithm. We will try to use it in our work in the future. In this paper, we mainly want to show whether the local invariant can be applied in deep architecture.
-------------------------------------------
“What parameters eta, lambda $k$ (for the graph construction) were chosen? How did this $k$ affect the results? ”
The parameters are chosen using a grid based search on the validation set. We give the performance with the best parameter configuration.
-------------------------------------------
“How expensive is the proposed technique compared to the alternative?”
The time complexity is similar to sparse auto-encoders, as it is a change of constraint. The main problem is space complexity of graph matrix. However, we use the sparse matrix to save the information. Since the graph connection is sparse, there are many zero entries.