Learning High-level Image Representation for Image Retrieval via Multi-Task DNN using Clickthrough Data
Authors: Wei-Ying Ma, Tiejun Zhao, Kuiyuan Yang, Wei Yu, Yalong Bai
ICLR 2014

Abstract: Image retrieval refers to finding relevant images from an image database for a query, which is considered difficult for the gap between low-level representation of images and high-level representation of queries. Recently further developed Deep Neural Network sheds light on automatically learning high-level image representation from raw pixels. In this paper, we proposed a multi-task DNN for image retrieval, which contains two parts, i.e., query-sharing layers for image representation computation and query-specific layers for relevance estimation. The weights of multi-task DNN are learned on clickthrough data by Ring Training. Experimental results on both simulated and real dataset show the effectiveness of the proposed method.


Peer review:
[Review]:
The paper presents an architecture to learn simultaneously multiple image rankers, all of which sharing low-level layers of a deep neural network, while keeping their last layer separate for each ranking task. Since the distribution of images per query is far from uniform, this helps 'poor' queries learning good image representations from 'rich' queries. The query specific model is a 2-way softmax (good or bad image for that query). This is not exactly what I would call a ranker, but rather a classifier. In fact, I didn't see how this architecture differs from a DNN trained with logistic regression (so that each query has one output unit, which should be 1 if the image corresponds to the query, and 0 otherwise); it seems that a softmax with 2 units is similar to a single sigmoid unit, no?
The paper shows two series of experiments, one on CIFAR-10, and one on MSR-Bing Image Retrieval challenge dataset, which is a real image ranking problem.
Results show that the proposed approach works better than the two compared approaches (the Binary DNN consists of one DNN per query, which cannot work when the data is heavy tail; the multi-class DNN, on the other hand, seems very similar to the proposed approach, and I did not understand why this one would not scale to millions of queries, while the proposed approach would). Finally, apparently, an SVM is trained after the features are fixed. I did not understand this part neither. Why is the SVM needed anyway?
The paper is not very well written, with a poor level of english, and several details missing, both in describing the algorithm and in the experimental section.