Learning High-level Image Representation for Image Retrieval via Multi-Task DNN using Clickthrough Data
Authors: Wei-Ying Ma, Tiejun Zhao, Kuiyuan Yang, Wei Yu, Yalong Bai
ICLR 2014

Abstract: Image retrieval refers to finding relevant images from an image database for a query, which is considered difficult for the gap between low-level representation of images and high-level representation of queries. Recently further developed Deep Neural Network sheds light on automatically learning high-level image representation from raw pixels. In this paper, we proposed a multi-task DNN for image retrieval, which contains two parts, i.e., query-sharing layers for image representation computation and query-specific layers for relevance estimation. The weights of multi-task DNN are learned on clickthrough data by Ring Training. Experimental results on both simulated and real dataset show the effectiveness of the proposed method.


Peer review:
[Review]:
This paper proposes a multi-task deep neural network approach to solve the task of image retrieval. A dataset of clickthrough  data is used to train the network.

The main contribution of this paper is the proposed multi-task DNN method for image retrieval and the ring training. This multi-task approach consists of adding weights specific to each query (basically a supplemental fully-connected layer) before binary classification. The authors compare their approach to binary DNNs and multi-class DNN. However, I would have like to see a multi-label DNN as well for comparison.

Quality:
I found the quality of the language passable. There were several syntactic, grammatical errors and typos which made the paper hard to understand. The paper could have used more polishing. In general, I have found the paper hard to read, and the authors did not get their points through clearly.

General comments:
What loss function was used? I don't understand why the authors did not give it explicitly.

In the ring training pseudo code, it is not mentioned how j is defined, is there an iteration on all values, or is it chosen randomly? It makes a lot of difference how j is treated if the classes are unbalanced. This should be explained clearly.

In section 4, the architecture of the network is presented, but nothing is mentioned about which weights were query-specific in the multi-task case for the CIFAR-10 experiment.

section 4.2: 'In general, binary DNN performs consistently worse for the severe overfitting problem.' This sentence does not make a lot of sense, and does not reflect the results very well.

I don't really see the point of using dataset 2. The difference in results between d1 and d2 are small, and do not, in my opinion, demonstrate what the authors claim it shows. I think it only clutters the paper.

The authors use the term significantly without showing confidence intervals. Perhaps another term should be used.

Section 5.1: 'multi-class DNN is infeasible for such large number of queries'. This would require more explanation. Why does the multi-task approach scale better than the multi-class approach? Since the parameters are not shared, doesn't the multi-task DNN require even more parameters than the multi-class? If this is not the case, please explain more clearly your point.

In section 5.3, why not use the activation of the classification layer as an affinity measure instead of training SVMs