Learning High-level Image Representation for Image Retrieval via Multi-Task DNN using Clickthrough Data
Authors: Wei-Ying Ma, Tiejun Zhao, Kuiyuan Yang, Wei Yu, Yalong Bai
ICLR 2014

Abstract: Image retrieval refers to finding relevant images from an image database for a query, which is considered difficult for the gap between low-level representation of images and high-level representation of queries. Recently further developed Deep Neural Network sheds light on automatically learning high-level image representation from raw pixels. In this paper, we proposed a multi-task DNN for image retrieval, which contains two parts, i.e., query-sharing layers for image representation computation and query-specific layers for relevance estimation. The weights of multi-task DNN are learned on clickthrough data by Ring Training. Experimental results on both simulated and real dataset show the effectiveness of the proposed method.


Peer review:
[Review]:
Pros: 
•	The use of multi-task DNNs for clickthrough data makes sense and provides good results


Cons: 
•	The idea of multi-task DNNs has been explored before in the literature and is not new. For example, in speech in multi-lingual speech processing this has been explored before. 
o	This paper does joint training of the shared and task-specific components: K. Vesley et al, “The Language-independent bottleneck features,” in Proc. ICASSP 2012. 
o	This paper also does joint training: Z. Tuske et al, “Investigation on Cross-and Multilingual MLP Features under Matched and Mismatched Acoustical Conditions,” in Proc. ICASSP 2013.
o	This paper does separate training of shared and task-specific components: Samuel Thomas, Sriram Ganapathy and Hynek Hermansky, Multilingual MLP Features For Low-resource LVCSR Systems, ICASSP, Kyoto, Japan, March 2012
•	Details of and Ring training are not clearly described. Also, relating this to prior art and justifying why this is the best approach is missing.
•	The paper could be written much better. There are many spelling/grammar mistakes in the paper 

Here are my comments per section:
•	Section 1, page 1: expand on SIFT, HOG and LBP
•	Section 1, page 1: Your statement on CNNs being good doesn’t fit with the rest of the sentence
•	Section 1, page 2: exiting → existing
•	Section 2: There needs to be a discussion on Multi-task DNNs relation to prior work and why your idea is novel. For example, see the papers listed above for speech processing.
•	Section 2, page 3: What is the loss function that you use. 
•	Section 3, page 3: I found the discussion of Ring Training very confusing. You should describe the algorithm 1 in more detail in the text. Also, ring training is not the only way to train Multi-task DNNs. Why did you use this approach as opposed to other ideas in the literature, and why is it potentially better?
•	Section 4, page 3 CIFRA→ CIFAR
•	Section 5.2, page 6: Why did you only add dropout to the first fully connect layer? Did you tune this optimally? Sometimes it helps to add dropout to multiple layers. You should state if it’s tuned properly or not.
•	Section 5.3 page 6: Provide a reference for DCG