Deep Convolutional Ranking for Multilabel Image Annotation
Authors: Sergey Ioffe, Alexander Toshev, Yangqing Jia, Thomas Leung, Yunchao Gong
ICLR 2014

Abstract: Multilabel image annotation is one of the most important challenges in computer vision with many real-world applications. While existing work usually use conventional visual features for multilabel annotation, the recent deep convolutional feature shows potentials to significantly boost performance. In this work, we propose to leverage the advantage of such features and analyze key components that lead to better performances. Specifically, we show that a significant performance gain could be obtained by combining convolutional architectures with an approximate top-$k$ ranking objective function, as such objectives naturally fit the multilabel tagging problem. Our experiments on the publicly available NUS-WIDE dataset outperforms the conventional visual features by about $10%$, obtaining the best reported performance in the literature.


Peer review:
[Review]:
*Summary*
This paper proposes to use convolutional networks for image annotation. Great care is given to the selection of an appropriate loss function as well as the comparison with reasonable baselines over the NUS/Flickr dataset. The paper reads well, gives enough context and references to related work. It
reports improvement with respect to the state of the art. In my opinion, this is a good paper, with the only drawback that the evaluation is conducted over a single dataset, with a vocabulary of only 80
 tags, which is small compared to realistic application.

 *Detailed review*
 I would like to clarify only two points regarding (i) small tag vocabulary in NUS, (ii) relationship with imageNET classification. 

 (i) the vocabulary of NUS/Flickr is only 80 different tags. This is very  small compare to web annotation or even personal photo gallery annotation. In particular, the fact that your network has 80 outputs make the evaluation of the output score of every tag for every forward/backward step very
 inexpensive (compared to evaluating the rest of the network). This is very different from the initial conditions in which the WARP loss was introduced. Loss functions which does not rely on sampling can perfectly be used and might be better. I notably think at  T. Joachims, A Support Vector Method
 for Multivariate Performance Measures, Proceedings of the International Conference on Machine Learning (ICML), 2005 or Ranking with ordered weighted pairwise classification from Usunier et al 2009. More fundamentally, I feel that focussing on 80 tags hides most of the interesting challenges in real tasks: reasonable 10k vocabularies implies greater perplexity and therefore require greater performance for the CNN. They also suggest giving greater importance on tag coocurences and language modeling to understand unlikely predictions like ocean and lake tag in the same image from your example.

 (ii) I appreciate that you highlight the difference between annotation and classification, and that you want a model trained from scratch for fair comparisons (Section 2.1). However, the CNN trained over ImageNET of [20] or subsequent work has spurred hopes for a universal vision machine. If large
 CNNs trained over 1k and 20k imagenet were available to you, it might be interesting to evaluate how a NUS model initialized from those would perform. This would be an additional result which would not replace the network trained from scratch but rather analyze the reusability of the
 imageNET network and give perspective on the importance of the imageNET breakthrough.

 *Comments along the text*
 Page 2. 'parametric model might not be sufficient to capture the complex distribution of the data' this sentence should be removed. Parametric model can model complex distribution for non linear problems. Use a different wording to introduce that nearest neighbor approaches are competitive.
 Page 3. 'staircase weight-decay' I am not familiar with this name, which is rather explicit though. You might want to sprincke references over neural net specific terms to allow other ML and core vision people to read your paper. E.g. references after momentum, asynchronous SGD, staircase weight
 decay might help. 
 Page 3 'posterior probability of an image x_i and class j' the wording is wrong, it should read posterior of class j given image x_i.
 Page 5 'weight kNN' should read 'weighted kNN'