Deep Convolutional Ranking for Multilabel Image Annotation
Authors: Sergey Ioffe, Alexander Toshev, Yangqing Jia, Thomas Leung, Yunchao Gong
ICLR 2014

Abstract: Multilabel image annotation is one of the most important challenges in computer vision with many real-world applications. While existing work usually use conventional visual features for multilabel annotation, the recent deep convolutional feature shows potentials to significantly boost performance. In this work, we propose to leverage the advantage of such features and analyze key components that lead to better performances. Specifically, we show that a significant performance gain could be obtained by combining convolutional architectures with an approximate top-$k$ ranking objective function, as such objectives naturally fit the multilabel tagging problem. Our experiments on the publicly available NUS-WIDE dataset outperforms the conventional visual features by about $10%$, obtaining the best reported performance in the literature.


Peer review:
[Review]:
This paper proposes to use deep convolutional neural network (DCNN)combined with ranking training criteria to attack the multi-label image annotation problem. DCNN is now widely used in image classification (annotation) problems. Applying it to multi-label image annotation problem is a natural extension of prior arts. The combination of DCNN with the ranking training criteria to solve the multi-label annotation problem, however, is new and is the main contribution of the paper.

The authors evaluated the proposed approach on the NUS-WIDE dataset, which is considered the largest multi-label image dataset available. They compared the proposed approach with baselines that use manually designed image features and showed that the proposed approach outperforms the baseline by 10%. They claim that this is mostly due to the features learned from DCNN. They also compared several different ranking criteria and demonstrated that the weighted Approximate Ranking (WARP) criterion performs the best.

While their results are not surprising given the recent success of DCNN on image classification tasks, this paper does show a novel usage of the DCNN on the multi-label image annotation problem. 

If the paper is to be improved, I would suggest to include published results on the same task as part of the baselines. This allows readers to understand better the position of the proposed approach.