Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: (1) The motivation
The paper argues that it is more suitable to use non-metric distances instead of metric distances. However, the distance function used in this work is cosine similarity between two l2 normalized features. It is known that in such a situation, cosine similarity is equivalent to Euclidean distance. The motivation should be further explained.

(2) In Eq. (5), I am not sure why not directly set y_ij = 1 if two images come from the same category, and set to 0 otherwise. It is weird to see the annotation is related to the input features considering that we already have the groundtruth labels.

(3) The whole pipeline is not trained in an end-to-end manner. It requires some other features as the input (RMAC used in this work), and three-stage training. It is interesting to see some more experiments where image pixels are the input.

(4) The algorithm is not comparable to the state-of-the-art. Some representative papers have reported much better performances on the datasets used in this paper. It is suggested to refer to some recent papers in top conferences.
review_title: Motivation is unclear and more evaluations are needed

final_score: rating: 7: Good paper, accept
review_content: The authors of this work propose learning a similarity measure for visual similarity and obtain, by doing that, an improvement in the very well-known datasets of Oxford and Paris for image retrieval. The work takes high-level image representations generated with an existing architecture (R-MAC), and train on top a neural network of two fully connected layers. 

The training of such network is performed in three stages: firstly approximating the cosine similarity with a large amount of random feature vectors, secondly using image pairs from the same class, and finally using the hard examples.


PROS

P1. Results indicate the benefit of this approach in terms of similarity estimation and, overall, the paper present results that extend the state of the art in well-known datasets. 

P2. The authors make a very nice effort in motivation the paper, relating it with the state of the art and funding their proposal on studies regarding human visual perception. The whole text is very well written and clear to follow.

CONS

C1. As already observed by the authors, training a similarity function without considering images from the target dataset is actually harmful. In this sense, the simple cosine similarity does not present this drawback in terms of lack of generalization. This observation is not new, but relevant in the field of image retrieval, where in many applications the object of interest for a query is actually not present in the training dataset.

C2. The main drawback of this approach is in terms of computation. Feed-forwarding the two samples through the trained neural network is far more expensive that computing the simple cosine similarity, which is computed very quickly with a GPU as a matrix multiplication. The authors already point at this in Section 4.3.

C3. I am somehow surprised that the authors did not explore also training the network that would extract the high-level representations, that is, a complete end-to-end approach. While I would expect to have the weights frozen in the first phase of training to miimic the cosine similarity, why not freeing the rest of layers when dealing with pairs of images ?

C4. There are a couple of recent papers that include results of the state of the art which are closer and sometimes better than the ones presented in this work. I do not think they reduce at all the contribution of this work, but they should be cited and maybe included in the tables:

A. Gordo, J. Almazan, J. Revaud, and D. Larlus. End-to-end learning of deep visual representations for image retrieval.
International Journal of Computer Vision, 124(2):237–254, 2017.

Albert Jimenez, Jose M. Alvarez, and Xavier Giro-i-Nieto. “Class-Weighted Convolutional Features for Visual Instance Search.” In Proceedings of the 28th British Machine Vision Conference (BMVC). 2017.

review_title: Pushing the performance in image retrieval by learning a non-metric similarity

final_score: rating: 3: Clear rejection
review_content: This paper presents a simple image retrieval method. Paper claims it is a deep learning method, however it is not an end-to-end network. The main issue of the paper is lack of technical contributions.

Paper assumes that image retrieval task can be reformulated at a supervised similarity learning task. That is fine, however image retrieval is traditionally an unsupervised task. 

Even after using supervised method and deep learning technique, still this method is not able to obtain better results than hand crafted methods. Why is that? See - paper from CVPR2012 -  Arandjelović, Relja, and Andrew Zisserman. "Three things everyone should know to improve object retrieval." Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, 2012.

Paper make use of external signal to obtain y_{i,j}. It is not clear to me how does this generalize to large datasets?

If features are L2 normalized, why you need to normalize the features again in equation 5?

In equation 5, why not simply use a max margin deep similarity metric learning method with slack variables to generalizability?

The performance of entire network really rely on the accuracy of y_{i,j} and it is not clear the obtained performance is simply due to this supervision.

Paper does not argue well why we need this supervision.

Technically, there is nothing new here.

review_title: Lack of technical contribution

Meta-review: Two reviewers recommended rejection, and the last reviewer votes for acceptance. The authors provided a rebuttal, including the end-to-end experiment (although the AC agrees with the authors that this experiment is not crucial to the paper). The AC read the paper and the reviews. While there are clearly interesting aspects of this work, it somewhat falls short in terms of the technical contribution. Perhaps a better writing would alleviate this issue: for example, explaining the visual features is somewhat a distraction from the main point, and could be put in the end. The 3 stage training is somewhat ad hoc (or less elegant). Since there are many excellent papers submitted to ICLR this year, this paper unfortunately did not make it above the bar.
Decision: Reject