Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: There has been substantial recent interest in representation learning, and specifically, using distance metric learning (DML) to learn representations where semantic distance between inputs can be measured. This is a topic of particular relevance / interest to ICLR. This paper poses a simple yet provocative question: can a standard SoftMax based approach learn features that match or even outperform recent state-of-the-art DML approaches? Thorough experiments seem to indicate that this is indeed the case. Comparisons are made to recent DML approaches including Magnet Loss (ICLR2016) and Lifted structure embedding (CVPR2016) and superior results are shown across a number of datasets / tasks for which the DML approaches were designed. 

This main result is a bit surprising since SoftMax is a natural and trivial baseline, so it should have been properly evaluated in previous DML literature. The authors argue that previous approaches did not fully/properly tune the softmax baselines, or that comparisons were not apples-to-apples. Also, one change in the current paper is the addition of L2 normalization, which is well motivated and helps improve SoftMax feature distances. Different dimensionality reduction approaches are also tested. These changes are minor, but especially the L2 normalization proves to be a simple but effective improvement for SoftMax features.

A big issue is with how pre-training is performed (in Magnet Loss the softmax baselines were pretrained for less time on ImageNet). The approach taken here is reasonable, but so is the approach in Magnet Loss (for different reasons). Ultimately, both are fine. Unfortunately, due to use of different schemes, the results are not comparable. Let me copy-paste what I wrote in an earlier comment: 

My main concern about the paper is that the comparisons in Tables 1 and 2 and Figure 4 to Rippel et al. are not apples-to-apples. Basically, the papers shows that absolute results of using SoftMax w full pre-training (PT) on ImageNet is superior to any of the results in Rippel's paper (including both the Softmax and Magnet results). But as the current results show, PT appears to be critical to obtaining such good numbers - The Rippel SoftMax numbers use only 3 PT stages and are dramatically worse than the full PT on ImageNet. As it stands, I am not convinced that SoftMax is actually better than Magnet. Here is the evidence we have (I'll use Stanford Dogs as an example, but any of the datasets have the same conclusion): (1) Softmax w 3 stages of PT: 26.6% (from Rippel paper) and 32.7% (from authors' reproduction) (2) Magnet w 3 stages of PT: 24.9% (3) Softmax w full PT: 18.3% (4) Magnet w full PT: not shown From this all I see is that PT is critical for getting absolute good results. However, what about Magnet w full PT? These results are not shown either here or in the original Rippel paper (I went back and looked). As such, I do not think it is justifiable to claim superiority of Softmax to Magnet based on available evidence. (Note: I looked back carefully at Rippel's paper, and it appears that the authors use 3 PT stages as a form of "warmup". There is a statement that using full PT would "defeat the purpose of pursuing DML". I'm not sure if I agree w Rippel's statement since in the present paper there is clear evidence that full PT is hugely helpful, at least for softmax. That being said, I did not see any evidence in the Rippel paper that PT is harmful or that DML wouldn't work with full PT.)

The authors responded to my concern by claiming that “from Rippel's results, it is no doubt that Magnet@3epochPT > Magnet@FullPT and Magnet@3epoch > Softmax@3epochPT.” However, I went back to Rippel’s paper, and simply the Magnet@FullPT experiment never appears. I further went and contacted Oren Rippel himself, and he verified he never ran the Magnet@FullPT experiment. I encourage the authors to contact Oren Rippel regarding this if they wish to verify (I have asked Oren Rippel to not reveal my identity). [Disclaimer: I am NOT Oren Rippel]. The authors mentioned that they are training Magnet@FullPT. If results were shown for Magnet@FullPT and also retrain the Magnet@3epochPT as a sanity check, that would help alleviate this concern. Alternatively, the language in Section 4 and the Tables could be altered to make clear that the methods use different pretraining and hence are not comparable.

Overall, I am actually quite sympathetic to this work. I think it could serve as an important sanity-check paper for the community and quite relevant to ICLR. Having proper and strong SoftMax baselines should prove quite useful to the DML community and to this line of work.

However, currently I find the main results (table 1, table 2, figure 4, etc.) to be misleading. If indeed it were the case that Magnet@3epochPT > Magnet@FullPT, then it would be fine. However, at this point as far as I know no one has actually tried Magnet@FullPT. And, given the general importance and effectiveness of pre-training, especially when transferring to small dataset, I would be hugely surprised if Magnet@FullPT was not superior by a large margin. I think either having this experiment in place or altering the writing / presentation of the results would be critical to allow for publishing.
review_title: review

final_score: rating: 7: Good paper, accept
review_content: I have a huge, big picture concern about this paper and the papers it most closely addresses (MagnetLoss and Lifted Feature Structure Embedding). I don't understand why Distance Metric Learning (DML) is being used for classification tasks (Stanford Cars 196, UCSD Birds 200, Oxford 102 flowers, Stanford Dogs, ImageNet attributes, etc). As far as I can tell, there is really only a single "retrieval"-like benchmark being used here - the Stanford Online Products database. All the other datasets are used in a "classification-by-retrieval" approach which seems contrived. While ostensibly evaluating "retrieval", the retrieval ground truth is totally defined by category membership so these are still classification tasks with many instances in each category. With the Online Products dataset the correspondence between queries and correct results is much more fine grained so it makes sense to think of it as a retrieval task.

It seems obvious that if your task is classification, a network trained with a classification loss will be best. Even when these datasets are used in a "retrieval" setting, the ground truth is still defined by category membership. It's still a classification task. 

I don't really see the point of using DML in these scenarios. I guess prior work claims to outperform SoftMax in these settings so this paper is fighting back against this and I should be thankful for this paper. But I think this paper's narrative is a bit off. The narrative shouldn't be "We can get good retrieval features from softmax networks with appropriate normalization". It should be "It never made sense to train or evaluate these things as retrieval tasks. Direct classification is better". For example, why are you taking the second to last layer or pool5 layer from these networks? Why aren't you taking the last layer? That should do well in these evaluations, right? Table 1 and 2 do show that using softmax probabilities directly tends to be better than doing classification-by-retrieval (works better or the same as doing retrieval with an earlier layer of features, except on Oxford flowers).

GoogLeNet is quite deep and gets auxiliary supervision. By the second-to-last layer of the network, the activations could look a lot like category membership already. And category membership is all that's needed for the tasks in 4.2 and 4.3. 
 
I don't think my pre-review question was adequately addressed. I was getting at this concern by pointing out numerous scenarios where distance metric learning makes sense because you have fine-grained associations between instances at training time, NOT categorical associations -- e.g. this product photo corresponds to this photo of the object in a scene [Bell et al. 2015], this 3d model correspond to this sketch [Wang et al. 2015], this sketch corresponds to this photo [Sangkloy et al. 2016], this ground view corresponds to this aerial view [Lin et al., 2015]. DeepFace and follow-up works on LFW could also fit into this space because there are few training samples per class (few training samples per person identity). You cite DeepFace and Bell et al. 2015 but you don't compare on those benchmarks. I think those are exactly the tasks where DML makes sense.

Maybe the "retrieval on classification datasets" would be a reasonable benchmark if the test and train classes were completely different. Then you could argue that softmax is learning a useful representation yet the last layer isn't directly useful since the categories change. But that's not the case here, is it?

With all of this said, I'm not sure whether I'm positive or negative about this paper. I think you're onto something significant -- people have been using DML where it is not appropriate -- but addressed it in the wrong way -- by using softmax for "classification by retrieval". But you don't need to do retrieval! Softmax is already telling you the class prediction! Why go through the extra step of finding nearest neighbors with some intermediate feature?

AnonReviewer3 also raises some good points and you should be thankful that a reviewer is willing to dig so deep to help make your experiments sound! I don't think his/her concerns are disqualifying for this paper, though, as long as it is fixed.

I look forward to hearing your response. I want this paper to be published, but I think it needs to be tweaked.


review_title: Interesting observations, but the paper seems misguided or incomplete

final_score: rating: null
review_content: L2 normalization seemed to very important for experiments in S4.4. Yet in table 1 the experiments show little difference w and w/o normalization. Why? (Did you use cosine distance in any of the experiments, or was it all L2?) 

Could you please report the pool5 / pool5+L2 results in table 2? (Or is there a reason these were not included in table 2?)

And one more question: if you train SoftMax but only pre-train for 3 epochs (as in Rippel), do you get the same number in table 1 / table 2 as Rippel (Softmax prob)? I would like to see this verified to understand if this is the only difference between these results. Otherwise I am concerned about the large gap here.
review_title: questions about experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: I agree with the other two reviewers that it is an interesting topic to investigate the feature learned by DML. For classification task though, I feel intuitively softmax should have advantages over distance metric learning method because the loss function is designed to assign the correct class for the given image. All the experimental results show that the softmax features work better than Rippel et al DML method. However, does it support the claim that softmax-based features work much better than DML learned features? I have doubts on this claim. 

Also the experiments are a little bit misleading. What is vanilla googleNet softmax finetuned results? It seems it is not Rippel et al. (softmax prob) result. I am wondering whether the improvement comes from a) using retrieval (nearest neighbor) for classification or b) adding a new layer on top of pool5 or c) L2 normalization of the features. It is not clear to me at all. It appears to me the comparison is not apple vs apple between the proposed method and Rippel et al. 

It would be great if we know adding feature reduction or adding another layer on top of pool5 can improve finetued softmax result. However, I am not sure what is the biggest contributing factor to the superior results. Before getting more clarifications from the authors, I lean toward rejection. 


review_title: Interesting discussion but sharing concern with reviewer 3 

final_score: rating: null
review_content: One area where DML is popular is Image Geolocalization, e.g. Lin et al. Learning Deep Representations for Ground-to-Aerial Geolocalization, CVPR 2015. In this domain the number of categories would be the number of locations, which could be in the tens of thousands to millions, with one pair of training examples per location. I suppose this would be an even more extreme situation than the Stanford Products (OP) dataset you evaluated on. Works in that domain have proposed improvements to the contrastive loss or triplet loss. See Vo and Hays, ECCV 2016. That work also proposes a way to address another problem you identify -- "DML optimization converges very slowly" -- by using all possible triplets in each training batch of N, you can get O(N^2) training samples instead of the typical O(N).

Another element of these particular geolocalization works is that they are cross domain (e.g. ground images and overhead images, also the case in Workman et al. Wide-Area Image Geolocalization with Aerial Reference Imagery. ICCV 2015, but they don't use DML). Another example is sketch-based image retrieval, e.g. Yu et al. Sketch Me That Shoe, CVPR 2016. Sangkloy et al. The Sketchy Database: Learning to Retrieve Badly Drawn Bunnies, Siggraph 2016. and sketch-based 3d model retrieval e.g. Wang et al. Sketch-based 3D Shape Retrieval using Convolutional Neural Networks. CVPR 2015. With DML it's trivial to have multiple networks, trained on different domains with independent weights, and have a loss on the final layer which asks them to place matching inputs nearby in feature space. What is the analog in the softmax approach? You can't have two fully independent networks, because the layers prior to the final layer won't learn comparable embeddings. You would need partially fused classification networks of some sort?
review_title: Applicability to Geolocalization and cross-domain DML

final_score: rating: null
review_content: The revision raises even more concern and red flags than the original version.

My main concern originally was that the comparisons to Rippel et al. were not apples-to-apples due to different ImageNet pre-training strategies. Those strategies were discussed in the original draft, but in the experiments were not highlighted, so the experiments felt quite misleading.

The authors now claim that in the paoer "These strategies are exactly the same as those of the previous method (Rippel et al., 2016)." So instead of modifying the experiments and the tone of the paper to account for the differences, now the differences are entirely removed from the paper! The authors provided extended discussion in the comment thread here, but it was inconclusive. The paper now contains no discussion of this absolutely critical issue.

While I was originally sympathetic to this work, at this stage I am concerned it will create much more confusion in the community rather than resolve it. As such, I cannot recommend acceptance.

My updated recommendation is: "4: Ok but not good enough - rejection"
review_title: revision more problematic than the original

Meta-review: The paper aims to compare the representations learnt by metric learning and classification objectives. While this is an interesting topic, the presented evaluation is not sufficiently clear for the paper to be accepted.
Decision: Reject