Title: Significance of Softmax-Based Features over Metric Learning-Based Features
Authors: Shota Horiguchi
Daiki Ikami
Kiyoharu Aizawa
Venue: ICLR 2017
Keywords: Computer vision
Deep learning

Abstract: The extraction of useful deep features is important for many computer vision tasks.
Deep features extracted from classification networks have proved to perform well in those tasks.
To obtain features of greater usefulness, end-to-end distance metric learning (DML) has been applied to train the feature extractor directly.
End-to-end DML approaches such as Magnet Loss and lifted structured feature embedding show state-of-the-art performance in several image recognition tasks.
However, in these DML studies, there were no equitable comparisons between features extracted from a DML-based network and those from a softmax-based network.
In this paper, by presenting objective comparisons between these two approaches under the same network architecture, we show that the softmax-based features are markedly better than the state-of-the-art DML features for tasks such as fine-grained recognition, attribute estimation, clustering, and retrieval.

TL;DR: We show softmax-based features are markedly better than state-of-the-art metric learning-based features by conducting fair comparison between them.

Review discussion:
L2 normalization seemed to very important for experiments in S4.4.
Yet in table 1 the experiments show little difference w and w/o normalization. Why? (Did you use cosine distance in any of the experiments, or was it all L2?)
Could you please report the pool5 / pool5+L2 results in table 2? (Or is there a reason these were not included in table 2?)
And one more question: if you train SoftMax but only pre-train for 3 epochs (as in Rippel), do you get the same number in table 1 / table 2 as Rippel (Softmax prob)? I would like to see this verified to understand if this is the only difference between these results.
Otherwise I am concerned about the large gap here.
Thank you very much for your review.
>>L2 normalization seemed to very important for experiments in S4.4.
Yet in table 1 the experiments show little difference w and w/o normalization. Why? (Did you use cosine distance in any of the experiments, or was it all L2?)
We consider that the difference between these results is caused by the difference of tasks and evaluation metrics.
Fig.1(b) shows the reason why the L2 normalization is especially important for specific tasks.
For classification, attribute estimation, and retrieval, the evaluation is based on nearest neighbor method thus unnormalized features, which are radially distributed like Fig.1(b), can perform well to some extent.
However, for clustering, unnormalized features cause serious decline of performance.
(We have used Euclidean distance in all of our experiments)
We finally note that L2 normalization could improve all of our experimental performances of pool5 features.
>>Could you please report the pool5 / pool5+L2 results in table 2? (Or is there a reason these were not included in table 2?)
We added the results to table 2.
For the ImageNet attribute dataset, pool5 features are better than Magnet, and the L2 normalization works effectively.
>>And one more question: if you train SoftMax but only pre-train for 3 epochs (as in Rippel), do you get the same number in table 1 / table 2 as Rippel (Softmax prob)? I would like to see this verified to understand if this is the only difference between these results.
Otherwise I am concerned about the large gap here.
There are no detailed information about training strategies of pre-training in Rippel's paper.
We tried to reproduce Rippel's experiments: first, we trained 3-epoch model using BN-x30 strategy of [1], and then fine-tuned for each dataset.
We obtained the following error rates in fine-grained recognition tasks.
Stanford dogs: 32.7%
Oxford 102 flowers: 12.5%
Oxford-IIIT Pet: 19.8%
These results are different (even worse) from Rippel's results.
There are two main reasons which make reproduction of their results difficult.
(i) 3-epoch trained models are very different between their training strategies. See Fig.2 of [1]. The difference of final performance between various training strategies are small, but at 3-epoch the loss values are very different.
(ii) The training is "on the way" at 3-epoch so that loss values are not stable.
For these reasons, we could not get exact values of Rippel's results.
However, it is obvious that the halfway models perform worse than the fully-trained model.
Thanks again for your feedback.
[1] S. Ioffe and C. Szegedy, "Batch normalization: accelerating deep network trained by reducing internal covariate shift."
Thank you for the clarification, that is quite helpful.
My main concern about the paper is that the comparisons in Tables 1 and 2 and Figure 4 to Rippel et al. are not apples-to-apples. Basically, the papers shows that absolute results of using SoftMax w full pre-training (PT) on ImageNet is superior to any of the results in Rippel's paper (including both the Softmax and Magnet results).
But as the current results show, PT appears to be critical to obtaining such good numbers - The Rippel SoftMax numbers use only 3 PT stages and are dramatically worse than the full PT on ImageNet.
As it stands, I am not convinced that SoftMax is actually better than Magnet. Here is the evidence we have (I'll use Stanford Dogs as an example, but any of the datasets have the same conclusion):
(1) Softmax w 3 stages of PT: 26.6% (from Rippel paper) and 32.7% (from authors' reproduction)
(2) Magnet w 3 stages of PT: 24.9%
(3) Softmax w full PT: 18.3%
(4) Magnet w full PT: not shown
From this all I see is that PT is critical for getting absolute good results.
However, what about Magnet w full PT? These results are not shown either here or in the original Rippel paper (I went back and looked).
As such, I do not think it is justifiable to claim superiority of Softmax to Magnet based on available evidence.
(Note: I looked back carefully at Rippel's paper, and it appears that the authors use 3 PT stages as a form of "warmup". There is a statement that using full PT would "defeat the purpose of pursuing DML". I'm not sure if I agree w Rippel's statement since in the present paper there is clear evidence that full PT is hugely helpful, at least for softmax. That being said, I did not see any evidence in the Rippel paper that PT is harmful or that DML wouldn't work with full PT.)
If the authors have any response to my comments (or proposed changes they could make to the paper) I'd be happy to take them into account before posting a full review.
Thanks for your reply.
Rippel wrote that "We find that it is useful to warm-start any DML optimization with weights of a partly-trained a standard softmax classifier. It is important to not use weights of a net trained to completion, as this would result in information dissipation and as such defeat the purpose of pursuing DML in the first place." "the purpose of pursuing DML" is in Rippel's abstract: "Distance metric learning (DML) approaches learn a transformation to a representation space where distance is in correspondence with a predefined notion of similarity."
Ripppel verified whether Magnet could "learn a transformation to a representation space where distance is in correspondence with a predefined notion of similarity" by calculating attribute precision, and they found that 3-epoch model is better than full PT model, which caused "information dissipation".
Thus, from Rippel's results, it is no doubt that Magnet@3epochPT > Magnet@FullPT and Magnet@3epoch > Softmax@3epochPT.
In our paper, Softmax@FullPT > Magnet@3epochPT, as shown in Fig.4.
As above, it is reported in Rippel's paper that Magnet@FullPT was worse than Magnet@3epoch.
And, at least in clustering and retrieval task, Softmax@FullPT significantly outperformed DML-based model(lifted)@FullPT.
For confirmation, we just started to train Magnet from FullPT model now.
We will report results if possible.
Thanks again.
One area where DML is popular is Image Geolocalization, e.g. Lin et al. Learning Deep Representations for Ground-to-Aerial Geolocalization, CVPR 2015. In this domain the number of categories would be the number of locations, which could be in the tens of thousands to millions, with one pair of training examples per location. I suppose this would be an even more extreme situation than the Stanford Products (OP) dataset you evaluated on. Works in that domain have proposed improvements to the contrastive loss or triplet loss. See Vo and Hays, ECCV 2016. That work also proposes a way to address another problem you identify -- "DML optimization converges very slowly" -- by using all possible triplets in each training batch of N, you can get O(N^2) training samples instead of the typical O(N).
Another element of these particular geolocalization works is that they are cross domain (e.g. ground images and overhead images, also the case in Workman et al. Wide-Area Image Geolocalization with Aerial Reference Imagery. ICCV 2015, but they don't use DML).
Another example is sketch-based image retrieval, e.g. Yu et al. Sketch Me That Shoe, CVPR 2016. Sangkloy et al. The Sketchy Database: Learning to Retrieve Badly Drawn Bunnies, Siggraph 2016. and sketch-based 3d model retrieval e.g. Wang et al. Sketch-based 3D Shape Retrieval using Convolutional Neural Networks. CVPR 2015. With DML it's trivial to have multiple networks, trained on different domains with independent weights, and have a loss on the final layer which asks them to place matching inputs nearby in feature space. What is the analog in the softmax approach? You can't have two fully independent networks, because the layers prior to the final layer won't learn comparable embeddings. You would need partially fused classification networks of some sort?
Thank you for your review.
For cross domain image retrieval, there is an existing work in CVPR 2016: Castrejon+ ''Learning Aligned Cross-Modal Representations from Weakly Aligned Data.'' They learned cross-modal representations of natural images, sketches, clip art, spatial text, and descriptions by using softmax-based classifier.
We agree that DML based features might be better in such as image geolocalization. Based on our results, in OP and data-restricted CAR and CUB, which the number of samples per class is limited, we showed the difference between the performance of softmax-based features and that of DML-based features is little. As far as we know, this paper is the first work which made clear about this tendency by equitable comparisons.
We finally note that lifted structured feature embedding, which we used in our paper as a baseline in clustering and retrieval tasks, used all possible pairs in a batch to use O(N^2) distances from O(N) samples.
Thanks.
There has been substantial recent interest in representation learning, and specifically, using distance metric learning (DML) to learn representations where semantic distance between inputs can be measured. This is a topic of particular relevance / interest to ICLR. This paper poses a simple yet provocative question: can a standard SoftMax based approach learn features that match or even outperform recent state-of-the-art DML approaches? Thorough experiments seem to indicate that this is indeed the case. Comparisons are made to recent DML approaches including Magnet Loss (ICLR2016) and Lifted structure embedding (CVPR2016) and superior results are shown across a number of datasets / tasks for which the DML approaches were designed.
This main result is a bit surprising since SoftMax is a natural and trivial baseline, so it should have been properly evaluated in previous DML literature. The authors argue that previous approaches did not fully/properly tune the softmax baselines, or that comparisons were not apples-to-apples.
Also, one change in the current paper is the addition of L2 normalization, which is well motivated and helps improve SoftMax feature distances. Different dimensionality reduction approaches are also tested. These changes are minor, but especially the L2 normalization proves to be a simple but effective improvement for SoftMax features.
A big issue is with how pre-training is performed (in Magnet Loss the softmax baselines were pretrained for less time on ImageNet). The approach taken here is reasonable, but so is the approach in Magnet Loss (for different reasons).
Ultimately, both are fine. Unfortunately, due to use of different schemes, the results are not comparable. Let me copy-paste what I wrote in an earlier comment:
My main concern about the paper is that the comparisons in Tables 1 and 2 and Figure 4 to Rippel et al. are not apples-to-apples. Basically, the papers shows that absolute results of using SoftMax w full pre-training (PT) on ImageNet is superior to any of the results in Rippel's paper (including both the Softmax and Magnet results).
But as the current results show, PT appears to be critical to obtaining such good numbers - The Rippel SoftMax numbers use only 3 PT stages and are dramatically worse than the full PT on ImageNet. As it stands, I am not convinced that SoftMax is actually better than Magnet. Here is the evidence we have (I'll use Stanford Dogs as an example, but any of the datasets have the same conclusion): (1) Softmax w 3 stages of PT: 26.6% (from Rippel paper) and 32.7% (from authors' reproduction) (2) Magnet w 3 stages of PT: 24.9% (3) Softmax w full PT: 18.3% (4) Magnet w full PT: not shown From this all I see is that PT is critical for getting absolute good results.
However, what about Magnet w full PT? These results are not shown either here or in the original Rippel paper (I went back and looked).
As such, I do not think it is justifiable to claim superiority of Softmax to Magnet based on available evidence. (Note: I looked back carefully at Rippel's paper, and it appears that the authors use 3 PT stages as a form of "warmup". There is a statement that using full PT would "defeat the purpose of pursuing DML". I'm not sure if I agree w Rippel's statement since in the present paper there is clear evidence that full PT is hugely helpful, at least for softmax. That being said, I did not see any evidence in the Rippel paper that PT is harmful or that DML wouldn't work with full PT.)
The authors responded to my concern by claiming that “from Rippel's results, it is no doubt that Magnet@3epochPT > Magnet@FullPT and Magnet@3epoch > Softmax@3epochPT.” However, I went back to Rippel’s paper, and simply the Magnet@FullPT experiment never appears. I further went and contacted Oren Rippel himself, and he verified he never ran the Magnet@FullPT experiment. I encourage the authors to contact Oren Rippel regarding this if they wish to verify (I have asked Oren Rippel to not reveal my identity). [Disclaimer: I am NOT Oren Rippel]. The authors mentioned that they are training Magnet@FullPT. If results were shown for Magnet@FullPT and also retrain the Magnet@3epochPT as a sanity check, that would help alleviate this concern.
Alternatively, the language in Section 4 and the Tables could be altered to make clear that the methods use different pretraining and hence are not comparable.
Overall, I am actually quite sympathetic to this work. I think it could serve as an important sanity-check paper for the community and quite relevant to ICLR. Having proper and strong SoftMax baselines should prove quite useful to the DML community and to this line of work.
However, currently I find the main results (table 1, table 2, figure 4, etc.) to be misleading. If indeed it were the case that Magnet@3epochPT > Magnet@FullPT, then it would be fine.
However, at this point as far as I know no one has actually tried Magnet@FullPT. And, given the general importance and effectiveness of pre-training, especially when transferring to small dataset, I would be hugely surprised if Magnet@FullPT was not superior by a large margin. I think either having this experiment in place or altering the writing / presentation of the results would be critical to allow for publishing.
Thank you for your review,
We're still doing experiments of Magnet loss.
We tried Magnet@fullPT and Magnet@3epochPT for comparison, but it reaches lower accuracies than those in Rippel's paper.
(For example, we got about 25% classification error on Oxford-102Flowers dataset for both Magnet@3epochPT and Magnet@FullPT.)
We have sent an email to Rippel, and waiting for his reply.
Please wait for a while.
We posted about the reproductive experiments of Magnet Loss.
please see our comments
I have a huge, big picture concern about this paper and the papers it most closely addresses (MagnetLoss and Lifted Feature Structure Embedding). I don't understand why Distance Metric Learning (DML) is being used for classification tasks (Stanford Cars 196, UCSD Birds 200, Oxford 102 flowers, Stanford Dogs, ImageNet attributes, etc). As far as I can tell, there is really only a single "retrieval"-like benchmark being used here - the Stanford Online Products database. All the other datasets are used in a "classification-by-retrieval" approach which seems contrived.
While ostensibly evaluating "retrieval", the retrieval ground truth is totally defined by category membership so these are still classification tasks with many instances in each category. With the Online Products dataset the correspondence between queries and correct results is much more fine grained so it makes sense to think of it as a retrieval task.
It seems obvious that if your task is classification, a network trained with a classification loss will be best.
Even when these datasets are used in a "retrieval" setting, the ground truth is still defined by category membership. It's still a classification task.
I don't really see the point of using DML in these scenarios. I guess prior work claims to outperform SoftMax in these settings so this paper is fighting back against this and I should be thankful for this paper.
But I think this paper's narrative is a bit off. The narrative shouldn't be "We can get good retrieval features from softmax networks with appropriate normalization". It should be "It never made sense to train or evaluate these things as retrieval tasks. Direct classification is better".
For example, why are you taking the second to last layer or pool5 layer from these networks? Why aren't you taking the last layer? That should do well in these evaluations, right? Table 1 and 2 do show that using softmax probabilities directly tends to be better than doing classification-by-retrieval (works better or the same as doing retrieval with an earlier layer of features, except on Oxford flowers).
GoogLeNet is quite deep and gets auxiliary supervision. By the second-to-last layer of the network, the activations could look a lot like category membership already. And category membership is all that's needed for the tasks in 4.2 and 4.3.
I don't think my pre-review question was adequately addressed. I was getting at this concern by pointing out numerous scenarios where distance metric learning makes sense because you have fine-grained associations between instances at training time, NOT categorical associations -- e.g. this product photo corresponds to this photo of the object in a scene [Bell et al. 2015], this 3d model correspond to this sketch [Wang et al. 2015], this sketch corresponds to this photo [Sangkloy et al. 2016], this ground view corresponds to this aerial view [Lin et al., 2015]. DeepFace and follow-up works on LFW could also fit into this space because there are few training samples per class (few training samples per person identity). You cite DeepFace and Bell et al. 2015 but you don't compare on those benchmarks. I think those are exactly the tasks where DML makes sense.
Maybe the "retrieval on classification datasets" would be a reasonable benchmark if the test and train classes were completely different.
Then you could argue that softmax is learning a useful representation yet the last layer isn't directly useful since the categories change.
But that's not the case here, is it?
With all of this said, I'm not sure whether I'm positive or negative about this paper. I think you're onto something significant -- people have been using DML where it is not appropriate -- but addressed it in the wrong way -- by using softmax for "classification by retrieval".
But you don't need to do retrieval! Softmax is already telling you the class prediction! Why go through the extra step of finding nearest neighbors with some intermediate feature?
AnonReviewer3 also raises some good points and you should be thankful that a reviewer is willing to dig so deep to help make your experiments sound! I don't think his/her concerns are disqualifying for this paper, though, as long as it is fixed.
I look forward to hearing your response. I want this paper to be published, but I think it needs to be tweaked.
We thank you for your helpful review.
1) classification and retrieval
We think there might be some misunderstanding about our experimental settings. In section 4.2 we evaluated the classification performances, as you pointed out, and it was not surprising that softmax-based features outperformed DML-based features.
However, in section 4.3 "we used only the images and their class labels during our training of the softmax classifier and did not use attributes" (described in our paper). And in section 4.4, the classes of training data are completely different from those of test data, as in table 3.
Therefore we evaluated the retrieval (clustering) performance, not the classification performance.
2) comparison to DML
We agree with that DML is suitable in cases of non-categorical problems (e.g. Bell+, Wang+, etc.) or problems with few training samples.
Our main concern is "learning beneficial deep features", i.e. useful for such as retrieval, clustering, one-shot and zero-shot learning. In these problems, we can often use class labels during the training, however the classes of test data is different from those of training.
Therefore we cannot use class prediction and extracting useful intermediate features is important for these problems.
In section 4.4, we showed that softmax-based features are superior to DML-based features in image retrieval and clustering if we have enough training samples. Even in the case of few training samples, softmax-based features achieved comparable performance in the OP dataset, which contains only about 5.5 training samples per class.
3) about Learning visual product design[Bell+, 2015], DeepFace [Taigman+, 2014], facenet [Schroff+, 2015]
As far as we know, these studies used private datasets for training.
Moreover, they used simple siamese or triplet based networks.
Therefore we compared with more sophisticated methods, Magnet loss and Lifted structed feature embedding, on publicly available datasets.
1) Ok, great, thanks for the clarifications. That does make your experiments more reasonable.
I guess I was confused about this line in Table 2 "Ours (Softmax prob) 7.68%". If it wasn't trained with attributes, what is this saying? You're using the activations after softmax as the feature for retrieval? Or you've trained another network to predict the attributes directly?
2) I don't think those examples have "few training samples" -- they have 10's of thousands or more. They have few samples per category, though.
But I think that's what you meant.
3) You can find some of the datasets for papers I mentioned (Sangkloy et al, Vo and Hays).
Anyway, I'll raise my score based on your clarifications. I still think you need to change the narrative of your paper a bit, though. It should be less about beating Magnet loss and more about the value of using class labels (when available) to learn good feature embeddings, and it would be better to take this to the extreme with a dataset of 2 instances per "class", a truly fine grained retrieval task, and see how things work.
1) Table 2:
ImageNet Attribute dataset, which we used in section 4.3, is a subset of ImageNet and it contains 116,836 samples and 90 classes with attributes. As we described, networks were trained by class labels, not attributes.
Table 2 is "Classification error rates for the ImageNet Attribute dataset", we mean classification error on 90 class prediction. The reason we show Table 2 is that the same table is shown in the paper (Rippel+, 2016) and we wanted to show the performance of the softmax classifier is different when properly used.
2) few training samples
You are right. It was misleading. As you wrote, we meant "few training samples per class," like Online Product dataset and data-restricted CUB and CAR.
We thank you again that your comments make the significance of our paper clear. Discussion on learning good feature embedding is the theme of this paper. We will revise our paper to clarify this point.
We would like to make sure your intentions about some previous studies you introduced, which used DML (Sangkloy et al, Vo and Hays, Bell et al, Wang et al, Taigman et al, etc.).
We think these studies address following problems:
i) huge number of categories (e.g. "more than 1 million matched pairs of street-view and overhead-view" in Vo and Hays)
ii) cross-domain representation (sketch-photo, aerial-ground)
iii) few training samples per class
We write our thoughts on these points one by one.
i) “huge number of categories” is not evaluated in our paper because fully connected layer with millions of outputs cannot be on the GPU memory. This is a softmax's limitation. (It is  possible to learn about ten thousands of classes on single K40 GPU, as shown in our experiment on the OP dataset.)
ii) DML can solve “cross-domain” problem by siamese-like network architecture without weight sharing. For softmax, we introduced the study of Castrejon et al, however their learning strategy is complicated. Our main concern is in the basic framework of feature embedding.
iii) “few training samples per class” is evaluated on Online Product (Fig. 7) and data-restricted CUB and CAR (Fig. 8 and 9).
For example, 5% of the CUB datasets contains 5864*0.05 training samples, about 3 samples per class. It is already very small number. We think the case of 2 samples per class maybe not very different, but if you want to see the case of 2 samples per class, we can do the extra experiments.
I'm certainly not going to demand new experiments, although more can only help.
I'm not really seeking clarifications, either, just trying to adjust the narrative of the paper.
It seems like there's a pool of tasks where DML strategies like contrastive loss / triplet loss / etc. make more sense. And there's a pool of tasks where softmax is the natural thing (because you're only interested in categorization). There is some overlap in these, though -- tasks where the number of categories isn't huge, where multiple types of labels are available and some of them are categorical, where you're interested in retrieval and not categorization, and your paper shows that you can use softmax to learn a good feature embedding. That's great.
But you want to define the boundaries of where this strategy is fruitful and cite situations where maybe it's not ideal (or even viable, as you say is the case with millions of labels).
I agree with the other two reviewers that it is an interesting topic to investigate the feature learned by DML. For classification task though, I feel intuitively softmax should have advantages over distance metric learning method because the loss function is designed to assign the correct class for the given image. All the experimental results show that the softmax features work better than Rippel et al DML method.
However, does it support the claim that softmax-based features work much better than DML learned features? I have doubts on this claim.
Also the experiments are a little bit misleading. What is vanilla googleNet softmax finetuned results? It seems it is not Rippel et al. (softmax prob) result. I am wondering whether the improvement comes from a) using retrieval (nearest neighbor) for classification or b) adding a new layer on top of pool5 or c) L2 normalization of the features. It is not clear to me at all. It appears to me the comparison is not apple vs apple between the proposed method and Rippel et al.
It would be great if we know adding feature reduction or adding another layer on top of pool5 can improve finetued softmax result.
However, I am not sure what is the biggest contributing factor to the superior results. Before getting more clarifications from the authors, I lean toward rejection.
Thank you for your review.
This is a paper that shows the equitable comparisons between softmax features and DML features under the same networks, which leads to clarification of performance of softmax based feature embedding. It was not clear in the previous studies.
Let us clarify questions you raised in the comments.
a) We used 1-NN classification for fair comparison to Rippel. Note that nearest neighbor is used for classification, such as Nearest Class Mean (Mensink et al., 2013). We think the difference between classification and retrieval is whether the class sets of the training data and test data are the same or not.
b) In section 4.2 and 4.3, we used vanilla GoogLeNet w/ batch normalization. We'll revise to specify the network architectures.
c) We show w/ and w/o L2 normalization in our results. For Magnet loss, we note that L2 normalization reduces the performance. This is a natural result we can expect from Fig 1.
The reason of our improvement of softmax is the strategy of fine-tuning (and this is only difference from Rippel's softmax result). Rippel used the networks pretrained on ImageNet for only three epochs, and we used those of fully pretrained, as we described in our paper.
We posted about the reproductive experiments of Magnet Loss.
Please see our comments.
We revised our paper responding to reviewers' comments.
Changes are as follows:
* We revise about Magnet Loss (Section 2.1, 4.2).
* We added limitations to the end of "Conclusion".
* We clarified network architectures in "Procedure".
* Other minor clarifications.
We asked Rippel about unclear points of his proposed method, and from his reply we came to know that the actual Magnet Loss contains another hyper parameter which is not explicitly described in his paper. We also requested him to give us his source codes but were refused because the code is internally and proprietary. We reproduced his Magnet loss and obtained results which seems similar to Rippel's under different interpretation of "3 epochs". We asked him about the interpretation of "3 epochs", but so far we have not received his reply. We will report our findings below.
At first we quote from Rippel's paper as follows:
"We find that it is useful to warm-start any DML optimization with weights of a partly-trained a standard softmax classifier. It is important to not use weights of a net trained to completion, as this would result in information dissipation and as such defeat the purpose of pursuing DML in the first place.
Hence, we initialize all models with the weights of a net trained on ImageNet (Russakovsky et al., 2015) for 3 epochs only."
(1) We had been interpreting this sentence as "the network is trained on ImageNet for 3 epochs." First we conducted the experiments based on this interpretation, but the results were completely worse than those Rippel reported in his paper.
Stanford Dogs: 52.6 % (Rippel's result: 24.9 %)
Oxford 102 Flower: 43.7 % (Rippel's result: 8.6 %)
Oxford-IIIT Pet: 32.0 % (Rippel's result: 10.6 %)
(2) However, the sentence can be interpreted as another way: "the network is fully trained on ImageNet, and then the network is trained on each datasets, e.g. Stanford Dogs, for 3 epochs." We did the experiment by this interpretation and we could reproduce the Rippel's results as:
Stanford Dogs: 25.5 % (Rippel's result: 24.9 %)
Oxford 102 Flower: 7.4 % (Rippel's result: 8.6 %)
Oxford-IIIT Pet: 12.2 % (Rippel's result: 10.6 %)
From these results, we believe that the second interpretation is true.
But if so, we cannot explain why Rippel's results of softmax were completely worse than our results:
Stanford Dogs: 18.3 % (Rippel's result: 26.6 %)
Oxford 102 Flower: 8.69 % (Rippel's result: 11.2%)
Oxford-IIIT Pet: 9.04 % (Rippel's result: 11.3 %)
According to our experiment, their softmax results do not seem the best ones. The training strategies for softmax were not described in his paper, so we cannot confirm whether his experiments were conducted in a fair manner.
Our claim is that softmax-based feature is underestimated in many DML-based studies and must be taken into consideration as the baseline when evaluating DML-based features. (In fact, new DML papers without comparison to softmax-based features appeared during this rebuttal period.) We show that softmax-based features outperform lifted structed embedding, one of the latest DML method, by equitable comparison. Our softmax-based features also outperform Magnet loss which are presented in Rippel's paper. We think this fact is important to make sure that softmax is a strong baseline for feature representation.
The revision raises even more concern and red flags than the original version.
My main concern originally was that the comparisons to Rippel et al. were not apples-to-apples due to different ImageNet pre-training strategies. Those strategies were discussed in the original draft, but in the experiments were not highlighted, so the experiments felt quite misleading.
The authors now claim that in the paoer "These strategies are exactly the same as those of the previous method (Rippel et al., 2016)." So instead of modifying the experiments and the tone of the paper to account for the differences, now the differences are entirely removed from the paper! The authors provided extended discussion in the comment thread here, but it was inconclusive. The paper now contains no discussion of this absolutely critical issue.
While I was originally sympathetic to this work, at this stage I am concerned it will create much more confusion in the community rather than resolve it.
As such, I cannot recommend acceptance.
My updated recommendation is: "4: Ok but not good enough - rejection"
What we found in our reproductive experiments is "when finetuned from fully-pretrained weights, softmax outperformed Magnet Loss."
We think this fact is very important for the community.
PS
He said that he could not release his codes in his first reply.
He did not respond about the details of his experiments.
The objective contains an additional hyperparameter, so perfect reproduction is very hard.
We tried a lot of experimental settings to reproduce his results, and managed to reproduce them.
Our result is still significantly better and our claim does not change.
Is it a confusing result?
This is a paper with essentially no novelty that aims to show that softmax can outperform DML.
As such, the experimental correctness is of fundamental importance.
However, the experiments are not up to par.
The authors have not been able to reproduce Rippel's result, and Rippel has not released code. This is unfortunate.
Nevertheless, it does not change the main concern with this paper.
And to reiterate: none of the discussion in these comments below appears in the paper. The authors' current hypothesis is that Rippel does full pre-training on ImageNet, which directly contradicts what is written in Rippel's paper (and the authors' experiments don't fully support this either, as it does not explain the softmax results in Rippel's paper).
But, let me not quibble about the details: the damning concern is that the paper simply hides all this and says that under identical conditions they outperform Rippel, which is likely false.
Regardless, let me reaffirm my recommendation that this paper should not be accepted.
Let us clarify the story.
In the first place, we can interpret Rippel's training strategy by two ways: (1) 3-epoch pretraining on ImageNet and (2) 3-epoch finetuning on each dataset after full pretraining on ImageNet.
Although we have not get Rippel's reply for about a month, we examined which interpretation is correct by our reproductive experiments.
We managed to reimplement Magnet loss and we can reproduce Rippel's results by the latter interpretation and can not reproduce by the former interpretation. This duplication suggests that the latter interpretation is right.
For the softmax results in Rippel's paper, no one can identify the cause of his worse results than ours, just because he does not describe the training strategies, such as learning rate, in his paper;
thus, the softmax results in Rippel's paper are not contradictions.

Meta-review:
role: PC
data: comment: The paper aims to compare the representations learnt by metric learning and classification objectives. While this is an interesting topic, the presented evaluation is not sufficiently clear for the paper to be accepted.

Decision: Reject