Title: SIGNIFICANCE OF SOFTMAX-BASED FEATURES OVER METRIC LEARNING-BASED FEATURES

Abstract: The extraction of useful deep features is important for many computer vision tasks. Deep features extracted from classification networks have proved to perform well in those tasks. To obtain features of greater usefulness, end-to-end distance metric learning (DML) has been applied to train the feature extractor directly. End-to-end DML approaches such as Magnet Loss and lifted structured feature embedding show state-of-the-art performance in several image recognition tasks. However, in these DML studies, there were no equitable comparisons between features extracted from a DML-based network and those from a softmaxbased network. In this paper, by presenting objective comparisons between these two approaches under the same network architecture, we show that the softmaxbased features are markedly better than the state-of-the-art DML features for tasks such as fine-grained recognition, attribute estimation, clustering, and retrieval.

1. INTRODUCTION:

Recent developments in deep convolutional neural networks have made it possible to classify many classes of images with high accuracy. It has also been shown that such classification networks work well as feature extractors. Features extracted from classification networks show excellent performance in image classification (Donahue et al., 2014), detection, and retrieval (Razavian et al., 2014; Liu et al., 2015) , even when they have been trained to classify 1000 classes of the ImageNet dataset (Russakovsky et al., 2015) . It has also been shown that fine-tuning for target domains further improves the features' performance (Wan et al., 2014; Babenko et al., 2014) .

On the other hand, distance metric learning (DML) approaches have recently attracted considerable attention. These obtain a feature space in which distance corresponds to class similarity; it is not a byproduct of the classification network. End-to-end distance metric learning is a typical approach to constructing a feature extractor using convolutional neural networks and has been the focus of numerous studies (Bell & Bala, 2015; Schroff et al., 2015) . Some DML methods have been reported to show state-of-the-art performance in fine-grained classification (Rippel et al., 2016) and clustering and retrieval (Song et al., 2016) contexts.

However, there have been few experiments comparing softmax-based feature extraction with DMLbased feature extraction under the same network architecture or with adequate fine-tuning. An analysis providing a true comparison of DML features and softmax-based features is long overdue. As we explain more fully in the following section, we contend that there is no reason that DML, which learns feature embedding explicitly, should outperform a softmax-based feature extractor.

Fig. 1 depicts the feature vectors extracted from a softmax-based classification network and a metric learning-based network. We used LeNet architecture for both networks, and trained on the MNIST dataset (LeCun et al., 1998) . For DML, we used the contrastive loss function (Hadsell et al., 2006) to map images in two-dimensional space. For softmax-based classification, we added a two-or three-dimensional fully connected layer before the output layer for visualization. DML succeeds in learning feature embedding (Fig. 1a ). Softmax-based classification networks can also achieve a result very similar to that obtained by DML: Images are located near one another if they belong to the same class and far apart otherwise (Fig. 1b , Fig. 1c ). Our contributions in this paper are as follows:

• We show methods to exploit the ability of deep features extracted from softmax-based networks, such as normalization and proper dimensionality reduction. This is not technically novel, but this must be useful for fair comparison between image representations.

• We demonstrate that deep features extracted from softmax-based classification networks show markedly better results on fine-grained classification, attribute estimation, clustering, and retrieval tasks than those from DML-based networks in almost all datasets.

• We show that DML-based methods offer performance competitive to softmax-based methods only when the training dataset consists of a very small number of samples per class.

2. BACKGROUND:

2.1 PREVIOUS WORK

2.1.1. SOFTMAX-BASED CLASSIFICATION AND REPURPOSING OF THE CLASSIFIER AS A FEATURE EXTRACTOR:

Convolutional neural networks have demonstrated great potential for highly accurate image recognition (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; Szegedy et al., 2015; He et al., 2016) . It has been shown that features extracted from classification networks can be repurposed as a good feature representation for novel tasks (Donahue et al., 2014; Razavian et al., 2014; Qian et al., 2015) even if the network was trained on ImageNet (Russakovsky et al., 2015) . For obtaining better feature representations, fine-tuning is also effective (Babenko et al., 2014) .

2.1.2. DEEP DISTANCE METRIC LEARNING:

Distance metric learning (DML), which learns a distance metric, has been widely studied (Bromley et al., 1994; Chopra et al., 2005; Chechik et al., 2010; Qian et al., 2015) . Recent studies have focused on end-to-end deep distance metric learning (Bell & Bala, 2015; Schroff et al., 2015; Li et al., 2015; Rippel et al., 2016; Song et al., 2016) . However, in most studies comparisons of end-to-end DML with features extracted from classification networks have not been performed using architectures and conditions suited to enable a true comparison of performance. Bell & Bala (2015) compared classification networks and siamese networks, but they used coarse class labels for classification networks and fine labels for siamese networks; thus, it was left unclear whether siamese networks are better for feature-embedding learning than classification networks. Schroff et al. (2015) used triplet loss for deep metric learning in their FaceNet, which showed performance that was state of the art at the time, but their network was deeper than that of the previous method (Taigman et al., 2014) ; thus, triplet loss might not have been the only reason for the performance improvement, and the contribution from adopting triplet loss remains uncertain. Rippel et al. (2016) used the Magnet Softmax Siamese iter 1

(1,0,0) (0,1,0) (0,0,1)

(1,0,0) (0,1,0) (0,0,1)

(1,0,0) (0,1,0) (0,0,1)

(1,0,0) (0,1,0) (0,0,1) iter 2 iter 3 iter 4 ... ... For softmax, the gradient is defined by the distance between a sample and a fixed one-hot vector, and for siamese by the distance between samples.

Loss function for their DML. They tried softmax-based features as a comparison, but their results are unfairly low from our results as shown in Section 4.2 and 4.3. Song et al. (2016) used lifted structured feature embedding, another state-of-the-art DML method; however, they only compared their method with a softmax-based classification network pretrained on ImageNet (Russakovsky et al., 2015) and did not compare it with a fine-tuned network.

2.2. DIFFERENCES BETWEEN SOFTMAX-BASED CLASSIFICATION AND METRIC LEARNING:

For classification, the softmax function (Eq. 1) is typically used:

EQUATION

where p c denotes the probability that the vector u belongs to the class c. The loss of the softmax function is defined by the cross-entropy

EQUATION

where q is a one-hot encoding of the correct class of u. To minimize the cross-entropy loss, networks are trained to make the output vector u close to its corresponding one-hot vector. It is important to note that the target vectors (the correct outputs of the network) are fixed during the entire training (Fig. 2 ).

On the other hand, DML methods use distance between samples. They do not use the values of the labels; rather, they ascertain whether the labels are the same between target samples. For example, contrastive loss Hadsell et al. (2006) considers the distance d between a pair of samples:

EQUATION

where α represents the margin and q ∈ {0, 1} indicates whether the images in a pair are in the same class (1) or not (0). Recent studies (Schroff et al., 2015; Rippel et al., 2016; Song et al., 2016) use pairwise distances between three or more images at the same time for fast convergence and efficient calculation. However, these methods have some drawbacks. DML methods sometimes require complicated operations such as hard negative sampling (Schroff et al., 2015; Rippel et al., 2016) and k-means clustering for every epoch (Rippel et al., 2016) . For DML, in contrast to optimization of the softmax cross-entropy loss, the optimization targets are not always consistent during training even if all possible distances within the mini-batch are considered. Thus, the DML optimization converges very slowly and is not stable and unsteadily. An additional problem is that methods for sampling positive pairs and negative pairs have not been established.

3.1. DIMENSIONALITY REDUCTION LAYER:

One of DML's strength in using fine-tuning is the flexibility of its output dimensionality. When using features of a mid-layer of a softmax classification network, on the other hand, the dimensionality of the features is fixed. Some existing methods (Babenko et al., 2014) use PCA or discriminative dimensionality reduction to reduce the number of feature dimensions. In our experiment, we evaluated three methods for changing the feature dimensionality. Following conventional PCA approaches, we extracted features from a 1024-dimensional pool5 layer of GoogLeNet (Szegedy et al., 2015; Ioffe & Szegedy, 2015) (Fig. 3a ) and applied PCA to reduce the dimensionality. In a contrasting approach, we made use of a fully connected layer: We added a fully connected layer having the required number of neurons just before the output layer (FCR 1, Fig. 3b ). We also investigated a third approach in which a fully connected layer is added followed by a dropout layer (FCR 2, Fig. 3c ). We intend to show that the features extracted from the pool5 layer of FCR 2 provide better performance than those from FCR 1 even though they differ only in the positions of their dropout layers.

3.2. NORMALIZATION:

In this study, all the features extracted from the classification networks were from the last layer before the last output layer. The outputs were normalized by the softmax function and then evaluated by the cross-entropy loss function in the networks. Assume that the output vector is p = {p i | i p i = 1}. For arbitrary positive constant α, y = {log αp i } returns the same vector p after the softmax function is applied. The features x we extract from the networks are given as x = W -1 y, where W denotes the linear projection matrix from the layer before the output layer to the output layer. The vector y has an ambiguity in its scale, thus vector x, a linear transform of y, also has an ambiguity in the scale; therefore x should be normalized. As Fig. 1b clearly indicates, the distance between features extracted from a softmax-based classifier should be evaluated by cosine similarity, not by the Euclidean distance.

Some studies used L2 normalization for deep features extracted from softmax-based classification networks (Taigman et al., 2014) , whereas many recent studies have used the features without any normalization (Krizhevsky et al., 2012; Rippel et al., 2016; Song et al., 2016; Wei et al., 2016) . In this study, we also planned to validate the efficiency of normalizing deep features.

4. EXPERIMENTS:

In this section, we compare the deep features extracted from classification networks to those reported from state-of-the-art deep metric learning methods (Rippel et al., 2016; Song et al., 2016) in their performance on several tasks.

4.1. PROCEDURE:

All our networks were fine-tuned from the weights that were pretrained on ImageNet (Russakovsky et al., 2015) . To evaluate fine-grained classification and attribute estimation performances, we used GoogLeNet with batch normalization (Ioffe & Szegedy, 2015) and did not use any dimentionality reduction layers described in Section 3.1. To evaluate clustering and retrieval performances we used GoogLeNet without batch normalization (Szegedy et al., 2015) and dimentionality reduction layers. We used the Caffe (Jia et al., 2014) framework for our experiments.

4.2. FINE-GRAINED CLASSIFICATION:

For the evaluation of deep features in fine-grained classification tasks, we used three image datasets: Stanford Dogs (Khosla et al., 2011) , Oxford 102 Flowers (Nilsback & Zisserman, 2008) , and Oxford-IIIT Pet (Parkhi et al., 2012) . For the softmax-base method we fine-tuned the classifier from weights that were pretrained on ImageNet. We defined the learning rate using validation data, setting the learning rate to 0.0001 for the Stanford Dogs dataset and the Oxford-IIIT Pet dataset and to 0.001 for the Oxford 102 Flowers dataset. Learning rates were not changed during the training. We rescaled all the input up by 30% and randomly cropped 224 × 224. These strategies are exactly the same as those of the previous method (Rippel et al., 2016) .

We show the mean error rates for the three datasets in Table 1 . All our results were evaluated using a 1-nearest neighbor search of the 1024-dimensional vectors extracted from the pool5 layer of GoogLeNet with batch normalization (Ioffe & Szegedy, 2015) . In all the experiments, the features extracted from the fine-tuned classification network show the best fine-grained classification performance. Our results of softmax-based classification are better than the results in Rippel et al. (2016) .

The experiments of softmax-based classification in Rippel et al. (2016) were not the best.

4.3. ATTRIBUTE ESTIMATION:

Rippel et al. ( 2016) evaluated features' expressiveness using mean attribute precision and showed that the features generated by their proposed method contain intra-class diversity. In this section, we investigate the intra-class diversity of softmax features. We use the ImageNet Attribute dataset (Rippel et al., 2016) , which consists of overlap between the ImageNet training set (Russakovsky et al., 2015) and the Object Attribute dataset (Russakovsky & Fei-Fei, 2010) . We used only the images and their class labels during our training of the softmax classifier and did not use attributes.

Table 2 shows the error rates of 90-way classification under different training methods. Our finetuned softmax classifier outperformed those of Rippel et al. (2016) by a considerable margin. Fig. 4 shows the mean attribute precision for the ImageNet Attribute dataset. Our fine-tuned softmax features markedly outperformed those from Rippel et al. (2016) . These results implicitly indicate that the features extracted from the pool5 layer contain intra-class diversity that is better than those from DML networks designed to keep intra-class diversity. 

4.4. CLUSTERING AND RETRIEVAL:

Here, we give our evaluation of clustering and retrieval scores for the state-of-the-art DML method (Song et al., 2016) and for the softmax classification networks. We used the Caltech UCSD Birds 200-2011 (CUB) dataset (Wah et al., 2011) , the Stanford Cars 196 (CAR) dataset (Krause et al., 2013) , and the Stanford Online Products (OP) dataset (Song et al., 2016) . For CUB and CAR, we used the first half of the dataset classes for training and the rest for testing. For OP, we used the training-testing class split provided. The dataset properties are shown in Table 3 . We emphasize that the class sets used for training and testing are completely different. We multiplied the learning rates of the changed layers (output layers for all models and the fully connected layer added for FCR 1 and FCR 2) by 10. The batch size was set to 128, and the maximum number of iterations for our training was set to 20,000. These training strategies are exactly the same as those used in the earlier study (Song et al., 2016) .

For clustering evaluation, we applied k-means clustering 100 times and calculated the average standard F 1 and NMI (Manning et al., 2008) ; the value for k was set to the number of classes in the test set. For retrieval evaluation, we used the Recall@K metric (Jegou et al., 2011) .

We show the results for the CUB dataset in Fig. 5 and for the CAR dataset in Fig. 6 . We notice that we have been able to reproduce nearly exactly the scores of lifted structured feature embedding (Song et al., 2016) . However, the deep features extracted from the softmax-based classification networks outperformed the lifted structured feature embedding in all the evaluation metrics.

For F 1 and NMI, all of the softmax models, including PCA, FCR 1, and FCR 2, show markedly better scores than does lifted structured feature embedding. It is clear that L2 normalization improves the scores of all the softmax-based models. The scores of PCA and FCR 1 drop slightly as the feature dimensionality decreases from 1024 for both the CUB dataset and the CAR dataset. On the other hand, FCR 2, which has a fully connected layer followed by a dropout layer, improves the scores in spite of the reduction in dimensionality, as shown in Fig. 6 . It may be that 1024 dimensions is too large to describe the image classes. This result may imply that to obtain the best features we need to first determine the optimum dimensionality of the feature space for the dataset and then apply PCA.

For the Recall@K metric, we used 1024-dimensional features for the CUB dataset and 256dimensional features for the CAR dataset. The softmax-based features outperformed the DMLbased features. The differences between PCA, FCR 1, and FCR 2 are very minor. Regarding feature normalization, features without normalization show worse scores than do L2-normalized features.

Fig. 7 shows the standard F 1 , NMI, and Recall@K for the Online Products dataset. We used 1024dimensional features for the Recall@K metric. As shown in Table 3 , the OP dataset is very different from the CUB and CAR datasets in terms of the number of classes and the number of samples per class; the number of samples per class in the OP dataset is limited to 5.3 on average. In contrast to CUB and CAR, in the OP dataset the scores for softmax and for lifted structured feature embedding are nearly the same.

From the results for these three datasets, we conjecture that the number of images contained in the dataset has a considerable effect on softmax-based classification. In other words, it is difficult for DML to make use of the rich information from a large number of samples because of the randomness described in the previous section. Hence, we changed the size of datasets by subsampling the images of CUB and CAR datasets for each class and ran the experiments again. We constructed seven datasets of different sizes, containing 5, 10, 20, 40, 60, 80, and 100 %, respectively, of the whole dataset. As shown in Fig. 8 and Fig. 9 , the difference between the scores for softmax and DML is small or close to zero if the size of the training dataset is small. The gap between softmax and DML becomes larger as the dataset size increases. It is surprising that the scores of lifted structured feature embedding on the CUB dataset did not increase even though we used more images for the training (Fig. 8 ). It can be said that DML cannot exploit large training datasets, whereas the softmax-based classifier can obtain features of high expressiveness.

5. CONCLUSION:

Because there was no equitable comparison in previous studies, we conducted comparisons of the softmax-based classifier and DML methods using a design that would enable the methods to objectively demonstrate their true performance capabilities. Our results show that the features extracted from softmax-based classifiers perform better than those from state-of-the-art DML methods (Rippel et al., 2016; Song et al., 2016) on fine-grained classification, clustering, and retrieval tasks, especially when the size of the training dataset is large. The experimental results also show that softmax-based features exhibit rich intra-class diversity even though the softmax classifier is not explicitly designed to do so, unlike to the previous method (Rippel et al., 2016) . It is obvious that the softmax-based features are still strong baselines. We hope that softmax-based features are taken into account when evaluating the performance of deep features.

Limitations. When the number of classes are huge, it is hard to train classification networks due to GPU memory constraints. DML-based methods are suitable for such cases because they do not need the output layer which is proportional to the number of classes. For cross-domain tasks, such as sketches to photos (Yu et al., 2016; Sangkloy et al., 2016) or aerial views to ground views (Lin et al., 2015) , DML is also effective. Classification-based learning needs complicated learning strategies like in Castrejon et al. (2016) . DML-based methods can learn cross-domain representation only by using a pair of networks.

:



Figures and Tables:

Figure 1: Depiction of MNIST dataset. (a) Two-dimensional features obtained by siamese network. (b) Two-dimensional features extracted from softmax-based classifier; these features are well separated by angle but not by Euclidean norm. (c) Three-dimensional features extracted from softmaxbased classifier; we normalized these to have unit L2 norm and depict them in an azimuth-elevation coordinate system. The three-dimensional features are well separated by their classes.

Figure2: Illustration of learning processes for softmax-based classification network and siamesebased DML network. For softmax, the gradient is defined by the distance between a sample and a fixed one-hot vector, and for siamese by the distance between samples.

Figure 3: GoogLeNet Szegedy et al. (2015) architecture we use in this paper. We extracted the features from the red-colored layers. For (a), we applied PCA to reduce the number of feature dimensions. For (b) and (c), the dimensionality is already reduced to the required number by the fc reduction layer.

Figure 4: Mean attribute precision for the ImageNet Attribute dataset.

Figure 7: F 1 , NMI, and Recall@K scores for the test set of the Online Products dataset.

Figure 9: F 1 , NMI, and Recall@K scores for test set of the Stanford Cars 196 dataset under different dataset sizes. The feature dimensionality is fixed at 256.

Error rates for various fine-grained image datasets.

Table content: <table><tr><td/><td/><td colspan="2">(b) Oxford 102 Flowers.</td><td>(c) Oxford -IIIT Pet.</td><td/></tr><tr><td>Approach</td><td>Error</td><td>Approach</td><td>Error</td><td>Approach</td><td>Error</td></tr><tr><td>Angelova &amp; Long (2014)</td><td>51.7%</td><td>Angelova &amp; Zhu (2013)</td><td>23.3%</td><td>Angelova &amp; Zhu (2013)</td><td>49.2%</td></tr><tr><td>Xie et al. (2015)</td><td>50.6%</td><td>Angelova &amp; Long (2014)</td><td>19.6%</td><td>Parkhi et al. (2012)</td><td>46.0%</td></tr><tr><td>Gavves et al. (2013)</td><td>49.9%</td><td>Murray &amp; Perronnin (2014)</td><td>15.4%</td><td>Angelova &amp; Long (2014)</td><td>44.6%</td></tr><tr><td>Gavves et al. (2015)</td><td>43.0%</td><td>Razavian et al. (2014)</td><td>13.2%</td><td>Murray &amp; Perronnin (2014)</td><td>43.2%</td></tr><tr><td>Qian et al. (2015)</td><td>30.9%</td><td>Qian et al. (2015)</td><td>11.6%</td><td>Qian et al. (2015)</td><td>19.6%</td></tr><tr><td colspan="2">Rippel et al. (2016) (Softmax prob) 26.6%</td><td colspan="2">Rippel et al. (2016) (Softmax prob) 11.2%</td><td colspan="2">Rippel et al. (2016) (Softmax prob) 11.3%</td></tr><tr><td>Rippel et al. (2016) (Triplet)</td><td>35.8%</td><td>Rippel et al. (2016) (Triplet)</td><td>17.0%</td><td>Rippel et al. (2016) (Triplet)</td><td>13.5%</td></tr><tr><td>Rippel et al. (2016) (Magnet)</td><td>24.9%</td><td>Rippel et al. (2016) (Magnet)</td><td>8.6%</td><td>Rippel et al. (2016) (Magnet)</td><td>10.6%</td></tr><tr><td>Ours (Softmax prob)</td><td>18.3%</td><td>Ours (Softmax prob)</td><td>8.69%</td><td>Ours (Softmax prob)</td><td>9.04%</td></tr><tr><td>Ours (Softmax pool5)</td><td>21.0%</td><td>Ours (Softmax pool5)</td><td>7.90%</td><td>Ours (Softmax pool5)</td><td>9.09%</td></tr><tr><td>Ours (Softmax pool5 + L2)</td><td>20.3%</td><td>Ours (Softmax pool5 + L2)</td><td>7.09%</td><td>Ours (Softmax pool5 + L2)</td><td>9.04%</td></tr></table>

Classification error rates for the ImageNet Attribute dataset.

Table content: <table><tr><td/><td/><td/><td>1</td><td>Rippel (Softmax prob)</td></tr><tr><td/><td/><td>precision</td><td>0.8</td><td>Rippel (Triplet) Rippel (Magnet) Ours (Softmax pool5 + L2)</td></tr><tr><td/><td/><td>attribute</td><td>0.6</td><td/></tr><tr><td>Approach</td><td>Error</td><td>Mean</td><td>0.4</td><td/></tr><tr><td colspan="2">Rippel et al. (2016) (Softmax prob) 14.1%</td><td/><td/><td/></tr><tr><td>Rippel et al. (2016) (Triplet) Rippel et al. (2016) (Magnet)</td><td>26.8% 15.9%</td><td/><td>2 0 0.2</td><td>Neighbourhood size 2 2 2 4 2 6</td><td>2 8</td></tr><tr><td>Ours (Softmax prob)</td><td>7.68%</td><td/><td/><td/></tr><tr><td>Ours (Softmax pool5)</td><td>11.9%</td><td/><td/><td/></tr><tr><td>Ours (Softmax pool5 + L2)</td><td>10.7%</td><td/><td/><td/></tr></table>



Table content: <table><tr><td/><td colspan="3">: Properties of datasets</td></tr><tr><td colspan="4">used in Section 4.4. Each cell</td></tr><tr><td colspan="4">shows the number of images</td></tr><tr><td colspan="4">(upper figure) and the number</td></tr><tr><td colspan="4">of classes (lower figure).</td></tr><tr><td>Dataset</td><td>Train</td><td>Test</td><td>Total</td></tr><tr><td>CUB</td><td>5,864 100</td><td>5,924 100</td><td>11,788 200</td></tr><tr><td>CAR</td><td>8,054 98</td><td>8,131 98</td><td>16,185 196</td></tr><tr><td>OP</td><td colspan="3">59,551 60,502 120,053 11,318 11,316 22,634</td></tr></table>

F 1 , NMI, and Recall@K scores for the test set of the Stanford Cars 196 dataset.

Table content: <table><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>1</td><td/><td/><td/></tr><tr><td/><td>0.3</td><td/><td/><td/><td/><td/><td>0.62</td><td/><td/><td/><td/><td/><td/><td>0.9</td><td/><td/><td/></tr><tr><td/><td>0.28</td><td/><td/><td/><td/><td/><td>0.61</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>F 1 score</td><td>64 0.18 0.2 0.22 0.24 0.26</td><td>128</td><td>256 PCA + L2 PCA FCR 1 + L2 FCR 1</td><td colspan="2">512 FCR 2 + L2 1024 FCR 2 Lifted struct</td><td>NMI score</td><td>64 0.55 0.57 0.58 0.59 0.6 0.56</td><td>128</td><td>256 PCA + L2 PCA FCR 1 + L2 FCR 1</td><td colspan="3">512 FCR 2 + L2 1024 FCR 2 Lifted struct</td><td>Recall@K</td><td>0.4 0.7 0.8 0.6 0.5</td><td>1</td><td>2</td><td>4</td><td>8</td><td>16 PCA + L2 PCA FCR 1 + L2 FCR 1 FCR 2 + L2 FCR 2 Lifted struct</td><td>32</td></tr><tr><td/><td/><td colspan="2">Embedding size</td><td/><td/><td/><td/><td colspan="2">Embedding size</td><td/><td/><td/><td/><td/><td/><td/><td>K</td></tr><tr><td colspan="19">Figure 5: F 1 , NMI, and Recall@K scores for the test set of the Caltech UCSD Birds 200-2011</td></tr><tr><td colspan="2">dataset.</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0.3</td><td/><td/><td/><td/><td/><td>0.64</td><td/><td/><td/><td/><td/><td/><td>1</td><td/><td/><td/></tr><tr><td/><td>0.28</td><td/><td/><td/><td/><td/><td>0.62</td><td/><td/><td/><td/><td/><td/><td>0.9</td><td/><td/><td/></tr><tr><td>F 1 score</td><td>0.2 0.22 0.24 0.26</td><td>PCA + L2 PCA FCR 1 + L2 FCR 1</td><td colspan="2">FCR 2 + L2 FCR 2 Lifted struct</td><td/><td>score NMI</td><td>0.54 0.56 0.58 0.6</td><td>PCA + L2 PCA FCR 1 + L2 FCR 1</td><td>FCR 2 + L2 FCR 2 Lifted struct</td><td/><td/><td/><td>Recall@K</td><td>0.7 0.8 0.6</td><td/><td/><td/><td>PCA + L2 PCA FCR 1 + L2 FCR 1 FCR 2 + L2 FCR 2 Lifted struct</td></tr><tr><td/><td>64 0.18</td><td>128</td><td>256</td><td>512</td><td>1024</td><td/><td>64 0.52</td><td>128</td><td>256</td><td colspan="2">512</td><td>1024</td><td/><td>0.5</td><td>1</td><td>2</td><td>4</td><td>8</td><td>16</td><td>32</td></tr><tr><td/><td/><td colspan="2">Embedding size</td><td/><td/><td/><td/><td colspan="2">Embedding size</td><td/><td/><td/><td/><td/><td/><td/><td>K</td></tr><tr><td>F 1 score</td><td colspan="2">128 Figure 6: 64 0.14 0.26 0.28 0.16 0.18 0.2 0.22 0.24 PCA + L2 PCA FCR 1 + L2 FCR 1</td><td>256 FCR 2 + L2 FCR 2 Lifted struct</td><td>512</td><td>1024</td><td>NMI score</td><td>64 0.85 0.875 0.88 0.855 0.86 0.865 0.87</td><td>128 PCA + L2 PCA FCR 1 + L2 FCR 1</td><td colspan="2">256 FCR 2 + L2 FCR 2 Lifted struct</td><td>512</td><td>1024</td><td>Recall@K</td><td colspan="2">10 0 0.9 1 0.7 0.8 0.6</td><td>10 1</td><td/><td>10 2</td><td>PCA + L2 PCA FCR 1 + L2 FCR 1 FCR 2 + L2 FCR 2 Lifted struct</td><td>10 3</td></tr><tr><td/><td/><td colspan="2">Embedding size</td><td/><td/><td/><td/><td colspan="2">Embedding size</td><td/><td/><td/><td/><td/><td/><td/><td>K</td></tr></table>

F 1 , NMI, and Recall@K scores for test set of the Caltech UCSD Birds 200-2011 dataset under different dataset sizes. The feature dimensionality is fixed at 1024.

Table content: <table><tr><td/><td>0.32</td><td/><td/><td/><td/><td/><td/><td>0.64</td><td/><td/><td/><td/><td/><td/><td colspan="2">0.56</td><td/><td/></tr><tr><td/><td>0.3</td><td/><td/><td/><td/><td/><td/><td>0.62</td><td/><td/><td/><td/><td/><td/><td colspan="2">0.54</td><td/><td/></tr><tr><td>F 1 score</td><td>0.22 0.24 0.26 0.28</td><td/><td/><td/><td>FCR 2 + L2 Lifted struct</td><td/><td>NMI score</td><td>0.56 0.58 0.6</td><td/><td/><td/><td>FCR 2 + L2 Lifted struct</td><td/><td>Recall@1</td><td colspan="2">0.46 0.52 0.48 0.5</td><td/><td/><td>FCR 2 + L2 Lifted struct</td></tr><tr><td/><td>0.2</td><td/><td/><td/><td/><td/><td/><td>0.54</td><td/><td/><td/><td/><td/><td/><td colspan="2">0.44</td><td/><td/></tr><tr><td/><td>0.18</td><td>5 10 20</td><td>40</td><td>60</td><td>80</td><td>100</td><td/><td>0.52</td><td>5 10 20</td><td>40</td><td>60</td><td>80</td><td>100</td><td/><td colspan="2">0.42</td><td>5 10 20</td><td>40</td><td>60</td><td>80</td><td>100</td></tr><tr><td/><td/><td colspan="4">Training dataset size (%)</td><td/><td/><td/><td colspan="4">Training dataset size (%)</td><td/><td/><td/><td/><td colspan="3">Training dataset size (%)</td></tr><tr><td colspan="3">0.1 0.25 0.3 Figure 8: 5 10 20 0.15 0.2 F 1 score</td><td>40</td><td>60</td><td>80 Lifted struct FCR 2 + L2</td><td>100</td><td>NMI score</td><td>0.35 0.6 0.65 0.4 0.45 0.5 0.55</td><td>5 10 20</td><td>40</td><td>60</td><td>80 Lifted struct FCR 2 + L2</td><td>100</td><td colspan="2">Recall@1</td><td>0.3 0.7 0.8 0.4 0.5 0.6</td><td>5 10 20</td><td>40</td><td>60</td><td>80 FCR 2 + L2 Lifted struct</td><td>100</td></tr><tr><td/><td/><td colspan="4">Training dataset size (%)</td><td/><td/><td/><td colspan="4">Training dataset size (%)</td><td/><td/><td/><td/><td colspan="3">Training dataset size (%)</td></tr></table>

Peer review (iclr2017):
There has been substantial recent interest in representation learning, and specifically, using distance metric learning (DML) to learn representations where semantic distance between inputs can be measured. This is a topic of particular relevance / interest to ICLR. This paper poses a simple yet provocative question: can a standard SoftMax based approach learn features that match or even outperform recent state-of-the-art DML approaches? Thorough experiments seem to indicate that this is indeed the case. Comparisons are made to recent DML approaches including Magnet Loss (ICLR2016) and Lifted structure embedding (CVPR2016) and superior results are shown across a number of datasets / tasks for which the DML approaches were designed. This main result is a bit surprising since SoftMax is a natural and trivial baseline, so it should have been properly evaluated in previous DML literature. The authors argue that previous approaches did not fully/properly tune the softmax baselines, or that comparisons were not apples-to-apples. Also, one change in the current paper is the addition of L2 normalization, which is well motivated and helps improve SoftMax feature distances. Different dimensionality reduction approaches are also tested. These changes are minor, but especially the L2 normalization proves to be a simple but effective improvement for SoftMax features. A big issue is with how pre-training is performed (in Magnet Loss the softmax baselines were pretrained for less time on ImageNet). The approach taken here is reasonable, but so is the approach in Magnet Loss (for different reasons). Ultimately, both are fine. Unfortunately, due to use of different schemes, the results are not comparable. Let me copy-paste what I wrote in an earlier comment: My main concern about the paper is that the comparisons in Tables 1 and 2 and Figure 4 to Rippel et al. are not apples-to-apples. Basically, the papers shows that absolute results of using SoftMax w full pre-training (PT) on ImageNet is superior to any of the results in Rippel*s paper (including both the Softmax and Magnet results). But as the current results show, PT appears to be critical to obtaining such good numbers - The Rippel SoftMax numbers use only 3 PT stages and are dramatically worse than the full PT on ImageNet. As it stands, I am not convinced that SoftMax is actually better than Magnet. Here is the evidence we have (I*ll use Stanford Dogs as an example, but any of the datasets have the same conclusion): (1) Softmax w 3 stages of PT: 26.6% (from Rippel paper) and 32.7% (from authors* reproduction) (2) Magnet w 3 stages of PT: 24.9% (3) Softmax w full PT: 18.3% (4) Magnet w full PT: not shown From this all I see is that PT is critical for getting absolute good results. However, what about Magnet w full PT? These results are not shown either here or in the original Rippel paper (I went back and looked). As such, I do not think it is justifiable to claim superiority of Softmax to Magnet based on available evidence. (Note: I looked back carefully at Rippel*s paper, and it appears that the authors use 3 PT stages as a form of *warmup*. There is a statement that using full PT would *defeat the purpose of pursuing DML*. I*m not sure if I agree w Rippel*s statement since in the present paper there is clear evidence that full PT is hugely helpful, at least for softmax. That being said, I did not see any evidence in the Rippel paper that PT is harmful or that DML wouldn*t work with full PT.) The authors responded to my concern by claiming that “from Rippel*s results, it is no doubt that Magnet@3epochPT > Magnet@FullPT and Magnet@3epoch > Softmax@3epochPT.” However, I went back to Rippel’s paper, and simply the Magnet@FullPT experiment never appears. I further went and contacted Oren Rippel himself, and he verified he never ran the Magnet@FullPT experiment. I encourage the authors to contact Oren Rippel regarding this if they wish to verify (I have asked Oren Rippel to not reveal my identity). [Disclaimer: I am NOT Oren Rippel]. The authors mentioned that they are training Magnet@FullPT. If results were shown for Magnet@FullPT and also retrain the Magnet@3epochPT as a sanity check, that would help alleviate this concern. Alternatively, the language in Section 4 and the Tables could be altered to make clear that the methods use different pretraining and hence are not comparable. Overall, I am actually quite sympathetic to this work. I think it could serve as an important sanity-check paper for the community and quite relevant to ICLR. Having proper and strong SoftMax baselines should prove quite useful to the DML community and to this line of work. However, currently I find the main results (table 1, table 2, figure 4, etc.) to be misleading. If indeed it were the case that Magnet@3epochPT > Magnet@FullPT, then it would be fine. However, at this point as far as I know no one has actually tried Magnet@FullPT. And, given the general importance and effectiveness of pre-training, especially when transferring to small dataset, I would be hugely surprised if Magnet@FullPT was not superior by a large margin. I think either having this experiment in place or altering the writing / presentation of the results would be critical to allow for publishing.