Title: SIGNIFICANCE OF SOFTMAX-BASED FEATURES OVER METRIC LEARNING-BASED FEATURES

Abstract: The extraction of useful deep features is important for many computer vision tasks. Deep features extracted from classification networks have proved to perform well in those tasks. To obtain features of greater usefulness, end-to-end distance metric learning (DML) has been applied to train the feature extractor directly. End-to-end DML approaches such as Magnet Loss and lifted structured feature embedding show state-of-the-art performance in several image recognition tasks. However, in these DML studies, there were no equitable comparisons between features extracted from a DML-based network and those from a softmaxbased network. In this paper, by presenting objective comparisons between these two approaches under the same network architecture, we show that the softmaxbased features are markedly better than the state-of-the-art DML features for tasks such as fine-grained recognition, attribute estimation, clustering, and retrieval.

1. INTRODUCTION:

Recent developments in deep convolutional neural networks have made it possible to classify many classes of images with high accuracy. It has also been shown that such classification networks work well as feature extractors. Features extracted from classification networks show excellent performance in image classification (Donahue et al., 2014), detection, and retrieval (Razavian et al., 2014; Liu et al., 2015) , even when they have been trained to classify 1000 classes of the ImageNet dataset (Russakovsky et al., 2015) . It has also been shown that fine-tuning for target domains further improves the features' performance (Wan et al., 2014; Babenko et al., 2014) .

On the other hand, distance metric learning (DML) approaches have recently attracted considerable attention. These obtain a feature space in which distance corresponds to class similarity; it is not a byproduct of the classification network. End-to-end distance metric learning is a typical approach to constructing a feature extractor using convolutional neural networks and has been the focus of numerous studies (Bell & Bala, 2015; Schroff et al., 2015) . Some DML methods have been reported to show state-of-the-art performance in fine-grained classification (Rippel et al., 2016) and clustering and retrieval (Song et al., 2016) contexts.

However, there have been few experiments comparing softmax-based feature extraction with DMLbased feature extraction under the same network architecture or with adequate fine-tuning. An analysis providing a true comparison of DML features and softmax-based features is long overdue. As we explain more fully in the following section, we contend that there is no reason that DML, which learns feature embedding explicitly, should outperform a softmax-based feature extractor.

Fig. 1 depicts the feature vectors extracted from a softmax-based classification network and a metric learning-based network. We used LeNet architecture for both networks, and trained on the MNIST dataset (LeCun et al., 1998) . For DML, we used the contrastive loss function (Hadsell et al., 2006) to map images in two-dimensional space. For softmax-based classification, we added a two-or three-dimensional fully connected layer before the output layer for visualization. DML succeeds in learning feature embedding (Fig. 1a ). Softmax-based classification networks can also achieve a result very similar to that obtained by DML: Images are located near one another if they belong to the same class and far apart otherwise (Fig. 1b , Fig. 1c ). Our contributions in this paper are as follows:

• We show methods to exploit the ability of deep features extracted from softmax-based networks, such as normalization and proper dimensionality reduction. This is not technically novel, but this must be useful for fair comparison between image representations.

• We demonstrate that deep features extracted from softmax-based classification networks show markedly better results on fine-grained classification, attribute estimation, clustering, and retrieval tasks than those from DML-based networks in almost all datasets.

• We show that DML-based methods offer performance competitive to softmax-based methods only when the training dataset consists of a very small number of samples per class.

2. BACKGROUND:

2.1 PREVIOUS WORK

2.1.1. SOFTMAX-BASED CLASSIFICATION AND REPURPOSING OF THE CLASSIFIER AS A FEATURE EXTRACTOR:

Convolutional neural networks have demonstrated great potential for highly accurate image recognition (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; Szegedy et al., 2015; He et al., 2016) . It has been shown that features extracted from classification networks can be repurposed as a good feature representation for novel tasks (Donahue et al., 2014; Razavian et al., 2014; Qian et al., 2015) even if the network was trained on ImageNet (Russakovsky et al., 2015) . For obtaining better feature representations, fine-tuning is also effective (Babenko et al., 2014) .

2.1.2. DEEP DISTANCE METRIC LEARNING:

Distance metric learning (DML), which learns a distance metric, has been widely studied (Bromley et al., 1994; Chopra et al., 2005; Chechik et al., 2010; Qian et al., 2015) . Recent studies have focused on end-to-end deep distance metric learning (Bell & Bala, 2015; Schroff et al., 2015; Li et al., 2015; Rippel et al., 2016; Song et al., 2016) . However, in most studies comparisons of end-to-end DML with features extracted from classification networks have not been performed using architectures and conditions suited to enable a true comparison of performance. Bell & Bala (2015) compared classification networks and siamese networks, but they used coarse class labels for classification networks and fine labels for siamese networks; thus, it was left unclear whether siamese networks are better for feature-embedding learning than classification networks. Schroff et al. (2015) used triplet loss for deep metric learning in their FaceNet, which showed performance that was state of the art at the time, but their network was deeper than that of the previous method (Taigman et al., 2014) ; thus, triplet loss might not have been the only reason for the performance improvement, and the contribution from adopting triplet loss remains uncertain. Rippel et al. (2016) used the Magnet Softmax Siamese iter 1

(1,0,0) (0,1,0) (0,0,1)

(1,0,0) (0,1,0) (0,0,1)

(1,0,0) (0,1,0) (0,0,1)

(1,0,0) (0,1,0) (0,0,1) iter 2 iter 3 iter 4 ... ... For softmax, the gradient is defined by the distance between a sample and a fixed one-hot vector, and for siamese by the distance between samples.

Loss function for their DML. They tried softmax-based features as a comparison, but their results are unfairly low from our results as shown in Section 4.2 and 4.3. Song et al. (2016) used lifted structured feature embedding, another state-of-the-art DML method; however, they only compared their method with a softmax-based classification network pretrained on ImageNet (Russakovsky et al., 2015) and did not compare it with a fine-tuned network.

2.2. DIFFERENCES BETWEEN SOFTMAX-BASED CLASSIFICATION AND METRIC LEARNING:

For classification, the softmax function (Eq. 1) is typically used:

EQUATION

where p c denotes the probability that the vector u belongs to the class c. The loss of the softmax function is defined by the cross-entropy

EQUATION

where q is a one-hot encoding of the correct class of u. To minimize the cross-entropy loss, networks are trained to make the output vector u close to its corresponding one-hot vector. It is important to note that the target vectors (the correct outputs of the network) are fixed during the entire training (Fig. 2 ).

On the other hand, DML methods use distance between samples. They do not use the values of the labels; rather, they ascertain whether the labels are the same between target samples. For example, contrastive loss Hadsell et al. (2006) considers the distance d between a pair of samples:

EQUATION

where α represents the margin and q ∈ {0, 1} indicates whether the images in a pair are in the same class (1) or not (0). Recent studies (Schroff et al., 2015; Rippel et al., 2016; Song et al., 2016) use pairwise distances between three or more images at the same time for fast convergence and efficient calculation. However, these methods have some drawbacks. DML methods sometimes require complicated operations such as hard negative sampling (Schroff et al., 2015; Rippel et al., 2016) and k-means clustering for every epoch (Rippel et al., 2016) . For DML, in contrast to optimization of the softmax cross-entropy loss, the optimization targets are not always consistent during training even if all possible distances within the mini-batch are considered. Thus, the DML optimization converges very slowly and is not stable and unsteadily. An additional problem is that methods for sampling positive pairs and negative pairs have not been established.

3.1. DIMENSIONALITY REDUCTION LAYER:

One of DML's strength in using fine-tuning is the flexibility of its output dimensionality. When using features of a mid-layer of a softmax classification network, on the other hand, the dimensionality of the features is fixed. Some existing methods (Babenko et al., 2014) use PCA or discriminative dimensionality reduction to reduce the number of feature dimensions. In our experiment, we evaluated three methods for changing the feature dimensionality. Following conventional PCA approaches, we extracted features from a 1024-dimensional pool5 layer of GoogLeNet (Szegedy et al., 2015; Ioffe & Szegedy, 2015) (Fig. 3a ) and applied PCA to reduce the dimensionality. In a contrasting approach, we made use of a fully connected layer: We added a fully connected layer having the required number of neurons just before the output layer (FCR 1, Fig. 3b ). We also investigated a third approach in which a fully connected layer is added followed by a dropout layer (FCR 2, Fig. 3c ). We intend to show that the features extracted from the pool5 layer of FCR 2 provide better performance than those from FCR 1 even though they differ only in the positions of their dropout layers.

3.2. NORMALIZATION:

In this study, all the features extracted from the classification networks were from the last layer before the last output layer. The outputs were normalized by the softmax function and then evaluated by the cross-entropy loss function in the networks. Assume that the output vector is p = {p i | i p i = 1}. For arbitrary positive constant α, y = {log αp i } returns the same vector p after the softmax function is applied. The features x we extract from the networks are given as x = W -1 y, where W denotes the linear projection matrix from the layer before the output layer to the output layer. The vector y has an ambiguity in its scale, thus vector x, a linear transform of y, also has an ambiguity in the scale; therefore x should be normalized. As Fig. 1b clearly indicates, the distance between features extracted from a softmax-based classifier should be evaluated by cosine similarity, not by the Euclidean distance.

Some studies used L2 normalization for deep features extracted from softmax-based classification networks (Taigman et al., 2014) , whereas many recent studies have used the features without any normalization (Krizhevsky et al., 2012; Rippel et al., 2016; Song et al., 2016; Wei et al., 2016) . In this study, we also planned to validate the efficiency of normalizing deep features.

4. EXPERIMENTS:

In this section, we compare the deep features extracted from classification networks to those reported from state-of-the-art deep metric learning methods (Rippel et al., 2016; Song et al., 2016) in their performance on several tasks.

4.1. PROCEDURE:

All our networks were fine-tuned from the weights that were pretrained on ImageNet (Russakovsky et al., 2015) . To evaluate fine-grained classification and attribute estimation performances, we used GoogLeNet with batch normalization (Ioffe & Szegedy, 2015) and did not use any dimentionality reduction layers described in Section 3.1. To evaluate clustering and retrieval performances we used GoogLeNet without batch normalization (Szegedy et al., 2015) and dimentionality reduction layers. We used the Caffe (Jia et al., 2014) framework for our experiments.

4.2. FINE-GRAINED CLASSIFICATION:

For the evaluation of deep features in fine-grained classification tasks, we used three image datasets: Stanford Dogs (Khosla et al., 2011) , Oxford 102 Flowers (Nilsback & Zisserman, 2008) , and Oxford-IIIT Pet (Parkhi et al., 2012) . For the softmax-base method we fine-tuned the classifier from weights that were pretrained on ImageNet. We defined the learning rate using validation data, setting the learning rate to 0.0001 for the Stanford Dogs dataset and the Oxford-IIIT Pet dataset and to 0.001 for the Oxford 102 Flowers dataset. Learning rates were not changed during the training. We rescaled all the input up by 30% and randomly cropped 224 × 224. These strategies are exactly the same as those of the previous method (Rippel et al., 2016) .

We show the mean error rates for the three datasets in Table 1 . All our results were evaluated using a 1-nearest neighbor search of the 1024-dimensional vectors extracted from the pool5 layer of GoogLeNet with batch normalization (Ioffe & Szegedy, 2015) . In all the experiments, the features extracted from the fine-tuned classification network show the best fine-grained classification performance. Our results of softmax-based classification are better than the results in Rippel et al. (2016) .

The experiments of softmax-based classification in Rippel et al. (2016) were not the best.

4.3. ATTRIBUTE ESTIMATION:

Rippel et al. ( 2016) evaluated features' expressiveness using mean attribute precision and showed that the features generated by their proposed method contain intra-class diversity. In this section, we investigate the intra-class diversity of softmax features. We use the ImageNet Attribute dataset (Rippel et al., 2016) , which consists of overlap between the ImageNet training set (Russakovsky et al., 2015) and the Object Attribute dataset (Russakovsky & Fei-Fei, 2010) . We used only the images and their class labels during our training of the softmax classifier and did not use attributes.

Table 2 shows the error rates of 90-way classification under different training methods. Our finetuned softmax classifier outperformed those of Rippel et al. (2016) by a considerable margin. Fig. 4 shows the mean attribute precision for the ImageNet Attribute dataset. Our fine-tuned softmax features markedly outperformed those from Rippel et al. (2016) . These results implicitly indicate that the features extracted from the pool5 layer contain intra-class diversity that is better than those from DML networks designed to keep intra-class diversity. 

4.4. CLUSTERING AND RETRIEVAL:

Here, we give our evaluation of clustering and retrieval scores for the state-of-the-art DML method (Song et al., 2016) and for the softmax classification networks. We used the Caltech UCSD Birds 200-2011 (CUB) dataset (Wah et al., 2011) , the Stanford Cars 196 (CAR) dataset (Krause et al., 2013) , and the Stanford Online Products (OP) dataset (Song et al., 2016) . For CUB and CAR, we used the first half of the dataset classes for training and the rest for testing. For OP, we used the training-testing class split provided. The dataset properties are shown in Table 3 . We emphasize that the class sets used for training and testing are completely different. We multiplied the learning rates of the changed layers (output layers for all models and the fully connected layer added for FCR 1 and FCR 2) by 10. The batch size was set to 128, and the maximum number of iterations for our training was set to 20,000. These training strategies are exactly the same as those used in the earlier study (Song et al., 2016) .

For clustering evaluation, we applied k-means clustering 100 times and calculated the average standard F 1 and NMI (Manning et al., 2008) ; the value for k was set to the number of classes in the test set. For retrieval evaluation, we used the Recall@K metric (Jegou et al., 2011) .

We show the results for the CUB dataset in Fig. 5 and for the CAR dataset in Fig. 6 . We notice that we have been able to reproduce nearly exactly the scores of lifted structured feature embedding (Song et al., 2016) . However, the deep features extracted from the softmax-based classification networks outperformed the lifted structured feature embedding in all the evaluation metrics.

For F 1 and NMI, all of the softmax models, including PCA, FCR 1, and FCR 2, show markedly better scores than does lifted structured feature embedding. It is clear that L2 normalization improves the scores of all the softmax-based models. The scores of PCA and FCR 1 drop slightly as the feature dimensionality decreases from 1024 for both the CUB dataset and the CAR dataset. On the other hand, FCR 2, which has a fully connected layer followed by a dropout layer, improves the scores in spite of the reduction in dimensionality, as shown in Fig. 6 . It may be that 1024 dimensions is too large to describe the image classes. This result may imply that to obtain the best features we need to first determine the optimum dimensionality of the feature space for the dataset and then apply PCA.

For the Recall@K metric, we used 1024-dimensional features for the CUB dataset and 256dimensional features for the CAR dataset. The softmax-based features outperformed the DMLbased features. The differences between PCA, FCR 1, and FCR 2 are very minor. Regarding feature normalization, features without normalization show worse scores than do L2-normalized features.

Fig. 7 shows the standard F 1 , NMI, and Recall@K for the Online Products dataset. We used 1024dimensional features for the Recall@K metric. As shown in Table 3 , the OP dataset is very different from the CUB and CAR datasets in terms of the number of classes and the number of samples per class; the number of samples per class in the OP dataset is limited to 5.3 on average. In contrast to CUB and CAR, in the OP dataset the scores for softmax and for lifted structured feature embedding are nearly the same.

From the results for these three datasets, we conjecture that the number of images contained in the dataset has a considerable effect on softmax-based classification. In other words, it is difficult for DML to make use of the rich information from a large number of samples because of the randomness described in the previous section. Hence, we changed the size of datasets by subsampling the images of CUB and CAR datasets for each class and ran the experiments again. We constructed seven datasets of different sizes, containing 5, 10, 20, 40, 60, 80, and 100 %, respectively, of the whole dataset. As shown in Fig. 8 and Fig. 9 , the difference between the scores for softmax and DML is small or close to zero if the size of the training dataset is small. The gap between softmax and DML becomes larger as the dataset size increases. It is surprising that the scores of lifted structured feature embedding on the CUB dataset did not increase even though we used more images for the training (Fig. 8 ). It can be said that DML cannot exploit large training datasets, whereas the softmax-based classifier can obtain features of high expressiveness.

5. CONCLUSION:

Because there was no equitable comparison in previous studies, we conducted comparisons of the softmax-based classifier and DML methods using a design that would enable the methods to objectively demonstrate their true performance capabilities. Our results show that the features extracted from softmax-based classifiers perform better than those from state-of-the-art DML methods (Rippel et al., 2016; Song et al., 2016) on fine-grained classification, clustering, and retrieval tasks, especially when the size of the training dataset is large. The experimental results also show that softmax-based features exhibit rich intra-class diversity even though the softmax classifier is not explicitly designed to do so, unlike to the previous method (Rippel et al., 2016) . It is obvious that the softmax-based features are still strong baselines. We hope that softmax-based features are taken into account when evaluating the performance of deep features.

Limitations. When the number of classes are huge, it is hard to train classification networks due to GPU memory constraints. DML-based methods are suitable for such cases because they do not need the output layer which is proportional to the number of classes. For cross-domain tasks, such as sketches to photos (Yu et al., 2016; Sangkloy et al., 2016) or aerial views to ground views (Lin et al., 2015) , DML is also effective. Classification-based learning needs complicated learning strategies like in Castrejon et al. (2016) . DML-based methods can learn cross-domain representation only by using a pair of networks.

:



Figures and Tables:

Figure 1: Depiction of MNIST dataset. (a) Two-dimensional features obtained by siamese network. (b) Two-dimensional features extracted from softmax-based classifier; these features are well separated by angle but not by Euclidean norm. (c) Three-dimensional features extracted from softmaxbased classifier; we normalized these to have unit L2 norm and depict them in an azimuth-elevation coordinate system. The three-dimensional features are well separated by their classes.

Figure2: Illustration of learning processes for softmax-based classification network and siamesebased DML network. For softmax, the gradient is defined by the distance between a sample and a fixed one-hot vector, and for siamese by the distance between samples.

Figure 3: GoogLeNet Szegedy et al. (2015) architecture we use in this paper. We extracted the features from the red-colored layers. For (a), we applied PCA to reduce the number of feature dimensions. For (b) and (c), the dimensionality is already reduced to the required number by the fc reduction layer.

Figure 4: Mean attribute precision for the ImageNet Attribute dataset.

Figure 7: F 1 , NMI, and Recall@K scores for the test set of the Online Products dataset.

Figure 9: F 1 , NMI, and Recall@K scores for test set of the Stanford Cars 196 dataset under different dataset sizes. The feature dimensionality is fixed at 256.

Error rates for various fine-grained image datasets.

Table content: <table><tr><td/><td/><td colspan="2">(b) Oxford 102 Flowers.</td><td>(c) Oxford -IIIT Pet.</td><td/></tr><tr><td>Approach</td><td>Error</td><td>Approach</td><td>Error</td><td>Approach</td><td>Error</td></tr><tr><td>Angelova &amp; Long (2014)</td><td>51.7%</td><td>Angelova &amp; Zhu (2013)</td><td>23.3%</td><td>Angelova &amp; Zhu (2013)</td><td>49.2%</td></tr><tr><td>Xie et al. (2015)</td><td>50.6%</td><td>Angelova &amp; Long (2014)</td><td>19.6%</td><td>Parkhi et al. (2012)</td><td>46.0%</td></tr><tr><td>Gavves et al. (2013)</td><td>49.9%</td><td>Murray &amp; Perronnin (2014)</td><td>15.4%</td><td>Angelova &amp; Long (2014)</td><td>44.6%</td></tr><tr><td>Gavves et al. (2015)</td><td>43.0%</td><td>Razavian et al. (2014)</td><td>13.2%</td><td>Murray &amp; Perronnin (2014)</td><td>43.2%</td></tr><tr><td>Qian et al. (2015)</td><td>30.9%</td><td>Qian et al. (2015)</td><td>11.6%</td><td>Qian et al. (2015)</td><td>19.6%</td></tr><tr><td colspan="2">Rippel et al. (2016) (Softmax prob) 26.6%</td><td colspan="2">Rippel et al. (2016) (Softmax prob) 11.2%</td><td colspan="2">Rippel et al. (2016) (Softmax prob) 11.3%</td></tr><tr><td>Rippel et al. (2016) (Triplet)</td><td>35.8%</td><td>Rippel et al. (2016) (Triplet)</td><td>17.0%</td><td>Rippel et al. (2016) (Triplet)</td><td>13.5%</td></tr><tr><td>Rippel et al. (2016) (Magnet)</td><td>24.9%</td><td>Rippel et al. (2016) (Magnet)</td><td>8.6%</td><td>Rippel et al. (2016) (Magnet)</td><td>10.6%</td></tr><tr><td>Ours (Softmax prob)</td><td>18.3%</td><td>Ours (Softmax prob)</td><td>8.69%</td><td>Ours (Softmax prob)</td><td>9.04%</td></tr><tr><td>Ours (Softmax pool5)</td><td>21.0%</td><td>Ours (Softmax pool5)</td><td>7.90%</td><td>Ours (Softmax pool5)</td><td>9.09%</td></tr><tr><td>Ours (Softmax pool5 + L2)</td><td>20.3%</td><td>Ours (Softmax pool5 + L2)</td><td>7.09%</td><td>Ours (Softmax pool5 + L2)</td><td>9.04%</td></tr></table>

Classification error rates for the ImageNet Attribute dataset.

Table content: <table><tr><td/><td/><td/><td>1</td><td>Rippel (Softmax prob)</td></tr><tr><td/><td/><td>precision</td><td>0.8</td><td>Rippel (Triplet) Rippel (Magnet) Ours (Softmax pool5 + L2)</td></tr><tr><td/><td/><td>attribute</td><td>0.6</td><td/></tr><tr><td>Approach</td><td>Error</td><td>Mean</td><td>0.4</td><td/></tr><tr><td colspan="2">Rippel et al. (2016) (Softmax prob) 14.1%</td><td/><td/><td/></tr><tr><td>Rippel et al. (2016) (Triplet) Rippel et al. (2016) (Magnet)</td><td>26.8% 15.9%</td><td/><td>2 0 0.2</td><td>Neighbourhood size 2 2 2 4 2 6</td><td>2 8</td></tr><tr><td>Ours (Softmax prob)</td><td>7.68%</td><td/><td/><td/></tr><tr><td>Ours (Softmax pool5)</td><td>11.9%</td><td/><td/><td/></tr><tr><td>Ours (Softmax pool5 + L2)</td><td>10.7%</td><td/><td/><td/></tr></table>



Table content: <table><tr><td/><td colspan="3">: Properties of datasets</td></tr><tr><td colspan="4">used in Section 4.4. Each cell</td></tr><tr><td colspan="4">shows the number of images</td></tr><tr><td colspan="4">(upper figure) and the number</td></tr><tr><td colspan="4">of classes (lower figure).</td></tr><tr><td>Dataset</td><td>Train</td><td>Test</td><td>Total</td></tr><tr><td>CUB</td><td>5,864 100</td><td>5,924 100</td><td>11,788 200</td></tr><tr><td>CAR</td><td>8,054 98</td><td>8,131 98</td><td>16,185 196</td></tr><tr><td>OP</td><td colspan="3">59,551 60,502 120,053 11,318 11,316 22,634</td></tr></table>

F 1 , NMI, and Recall@K scores for the test set of the Stanford Cars 196 dataset.

Table content: <table><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>1</td><td/><td/><td/></tr><tr><td/><td>0.3</td><td/><td/><td/><td/><td/><td>0.62</td><td/><td/><td/><td/><td/><td/><td>0.9</td><td/><td/><td/></tr><tr><td/><td>0.28</td><td/><td/><td/><td/><td/><td>0.61</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>F 1 score</td><td>64 0.18 0.2 0.22 0.24 0.26</td><td>128</td><td>256 PCA + L2 PCA FCR 1 + L2 FCR 1</td><td colspan="2">512 FCR 2 + L2 1024 FCR 2 Lifted struct</td><td>NMI score</td><td>64 0.55 0.57 0.58 0.59 0.6 0.56</td><td>128</td><td>256 PCA + L2 PCA FCR 1 + L2 FCR 1</td><td colspan="3">512 FCR 2 + L2 1024 FCR 2 Lifted struct</td><td>Recall@K</td><td>0.4 0.7 0.8 0.6 0.5</td><td>1</td><td>2</td><td>4</td><td>8</td><td>16 PCA + L2 PCA FCR 1 + L2 FCR 1 FCR 2 + L2 FCR 2 Lifted struct</td><td>32</td></tr><tr><td/><td/><td colspan="2">Embedding size</td><td/><td/><td/><td/><td colspan="2">Embedding size</td><td/><td/><td/><td/><td/><td/><td/><td>K</td></tr><tr><td colspan="19">Figure 5: F 1 , NMI, and Recall@K scores for the test set of the Caltech UCSD Birds 200-2011</td></tr><tr><td colspan="2">dataset.</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0.3</td><td/><td/><td/><td/><td/><td>0.64</td><td/><td/><td/><td/><td/><td/><td>1</td><td/><td/><td/></tr><tr><td/><td>0.28</td><td/><td/><td/><td/><td/><td>0.62</td><td/><td/><td/><td/><td/><td/><td>0.9</td><td/><td/><td/></tr><tr><td>F 1 score</td><td>0.2 0.22 0.24 0.26</td><td>PCA + L2 PCA FCR 1 + L2 FCR 1</td><td colspan="2">FCR 2 + L2 FCR 2 Lifted struct</td><td/><td>score NMI</td><td>0.54 0.56 0.58 0.6</td><td>PCA + L2 PCA FCR 1 + L2 FCR 1</td><td>FCR 2 + L2 FCR 2 Lifted struct</td><td/><td/><td/><td>Recall@K</td><td>0.7 0.8 0.6</td><td/><td/><td/><td>PCA + L2 PCA FCR 1 + L2 FCR 1 FCR 2 + L2 FCR 2 Lifted struct</td></tr><tr><td/><td>64 0.18</td><td>128</td><td>256</td><td>512</td><td>1024</td><td/><td>64 0.52</td><td>128</td><td>256</td><td colspan="2">512</td><td>1024</td><td/><td>0.5</td><td>1</td><td>2</td><td>4</td><td>8</td><td>16</td><td>32</td></tr><tr><td/><td/><td colspan="2">Embedding size</td><td/><td/><td/><td/><td colspan="2">Embedding size</td><td/><td/><td/><td/><td/><td/><td/><td>K</td></tr><tr><td>F 1 score</td><td colspan="2">128 Figure 6: 64 0.14 0.26 0.28 0.16 0.18 0.2 0.22 0.24 PCA + L2 PCA FCR 1 + L2 FCR 1</td><td>256 FCR 2 + L2 FCR 2 Lifted struct</td><td>512</td><td>1024</td><td>NMI score</td><td>64 0.85 0.875 0.88 0.855 0.86 0.865 0.87</td><td>128 PCA + L2 PCA FCR 1 + L2 FCR 1</td><td colspan="2">256 FCR 2 + L2 FCR 2 Lifted struct</td><td>512</td><td>1024</td><td>Recall@K</td><td colspan="2">10 0 0.9 1 0.7 0.8 0.6</td><td>10 1</td><td/><td>10 2</td><td>PCA + L2 PCA FCR 1 + L2 FCR 1 FCR 2 + L2 FCR 2 Lifted struct</td><td>10 3</td></tr><tr><td/><td/><td colspan="2">Embedding size</td><td/><td/><td/><td/><td colspan="2">Embedding size</td><td/><td/><td/><td/><td/><td/><td/><td>K</td></tr></table>

F 1 , NMI, and Recall@K scores for test set of the Caltech UCSD Birds 200-2011 dataset under different dataset sizes. The feature dimensionality is fixed at 1024.

Table content: <table><tr><td/><td>0.32</td><td/><td/><td/><td/><td/><td/><td>0.64</td><td/><td/><td/><td/><td/><td/><td colspan="2">0.56</td><td/><td/></tr><tr><td/><td>0.3</td><td/><td/><td/><td/><td/><td/><td>0.62</td><td/><td/><td/><td/><td/><td/><td colspan="2">0.54</td><td/><td/></tr><tr><td>F 1 score</td><td>0.22 0.24 0.26 0.28</td><td/><td/><td/><td>FCR 2 + L2 Lifted struct</td><td/><td>NMI score</td><td>0.56 0.58 0.6</td><td/><td/><td/><td>FCR 2 + L2 Lifted struct</td><td/><td>Recall@1</td><td colspan="2">0.46 0.52 0.48 0.5</td><td/><td/><td>FCR 2 + L2 Lifted struct</td></tr><tr><td/><td>0.2</td><td/><td/><td/><td/><td/><td/><td>0.54</td><td/><td/><td/><td/><td/><td/><td colspan="2">0.44</td><td/><td/></tr><tr><td/><td>0.18</td><td>5 10 20</td><td>40</td><td>60</td><td>80</td><td>100</td><td/><td>0.52</td><td>5 10 20</td><td>40</td><td>60</td><td>80</td><td>100</td><td/><td colspan="2">0.42</td><td>5 10 20</td><td>40</td><td>60</td><td>80</td><td>100</td></tr><tr><td/><td/><td colspan="4">Training dataset size (%)</td><td/><td/><td/><td colspan="4">Training dataset size (%)</td><td/><td/><td/><td/><td colspan="3">Training dataset size (%)</td></tr><tr><td colspan="3">0.1 0.25 0.3 Figure 8: 5 10 20 0.15 0.2 F 1 score</td><td>40</td><td>60</td><td>80 Lifted struct FCR 2 + L2</td><td>100</td><td>NMI score</td><td>0.35 0.6 0.65 0.4 0.45 0.5 0.55</td><td>5 10 20</td><td>40</td><td>60</td><td>80 Lifted struct FCR 2 + L2</td><td>100</td><td colspan="2">Recall@1</td><td>0.3 0.7 0.8 0.4 0.5 0.6</td><td>5 10 20</td><td>40</td><td>60</td><td>80 FCR 2 + L2 Lifted struct</td><td>100</td></tr><tr><td/><td/><td colspan="4">Training dataset size (%)</td><td/><td/><td/><td colspan="4">Training dataset size (%)</td><td/><td/><td/><td/><td colspan="3">Training dataset size (%)</td></tr></table>

Peer review (iclr2017):
I have a huge, big picture concern about this paper and the papers it most closely addresses (MagnetLoss and Lifted Feature Structure Embedding). I don*t understand why Distance Metric Learning (DML) is being used for classification tasks (Stanford Cars 196, UCSD Birds 200, Oxford 102 flowers, Stanford Dogs, ImageNet attributes, etc). As far as I can tell, there is really only a single *retrieval*-like benchmark being used here - the Stanford Online Products database. All the other datasets are used in a *classification-by-retrieval* approach which seems contrived. While ostensibly evaluating *retrieval*, the retrieval ground truth is totally defined by category membership so these are still classification tasks with many instances in each category. With the Online Products dataset the correspondence between queries and correct results is much more fine grained so it makes sense to think of it as a retrieval task. It seems obvious that if your task is classification, a network trained with a classification loss will be best. Even when these datasets are used in a *retrieval* setting, the ground truth is still defined by category membership. It*s still a classification task. I don*t really see the point of using DML in these scenarios. I guess prior work claims to outperform SoftMax in these settings so this paper is fighting back against this and I should be thankful for this paper. But I think this paper*s narrative is a bit off. The narrative shouldn*t be *We can get good retrieval features from softmax networks with appropriate normalization*. It should be *It never made sense to train or evaluate these things as retrieval tasks. Direct classification is better*. For example, why are you taking the second to last layer or pool5 layer from these networks? Why aren*t you taking the last layer? That should do well in these evaluations, right? Table 1 and 2 do show that using softmax probabilities directly tends to be better than doing classification-by-retrieval (works better or the same as doing retrieval with an earlier layer of features, except on Oxford flowers). GoogLeNet is quite deep and gets auxiliary supervision. By the second-to-last layer of the network, the activations could look a lot like category membership already. And category membership is all that*s needed for the tasks in 4.2 and 4.3. I don*t think my pre-review question was adequately addressed. I was getting at this concern by pointing out numerous scenarios where distance metric learning makes sense because you have fine-grained associations between instances at training time, NOT categorical associations -- e.g. this product photo corresponds to this photo of the object in a scene [Bell et al. 2015], this 3d model correspond to this sketch [Wang et al. 2015], this sketch corresponds to this photo [Sangkloy et al. 2016], this ground view corresponds to this aerial view [Lin et al., 2015]. DeepFace and follow-up works on LFW could also fit into this space because there are few training samples per class (few training samples per person identity). You cite DeepFace and Bell et al. 2015 but you don*t compare on those benchmarks. I think those are exactly the tasks where DML makes sense. Maybe the *retrieval on classification datasets* would be a reasonable benchmark if the test and train classes were completely different. Then you could argue that softmax is learning a useful representation yet the last layer isn*t directly useful since the categories change. But that*s not the case here, is it? With all of this said, I*m not sure whether I*m positive or negative about this paper. I think you*re onto something significant -- people have been using DML where it is not appropriate -- but addressed it in the wrong way -- by using softmax for *classification by retrieval*. But you don*t need to do retrieval! Softmax is already telling you the class prediction! Why go through the extra step of finding nearest neighbors with some intermediate feature? AnonReviewer3 also raises some good points and you should be thankful that a reviewer is willing to dig so deep to help make your experiments sound! I don*t think his/her concerns are disqualifying for this paper, though, as long as it is fixed. I look forward to hearing your response. I want this paper to be published, but I think it needs to be tweaked.