17th International Conference on Machine Vision Applications (MVA)
Fully Online, July 25–27, 2021.

P2-6

Analysis of Evaluation Metrics with the Distance between
Positive Pairs and Negative Pairs in Deep Metric Learning
Hajime Oi† ,Rei Kawakami‡ ,Takeshi Naemura†† The University of Tokyo,‡ Tokyo Institute of Technology{hoi,naemura}@hc.ic.i.u-tokyo.ac.jp,
reikawa@c.titech.ac.jp
Abstract
Deep metric learning (DML) acquires embeddings
via deep learning, where distances among samples ofthe same class are shorter than those of diﬀerent
classes. The previous DML studies proposed new metrics to overcome the issues of general metrics, but they
have the following two problems; one is that they consider only a small portion of the whole distribution of
the data, and the other is that their scores cannot bedirectly compared among methods when the number ofclasses is diﬀerent. To analyze these issues, we consider the histograms of the inner products between arbitrary positive pairs and those of negative pairs. We
can evaluate the entire distribution by measuring the
distance between the two histograms. By normalizing
the histograms by their areas, we can also cancel theeﬀect of the number of classes. In experiments, visualizations of the histograms revealed that the embeddingsof the existing DML methods do not generalize well tothe validation set. We also conﬁrmed that the evaluation of the distance between the positive and negative
histograms is less aﬀected by the variation in the number of classes compared with Recall@1 and MAP@R.

1 Introduction
Metric learning aims to learn an embedding spacein which the similarity of data corresponds to the distance between vectors; i.e., the more similar the data
are, the shorter the distance between them in the spacebecomes. In the case of images, the embedding spaceis acquired such that the distances are shorter for images of the same class and longer for images of diﬀerent
classes. Many applications can beneﬁt from a better
evaluation of the semantic distance, such as contentbased image retrieval and face recognition. Metriclearning that relies on deep learning is called deep
metric learning (DML), and its performance has been
higher than those based on non-linear kernels and classiﬁers.
The literature on DML has few discussions on evaluation metrics. One such metric, normalized mutual
information (NMI) [9], is used to evaluate clustering; it
examines the information gain of the estimated labels
for the true labels. Clustering is necessary to estimate
the labels in the learned space; thus, NMI depends
on the clustering accuracy. Another evaluation metric, Recall@K [5], is used in information retrieval; it
represents the percentage of total data in the dataset
containing at least one positive among their K nearest neighbors; thus, Recall@K does not consider thenumber of positives in the neighbors or how close each© 2021 MVA Organization

positive is to the query. R-Precision and Mean Average
Precision@R (MAP@R) were used in [10] as alternatives, but these also have two issues. First, they focus
only on the accuracy of R nearest neighbors, which is atiny part of the whole data distribution. Second, when
the number of classes increases, their scores probably
decrease because the proportion of negatives in the Rnearest neighbors increases; therefore, their scores cannot be directly compared if the number of classes is
diﬀerent.
To analyze the issues of evaluation metrics in DML,
we consider the distribution of inner products between
positive pairs as well as that of negative pairs. We
also measure the distance between these two distributions by using the Jensen-Shannon divergence (JSD)to evaluate the performance of DML on the entire distribution. Additionally, we normalize the distribution
by its area to cancel out the variation in the numberof classes.
In the experiments, we observed changes in themean value of the inner products between positive pairs
and those of negative pairs for the training and validation sets during training. We also determined whether
these changes are consistent with those of JSD, Recall@1, and MAP@R. As the training progressed, theaverage value of the inner products between positives
in the training set increased, but those in the validation
set remained constant or slightly decreased, suggestingthat the learned space overﬁts the training set and is
not well generalized. The changes in JSD were similar
to those of the existing metrics, indicating that it does
not show any clear advantages in using the entire distribution compared with R nearest neighbors with theexamined dataset. However, JSD showed signs of overﬁtting to the training set and synchronized with it. It
starts with the same value for the training and validation sets, and later, only the value for the training set
increases. Since the numbers of classes in the training
and validation sets are diﬀerent, other metrics cannot
compare their scores directly or capture the overﬁtting.
The experiments with diﬀerent data splitting methods also revealed problems with those used in the previous studies.

2 Preliminaries
R-Precision and MAP@R Musgrave et al. [10]propose to use R-Precision and MAP@R to evaluate
DML, where R is the total number of positives foreach query. These two measures are related to Precision@K, which refers to the percentage of positives
in the K nearest neighbors. R-Precision is the averageof Precision@R over all of the queries. MAP@R is theaverage of Average Precision@R (AP@R) over all ofthe queries, where AP@R is given by 1
1 ∑AP @R =
P recision@k × p(k)
R
R

(1)

k=0

{
p(k) =

10

if k-th nearest neighbor is positive,
if k-th nearest neighbor is negative.
(a) Before normalization

MAP@R and AP@R have two problems. The ﬁrst
is that they do not evaluate the distribution of the entire data; namely, they ignore the improvement when acorrect positive sample not included in its R neighbors
approaches the query. The second is that the scores
are sensitive to changes in the number of classes. In
other words, when the number of classes increases, thepossibility of having negatives in the R nearest neighbors increases, and the score decreases because all theadded classes are negative.
Loss functions in DML Early losses in DML attempted to obtain the global metric space by optimizing distances within small local groups [3, 17, 4],which was ineﬃcient because of the enormous numberof possible group combinations. One way to meet thischallenge is to prepare proxies for each class and toput constraints only on the distances between the data
and the proxies. Most of the losses in this category are
similar to the softmax loss.
Here, we will review the softmax loss and the normalized softmax loss (NSL) to clarify the distance andinner products between any pair. Let us deﬁne xi tobe the input to the last fully connected (FC) layer ofthe i-th sample; it is a d-dimensional vector belonging
to the yi -th class. The softmax function, which also
serves as a loss function in metric learning, outputs
a vector whose j-th element represents the predictedprobability that xi belongs to class j. The FC layer
consists of a weight matrix W ∈ Rd×C and a bias vector b ∈ RC . Wj denotes the j-th column of W, and C
is the number of classes. The softmax loss maximizes
the yi -th element of the ﬁnal vector as it is the trueclass of xi .
T

eWyi xi +byi

Sof tmax = − log ∑C

j=1 e

WjT xi +bj(2)

In DML, Wj is often treated as a proxy of the j-thclass and b is ignored; therefore, the inner product of xi
and Wj , both of which are ℓ2 -normalized, is considered
to be a distance [2, 12, 6]. The simplest loss [16, 19] iscalled the normalized softmax loss (NSL) in this paper,
following [10]:TesW̄yi x̄i

N SL = − log ∑C

j=1 esW̄jT x̄i,(3)where .̄ is ℓ2 -normalized and s is a scaling parameter.

(b) After normalization

Figure 1: Distribution of inner products before and after normalization. The histogram in red shows the inner products between every positive pair after training,
and the histogram in blue shows those of the negative
pairs. Best viewed in color.

3 Distribution of inner products between
positives and negatives
In many DML losses, the inner product between an
embedded vector of a given sample and a proxy vector
for its class, which are ℓ2 -normalized, are evaluated andmaximized during training. Fig. 1 (a) depicts two histogram distributions of inner products; the one in red
shows the inner products between every positive pairand the one in blue is for those between every negative
pair in the dataset. It is clear that the number of negative pairs is much larger than the number of positive
pairs. If the number of classes is c and the average
number of samples per class is n, the number of positive pairs is O(cn2 ) and the number of negative pairs isO((cn)2 ); thus, the latter is about c times larger. This
imbalance may make it impossible for metrics based on
rankings arranged in order of increasing inner product,such as MAP@R, to evaluate the true performance ofmodels.
To resolve this issue, we consider a distribution thatis normalized so that the areas of the histograms of thepositive and negative pairs are each 1.0, as shown in
Fig. 1 (b). In this way, we cancel out the imbalance inthe number by normalization and obtain distributions
independent of the number of classes.
The two histograms in Fig. 1 (b) can be regarded
as discrete probability distributions since they sum to
1. We can evaluate the distance between them by using tools to evaluate the distance between probability distributions, such as the Kullback-Leibler divergence (KLD). KLD, as shown in Eq. (4), represents
the diﬀerence between the true probability distribution P and its predicted probability distribution Q,
and it is equal to the cross-entropy of P and Q minus the self-information content of P. In general, KLD
is not symmetric between P and Q, i.e., DKL (P ||Q) ̸=
DKL (Q||P ). Thus, we employ the JSD as shown in
Eq. (5), which a modiﬁcation to KLD that is symmetric. JSD is obtained by adding the probability distribution M averaged over P and Q and averaging the KLD
of P and M and the KLD of Q and M. The minimum
1 Note that Eq. (3) in [10] should be a typo.

(a) Cars196 [7](b) CUB-200-2011 [15]Figure 2: Mean value of inner products between every
positive pair and those between every negative pair in
the training and validation sets. Overﬁtting occurredfor positive pairs during training.

value of JSD is 0 when P and Q coincide perfectly, andits maximum is 1 when P and Q do not overlap at all.
DKL (P ||Q) =∑i

DJS (P ||Q) =P (i) log

P (i)
,
Q(i)

DKL (P ||M ) + DKL (Q||M ),2()
P (i) + Q(i)M (i) =
.
2

(4)
(5)

4 Experiments
We implemented our model in the PyTorch framework [11].
Dataset Our evaluation used the Cars196 [7], CUB200-2011 [15] datasets. The validation set was prepared for model selections aligned with [10]; therefore,
the datasets were split into training, validation, andtest sets in a ratio of 4:1:5 in the number of classes.
We also conducted experiments in which we split theclasses in the default order or in random order. The
elements of the two datasets are originally arranged in
the order of their labels names; cars from the same
company are placed near each other in the Cars196
dataset [7] and birds belonging to the same family
and genus are near each other in the CUB-200-2011
dataset [15]. Since there seems to be many similarities
among the items belonging to the same domain in thehigher level classiﬁcations, there is possibly a domain
bias among the training, validation, and test sets when
they are divided up according to the default order ofthe classes. We set the random seed to 1.
Augmentation The input images in the training set
were ﬁrst randomly cropped in the scale-ratio range of[1/8, 1] and aspect ratio [3/4, 4/3]; then, they were resized to 299 pixels square. Though the previous studies set the size 244 pixels, we used a pre-trained model
provided PyTorch framework [1] and could not choose
the size. After that, they were ﬂipped horizontallywith a probability of 50 percent. Finally, the values of
their RGB channels were normalized according to the

Figure 3: Comparison of the histograms in the test
set before and after training with two datasets where
overﬁtting occurs. The variances of the histograms fornegative pairs decreased, and the peak is closer to 0
after training; thus, although the overﬁtting occurs,
the network are successful to separate the negatives
from each other. It is more diﬃcult to let positives becloser.

mean = (0.485, 0.456, 0.406) and standard deviation= (0.229, 0.224, 0.225). The images in the validationand test sets were resized to 299 pixels square and thennormalized in the same way as above.
Training and Evaluation We used the Inceptionv3 network [14] pre-trained on ImageNet [13] as theembedding network, whose last fully-connected (FC)layer was replaced by a new one with 2064 input and 64
output dimensions. For the loss, we used the NSL expressed by Eq. 3 with s = 10. Training was performedfor 20 epochs in all cases. The initial learning ratesof the network and NSL were set to 10−4 and 10−2 ,respectively, and they were tuned with the AdamW
optimizer [8]. In the evaluation, we used Recall@1 and
MAP@R as representatives of the metrics used in theprevious studies and JSD as proposed.
Results Fig. 2 shows the mean values of the innerproducts in the training and validation sets. The averages of the two sets for the negative pairs were almost
the same during training and converged to around 0.
The negative pairs were suﬃciently separated because
the Euclidean
distance between negative pairs is likely√
to be 2 for negative pairs uniformly distributed on
the unit hypersphere [18]. On the other hand, although
the positive mean was initially small and subsequently
increased in the training set, it remained constant orslightly decreased in the validation set, indicating that
overﬁtting occurred.
To clarify the diﬃculty of letting positives be closer
in metric learning, we also visualize the histograms in
the test set before and after the training in Fig. 3.
The histograms for the positive pairs show a slight decrease of the inner products though we hope them tobe larger, and this tendency is the same for the mean

Figure 4: Scores of existing metrics and JSD for training and validation set. Though the changes in the individual
metrics were similar on validation set, the scores for the training set were diﬀerent from those of the validation
set. On the one hand, in JSD, these score were initially almost the same; then, the training score started to
exceed the validation score when the overﬁtting began in Fig. 2. On the other hand, regarding the metrics used in
the previous studies, the validation scores were initially larger than the training scores; moreover, the large-small
relationship reversed after the overﬁtting started.
value of them, although the inner products for training set indeed become larger as in Fig. 2. On the other
hand, the variance of the distribution of negative pairs
becomes smaller, and the inner products between thenegatives are closer to 0 after training.
Fig. 4 depicts the dissimilarity between the metrics
used in the previous studies and JSD. There was no
signiﬁcant diﬀerence between them on the validation
set; therefore, the necessity of evaluating the whole
data distribution could not be veriﬁed. By contrast,
comparing the training with validation results, we cansee that the scores of JSD were equal before the overﬁtting began in Fig. 2; then, the training scores exceeded
the validation scores. For Recall@1 and MAP@R, thescores for the validation set were initially larger; moreover, the reversal of the large-small relationship waslater than the start of overﬁtting. These results arelikely due to the fact that the number of classes in the
training set was about four times larger than that in
the validation set, and the number of negatives waslarge, suggesting that the existing evaluations are easily aﬀected by variations in the number of classes.
Finally, Table 1 reveals the diﬀerences in the scoresfor the test set between classes split in default orderand in random order. These scores were calculatedwith embeddings that were extracted using the modelof the epoch having the highest score for the validation
set. The scores for all metrics were lower when theclasses were partitioned in default order; thus, there
seemed to be a domain bias between the training, validation, and test sets in this partition.

5 Conclusion
We analyzed DML, focusing on the histogram ofinner products between positive pairs and that of negatives. Measuring the distance between the two his-Table 1: Diﬀerence in scores of evaluation metrics onthe test set among class partitioning methods. For allof the datasets, the scores were lower when the classes
were split in the default order than in a random order, suggesting that there is a domain bias between
the training, validation, and test sets in the default
order.
DatasetCars196
CUB-200-2011

Splittingdefault
randomdefault
randomRecall@1
0.673
0.721
0.481
0.659

MAP@R
0.168
0.226
0.164
0.260

JSD
0.439
0.472
0.401
0.579

tograms with JSD enabled us to consider all the data
other than R nearest neighbors, unlike the evaluation
metrics used in the previous studies. Moreover, normalization of the histograms removed the eﬀect of thevariation in the number of classes. As a result, we suggest that the existing methods are likely to overﬁt the
distance between positives. Although we cannot verify
the necessity of considering data other than R nearest
neighbors, we can conﬁrm that JSD reduces the eﬀectof the variation in the number of classes compared withthe existing metrics. In addition, there is a possibly local domain bias with the default order of labels in thedatasets, suggesting that we should assign the classes
randomly in the training-validation-test splitting.
One of our future tasks will be to remove the overﬁtting. Disentangled representations perhaps resolve thisissue, each variables of which are lower dimensional features and independent of each other. We expect thatthe use of disentangled representations will improveclass discrimination by including only features whose
inter-class variances are large and intra-class variances
are small.

References[1] Inception v3 |PyTorch. https://pytorch.org/hub/pytorch_vision_inception_v3/. Accessed: 2021-0611.
[2] J. Deng, S. Zafeririou. Arcface for disguised face recognition. In IEEE International Conference on Computer Vision Workshop, pages 485–493, 2019.
[3] R. Hadsell et al. Dimensionality reduction by learning
an invariant mapping. In IEEE Conference on Computer Vision and Pattern Recognition, volume 2, pages
1735–1742, 2006.
[4] E. Hoﬀer, N. Ailon. Deep metric learning using triplet
network. In Similarity-Based Pattern Recognition, pages
84–92, 2015.
[5] H. Jégou et al. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis
and Machine Intelligence, 33(1):117–128, 2011.
[6] S. Kim et al. Proxy anchor loss for deep metric learning. In IEEE Conference on Computer Vision and
Pattern Recognition, pages 3235–3244, 2020.
[7] J. Krause et al. 3d object representations for ﬁnegrained categorization. In International IEEE Workshop on 3D Representation and Recognition, 2013.
[8] I. Loshchilov, F. Hutter. Decoupled weight decay regularization. In International Conference on Learning
Representations, 2019.
[9] C. D. Manning et al. Introduction to Information Retrieval. Cambridge University Press, 2008.
[10] K. Musgrave et al. A metric learning reality check.
In European Conference on Computer Vision, pages

681–699, 2020.
[11] A. Paszke et al. PyTorch: An Imperative Style, HighPerformance Deep Learning Library. In Advances in
Neural Information Processing Systems, pages 8024–
8035. 2019.
[12] Q. Qian et al. Softtriple loss: Deep metric learning
without triplet sampling. In IEEE International Conference on Computer Vision, pages 6449–6457, 2019.
[13] O. Russakovsky et al. ImageNet Large Scale Visual
Recognition Challenge. International Journal of Computer Vision, 115(3):211–252, 2015.
[14] C. Szegedy et al. Rethinking the inception architecture for computer vision. In IEEE Conference on
Computer Vision and Pattern Recognition, pages 2818–
2826, 2016.
[15] C. Wah et al. The Caltech-UCSD Birds-200-2011 Dataset.
Technical report, 2011.
[16] F. Wang et al. Normface: L2 hypersphere embedding
for face veriﬁcation. In ACM International Conference
on Multimedia, page 1041–1049, 2017.
[17] J. Wang et al. Learning ﬁne-grained image similarity with deep ranking. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1386–
1393, 2014.
[18] C. Wu et al. Sampling matters in deep embedding
learning. In IEEE International Conference on Computer Vision, pages 2859–2867, 2017.
[19] A. Zhai, H. Wu. Classiﬁcation is a strong baseline
for deep metric learning. In British Machine Vision
Conference, page 91, 2019.