Improve distance metric learning by learning positions of class centers
Authors: Kun Song
Lantian Chu
Junwei Han
Fakhri Karray
Venue: Submitted to ICLR 2023 (2023)
Keywords: distance metric learning
skewed mean function

Abstract: Deep metric learning aims at learning a deep neural network by letting similar samples have small distances while dissimilar samples have large distances. To achieve this goal, the current DML algorithms mainly focus on pulling similar samples in each class as closely as possible. However, pulling similar samples only considers the local distribution of the data samples and ignores the global distribution of the data set, i.e., the center positions of different classes. The global distribution helps the distance metric learning. For example, expanding the distance between centers can increase the discriminant ability of the extracted features. However, how to increase the distance between centers is a challenging task. In this paper, we design a genius function named the skewed mean function, which only considers the most considerable distances of a set of samples. So maximizing the value of the skewed mean function can make the largest distance larger. We also prove that the current energy functions used for uniformity regularization on centers are special cases of our skewed mean function. At last, we conduct extensive experiments to illustrate the superiority of our methods.

Under review as a conference paper at ICLR 2023
COMPACT BILINEAR POOLING VIA GENERAL BILIN- EAR PROJECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep metric learning aims at learning a deep neural network by letting similar samples have small distances while dissimilar samples have large distances. To achieve this goal, the current DML algorithms mainly focus on pulling similar samples in each class as closely as possible. However, the action of pulling similar samples only considers the local distribution of the data samples. It ignores the global distribution of the data set, i.c., the center positions of different classes. The global distribution helps the distance metric learning. For example, expanding the distance between centers can increase the discriminant ability of the extracted features. However, how to increase the distance between centers is a challenging task. In this paper, we design a genius function named the skewed mean function, which only considers the most considerable distances of set of samples. So maximizing the value of the skewed mean function can make the most significant distance larger. We also prove that current energy functions used for uniformity regularization on centers are special cases of our skewed mean function. At last, we conduct extensive experiments to illustrate the superiority of our methods.
INTRODUCTION
Deep metric learning (DML) is a branch of supervised feature extraction algorithms that con- strain the learned features, such that similar samples have small distance and dissimilar sam- ples have a large distance. Because having the ability to learn a deep neural network for un- seen classes, distance metric learning, i.e., the classes of testing classes do not appear in the training data set, DML are widely used in the applications of image classification & clus- tering, face re-identification, or general supervised and unsupervised contrastive representa- tion learning [Chuang et al. (2020). The goal of DML is to optimize deep neural networks to span its projection space lar samples have small distances, and the semantically dissimilar samples have large distances. This purpose can be formulated as a of triplets How- ever, because of the exponential amount of those triplets,
surface of hyper-sphere, in which the semantically simi-
distance metric learning needs an additional procedure
called as information sample selection, such as hard sam-
ple mining and semi-hard sample mining. With the advance of the mining techniques, state-of-the-art met- ric learning algorithms Waan et al. (2019b); Kim et al. (2020); Roth al. (2022); Wang & Liu (202 1); Schroff et al. (2015): [Sun et al. (2020): Deng et al. (2019);
Wang al. 2018) use the log-exp function (x)
1o9(>>; ( Oh Song et al. (2016) to combine the
distance metric learning and the information sample se-
The illustration of assigning the location of centers. eı is only pushed away by the six nearest centers. Because the push- ing directions is easy to stick. Therefore, the location as- signment fails.
Figure I:
lection together. an
Where 0 is the parameter of the deep metric, d1 and 52 are two tuned parameters, and S; and D; are the sets of sim- ilar and dissimilar samples of the query x,, respectively.
contrary, the position of eı
Although achieving excellent performance, those log - erp runated related algorithms fail to as- sign different classes' centers. Assigning the locations of class centers facilitates distinguishing the

---

Under review as a conference paper at ICLR 2023
features from different. For example, if can let the centers have significant distance in the features samples, thus the distance of two samples from those two classes will also be increased, making them easily distinguished.
There are several methods designed to enlarge the distance between centers. The most representative potential energy functions. However, those functions only consider the nearest centers of the query samples. Because the nearest centers are distributed around the query, the pushing actions shown from those nearest centers will let the query sample be stuck, i.e, the position of the query is hard to move in the training stage. As seen from the figure [ we consider the с₁ as the query centers, and the six centers in the circle are the nearest centers of the center C1. Because the current energy function only lets the nearest samples push the query samples away. the six nearest samples will let the query sample stuck because the directions of those pushing actions contradict each other. This makes the potential energy function fails to assign the locations of centers.
In this paper, we propose set of functions named skewed mean function, which can consider the largest values of a set of samples. In this way, we can let the sample pairs with largest distance be away from each other. Because those samples are on the boundary of the cluster of each sample. Those samples are less to be stuck. Using this finding, we design regularization term assign the centers of different classes. The contents include the following aspects:
We give a unified framework of proxies- distance metric learning. From our framework, summarize the traditional distance metric learning algorithms and the clssifiication-bssel distance metric learning algorithm together. Therefore, the math- ematical proof of their connections gives theoretical base for performing ablation experiments to support our comment that all problems of distance metric learning are about the Lipschitz constant.
We reveal that the potential energy-based methods have less power to push centers away from each other since they only consider local data information. To alleviate this problem, we adopt the log-exp mean functions and power mean function design the term to pull the centers of each class. Because we prove the potential energy methods are special cause of ours, our algorithms have the power to push centers of different classes.
We conduct extensive experiments on challenging data sets such CUB-200-2011, Cars196, Aircraft, and Inshop to illustrate the effectiveness of our algorithms.
Notation. X {(x, mi C'-class dataset where x; Rd the i-th sample and y; {1.-.., C} the label of x; z, fo(x:) Rd - Rd is deep neural networks parameterized by 0. The similarity between x; and x, is denoted as Aij cos(Jn x,), fo(x))) The set of proxies is denoted by X ((Wk, (.Wk:,))): ) where We R and yk {1. .C} is the corresponding cos(fa(x;), w;). Because proxy-based DML does not calculate the similarity between samples within X' or XP, the similar relationship between samples X + can be depicted by a bipartite graph. For x; E X2,, its similar samples are only in X and denoted as S. For w; E X its similar samples are only in X and denoted by S?. Likewise, dissimilar sample sets of x; E X and w, E X2 are denoted by D) and DR, respectively.
label of Wk. The similarity between x; and w, is denoted by B.j-
2 DISTANCE METRIC LEARNING REVISITED
3 SHORTCOMINGS OF DISTANCE METRIC LEARNING
In this section, we adopt the proxy anchor loss as the baseline to analyze current distance metric learning model. The objective function of proxy anchor loss is presented as follows.
J
log PEP+
1+
Σ e a((((x)))-
+
Σ log
1+
Σ
(1)
P
X
PEP
xEXt
2

---

Under review as a conference paper at ICLR 2023
where 8 >> 0 is margin, a >> 0 is a scaling factor, P indicates the set of all proxies, and Pt denotes the set of positive proxies of data in the batch. Also, for each proxy p. a batch of embedding vectors X is divided into two sets: X", the set of positive embedding vectors of p. and X, X X
The gradient of the loss function with respect to s(x, is given by
de a(s(a,))
Vx E Xp
o((x)
PT 1+ Fex! --
ps(8.p)
V e -l Vx E
-ae a(
In practice, the best performance of distance metric learning algorithms set a a large value. Nor- mally, 32. According to Eq.(2), we know large only focuses the farthest similar sample and nearest dissimilar samples in the optimization procedure. Only consider the nearest dissimilar samples means the distance metric leaning only consider the local distribution of the trainging data, and does consider the global information of the training set. As a consequence, the moving of the closest dissimilar sample will be stuck by other ignored dissimilar samples. Because those nearest samples will give each element a pushing force from its anchor or on is its. Therefoee,, we can claim that the goal of distance metric learning mainly depends on the shrinking of similar samples each class. In this way, the distance metric learning does not have the power to assign the centers of each classes. Geometrically, pushing centers away from each other will benefit the distinguish samples between different classes. For example, suppose the radius of the cluster region of each class be fixed as r, and the gap between two classes be If we let the centers of each class be pushing away from each other, the gap between two classes will also be enlarged, i.e., 5 e where is the amount increased by the pushing action for centers. In this way, the features extracted by neural networks will be easy to distinguish. Besides, if the centers are not assigned by the algorithm. When we want the gap between two classes still to be ö + +e, we should shrink samples in the cluster of each class significantly. However, the training samples of distance metric learning are not very much. For ex- ample, the widely used dataset in metric learning is CUB-200 200-2011111 has 200 classes with each class 69 samples on average. Compared with dimension of features extracted by neural network, normally being 512 or 1024, the number 69 is very smaller. In this way, it is hard to shrink so less samples in the high dimensional feature space without the overfitting. When the overfitting happens, the performance of distance metric learning will be hurt. Therefore, how to assign the centers of each class is an very important issue.
3.1 SHORTCOMINGS OF ENERGY FUNCTION
Several works proposed to assign the centers of different classes for classification problem. The well-known ones are the energy function based ones whose formulations are presented as follows.
C Eg.d (w: ( - Σ (.(((w,
{- E Eiey S(Wi, W Eiz log(S((:,, wy)) 8 0
0
(3)
where S(w,, wy) is a similarity function between W; and w,. Commonly, there S(w, w3) = |w; w,1B.
Let us calculate the gradient descent of energy function with respective to s(w,, Wy). there is w - - -ing S(w., w
aE,a( 1 Ōs(w; w;)
5 >
(4)
5 =
As seen from the above Eq. 4.. we know that if 5(W;, w3) is small, the value of gradient descent is large. It means the algorithm would give large weight to the sample pairs with smaller distance. Thus, the energy function only consider the closest samples of each query, and ignore the farther samples. This have two shortcomings: If only closest samples of each query considered, so the pushing action on this query sample is easy to be climinated by the samples around it. Considering there are hundreds of classes in each distance metric learning task, thus, such phenomenons easy encounter. And the centres can not be assigned to the whole surface of the hyper-sphere in the feature space.
In the following content, we design a new mechanism to solve this problem. That is we let the farthest samples of each query push the query. Because the farthest samples are always located
3

---

Under review as a conference paper at ICLR 2023
in the boundary of the region of the features, when we let the distance between them and query samples, it is hard to be stuck.
3.2 DISTANCE METRIC LEARNING SURVEY BY SKEWED MEAN FUNCTIONS
Definition 1. Given set of numbers S (2,,8 sN.,, without loss generality, by setting 0 <88 81 < <82 < .... < 8N, we can define K skewed mean of the numbers S as follows.
<<0
(S)
(5)
Lim1 8N-141
K>0
where K E (2,,+2,... +NJ. Obviously, MI-1] 81, M sN. and M(N
MI-MI(S) -
EE
As seen from the definition of the skewed mean functions, it is easy find the largest value from a set of numbers. If those numbers are the distances between a pair of centers, we can enlarge the skewed mean functions assign the position of centers. However, those skewed mean function involves the operation of ranking the numbers, which make the skewed mean function is not
continuous function with respective to the distance S(w,, w,). ) To solve this problem, we design series of continuous skewed mean function by introducing the following Theorem.
Theorem 1. Given a monotonously continuous increasing function fx(x): R : where R. and its inverse function = =fx (x)( R : IR,, we define a function presented as follows.
bs( = (A) EA(s)))
We calculate the K skewed mean of the numbers S = {81, 82. SN) defined in Eq. by using (q.16) with an appropriate selected X, if bs)) satisfies the following rules:
(A) a monotonously increasing function with respective to X
(2) limx bs(A) max and 1im (A) min
For the Theorem 1, give two examples of fx(x), ie., fx exx and fx(x) = x², which respectively corresponds to
N
bs(A,a) = = loga
1
Σ
N
N
bs(A) = N
Σ
Property 1. The functions b(X) and g(A) has the following properties:
(1) Both b(A) and g(X) are monotonically increasing functions with respective to X;
(2) limx-++oo b()) = and lim ((2) = O..
(3) lim 0a b(A) = a1 and 1im g(A) = 01, thus, there is an appropriate number X*
to let b(X") 0k g(X*) as where 0g is the k-th largest number in {a,)T
(4) Let a; = (x: x ' x) where M
pd is a distance metric, b(X) and -b(-))
are convex with respective to the matrix M when X > 0.
Remark 1. By using the skewed mean function, we can automatically select the k largest values of set of numbers or the smallest values. If we want to assign the centers of different classes in the features space, we should select the center pairs whose distance are large, and let those sample pairs with large distance be pushed away from each other. Because all of those samples on the surface of a sphere, the distances between those selected sample pairs have a maximal values. In this way, the algorithms will convergence.
4

---

Under review as a conference paper at ICLR 2023
3.3 REGULARIZATION PUNISHING LARGE SIMILARITY BETWEEN CLASS CENTERS
In this section, we design term to push centers of classes away from each other. Suppose {p, ) are centers classes and the similarity between P; and p, is denoted as s(p;. Then, we collect all similarities related to P as set denoted by MM.- {p P:]lī * c}. In Mes the r-th largest (r) element of is denoted by e M. In this way, there is a constraint e V.M. 8g to fulfill the above goal, whose continuous version is presented as follows.
(1)
R1 log
Σ
73
(8)
R2 = C(C ΣΣ PP..,,)))
1
izj
Thus, if we add the above regularization for the metric learning algorithm, we can achieve a new optimization problem which have the ability constrain the location of centers of different classes.
3.4 THE RELATIONSHIP BETWEEN EXISTING METHODS.
Let us introduce a relaxation of the constraint (1)
"Mo 83. Let us combine all (Ma) to one large set M = - Mc. the set {eM, |c= 1, .., C is subset of M. Therefore, the constraint (1) e.Me < 83 can be relaxed as & Σe= UMM < d3. In this way, by constructing continuous version of it, we can have an new regularization. Excepting the log-exp mean function, there another skewed
mean function can be used, i.e., g(A)
a²)x Thus, the new continuous constraint is
Σ
Σ s (P.
<
(9)
C - x=1 171
we set the similarity function as the negative distance, the constraint in Eq. 1 is presented as
CCC Σ d d(...,,)
)-))
> 83
(10)
If set À = -1, the left term in Eq. 10 is the energy based regularization proposed by in Uniform- face Duan et al. (2019). If we perform the operation (z) - on left term of Eq. 10). the minimum hyperspherical energy сπC- D Enzj d(... p,)) is obtained Liu et al. 2018). Because (æ)-> is monotonous decrease function with respect to X, the minimum hyperspherical energy has the same goal of Eq.(TOp.
For the regularization term used in Uniformface, to let À = -1 will reduce the flexibility of the algorithm to suit different types of data, because we know the À is a parameter related to the class number C. Different from Uniformface, the minimum hyperspherical energy term has parameter on sP, Py). However, > ... p,) will be very large with a relative small À s (P,, Pý) is small. Such large value will make its coefficient in the objective function hard tune. Thus, in practice, could not be selected too large. Actually, is set to 0, 1, 2 in Liu et al.) (2018). This means minimum hyperspherical energy term also lacks enough flexibility deal with different types of samples.
Besides the flexibility, the above mentioned two methods should calculate C(CC - 1)/2 times simi- larity, which is extremely large when the class number of the task is large. So many calculation will make the gradient update very slow. For example, in the face recognition, the class number can be more than 690K, so such the terms used in Uniformface and minimum hyperspherical energy term will cost plentiful computational resources. However, in our algorithm, we consider the [p.]c as nodes in the bipartite graph. Similar to samples in X. we can also only select small part of [p. to construct the objective function. In this way, our algorithm
lot of computational resource.
5

---

Under review as a conference paper at ICLR 2023
4
EXPERIMENTAL RESULTS
In this section, our method is evaluated and compared to current state-of-the-art methods on the four benchmark datasets for deep metric learning. We also investigate the effect of hyperparameters and embedding dimensionality of our loss to demonstrate its robustness.
4.1 DATASETS
We employ CUB-200-201 [Wah et. (2011), Cars-196Kraune et al. (2013), Stanford Online Product (SOP(Oh Song et al. (2016) and In-shop Clothes Retrieval (In-Shop) datasets Liu et al. (2016) for evaluation. For CUB-200-201 200-2011, we use 5864 images of its first 100 classes for training and 5,924 image of the other classes for testing. For Cars-196, 8054 images of its first 98 classes are used for training and 8131 images of the other classes are kept for testing. For SOP, we follow the standard dataset split in Oh Song al. (2166) using 59551 images of 1 11,318 classes for training and 60,502 images of the rest classes for testing. Also for In-Shop, we follow the setting Oh Song el al. (2016) using 25882 images of the first 3,997 classes for training and 28,760 images of the other classes for testing; the test set is further partitioned into query set with 14,218 images of 3,985 classes and a gallery set with 12,612 images of 3,985 classes.
4.2 IMPLEMENTATION DETAILS
For fair comparison previous work, the inception network loffe & Szegedy (2015) with batch normalization pre-trained for ImageNet classification is adopted as our embedding network. We change the size of its last fully connected layer according to the dimensionality of embedding vectors, and Lo-nmmlleie the final output.
Embedding network:
Training: In every experiment, we employ AdamW optimizer Loshchilov & Hutter (2017),
which has the same update step of Adam Kingma Ba (014) yet decays the weight separately. Our model is trained for 40 epochs with initial learning rate 10-4 on the CUB-200-201 and Cars-196, and for 60 epochs with initial learning rate 6 10 on the SOP and In-shop. The learning rate for proxies is scaled up 100 times for faster convergence. Input batches are randomly sampled during training.
Image setting: Input images are augmented by random cropping and horizontal flipping during training while they center-cropped in testing. The default size of cropped images is 224 x 224 as in most of previous work, but for comparison to HORDEJJaob et al. (2019), we also implement models trained and tested with 256 x 256 cropped images.
4.3 ABLATION EXPERIMENT ON DIFFERENT METHODS
To demonstrate the importance of the neighborhood parameter learning, we conduct an ablation study on CUB-200-201 I. Since the outer objective is a standard metric learning based on the log-exp function, therefore, we could instead it with the objective function of other type of metric learning algorithm, such as multi-similar loss Wang et al. (2019b), N-pair los4Sohn (201)). lifted-structure los+Oh Song et al. (2016). Proxy-nca loss Mosthovitz-AAttas et al. (2017) and the adaptive neigh- borhood metric learning Song et al. (2021).
The reason why we select those three methods to conduct the ablation experiment, is they are the special cases of the adaptive neighborhood metric learning [Song et al.] (2021). We adopt the re- formulated metric learning methods as the outer objective function of our methods, our bi-level learning framework could solve the neighborhood parameters and the metric parameter according to the algorithm ??. We utilize the dataset CUB-200-2011 to training those methods, and adopt the Recall @1 to evaluate the performance of them. The results shown in the Table As seen from the Table with the help of the neighborhood parameter learning, those well-known metric learning algorithms could be improved further in terms of performance.

---

Under review as a conference paper at ICLR 2023
Table Performance on the CUB-200-201 of the three state-of-the-ard methods and their improved versions with 512 dimension.
Recall@k
1
2
4
8
Lifted structure
Original
45.4
58,4
69.5
79.5
bi-level
47.1
60.3
71.6
81.8
Multi-similar loss
Original
49.2
61.9
67.9
72.4
bi-level
52.7
65.4
68.3
75.7
Original
43.6
56.6
68.6
79.6
N-pairs loss
bi-level
46.8
60.7
72.6
83.8
Proxy-NCA
Original
44.6
53.6
68.6
84.6
loss 512
bi-level
47.8
62.7
72.5
87.8
Original
65.4
76.8
85.7
90.7
DANMI 612
Improved
67.6
79.1
88.2
93.4
4.4 COMPARISON TO STATE-OF- THE-HE- METHODS
With further comparison the performance of our method with state-of-the-art techniques on image retrieval task, conduct the proposed methods on the CUB-200-2011 Car- 196, Stanford Online Product (SOP) and In-shop Clothes Retrieval (In-Shop) datasets. We adopt the recall@k as the metric to evaluate the performance of the related metric learning methods. The result shown in Table 2
As shown in Table our DANML improves Recall by 1.9% on the CUB-200-2011, and 1.5% on the Cars-195 196 over the recent state-of-the-art multi-similarity loss. This may be because the logistic loss function is more powerful than the linear function for generalization. Meanwhile, for recently proposed method Circle Loss, our DANML outperforms it about 0.9% on the CUB-200-200 I and 2.2% on the Cars-196 dataset. Compared with ABE which an ensemble method with much heavier model, our method achieves a higher @ecall@) 1 by 7.0% improvement on the CUB-200- 2011 and 0.4% on the Cars-196, dataset.
For the Stanford Online Products (SOP) and the In-Shop Clothes Retrieval (In-Shop), as seen from Tables and 3 our method outperforms multi-similarity loss by 1.7% on the In-Shop dataset and by 0.4% the SOP dataset, respectively. Furthermore, when compared with ABE, method increases Recall@ 1 by 3.6% and 2.8% on the In-Shop and SOP dataset, respectively. For the Circle Loss which is recent state-of tatte-of-theaasd art method on SOP dataset, our DANML achieves better per- formance about 1.6% on it.
4.5
IMPACT HYPERPARAMETERR
264
1524
2043
Déveraion
0.30 0.5
0.43
0.43
050
835
0.00
5366
Figure 2: The Recall@1 corresponds to different dimensions.
Figure The Recall@ corresponds to different r.
Embedding dimension: The dimension of embedding vectors is crucial factor that controls the trade-off between speed and accuracy in image retrieval systems. We thus investigate the effect of embedding dimensions the retrieval accuracy in our Bi-level metric learning framework. We

---

Under review as a conference paper at ICLR 2023
Table 2: Recall@ performance on CUB-200-2011 dataset and Cars-190 196 dataset. Superscript denotes embedding size.
CUB 200-2011
Cars- 196
Recall@KK
1
4
8
16
32
2
4
16
32
Clustering Oh Song al. 2017
48.2
61.4
71.8
81.9
58.1
70.6
80.3 87.8
ProxyNCA o Movshovitz Attias et al. (017))
49.2
61.9
67.9
72.4
73.2 82.4
86.4 87.8
Smart Mining Harwood et al. 2017
49.8
62.3
74.1
83.3
64.7 76.2 84.2 90.2
Margin Wu et al. (2017
63.6
74.4 83.1 90.0 94.2
79.6 86.5 91.9 95.1 97.3
HDC Wu et al.
2017)
53.6 65.7 77.0 85.6 91.5 95.5
73.7 83.2 89.5 93.8 96.7 98.4
HTL Ge 2018
57.1 68.8 78.7 86.5
95.5
81.4 88.0 92.7 95.7 97.4 99.0
ABIER Opitz et al. (0118)
57.5 68.7 78.3 86.2 91.9 95.5
82.0 89.0 93.2 96.1 97.8 98.7
ABE
2018)
60.6 71.5 79.8 87.4
85.2 90.5 94.0 96.1
Multi-similarity loss "1 Wang el al. 2019b
65.7 77.0 86.3 91.2 95.0 97.3
84.1 90.4 94.0 96.5 98.0 98.9
Hardness
"ardness-awwre" aware Zhhnng al 2019
53.7 65.7 76.7 85.7
79.1 87.1 92.1 95.6
Circle Loss Yifan Sun (2020 06.7
77.4 86.2
91.2
83.4
89.8
94.1 96.5
Ranked list loss" Wang al.
2019a
61.3 72.7 82.7 89.4
82.1 89.3 93.7 97.7
Ours
67.6 79.1 86.4
91.2 97.1
98.1
85.6 92.1
94.1 97.7 98.1 99.3
Table Recall@ K(%) performance on SOP dataset. Superscript denotes embedding size.
SOP
RecallOk
1000
Clustering Oh Song ct al. 2017
67.0
83.7
93.2
ProxyNCA Movshovitz- Catnnizz-tiii Attias et al. 2017
73.7
Smart Mining Harwood
2017
Margin Wu et al (2017)
72.7
86.2
93.8
98.0
HDC Wu ct 2017
69.5
84.4
92.8
97.7
HTT. 51. Go (2018)
74.8
88.3
94.8
98.4
ABTER Opitz
2018
74.2
86.9
94.0
97.8
ABE Kim et al. (2018)
76.3
88.4
94.8
98.2
Multi-similarity loss "ang et al. (019b)
78.2
90.5
96.0
98.7
Hardness-aware"" ZZheng al. 2019)
83.5
92.3
Ranked list loss 01 Wang al.
2019a
79.8
91.3
96.3
Cirele Loss 614
Yifan Sun
(2020)
78.3
90.5
96.1
98.6
512
79.9
98.9
Table 4: Reck)l@K(%%) performance In-Shop dataset. Superseript denotes embedding size.
In-Shop
Recall@k
10
20
30
40
50
NeshionNet 409K
Oh Song et al.
2017
53.0
73.0
76.0
77.0 79.0 80.0
HDC et 2017
84.9
89.0
91.2 92.3
HTL 522
Ge
2018
80.9
94.3 95.8 97.2 97.4 97.8
ABIER Opitz
2018
83.1 95.1 96.9 97.5 97.8 98.0
ABE
Kim el al.
2018
87.3
96.7
97.9 98.2
98.5 98.7
Multi similiiii loss
20196
89.7
97.9 98.5 98.8 99.1 99.2
Ours
90.1
98.2
98.9
99.0 99.3 99.4
8

---

Under review as a conference paper at ICLR 2023
test our loss with embedding dimensions varying from 64 to 1, 024 following the the experiment in Wang et al. (2019b), and further examine that with 32 embedding dimension. The result of analysis quantifted in Figure 2] in which the retrieval performance of our loss is compared with that of MS loss Wang et al. 2019bj. The performance of our loss is fairly stable when the dimension is equal to larger than 128. Moreover, loss outperforms MS loss in all embedding dimensions, and more importantly its accuracy does not degrade even with the very high dimensional embedding unlike MS loss.
Parameter in our method: We also investigate the effect of the hyperparameter of our method on the Cars-196 dataset and CUB-200-211 The results of our analysis are summarized in Figure 3 in which we examine Recall @ 1 of the proposed bi-level metric learning loss by varying the values of the parameter r E (0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.66, 0.7}. For CUB-200-21 and Cars- 196, the results suggest that when near 0.55 and 0.45, the proposed bi-level metric learning achieve the best performances, respectively. The results indicate the performance of the proposed bi-level methods is sensitive to the parameter X, so we should carefully chose r in the proposed bi-level distance metric learning algorithm. The r the gap between two negative classes which determines the lower- bound of the Lipschitz constant of the learned deep neural network network. That is why the performance of the proposed methods is sensitive to the value of r.
CONCLUSION
In this paper, we reveal that learning the position centers for each class is very important to metric learning. However, current potential energy- beaeey regularization has less ability constrain the position of centers because it considers the nearest centers each query center. The pushing actions given by the nearest centers the query center contradict each other. To overcome this shortcoming, we design a function named skewed mean function, which can be used to calculate the most considerable distances of of numbers. Using the skewed mean function, give new center regularization, which considers center pairs with farthest centers. The conducted experiments illustrate the effectiveness of our proposed method.
REFERENCES
Ching- Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. De- biased contrastive learning. Advances in neural information processing systems, 33:8765-8775, 2020.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4690-6699 2019.
Yueqi Duan, Jiwen Lu, and Jie Zhou. Uniformface: Learning deep equidistributed representation for face recogmition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2019.
Weifeng Ge. Deep metric learning with hierarchical triplet loss. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 269-285, 2018.
Ben Harwood, BG Kumar, Gustavo Carneiro, Ian Reid, Tom Drummond, et al. Smart mining for deep metric learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2821-2829 2017.
Sergey loffe and Christian Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. In International Conference on International Conference on Machine Learning, pp. 249-256, 2015.
Pierre Jacob, David Picard, Aymeric Histace, and Edouard Klein. Metric learning with horde: High- order regularizer for deep embeddings. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6539-65444 2019.

---

Under review as a conference paper at ICLR 2023
Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 2238-3245 2020.
Wonsik Kim, Bhavya Goyal, Kunal Chawla, Jungmin Lee, and Keunjoo Kwon. Attention-based ensemble for deep metric learning. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 736-751, 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:1412.6980 2014.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- ane-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Work- shops, pp. 554 2013.
Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning towards minimum hyperspherical energy. Advances in neural information processing systems, 31, 2018.
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1096-1104, 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXhvv1711.05001, 2017.
Yair Movshovitz-Attias Alexander Toshev, Thomas K Leung, Sergey loffe, and Saurabh Singh. No fuss distance metric learning using proxies. In Proceedings of the IEEE International Conference on Computer Vision, pp. 360-368, 2017.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4004 4012, 2016.
Hyun Oh Song, Stefanie Jegelka, Vivek Rathod, and Kevin Murphy. Deep metric learning via facility location. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5382-5390, 2017.
Michael Opitz, Georg Waltner, Horst Possegger, and Horst Bischof. Deep metric learning with bier: Boosting independent embeddings robustly. IEEE transactions on pattern analysis and machine intelligence, 2018.
Karsten Roth, Oriol Vinyals, and Zeynep Akata. Non-isotropy regularization for proxy-based deep metric learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7420-7430, 2022.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. 2015.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Advances in Neural Information Processing Systems, pp. 1857-1865, 2016.
Kun Song, Junwei Han, Gong Cheng, Jiwen Lu, and Feiping Nie. Adaptive neighborhood metric learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss: unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6398-6407, 2020.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2495-2504, 2021.
10

---

Under review as a conference paper at ICLR 2023
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2265-5274, 2018.
Xinshao Wang, Yang Hua, Elyor Kodirov, Guosheng Hu, Romain Garnier, and Neil M Robertson. Ranked list loss for deep metric learning. arXiv preprint arXiv:1903.03238, 2019a.
Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott. Multi-similarity loss with general pair weighting for deep metric learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 5022-5030, 2019b.
Chao-Yuan Wu, R Manmatha, Alexander J Smola, and Philipp Krahenbuhl. Sampling matters in deep embedding learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2840-2843 2017.
Yuhan Zhang Chi Zhang Liang Zheng Zhongdao Wang Yichen Wei Yifan Sun, Chang- mao Cheng, Circle loss: A unified perspective of pair similarity optimization. arXiv preprint
arXiv:2002.10857, 2020.
Wenzhao Zheng, Zhaodong Chen, Jiwen Lu, and Jie Zhou. Hardness-ane wware In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 72-81, 2019.
aware deep metric learning-


Reviews:
[{"review_id": "v5AeJATVOeB", "reviewer": "Reviewer_CgdK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors claim that existing distance metric learning methods update the distance metric based on local relationships and ignore the more global ones (however, this claim is not right). The authors argue that updating distance metric based on the closest class centers has some drawbacks and they claim that the furthest distances must be also used for distance metric learning. To maximize the distances between the class centers they utilize skewed mean function, but it is not clear how this information is used. The authors also show some relations between their proposed methodology and the existing ones and they claim that the existing methods focusing on uniform distributions through energy functions are special cases of their proposed method. Lastly, the authors conduct some experiments on different datasets and report better accuracies compared to the existing methods in the literature.", "summary_of_the_review": "This is badly written paper. There are frequent English Grammar mistakes, typos and annotation mistakes. Some of the equations is not complete and there are inconsistencies among the notations. All these things made reading the paper extremely difficult. Also, some of the claims are not correct. This paper is clearly not ready and it needs a serious proofread and rewriting. Experimental evaluations is also not complete. Therefore, the paper must be rejected in my opinion.", "strength_and_weaknesses": "The only strength of the paper is the reported better accuracies compared to the existing methods. It seems the authors proposed a new method, but the novelty is limited.\nHowever, there are many weaknesses regarding the paper and the proposed methodology. These can be summarized as follows:\ni) First of all, the paper is written very badly. The mistakes start at the first page, the authors forget to include an equation, but they describe the parameters of the missing equation. Also, there are many annotation mistakes as well. The following statements describing the motivations do not make any sense at all: “Because those samples are on the boundary of the cluster of each sample. Those samples are less to be stuck. Using this finding, we design a regularization term to assign the centers of different classes.” “… to support our comment that all problems of distance metric learning are about the Lipschitz constant.”\nii) The claim that the most existing methods only consider the local relations between classes and ignore the global ones is incorrect. For example, the UniformFace discussed in the paper targets to learn equi-distributed representations for face recognition. It considers all the relations between class centers. However, the distance metric is updated based on the closest class centers, and it makes perfect sense the closest samples are the hardest ones for separation. The classes far from each other can be already separated easily.\niii) There are methods that update the centers based on the feature representations such as the methods using center loss [1] and discriminative centers [2]. They also consider both local and global relations among the classes and centers are updated dynamically. These methods should be also discussed in the paper. \niv) The authors criticize the methods such UniformFace and methods that target diverse deep neural network weights that are uniformly distributed on a hypersphere. Yet, they do not make any experimental comparison. Especially, comparing the proposed method to UniformFace on large-scale face recognition problems is necessary to support their claims.\nv) There are also mistakes in the tables. There are better accuracies outperforming the authors’ proposed method that must be indicated in bold characters in the tables.\n\nReferences\n[1] Y. Wen, K. Zhang, Z. Li, and Y. Qiao. A comprehensive study on center loss for deep face recognition. International Journal of Computer Vision, 127:668–683, 2019.\n[2] B. Uzun, H. Cevikalp and H. Saribas, \"Deep Discriminative Feature Models (DDFMs) for Set Based Face Recognition and Distance Metric Learning,\" in IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ARa4OMxiJw", "reviewer": "Reviewer_tGz6", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method to expand the distance between centers of different classes. However, the contribution is not clear, and the article is an unfinished work. In section 2, nothing is revisited but the title. The writing is terrible. There are too many spelling and grammatical errors in Section 3. In general, this paper is far from acceptance.", "summary_of_the_review": "As I stated above.", "strength_and_weaknesses": "Strength: the studied problem is interesting.\n\nWeaknesses: the contribution is not clear, and the article is an unfinished work. In section 2, nothing is revisited but the title. The writing is terrible. There are too many spelling and grammatical errors in Section 3. In general, this paper is far from acceptance.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7nhBqnm7YId", "reviewer": "Reviewer_SAjL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the global distribution of the dataset in deep metric learning, and introduces the skewed mean function to only considers the most considerable distances of a set of samples. The paper also proves that current energy functions are special cases of the skewed mean function. Extensive experiments are conducted to illustrate the superiority of the proposed method.", "summary_of_the_review": "See major weaknesses in Strength And Weaknesses.", "strength_and_weaknesses": "Strength:\n+ This paper designs the skewed mean function used in deep metric learning to only considers the most considerable distances of a set of samples.\n\nWeaknesses:\n- The title of this paper is \"COMPACT BILINEAR POOLING VIA GENERAL BILINEAR PROJECTION\", which is not appropriate for the content.\n- The Section \"2 DISTANCE METRIC LEARNING REVISITED\" is not complete.\n- The results of the proposed method on Table 2 & Table 3 are not SOTAs.\n- The paper is not well-written.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject