Determinant regularization for Deep Metric Learning
Authors: Kun Song
Ruben Solozabal
Martin Takáč
Fakhri Karray
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep Metric Learning
Generalization
Jacobi Matrix

Abstract: Distance Metric Learning (DML) aims to learn the distance metric that better reflects the semantical similarities in the data. Current \textit{pair-based} and \textit{proxy-based} methods on DML focus on reducing the distance between similar samples while expanding the distance of dissimilar ones. However, we reveal that shrinking the distance between similar samples may distort the feature space, increasing the distance between points of the same class region and, therefore, harming the generalization of the model. Traditional regularization terms (such as $L_2$-norm on weights) cannot be adopted to solve this issue as they are based on linear projection. To alleviate this issue, we adopt the structure of normalizing flow as the deep metric layer and calculate the determinant of the Jacobi Matrix as the regularization term. At last, we conduct experiments on several \textit{pair-based} and \textit{proxy-based} algorithms that demonstrate the benefits of our method.



Under review as a conference paper at ICLR 2023
DETERMINANT REGULARIZATION FOR DEEP METRIC LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Distance Metric Learning (DML) aims to learn the distance metric that better re- flects the semantically similarities in the data. Current pair-based and proxy-based methods on DML focus on reducing the distance between similar samples, while expanding the distance of dissimilar ones. However, we reveal that shrinking the distance between similar samples may distort the feature space, increasing the distance between points of the same class region, and therefore, harming the gen- eralization of the model. The regularization terms (such as L2 L2-norm on weights) cannot be adopted to solve this issue as they are based on linear projection. To alleviate this issue, adopt the structure of normalizing flow the deep metric layer and calculate the determinant of the Jacobian matrix as a regularization term that helps in reducing the Lipschitz constant. At last, conduct experiments several pair-based and proxy-based algorithms that demonstrate the benefits our method.
INTRODUCTION
Deep metric learning (DML) is branch of learning algorithms that parameterizes deep neural network to capture highly non-linear similarities between images according to a given semantical relationship. Because the learned similarity functions can measure the similarity between samples that do not appear in the training data set, the learning paradigm of DML is widely used in many applications such as image classification & clustering, face re-identification. or general supervised and unsupervised contrastive representation learning Chuang et al. (2020). Commonly, DML aims to optimize deep neural networks to span the projection space on a surface of hyper-sphere in which the semantic similar samples have small distances and the semantic dissimilar samples have large distance. This goal can be formulated as the discriminant criterion (and its many variants that appear in the literature) we summarize as follows.
max (do xÿ)l E S.) < & < 6g < min(fo x:) E D,}
(1)
where 0 are the parameters of the deep metric model, 6, and 82 are two tunable hyperparameters, and S; and Dj are the sets of similar and dissimilar samples of the query x,, respectively.
Commonly the log-exp function (0) 1 eaa(())) Oh Song et al. (016 is used to define the objective function in DML. Besides the definition of the objective function, many works point out that the performance of DML crucially depends on the informative sample mining (HSM) procedure and therefore focus their research direction on improving the HSM. Unfortunately, the explicit definitions of informative samples is still unclear, and the problem seems to be unsolved. This leads us to the following question: what is the real reason that makes DML model so erucially depend hard sample mining? In this paper, we try to answer this question studying the local I Lipschitz constant of the learned projection fx().
Figure 1: The illustration represents the feature space spanned by fx(x) learned by deep metric learning.
the radius i-th class of samples in training dataset and r. the radius of unknown class. ó2 ô, reflects the distance between the two closest samples in class and class 2.

---

Under review as a conference paper at ICLR 2023
Many recent advances have been presented on DML in many new excellent works Wang et al. (2019): Kim et al. (2020); Roth et al. (2022); Wang & Liu (2021); Schroff et al. (2013); Sun et al.
2020 Deng al. (2019;; Wang et al. 2018) since the first DML model was proposed in 2015
Schroff et al. (2015) J. Current methods present good generalization i.e., the learned metric can work well on the unseen classes. We attribute the good generalization performance of DML, models to the fact that the learned function fx(x) extends in a continuous manner on the projecting region presenting small Lipschitz constant w.r.t the origin sample space. As we know, DML reduces the distances between similar samples which also reduces the local Lipschitz constant of fo fx) of the region surrounded by training samples. The continuity of fo fx(x) prevents the Lipschitz constant from changing fast, so (x(x) presents small Lipschitz constant points close the region where training samples are located. The learned projection having small Lipschitz constant induces a smaller upper bound on the empirical loss, which results on better generalization performance. This interpretation explains why the projection fo (x) learned by distance metric learning has good generalization even unseen classes.
However, DML presents some drawbacks. Shrinking the cluster of each class increases the local Lipschitz constant of fx(x). This phenomenon can be divided into two effects. The first corre- sponds to the increase of the Lipschitz constant caused by enlarging the distances between dissimilar samples, which was found by Song al. (202T). This effect increases the local Lipschitz constant of fo(x) at the region between classes. The author of Song et al. (2021) commented that the failure of training triplet loss without semi-hard sample mining can be attributed to it. In the second case, the regions with large Lipschitz constant are a priori unknown. This effect may occur in the region that belongs to unknown regions or regions that belong to a cluster for a class. In the first case, the negative effect can be alleviated reducing the distance between dissimilar samples. Based on this strategy, (Song et al. (202T) designs an KKN decision boundary to formulate a general framework of distance metric learning and proved that current state-of-the-art algorithms such as lifted struc- ture loss, multi-similarity loss, circle loss or N-pair loss are special cases of it. Those implicitly mean that the sample mining strategies are designed to reduce the Lipschitz constant of the learned projection.
Regarding the second case, there are a few works that address this issue. A common strategy is to assign the position of centers of each class by minimizing designed energy functions Duan et al.) (2019) et al. (2018). The main assumption of this method is that if distances between centers large, there is no need to shrink each class too much while still preserving the classes gap. However, this method has only positive results if it increases the Lipschitz constant at the region of unknown cases. If the increasing regions occur within each class, this routine will fail to work.
In summary, we claim that current methods in distance metric learning can improve its dis- criminant ability and reduce the Lipschitz constant of the learned problem at the same time by applying the right regularization factor. In this paper, we design a framework demonstrate this. The contents include the following aspects:
We give an unified framework of proxy-based distance metric learning. In our framework, we summarize the traditional distance metric learning algorithms and classification-basee distance metric learning algorithms together. Therefore, we present the mathematical framework that proves the connections between these methods and it gives theoretical base to support our hypothesis on the effects of the Lipschitz constant on distance metric learning.
We reveal that potential energy-based methods are not very effective in pushing centers away from each other since it only consider local information of the data. To alleviate this problem, we adopt the log-exp mean functions and power mean function to design a loss term that pulls away centers of each classes. Because we prove that potential energy methods are a special realization of our algorithm, ours also has the power to separate centers of different classes.
To further solve the Lipschitz constant problem on distance metric learning we design a deep neural network structure that allow to minimize the Lipschitz constant of the deep neural network directly. This structure contains two parts: the first part extracts features using traditional backbone networks, such Resnet, VGG, and Inceptions; the second
2

---

Under review as a conference paper at ICLR 2023
part learns the non-linear metric by invertible deep neural layer used in the Normalizing Flows, whose gradients with respect to the input are easy to compute.
We conduct extensive experiments on challenging data sets such as CUB-200-2011 I, 1ars196, Standord Online Products, and In-Shop Clothes Retrieval to illustrate the effec- tiveness our algorithm.
Notation. We denote as X = {(x, 1i)
Ni the C'-llas dataset where x; E Rd is the i-th sample and ys E {1.... .C} is the label of x, Z, = fo(x;) R + R is a deep neural networks parameterized by 8. The similarity between x; and x, is denoted A. cos(fo(X;), fo (xz)). The set of proxies is denoted by X = [(W+)::))k= k where W1: E RR and yk {1, C] is the corresponding label of We. The similarity between x, and w, is denoted by Bıy = (xx(f((x)) wyy) Because proxy-based DML does not calculate the similarity between samples within X' or X', the similar relationship between samples X X can be depicted by bipartite graph. For X; E X its similar samples are only in X and denoted as St. For W, E X., its similar samples are only in X and denoted by S?. Likewise, dissimilar sample sets of X, X and w, E XP are denoted by D! and D,,, respectively.
MOTIVATION
Let us consider deep neural network fx) RR : Rd where 0 is the learnable parameter, x is the image vector, and z is the feature vector of the image vector x. Normally, in DML the norm of is equal to because the Lz-normalization is often used on the last layer of fo (x).. Therefore, for a training dataset X₁ {xx y:) ) and a testing data Nee = (x. {x)) the samples of them are all projected the surface of d-dimensional sphere centered the origin of the feature space.
Let us denote the surface of the considered d-dimensional sphere by S. Because in classification- based tasks, the features vectors of different classes are desired to be separated from each other, the features from different classes are located in different clusters on S. Without loss of generality, we suppose that each class of samples belong only one cluster, and the cluster region located by the k-th class is denoted by Sy. Therefore, the set S is divided into C' + 1 parts. Besides the C' regions {S&) there one region without any samples located in, which called as blank region. In the open set problem, the blank region is specified to the unknown classes. In distance metric learning, the blank region of the training features may be located within the region of the testing features. Because there is no overlapping between classes of the training and the testing sets, therefore, S = B S
Then, would like to demonstrate why shrinking the samples will increase the Lipschitz constant of the learned projection. Before doing this, we introduce the definition of the Lipschitz constant.
Definition. Let (X, ) and (J), d²)) be two metric spaces, the Lipschitz constant of function is defined as:
d f(x)),
Lip(f) = max X.EEAA
(2)
d (x1,
Let us consider two projection (x) and fx(), their projecting regions on S are (SLIC + B and (S)) C
k= + B2, respectively. Thus, if the area of {S is larger than the area of ( the area of B² is smaller than that of B², Therefore, we can find two samples x, x,, whose projections are in 1 and 2 Thus, the following constraint holds
dy (f1 (x
dy (x((x))))
(3)
Exxxx,X) N6)
(x(xxxx)) X6)
The Eq. 3 indicates that the Lipschitz constant of the fi is smaller than that of f2.
However, for distance metric learning tasks, the dimension of S is very small when compared to the number of training samples. For example, the CUB200-2021 dataset has 11,788 samples with 200 classes. Each class has less than 60 images on average. Thus, Sg, the region of the k-th class may be not a connected to the neighborhood of another class, otherwise we consider that both belong to the same one. As seen from Figure between two similar samples there could be space belonging
3

---

Under review as a conference paper at ICLR 2023
to the blank region. Thus, when we shrink the distance between samples of each class, the regions involving the increasing Lipschitz constant may probably be within the blank areas between two similar samples. When this happens, the generalization ability of fi would be harmed.
For a good training on the deep neural network, we want to learn the distance metric learning without increasing the Lipschitz constant within the cluster of each class.
3 ANALYSIS ON THE LIPSCHITZ CONSTANT OF NEURAL NETWORK
In this section, we introduce how to control the Lipschitz constant of our deep network. Before doing this, introduce useful lemma about the Lipschitz constant of deep layer-based projection.
Lemma 1. [eever [2018) Given a T Tlayer deep projection fo(x") [T( (xx)) param- elerized with U, the i-th layer is x [x(x"). Let L5, and Li denote the Lipschitz constant of fe(x') and fx there is an equation of T.fo constrained by
L. -II L
(4)
According to 4), the Lipschitz constant of (x) can be constrained calculating the individual contribution of each layer {L = 1, ,,..,.... ,T). Let us introduce another lemma which gives a tighter upper bound to the Lipschitz constant of the projection Lic.
Lemma 2. Given deep neural network, the Lipschitz constant of its i-th layer presents the follow- ing relationship.
lff((1)
fx2)111 | x2
0f: 8x
Lf.
max XLA
max X'E
|1(
)|= 11
max
(25')2
(5)
where XY is the i-th singular value of the matrix
8f 0x x=x'
Proof. Suppose fi is a continuous function, according to the Taylor equation there is a x' such 0x.(x) dx x=x' Thus, maxx,,22 N1 N2-
that fi(x)) fi(X1) Ax (X2 x1), where Ag
[[
maxx || Denoting the singular values of A by (N) 1d (x2-x1)² " A,
maxx x1.3g
[1] x2ll
JiE1
there is ||Ax Tr Av) = E Thus, L5, = max EE=(
Remark 1. The above Lemma estimates the Lipschitz constant of a projection by calculating partial gradient of f((x) with respect x. Thus, if we reduce the F-norm of the partial gradient matrix at all the samples in the training dataset, we can let the learned projection have a smaller Lipschitz constant. If fs represents a linear projection: y y= LL'xx. Because Df. constant corresponds to Lt. | which is widely-used regularization term to improve the generalization ability on deep learning models.
dx = LL,, the Lipschitz
Distance metric learning learns a representation where samples in the same class present small distance, and samples from different classes a large distance. Thus, for the deep neural networks fo trained on this metric, the lipschitz constant of fo will naturally increase in the black region and decrease in clusters. However, if we use the term in Fq. to minimize the Lipschitz constant of fg. then we would reduce the Lipschitz constant of the overall problem. Such a result contradicts the goal of distance metric learning. Because it harms the discriminant ability of the model, but in return it improves generalization.
Thus, instead of using Σd= (dr;), we introduce the following term to regularize the Lipschitz constant.
d F. = log [I ( = 2
log (x7
(6)
1=1
Geometric meaning. Suppose A, is the Jacobian matrix of = fx( at 0(x,r) = [x||x'
r) is neighhorhood of x', the volume of 0(x,r) [ if 0. Suppose the singular
4

---

Under review as a conference paper at ICLR 2023
values of Ax are Xi Xσ ....>> X/. The volume of fi(O(x,)) r)) is IT (dzz), thus, I X the volume changed after projection.
Therefore, if Ry is reduced, the value || = Ei (> )² is reduced. But the difference from A is that minimizing Ry. will let small (X become smaller at the procedure of minimizing
Therefore, propose regularization term to minimize the Lipschitz constant according to the metric learning requirement.
R(x,) - log 22 +1)
4 DEEP METRIC LAYER
From the implementation point of view, the formulation involved in the calculus of det( (xx()) intractable for traditional neural networks. To alleviate this problem, we design an non-linear projection layer for whom the determinant of its Jacobian matrix is easy to solve. Here, we adopt the deep neural network used in Normalizing Flows Rezende Mohamed (2015) where it is efficient to solve det( ()((())
8x
Let h(:;0) - R be a bijection parameterized by 8. Then, the desired projection is R + R.D. which projects each sample x E TR. as y = gx).. Let the 1-th entries of x and y be xa and yt. the projection is defined as
Vt h O, (xi+-))
where X1:1 (#1, the set of all parameters, and O is a constant.
21). For t 2,... D choose arbitrary functions O₂(-) mapping R to
The jacobian matrix of the Eq. (9) is triangular. Each output ye only depends on X1 and so the determinant is just the product of its diagonal entries,
dy: Or,
det(Dg) -e
The structure of Normalizing Flows is an invertible projection which can not be used to reduce the dimension of the input samples. So the input of the Normalizing Flows not be images, thus need employ a backbone based on convolutional neural networks to extract the features of images and at last those features fed into the Normalizing Flow layers. Therefore, the regularization pre- sented in the previous section can not be used to constrain the Lipschitz constant of the convolutional neural networks. Let consider the Eq. of the structure of Normalizing Flows, two results may occur. (1) the Lipshcitz constant of whole structure of our deep neural network will be reduced. (2) the Lipschitz constant of the convolutional neural works will be larger. To avoid the second situation, we use dueling Normalizing Flows architecture connecting to the backbone. For the first one, the regularization minimized, while for the second one, its regularization term is maximized. During the inference, only the minimizing deep metric layer are used. By doing this, we can let the Lipschitz constant the backbone be stable while reducing Lipschitz constant of the combined network. The Structure of the proposed method depicted in the Figure According to the property of the invertible structure of normalizing flow, is an upper triangle matrix. Thus, there is an equation that
find that if only constrain the Lipschitz constant
the Jacobi matrix dy: 0x,
Oy: + +1)
R(x,)
log { +1) d
log
(10)
Ox,
where is an identity matrix.
Because the R(x;) is function of x, If we have the sample set 2 {21, 2, then R(2) = Σi R(z,).
The Eq.(I) lets the regularization of the Lipscthiz constant be small for each training sample. How- ever it does not reduce the Lipsethiz constant between dissimilar samples. To solve this problem, sample augmentation reduce Lipschitz constant in those regions.
5

---

Under review as a conference paper at ICLR 2023
Loss Function
Regularization
Loss Function
Regularization
Backbane
Image
Deep metric layer
For feature extraction
Structure of the dueling deep metric neural network used to reduce the overall Lipschitz constant of the network.
Figure 2:
Sample augmentation. Let = fx;) be the output feature embedding of the backbone. Then, for point }z, we can be compute the direction eij Z; Zj. Then, a new sample can be generated sampling from Zij = Z, + &.j.. By selecting an appropriate value E, we can find that the new sample Zy belongs to the blank region between 7, and 2j- In this way, if we minimize the Lipschitz constant on the generated samples, we can reduce the Lipscthiz constant of points within the cluster of each class, In order to efficiently select the number of samples, neighborhood of x,. In this way, we produce extra training samples for the batch. Because those augmented samples are located between reduced, the distance between the augmented sample and each of the two similar ones also does. Therefore, there is need to perform the distance metric learning the augmented samples.
only select the samples X, in the
similar samples, if the distance between the samples
Thus, the objective function of the proposed distance metric learning is presented as follows.
min 10L00..00 (a)
loss((x,, D.), + Ay (2,) AgR(Z2)
(11)
where 0 the learnable parameter of the backbone, 02 is the learnable parameter of the parameter of the first deep metric layer, 0g is the second deep metric layer. X and X₂ 0 are the coefficient factors of the regularization terms. 2, is the training samples and their augmented samples for the first invertible neural network, and Zz is the training samples and the augmented samples for the second invertible neural network.
After the training of the model, the outputs of the first invertible networks are considered as the features of images.
5 EXPERIMENTS
We evaluate the effectiveness the proposed method on four datasets for fine-grained image re- trieval. In the conducted experiments compare the performance of the proposed regularization factor when applied on current state-of-thh-ar models in DML.
5.1 SETTINGS
Fine-grained image retrieval. We benchmark our model in four datasets on fine- ine-ggained image retrieval: 1ars196, CUB-200-2011, Standord Online Products (SOP) and In-Shop Clothes Retrieval. CARS-196 contains 16, 183 images from 196 classes of cars. The first 98 classes are used for training and the last 98 classes are used for testing. CUB-200 CUB-200-2011 contains 200 different class of birds. We use the first 100 class with 5,864 images for training and the last 100 class with 5,924 images for testing. SOP is the largest dataset and consists of 120,053 images belonging to 22,634 classes of online products. The training set contains 11,318 classes and includes 59,551 images, the rest 11,316 classes with 60,499 images used for testing. Lastly, the In-shop dataset consists of a total of 54,642 images which are divided in 25,882 images from 3997 classes for training and 28,760 images for testing.

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison of the Recall@k (in percentile) on the CUB-200-2011 and the Cars-196 fine- grained image datasets. The backbone network is denoted by: G for GoogleNet, R50 for ResNet50 and BN for Inception with Batch Normalization. The superseript indicates the size of the final embedding layers used in the backbone network. Source: Kim et al. 2020).
CUB-200-2011
Cars-196
Recall
k=
k=2
k=4
k=8
k=
k=2
k=4
k=8
Clustering
BN
48.2
61.4
71.8
81.9
58.1
70.6
80.3
87.8
Proxy-NCA
BN
49.2
61.9
67.9
72.4
73.2
82.4
86.4
87.8
Smart Mining
G
49.8
62.3
74.1
83.3
64.7
76.2
84.2
90.2
MS
BN
57.4
69.8
80.0
87.8
77.3
85.3
90.5
90.2
SoftTriple
BN
60.1
71.9
81.2
88.5
78.6
86.6
91.8
95.4
Proxy-Anchors
BN
61.7
73.0
81.8
88.8
78.8
87.0
92.2
95.5
Margin 128
R50
63.6
74.4
83.1
90.0
79.6
86.5
91.9
95.1
HDC
53.6
65.7
77.0
85.6
73.7
83.2
89.5
93.8
A-BIERS12
G
57.5
68.7
78.3
86.2
82.0
89.0
93.2
96.1
ABE
G
60.6
71.5
79.8
87.4
85.2
90.5
94.0
96.1
HTL
BN
57.1
68.8
78.7
86.5
81.4
88.0
92.7
95.7
RLL-H
BN
57.4
69.7
79.2
86.9
74.0
83.6
90.1
94.1
5S
BN
65.7
77.0
86.3
91.2
84.1
90.4
94.0
96.5
SoftTriple
BN
65.4
76.4
84.5
90.4
84.5
90.7
94.5
96.9
Proxy-Anchor-12
BN
68.4
79.2
86.8
91.6
86.1
91.7
95.0
97.3
Ours 512
BN
69.1
80.1
87.2
92.2
87.2
92.2
95.3
97.4
5.2 IMPLEMENTATION DETAILS
We use Inception as the backbone for our model. Similar to previous works in the literature, we use pre-trained model on ImageNet for classification and select a final 512-D embedding layer that would correspond to the dimension of the hidden layers on the deep metric model. For the backbone we freeze the batch normalization layers during the training and we add an activation layer for the connection to the deep metric layers. The deep metric layer use are invertible normalizing flows layers, as the determinat of the Jacobian matrix required in regularization factor is efficient to compute. Particularly, implements normalizing flows using affine coupling layers that combine a scaling term with a shit term in the transformation. Despite its simplicity Real-NVP has shown be effective estimating complex density distributions without requiring a high number of layers. In our experiment we rely on 12 layers for each of the dueling deep metric modules capture the radial distribution proceed by the cosine similarity in DML. Finally, we use Lg-normalization on the final output of the normalizing flows.
rely Real-NVP Dinh et al. (2016), model that
Regarding the loss function, we rely on the state-of ofte--of-the-art Proxy Anchor loss Kim et al. (020 for our experimentation. Proxy Anchor proposes a proxy-based anchor method that associates the en- tire data in the batch with proxies for each class. This method has shown advantages w.r.t. previous proxy-based method that do exploit data-to-data relations and has also shown be more effi cient than pair- based methods. In particular, we define the same number of proxies as classes in the dataset. The loss function is completed with the determinant-ban regularization factor presented Eq.
Training: To train our model we rely on the weight decay AdamW optimizer Cewnniizzeroshhhhhiee Loshchilov & Hutter 2017). The initial learning rate is 10-4 for the backbone network and 10 for the metric learning layers and the proxies.
use lincar decay in both cases and we train the model for 30 epochs. Similarly to Kim al.) (2000) initialize the proxies using normal distribution and bigger learning rate used on them for a faster convergence.
We maintain the hyperparameters 8 for the margin and scaling factor of Proxy Anchor 32 and 0.1, respectively, all experiments. We set the coefficients X and 22 to 0.05 for the dueling deep metric layers.

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison of the Recall @k (in percentile) the Standord Online Products (SOP).
SOP
Recall
k=1
k=10
k=100
k=1000
Clustering
67.0
83.7
93.2
Proxy-NCA*
73.7
MS
74.1
87.8
94.7
98.2
SoftTriple
76.3
89.1
95.3
Proxy-Anchor
76.5
89.0
95.1
98.2
Margin 128
72.7
86.2
93.8
98.0
HDC
69.5
84.4
92.8
97.7
A-BIERS12
74.2
86.9
94.0
97.8
ABE$12
76.3
88.4
94.8
98.2
HTL 512
74.8
88.3
94.8
98.4
RLL--H512
76.1
89.1
95.4
MS
78.2
90.5
96.0
98.7
SoftTriple
78.3
90.3
95.9
Proxy-Anchor
79.1
90.8
96.2
98.7
Ours 512
79.0
90.7
96.1
98.7
Table 3: Comparison of the Recall@k (in percentile) on the In-Shop Clothes Retrieval dataset.
In-shop
Recall
k=/
k=10
k=100
k= (=0000
HDC
62.1
84.9
89.0
92.3
HTL!18
80.9
94.3
95.8
97.4
MS
88.0
97.2
98.1
98.7
Proxy-Anchor
90.8
97.9
98.5
99.0
Ne
53.0
73.0
76.0
79.0
A-BIERS12
83.1
95.1
96.9
97.8
ABE
87.3
96.7
97.9
98.5
MS
89.7
97.9
98.5
99.1
Proxy- Pn
91.5
98.1
98.8
99.1
Ours 312
91.5
98.2
98.7
99.1
5.3 RESULTS
To evaluate the benefits of the proposed regularization factor follow Kim et (2020) and per- form an experimentation on the CUB200-2011 I, Cars-196, Standord Online Products and In-Shop Clothes Retrieval datasets for image size of 224x224 pixels.
The results for the CUB-200-2011 and the Cars-199 196 datasets are summarize in Table Our method achieves competitive results on both dataset, improving the recall @k metric of the baseline imple- mentation of Proxy Anchor. Significant results are obtained on both cases, where we beat by margin of 0.7% the recall @ in the CUB-200-2011 dataset and by a margin of 1.1 in the Cars- 196. We recreate the T for Kim et al.] (2020) to put in perspective the obtained results. In the SOP dataset, the model shows slightly worse performance that the baseline, the results are summarized in Table 2 We attribute these results to the fact that the SOP dataset is the largest of the four. Despite this fact we expect that with further tunning of the parameters this result can be also improved. At last, the results the In-shop dataset also modest in comparison to the baseline scenario (see Table 3), and we cannot attribute a significant benefit on applying the regularization factor in that
8

---

Under review as a conference paper at ICLR 2023
5.4 ABLATION STUDIES
Deep metric layers. In the ablation studies we test two types of Normalizing Flows for the distance metric layers: Real-NVP Dinh et al. (2016) and NICE (Non-linear Independent Component Esti- mation) Dinh et al. (2014) considered the predecessor of Real-NVP. We observe slight benefit in using Real-NVP and therefore this is the metric that we report in the experimentation.
Coeflicients of regularization. Regarding the regularization coefficient, we perform a test com- prising the following values of À E {0.01, 0.05, 0.1, 0.2, 0.5). Particularly, we observe better results when the coefficients λ and X2 of the dueling deep metric layers take the same value. The reported results correspond to a value of A = X2 0.05 that achieved overall best performance.
Data augmentation. Finally we also experiment with different values for the e-coefficient in the generation of new samples for data augmentation. The e-coefficient controls how far from the cur- rent sample is the augmented one be created. Because the last layer applies an Izz-normalization, the output samples are restricted to be in the unitary hyper-sphere and the data augmentation occurs in the tangent hyperplane to it. We test several coefficients E {0.01, 0.05, 0.1, 0, 2, 0.4, 0.6, 0.8) ranging from the close proximity of the original sample to the neighbour sample. In order to fur- should carefully select the c, in experimentation
ther increase the generalization corresponds to the value that achieved best performance.
0.4
We keep the ablation studies concise because the aim in the experimentation is determine whether the regularization benefits the proxy-anchor algorithm used as a baseline. Therefore, in the shake of presenting a fair comparison, maintain the same 512-D embedding dimension the original work does, and also we report the same values for the margin ò = 0.1 and scaling factor a 32 of Proxy Anchor.
CONCLUSIONS
This paper presents a novel learning paradigm for Distance Metric Learning (DML). Differently from the other DML methods that only focus on designing different loss functions, our work focuses on regularizing the Lipschitz constant as a way to improve the generalization capabilities of DML models. We adopt invertible layers from Normalizing Flows to construct deep metric model in which the computation the Jacobian matrix is efficient. At last, we minimize the determinant of Jacobian matrix to reduce the Lipschitz constant of the deep neural network. Conducted experiments on fine-gre mne-grained Cars196, CUB-200-201 Standord Online Products (SOP) and In-Shop Clothes Retrieval datasets show that the proposed architecture benefits the baselines proxy-basee architecture on achieving better generalization.
REFERENCES
Ching-Yac Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. De- biased contrastive learning. Advances in neural information processing systems, 33:8765-8775, 2020.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEF/CVF conference on computer vision and patlern recognition, pp. 4690-4699, 2019.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint arxiv: 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: 08803, 2016.
Yueqi Duan, Jiwen Lu, and Jie Zhou. Uniformface: Learning deep equidistributed representation for face recognition. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3415-3424, 2019.
Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 1238-3247, 2020.

---

Under review as a conference paper at ICLR 2023
Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning towards minimum hyperspherical energy. Advances in neural information processing systems, 31, 2018.
lya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.0501,, 2017.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 4004-4012, 2016.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Interna- tional conference on machine learning, pp. 1530-1538. PMLR, 2015.
Karsten Roth, Oriol Vinyals, and Zeynep Akata. Non-isotropy regularization for proxy-based deep metric learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7420-7430, 2022.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 815-823, 2015.
Kun Song, Junwei Han, Gong Cheng, Jiwen Lu, and Feiping Nie. Adaptive neighborhood metric learning. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2021.
Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss: unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2495-2504, 2021.
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5265-5274, 2018.
Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott. Multi-similarity loss with general pair weighting for deep metric learning. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5022-5030 2019.
Nik Weaver. Lipschitz algebras. World Scientific, 2018.
10


Reviews:
[{"review_id": "v6YdpQ_0Yz", "reviewer": "Reviewer_obNj", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is about deep metric learning for fine-grained image retrieval. The task consists in learning a parameterized embedding function modelled as a neural network to project images into a deep space where images from the same categories are similar and images from different categories are not similar. The authors propose to train their model with a regularization term minimizing the Lipschitz constant around a set of data points from the same class. They approximate the Lipschitz constant using a Normalizing-Flow network.", "summary_of_the_review": "I would not accept this paper because too many elements needs to be changed and the contribution is too incremental. ", "strength_and_weaknesses": "Strengths:\n- The strength of this paper resides in the strong theoritical analysis of the effect the Lipschitz constant has on the embedding function modelled by the deep neural network.\n- The authors also show some improvement on the CUB-200-2011 and Cars-196 Dataset\n- The utilization of Normalizing flow network seems to be appropriate for approximating the Jacobian.\n\nWeaknesses:\n-  The main weakness of this paper is that it seemed to have been unfinished. They are a lot of typos and mistakes that degrades the overall quality of the paper. Most importantly the ablation study tables are missing.\n- Incremental contribution. The main contribution is not significant enough in terms of motivation and performance. While the regularization term shows improvement on 2 datasets, it does not on 2 others that are much larger and more complicated. The performances seems to be mostly achieved by the baseline.\n- The figure 2. is not informative, there is no disctinction between the 2 dueling branches. The captions are lacking information. What makes the top branch different from the bottom one?\n- The main contribution is the presentation of  the regularization term where the Lipschitz constant is approximated and minimized for similar sample (as stated end of section 4 and with Eq. 11). \n- In the conclusion the authors mention that other work focus on loss function design and not them. I would argue that design of regularization terms fall within the design of loss functions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "nn629kLE9Ip", "reviewer": "Reviewer_n2tf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the deep metric learning field. The motivation of this paper is that Current pair-based and proxy-based methods harm the generalization ability as shrink the distance between positive pairs. As a result, it adopts the structure of normalizing flow as the deep metric layer and calculates the determinant of the Jacobian matrix as a regularization term that helps in reducing the Lipschitz constant.", "summary_of_the_review": "The motivation of this paper doesn’t make sense to me, which is my main concern about it, hence I recommend rejection.", "strength_and_weaknesses": "Pros:\nThis paper focuses on a significant topic.\nThe organization of this paper is reasonable and the expression is clear, which make it easy to follow.\n\nCons.\nThe authors summarized four contributions of this paper, however, I did not find the first two.\nI do NOT think the motivation of this paper holds, which is the crucial concern from me about this paper. This paper claims that shrinking the feature space will increase the Lipschitz constant (LC), and give some deductions in Eq (2) and Eq(3). It’s true you can find some x that satisfy Eq (3), but not for all x. Therefore, this can not prove the increasing or decreasing property of LC. Since the motivation does not hold, I do not see the meaning of the proposed method.\nThere are unclear statements, such as “the first/second deep metric learning layer”, “the learnable parameter of the parameter of”, “may be not a connected to the neighborhood of another class” which should be “may be connected to the neighborhood of another class” in my opinion.\nThe experimental part is very confusing. The proposed method compares with other methods based on the different backbone and embedding dimensions, which I don’t think can be compared together. Since your main strategy is to modify the learning objection, by adding a new regularization term, I advise you to combine multiple loss functions with your regularization term, which will be more convincing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_r6f54hPvL", "reviewer": "Reviewer_VbA6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the over-fitting issue caused by reducing intra-class distances in metric learning. Specifically, when the intra-class distances are excessively reduced, the Lipschitz constant of network embedding would be significantly enlarged. Then the decision boundary comes unsmooth and is not good for generalization performance. To address this issue, the authors propose a normalizing flow as a new network layer to regularize the learning. Experiments on popular datasets validate the effectiveness of the proposed method.", "summary_of_the_review": "Overall, this paper is well-written and the idea is novel, but I do not agree with some critical claims in this paper. Since these claims are rather critical to the motivation, I would like to vote for a \"reject\", and I think the authors should further clarify them in the new version.", "strength_and_weaknesses": "**Strengths:**\n\n(1). This paper is clearly written and easy to follow.\n\n(2). The high-level motivation is good. I agree with the authors that excessively reducing intra-class distance may incur over-fitting.\n\n(3). Theoretical analysis is provided.\n\n\n**Weaknesses:**\n\nThis paper has two critical flaws.\n\n(1). The authors claim that the L2 normalization is only available for linear layers, so they cannot be used in deep metric learning. I don't think so. Actually, there is a weight decay technique in deep learning, which also effectively reduces the Lipschitz constant of the neural network. In this case, the motivation for proposing a new regularization technique in this paper is not solid.\n\n(2). For pair-based loss functions such as contrastive similarity loss, I agree with the authors that they may aim to reduce the intra-class distance as much as possible. However, for many relative similarity loss functions, we only encourage that it is okay when inter-class distances are larger than intra-class distances. We would not excessively reduce the intra-class distances as much as possible.\n\n(3). Furthermore, I found the compared methods in experiments are before 2020, which means they cannot be regarded as sota.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DQWEpZvuDn-", "reviewer": "Reviewer_F9Bh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper summarizes the traditional distance metric learning algorithms and classification-based distance metric learning algorithms together and solves the Lipschitz constant problem on distance metric learning. The authors design a deep neural network structure that allows us to minimize the Lipschitz constant of the deep neural network directly. They also minimize the determinant of the Jacobian matrix to reduce the Lipschitz constant of the deep neural network.", "summary_of_the_review": "The idea of this paper show some novelty, but the overall writing quality is limited. Also, the experimental result is not convincing and competitive.", "strength_and_weaknesses": "Strength:\n- Clear motivation and identification of the problem in existing methods.\n- Proposed method is easy to understand and the description is clear.\n- Theoretical proof is sufficient and complete.\n\nWeaknesses:\n- Experiments are not convincing:\n1. An ablation experiment should be added to analyze the validity and necessity of each block. \n2. The effectiveness of sample augmentation is necessary to discuss. \n3. Visualization results should also be added.\n4. The performances of the proposed methods are not good enough.\n- Why Eq.(6) is better than Eq.(5)? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject