Triplet Similarity Learning on Concordance Constraint
Authors: Jiansheng Fang
Jiajian Li
Jiang Liu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Metric Learning
Triplet Loss
Concordance
Hard samples

Abstract: Triplet-based loss functions have been the paradigm of choice for robust deep metric learning (DML). However, conventional triplet-based losses require carefully tuning a decision boundary, i.e., violation margin. When performing online triplet mining on each mini-batch, choosing a good global and constant prior value for violation margin is challenging and irrational. To circumvent this issue, we propose a novel yet efficient concordance-induced triplet (CIT) loss as an objective function to train DML models. We formulate the similarity of triplet samples as a concordance constraint problem, then directly optimize concordance during DML model learning. Triplet concordance refers to the predicted ordering of intra-class and inter-class similarities being correct, which is invariant to any monotone transformation of the decision boundary of triplet samples. Hence, our CIT loss is free from the plague of adopting the violation margin as a prior constraint. In addition, due to the high training complexity of triplet-based losses, we introduce a partial likelihood term for CIT loss to impose additional penalties on hard triplet samples, thus enforcing fast convergence. We extensively experiment on a variety of DML tasks to demonstrate the elegance and simplicity of our CIT loss against its counterparts. In particular, on face recognition, person re-identification, as well as image retrieval datasets, our method can achieve comparable performances with state-of-the-arts without tuning any hyper-parameters laboriously.

Under review as a conference paper at ICLR 2023
TRIPLET SIMILARITY LEARNING ON CONCORDANCE CONSTRAINT
Anonymous authors Paper under double-bling review
ABSTRACT
Triplet-based loss functions have been the paradigm of choice for robust deep met- ric learning (DML). However, conventional triplet-based losses require carefully tuning a decision boundary, i.e., violation margin. When performing online triplet mining on each mini-batch, choosing good global and constant prior value for violation margin challenging and irrational. To circumvent this issue, we pro- pose novel yet efficient concordance-induce function to train DML models. We formulate the similarity of triplet samples as a concordance constraint problem, then directly optimize concordance during DML model learning. Triplet concordance refers to the predicted ordering of intra-class and inter-class similarities being correct, which is invariant to any monotone trans- formation of the decision boundary of triplet samples. Hence, our CIT loss is free from the plague of adopting the violation margin a prior constraint. In addition, due to the high training complexity of triplet-based losses, we introduce a partial likelihood term for CIT loss impose additional penalties on hard triplet sam- ples, thus enforcing fast convergence. We extensively experiment on a variety of DML tasks demonstrate the elegance and simplicity of our CIT loss against its counterparts. In particular, on face recognition, person re-identification, as well as image retrieval datasets, our method can achieve comparable performances with tite-offthheartt without tuning any hyper-parameters laboriously.
triplet (CTT) loss as objective
INTRODUCTION
Deep metric learning (DML) for visual understanding tasks, e.g., face recognition Schroff et al.
(2055) Taigman et al. (014). person re-identification (ReID) Shi et al. (206);; Ustinova & Lem-
pitsky 2016). image retrieva Fang et al.] (202T); Revaud et al. (011)), aims at learning embedding
representations of images with class-level labels by ranking loss function Kaya & Bilge (2019);
Sohn (2016); Wang et al. (2017). There are two representative ranking loss functions developed for DML to minimize between-class similarity and maximize within-class similarity, i.e., pair-based loss Sun et al. (2014) and triplet-based basee loss Zhao et al. (2019). Compared pairwise constraints, the optimization pattern of triplet-basee losses additionally captures the relative similarity information, thus yielding impressive performances [Lang et al. (2021); Zhuang et al.) (016)). With triplet con- straints, images from the same class are projected into neighboring embedding spaces, and images with different semantic contexts are mapped apart. However, under such an optimization objec- tive, triplet-based losses suffer from following two problems when training DML models with the stochastic gradient descent (SGD) algorithm and sampling triplets within a mini-batch.
Irrational to set an absolute margin. Triplet constraint relies on a decision boundary to partition the embedding space of intra-class and inter-class, i.e.. violation margin for reinforcing optimization Wang et al. (2018a]b) b). However, the violation margin is sensitive to scale change. and choosing an identical absolute value for clusters in different scales of intra-class variation is inappropriate Wang et al. (2017). Hence, triplet-based losses need to regulate this syper-parameter attentively to impose appropriate penalty strength Qian et al. 2019); ; Sun et al. (2020). The performance of Circle loss Sun et al. (2020) on the varying circular decision boundary can prove such claim. The performance of the same task exhibits a significant difference by setting different violation margins. And Circle loss with the same violation margin varies from superior to inferior various tasks. For circumventing this issue, Angular loss is proposed to push the negative point away from the center of the positive cluster and drag the positive points closer each other by constraining

---

Under review as a conference paper at ICLR 2023
the upper bound of the angle at the negative point Wang et al. 2017). In hierarchical triplet loss (HTL) Ge (018). the violation margin is automatically updated over the constructed hierarchical tree to identify& a margin that generates gradients for violated triplets. However, existing methods of mitigating the issue still depend on the setting of the decision boundary, only substituting hyper- parameter. Angular loss needs to specify the angle degree and HTL needs to design hierarchical class tree. Since choosing a global and constant prior value for the decision boundary is irrational, we innovatively formulate triplet similarity learning as concordance constraint problem without an assumed decision boundary.
-Suffering from slow convergence. Triplet-based losses can provide a strong supervisory signal for training DML models by mining rich and fine-grained inter-sample relations. However, since the number of tuples (each tuple contains an anchor sample and its positive and negative samples) increases polynomially with the number of training samples, they suffer from prohibitively high training complexity, thus causing significantly slow convergence [Ebrahimpour et al.) (2022): Kim et al. (2020). Another potential issue for triplet-based losses is that large amount of tuples make a limited contribution to the learning algorithm and sometimes even diminishes the quality of the learned embedding space Wu et al. (017). Many works have been devoted to studying the effective triplet sampling strategy within a mini-batch to utilize hard triplet samples that improve convergence speed or the final discriminative performance Hermans al.. (2017); Oh Song et al. (2016); (Shnn (016);; Wu et al. (2017) For example, HTI. Ge (2018) is proposed to automatically collect infor- mative training triplets via an adaptively-earrned hierarchical class structure. However, these hard triplet sample mining techniques involve tuning hyper-paranee syper--arameters and may occur the risk of over- fitting when performing online triplet mining within a mini-batch Ebrahimpour et al. (2022) Kim et al. (2020). Given three tuple types (hard, semi-hard, and easy triplet samples), we need to con- sider how to achieve the trade-off between them during DML optimization. Leveraging hard triplet samples alone may occur bad local minima Do et al. (2019). Overwhelming easy triplet samples affect the training efficiency Schrooff et al. (2015). Inspired by SoftTriplet loss QQia et al. (2019) introduced to learn embeddings without triplet sampling, explore laying more emphasis on hard triplet samples by relaxing concordance constraints, thus accelcrating convergence speed.
In each triplet sample, the intra-class similarity is naturally higher than the one of inter-class. The predicted ordering of similarities of intra-class and inter-class needs to be on par with the observed ordering. Such an ordering concordance not only takes effect mini-batch but also the whole sample. Such intrinsic concordance constraint is invariant to any monotone transformation of the decision boundary of triplet samples. Hence, we develop a novel concordance-induces triplet (CIT) loss function to optimize triplet similarity. Existing triplet-based losses explicitly give a global and constant violation margin as a decision boundary based on apriori knowledge. Unlike them, our CIT loss exploits the concordance constraint of triplet similarity to avoid falling into the plague of tuning the violation margin. It is an elegant, simple, and efficient way to learn the intrinsic simi- larity between all samples and is insensitive to the triplet sampling within a mini-batch, We further introduce a partial likelihood term to enforce different penalty strengths on different tuple types, pri- marily laying more penalties on hard triplet samples. This term mainly helps improve convergence speed and exhibits a slight impact on performance, thus avoiding the plague of elaborative tuning. Based on thoroughly and randomly mini-batches and triplet sampling, this term can regulate the penalty strength keeping consistency with the degree of the discordance or concordance of triplet similarity. The higher discordance of hard triplet samples brings more penalty strengths, thus arising more contributions to gradients.
The main contributions of this work summarized as follows:
We propose a novel, simple, elegant concordance-induce triplet (CIT) loss function for deep metric learning (DML.. Our CIT loss frees DML training from tuning the decision boundary by directly maximizing concordance of triplet similarity.
In addition, we introduce partial likelihood term to impose loose concordance constraints to focus on the informativity of hard triplet samples, thus helping speed up convergence.
Using two popular backbones, we conduct extensive experiments on various DML tasks, in- cluding face recognition, person re- re-identification (Reid), and image retrieval. On all tasks, we demonstrate the effectiveness and elegance of our CIT loss and gained performance on par with the
2

---

Under review as a conference paper at ICLR 2023
2 METHODOLOGY
2.1 WARM-UP
Given training set D {a 31) with K classes, æ; with label ys E {0, 10,1. K K-1] the feature embedding projected from the i-th image with DML models. Let the feature embedding of a;; IR be represented as (F,), where I; is the i-th input image, 0 indicates the learnable parameters of a differentiable DML model F, and D is the dimension of feature embedding. With DML model F, we can map the image space I; into low-dimension embedding space æ; used for similarity measurement. 2; is usually normalized into unit length for the training stability and comparison simplicity. DML models primarily utilize various ranking loss functions learn the embedding space from the image space in supervised way. Apart from a suitable neural network, is essential to design ranking loss function optimize the embedding space. Our work aims to improve triplet-based loss to help DMI models train elegantly and efficiently by avoiding tuning the violation margin and improving convergence speed, and can gain impressive performance.
1 denotes
DML models are commonly trained using online SGD algorithms, where the gradients for optimiz- ing network parameters are computed locally with mini-batches. Hence, triplet samples are selected and formed in mini-batch during each training iteration. Each triplet sample T (æa Tp, tx,) consists of an anchor sample Ta. a positive sample Ty and a negative sample In, whose labels sat- isfy Yo Up * Vn. The goal of triplet- based losses to push away the negative sample T,, from the anchor sample F. by a violation margin m > compared to the positive sample Tp:
San + m S Sap-
(1)
We define Sup E [0, 1] as the intra-class similarity of æt and Tp. and San [0, 1] the inter-class similarity of x, and F,, We seek to minimize San and maximize Sap- To enforce this constraint in the embedding space, we define the optimization target of the standard triplet loss as:
Let
Σ San Sup m
Ny.
where the operator F++ maz(0, -) represents the hinge function and the symbol Ny denotes the number of all triplet samples mini- batch.
given globally constant value for the violation margin m, we can group all triplet samples into three categories:
According to Equation 2]
Hard triplet samples: San > Sap-
Semi-hard triplet samples: if Sap
San Sap-
Easy triplet samples: if S.n + m < Sap
Among these three tuple types, easy triplet samples generate zero loss, while hard triplet samples contribute the most losses. An effective sampling strategy combining an appropriate violation mar- gin can help mine hard triplet samples. In fact, the violation margin in the triplet-based losses plays key role to sample selection during model training Ge (2018). However, the violation margin needs to be carefully tuned. On the other hand, the absolute violation margin is an irrational decision boundary for multi-classes with different cluster centroids.
2.2 CIT Loss
Being simple, we intuitively explore no need to consider the violation margin for learning hard triplet samples effectively. Since the concordance is invariant to any monotone transformation of the decision boundary of triplet samples, it is natural to formulate the metric learning problem to maximize the concordance. We turn the predicted similarity correlation into comparable pairs, i.e., intra-class similarity Sap and inter-class similarity San- The set of comparable pairs mined from the whole training set D is E, == {(Sap San]]. }. The number this set is Ny. A comparable pair (Sap. San) E ET is concordant if Sap > San- Otherwise, the pair is discordant with the ground truth. We calculate the ratio of undergoing any pairs by taking the exponential form of intra-class and inter-class similarities:
each triplet sample T
R
1 NT
Σ
E Sun
(3)
Sax
San
3

---

Under review as a conference paper at ICLR 2023
Obviously, the average ratio R lies in the range [e-1,e]. The lower and upper ranges correspond to complete concordance and discordance of all comparable pairs, respectively. With such bounds, we define our CIT loss an exponential lower-bound form:
Le
Σ
[] 1- (San Son)
(4)
NT (Sag.San))EF
With minimizing CIT loss, we address the metric learning problem as concordance optimization between the predicted and observed similarity of comparable pairs. Similarity concordance building in whole triplet samples is invariant to any monotone transformation of the decision boundary of triplet samples. In other words, concordance optimization pays emphasis on the ordering correlation of distance without considering the correlation degree, thus avoiding imposing margin constraints.
The empirical error induced by pairwise discordance with respect to DML model Fo is denoted by (FFo) and defined by:
g(Fo) = NT
Σ ISp 2 Les
(5)
Sen)EEE
where the indicator function = if Sup < SSan (discordance), and 0 otherwise (concordance). According to Equation 5 we can estimate the learning parameters 0 of the DML model F by mini- mizing CIT loss L. We We leverage pairwise similarity.
insordancc--nducee penalty in our CIT loss to optimize the target
We assume that the violation margin is 0 in CIT loss, then there triplet samples Sap < Son and easy triplet samples Sap > Son- For easy triplet samples, the DML model F faultlessly predicts the target pairwise ranking, thus leaving penalty-free And hard triplet samples contribute to the informativity for gradient-based optimization. To speed up con- vergence. we further introduce a partial likelihood term for our CIT loss to focus on the discordant penalty of hard triplet samples. From the standpoint of concordance, the pairwise similarity in each triplet sample T is not always transitive. The transitivity of concordance-induced order by the DMI. model Fo: Sap > San but may Sap < Spm for each triplet sample TT Spm indicates inter-class sim- ilarity between u and Fy Considering transitivity of triangle edge, we define the partial likelihood form as:
tuple types: hard
Cp II (enp)Sen)EE
SSon + Spm
Theoretically, the higher proportion of the edge Sop in the three sides can better account for the concordance and transitivity. Hence, the product of the ratio in Lp can suggest the degree of concor- dance or discordance. We quantize such degree to construct loose concordance constraint term by placing the negative log partial likelihood of Equation [6)
Cp
Σ
{ Sap - log(e + e
NT
(Sa
With Equation 7 the penalty on predictive error for hard triplet samples can be boosted, thus helping gradient optimization. And the best predictive outcome for easy triplet samples is Sap 2 log(e + eS there is no penalty. If not, the slight penalty for easy triplet samples can help enhance the clustering effect within the class. It can be seen from this that this term is subject to the principles of maximizing inter-class similarity and minimizing intra-class similarity.
Combining Equations and 7 we present our CIT loss as:
Loit = GLe + (1
where the hyper-parameter the magnitudes of the two losses at the same level stabilize the model training. By comparing the two terms in our CIT loss, we can find that the former term requires more rigorous concordance. However, it impossible to ensure the complete transitivity and concordance. Hence, we can not fully replace the former term with the latter term. Generally, > = 0.5 can trade off the discordant penalty and fast convergence. Schematically, (a) and (b) in Figure I show that our CIT loss can bring fast convergence by utilizing the hyper-parameter y boost the contribution of hard triplet samples to gradient update of network parameters. And it helps reduce the training consumption of easy triplet samples, thus reaching up to the last performance in advance.
[0, 1] regulates the loss value between them. Specifically, controls
4

---

Under review as a conference paper at ICLR 2023
(a) Convergence hard triplet samples
(b) Convergence all triplet samples
hard triplet samples, y=0.5
y=
hard triplet samples, y-1.0
0.9
y-1.0
1.4
mcan of hard triplct samples, y-0.5
mean triplet samples, y-1.0
0.8
0.7
0.9
0.5
0.4
1.0
20
30
40
10
20
40
50
Epoch
Epoch
Figure 1: (a) For hard triplet samples, the epoch increases, their number significantly decreases,
and the average loss on 7 = 0.5 (heavier penalties) gradually approaches 7 1.0. (b) The con- vergence speed on 0.5 is faster than 1.0 due to imposing more penalties on hard triple samples.
EXPERIMENTS
3.1 SETTINGS
Datasets. (1) Person RelD aims to spot the appearance of the person in different cameras the same camera on different occasions. We evaluate our method on a popular dataset, Market1501 Zheng al. 2013), containing 1,501 identities, 12,396 training images, and 19,732 gallery images captured with six cameras.
(2) We use two datasets for evaluation image retrieval, i.e., Car196 Krause et al. (2013) and In-shop Clothes Liu al.] (2016). The Cars196 dataset is composed of 16,185 car images of 196 classes. And the In-shop Clothes Retrieval dataset has 1 11,735 classes of clothing items and 54,642 images. (3) The CASlA-WebFace dataset Yi et al. (2014) with 10,757 real identities and 494,441 face images is the most popular dataset for the training of face recog- nition. For evaluation, we adopt the face verification results on LFW Huang et (2008), AgeDB Moschoglou al. 2017 IJB-C Maze et al. (2018), and CFP-FP Sengupta et al. (2016) datasets.
Comparable methods. Five comparable triplet-based loss functions involve (1) standard triplet loss (Triplet) with violation margin repressentigg the difference between the anchor-positive dis- tance and the anchor- neggative distance |Schroff et al. (2013). (2) Angular loss with an angular prior used for separating different classes Wang et al. 70, (3) centroid triplet (CT) loss with margin hyper-parameter used as the decision boundary of centroids of positive and negative classes Wiec- zorek et (202I). (4) Circle loss with a relaxation margin used for controlling the radius of the decision boundary Sun et al. (2020), and (5) soft triple (ST) loss without triplet sampling and viola- tion margin Qian al. (2019). Both ST and our CIT are exempt from the violation margin setting. Our CIT leverages the intrinsic concordance between the predictive and observed similarities of triplet samples, while ST extends softmax loss with multiple centers for each class.
Implement details, We implement all loss functions on the pytorch-metrie-lammiin 2020) platform and experiment with them on two different network structures. Two networks are convolutional neural network (CNN) of ResNet50 He et al.) (2016) and vision transformer (ViT) Dosovitskiy et al. (2020). We set the hyper-parameters for comparable methods according to the default reported in their works, and the default of the hyper-parameter in our CIT loss is 1.0, if not specified. We extract the 512-D feature embeddings for computing distances and use Euclidean distance the metric during inferences. We adopt FastRelD He et al.] (2020) platform to train the two networks with different loss functions for three DML tasks. The CNN optimizer Adam Kingma & Ba 2014), with learning rate of 3.5e-4, while VIT is SGD, with learning rate of 8e-3.
Musgrave et al.

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison of different loss functions for person RelD task CNN and ViT networks in terms of rank-k (k=1,5,10, 10, in %) accuracy, mean average precision (mAP, in %), and mean inverse negative penalty (mINP, %).
Market1501
Method
Margin
Network
R@l
R05
R@10
mAP
mlNP
0.01
95.58
98.28
98.87
89.08
67.05
Triplet
m = 0.05
95.75
98.52
98.87
89.26
67.48
0.1
95.75
98.49
98.99
89.19
66.65
a = 20
95.61
98.37
99.08
89.74
69.56
Angular
40
94.98
98.13
99.05
87.29
62.75
= 60
67.87
84.02
89.54
61.87
36.15
0.01
95.52
98.37
98.90
88.68
66.11
CNN
CT
m = 0.05
95.61
98.31
99.02
89.05
66.28
m 0.1
95.16
98.22
98.96
88.65
66.27
m = 0.1
92.10
96.82
97.60
81.84
52.83
Circle
m 0.4
91.18
96.59
97.71
79.65
48.91
m 0.6
93.05
97.54
98.40
82.37
52.97
ST
95.43
98.34
98.87
88.66
66.04
CIT (Ours)
95.87
98.22
98.93
89.41
68.27
m 0.01
93.62
98.
99.08
85.07
58.88
Triplet
0.05
94.06
98.40
99.11
85.87
60.74
m 0.1
93.68
98.01
99.17
86.02
61.94
20
93.47
97.71
98.57
85.35
60.70
Angular
a = 40
91.95
96.47
97.86
80.68
52.74
60
90.38
96.26
97.65
78.95
51.01
m = 0.01
92.19
97.71
98.96
81.75
51.91
VIT
CT
0.05
92.58
97.60
98.96
82.27
52.87
m = 0.1
93.38
97.98
98.93
83.00
54.31
m 0.I
87.68
93.88
95.46
72.80
42.64
Circle
m = 0.4
93.32
97.71
98.46
85.13
61.62
m 0.6
92.37
97.92
98.93
80.90
49.63
ST
91.81
97.51
98.78
80.69
50.00
CIT (Ours)
93.88
97.74
98.93
86.23
61.88
Specifically, the learning rate is decay scheduled according to cosine annealing strategy Loshchilov & Hutter 2016). The input size for CNN is 384 128 and VIT is 256 128. The batch size of both CNN and VIT is 64. The parameters of CNN are optimized in 60 epochs, while ViT is 120 epochs.
3.2 PERSON RE-IDENTIFICATION
We evaluate comparable loss functions on the RelD task in Table We can make three observations from the reported performances. First, mances against the state-of-the-arr. CIT obtains the best (95.81) on CNN and the second-highest (93.88) on VIT in terms of R@1. And CIT achieves the best (86.23) ViT and the second-highest (89.41) on CNN in terms of AP. Moreover, ml N P of our CIT are on par with the second-highest methods (underline), showing the competence of staying in the first tier among all methods. Sec- ond, we report the performances on varying violation margins for those loss functions relying on the decision boundary (See column Margin Table [.. We can discern that different violation margins can bring significant differences in performance, and the most prominent are Angular and Circle. Angular achieves the best mAP and m/NE P on CNN when the angle specified degrees (a) is 20 (its default is 40). But when 60, the performance of Angular is pretty bad. In particular, conduct repeated experiments to verify the reliability of these results. Such results entail that is challenging choose appropriate violation margin. Third, both ST and our CIT are free from the plague setting the violation margin. In terms three key metrics, R@ 1, mAP, and mIN P on CNN and VIT, CIT exhibits significant advantages against ST. In fact, there is crucial hyper-
find that CIT achieve competitive perfor-

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison of different loss functions for image retrieval tasks CNN and ViT networks in terms of rank-k (k=1,5,10, in %) accuracy and mAP (in %).
Cars196
In-shop Clothes
Method
Network
R@1
R055
R@10
mAP
R@l
R@5
R@10
mAP
Triplet
90.85
96.84
98.32
51.08
93.37
97.80
98.52
78.47
Angular
89.36
96.70
97.97
43.12
90.34
95.91
97.00
71.71
CT
89.40
96.90
98.03
42.56
93.75
97.87
98.56
79.17
CNN
Circle
87.05
95.93
97.27
41.26
87.81
94.90
96.51
70.00
ST
90.05
97.04
98.16
43.95
93.49
97.91
98.64
78.87
CIT (Ours)
90.87
97.63
98.55
46.18
93.68
97.76
98.59
78.93
Triplet
86.55
95.94
97.81
39.14
92.09
97.37
98.26
73.44
Angular CT
86.94
96.74
98.13
39.75
91.74
97.05
97.87
72.78
86.93
97.04
98.71
38.52
91.50
97.43
98.30
73.13
VIT
Circle
81.55
94.35
96.68
36.67
92.73
97.66
98.32
76.97
ST
87.09
96.83
98.61
36.96
91.10
97.02
98.00
71.12
CIT (Ours)
88.54
96.32
98.00
43.57
92.73
98.03
98.67
76.62
parameter needed to be tuned carefully for ST. i.e., the number of weight vectors per class. By contrast, our CIT avoids attentively regulating any hyper-parameters. The above three observations can demonstrate the elegance and effectiveness of our CIT, achieving comparable performance with carefully.
the state-of-the-arr and no need to tune
3.3 IMAGE RETRIEVAL
We evaluate all comparable loss functions adopting default hyper-parameter settings on two image retrieval datasets, i.e., Cars and In-shop Clothes. We compare CIT against those state-of- the-art methods in Table 2 Given metrics R@1 and mAP of two datasets on two networks, CIT gets four number (bold) and four number again demonstrate the superiority of our CIT to other methods conditioned on no careful decision boundaries and other hyper- paremeerrsssssss tuning. There are other two interesting points implied in Ta- ble First, for those comparable methods with the violation margin, Table illustrates that various violation margins bring significant performance differences on the same dataset. Whereas Table indicates that different datasets with the same violation margin exhibit significant gain differences. can be proved by the performance 76.97 on VIT for the In-shop Clothes dataset but achieves the worst VIT for the Cars 196 dataset. Second, for the Cars196 dataset, our CIT lay a lot behind mAP on CNN compared to Triplet. The metric R@1 only measures how many items rank the hit items to reflect the ranking quality. From the standpoint of ranking quality, our CIT fails to quantize the concordance between the predicted and observed similarities. In other words, CIT implicitly models the relations of Sap and San as inequality, while Triplet explicitly formulates their relations tion margin for Triplet fits related tasks, such a prior constraint can help boost performance, and the obtained result is superior to our CIT. However, choosing an appropriate task-specific violation margin is challenging and laborious. Our CIT marginally outperforms its counterparts and exhibits an elegant training manner that does not need to adjust the violation margin elaboratively.
(underline) among all eight. Such results
Circle which gains the best R@l of 92.73 and mAP of
hit, while the metric mAP also considers the
identical equation with the help of the violation margin. If the chosen viola-
3.4 FACE RECOGNITION
For the face recognition task, according to the ROC curve on TPR (True Positive Rate) at FPR (False Positive Rate) from le-6 to 1, we report AUC performances of four verification datasets on CNN and VIT in Figure 2 Our CIT gains the four highest AUC among the eight sub-tasks, including AgeDB on CNN (98.33), CFP-FP on CNN (97.39), IJB-C on ViT (99.32), and CFP- FP on ViT (96.85). The overall performance of this task again affirm the availability of our CIT which directly optimizes the similarity concordance of triplet samples and free from the annoyance of introducing the prior of the decision boundary. While regarding those loss functions

---

Under review as a conference paper at ICLR 2023
an
Agx03
OTh/
5.0
(ole PA600
Halectau 36.00
ye Jas mocee
100
Faithre
Rata
Tilns Fosttie
1.50
1990
(i-Can
C3P49P VIT
Aupdl (sac
Angiee taid
9730
C5
Eacoutt 5530 EnDweSS
STUMC Eftcce
O11hke RoC 16.00
Nodiie tale
Rase Nothvellat
Farse Roother Rate
fabe Postiee Rote
Figure 2: Comparison of different loss functions for LFW Huang et al. 20088, AgeDB Moschoglou et al. (017), IJB-C Maze et al. (2018), and CFP-FP Sengupta et al. (2016) datasets on CNN (upper row) and VIT (down row) networks in terms of AUC (in %).
Table 3: Performance of our CIT loss function with different sizes of the peper-parameter on the M5r and In-shop Clothes datasets.
Market1501
In-shop
Clothes
Y
Network
R&l
R
R010
MAP
RØ1
R85
R010
mAP
0.1
95.72
98.69
99.23
89.51
93.14
97.76
98.43
78.68
0.2
95.25
98.16
98.93
88.92
93.71
97.98
98.54
78.97
0.5
95.34
98.28
98.99
88.78
93.60
97.83
98.52
79.09
CNN
0.8
95.34
98.52
99.05
88.88
93.60
97.78
98.51
78.89
1.0
95.87
98.22
98.93
89.41
93.68
97.76
98.59
78.93
0.1
93.74
97.86
99.02
86.09
92.76
98.02
98.40
76.64
0.2
93.53
97.60
98.72
85.64
92.28
97.57
98.39
75.43
0.5
93.53
97.57
98.66
85,88
92.11
97.48
98.29
75.85
VIT
0.8
93.71
97.74
98.75
86.06
92.21
97.66
98.45
75.41
1.0
93.88
97.74
98.93
86.23
92.73
98.03
98.67
76.62
with the violation margin, we must diligently seek good violation margins for specific datasets and backbones obtain comparable performance. And for ST without the violation margin, still need to find an appropriate number of centers for each class, and the over-large number of centers raises an efficiency problem. Hyper-parameters are task-specific, so regulating typer-parameters is indispensable and de-facto laborious and time-consume cone-consuming. Based on the concordance relations of similarity, our CIT exhibits more flexibility modeling triplet constraints than existing triplet-based losses. The violation margin enforces constant restrictions for triplet samples, while our CIT only abides unequal relationship. Without any bells and whistles on tuning hyper-parameters CIT can yield advanced performances. And the only hyper-paramete in CIT is task-agnostic.
3.5 ABLATION STUDY
Here we want to analyze the impact of the hyper-parameter y in Equation g Table shows that the hyper-parameter 7 in our CIT loss has little effect on the performance. Both accuracy and mAP on different sizes of exhibit consistency and coherence. For two datasets on CNN and VIT, the best of R@1 and m.AF P (bold) scatter varying sizes of 7. Since Equations
and 7 can be
8

---

Under review as a conference paper at ICLR 2023
Markat1501 on CNN
Moract1b31 on VIT
V=0.1
y-1.
16.0
80
15:3
7
ng1
Rg1
535
1
K5
53.0
Y-01
y11
Epoch
Epoch Ir Ir-shop Clothes VTT
In-shop Clothes CNN
V-0.1
V-0.5
80
y-1.0
77
PMAP
mAp
20
70
90
10
Epoch
Epoch
Figure 3: Rank-1 accuracy (R 1) (Market1501 dataset) and mAP (In-shop Clothes dataset) versus epochs our CIT with different sizes the arper-parameter. ry on CNN and ViT.
interchangeable, and the difference is that Equation lays penalty on hard triplet samples. The reported stable performance can demonstrate that y has no impact on learning the concordance of triplet similarity in the mini-batch sampling. We can claim that this hyper-pane pepee-parameter is task-agnostic. From the performance perspective, our CIT can pay no attention to this only hyper-parameter. On the other hand, from the convergence viewpoint, introducing 7 aims help DML models speed up the convergence by laying emphasis on hard triplet samples. As Figure shows, CIT with lesser y can be fast convergence with fewer epochs achieving the last performance. The lesser y entails that Equation 7 with more loose concordance contributes more to the loss value. Our CIT pays more penalty on hard triplet samples and little penalty on easy triplet samples by setting lesser 7 thus advancing the convergence speed. This ablation study can demonstrate that our CIT with y can alleviate the training complexity of triplet-based loss functions.
4 CONCLUSIONS AND DISCUSSIONS
Building on the concordance constraint of triplet similarity, we propose a novel and elegant concordance-indees triplet (CIT) loss function simplify the optimization process for deep metric learning (DML). Our CIT loss can free DML training from the laborious tuning of the violation mar- gin in conventional triplet-based loss functions and encourage model training fast convergence. The violation margin task-specific, while the concordance constraint is task-agnostic and monotonous. Hence, the concordance between the predicted and observed similarities can help our CIT loss push far away from the plague of giving prior constraints for decision boundaries. We further utilize the degree of concordance of triplet samples pay more penalties hard triplet samples speed up gradient optimization. The extensive experiments on three popular DML tasks with two networks can demonstrate the elegance and availability of our proposed CIT, yielding performances on par with other triplet-based loss functions.
worthy to emphasize that our CIT loss intends achieve comparable performance with its counterparts but not pursue superior performance, CTT can favor DML training simply and elegantly by modeling the concordance of triplet similarity. When DML task is challenging to offer the violation margin and needs to alleviate the training complexity, our CIT loss is a reliable alternative to conventional triplet- based loss functions. is interesting to explore the degree concordance of triplet samples to bring the best performance in the future. If the degree of concordance, we can avoid the excessive constraints of the partial likelihood term in CTT loss.
design method to measure

---

Under review as a conference paper at ICLR 2023
REFERENCES
Thanh-Toue Do, Toan Tran, Ian Reid, Vijay Kumar, Tuan Hoang, and Gustavo Carneiro. A the- oretically sound upper bound on the triplet loss for improving the efficiency of deep distance metric learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10404-10413, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Mohammad K Ebrahimpour, Gang Qian, and Allison Beach. Multi-head deep metric learning ing global and local representations. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3031-3040, 2022.
Jiansheng Fang, Huazhu Fu, and Jiang Liu. Deep triplet hashing network for case-based medical image retrieval. Medical Image Analysis, 69:101 981, 2021.
Weifeng Ge. Deep metric learning with hierarchical triplet loss. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 269-2855 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Lingxiao He, Xingyu Liao, Wu Liu, Xinchen Liu, Peng Cheng, and Tao Mei. Fastreid: A pytorch toolbox for general instance re-identification. arXiv preprint arXiv:2006.0266 2020.
Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense the triplet loss for person re- identification. arXiv preprint arXiv: 1103:0770007733 2017.
Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned- Labeled faces in the wild: A datahase forstudying face recognition in unconstrained environments. In Workshop on faces in Real-Life Images: detection, alignment, and recognition, 2008.
Mahmut Kaya and Hasan Şakir Bilge. Deep metric learning: A survey. Symmetry, 11(9):1066, 2019.
Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 3238-33 3247,, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine -re-ggained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013.
Yuchen Liang Yan Pan, Hanjiang Lai, Wei Liu, and Jian Yin. Deep listwise triplet hashing for fine-grained image retrieval. IEEE Transactions on Image Processing, 2021.
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1096-1104, 2016.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608. 03983, 2016.
Brianna Maze, Jocelyn Adams, James A Duncan, Nathan Kalka, Tim Miller, Charles Otto, Anil K Jain, W Tyler Niggel, Janet Anderson, Jordan Cheney, et al. larpa janus benchmark-c: Face dataset and protocol. In 2018 international conference on biometrics (ICB), pp. 158-165. IEEE, 2018.
10

---

Under review as a conference paper at ICLR 2023
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, in-the-wild age database. In proceed- ings of the IEEE conference on computer vision and pattern recognition workshops, pp. 51-59, 2017.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. Pytorch metric learning, 2020.
Hyun Oh Song. Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 4004-4012, 2016.
Qi Qian, Lei Shang, Baigui Sun, Juhua Hu, Hao Li, and Rong Jin. Softtriple loss: Deep metric learning without triplet sampling. Proceedings of the IEEE/CVF International Conference on 2019.
Computer Vision, pp.
Jerome Revaud, Jon Almazán, Rafael S Rezende, and Cesar Roberto de Souza. Learning with average precision: Training image retrieval with a listwise loss. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5107-5116, 16, 2019.
Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Bjorn Ommer, and Joseph Paul Co- hen. Revisiting training strategies and generalization performance deep metric learning. In International Conference on Machine Learning, pp. 8242-8252. PMLR, 2020.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 815-823, 2015.
Soumyadip Sengupta, Jun-Cheng Chen, Carlos Castillo, Vishal M Patel, Rama Chellappa, and David W Jacobs. Frontal profile face verification in the wild. In 2016 IEEE winter confer- ence on applications of computer vision (WACV), pp. 1-9. IEEE, 2016.
Hailin Shi, Yang Yang, Xiangyu Zhu, Shengcai Liao, Zhen Lei, Weishi Zheng, and Stan Li. Embedding deep metric for person re-identification:: A study against large variations. In European conference on computer vision, pp. 732-748. Springer, 2016.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016.
Yi Sun, Yuheng Chen, Xiaogang Wang. and Xiaoou Tang. Deep learning face representation by joint ddentffiationnvveriiiattto Advances in neural information processing systems, 27, 2014.
Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss: A unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6398-6407, 2020.
Yaniv Taigman, Ming Yang. Marc' Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to
human-leve performance in face verification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1701-1708. 2014.
Evgeniya Ustinova and Victor Lempitsky. Learning deep embeddings with histogram loss. Advances in Neural Information Processing Systems, 29, 2016.
Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu. Additive margin softmax for face verifica- tion. IEEE Signal Processing Letters, 25(7):926-930. 2018a.
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5265-5274, 2018b.
Jian Wang, Feng Zhou, Shilei Wen, Xiao Liu, and Yuanqing Lin. Deep metrie learning with angular loss. In Proceedings of the IEEE international conference on computer vision, pp. 2593-2601
2017.
11

---

Under review as a conference paper at ICLR 2023
Mikolaj Wieczorek, Barbara Rychalska, and Jacek Dabrowski. On the unreasonable effectiveness of centroids in image retrieval. In International Conference on Neural Information Processing, pp. 212-223. Springer, 2021.
Chao-Yual Wu, Manmatha, Alexander Smola, and Philipp Krahenbuhl. Sampling matters in deep embedding learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2840-28444 2017.
Dong Yi, Zhen Lei, Shengeai Liao, and Stan Z Li. Learning face representation from scratch. arXiv preprint 1rxivv:411.. 7923, 2014.
Xiaonan Zhao, Huan Qi, Rui Luo, and Larry Davis. A weakly supervised adaptive triplet loss for deep metric learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification: A benchmark. In Proceedings of the IEEE international conference on computer vision, pp. 1116-1124, 2015.
Bohan Zhuang, Guosheng Lin, Chunhua Shen, and Ian Reid, Fast training of triplet-based deep binary embedding networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5955-5964, 2016.
A GENERALIZATION BOUND ANALYSIS
Theorem 1. Let L represents family of loss function associated to Ceit given DML model Fe-
Then for A8 > 0 over Ey, each of the following holds for any Last E L with probability at least 1 1-60 6.
In(1/5) 2NT
E E(Coir] < ED Coiie + RD(L) +
where Ep Ceit] is the empirical error (FF)) in Equation 5 over the training set D (in-of-samples), and [[Ccit] denotes the generalization or expectation error on out-of-samples. Ey with length Ny sampled from the training set DD. (DL) signifies the Rademacher complexity of L with respect to the training set DD
Proof Sketch. The generalization error bound based McDiarmid inequality and Rademacher complexity for concordance learning can be proved by Theorem 3.5 described in Mohri et al.) (2018).
As shown in Equation 9 the supremum of the generalization bound consists of three terms. The first term is the empirical error relating to training, the lower empirical error brings smaller supremum of the generalization bound. And in the third term, the more considerable amount of triplet samples T can reduce the upper bound of generalization error. The generalization bound of our loss function Lou given the DML model Fg largely determined by the second term, Rademacher complexity of TL.
Remark 1.
Theorem 2. For the hypothesis space [L T + {o, 1}}. we state (Ti T2, TN) (Coa (Ti), (oir(T)),... Lest (TN)) E 10, is a dichotomy. We further define the growth func- Then the following holds:
tion of is ML(NT) = maxN, [T. T2, T.
2 (NT)
(D S
(10)
NT
Proof Sketch. Relating the Rademacher complexity to the growth function, Equation 10 can be derived by Theorem 3.7 (Massart's lemma) and Corollary 3.8 in Mohri et al. (2018).
Remark 2. The Rademacher complexity can be bounded in terms of the growth function, which is distribution independent and purely combinatorial. The growth function Mr (Nr) is the maximum number of distinct ways in which triplet samples of Nr can be predicted as concordance (0) or discordance (1) by using hypotheses in L. It suggests the number of dichotomies realized by the
12

---

Under review as a conference paper at ICLR 2023
hypothesis and its upper bound is 2Nr, The growth function (N() suggests the representation power of the hypothesis space L, thus reflecting the complexity of the hypothesis space.
Combining Equations
9 and 10
we rewrite the supremum of the generalization error
In(1/6)
[[ sup [Ep[Ceir + (LT) +1 D
}.
(11)
2NT
Based on the above theorems, we would like to compare the supremum of the generalization bound of two items in the loss function Leit- The relation of triplets is modeled as Sep San in Equation and we reorganize the modeling relation of Equation /
with a violation margin, as follows:
Sap
San
(San log(e Sun l Smm)
(12)
margin
Similar to the standard triplet loss in Equation 2 the loss function Cp utilizes a margin to constrain triplet similarities, as Equation 12 shows. Obviously, from the standpoint of modeling complexity, the representation power of Lp is stronger than Le.
Corollary 1. Given the same DML model Fø. let The represents family of loss function sociated to Le and Lp represents a family of loss function associated to Ep. Then the following holds:
(NT) < ML,(N) S 2
(13)
For any hypothesis set I., the trivial bound is 2NT, The growth function measures the richness or complexity of the hypothesis set LL. Hence, the growth function Mr., (NT) for the hypothesis set ILe smaller than the growth function Mi,, (Nr) for the hypothesis set L.p
Corollary 2. By virtue of Theorem and Theorem 2, we further bound the difference between the empirical emrrr and generalization error for loss functions L.g and T., associated the DML model Fe.
In(1/8)
B|C.| - ED[Co] (NT)+ +
(14)
2NT
In(1/6)
B|C,] - EE[[pp & (NT)
V
(15)
2NT
With Equation 13 we can obtain:
R[Ca]- - [D.]..] < [[Cp) 1ppCpp
(16)
Supposing the amount of triplet samples is enough to make both expectation errors theoretically infinite close. Then, we can further derive ED(4,) < ED[C.)
Remark 3. Due Lp introducing extra decision boundary, the complexity of modeling triplet similarities can better fit the training set, thus helping speed training convergence. Compared to the loose constraint of Cp. C. enforces tight constraint on triplet similarities. As shown in in (b) of Figure D our loss function Cost fulfills lower loss by setting ? = 0.5.
However, we need to make trade-off between the training error and generalization error on the complexity of the hypothesis space. The larger complexity of the hypothesis space easily leads to higher generalization errors. Such as, one of the popular strategies for controlling the complexity to avoid over- over-fitting is introducing regularization terms. Therefore, it is not that the higher the complexity and the smaller the training error can surely bring the better generalization.
'abble 3 can account for the necessity of holding the trade-off between the training error and gen- eralization error. Increasing complexity by setting 7 < 1.0, i.e., introducing C, in Lest favors the In-shop Clothes dataset with 11,735 classes. On the contrary, the best performances of the Mar- ket1501 dataset containing 1,501 identities are achieved by setting 7 = 1.0, i.e., without Cp.
Through generalization bound and experiment result analysis, we can conclude:
To free DML training from tuning the decision boundary, we present a tight constraint on triplet similarities and achieve comparable performances with its counterparts.
13

---

Under review as a conference paper at ICLR 2023
We further introduce loose strategy to increase the complexity of the hypothesis space to speed convergence.
Increasing complexity is favorable to a large amount of training set. When the number of triplet samples is enough, regulate relatively small 7 enjoy two benefits simultaneously: speeding convergence and better generalization.
B GRADIENTS ON SIMILARITIES OF COMPARABLE PAIRS
The partial derivatives of our CIT loss in Equation with respect to Sap and San can be written as Equations 17 and 18 and the schematics of both corresponding gradients are shown in Figure 4
OCoit
San
7 -San-Sap) (1 7) (e v
(17)
0San
0h
-y (San -San +(1 (1--9) (1)
(18)
OSop
14
26
100
0.8
0.5
6.4
60
Figure 4: Gradients of CIT Loss Function with respect to Sap and San- CIT loss lays emphasize on hard samples by setting 0.5 compared y 1.0.
C PERFORMANCE OF FACE RECOGNITION
In Figure show the full ROC curves for comparable loss functions. Here, we additionally report their true accepted rate (TAR) at le-5 and 1e-3 false accepted rate (FAR) Tables and Besides, we also compare their face verification accuracy in Table 6
D EMBEDDING SPACE METRIC
An image can be encoded into an embedding feature by a differentiable DML model trained by triplet loss function. In Tables and g] we report performances for Triplet and CT on 550 and In-shop Clothes datasets by tuning more margins. Besides, we also provide two metrics to measure embedding space, i.e., normalized mutual information (NMI), spectral variance (SV) Roth (et al. (2020). Lower values SV indicate more directions of significant variance and suggest more discriminative embedding features.
14

---

Under review as a conference paper at ICLR 2023
Table 4: Comparison of different loss functions for LFW Huang al. (2008) AgeDB Moschoglou
et al. (017)) Maze et al. 20188. and CFP-FP Cep-rPSenggptaa et al. (2016 datasets on CNN in terms
of TAR @ @AR (in %).
Method @CNN
LFW
AgeDB
JJB-C
CFP-FP
TAR@
FAR
TAR
@
FAR
TAR@
FAR
TAR@
FAR
le-5
1e-3
le-5
1e-3
le-5
1e-3
le-5
1e-3
Triplet
92.73
98.87
64.07
74.00
80.20
92.77
67.89
80.23
Angular
88.07
98.57
65.30
67.93
80.80
93.05
77.54
82.03
CT
95.37
97.97
53.77
74.33
81.24
93.04
72.97
84.69
Circle
91.43
98.27
51.27
73.13
80.82
92.93
79.03
83.97
ST
92.57
98.17
56.30
70.20
81.75
93.08
80.37
83.00
CIT (Ours)
91.67
98.33
71.37
74.70
82.23
93.24
76.09
84.46
Table 5: Comparison of different loss functions for LFW Huang et al. 2008 Age Moschoglou al. (207), IJB-C al. (2018), and CFP-FP Sengupta et al. (2016) datasets on ViT in terms of TAR@FAR (in %).
Method
LFW TAR@ @
AgeDB
IJB-C
CFP-FP
FAR
TAR
@
FAR
TAR
FAR
T
FAR
@VT
le-5
1e-3
le-5
1e-3
le-5
1c-3
le-5
le-3
Triplet
87.13
97.80
56.63
62.90
67.01
87.96
54.94
69.63
Angular
86.00
98.53
40.30
53.17
68.02
88.43
49.54
76.57
CT
87.07
97.57
35.10
41.67
68.49
88.62
39.06
73.54
Circle
91.17
98.03
42.83
49.70
68.10
88.23
59.29
70.43
ST
91.50
98.30
50.50
63.33
70.40
88.74
42.00
75.46
CIT (Ours)
83.77
98.53
51.93
58.70
72.45
90.43
60.89
65.40
Table 6: Comparison of different loss functions for LFW Huang al. et al. (2017), IJBC Maze et al.) (2018), and CFP-FP (Sengupta et al. VIT networks in terms face verification accuracy (in %).
2008
AgeDB Moschoglou
(0016)
datasets on CNN and
CNN
VIT
Method
LFW
AgeDB
IJB-C
CFP-FP
LFW
AgeDB
IJB-C
CFP-FP
Triplet
99.37
93.97
98.23
94.93
99.27
92.12
98.25
92.90
Angular
99.10
93.65
98.27
94.80
99.07
91.50
98.24
92.30
CT
99.30
94.25
98.52
94.83
99.40
92.77
98.28
93.39
Circle
99.35
94.22
98.32
93.99
99.26
92.63
98.35
93.19
ST 99.37
94.52
98.28
95.19
99.35
93,07
97.99
93.27
CIT (Ours)
99.40
94.57
98.40
94.76
99.37
93.18
98.39
93.34
E CONVERGENCE CURVES
Figure Soure5lance and help analyze the convergence of loss functions.
QUALITATIVE RESULTS
In Figure 7 exhibit some retrieval cases for our CIT loss on different datasets.
G GRADIENTS ON FEATURES OF TRIPLET SAMPLES
In Figure visualize the gradients of three features Ta. Zp. Tx- The three features may be- long to different classes under randomly mini-batch sampling, thus incomparable. Hence, from this visualization, discern the contributions of features DML optimization.
15

---

Under review as a conference paper at ICLR 2023
Table 7: Comparison of Triplet and CT with different margins for the M5r dataset on CNN
and VIT networks in terms of rank-1 (in %) accuracy, normalized mutual information (NMI, in %),
spectral variance (SV, in %), mean average precision (mAP, in %), and mean inverse negative penalty (mINP, in %).
Market1501
Method
Margin
Network
R
m.P
mINP
SV
NMI
m 0.0
95.64
88.73
66.60
44.37
95.59
0.01
95.58
89.08
67.05
44.45
95.56
= 0.05
95.75
89.26
67.48
44.44
95.69
Triplet
m 0.1
95.75
89.19
66.65
43.83
95.50
0.5
95.40
88.78
66.46
45.86
95.57
m 1.0
95.37
88.73
66.64
49.08
95.62
0.0
CNN
95.81
88.77
66.32
44.02
95.54
m = 0.01
95.52
88.68
66.11
44.54
95.55
0.05
95.61
89.05
66.28
44.84
95.61
CT
m = 0.1
95.16
88.65
66.27
46.90
95.51
0.5
95.43
88.79
67.07
44.81
95.51
m 1.0
95.64
88.74
66.24
44.10
95.65
CIT (Ours)
95.87
89.41
68.27
43.62
95.70
m 0.0
93.56
84.76
57.45
38.00
95.15
0.01
93.62
85.07
58.88
38.18
95.19
m = 0.05
94.06
85.87
60.74
36.25
95.32
Triplet
m 0.1
93.68
86.02
61.94
36.96
95.19
m = 0.5
93.82
84.76
60.60
36.86
95.27
m 1.0
93.68
84.64
58.81
37.95
95.20
m = O)O
VIT
92.37
81.79
51.82
51.79
95.01
m 0.01
92.19
81.75
51.91
54.50
95.08
0.05
92.58
82.27
52.87
47.84
95.11
CT
m 0.1
93.38
83.00
54.31
38.24
95.06
0.5
92.99
84.62
60.03
48.71
95.31
m 1.0
93.47
84.16
58.44
46.43
95.22
CIT (Ours)
93.88
86.23
61.88
36.65
95.38
16

---

Under review as a conference paper at ICLR 2023
Table 8: Comparison of Triplet and CT with different margins for the In-shop Clothes dataset on CNN and VIT networks in terms of rank-1 (in %) accuracy, normalized mutual information (NMI, in %),, spectral variance (SV, in %), mean average precision (mAP, in %),, and mean inverse negative penalty (mINP, in %).
In-shop Clothes
Method
Margin
Network
R&1
mAP
m/NP
SV
NMT
m 0.0
93.53
78.21
61.71
47.99
94.48
m 0.01
93.44
78.60
62.34
47.83
94.47
0.05
93.37
78.47
62.58
47.91
94.54
Triplet
m 0.1
93.66
78.36
62.09
47.63
94.51
0.5
93.39
78.61
62.40
48.85
94.54
m = 1.0
93.37
78.07
61.57
50.31
94.39
0.0
CNN
93.44
78.73
62.37
48.72
94.53
m = 0.01
93.37
78.33
61.88
50.45
94.45
0.05
93.75
79.17
62.87
46.64
94.62
CT
m = 0.1
93.44
78.27
61.87
48.33
94.51
m 0.5
93.57
78.33
61.92
48.31
94.46
m 1.0
93.61
78.43
62.12
48.09
94.49
CIT (Ours)
93.68
78.93
62.92
46.76
94.58
m 0.0
92.31
74.84
56.50
40.00
93.59
0.01
92.52
75.97
57.95
37.81
93.93
= 0.05
92.09
73.44
55.62
39.66
93.81
Triplet
m 0.1
92.70
76.58
59.19
37.62
93.93
0.5
92.55
75.84
58.64
38.99
93.85
m 1.0
92.45
74.50
56.82
38.33
93.59
0.0
VIT
91.35
72.43
53.40
39.54
93.12
m = 0.01
91.49
72.63
53.63
38.99
93.19
0.05
91.50
73.13
54.40
38.49
93.30
CT
m = 0.1
91.67
74.08
55.52
38.40
93.50
0.5
92.64
75.43
58.01
38.16
93.75
m 1.0
92.47
74.65
56,91
38.37
93.64
CIT (Ours)
92.73
76.62
59.83
37.51
93.94
17

---

Under review as a conference paper at ICLR 2023
120
Miplet
CT Dm3. km000009
Gircle
y=1.0.
300
ST
0.6
20
022
19
10
20
50
40
25 30
20
0.0
Tpech
Epocs
Figure 5: Convergence curves of CNN for different loss functions on the Market1501 dataset. Left shows that our CIT can achieve the same convergence effect as Triplet and CT, smoother than Circle, ST. and Angular, which need tuning more hyper-pane paeer-paramterss. And right plot again proves that the hyper-parameter can help speed the convergence of our CIT. signifies the angle factor straight line and is used to measure the decline speed of loss curves roughly.
Markot1501
In-shop Clothers
Upplet m=0
triplet m=O
0.6
g.mms-0.3
camma-05
gammau1
ct gamman1
0.4
0.3
0.3
0.2
0.1
0.1
0.0
0.0
20
30
10
Epoch
Epoch
Figure 6: Convergence curves of CNN for CIT and Triplet loss functions CNN for the Mar- ket1501 and In-shop Clothes dataset. Triplet of m = 0.0 falls into the local optimum prematurely by observing occurence epochs of the zero loss. Triplet of m=0.0 has the same mining strategy CIT, but its L, form is weaker than CIT in the exponential form driving DML model optimization due insufficient exploitation of hard triplet samples. By setting γ 1.0, our CIT building similarity concordance can approach the convergence of Triplet with m=0.05.
18

---

Under review as a conference paper at ICLR 2023
Query
Top-4 retrieved images
(a)
(b)
(c)
(d)
Figure 7: Qualitative results of our CIT loss for (a) Market1501 dataset CNN, (b) Market dataset on VIT, (c) In-shop Clothes dataset on CNN, and (d) In-shop Clothes dataset on ViT. For each query image (leftmost), top 4 retrievals are exhibited. The results with red boundaries are false cases but they are substantially similar to the query images in terms of appearance.
19

---

Under review as a conference paper at ICLR 2023
Figure 8: Gradients on features of triplet samples.
20


Reviews:
[{"review_id": "w3ZS-PiyOh", "reviewer": "Reviewer_nK2W", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a Concordance-induced triplet(CIT) loss for Deep Metric Learning tasks. The major hypothesis is that the ordering concordance should be invariant to any monotone transformation of the decision boundary of triplet loss. Therefore, CIT loss should with concordance can help avoid the plague of turning the violation margin. Moreover, the paper also introduces the partial likelihood term on hard triplets to speed up convergence. Multiple benchmark experiments are conducted for the new loss.", "summary_of_the_review": "In sum, SOTA is important for DML research but it's not the only type of metric to evaluate the new loss function. Please check out the Strength And Weaknesses. ", "strength_and_weaknesses": "Strength\nThe proposed CIT triplet loss avoids tuning the margin hyper-parameter used in the regular triplet loss. \nThe experiments are well-conducted across different types of datasets\nThe paper is well-organized and written.\n\nWeaknesses\nThough multiple datasets results are provided, they are all about retrieval performance. There should be other types of evidence to support the improvement of the new loss functions such as the embedding space structure(TSNE plots), embedding space density, and spectral decay[1] which helps understanding and metrics to show where is gained coming from with the new loss function.\n\nThe author proposes to avoid the tuning of the margin hyper-parameter. But the new loss function introduces another hyper-parameter \\gama, which is dimming the novelty.\n\nIn figure 3, why all the ending points are not with the same epoch size? The training of VIT on In-shop Clothes is not stable. Suggestion: it's better to train the model multiple times and average the curve for these plots.\n\nit's much more interesting to see the gradient difference for the CIT loss and have more discussion on the gradient level and whether the proposed idea fits the explanation of the gradient. In the appendix, it's better to show the gradient in the format of the loss term w.r.t to features x_a, x_p,x_n\n\n\nsome minor issues\nin Section 2.1 the similar hard triplet samples definition should be written in Sap-m<San<Sap. Although it's the same equation, semi-hard mostly describes the negative example. Put San in the relation to Sap is clear.\nmissing the definition of Spn\n\n\n[1] Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Björn Ommer, and Joseph Paul Cohen.\nRevisiting training strategies and generalization performance in deep metric learning. In Proceedings of the\n37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume\n119 of Proceedings of Machine Learning Research, pages 8242–8252, 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "i0dAvtMohjP", "reviewer": "Reviewer_FXz5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Conventional triplet-based losses require carefully tuning a decision boundary. To circumvent this issue, this paper proposes a novel yet efficient concordance-induced triplet (CIT) loss as an objective function to train deep metric learning model. Furthermore, it introduces a partial likelihood term for CIT loss to impose additional penalties on hard triplet samples, thus enforcing fast convergence. Extensive experiments are conducted on a variety of deep metric learning tasks to demonstrate the advantages of the proposed method.", "summary_of_the_review": "This paper proposes a novel concordance-induced triplet (CIT) loss to train deep metric learning model, embracing the advantages of no need for margin tuning and fast convergence. However, there are some limitations as mentioned in the Weakness. I would like to hear the feedback from the authors.", "strength_and_weaknesses": "**Strength**\n1. The proposed CIT loss is new to me, and shows the advantages of elegance and simplicity.\n2. The paper is easy to understand and well written. The motivation of this paper is solid.\n3. Extensive experiments are conducted to demonstrate the superiority of the proposed method.\n\n**Weakness**\n1. In Eq. 4, does it need to tune for the term of ‘1’ in the range of (e$^{-1}$, e) or will there be gains from tuning it?\n2. Figure 1 only shows the property of fast convergence compared with the variant of itself, how about compared with other losses (e.g., circle loss, soft triple loss) ?\n3. Hard mining usually shows benefits to model training, but it seems inconsistent with the results from Tab.3, could the author provide some explanations or discussions?\n4. Results in Tab. 1 and 2 are not always competitive compared with others. From Tab. 1, results of Circle and Angular are fluctuant with different values of margin, but those of Triplet and CT are much more stable. What are the values of margin used in Tab. 2. I am wondering, among these three tasks, if methods like Triplet or CT really need a careful tuning \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GUKcOfkLx_E", "reviewer": "Reviewer_mdeV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims at addressing two issues with the triplet loss: 1) needs to set a global violation margin, and 2) slow convergence during training. The paper proposes a \"Concordance-Induced Triplet\" (CIT) loss, which consists of two parts, one can be considered an exponential form of the conventional triplet loss with margin = 0; the other can be considered a variant of the softmax cross-entropy classification loss within each triplet.\n\nExperiments on three tasks (person re-identification, image retrieval, face recognition) show that the proposed CIT loss leads to comparable accuracy and convergence rate with the conventional triplet loss, without having to tune the margin hyperparameter.", "summary_of_the_review": "The paper presents some interesting improvements to the triplet loss, but as elaborated above, 1) it lacks some definitions/discussions to make the proposed improvements more theoretically grounded, and 2) the significance of the proposed method is arguably marginal according to the experiments. Thus, I would recommend a reject of the paper in its current version.", "strength_and_weaknesses": "### Strengths\n\n1. The paper is quite well motivated at the high-level. It states clearly what issues to be addressed; how they propose to address them; and the experiments validate the proposal.\n\n2. The related works are discussed quite thoroughly.\n\n3. Experiments are conducted on three different tasks, which validate the effectiveness of the proposed method in more applications.\n\n### Weaknesses\n\n1. The paper is written based on the concept of \"concordance\". What is the formal definition of \"concordance\"? Can the conventional triplet loss in Eq(2) be considered a \"concordance-induced triplet\" loss? The paper presents several fancy loss functions, but the foundation does not seem to be that solid.\n\n2. Some detailed tech choices are not well motivated or discussed. For example,\n\n    1. Eq(4), why is it called the exponential lower-bound form? I assume it is because $1 - e^{-x} \\leq x$?\n    2. Eq(4), why do we have to take the exponential lower-bound form? Why cannot we just set the margin = 0 for the conventional triplet loss in Eq(2)?\n    3. Eq(6), what is the \"partial likelihood\" for? Why cannot it be computed as $\\frac{e^{S_{ap}}}{e^{S_{ap}} + e^{S_{an}} + e^{S_{pn}}}$ or other forms inspired by the cross-entropy softmax classification loss?\n    4. The proposed CIT loss formula in Eq(8) reminds me of the classification + triplet + center loss used in [a]. How do they compare?\n\n[a] Bag of Tricks and A Strong Baseline for Deep Person Re-identification, CVPRW 2019\n\n3. There is a typo / inaccurate definition of the set of comparable pairs $E_T$ in section 2.2. Currently it enforces $S_{ap} > S{an}$, which does not seem to be right? Otherwise, the Eq(4) and Eq(5) will become meaningless.\n\n4. The empirical significance of the proposed method is arguably marginal:\n\n    1. In Table 1, the conventional triplet loss seems to be quite robust against the margin choices, and the results are all quite competitive. Then is tuning the margin really a trouble in practice? And how about we just set the margin = 0 as default?\n    2. In Figure 3, $\\gamma < 1.0$ enables the partial likelihood loss Eq (7) and $\\gamma = 1.0$ disables it. However, the figures suggest that the convergence rates are not that different. The difference is slightly more obvious in ViT but still not very convincing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-H1wmAz34v", "reviewer": "Reviewer_TC7M", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles triplet loss learning by introducing a novel loss function: CIT. The approach is demonstrated for different applications.", "summary_of_the_review": "Overall, the approach seems to be feasible and is based on theoretical foundation. The paper is well written and thus easy to read. On the downside, however, the experiments do not show clear benefits to the state-of-the-art.", "strength_and_weaknesses": "Strength:\n\nThe introduction gives a clear problem statements and a motivation for the proposed approach. In this way, also the related work is caputed in a meaningful way.\n\nThe technical description is simple but seems to be reasonable.\n\nWeaknesses:\n\nThere is only a slight (if any) improvement compared to the used baselines?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject