Threshold-Consistent Margin Loss for Open-World Deep Metric Learning
Authors: Qin ZHANG
Linghan Xu
Jun Fang
Qingming Tang
Ying Nian Wu
Joseph Tighe
Yifan Xing
Venue: ICLR 2024 poster (2024)
Keywords: Deep metric learning
Open-world visual recognition
Threshold consistency
Primary area: metric learning, kernel learning, and sparse coding
TL;DR: Quantify the notion of threshold inconsistency in deep metric learning through a novel variance-based metric, and introduce a simple yet effective regularization loss to improve threshold consistency for DML models.

Abstract: Existing losses used in deep metric learning (DML) for image retrieval often lead to highly non-uniform intra-class and inter-class representation structures across test classes and data distributions. When combined with the common practice of using a fixed threshold to declare a match, this gives rise to significant performance variations in terms of false accept rate (FAR) and false reject rate (FRR) across test classes and data distributions. We define this issue in DML as threshold inconsistency. In real-world applications, such inconsistency often complicates the threshold selection process when deploying large-scale image retrieval systems. To measure this inconsistency, we propose a novel variance-based metric called Operating-Point-Inconsistency-Score (OPIS) that quantifies the variance in the operating characteristics across classes. Using the OPIS metric, we find that achieving high accuracy levels in a DML model does not automatically guarantee threshold consistency. In fact, our investigation reveals a Pareto frontier in the high-accuracy regime, where existing methods to improve accuracy often lead to degradation in threshold consistency. To address this trade-off, we introduce the Threshold-Consistent Margin (TCM) loss, a simple yet effective regularization technique that promotes uniformity in representation structures across classes by selectively penalizing hard sample pairs. Large-scale experiments demonstrate TCM's effectiveness in enhancing threshold consistency while preserving accuracy, simplifying the threshold selection process in practical DML settings.

Published as a conference paper at ICLR 2024
THRESHOLD-CONSISTENN MARGIN LOSS FOR OPEN- WORLD DEEP METRIC LEARNING
Qin Zhang's Linnhan Qingming Tang", Jun Fang', Ying Nian Wu', Joe Tighe', Yifan Xing'
1 AWS AI Labs, 2 Alexa Al iqzaamz, Linghax, gmtang, junfa, wunyin, yifax (5ffaxeeamzzen @amazon. com, jtighelcs unc. edu
ABSTRACT
Existing losses used in deep metric learning (DML) for image retrieval often lead to highly non-uniform intra-class and inter-class representation structures across test classes and data distributions. When combined with the common practice of using a fixed threshold to declare match, this gives rise to significant perfor- mance variations in terms of false accept rate (FAR) and false reject rate (FRR) across test classes and data distributions. We define this issue in DML thresh- old inconsistency. In real-world applications, such inconsistency often compli- cates the threshold selection process when deploying commercial image retrieval systems. To measure this inconsistency, we propose a novel variance-based metric called Operating-Poinn Point-Inconsistence the operating characteristics across classes. Using the OPIS metric, we find that achieving high accuracy levels in DML model does not automatically guaran- tee threshold consistency. In fact, our investigation reveals Pareto frontier in the high-accuracy regime, where existing methods improve accuracy often lead to degradation in threshold consistency. To address this trade-off, we introduce the Crrssoold-Conssstentt Margin (TCM) loss, simple yet effective regulariza- tion technique that promotes uniformity in representation structures across classes by selectively penalizing hard sample pairs. Extensive experiments demonstrate TCM's effectiveness in enhancing threshold consistency while preserving accu- racy, simplifying the threshold selection process in practical DML. settings.
(OPIS) that quantifies the variance in
INTRODUCTION
Deep metric learning (DML) has shown success in various open-world recognition and retrieval tasks (Schroff et al. 2015a; Wu et al. 2017 Deng et al. 20199 Wang et al.) 2018).. Nevertheless,
the common DML losses, such as contrastive loss (van den Oord al. 2018 Chen et al. 2020).
pairwise loss (Brown et al.] 2020; Patel et al. 2022) and proxy-based losses (Kim et al. 2020
Movshovitz-Attias al. 2017 2 Qian al. 2019: Deng et al. 2019), often yield highly varied intra-
class and inter-class representation structures across classes (Liu et al., 2019 Duan et al., 2019 Zhao et al. 2019). Hence, even if an embedding model has strong separability, distinct classes may still require varying thresholds to uphold a consistent operating point in terms of false reject rate (FRR) or false acceptance rate (FAR). This challenge is particularly important in real-world image retrieval systems, where threshold-basee retrieval criterion is preferred over a top-k approach due to its ability to identify negative queries without matches in the gallery. However, selecting the right threshold is difficult, especially when systems must cater diverse use-cases. For instance, in clothing image retrieval for online shopping, the similarity between two T-shirts can be significantly different from that between
coats. A threshold that works well for coats may lead to poor relevancy and give many false positives in the retrieved images for T-shirts, shown in Figure I These difficulties are more pronounced in the open-world scenarios (Scheirer al.) [2012] Bendale & Boult 2015 2016), where the test classes may include entirely new classes not seen during training.
We define the phenomenon in DML, where different test classes and distributions require varying distance thresholds to achieve a similar retrieval or recognition accuracy, as threshold inconsis- tency. In commercial environments, particularly under the practical evaluation and deployment
Equal contribution.

---

Published as a conference paper at ICLR 2024
(a) Threshold selection withour threshold consistent representution.
Threshold selection threshold representation.
baa
baccha
Dast
Pealty
Made
040
Madel
Deady
1.30
Deatty
A
Figure I: Here we show that (a) without threshold-conee consistent representation, selecting the right threshold for a commercial image retrieval system that serves a diverse range of test classes and distributions is challenging. It requires careful manual tuning of retrieval thresholds to strike balance across multiple datasets. However, (b) with threshold-consistent representation, different test distributions yield similar distance thresholds at the performance target, effectively simplifying the otherwise complicated manual threshold tuning process. In the plots, d" denotes the distance threshold selected to align the False Positive (FP) rate with a pre-defined target.
setting with one fixed threshold for diverse user groups (Liu et al. 2022). the significance of thresh- old inconsistency cannot be overstated. Accurate quantification of this inconsistency is essential for detecting potential biases in the chosen threshold. To this end, we introduce novel evaluation -Serre (OPIS), which quantifies the variance in the
metric, named Conntting-Poin--nnoonssi operating characteristics across classes within a target performance range. Using OPIS, we observe an accuracy- settraccyythehholdd consistency Pareto frontier in the high accuracy regime, where methods to improve accuracy often result in a degradation in threshold consistency, as shown in Figure 3 This highlights that achieving high accuracy does not inherently guarantee threshold consistency.
One solution to this problem is using posthoc calibration methods (Platt et al. 1999; Zadrozny &
Elkan 2002; Guo et al.] 2017a), which adjust trained model's distance thresholds to align with specific operating points in FAR or FRR. However, in real-world settings, these methods can be inefficient and lack robustness, as they involve constructing separate calibration datasets and may require prior knowledge about the test distribution for effective calibration (Naeini et al. 2013 Guo jt al, 2017 Moreover, they do not address the threshold inconsistency problem unless customized calibration is done for each user. Another option is employing conformal prediction (Romano et al.) 2020 Gibbs & Candes) 2021), which guarantees confidence probability coverage and can handle complex data distributions as well as covariate and label shifts. However, conformal prediction in- herently assumes a closed-wort world setting, where training and test samples share the same label space. In contrast, real-world image retrieval systems typically operate in open-world environment, pre- senting a more complex and realistic setting with unknown classes at test time.
Given these challenges, an essential question arises: Can we train embedding model for open- world image retrieval that sustains a consistent distance threshold across diverse data distributions, thus avoiding the complexities of posthoc threshold calibration? This objective falls within the scope of calibration-aware training. In closed-set classification, the goal of calibration-awar align predicted confidence probabilities with empirical correctness of the model (Guo et al.) 2017a Müller et al. 2019 Mukhoti et al. 2020). However, our focus lies on what we term as threshold- consistent DML, paradigm that trains an embedding model with reduced threshold inconsistencies, such that a universal distance threshold can be applied to different test distributions to attain a similar level of FAR or FRR. This differentiation is crucial because in DML the output similarity score does not strictly reflect the empirical correctness of the model (Xu et al., 2023) and may exhibit strong variations across test data distributions. To address the unique challenges of threshold inconsistency in DML, we propose a simple yet effective regularization technique called Threshold-Consisteer Margin (TCM) loss. Through experiments on four standard image retrieval benchmarks, we vali- date the efficacy of the TCM regularization in improving threshold consistency while maintaining accuracy. To summarize, our contributions are as follows:
wainiig is to
We propose novel variance- based metric, named Operating- www---oonnnnnnistteeeeeeteeeeeee (OPIS), to quantify the threshold inconsistency of a DML model. Notably, OPIS does not need separate hold-out dataset besides the test set, enhancing flexibility in evaluation.
2

---

Published as a conference paper at ICLR 2024
We observe securacy-thresheee consistency Pareto frontier in the high accuracy regime. This finding underscores that achieving high model accuracy in DML does not automati- cally guarantee threshold consistency, necessitating dedicated solutions.
We introduce the Crreshold-Consister Margin (TCM) loss, a simple yet effective regular- ization technique, that can be combined with any base losses and backbone architecture to improve threshold consistency in DML. Our approach outperforms SOTA methods across various standard image retrieval benchmarks, demonstrating substantial improvements in threshold consistency while maintaining or even enhancing accuracy.
2 RELATED WORKS
DML losses for image retrieval Advancements in DML losses for image retrieval have focused on improving accuracy, scalability and generalization Brown et al. (2020); Patel et al. (2022): Deng
et al. (2020); Kim et al. (2023); Roth et al.] (2020g Kan et al. 2022 Ypsilantis al. (2023) The
pioneering work of the Smooth-AP loss (Brown et al. 2020) optimizes a smoothed approximation
for the average precision. Similarly, the Recall@k k Surrogate loss (Patel et al. 2022) approximates the recall@t & metric. Leveraging vision-transformer backbones and large batch sizes, Recall@k Surrogate has achieved remarkable performance in several image retrieval benchmarks. However, these pairwise methods are inefficient when dealing with a large number of classes. To reduce the computational complexity, proxy-based methods such as ProxyAnchor (Kim et al., 2020). Prox- yNCA (MovshovizzAAttias et al., 2017), SoftTriple (Qian et al.) 2019), ArcFace (Deng et al. 2019), and HIER (Kim al. 2023) are employed, where sample representations are compared against class prototypes. Despite high accuracy, these methods still face challenges in biases and fairness (Fang cet al. 2013; [[vento 2019; Dullerud et al.]| 2022) and display inconsistencies distance thresholds
when applied in real-world scenarios (Liu et al. 2022).
Evaluation Metries for threshold consistency (inconsistency) In closed-set classification, thresh- old consistency is usually evaluated through calibration metrics, such Expected Calibration Error (ECE) (Nacini et al., 2015), Maximum Calibration Error (MCE) (Guo et al.] 2017b) and Adaptive ECE Nixon et 2019). These metrics gauge how well a model's predictions match actual correct- ness. However, directly applying them to evaluate threshold consistency in DML (e.g.. by replacing confidence probability with similarity measures) is not straightforward. A key hurdle that DML uses distance measurements to represent semantic similarities, and these distances can vary widely across different classes due to the intrinsic bojetijectsvenees of semantic similarity in the data (Roth iet al., 2022). In the context of DML, OneFace (Liu et al. (022) introduced the calibration threshold for face recognition systems, which corresponds to the distance threshold at given FAR of a sepa- rate calibration dataset. They further propose the mme-Threshood-for
(OTA) evaluation protocol to measure the difference in the accuracy performance across datasets at this calibration threshold as an indicator for threshold consistency. However, this approach requires dedicated calibration dataset, which can difficult to acquire practice. To our knowledge, there is no widely accepted and straightforward metric for threshold consistency in DML.
Calibration-awa aware training vs Posthoe threshold calibration Calibration-wwac training has been well studied in closet-set classification, where the goal is to align predicted probabilities with empir- ical correctness (Guo et al. 2017a| Müller et al., 2019 Mukhoti et al. 2020). Common approaches use regularizer to guide the model in generating more calibrated predictions (Pereyra al.) 2017
Liang et al. 2020 Hebbalaguppe et al.) ).22). Yet, thresholld-onnsisten training for DML dif- fers from calibration-aware training. Instead of aligning model output with empirical correctness, threshold-consisten DML seeks to maintain a consistent distance threshold across classes and data distributions. In face recogmition, [Liu et al.) (2022) introduces the Threshold Consistency Penalty to improve threshold consistency among various face domains. The method divides mini-bur batch data into 8 domains and computes each domain threshold using a large set of negative pairs from a fea- ture queue. It then adjusts the loss contribution from each sample based on the ratio of its domain threshold to the in-batch calibration threshold. However, this method is designed for face recog- nition- more constrained scenario. In contrast, our target is general image retrieval tasks which can involve significantly more domains, making it impractical to construct negative pairs for all domains. Besides train-time methods, another approach is posthoc threshold calibration, such as Platt calibration (Platt et al. 1999), isotonic regression (Zadrozny & Elkan, 2002) and temperature scaling (Guo al. 2017a), which seeks to calibrate the operating point of trained model using
3

---

Published as a conference paper at ICLR 2024
Amooth-AP, VIT-B/16
Smooth-AP TCM, VIT-8/66
SmoothsAP, VIT-8/16
Smooth-AP TOM, VIT-B/16
0.70
Rocal 79.4
0.70
Reodigg: 0.
10
OPSS 0.30e-3
0:6
OPIS: 0 200-
0.0
6:5
00-
0.7
E1 ST
0.7
Rood 0
OP1 3.50
0.75
1.00
150
175 2.00
(a) INaturalist-2000 2018
(b) Cars-196
Figure 2: Utility- Distance Threshold Curves are presented for test classes in the iNaturalist and Cars- 196 datascts. Each curve represents a class in its respective datasct. The calibration range is underscored by the red lincs. As defined in Eq the calibration range is based on pre-defined FAR FRR target. Thus, changing the loss can result in minor shifts in the calibration range. After integration of the TCM regularization, a significant enhancement in the alignment of utility curves across various classes is observed, accompanied by a notable enhancement in threshold consistency, as indicated by the reduction in OPIS by up to 55%.
hold-out calibration datasets. However, it cannot solve threshold inconsistency unless customized calibration is conducted for each user. Another category of posthoc calibration method conformal prediction (Tibshirani et al.) Rooaaoo et al. 2020; Gibbs & Candes) 2021 Barber et al.) 2023), which can be applied beyond the setting of exchangeable data even when the training and test data are drawn from different distributions. However, conformal prediction relies closed-set setting where the training and test data share the same label space, which does not apply to open-world im- age retrieval. Thus, in this work, we focus on developing threshold-consister training technique tailored for DML, with the goal of simplifying the posthoc calibration process in practical settings.
3 THRESHOLD INCONSISTENCY IN DEEP METRIC LEARNING
Visualizing threshold inconsistency in image retrieval We visually illustrate the issue of thresh- old inconsistency in DML using image retrieval datasets. First, we borrow the widely-used F- score (Sasaki al., 2007) to define the utility score, incorporating both sides of the accuracy metric (e.g. precision and recall, or specificity and sensitivity). Specifically, we denote one side as o and the other side as v and define the utility score, denoted as U, as follows:
U(d) (1+ ((d) (d)
c + (dd)
where d is the distance threshold (d e [0,2] for hyperspherical embeddings), and e is the relative importance of vb over & (c 1 if not specified). Without loss of generality, we let & be specificity (same as TNR or 1 FAR) and 1a be sensitivity (same recall or 1 FRR)
In Figure 2 we present the accuracy uiility-disannce threshold curves for the test classes using mod- els trained on the iNaturalis-20018 (Horn et al., (017)) and Cars-196 (Krause et al. (013)) datasets. In the left column of each subfigure, we observe considerable variations in the operating character- istics among distinct classes for models trained with the popular Smooth-AP loss. These variations make it difficult to select a single distance threshold that works well across the entire spectrum of test distributions. However, while will elaborate on in later sections, incorporating our proposed TCM regularization during training visibly improves the threshold consistency across classes, as evidenced by the more aligned utility curves compared to those without the TCM regularization.
OPIS for overall threshold inconsistency To quantify threshold inconsistency in DML, we intro- duce a variance-based metric, Operating-Pointe Pwwnt--Inonnnssstenettttttetttt ation proposed in Liu et al. (2022). OPIS does not require separate calibration dataset. It quantifies the variance in the operating characteristics across test classes in a predefined calibration range of distance thresholds. This calibration range, denoted as [dmin ] is typically determined based on the target performance metric operating ranges (e.g.. a <FAR< < b, where a, are determeedddedd error constraints). Formally, the OPIS metric can be expressed as follows:
Score (OPIS). Unlike the OTA evalu-
ΣJ Samn [[U,(d) (dd)l dd T (dmmx dmin)
OPIS
(2)
We employ the specificity and sensitivity pair because they are particularly relevant for visual recognition applications and are not sensitive changes in test data composition.
4

---

Published as a conference paper at ICLR 2024
where i = .., ((d) is the average utility for the entire test dataset.
T is the index for the test classes, (,)d) is the accuracy utility for class i, and
c-OPIS for utility divide between groups The overall OPIS metric does not emphasize the outlier classes. For applications where outlier threshold consistency is essential, we also provide a more fine-gr grained metric that focuses the utility disparity between the best and worst sub-groups. First, we define the utility of the percentile of best-performing classes as follows:
Dbbert (d) 3'Eben (d)
U U (d)
(3)
bane (d) + Vebare (d)
where Sesma (d), ebe (d) are the expected accuracy metrics for the entirety of the percentile of the best-performing classes. By replacing Ebest with Eworst. the same can be defined for U (d). Then, we define the E-OPIS metric as the following:
1
ddmax
E-OPIS = dmax dmin
eearrae (d) Uswea Sbeat (d)ll dd
(4)
Amin
By definition, the e-OPIS metric is maximized at & - O, and eventually becomes zero when & 100% as the best-performing and worst-performing set become identical.
High accuracy jf High threshold consistency In Figure 3 employ the OPIS metric to ex- amine the relations between threshold inconsis- tency and recognition error in embedding mod- els trained with various DML losses, backbones and batch sizes. Notably, we observe distinct behaviors across different accuracy regimes. In the low-accuracy regime, located in the right of the plot, we notice a simultaneous improvement of accuracy and threshold consistency. This aligns with the established notion that improv- ing model discriminability helps threshold con- sistency by strengthening the association be- tween samples and their corresponding class centroids. However, as the error decreases, trade-off surfaces in the high-accuracy regime. Here, the reduction in error is correlated with increased threshold inconsistency, leading to the formation of a Pareto frontier.
Ka.
Recognition Error vs Threshoid Inconsistency
-32
stusee rur rad win 100 aealtiation of fecomnc semppetinn fecoonee 3 norasaa meriggg dacressas
Threshold incesalliicoratttotoo incorsistency (OPIS)
-3.4
GAP
-38
377-
cuf Prti
-300
juant)
CRNM
1411))
RSOAITEM
-3.8
Recagnition error (100 recalig))
Figure 3: The plot depicts the relations between recog- nition (measured by 100 recall 1, the lower the better) and threshold inconsistency (measured by OPIS, the lower the better) across low- and high-accuracy regimes. Each represents trained DML model. with its size indicating the batch size used during training. In the low accuracy regime, located in the right side of the plot, there is a simultancous improvement threshold consistency and accuracy. as highlighted by /. How- ever, beyond a certain point, Pareto frontier emerges (indicated by K). where enhancing accuracy comes the expense threshold consistency. Notably, the in- clusion of our proposed TCM regularization (marked in A) leads to substantial OPIS reduction, well below the marked Pareto frontier. Best viewed in color.
The trade-off between recognition error and threshold inconsistency highlights that achiev- ing high accuracy alone does not automatically guarantee threshold consistency. In this con- text, introducing the proposed OPIS metric as an additional evaluation criterion alongside re- call (okk is crucial for streshold-basce commer- cial DML applications, where the ability to identify negative queries without matching classes in the gallery is of importance. To explain further, we compare OPIS with the widely-used accuracy metric, recall@k. These two metrics evaluate different aspects of model and can be used com- plementarily: recall@ k focuses on top-k relevancy (retrieving top-k similar samples the query from a collection), and OPIS measures the inconsistency in threshold-relevanccy (retrieving similar examples above threshold from collection). Moreover, unlike recall@ that solely gauges recall, OPIS evaluates both the FAR and FRR (-recall), offering more holistic error assessment.
4 TOWARDS THRESHOLD-CONSITTIE DEEP METRIC LEARNING
Margin (TCM) loss. TCM specifically penalizes hard positive and hard negative sample pairs the decision boundaries outlined by a pair of cosine margins. This strategy is in line with several stud- ies (Dong et al. 2017 Xuan al. 2020 Robinson et al. 2020) that emphasize hard mining for
To tackle the threshold inconsistency problem, we introduce the
5

---

Published as a conference paper at ICLR 2024
M. comcoldingg
DNW
TOM meg Jertine
Margerbased Softra
O Pa
[4] Figure 4: (a) An overview of the regularization are combined in an additive fashion to reduce the trade off between accuracy and threshold consistency, (b) Distinguishing TCM from Stagin-BBase Softmax Loss such as Deng et al. (2019). See TCM vs Margin-based Softmax loss section for detailed explanation of the differences. 0g represent the intra-class are lengths for the blue and red classes, respectively. æ1 and 2g are both instances of the blue class with class centroid W1, whereas 72 belongs to the red class with centroid W2. In this case, are hard negative sample pairs, and æı, I3 are hard positive sample pairs. Best viewed in color.
consistent DML training framework. Here, the base loss and TCM
this illustration. O1 and
extracting more informative samples. Let St and S be the sets of cosine similarity scores for positive and negative pairs in mini-batch, respectively, the TCM loss is formulated as follows:
LTCM x E.es+ (m+ - s) 1 1asm + Exes- (s m 1 =2m=
Exes+ 1.gm!
EvEs- 1.2"
where condition if the condition is true, and otherwise. X and A are the weights assigned to the positive and negative regularizations, respectively. The TCM regularizer can be combined with any base loss Lasse resulting in the final objective function:
Linull Liase + LTCM
(6)
Design justification: representation structures Several works have shown strong correlation between model accuracy and representation structures YYu et al. 2020 Chan et al. 2022). Indeed, SOTA DML losses are designed to optimize this relationship by encouraging intra-class compact- ness and inter-class discrimination. However, when considering threshold consistency, the focus shifts towards achieving consistent performance in FAR and FRR in the calibration range, with an emphasis on local representation structures near the distance threshold. In this context, the TCM regularization serves as a "local inspector" by selectively adjusting hard samples to prevent over separateness and excessive compactness in the vicinity of the margin boundaries. This strategy also aligns with previous work that found excessive feature compression actually hurts DML general- ization (Roth et al. 2020). Since the margin constraints are applied globally. this helps encourage more equidistant distribution of class centroids and more uniform representation compactness across different classes in the embedding space.
Hard mining strategy TCM regularizes on hard samples, distinguishing from techniques that encourage similarity consistency by minimizing marginal variance (Kan et al. 2022 Specifically, TCM's hard mining strategy is different from the semi-hard negative mining strategy (Schroff et al. 2015b) and its variants (Oh Song et al. 2016 Wu et al., 2017 Wang al.) 2019), TCM's hard mining is based on the absolute cosine similarity values, rather than their relative differences. Mean- while, TCM also differs from ROADMAP (Ramzi al., (021)) in that TCM utilizes hard positive and negative counts, whereas ROADMAP uses the total positive and negative counts. This makes TCM well-suited for scenarios involving large batch sizes (as is the standard in DML) and significant imbalances between the counts of positive and negative pairs?
Connection the calibration range TCM is implicitly connected to the calibration range of the OPIS metric through the two cosine margins. Since cosine similarity is bijective with the Lg dis- tance for hyperspherical embeddings, these margin constraints ensure that the model's intra-class and inter-class representation structures adhere to the desired distance threshold range, which is
[V2- - 2m 22 - 2m However, due to the inevitable distributional shift between the training and testing datasets, the selection of the margin constraints requires some hyper-parameter tuning and cannot be directly estimated from the calibration range. In Figure 6 we give guidance on how to select the margins, with details discussed in the ablation of TCM margin hyperparameters.
TCM vs Margin-based Softmax loss TCM has distinguishing characteristics when compared to margin-based softmax losses (Deng et al. 2019 Qian et al., 2019), as illustrated in Figure {b).
An detailed comparison between TCM and the method of Ramzi al. 2021 is given in appendix A.2.2

---

Published as a conference paper at ICLR 2024
First, TCM is designed regularizer that operates in conjunction with base loss. It specifically applies to hard sample pairs that are located near the margin boundaries. Second, TCM employs two cosine margins to regularize the intra-class and inter-class distance distributions simultancously. This allows TCM to capture both hard positive and hard negative examples, resulting in more hard pairs within mini-batch. Secondly, the TCM loss is specifically applied to the hard pairs, contrast- ing with Areface, which is applied to all pairs. Last, TCM is a sample-level pair-wise loss, which better models the relationships between individual samples compared to proxy-based methods.
Visualization of TCM effect We visualize the effect of the TCM regularization on represen- tation structures across the 10 classes in the MNIST dataset of handwritten digits (LeCun et (9988) by training shallow CNN using the Areface loss (Deng al.) 2019). For clearer visualization,
ArcFace
ArcFace + TCM
use two-dimensional features and employ kernel-density estimation (Chen 2017) to model the probability density function for the embeddings of each class. As shown in Figure 5] compared using ArcFace (Deng let al., (019)) only, the incorporation of TCM (ArcFace+TCM) enhances the separation be- tween digits 2 and 5 (lower middle), O and 8 (lower right), and 4 and 9 (upper left). This observation supports our claims about TCM's ability in refining the representation structures for improved threshold consistency.
Tog-1 acc: 96.11
Top- I boc: 97.36
Moan PR-AUC: 96.43
Mean PR-AUC: 97.29
Figure 5: We present visualizations of 2D embed- ding distributions for the MNIST dataset, both with and without TCM regularization. In the figure, each arrow's direction corresponds to a class centroid and is labeled with its respective digit in white. The width of the line perpendicular to each arrow reflects the intra-class rep- resentation compactness, with narrower lines indicating more compact embeddings. The color intensity con- veys the probability density distribution of embeddings within each class, with higher density depicted in red.
5 EXPERIMENTS
5.1 DATASETS AND IMPLEMENTATION DETAILS
Datasets For training and evaluation, we use four commonly-used image retrieval benchmarks, namely iNaturalist- 2018 (Horn et al. 2017). Stanford Online Product (Song et al. 2015). CUB-200- 2011 (Wah et al., 2011) and Cars- 196 (Krause et al. 2013). These benchmarks cover a diverse set of data domains including natural species, online catalog images, birds, and cars. As in previous works (Brown et al., 2020 Patel et al. 2022; An al. 2023), , the iNaturalist and Stanford Online Product datasets use an open-world train-test-split,, where the training classes are disjoint from the ones in testing. For CUB and Cars, use shared train-test classes to make fair comparisons with [7 The details to each dataset can be found in Table |
prior DML m
Evaluation metries We measure model accuracy using the recall@k metric and assess threshold inconsistency using the OPIS and e-OPIS metrics as defined earlier. Similar to previous works (Veit & Wilber 2020 Liu et al. 2022), we estimate threshold inconsistency by comparing normalized features of image pairs in I:1 comparisons. In the case of the N 88 and Stanford Online Product datasets, given the large number of classes, we only sample positive pairs exhaustively and randomly sample negative pairs with fixed negative-to-pont topossiive ratio of 10-to-1 for each class. All positive and negative pairs the CUB and Cars datasets are exhaustively sampled.
Implementation details We use two backbone architectures, namely ResNet (He et al. 2016) and Vision Transformer (Dosovitskiy et al. 2020), both pretrained on ImageNe Since the original papers do not report OPIS, we train both baseline models (without TCM) and TCM-regularized models using the same configuration. The hyperparameters for each base loss are taken from the original papers. For TCM, we set A x 1. For OPIS, the calibration range is set to le-2 < FAR < le-1 for all benchmarks. The margin parameters (m m ") are tuned using grid search on 10% of the training data for each benchmark. We adopt the same optimization schemes as specified
3 We also provide results for CUB and Cars in the open-world setting in Appendix A.2.5
For ResNet, we follow Brown al. (200)) and use ImageNNtt-retraing hackhones. For ViTs, we fol- low Patel et al. 2022 and use ImageNet-211 2 pretrained backhones released by timm library (Wightman 2019
7

---

Published as a conference paper at ICLR 2024
Table 1: Dataset statisties. The datasets with open- Table 2: The influence of TCM regularization on dif- ferent base losses for ResNet50 backbones.
world train-test split are highlighted in light gray.
Train
Test
Dataset
* Ims
+Cs
Ims//C
* Ims
Cis
+Im/CI
Liuus + Lms
Re1
R48
16
iNat
325846
5,690
57.3
136095
2,452
55.5
ProxyNCA TCM
63.1
(1.4
88.3 (11.0)
(10,60)
59551
11318
60502
11316
Auclace ICM
63.6 (7.00
(199)
(10.5)
(10,05)
CUB
5994
200
30.0
5791
200
29.0
SAP + TCM
68.1 (177)
82. (110)
91.1 (10.7
Cars
8054
196
8131
196
41.5
RSOK+ + TCM
72.2 C155)
84.9 (1.(
0.11 (11.0
Table 3: Impact of TCM regularization on various DNN models trained with the Recall @k Surrogate loss
Table 4: Time complexities of TCM in compari- son to the Recall@k Surrogate loss the 196s-196 dataset. The ViT-B/16 backbone is utilized with 8x Tesla V100 GPUs and a batch size of 392.
batch size of 4000 in Patet et al. (2022)
Arch dem
R@l
R@4
Rerlo
OPISx 1e-3
ResNe150
(1.)))
(1.9)
(1100
0.11
ResNet10p913
73.8 (1L.)
(1.69
0.14
Method
Complexity
Nins
Aeekhiin (5)
Repsch (s)
WTT-S/6611
(1.6))
(10.5
0.17 (10.)
RS6k
0(a")
19.9
102.6
1313
WIT-B/16912
(10.8)
(10.))
96.5 (10.4)
0.17 (.0.20)
RSAK TCM
0
104.0
133.2
VIT-L/16912
85.7 (10.)
(10.7)
96.6 (10.7)
0.21 (.013)
Delm
0(1)
6.00%
+1.6%
+1.4%%
in the original papers for each base loss. During training, mini-batches are generated by randomly sampling 4 images per class following previous works (Brown et al. 2020 Patel et al. (022).
5.2 ABLATION AND COMPLEXITY ANALYSIS
Unless stated otherwise, all ablation studies are conducted using the iNaturalist dataset. Owing to space constraints, further ablations can be found in the appendix.
Effect of TCM margins We examine the impact of the cosine margins m m on aceuracy and OPIS. As shown in Figure 6 adding TCM consistently enhances threshold consistency compared to the baseline Smooth-AP loss across all combinations of margins, with up to 50% of reduction
in OPIS. Regarding accuracy, observe that the negative margin (m- has a greater in- fluence than the positive margin (m ), which aligns with previous works (Dong et al., 2017: Xuan al., 2020 Robinson et al.) 2020). How- ever, when the negative margin becomes exces- sively stringent, such as m 0.25, the ac- curacy drops below the baseline. We hypoth- esize that an overly restrictive negative mar- gin may interfere with the base loss, leading to decreased accuracy. For imageNet--retrained backbones, the recommended values for m and m are around 0.9 and 0.5, respectively.
Recalld91
OP15
2/00
0.50
0.
0.696
0.591
0.00
0.202
0.210
0.264
166
0.695
0.600
0.197
0.214
0.175
0.00
0.8
0.05
0.834
0.090
0.219
020
088
0.663
0.694
0.702
6.0
9.214
0.214
0.950
0200
0.82
0.030
0.692
0.700
0.80
0.255
0.220
0.109
0.35
0.55
0.50
025
0.575
0.50
11
Figure 6: Impact of TCM margins on recall and OPIS for ResNet50 models. The Smooth-AP P hase loss is utilized, yielding baseline results without TCM recall@ = 67.4% and OPIS- 3.
Compatibility with various base DML losses We select the most representative DML losses for each method category, including proxy-based methods Movshovitz-Attias et al. 2017 Deng et al., 2020) and baiewise-based methods (Brown al. 2020: Patel et al.) 2022). Notably, the Recall@k surrogate loss (Patel et al., (022) represents the SOTA loss for fine-grained image retrieval tasks. We run experiments using these base losses with and without the TCM regularization. As shown in Table 2 there is a consistent improvement in both accuracy (> 1.0% increase in recall 1) and threshold consistency (up to 60.7% in relative reduction) when TCM regularization is applied in conjunction with different serforerfon base losses.
Compatibilityy with different architectures We investigate the compatibility of TCM regularization with different backbone architectures including ResNet variants and Vision Transformers. As shown in Table we observe significant improvements threshold consistency across backbone architec- tures when TCM is incorporated. On accuracy, ResNet models exhibit more notable improvements in accuracy (> 1.5%) compared to Vision Transformers, which see a < 1.0% boost.
Time Complexity In mini-batch with size the complexity of TCM is O(n²) as it compares every sample with all samples in the mini-batch. For image retrieval benchmarks where the number of training classes K is significantly greater than the batch size n. i.e., K >> n, this complexity is comparable to most pase- losses (0(z²) and proxy-based losses (O(nK)). In Table 4 we provide time complexities for the loss computation, the forward and backward passes and the overall
8

---

Published as a conference paper at ICLR 2024
Table 5: Performance of supervised image retrieval after incorporating TCM regularization in @ecall@k (the higher the better) and OPIS (the lower the better) on 4 image retricval datascts. The numbers in black represent models trained with baaee - LTCM, while the colored numbers indicate improvement/ / degradation in absolute magnitude over models trained with Liaare alone. For Cars, with the same DINO backbone and Proxy Anchor base loss in Kim al. (2023), TCM achieves R @1 1 of 91.9, with 46.3% relative OPIS reduction.
Benchmark
Arch tim
Tome LTCM
BS
OPIS x
10%-OPIS x
R@1
Previous SOTA with ImageNet pretraining
Re
SAP + TCM
384
0.17
1.77
1000
69.1 TX
INaturalist-2018
RS@k TCM
4000
1007 775
1.25
72.2 11.
VIT-B/16912
SAP ICM
81.2
(ateel et 2022
RS@k + TCM
4000
0.17
1- 196
2.03
2500 mTL
84.8 10.
Res
SAP + ICM
384
0.06
U- 770
0.52
1117 t- 2010
82.7 12.
Stanford
RS@k + TCM
4000
0.07
880
0,74
83.3
R 1: 88.0 vilans
Online Product
VIT-B/16222
SAP + TCM
384
0.04
0.33
po e-
87.3 m0.
Patel et al. 2022
HMC
0.04
0,38
pour y- 37.4
88.4
ResNer50
SAP TCM
384
0,11
1,00
80.8
CUB-200-2011
RS@K TCM
384
0.10
1- 6383
11:00 (- a39)
80.0 mm
85.7 wmaw 2023
VIT-B/16412
SAP TCM
11000 (-005)))
88.4
RSØk + TCM
384
0.10
0.91
$300 (-7.580
87.6 Usu
ResSo150
SAP + TCM
384
0.39
Jour (- 33.3%)
3.33
$1.04
(- 2..0%
89.6 +2.7
1ars-196
RS@k+ TCM
392
0.45
2.93
p000 q- (9.0)
89.7 un
R@1: 91.3
VIT-B/16212
SAP TCM
384
0.54
(006 935
0.83
(35)
87.8
(Kim al. 2023
RS@k+ + TCM
392
0.60
0.98
11.21 6a.41)
87.7 -9.
time per epoch. The results suggest that adding TCM regularization results in a negligible (< 1.5%) increment in the overall training time per epoch.
5.3 IMAGE RETRIEVAL EXPERIMENT
The results for supervised fine-tuning for image retrieval benchmarks with and without the TCM regularizer are summarized in Table As is shown, our TCM loss is effective in improving threshold consistency (measured by OPIS and E-OPIS, the lower the better), by up to 77. 77.3%. compared the various baseline losses considered. Meanwhile, adding TCM regularization consistently improves accuracy across almost all benchmarks, base losses and backbone architectures. While we notice a slight decrease in recall@ I on the two smaller datasets (as marked in red), namely CUB and Cars, these are at the same magnitude as non-significant variations due to random initialization during training. It's worth highlighting that on i iNaturalist-2018, arguably the largest public image retrieval benchmark, adding our TCM regularization is shown to out-perform SOTA DML loss, recall@ k surrogate, reducing the OPIS threshold inconsistency score from 0.37 10 to 0.17 x 10 while improving the recall@1 accuracy metrics from 83.9% to 84.8%.
CONCLUSION
In this work, we comprehensively study the issue of threshold inconsistency in deep metric learning- We introduce novel based metric named Operating- Point- Inconsistency- Score (OPIS) to quantify threshold inconsistency among different classes. Distinct from the Tereshohsshoooor evaluation protocol proposed by Liu al. (2022). key advantage of OPIS is its elimination of the need for a separate calibration dataset. As a result, OPIS can be easily utilized alongside existing accuracy metrics, providing an added dimension for evaluating the threshold robustness of trained DML models. With the OPIS metric, we find that achieving high accuracy in DML model does not necessarily guarantee threshold consistency. To address this issue, we propose the Threshold- Consistent Margin loss (TCM), a simple and versatile regularization technique that can be integrated with any base loss and backbone architecture improve the model's threshold consistency during training. TCM is designed to enforce more uniform intra-class compactness and inter-class sep- arability across diverse classes in the embedding space. By incorporating TCM, we demonstrate state-of-the-arr performance in both threshold consistency and accuracy across various image trieval benchmarks. We hope that our work serves as a catalyst to encourage more explorations in developing threshold-consistenn DML solutions for practical open-world scenarios.
Limitations of OPIS The OPIS and e-OPIS metrics necessitate a sufficient number of samples per class to ensure statistical significance, making them unsuitable for few-shot evaluation scenarios. Linitations of TCM Like other inductive deep learning methods, TCM can fail when there's 'ss a significant distribution shift between the training and test sets when strong label noise is present.
9

---

Published as a conference paper at ICLR 2024
REFERENCES
Xiang An, Jiankang Deng, Kaicheng Yang, Jiawei Li, Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Unicom: Universal and compact representation learning for image retrieval. In ICLR, 2023.
L.C. Andrews, Special Functions of Mathematics for Engineers. Online access with subscrip- tion: SPIE Digital Library. SPIE Optical Engineering Press, 1998. ISBN 9780819426161. URL
https://books wtt.oooooooooooooooooooooooooo.e com com/books?Ld-z2AAge
ReboC
Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, (1 2023.
Abhijit Bendale and Terrance Boult. Towards open world recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1893-1902,, 2015.
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1563-1572, 2016.
Andrew Brown, Weidi Xie, Vicky Kalogeiton, and Andrew Zisserman. Smooth-ap: Smoothing the path towards large-scale image retrieval. CoRR, abs/2007.12163 2020. URL https::/
Kwan Ho Ryan Chan, Yaodong Yu, Chong You, Haozhi Qi, John Wright, and Yi Ma. Redunet: A white-box deep network from the principle of maximizing rate reduction. The Journal of Machine Learning Research, 23(1):4907-5009, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning. pp. 1597-1607. PMIR, 2020.
Yen-Chi Chen. A tutorial on kernel density estimation and recent advances, 2017.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4690-4699, 2019.
Jiankang Deng, Jia Guo, Tongliang Liu, Mingming Gong, and Stefanos Zafeiriou. Sub-center are- face: Boosting face recognition by large-scale noisy web faces. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 741-757, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58621-88
Qi Dong, Shaogang Gong, and Xiatian Zhu. Class rectification hard mining for imbalanced deep learning. In Proceedings of the IEEE international conference on computer vision, pp. 1851- 1860, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11922 2020.
Yueqi Duan, Jiwen Lu, and Jie Zhou. Uniformface: Learning deep equidistributed representation for face recognition. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3415-3424, 2019.
Natalie Dullerud, Karsten Roth, Kimia Hamidieh, Nicolas Papernot, and Marzyeh Ghassemi. Is faimess only metric deep? evaluating and addressing subgroup gaps in deep metric learning. arXiv preprint arXiv:2203, 12748, 2022.
Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. Proceedings of the IEEE International Conference Computer Vision pp. 1657-1664, 2013.
10

---

Published as a conference paper at ICLR 2024
Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Ad- vances Neural Information Processing Systems, 14:1660--672,, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. CoRR, abs/1706.04599, 7a. (tL::ttp///rx
04599
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. PMLR, 06-11 Aug 2017b. URL https: httpas:/pppcceeeinngee mlr ptw.s.p/es//11
guol 7a. html.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recogni- tion (CVPR') 066, volume 2, pp. 1735-1742. IEEE, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Ramya Hebbalaguppe, Jatin Prakash, Neelabh Madan, and Chetan Arora. A stitch in time saves nine: A train-time regularizing loss for improved neural network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16081- 16090, June 2022.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The inaturalist challenge 2017 dataset. CoRR, abs/1707.06642, 2017. URL BtRLhttt////xxv.coo org/ abs/ 06642
Christina Ilvento. Metric learning for individual fairness. arXiv preprint 1xxiv:906.00025:0 2019.
Shichao Kan, Zhiquan He, Yigang Cen, Yang Li, Vladimir Mladenovic, and Zhihai He. Contrastive bayesian analysis for deep metric learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Patlern Recogni- fion (CVPR), June 2020.
Sungyeon Kim, Boseung Jeong, and Suha Kwak. Hier: Metric learning beyond class labels via hierarchical regularization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19903-19912. 2023.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 2013 IEEE International Conference Computer Vision Workshops, pp. 554 561, 2013. doi: 10. 109/CCCVW.201..77.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (((11)):277-22222 1998.
Gongbo Liang, Yu Zhang, Xiaoqin Wang, and Nathan Jacobs. Improved trainable calibration method for neural networks medical imaging classification. In British Machine Vision Conference (BMVC), 2020.
Hao Liu, Xiangyu Zhu, Zhen Lei, and Stan Z. Li. Adaptiveface: Adaptive margin and sampling for face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Jiaheng Liu, Zhipeng Yu, Haoyu Qin, Yichao Wu, Ding Liang, Gangming Zhao, and Ke Xu. One- face: one threshold for all. In Computer Vision-ECC 2022: 17th European Conference, Tel Aviy, Israel October 23-27, 2022, Proceedings, Part XII, pp. 545-561. Springer, 2022.
11

---

Published as a conference paper at ICLR 2024
Yair Movshovitz-Atiia, Alexander Toshev, Thomas K. Leung. Sergey loffe, and Saurabh Singh. No fuss distance metric learning using proxies. CoRR, abs/1703.07464, 2017. URL http:
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Doka- nia. Calibrating deep neural networks using focal loss. Advances in Neural Information Process- ing Systems, 33:15288-15299 2020.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Ad- vances neural information processing systems, 32, 2019.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-Nimth AAAI Conference on Artificial Intelligence, 2015.
Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measur- ing calibration in deep learning. In CVPR Workshops, volume 2, 2019.
Hyun Oh Song. Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4012-401, 2016.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arX 2023.
Yash Patel, Giorgos Tolias, and Jiri Matas, x loss with large batches and similarity mixup. CVPR, 2022. URL
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. In 5th International Conference on Learning Representations (ICLR 2017), 2017.
John Platt al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
Qi Qian, Lei Shang, Baigui Sun, Juhua Hu, Hao Li, and Rong Jin. Sofuriile loss: Deep metric learning without triplet sampling. CoRR, (ed//199.05235,. 2019.
abs/1909 05235
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
Elias Ramzi, Nicolas THOME, Clément Rambour, Nicolas Audebert, and Xavier Bitot. Robust and decomposable average precision for image retrieval. In Thirty-Fffth Conference on Neu- ral Information Processing Systems, 2021. / net/ for
V #1QW3v3FFpJ
Joshua Robinson, Ching-Yas Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592. 2020.
Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive cover- age. Advances in Neural Information Processing Systems, 33:3581-3591, 2020.
Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Bjorn Ommer, and Joseph Paul Co- hen. Revisiting training strategies and generalization performance in deep metric learning. In International Conference on Machine Learning, pp. 8242-8252, PMLR, 2020.
Karsten Roth, Oriol Vinyals, and Zeynep Akata. Non-isotropy regularization for proxy-based deep metric learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7420-7430, June 2022.
12

---

Published as a conference paper at ICLR 2024
Yutaka Sasaki al. The truth of the f-measure. Teach nutor mater, 1(5):1-5, 2007
Walter Scheirer, Anderson Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. IEEE transactions pattern analysis and machine intelligence, 35(7):
1757-1772, 2012.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015a.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 815-823, 2015b.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. CoRR, abs/1511. 1,06452, 2015. URL http://arxiv.org
abs/1511.06, .6452
Eu Wern Teh, Terrance DeVries, and Graham W. Taylor. Proxynca++: Revisiting and revital- izing proxy neighborhood component analysis. CoRR, abs/2004.01113,, 2020. URL https:
arxiv. wwx.v.org/abs///oooooooooo 22w 01113
Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal pre- diction under covariate shift. Advances in neural information processing systems, 32, 2019.
Aäron van den Oord, Yazhe Li. and Oriol Vinyals. Representation learning with contrastive predic- 037488
tive coding. CoRR, abs/1807.00 01211187.0333 807.03748, 2018.
Andreas Veit and Kimberly Wilber. Improving calibration in deep metric learning with cross- example softmax. arXiv preprint arXiv:2011.08824 2020.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech- uesd birds-200-2011 dataset. 2011.
Hao Wang, Yitong Wang, Zheng Zhou. Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5265-5274, 2018.
Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott. Multi-similarity loss the IEEE/CVE conference
with general pair weighting for deep metric learning. In Proceedings the on computer vision and pattern recognition, pp. 5022-5030, 2019.
Ross Wightman. Pytorch image models.
httppss/ggtthbb..om//riggg.....
waw.:chh-immaammmmdeee 2019.
Chao-Yual Yuan Wu, Manmatha, Alexander J. Smola, and Philipp Krähenbühl. Sampling matters in deep embedding learning. CoRR, abs/ (bs/1706.07567, 2017. URL httt::///xxxi/// abs/
1706. 07567
Zhenlin Xu, Yi Zhu, Tiffany Deng, Abhay Mittal, Yanbei Chen, Manchen Wang, Paolo Favaro, Joseph Tighe, and Davide Modolo. Challenges zero-shot recognition with vision-language models: Granularity and correctness. arXiv preprint arXiv:2306. 2023.
Hong Xuan, Abby Stylianou, Xiaotong Liu, and Robert Pless. Hard negative examples are hard, but useful. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pp. 142. Springer, 2020.
Nikolaos-Antonios Ypsilantis, Kaifeng Chen, Bingyi Cao, Mário Lipovský, Pelin Dogan- Schönberger, Grzegorz Makosa, Boris Bluntschli, Mojtaba Seyedhosseini, Ondřej Chum, and André Araujo. Towards universal image embeddings: A large-scale dataset and challenge for generic image representations. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11290-1130) 2023.
13

---

Published as a conference paper at ICLR 2024
Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. Advances in Neural Information Processing Systems, 33:9422-9444 9434,, 2020.
Bianca Zadrozny and Charles Peter Elkan. Transforming classifier scores into accurate multiclass probability estimates. Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, 2002.
Kai Zhao, Jingyi Xu, and Ming-Ming Cheng. Regularface: Deep face recognition via exclusive regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1136-1144, 2019.
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
In the appendix, offer in-depth theoretical analyses of proposed OPIS metric (refer to Ap- pendix A. conduct additional ablation studies for the TCM regularization (see Appendix [A.)] and provide more implementation details (see Appendix A.3
A.1 CLLAATING-PPINT-INNCONNII
SCORE
In this section, provide theoretical analyses for the utility score and the OPIS metric, grounded in Gaussian assumptions. Leveraging these assumptions, we establish upper and lower bounds for OPIS benchmark values.
A.1.1 UTILITY SCORE ANALYSIS: A GAUSSIAN MODEL
For specific class, we assume that its L2 distance distributions between the embeddings of positive sample pairs and between the embeddings negative sample pairs follow Gaussian distribution. We represent the L2 distance distribution for positive pairs d N(ppos 0 pos) and that for negative pairs as ([Jnngg Nneg), with the constraint 0 < Apos < Pneg < 2 for hyperspherical embeddings. Given distance threshold d, can estimate the fractions of true positive (TP), false negative (FN), false positive (FP) and true negative (TN) as follows:
Predicted
Positive
Negative
Actual
Positive
TP [eorf((poo)
FN 1- pro
Negative
FP
[+ff((nncu)
TN crfí
2
Here, erf(t) represents the Gaussian error function Andrews 1998), and Epos d- and tneg /2r
d-pneg Thus we can express the accuracy metrics for precision, recall, specificity, and sensitivity
Vienco
as follows:
+ pro(()pos)
recall 1 [1 + erf(tpos))
precision
+ crf(tpos) + erl(tneg)
specificity 2 (1--e((tneg)). sensitivity
(1 eer((tpos))
Plugging these into the utility score Uroe defined by the specificity and sensitivity pair yields:
(roc(d) (d) (1 + erf(tpoz)) (1 erf((neg))
(9)
erf(t erf((neg)
Similarly, the utility score defined based on the precision and recall pair, denoted as Upr (equivalent to F1 score), can be expressed the following:
2. 21 + erf(()pas)
(d) = 4 + +er((tpos) + erf(tneg)
(10)
Figure 7 gives typical utility score curves. These curves are derived from representative values (indicated in the subtitles) for the mean and standard deviation of the intra-class and inter-class L2 distance distributions of embeddings generated using the Gaussian distribution model. Notably, the left part of the Upr curve and the Uroc curve are nearly indistinguishable. This similarity arises because erf(tneg) + -1 when fneg < -2, leading to (d)(d) s Uroe(d) s 3+erf((poo]] when d 1.0. Meanwhile, both Uroc and Upr utility score curves exhibit concave shape, indicating maximum utility attainable at an optimal distance threshold.
2
15

---

Published as a conference paper at ICLR 2024
4
pos
5 J.Hneg =
2" 200
Oneg
Poos
g.nneg
3'pps = Oneg
1.0
Upr = Fiscore
Upr F1 score
0.8
Uroc
U/oc
Utility score
Utility score
0.6
0.6
0.4
0.2
0.2
0.0
1.0
2.0
1.0
L2 distance threshold
L2 distance threshold
Figure 7: Utility score curves Up (defined as the harmonic mean of precision and recall) and Uroc (defined as the harmonic mean of specificity and sensitivity) plotted as functions of the L2 distance thresholds. The 1.2 distance distributions of positive and negative embedding pairs from Gaussian distribution models whose means (11) and standard deviations (o) are shown in the subtitles. In general. both Ure and UPr exhibit concavity, indicating maximum utility is achieved at an optimal distance threshold.
derived
A.1.2 LOWER AND UPPER BOUNDS OF OPIS
Lower bound The OPIS threshold inconsistency metric measures the variance in the utility curves across different test classes. Consequently, its theoretical lower bound is zero, a value achieved when utility curves for all test classes perfectly aligned within the calibration range. However, it's essential to recognize that achieving a zero-variance world datasets. From our evaluations on public image retrieval datasets, the lower bound of OPIS for prevailing DML losses coupled with VIT backbones is around 1e-5.
an idealistic scenario, unrealistic for real-
Absolute upper bound The upper bound OPIS depends on the degree of dispersion present the utility score curves, making it largely dataset-specific. Here, we provide an absolute upper bound. Let's consider an extreme case where fraction (= a) of the test classes are perfectly accurate within the calibration range, i.e., U Uoorver
1, d [dmin: dmax), while the remainder are entirely inaccurate, i.e., U Unrong (d) = 0, d E [dmin dinax). This leads to an average utility of ((d) a and an overall OPIS score of 2n(1 ax). This OPIS score has an absolute upper bound of 0.5, which is achieved at a = 50%.
Realistic upper bound Here we consider a more practical, realistic upper bound for OPIS. As- suming a fraction (= a) of the classes have perfect utility score in the calibration range, i.e., U Ucorrect()) (d) 1, d [duin dmax), and the rest perfectly "random". meaning their pairwise L2 distances are randomly drawn from uniform distribution with a probability of p(d = x)) = 2, 0 =
2. Given these assumptions, the sensitivity and specificity for random class are described as:
sensitivity(d) d
d
specificity(d)
(11)
2
Incorporating these assumptions into the utility score definition, the utility curves for a "random" class can be expressed follows
Uroc(d) d(1
(12)
Thus, the average utility score, represented by Uroc can be written as the following:
Cro (d) = =(1 a)(1 ア +a
(13)
'We focus Ur),(d), the definition used in the main paper, establish the realistic upper bound of OPIS.
16

---

Published as a conference paper at ICLR 2024
We can then determine the upper bound of OPIS using the equation below:
(1 - a) Sdmas [|U(d) - d(1 2)1 cd
a -dmm ||1 - U(d)ll dd
Jdmax
OPIS
dmax
dmin
dmax
duni
Sdma (d - (d=1)22d
(14)
a((1 - -a) (1+
dmax dmin
a(1 - a) ((dmax)² + (dmin)² dmax dmin)
3
This realistic upper bound a(l=a) ((dmax)2 + (dmin)² dmax dmin) is a function of and the calibration range. To illustrate, consider scenario where 10% and [dmins dmaxl [0.8, 1.2]. this case, the computed OPIS upper bound stands 12e-2.
A.1.3 SENSITIVITY OF OPIS TO CALIBRATION RANGES
In Figure we show the variation of the OPIS metric across different FAR ranges cach repre- senting distinct calibration range. Notably, while the absolute values of the OPIS metric might fluctuate across different calibration ranges, the relative rank orderings mostly stay consistent. The only deviation observed is a rank flip between positions and 6 when transitioning from cal- ibration range of 0.001 < FAR < 0.05 to 0.01 FAR < 0.1; meanwhile, ranks 1, 2, 3 and persistently remain stable. It's worth noting that at high FAR, the magnitude OPIS decreases and variances in relative values will naturally increase when absolute metric values get small. This behavior attributed to the general tendency for relative ranks to become more unstable as absolute metric values decrease phenomenon observed across numerous evaluation metries including call & as shown in [Bronn et al.) (2020); Patel et al. 2022); Kim et al. (2020); Movshovitz-Attias al. (2017; Teh al. (2020). Given these observations, we consider OPIS be largely unaffected by the calibration range.
3.5E-03
Rank: 6
3.0E-03
5
2.5E-03
ProxyNCA
Smooth-AP
2.0E-03
OPIS
CENM
1.SE-03
RS@K
1.0E-03
ProxyAnchor
RS@K+TCM (ours)
3.0E-04
0.0E+00
1o-3FAR<<lo-1
1e-3<FAR-5e-2
1o-2<FAR<le-1
Figure 8: Sensitivity of OPIS the calibration range: Evaluation of ResNe150512 models trained using different losses on the iNaturalist-0015 dataset with a batch size of 384. Different methods are ranked in ascending order according to their OPIS values, with the method exhibiting the lowest OPIS (indicating the best threshold consistency) ranked first.
17

---

Published as a conference paper at ICLR 2024
A.2 ADDITIONAL ABLATION STUDIES FOR TCM
In this section, we present comprehensive ablation studies, including sensitivity analyses, compari- son between TCM and other losses, TCM design variations, and investigation into the compati- bility of TCM with self-supervised pretraining. We also give results for the CUB and Cars datasets an upen-world setting.
A.2.1 SENSITIVITY OF TCM TO RANDOM SEEDS
Table gives the @ and OPIS evaluation results for ResNet50 models trained using the Smooth-AP base loss with the TCM regularization for different random seeds. As demonstrated, the improvements in aceuracy and calibration consistency due to the TCM regularization remain stable across varied random seeds, exhibiting a small variation of 0.1 for recall @K and 4e-6 for OPIS.
Table 6: Effect of different random seeds on @ecall@k and OPIS for models trained on the iNaturalist-2011 2018 dataset, both with and without the TCM regularization, using Smooth-AP (SAP) as the base loss. SAP* represents results taken from the original paper (Brown et al.) 2020). while SAP+TCM corresponds to the results we reported in Table of our main paper.
Method
Recall@1 1
@4
@16
@32
OPIS x
SAP*
67.2
81.8
90.3
93.1
SAP
67.4
81.9
90.4
93.1
0,330
1AP+TCM1
69.1
82.9
91.1
93.8
0.168
SAP+TCM
69.0
82.9
91.0
93.7
0.172
SAP+TCM
69.2
83.0
91.2
93.9
0.175
Average
69.1+ 0.1
82.9+0.11
91.1 +0.1
93.8+0.1
0.172+00.04
A.2.2 COMPARISON WITH OTHER REGULARIZATIONS
In T we demonstrate the superiority of our proposed TCM loss over other DML losses, includ- ing the ArcFace loss Deng et al. 2019), the Triplet loss (Schroff et al. 2015a). and the contrastive loss (Hadsell et al., 2006)), when used as regularizer. It's worth noting that these losses can function as stand-alone DMI. losses, whereas TCM is specifically designed the table, although adding contrastive loss as the regularizer leads to the best threshold consistency, it also causes some degradation in recall @1 1. In contrast, our TCM regularization concurrently en- hances both accuracy and threshold consistency, resulting in a 1.7% boost in recall and relative OPIS improvement of 48%.
regularizer. As indicated in
Table 7: Comparison of our proposed TCM loss with other DML losses (Deng et al. 2019; Schroff ct al. 2015a; Hadsell et al. 2006) when employed as a regularizer on the iNaturalist- 2018 dataset.
Method
Archdom
Batch size
R@1
@4
OPIS xlea
SAP
ResNet50
384
67.4
81.9
90.4
0.330
SAP+ArcFase
ResNet50012
384
62.3 s.1
77.9 4.0.0
87.6 2.8
0.36 0.000
SAP+Trriph Trripeet
ResNet50012
384
66.2 1.2
81.3 u.6.
90.1
0.35
JAP+Contrassie
ResNet50512
384
67.1 o..
82.1
90.6
0.2
0.10 1.2.3
SAP+TCM (ours)
ResNet50
384
69.1 1177
82.9 1..0
91.1 0.7
0.17 0.30
We also compare our TCM loss with the calibration loss introduced in ROADMAP (Ramzi et al. 2021], which aims to address the decomposability gap for average precision during training. Their approach, akin to ours, involves the imposition of constraints to regulate pairwise similarity scores. However, a notable difference lies the denominator's composition: they use the total counts of positive and negative pairs, whereas we employ hard positive and negative counts. An apparent limitation of their design is that in scenarios with a large negative-to -otoiiiiee ratio, the importance of the negative term diminishes. Such scenarios often arise with large batch sizes (which is the standard of DML), such as the case PatPa et al. (2022) where a batch size of 4000 employed. This limitation can have adverse effect on the model performance as body of research (Schroff al.
18

---

Published as a conference paper at ICLR 2024
2013b Oh Song al., 2016 Wu et al. 2017 Wang et al. 2019) has consistently demonstrated that
hard negative pairs generally convey more information than hard positive pairs. As shown in Table when comparing RS@K K+TCM and RS @ K+RAADMAP at batch size of 4000, using ROADMAP as regularization results in a significant reduction in accuracy, aligning with our earlier analysis.
Table 8: Comparison of our proposed TCM loss with ROADMAP at a large batch size.
Method
Batch size
R@1
@4
@ 16
OPIS X
RS@K
ViT-B/16-12
4000
83.9
92.1
96.1
0.37
RS@ AORADMAP
ViT-B/16"12
4000
79.4
89.2
2.9
94.3
0.24 o.18
RS@K+TCM (ours)
ViT-B/16*12
4000
84.8 1.9.
92.7 1.4.
96.5 0.4
0.17 y 0.200
A.2.3 COMPARISON OF TCM AND ITS DESIGN VARIANTS
We examine the impact of various components within TCM, namely the indicator function, the positive term, and the negative term, to provide deeper insights into the design of the TCM regu- larization. Specifically, we consider three design variations, each omitting just one of its three core components: the positive term, the negative term, and the indicator functions. These TCM design alternatives are described as follows:
LTCM
Σ (s m m-) 1.2m SES-
(15)
1
LTCM+
(m+ 5) 1 1.5m+
(16)
15m+
SESt
LTCM
Σ (m
8)
(s m
(17)
S
5
SES+
SES
where I-CM+ denotes the variant using only the positive term, LTCM- represents the variant with just the negative term, and LTCM' stands for the variant without the indicator functions.
Table 9: Performance of supervised image retrieval on the -Ntuuralist-2018 dataset using recall@k surrogate loss with TCM regularization and its alternative designs at a batch size of 4000. The results highlight the efficacy of the TCM regularization in comparison to its design alternatives in terms of retaining the accuracy performance while improving OPIS.
Method
dim
Batch size
R@1
@4
@16
OPIS xle3
RS@K
VIT-B/16612
4000
83.9
92.1
96.1
0.37
RS@K+TCMM
ViT-B/16512
4000
79.0 .4.
88.9 432
94.2 119
0.06 Jn31
RS@ K+TCM
VIT-B/16512
4000
72.5 4114
86.0 (&i
93.0 j3l
0.14 1023
RS@ K+TCM'
VIT-B/16512
4000
78.1 1 1ss
88.4 13.7
93.7 124
0.31 10os
RS@ K+TCM (ours)
VIT- 5/161/
4000
84.8 na.9
92.7 n0.6
96.5 10.4
0.17 10.20
In Figure we give the progression of accuracy across training epochs. The accuracy and OPIS corresponding to the epoch with the peak accuracy are detailed in Table 9 In general. excluding any of the three components of TCM leads to a marked decrease in recall @ of up to 12%. Moreover, in scenarios where only the positive term is employed or when TCM operates without the indicator function, we observe that the model stops improving its accuracy after the first epoch. When relying solely TCM's ss negative term, we observe an initial dip in recall @ 1, which shows signs of recovery after 20 epochs. Nevertheless, the eventual recall 1, even post-recovery, lags significantly behind that achieved with the full TCM regularization with all three components included. In terms of threshold consistency, our observations suggest that penalizing the TCM positive term, the negative all improve the OPIS metric. Notably, regularizing solely with the positive TCM term (LTCM+) ) yields the best OPIS result. However, omitting the indicator function results in minimal improvements in OPIS, aligning closely with the baseline outcome observed
term, or both simultaneously
19

---

Published as a conference paper at ICLR 2024
RS@K
FTM negative term only
Recali@i
RS@K TTCM posilive only
RS@K-TCM indicator function
RS@K-TCM
10 20
30
40
50
60
70
80
90
Epoch
Figure 9: Evolution of recall @1 across training epochs for ViT-B/16 backbones trained on the iNaturalist-2018 dataset at batch size of 4000. The Recall@k surrogate loss is used as the base loss along with the TCM regularization and its design alternatives.
TCM proposed
only
Intre-cless, RS@K+reg intenclaass, R5@K+req
Densty
5.
Dencity
2-
2-
L2 distance
L. distance
TCM indicator function
TCM negative term only
2.5
Oensty
Denoity
1.0
2-
0.5
0.0
0.5
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Figure 10: Distributions of L.2 distances for intra-class and inter-class test embeddings of ViT- B/16 models trained on the iNtuuralist20018 dataset using recall@k surrogate loss with TCM regularization and its variants. It shows that excluding any of the TCM loss components introduces irregularities in the metric space, therefore diminishing accuracy.
without any regularization. This showcases the importance of the indicator function as an effective quantile estimation mechanism filter out the hard sample pairs.
We also visualize the intra-class and inter-class L2 distance distributions in Figure 10 for models trained with TCM regularization and the other four alternative designs. The results reveal that regularizer focusing solely on penalizing hard negative pairs (RS@) K+TCM causes the inter-class embeddings to become overly dispersed, compromising the desired intra-class compactness. Con-
20

---

Published as a conference paper at ICLR 2024
it results in the collapse of versely, when only hard positive pairs are penalized unchecked. On the other hand, in the ab- intra-class embeddings, leaving the inter-class distribution sence of the indicator function, the inter-class distribution manifests multi-modes with dual peaks with significantly increased overlap between the inter-class and intra-class 1.2 distance distributions. This phenomenon may be attributed to the failure distinguish hard negative pairs from the easy ones. To conclude, to maintain accuracy on par with the baseline (without regularization), all three components of TCM are indispensable.
A.2.4 COMPATIBILITY OF TCM WITH SEL.FSUUPERVISED PRETRAINING
We examine the compatibility of our proposed TCM regularization with self- pretraining methods, including CLIP (Radford et at. 202T). DINOv2 (Oquab et al. 2023), and Unicom (An)
et al. 2) In alignment with the approach in Unicom An al. (2023) adopt the ArcFace
loss Deng et al. 2019) for this supervised fine-tuning. The results are presented in Table IO
Table 10: Performance of supervised image retrieval on the Stanford Online Product dataset with different pretraining methods including CLIP (Radford et al., 202T), DINOv2 (Oquab et al. 2023) and Unicom (An et al. 2023). The pretraining dataset for each backbone model is specified in the text next each method. Following Unicom (An al. 2023), we use the ArcFace loss (Deng et al.) 2019) for supervised fine-tuning. Same as Table 5 in the the main paper, the numbers displayed in black represent models trained with Lbuse + LTCM. while the colored numbers indicate the im- provement/ degradation in absolute magnitude over models trained with the base loss alone.
Pretraining
Pretraining data
Archatm
R@1
OPIS x10-
CLIP
Private400M
VTT-B/16212
88.4 14.50 (+5.39)
0.94 10.06 (-6.2%)
DINOv2
LVD- 142M
ViT-B/14
84.2
10.20 (10.2%)
0.84 10.09 ( 9.7%%)
Unicom
LAION-400M
VIT-B/16 768
89.1 10.30 (+0.3%)
0.82 (0.00-20.99)
As shown in the table, incorporating TCM regularization in models initialized through various self- supervised pretraining methods consistently delivers improvements in the OPIS metries, observ- ing gains of up to 10%. However, the relative improvement in OPIS doesn't quite reach the lev- els achieved with supervised pretraining as reported in the main paper. This discrepancy is likely caused by the distinctive learning objectives of self-supervised pretraining and its use of completely different pretraining datasets compared to ImageNet. As for accuracy, adding TCM is shown to significantly boost recall @1 I by up to 4.5% when paired with CLIP pretraining. Yet, for approaches such as DINOv2 and Unicom, the enhancement in recall@ I marginal.
A.2.5 OPEN-WORLD SETTING RESULTS FOR CUB AND CARS
In the main paper, we utilize shared train-test classes in the closed-set setting for the CUB and Cars datasets to ensure fair comparisons with previous works. However, in this section, we present results for these two datasets under the open-world setting, where training and testing classes are disjoint, to align with our central focus on open- world scenarios. Specifically, we partition the datasets based on the class indices: for CUB, classes 1-100 serve as the training set while the rest constitute the test set; similarly, for Cars, classes 1-98 are used for training, with the remaining classes reserved for testing. In [T] we present evaluation results for the CUB and Cars datasets under the open- world setting. The results demonstrate that integrating TCM leads to notable relative improvement of 18% in OPIS for the Cars- 196 dataset and a substantial 45% enhancement for the CUB dataset, while preserving and improving the accuracy compared to the baseline without regularization.
Table 11: Evaluation results of incorporating TCM regularization on @ and OPIS for the CUB and Cars-196 196 datasets in the open-world setting.
Dataset
Arch dem
R@11 f
OPIS x1o-*|
10%-OPIS x
CUB
ViT-B/16'12
86.5 10n
0.92 1o2s (54.229)
10.43 (-40.4%)
Cars-196
VIT-B/16212
90.2 pa7
0.63 (-1898))
8.07 1233 (-22.45)
21

---

Published as a conference paper at ICLR 2024
A.3
IMPLEMENTATION DETAILS
A.3.1 CONNECTIONS BETWEEN CALIBRATION RANGE AND COSINE MARGINS
We clarify the connection between the calibration range and the cosine margins introduced in TCM. First, calibration range is determined by the FAR and FRR requirement specified by the use- case. For well-trained embedding model, optimal performance is typically achieved within an intermediate range of distance thresholds. Thus, the left bound of the calibration range is determined by the maximum FRR, as defined by the use-case. On the flip side, the right bound is determined by the maximum FPR set by the same use-case. The margins m and m of the TCM regularization are defined in the cosine space. Given hyperspherical embeddings, there a bijection between the cosine similarity and the L2 distance. In the absence of distribution shift between the training
and testing datasets, the cosine margins can adeptly pinpointed as m+ 2--dinn" and m
22-dmax to align the representation structures the desired calibration range. However, while the calibration range can serve as a useful reference, due to the inevitable distribution shifts between training and testing data, the selection of the margin constraints requires hyper-parameter tuning and cannot be directly estimated from the calibration range.
A.3.2 ELABORATION ON FBG.5(B) OF THE MAIN PAPER
In Fib. of the main paper, 0, and 02 represent the angular are lengths of the red and blue classes, respectively. Applying TCM regularization promotes intra-class compactness, targeting a condition where c0s m with representing the angular span of the given class in the embedding. Thus training with TCM regularization would encourage 01 <- arccos(m+t) and 02 <- arccos(m:)
A.3.3 IMPLEMENTATION DETAILS FOR OPIS METRIC AND TCM REGULARIZATION
In the following, we provide brief description of our implementation for both the OPIS metric and the TCM regularization in pseudo-code format.
Algorithm Computation for OPIS metric
Require: Pairwise evaluation protocol for test embeddings with a total of T classes, calibration range [dmin dmax] with a grid number N.
Initialize OPIS
2: for j to N
D Iterate over calibration grid
3: d 4- dmin + dinin .j
4: for i in T do
b. Iterate test classes
5:
Gather all L2 distances for class i Compute specificity (Ø) and sensitivity (t)) at d Uz(d) + 2-0(d)-9(())
6:
7:
((d)+a(2]
8: Compute average utility score ((d) across all classes
9: OPIS < OPIS + -|U O(4)
T-fikmux Amin)
return OPIS
In Algorithm 1, the number of grids in the calibration range, denoted N, should be selected in accordance with the maximum sharpness observed in the utility-distance curves within the calibration range. It is recommended to maintain N > 10. The calibration range itself is determined by the desired False Acceptance Rate (FAR) range of the entire dataset, as specified by the user case. Additionally, it is bound by the least achievable FAR of the dataset. In Algorithm 2, |Sm**) represents the number of hard positive pairs in S with similarity below m and |Smh represents the number of hard negative pairs in S with similarity above m.
22

---

Published as a conference paper at ICLR 2024
Algorithm 2 Training with TCM regularization
Require: m m_ A AT, base loss Lbase.
for number of epochs do
2: for steps do
In each training iteration
3: LrcM 4-
4: Sample mini- mani-batch data (311 U1), (Irms Mmm)
5:
Extract embeddings (denoted by f) for each æ Compute cosine similarities for the entire mini-batch data. Let S be the set of positive pair similarities and 5 be the set of negative pair similarities Compute |ST* and 1s
6:
7:
8:
for in S do LTOM LTCM Xt. -s)- 1m5m!
9:
Sum all hard positive pairs
[5mh
10:
for in S do
11:
LTOM < LICM x (5-m mm
Sum over all hard negative pairs
18th
12:
Lowerall <- Lhman + LTCM
13:
Update model using Loverall
A.3.4
DETAILS ON THE BASE DML LOSSES EMPLOYED
Smooth-AP Smooth-AP (Brown et al. 2020 is a pair-based DML loss that optimizes a smoothed approximation of average precision (AP) using sigmoid function. It is expressed
1 1+1 EVES! Gshn
1
LSAP
(18)
1+ ExeS, Gjn.s
1=
JEST
Here, 9Jn.;= = (1 + rank similarity scores of mini-batch samples against anchor sample 2y. represents the set of cosine similarity scores for the entire mini-batch against Iú, while S7 denotes the subset S; for the positive samples against 24.
is a softmax function incorporating a temperature parameter, X,
@ Surrogate The Recall@k Surrogate loss mirrors Smooth-AP's approach but approxi- mates the Recall @k metric rather than the average precision metric. Its detailed formulation is available in Patel et al. 20222).
AreFace ArcFace (Deng et al. 2019) introduces additive margin penalties based on the angle, 0, between mini-batch samples and prototype representations, aims to contrast samples against class
prototypes, and is formulated as:
LAveFace
log
Σ
Aaas-a t.t
(19)
72
j=1jfi
where Siy is the cosine similarity with inter-class margin penalty between the representation of an image with class and the prototypical representation for another image whose class j. with mi, m2 denoting different angular margins:
[cos(m:00) O.j + m2) for i
$ij =
cos Coo
for i +
(20)
23


Reviews:
[{"review_id": "W90516ZzLZ", "reviewer": "Reviewer_4pGE", "summary": "This paper addresses the problem of threshold inconsistency in deep metric learning (DML) for image retrieval. Existing DML methods often result in uneven representation structures within and between classes, leading to significant variations in performance across different test classes and data distributions, measured by false accept rate (FAR) and false reject rate (FRR). To tackle this issue, the authors propose a novel variance-based metric called Operating-Point-Inconsistency-Score (OPIS) to quantify the inconsistency in threshold performance across classes. They observe a trade-off between accuracy and threshold consistency, where improving accuracy can negatively impact threshold consistency. To mitigate this trade-off, they introduce the Threshold-Consistent Margin (TCM) loss, a simple yet effective regularization technique that penalizes difficult sample pairs to encourage uniform representation structures across classes. Extensive experiments on large-scale datasets demonstrate that TCM enhances threshold consistency while maintaining or even improving accuracy, simplifying the threshold selection process in practical DML applications. The key contributions of the paper include the introduction of the OPIS metric, the identification of the accuracy-threshold consistency trade-off, and the proposal of the TCM loss as a solution to improve threshold consistency in DML. The approach outperforms state-of-the-art methods on various large-scale image retrieval benchmarks, achieving significant improvements in threshold consistency.", "questions": "1. Since you conducted experiments on the large-scale iNaturalist-2018 dataset, what are the differences between open-set metric learning and face recognition or re-identification (re-ID)? Can your method be applied in the field of face recognition?\n2. If your method can use a single model to maintain the same threshold across multiple test sets, would it be meaningful, such as in this work[1].\n\n[1] https://cmp.felk.cvut.cz/univ_emb/", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed Operating-Point-Inconsistency Score (OPIS) and ϵ-OPIS provide valuable insights.\n2. The experiments comparing high accuracy with high threshold consistency are objective.\n3. The proposed Threshold-Consistent Margin (TCM) loss is relatively simple and easy to understand.\n4. The visualization of the TCM effect is interesting.\n5. The experiments are comprehensive, with detailed implementation and coverage of mainstream metric learning settings.\n6. The ablation experiments are extensive, exploring margin, DML losses, different architectures, and time complexity. They also validate the proposed method against state-of-the-art approaches such as RS.", "weaknesses": "It is meaningful to pull the scores of positive pairs towards a fixed value and the scores of negative pairs towards another fixed value, even though it sounds simple.\n\nApart from that, I did not see any other weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1IYQUqRV5y", "reviewer": "Reviewer_52BZ", "summary": "The paper introduces and addresses the threshold inconsistency problem in Deep Metric Learning (DML). To tackle this issue, the authors present the Operating-Point-Inconsistency-Score (OPIS) metric, which is based on the variance of utility score derived from the F-score. Additionally, they propose the Threshold-Consistent Margin (TCM) loss, which selectively penalizes hard sample pairs. The experimental results on various deep metric learning benchmarks validate the efficacy of their proposed method.", "questions": "See the weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper effectively identifies and defines the threshold inconsistency problem within the context of Deep Metric Learning (DML). \n\n2.To address this issue, the authors introduce a novel loss function, the Threshold-Consistent Margin (TCM) loss. \n\n3.Their proposed method is rigorously evaluated through comprehensive experiments.", "weaknesses": "1. The use of the term \"large-scale\" in this paper may be misleading as the experiment datasets do not contain a sufficiently large number of samples to be accurately characterized as \"large-scale.\" Typically, datasets with more than 10 million or 1 billion samples could be considered as large-scale.\n\n2. The threshold inconsistency problem, as described in this paper, is also referred to as the generalization problem and has been previously discussed in the deep metric learning (DML) literature [r1, r2]. In reference [r1], the authors proposed the adoption of a metric variance constraint (MVC) to enhance generalization ability, which is essentially a variance-based metric. Reference [r2] provided an in-depth discussion of the generalization problem in DML. It would be beneficial for this paper to incorporate discussions and comparisons with these existing works in the context of addressing the threshold inconsistency problem. \n\n[r1] Kan, Shichao, et al. \"Contrastive Bayesian Analysis for Deep Metric Learning.\" IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).\n\n[r2] Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Björn Ommer, and Joseph Paul Cohen. Revisiting training strategies and generalization performance in deep metric learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 8242–8252, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h4K2hfjC3e", "reviewer": "Reviewer_3dmW", "summary": "This paper addresses the issue of inconsistency in threshold determination for negative samples in threshold-based image retrieval. The authors propose a new metric called Operating-Point-Inconsistency-Score (OPIS) to measure inconsistency and introduce the Threshold-Consistent Margin (TCM) loss as a regularization technique to enhance consistency. The key contributions include identifying the problem with existing method, introducing an intuitive evaluation metric and regularization approach, and demonstrating improved threshold consistency without sacrificing accuracy in large-scale experiments.", "questions": "Figure 3 shows ProxyAnchor (ResNet50) with a low threshold consistency. It would be beneficial to compare the improvement in R@K and OPIS when using the proposed method. Ideally, the proposed method should show a significant OPIS improvement compared to others.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written, making it easy to understand while offering comprehensive comparisons with current methods.\n\n- It clearly highlights issues in existing models and presents an intuitive metric and regularization technique to tackle them.\n\n- The research goes a step further by demonstrating not just improved threshold consistency but also better performance in several instances.", "weaknesses": "- The biggest weakness in the paper is the lack of experiments related to face verification, where threshold importance is evident. While image retrieval mostly uses metrics like mAP or Recall@k, face verification relies heavily on thresholds and uses metrics like TAR@FAR. The introduced method appears more suited for face verification than image retrieval.\n\n- The paper suggests that high accuracy doesn't always mean high threshold consistency. However, in face verification tasks, consistency in threshold often translates to high accuracy. This amplifies the sense that the paper might be focusing on an unrelated task.\n\n- The paper mentions related works like Liu et al. (2022) and OneFace, but experiments comparing the proposed method to these in the realm of face recognition are missing. Such comparisons are necessary to understand the proposed method's improvements in threshold consistency.\n\n- The paper needs to update state-of-the-art results on the CUB and Cars-196 datasets [1].\n[1] Kim et al., HIER: Metric Learning Beyond Class Labels via Hierarchical Regularization, CVPR 23", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)