Characterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning
Authors: Timo Milbich
Karsten Roth
Samarth Sinha
Ludwig Schmidt
Marzyeh Ghassemi
Björn Ommer
Venue: NeurIPS 2021 Poster (2021)
Keywords: Deep Metric Learning
Representation Learning
Transfer Learning
Zero-Shot Generalization
Out-of-Distribution Generalization

Abstract: Deep Metric Learning (DML) aims to find representations suitable for zero-shot transfer to a priori unknown test distributions. However, common evaluation protocols only test a single, fixed data split in which train and test classes are assigned randomly. More realistic evaluations should consider a broad spectrum of distribution shifts with potentially varying degree and difficulty.
In this work, we systematically construct train-test splits of increasing difficulty and present the ooDML benchmark to characterize generalization under out-of-distribution shifts in DML. ooDML is designed to probe the generalization performance on much more challenging, diverse train-to-test distribution shifts. Based on our new benchmark, we conduct a thorough empirical analysis of state-of-the-art DML methods. We find that while generalization tends to consistently degrade with difficulty, some methods are better at retaining performance as the distribution shift increases. Finally, we propose few-shot DML as an efficient way to consistently improve generalization in response to unknown test shifts presented in ooDML.

Characterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning
Timo Milbich"
Karsten Roth +2
LMU Munich & IWR, Heidelberg University timo. mi wwiihbiiiiii iwr. uni wei-heidelbergeeeeeeeeeeeneeie-e
IWR, Heidelberg University karsten. rh1@gmaill com
Samarth Sinha
Ludwig Schmidt University of Washington schmidt ww. edu
Marzyeh Ghassemi
University of Toronto, Vector s sinhasame fb. com
MIT, University of Toronto, Vector
mghassememit edu
Björn Ommer
LMU Munich & IWR, Heidelberg University ommer@uni -he deeiidebee berg..d.
Abstract
Deep Metric Learning (DML) aims to find representations suitable for zero-shot transfer to priori unknown test distributions. However, common evaluation protocols only test a single, fixed data split in which train and test classes are assigned randomly. More realistic evaluations should consider broad spectrum distribution shifts with potentially varying degree and difficulty. In this work, we systematically construct train-test splits of increasing difficulty and present the MODML. benchmark to characterize generalization under out-of-distribution shifts in DML. oolMM. is designed to probe the generalization performance much more challenging, diverse train-to-test distribution shifts. Based on our new benchmark, conduct thorough empirical analysis of state-of-the-art DML methods. We find that while generalization tends to consistently degrade with difficulty, some methods better retaining performance as the distribution shift increases. Finally, we propose few-shot DML as an efficient way to consistently improve generalization in response to unknown test shifts presented in ooDM!
Introduction
Image representations that generalize well are the foundation of numerous computer vision tasks,
such image and video retrieval [61 71 54 38, [I] face (re- (re))identiiicttion [57]
34 g and image
classification [65] 19 40 37 Ideally, these representations should not only capture data within the (OOD) data. However, in practice,
training distribution, but also transfer to new, out-of-distribune
achieving effective OOD generalization more challenging than in-distribution (28] [2 21 31 (55. In the case of zero-shot generalization, where train and test classes completely distinct, Deep Metric Learning (DML) is used to learn metric representation spaces that capture and transfer visual similarity to unseen classes, constituting priori unknown test distributions with unspecified shift. To approximate such a setting, current DML benchmarks use single, predefined and fixed data splits of disjoint train and test classes, which assigned arbitrarily [71] 8 61 24 33 74 [1] 54 42
26 64 58). This means that (i) generalization is only evaluated on a fixed problem difficulty, (ii)
CCode available here:
Generalization, in DML Equal contribution, alphabetical order, equal supervision, now University Tuebingen.
35th Conference on Neural Information Processing Systems (NcurIPS 2021), virtual.

---

generalization difficulty is only implicitly defined by the arbitrary data split, (iii) the distribution shift not measured and (iv) cannot be not changed. As a result, proposed models can overfit these singular evaluation settings, which puts into question the true zero-shot generalization capabilities of proposed DML. models.
In this work, we first construct a new benchmark ooDML to characterize generalization under out- of-distribution shifts in DML. We systematically build ooDML as comprehensive benchmark for evaluating OOD generalization in changing zero-shot learning settings which covers much larger variety of zero-shot transfer learning scenarios potentially encountered in practice. We systematically construct training and testing data splits of increasing difficulty as measured by their Frechet-Inception Inception Distance (23] and extensively evaluate the performance of current DML approaches.
Our experiments reveal that the standard evaluation splits are often close to i.i.d. evaluation settings. In contrast, our novel benchmark continually evaluates models on significantly harder learning problems, providing a more complete perspective into OOD generalization in DML.. Second, we perform large-scale study of representative DML methods on ooDML, and study the actual benefit of underlying regularizations such as self-supervision (38). knowledge distillation [53]. adversarial
regularization [59] and specialized objective functions 77 g 26 54]. We find that conceptual
differences between DML approaches play a more significant role as the distribution shift to the test
split becomes harder.
Finally, we present a study on few-shot DML as simple extension to achieve systematic and consistent OOD generalization. As the transfer learning problem becomes harder, even very little in-domain knowledge effectively helps to adjust learned metric representation spaces to novel test distributions. We publish our code and train-test splits on three established benchmark sets, CUB200- 2011 1 (68) CARS1 196 BQ and Stanford Online Products (SOP) 44]. Similarly, we provide training and evaluation episodes for further research into few-shot DML. Overall, our contributions can be summarized as:
Proposing the ooDML benchmark create a set of more realistic train-test splits that evaluate DML generalization capabilities under increasingly more difficult zero-shot learning tasks. Analyzing the current DML method landscape under ooDML to characterize benefits and drawbacks of different conceptual approaches to DMI..
Introducing and examining few-shot DML a potential remedy for systematically improved OOD generalization, especially when moving to larger train-test distribution shifts.
2
Related Work
DML has become essential for many applications, especially in zero-shot image and video retrieval
71] 22 36), Proposed approaches most commonly rely on surrogate ranking task over
tuples during training (62). ranging from simple pairs [17] and triplets (57) higher-order quadruplets
[S] and more generic n-tuples [6] 43 22 77). These ranking tasks can also leverage additional
context such as geometrical embedding structures 69 However, due the exponentially increased complexity of tuple sampling spaces, these methods are usually also combined with tuple sampling objectives, relying on predefined or learned heuristics to avoid training over tuples that are too easy too hard [57 [22 or reducing tuple redudancy encountered during training [71 [5] [8 52
More recent work has tackled sampling complexity through the usage of proxy-representations
utilized as sample stand-ins during training. following a NCA [16] objective [4 26 64] leveraging
softmax-stye -tyle training through class-proxics
[8] [33 or simulating intraclass structures [46].
Unfortunately, the true benefit of these proposed objectives has been put into question recently, with (54] and 42) highlighting high levels of performance saturation of these diseriminative DML objectives on default benchmark splits under fair comparison. Instead, orthogonal work extending
the standard DML training paradigm through multi-tast approaches [56]5] 39]. boosting [44 45
[1
attention (27,, sample generation [1] 33 [74, multi-feature learning (38 or self self-distillatio
53 have
shown more promise with strong relative improvements under fair comparison (54 38 however still
only in single split benchmark settings. It thus remains unclear how well these methods generalize in
more realistic settings (2 under potentially much more challenging, different train-to-test distribution shifts, which we investigate in this work.
2

---

Table 1: FID scores between i.idd subsampled training and test sets in comparison to FID scores measured on default splits used in standard DML evaluation protocols, As can be seen, the train-test distribution shift of two out of three benchmarks are actually close i.i.d. settings, in particular when compared to the train-test splits evaluated in Fig. 1 reaching scores over 200.
Dataset
CUB
CARS
SOP
Default different classes train/test
52.62 4.87 1 0.05
8.59 2.33 1 0.03
3.43 0.98 1 0.01
i.id. same classes train/test
3 ooDML: Constructing Benchmark for OOD Generalization in DML
An image representation ((x) learned on samples E Xaain drawn from some training distribution generalizes well if can transfer to test data Nest that are not observed during training. In the particular case of OOD generalization, the learned representation is supposed to transfer to samples &est which are not independently and identically distributed (i.i..)) to X A successful approach to learning such representations is DML, which is evaluated for the special case of zero-shot generalization, i.e. the transfer of ó to distributions of unknown classes [57] 71 24 & 54, 42). DML models aim to learn embedding ó mapping datapoints r into embedding space D which allows to measure similarity between I; and æj as g(((z;); ((z))) Typically, y is a predefined metric, such as the Euclidean Cosine distance and Φ is parameterized by deep neural network.
In realistic zero-shot learning scenarios, test distributions are not specified a priori. Thus, their respec- tive distribution shifts relative to the training, which indicates the difficulty of the transfer learning problem, is unknown as well. To determine the generalization capabilities of o, we would ideally measure its performance on different test distributions covering large spectrum of distribution shifts, which we will also refer to as "problem difficulties" in this work. Unfortunately. standard evaluation protocols test the generalization of 6 single and fixed train-test data split of predetermined difficulty, hence only allow for limited conclusions about zero-shot generalization.
To thoroughly assess and compare zero-shot generalization of DML models, we aim to build an evaluation protocol that resembles the undetermined nature of the transfer learning problem. In order to achieve this, we need to be able to change, measure and control the difficulty of train-test data splits. To this end, we present an approach to construct multiple train-test splits of measurably increasing difficulty to investigate out-of-distributio generalization in DML, which make up the ooDML benchmark. Our generated train-test splits resort to the established DML benchmark sets, and are subsequently used in Sec. fo to thoroughly analyze the current state-of-the-art in DML. For future research, this approach is also easily applicable to other datasets and transfer learning problems.
3.1 Mcasuring the gap between train and test distributions
To create our train-test data splits, we need a way of measuring the distance between image datasets. This is difficult task due to high dimensionality and natural noise in the images. Recently, Frechet Inception Distance (FID) 23 was proposed to measure the distance between two image distributions by using the neural embeddings of an Inception-v3 network trained for classification on the ImageNet dataset. FID assumes that the embeddings of the penultimate layer follow Gaussian distribution, with given mean ya and covariance Ex for a distribution of images X. The FID between two data distributions X₁ and X₂ is defined as:
(I) In this paper, instead of the Inception network, use the embeddings of a ResNet-50 classifier (Frechet ResNer Distance) for consistency with most DML studies (see e.g. [7I] 64 26 [6
X2) A ||xx, Mxx 1x.1 1 TT(Ex, +xx 2(xx, )x,)+)
51
38
54 58)
For simplicity, in the following sections will still use the abbreviation FID.
3.2
On the issue with default train-test splits in DML
To motivate the need for more comprehensive OOD evaluation protocols, we look at the split difficulty as measured by FID of typically used train-test splits and compare to i.i.d. sampling of training and test sets from the same benchmark. Empirical results in Tab. show that commonly utilized DML
3

---

CUB20D-2013
CARS196
(SOP)
RID Erogiisioo
Syli
130
60-
Teainy Ttee RID
100
Oconsse HIL
[iceaae RID
-0-
20 Recations
0-
20
0-
www.0000
20
lterations
Derations
Cless Recuoved Spilts
Figure 1: FID progression with iterative class swapping and removal for train-test split generation. (Col. 1-3) FID per swapping iteration on all benchmarks. (Rightmost) FID of data splits obtained by additional k iterations of removing classes. The blue bar denotes the maximal FID after swapping.
train-test splits are very close to in-distribution learning problems when compared to more out-of- distribution splits in CARSI96 and SOP (see Fig. [ This indicates that semantic differences due to disjoint train and test classes, do not necessarily relate to actual significant distribution shifts between the train and test set. This also explains the consistently lower zero-shot retrieval performance on CUB200-201 as compared to both CARS196 and SOP in literature [7T. 70
38], despite
SOP containing significantly more classes with fewer examples per class. In addition to the previously discussed issues DML evaluation protocols, this further questions conclusions drawn from these protocols about the OOD generalization of representations ø.
3.3 Creating train-test splits of increasing difficulty
Let Xaain and Aes denote the original train and test set of a given benchmark dataset D X Nest. To generate train-test splits of increasing difficulty while retaining the available data D and maintaining balance of their sizes, we exchange samples between them. To ensure and emphasize semantic consistency and unbiased data distributions with respect to image context unrelated to the target object categories, we swap entire classes instead of individual samples. Measuring distribution similarity based on FID, the goal is then to identify classes Cirain C Aurain and Cies C Nies whose exchange yields higher FID di Miain M.s)). To this end, similar to other works [33]5] 38), we find resorting to an unimodal approximation of the intraclass distributions sufficient and approximate FID by only considering the class means and neglect the covariance in Eq. [
We select Crnin and Clest as
C train
arg max ||
M.Xis 1l2
||PCi. P.
(2)
Cruin E Xxii
C arg max ||NA Me ll2
||
M. l12
Cua Xup
where we measure distance to mean class class-represenntations Pxa. By iteratively exchanging classes between data splits, i.c. x = (Kt \ C(imi)) U Cist and vice versa, we obtain a more difficult train-test split (2++1) xt+1) ) at iteration step t. Hence, we obtain a sequence of train-test splits Mim)..... (XTT x(r))) )). with 2 و X and 2 &est Fig. (columns 1-3) indeed shows that our FIID approximation yields data splits with gradually increasing approximate FID scores with each swap until the scores cannot be further increased by swapping classes.
X' = ((x train (ita)...,,,..8
UMAP visualizations in the supplementary verify that the increase corresponds to larger OOD shifts. For CUB200-2011 and CARS196 we swap two classes per iteration, while for Stanford Online Products we swap 1000 classes due to a significantly higher class count. Moreover, to cover the overall spectrum of distribution shifts and ensure comparability between benchmarks we also reverse the iteration procedure on CUB200-2011 to generate splits minimizing the approximate FID while still maintaining disjunct train and test classes.
To further increase d( (TT XT) beyond convergence (see Fig. of the swapping procedure, we subsequently also identify and remove classes from both x and X More specifically, we
remove classes Cuain from XT that are closest to the mean of X and vice versa. For & steps, we successively repeat class removal as long 50% of the original data is still maintained in these

---

2011
Cars196
Online Products
Ditaua Thate Tex Spit
XMMggh 200
11775
Manga (S)
Maltorstatty
5250
Unforn Palce
Recat
Axtiiie
DIA
0.50
0.76
0.74
0.70
0.72
0.70
0.50
0075
0.000
Change against moant
0025
0.00
0.00
0.025
-002
-012
-004
0.003
Train//
Trann/
Figure 2: Zero-Shot Generalization performance under varying distribution shifts. (top row) Absolute Recall @1 performance for each increasingly more difficult train-test split in the goodDML benchmark (cf. Sec. B) on CUB200-2011, CARSI and SOP. We report mean Recall performances and standard deviations over 5 runs. For results based on mAP@ 1000 see supplementary. (bottom) Differences performances against the mean over all methods for each train-test split.
additional train-test splits. Fig. [ (rightmost) shows how splits generated through class removal progressively increase the approximate FID beyond what achieved only by swapping. To analyze if the generated data splits are not inherently biased to the used backbone network for FID computation, also repeat this procedure based representations from different architectures, pretraining methods and datasets in the supplementary. Note, that comparison of absolute FID values between datasets may be meaningful and given dataset distribution. Overall, using class swapping and removal we select splits that cover the broadest FID range possible, while still maintaining sufficient data. Hence, our splits are significantly harder and more diverse than the default splits.
mainly interested in distribution shifts within
Assessing the State of Generalization in Deep Metric Learning
This section assesses the state of zero-shot generalization in DML via a large experimental study of ooDML benchmark, offering much more complete and thorough
representative DML methods perspective on zero-shot generalization in DML as compared to previous DML studies [13 54 42 39].
For our experiments we use the three most widely used benchmarks in DML, CUB200-201 [68].
CARS196 30 and Stanford Online Products 45 For complete list of implementation and training details see the supplementary if not explicitly stated in the respective sections. Moreover, to measure generalization performance, we resort the most widely used metric for image retrieval DML, Recall @ k 25 Additionally, we also evaluate results over mean average precision (mAP@ 1000) [54]
42. but provide respective tables and visualizations in the supplementary when the interpretation of results is not impacted.
The exact training and test splits ultimately utilized throughout this work are selected based on Fig. [] to ensure approximately uniform coverage of the spectrum of distribution shifts within intervals ranging from the lowest (near i.i.d. splits) to the highest generated shift achieved with class removal. For experiments on CARS196 and Stanford Online Products, eight total splits were investigated, included the original default benchmark split. For CUB200-2011 we select nine splits to also account for benchmark additions with reduced distributional shifts. The exact FID ranges are provided in the supplementary. Training on CARS196 and CUB200-200 was done for a maximum of 200 epochs following standard training protocols utilized in (54, while 150 epochs were used for the much larger SOP dataset. Additional training details not directly stated in the respective sections can found in the supplementary.
5

---

Cub2000 2011
Carsi 96
Online Produrts
AUC Bealligi
0.65
0.67
0.80
0755
Multiiilerity
Margin
0.75
0750
Arcace
0.66
Proa//ncho
0.6
0.76
0745-
R-Margin
0740
Uniborn
0.64
0.3
0.71
075
S2SD
DIVA
Figure 3: Comparison of DMI. methods via AGS based on Recall@) I across benchmarks. To compute AGS (cf. S.I. we aggregate the performances from Fig. 2 across train-test distribution shifts of our proposed ooDML benchmark using the Area- Srea-Under-the Searr-thhCCurre Curve metric.
4.1 Zero-shot generalization under varying distribution shifts
Many different concepts have been proposed in DML learn embedding functions & that generalize from the training distribution to differently distributed test data. To analyze the zero-shot transfer capabilities of DML models, consider representative approaches making use of the following concepts: (i) surrogate ranking tasks and tuple mining heuristics (Margin loss with distance-based sampling [7T and Multisimilarity loss (70),, (ii) geometric constraints or class proxies (AreFace [8] Proxy Anchor (26)), (iii) learning of semantically diverse features (R-Margin [54]) and self-
supervised training (DiVA (38), adversarial regularization (Uniform Prior [59] and (iv) knowledge
self-distillation (S2SD [33
Fig. 2 (top) analyzes these methods for their generalization to distribution shifts the varying degrees represented in ooDML. The top row shows absolute zero-shot retrieval performance measured on Recall (results for mAP@ 1000 can be found in the supplementary) with respect to the FID between train and test sets. Additionally, Fig. 2 (bottom) examines the relative differences of performance to the performance mean all methods for each train-test split. Based these experiments, make the following main observations:
(i) Performance deteriorates monotonically with the distribution shifts. Independent of dataset, approach or evaluation metric, performance drops steadily as the distribution shift increases.
(ii) Relative performance differences are affected by train-test split difficulty. We see that the overall ranking between approaches oftentimes remains stable on the CARS196 and CUB200- 2011 datasets. However, we also see that particularly on a large-scale dataset (SOP), established proxy-based approaches ArcFace [8] (which incorporates additional geometric constraints) and ProxyAnchor (26) are surprisingly susceptible to more difficult distribution shifts. Both methods perform poorly compared to the more consistent general trend of the other approaches. Hence, conclusions on the generality of methods solely based the default benchmarks need be handled with care, as at least for SOP, performance comparisons reported on single (c.g. the standard) data splits do not translate to more general train-test scenarios.
(iii) Conceptual differences matter at larger distribution shifts While the ranking between most methods is largely consistent on CUB200-201 1 and CARS196, their differences in performance becomes more prominent with increasing distribution shifts. The relative changes (deviation from the mean of all methods at the stage) depicted in Fig. 2 (bottom) clearly indicates that particular methods based on machine learning techniques such as self-supervision, feature diversity (DiVA, R-Margin) and self-distlllation (S2SD) are among the best at generalizing in DML. more challenging splits while retaining strong performance on more i.i.d. splits as well.
While directly stating performance in dependence to the individual distribution shifts offers a detailed overview, the overall comparison of approaches is typically based on single benchmark scores. To further provide a single metric of comparison, we utilize the well-known (AUC) score to condense performance (either based on Recall@) or mAP@ 1000) over all available distribution shifts into a single aggregated score indicating general zero-shot capabilities. This Aggregated Generalization Score (AGS) computed based on the normalized FID scores of our splits to the interval (0, 1]. As Recall @ k or mAP@k scores are naturally bounded to [0. 1]. AGS is similarly bound [0, 1] with higher being the better model. Our corresponding results are visualized in Fig. Indeed, we that AGS reflects observations from Fig. 2 with sulf-supervision (DiVA)
Prea-under-Curye

---

and self- diffiissiilation (S2SD) generally performing best when facing unknown train-test shifts, Exact scores are provided in the supplementary.
4.2 Consistency of structural representation properties on ooDML
Aud BAAI
Jue
Minetied Rualle
CIROW
A-ia EDc
635
022
400
133
E50
6355
125
2335
103
375
100
123
2.30
123
Sule livi1
Sola tiry
Syle lit
Sple IDC.
Soak
3.00
1032
1,30
6:55
120
530
1.25
1155
372
1.00
2.50 0.55 1.0
Nassadaal
Figure 4: Generalization metrics computed on ooDML benchmark for CUB200-2011 and SOP. Each column plots one of the (normalized) measured structural representation property (cf. (42) over the corresponding Recall @ 1 performance for all examined DMI. methods and distribution shifts. Trendlines are computed as least squares fit over all datapoints (overall), respectively only those corresponding to default splits (default).
Roth et al. -4 attempts to identify potential drivers of generalization in DML by measuring the
following structural properties of representation d: (i) the mean distance Tinter between the centers of the embedded samples of each class, (ii) the mean distance Tintm between the embedded samples within class Tinma (iii) the "embedding space density' measured as the ratio Taaioo maa
Num and (iv)
'spectral decay' p( ()) measuring the degree of uniformity of singular values obtained by singular
value decomposition on the training sample representations, which indicates the number of significant directions of variance. For a more detailed description, we refer to (54). These metries indeed are empirically shown to exhibit a certain correlation to generalization performance on the default benchmark splits. In contrast, we are now interested if these observations hold when measuring generalization performance on the ooDML train-test splits of varying difficulty.
We visualize our results in Fig. 4 for CUB200-2011 and SOP, with CARS196 provided in the supplementary. For better visualization we normalize all measured values obtained for both metrics (i)-(iv) and the recall performances (Recall @ 1) the interval [o, for each train-test split. Thus, the relation between structural properties and generalization performance becomes comparable across all train-test splits, allowing us to examine if superior generalization still correlated to the structural properties of the learned representation ø. i.e. if the correlation is independent of the underlying distribution shifts. For a perfectly descriptive metric, one should expect to see heavy correlation between normalized metric and normalized generalization performance jointly across shifts. Unfortunately, our results show only a small indication any structural metric being consistently correlated with generalization performance observed when evaluating only against basic, purely discriminative DML objectives as was done in [54] for the default split, as well when incorporating methods that extend and change the base DML training setup (such as DiVA 38 or adversarial regularization (59).
varying distribution shifts. This is also
This not only demonstrates that experimental conclusions derived from the analysis of only single benchmark split may not hold for overall zero-shot generalization, but also that future research should consider more general learning problems and difficulty to better understand the conceptual impact various regulatory approaches. To this end, our benchmark protocol offers more comprehensive experimental ground for future studies to find potential drivers of zero-shot generalization in DML.
7

---

Multisimilarity
ArcFace
S2SD
DIVA
0.8
KN36
KN101
RNIS
RN18
KNJ0I
Recalee)
KNS0
0.6
Figure 5: Generalization performance for different backbone architectures for varying distribution shifts on CUB200-2011 I. We show absolute Recall @ performances averaged over 5 runs for each train-test split. Other datasets show similar results and are provided in the supplementary.
CUB200-2011
Cars 196
Stunford Online Products
0.8
0.8
0.8
BIT-21k
Rocalle
01.6
0.6
0.5
Eflaizathes 80
0.4
ResNetti
0.5
0.2
Tranod: Artisce
Train/Test Split ID
Train/esst Split ID
Train/es Split
Figure 6: Comparison of DML to various non-adapted generic representations pretrained on large amounts unlabelled data and state-of-the-de architectures. For DML, show best and worst DML objectives based on results in Fig. 2 on target dataset, architecture, amount of training data and learning objective.
Performance of generic representations are heavily dependent
4.3 Network capacity and pretrained representations
common way to improve generalization, as also highlighted in (54] and [4], to select stronger backbone architecture for feature extraction. In this section, we look at how changes in network capacity can influence OOD generalization across distribution shifts. Moreover, we also analyze the zero-shot performance of diverse set of state-of-the-arr pretraining approaches.
Influence of network capacity. Fig. we compare different members the ResNet architecture family 20 with increasing capacity, each of which achieve increasingly higher performance on i.i.d. test benchmarks such as ImageNet 77. going from small ResNet18 (R18) over ResNet50 (R50) to ResNet101 (R101) variants. As can be seen, while larger network capacity helps to some extent, we observe that performance actually saturates zero-shot transfer settings, regardless of the DML approach and dataset (in particular also the large scale SOP dataset). Interestingly, we also observe that the performance drops with increasing distribution shifts are consistent across network capacity, suggesting that zero-shot generalization is less driven by network capacity but rather conceptual choices of the learning formulation (compare Fig.
Generic representations versus Deep Metric Learning. Recently, self-supervised representation learning has taken great leaps with ever stronger models trained on huge amounts of image 29
and language data [9 35 2] These approaches are designed to learn expressive, well-transferring
features and methods like CLIP 47 even prove surprisingly useful for zero-shot classification. We now evaluate and compare such representations against state-of-the-art DML models to understand if generic representations that are readily available nowadays actually pose an alternative explicit application of DML. We select tiee-of-theerrt self-supervision models SWAV (3) (ResNet50 back- bone), CLIP [47] trained via natural language supervision on a large dataset of 400 million image and sentence pairs VVisionTransfommn 29 on both the standard ImageNet
[10 backbone), BiT(-M) 29), which trains a ResNet50-V2
million training samples) and the ImageNet- 2 lk dataset 7 50 with 14 million training samples and over 21 thousand classes, EfficientNet-B0 63 trained on ImageNet, and standard baseline ResNet50 network trained on ImageNet. Note, that none of these representations has been additionally adapted to the benchmark sets, in contrast the DML approaches which have been trained on the respective train splits.
The results presented in Fig. 6 show large performance differences of the pretrained representations, which are largely dependent on the test dataset. While BiT outperforms the DML. state-of-the- on CUB200-2011 without any finetuning. it significantly trails behind the DML models the other
8

---

ArcFacee
Multiimiladlyy
SSSD
TIVA
0.75
1.35
1.34
1.30
am
046
400
0000
007
4.10
1.10
048
0.300e
3.6
250
120
150
Datey Tost
Trator Test FID
Trin: Text 710
Tean: Tect FED
Figure 7: Few-Shot adaptation of DML representations on CUB200-200 2011 Columns show average Recall @1 performance over 10 episodes of and 5-shot adaption for various DML approaches (fewshot and zeroshot), highlighting substantial benefit of few-shot adaptation for a priori unknown distribution shifts (see black line highlighting relative improvements).
two datasets. On CARS196, only CLIP comes close to the DML approaches when the distribution shift sufficiently large. Finally, on SOP, none of these models comes even close to the adapted DML methods. This shows how although representations learned by extensive pretraining can offer strong zero-shot generalization, their performance heavily depends on the target dataset and specific model. Furthermore, the generalization abilities notably depend on the size of the pretraining dataset (compare e-g. BiT- Ik vs BiT-21k or CLIP), which significantly larger than the number of training images seen by the DML methods. We see that only actual training on these datasets provides sufficienlly reliable performance.
4.4 Few-shot adaption boosts generalization performance in DML
Since distribution shifts can be arbitrarily large, the zero-shot transfer of & can be ill-posed. Features learned on training set Xuaan will not meaningfully transfer to test samples Vust once they are sufficiently far from Xrrai as also already indicated by Fig. [ As remedy few-shot learning [60] 67
14 32 [66) assumes few samples of the test distribution to to be available during training, i.e. adjusting previously learned representation. While these approaches are typically explicitly trained for fast adaption to novel classes, we are now interested if similar adaptation of DMI. representations helps to bridge increasingly large distribution shifts.
To investigate this hypothesis, follow the evaluation protocol of few-shot learning and use representatives (also referred to as shots) of each class from test set Nest as a support set for finetuning the penultimate embedding network layer. The remaining test samples then represent the new test set to evaluate retrieval performance, also referred to as query set. For evaluation we perform 10 episodes, i.e. we repeat and average the adaptation of & over 10 different, randomly sampled support and corresponding query sets. Independent of the DML model used for learning the original representation & on Xrrain adaptation to the support data conducted using the Marginloss (71) objective with distance-based sampling [71 due to its faster convergence. This also ensures fair comparisons to the adaptation benefit to Φ and also allows to adapt complex approaches like self- supervision (DiVA (38] to the small number of samples in the support sets.
Fig. 7 shows and shot results on CUB200-2011 with CARSI96 available in the supplementary. SOP is not considered since each class is already composed of small number of samples. As we see, even very limited in-domain data can significantly improve generalization performance, with the benefit becoming stronger for larger distribution shifts. Moreover, we observe that weaker approaches like ArcFace [8] seem benefit more than state-of-the-a methods like S2SD [53] or DiVA (38]. We presume this to be caused by their underlying concepts already encouraging learning of more robust and general features. To conclude, few-shot learning provides substantial and reliable benefit when facing OOD learning settings where the shift is not known priori.
5 Conclusion
In this work we analyzed zero-shot transfer of image representations learned by Deep Metric Learn ing (DML) models. We proposed a systematic construction of train-test data splits of increasing

---

difficulty, as opposed to standard evaluation protocols that test on single data splits of fixed difficulty. Based on this, we presented the novel benchmark ooDML and thoroughly assessed current DMI. methods. Our study reveals the following main findings:
generalization only
Standard evaluation protocols are insufficient to probe general out-of-distributtio transfer: Prevailing train-test splits in DML are often close to i.i.d. evaluation settings. Hence, they only provide limited insights into the impact of train-test distribution shift on generalization performance. Our benchmark ooDML alleviates this issue by evaluating large, controllable and measurable spectrum of problem difficulty to facilitate future research.
Larger distribution shifts show the impact of conceptual differences in DML approaches: Our study reveals that generalization performance degrades consistently with increasing problem difficulty for all DML methods. However, certain concepts underlying the approaches are shown to be more robust to shifts than others, such as semantic feature diversity and knowledge-distilltton Generic, self-supervised representations without finetuning can surpass dedicated data adap- tation: When facing large distribution shifts, representations learned only by self-supervvsion on large amounts of of unlabelled data competitive to explicit DML. training without any finetuning. However, their performance is heavily dependent on the data distribution and the models themselves. Few-shot adaptation consistently improves -ittiiiiistiittiionn generalization in DML: Even very few examples from a target data distribution effectively help to adapt DML representations. The benefit becomes even more prominent with increasing train-test distribution shifts, and encourages further research into few-shot adaptation in DML.
Funding transparency statement This research has been funded by the German Federal Ministry for Economic Affairs and Energy within the project "KI-Absicherung Safe Al for automated driving" and by the German Research Foundation (DFG) within projects 371923335 and 421703927. Moreover, it was funded in part by CIFAR AI Chair at the Vector Institute, Microsoft Research, and an NSERC Discovery Grant. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute www.vectorinstiuutea//tttp aapprrrtes
Acknowledgements We thank the International Max Planck Research School for Intelligent Sys- tems (IMPRS-IS) for supporting K.R; K.R. acknowledges his membership in the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program.
References
[1] Biagio Brattoli, Joseph Tighe, Fedor Zhdanov. Pietro Perona, and Krzysztof Chalupka. Rethinking zero- shot video classification: End-to-end training for realistic applications. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child. Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, 102., Inc., 2020.
[3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 9912-9924. Curran Associates, Inc., 2020.
[4] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1599-16077 PMLR, 13-18 Jul 2020.
[5] Weihua Chen, Xiaotang Chen, Jianguo Zhang, and Kaiqi Huang. Beyond triplet loss: deep quadruplet. network for person re-identification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[6] Yinbo Chen, Xiaolong Wang. Zhuang Liu, Huijuan Xu, and Trevor Darrell. A meta-baselin for few-shot learning. CoRR, abs/2003. .043900 2020.
[7] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li. and L. Fei-Fei, ImageNet: A Large-Scor Scale Hierarchical Image Database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
10

---

[8] Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[9] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pages 4171-41866 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai. Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Hcigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[1]] Yueqi Duan, Wenzhao Zheng, Xudong Lin, Jiwen Lu, and Jie Zhou. Deep adversarial metric learning. In The IELE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[12] Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Exploring the landscape of spatial robustness. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1802-1811, PMLR, 09-15 Jun 2019.
[13] Istvan Fehervari, Avinash Ravichandran, and Srikar Appalaraju. Unbiased evaluation of deep metric learning algorithmss. 2019.
[14] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Tch, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 Proceedings of Machine Learning Research, pages 1 126-1135. PMLR, 06-11 Aug 2017.
[15] Weifeng Ge. Deep metric learning with hierarchical triplet loss. Proceedings of the European Conference on Computer Vision (ECCV). pages 269-285, 2018
[16] Jacob Goldberger, Geoffrey Hinton. Sam Roweis, and Russ R Salakhutdinov. Neighbourhood components analysis. In I... Saul, Y. Weiss, and I.. Botton, editors, Advances in Neural Information Processing Systems, volume 17. MIT Press, 2005.
[17] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2006.
[18] Ben Harwood, BG Kumar, Gustavo Carneiro, lan Reid, Tom Drummond, et al. Smart mining for deep metric learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 2821- - 2829, 2017.
[19] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[20] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[21] Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
[22] Alexander Hermans, Lueas Beyer, and Bastian Leibe. In Defense of the Triplet Loss for Person Re- Identification. arXiv e-prints, page arXiv: M.XKvv117330773::5 Mar. 2017.
[23] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium, 2017.
[24] Pierre Jacob, David Picard, Aymeric Histace, and Edouard Klein. Metric learning with horde: High-order
regularizer for deep embeddings. The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
[25] Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 33(1): 17--181 2011.
[26] Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June
2020.
[27] Wonsik Kim, Bhavya Goyal, Kunal Chawla, Jungmin Lee, and Keunjoo Kwon. Attention- ensemble for deep metric learning. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
[28] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Farnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in-the-wii thel distribution shifts, 2021.
[29] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Andrea Vedaldi, Horst Bischof,

---

Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020, pages 491-507. Cham, 2020. Springer International Publishing.
[30] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops. pages 554-561, 2013.
[31] David Krueger, Ethan Caballero, Joern- Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville, of-ttriibbiiito generalization via risk extrapolation (rex), 2021. [32] Kwonjoon Lee. Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differen- tiable convex optimization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 2019.
[33] Xudong Lin, Yueqi Duan, Qiyuan Dong, Jiwen Lu, and Jic Zhou. Deep variational metric learning. In The European Conference on Computer Vision (ECCV), September 2018.
[34] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. IEEE Conference on Computer Vision and Pattern Recognition
(CVPR). 2017.
[35] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeulemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach, 2019.
cite arxiv:1907.11633
[36] Timo Milbich, Miguel Bautista, Ekaterina Sutter, and Bjöm Ommer. Unsupervised video understanding by reconciliation of posture similarities. In Proceedings of the International Conference on Computer Vision
(ICCV), 2017.
[37] Timo Milbich, Omair Ghori, and Björ Ommer. Unsupervised representation learning by discovering reliable image relations. Puttern Recognition, 102, June 2020.
[38] Timo Milbich, Karsten Roth, Homanga Bharadhwaj. Samarth Sinha, Yoshua Bengio, Björn Ommer, and Joseph Paul Cohen. Diva: Diverse visual feature aggregation for deep metric learning. In Andrea Vodaldi, Horst Bischof, Thomas Brox, and Jan- Michael Frahm, editors, Computer Vision ECCV 2020, pages 590-607, Cham, 2020. Springer International Publishing.
[39] Timo Milbich, Karsten Roth, Biagio Brattoli, and Bjöm Ommer. Sharing matters for generalization in deep metric learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 2020.
[40] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext- invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[41] Yair Movshovitz-Ares distance metric learning using proxies. In Proceedings of the IEEE International Conference on Computer Vision, pages 360-368, 2017.
Alexander Toshev, Thomas K Leung, Sergey loffe, and Saurabh Singh. No fuss
[42] Kevin Musgrave, Serge Belongie, and Ser-Nam Lim, A metric learning reality check, 2020.
[43] Hyun Oh Song. Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition,
pages 4004-4012, 2016.
2016.
[44] Michael Opitz. Georg Waltner, Horst Possegger, and Horst Bischof. Bier-boosting independent embeddings robustly. In Proceedings of the IEEE International Conference on Computer Vision, pages 5189-5198, 2017.
[45] Michael Opitz, Georg Waltner, Horst Possegger, and Horst Bischof. Deep metric learning with bier: Boost- ing independent embeddings robustly. IEEE transactions on pattern analysis and machine intelligence, 2018.
[46] Qi Qian, Lei Shang, Baigui Sun, Juhua Hu, Hao Li, and Rong Jin. Softrriple loss: Deep metric learning without triplet sampling. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019.
[47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020. 2021.
[48] Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In International Conference on Learning Representations, 2020.
[49] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages $389-5400. PMLR. 09-15 Jun 2019.
[50] Tal Ridnik, Emanuel Ben Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-211 pretraining for the masses. CoRR, abs/2104. 10972, 2021.
[51] Karsten Roth, Biagio Brattoli, and Bjorn Ommer. Mic: Mining interclass characteristics for improved metric learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 8000-
8009, 2019.
12

---

[52] Karsten Roth, Timo Milbich, and Bjom Ommer. Pads: Policy-adapted sampling for visual similarity learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[53] Karsten Roth, Timo Milbich, Björn Ommer, Joseph Paul Cohen, and Marzyeh Ghassemi. S2SD: simulta- neous similarity-based self-distillation for deep metric learning. CoRR, abs/2009. 0000009 2020.
[54] Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Bjom Ommer, and Joseph Paul Cohen. Revisiting training strategies and generalization performance in deep metric learning. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 8242-8252. PMLR, 13-18 Jul 2020.
[55] Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf. Thomas Brox, and Peter V. Gehler. Towards total recall in industrial anomaly detection. CoRR, abs/2106.08266 2021.
[56] Artsiom Sanakoyeu, Vadim Tschernezki, Uta Buchler, and Bjorn Ommer. Divide and conquer the embedding space for metric learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[57] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for facc recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823, 2015,
[58] Jenny Seidenschwarz, Ismail Elezi, and Laura Leal-Taixé. Learning intra-batch connections for deep metric learning, 2021.
[59] Samarth Sinha, Karsten Roth, Anirudh Goyal, Marzych Ghassemi, Hugo Larochelle, and Animesh Garg. Uniform priors for data-efficient transfer, 2020.
[60] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in
Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[61] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Advances in Neural Information Processing Systems, pages 1857-1865, 2016.
[62] Juan Luis Suárez, Salvador García, and Francisco Herrera. A tutorial on distance metric lcarning: Mathematical foundations, algorithms and experiments, 2018.
[63] Mingxing Tan and Quoe Lc. EfficientNet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6105-6 65-61 14. PMLR, 09-15 Jun 2019.
[64] Eu Wern Tch, Terrance DeVries, and Graham W. Taylor. Proxynca++: Revisiting and revitalizing proxy neighborhood component analysis. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and MachMcl Frahm,, editors, Computer Vision ECCV 2020, pages 448-464, Cham, 2020. Springer International Publishing.
[65] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In International Conference on Learning Representations, 2020.
[66] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: A good embedding is all you need? In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Man-Mihhael Frahm, editors, Computer Vision ECCV 2020, pages 266-282, Cham, 2020. Springer International Publishing.
[67] Eleni Triantafillou, Richard Zemel, and Raquel Urtasun. Few-shot learning through an information retrieval lens. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus. S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. [68] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 I Dataset. Technical Report CNS- TR- 201200112 1001, California Institute of Technology. 2011.
[69] Jian Wang. Feng Zhou, Shilei Wen, Xiao Liu, and Yuanqing Lin. Deep metric learning with angular loss. In Proceedings of the IEEE International Conference Computer Vision, pages 2593-2601, 2017.
[70] Xun Wang. Xintong Han, Weilin Huang, Dengke Dong, and Mathew R. Scott. Multi-similarity loss with general pair weighting for deep metric learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[71] Chao-Yuna Yuan Wu, R Manmatha, Alexander Smola, and Philipp Krahenbuhl. Sampling matters in deep embedding learning. In Proceedings of the IEEE International Conference on Computer Vision, pages
2840 2800-2888,222 2017.
[72] Hong Xuan, Abby Stylianou, and Robert Pless. Improved embeddings with easy positive triplet mining. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision ( (WACV), March 2020.
[73] Andrew Zhai and Hao-Y Yu Wu. Classification is a strong baseline for deep metric learning. 2018.
[74] Wenzhao Zheng, Zhaodong Chen, Jiwen Lu, and Jie Zhou. Hardness-aware deep metric learning. The
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
13


Reviews:
[{"review_id": "YfxFOsNbvVa", "reviewer": "Reviewer_de3V", "summary": "This work proposes a new way to measure OOD performance of metric-learning algorithms. It introduces a proxy-metric for measuring ranking dataset generalization difficulty based on FID score. They construct several train/test splits of increasing difficulty based on this score. They show that splits constructed in such a way are well correlated with performance, thus validating the proposed proxy-metric of OOD generalization difficulty.  Several experiments are performed to assess the performance of modern metric learning methods. An AUC type score over all splits (difficulties) is proposed as an overall score for various methods. Finally, additional experiments that include increasing network capacity, using self-supervised pre-trained models, and fine-tuning with few-shot learning are presented. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed proxy-metric of difficulty makes sense, and is nicely empirically validated. I am glad to see that the splits/code will be open sourced because these will be very valuable to the community. Particularly moving beyond the recent negative results  [52, 40]. \nBecause this paper presents a new benchmark it may be well suited to the Benchmarks & Datasets track at NeurIPS (although I am happy to have it accepted via the main submission route as well). Although some of the conclusions are not particularly surprising (e.g. few-shot learning on the target distribution helps, self-supervised learning doesn't always work), they are nevertheless useful experiments to the community.\nSome questions:\n1-How were the error bars in Figure 3 computed, aren't the splits deterministic? \n2-According to Figure 1 there is a wide, but limited region for each dataset where swapping affects the FID. Do you only use these dataset splits in your experiments as the early/late ones lead to saturated FID scores ? I know that these are further extended using removals. \n3- Would the methods presented in the paper have impact on the few-shot learning community as well (e.g. MetaDataset uses fixed splits)? \n4-It would be nice to see other methods that are known to increase generalization i the analyses, data augmentation is an obvious one that comes to mind. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Because this work presents results on three particular datasets, it is not clear if the methods will generalize to others, though the results of the paper suggest that they will. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "6h75OU1ZSi3", "reviewer": "Reviewer_VKkF", "summary": "The authors present a new benchmark for deep metric learning algorithms, particularly for zero-shot tasks, in order to reflect their generalization under out-of-distribution shifts performance. They propose a novel way of splitting train and test data with increasing difficulty(the gap(distribution shift) between train and test get higher). FID score is chosen to determine the distribution shift between train and test. After an initial split is chosen, the classes are exchanged between the train and test data to obtain a higher FID measure at each time iteratively. The authors suggest to use ROC curve to obtain a one single criteria, while keeping the results for each split in order to observe the performance based on task difficulty. They include experimental analysis on their proposed criteria and show results for some existing methods and architectures. The authors also analyze query-support framework in few-shot learning and its effect on the generalization performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### *Originality:*  \n\n -  There are a few recent previous attempts to analyze quantify the difficulty of few/zero shot tasks and the it is still an important open question. Progressively increasing the difficulty of train-test split and obtaining a range of tasks as a benchmark is novel as of my knowledge. They used Resnet FID distance to achieve this in order to cover many of the existing methods.\n    - The related work contains sufficient citations of the previous contributions(especially 52,40,27), however the authors could explain them in a couple of more sentences to create a better set up for the current progress and their contribution.\n    - Suggested citations: Two Sides of Meta-Learning Evaluation:In vs. Out of Distribution(Amrith Setlur et. al.). Huang, Gabriel, Hugo Larochelle, and Simon Lacoste-Julien. \"Are Few-shot Learning Benchmarks Too Simple?.\" (2019).\n\n - The paper also switches from zero shot case to few shot by including some class samples into their test split and show that the generalization improves (as expected) as another novel contribution.\n\n#### *Quality*\n\n- Choosing FID metric can be unreliable as the authors are aware of. Figure S2 is sufficient proof of concept for me at this stage. Although a comparison between the datasets might not be reliable(Table 1 suggests the otherwise), it can still be a relative measure for a given fixed dataset.\n\n\n- Why didn't you simultaneously decrease FID to obtain simpler/simplest tasks given that your initialization is random? Is it guaranteed that random initialization is simple enough to cover a wider difficulty range?\n\n\n- The methods seem to act similar in recall criteria(Figure 2), is it necessary to use splitting via FID in this case? The authors could provide better insights on why mAP and recall behaves very differently(Figure 2).\n\n\n- The authors mention their weaknesses in having similar max FID value for different datasets, being flawed in its descriptiveness(Figure 4) and provided sufficient explanations for the possible reasons.\n\n\n- Can this method be used in domain adaptation? The method might be costly for the large datasets, isn't? \n\n\n- Adapting the method directly might cause a bias for example if minimizing FID measure alone helps to get better results in more difficult data splits.\n\n#### *Clarity:*\n\n- The paper is clearly written, the figures are very helpful. \n\n\n - As I mentioned above, the related work section could be more informative.\n\n#### *Significance:*\n\n- The proposed method is applicable to any zero/few shot dataset(as long as not very large and checking FID scores before using as a benchmark might be needed). This makes the method widely useable. \n\n- The problem itself is important and hard to solve, I like the idea of creating different distribution shifts for a given dataset. The method has a potential to be further improved or inspire future works, both experimental or theoretical approaches.  \n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Proposing a benchmark is always important in fairness in the evaluation of the algorithms. The authors already mention weak points.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "6X95lvTIjvo", "reviewer": "Reviewer_kgM7", "summary": "This paper proposed a new benchmark for deep metric learning under varying degree of distribution shift. First, it introduces a method for creating train/test split with increasing FID --- thereby increasing distribution shift between train and test. Second, various metric learning methods are compared under the benchmarks of varying difficulties.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\n- It is interesting and novel to quantify (a) the degree of distribution shift & (b) the effect of such shift.\n- The paper is largely well-written and easy to understand.\n\n[Weaknesses]\n- It introduces the problem, but it does not tell much about the problem. From machine learning theory, we know that out-of-distribution benchmark is hard. Introducing a simple method to mitigate the problem can be a plus to this paper.\n- It does not provide much insight on why different method performs better or worse under out-of-distribution scenarios.\n- I'm not sure how useful this benchmark is. We know that machine learning does not aim to work under out-of-distribution shift. Also, I think this benchmark is meaningful when the ranking between methods under i.i.d benchmark is different from out-of-distribution benchmark. From what I observed, the relative ranking is largely preserved under different difficulties of data split.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations or potential negative societal impact are not included in this paper. Including limitations as well as the position of this paper against prior literature will greatly enhance the quality of this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "-iAk9AlPsLH", "reviewer": "Reviewer_vS38", "summary": "The authors of this paper investigate how generalization is affected by varying the training and testing data splits in deep metric learning (DML). Specifically, they create train/test splits (splitting on classes) which increase in difficulty (meaning that the distributions are more different) and evaluate DML generalization. To measure the gap between train and test distributions, they use FID with ResNet-50. Experimentally, they use popular benchmarks and the Recall@k metric to show that performance decreases as the train/test FID distances become larger. This, and other experiments, show that using a fixed train/test split can lead to misleading conclusions about generalization.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nThe idea is very interesting and original. It makes sense that different train/test splits will make the training harder and this should be investigated.\n\nQuality:\n\nOverall, the paper is of high quality and is relatively easy to understand.\n\nThere are some mistakes in the paper. For example, on line 107, the authors state that FID uses Inception-v2 when it actually is Inception-v3. \n\nSome things are not well justified. For example, the approximation of FID using only the mean is not well justified. Just because the mean-FID is monotonically increasing per iteration does not mean that the FID will behave the same. More justification is needed here. Another justification that is needed is why ResNet-50 was used and why the authors think that FID with this network will work well on the CUB200-2011, CARS196, and Stanford Online Products datasets, assuming that it was trained on ImageNet.\n\nThere might be an issue with the FID calculation as well. If the train/test splits have varying (or small) sizes then FID will become biased. See “Effectively Unbiased FID and Inception Score and where to find them” for more details.\n\nClarity:\n\nOverall, the paper is easy to read and understand.\n\nThe fonts in the figures are too small, making them hard to read. The captions and descriptions are also hard to understand. Moreover, several figures do not print well in black and white.\n\nSignificance:\n\nI am not sure about the significance, because I don’t specialize in deep metric learning. However, the authors make a great point that if the train/test splits make a huge difference, then they should be investigated. However, the authors also split on classes, which may be causing the huge difference in performance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations:\n\nThe main limitation I see is the application of a mean-only FID using ResNet-50 instead of Inception-v3 on different datasets than ImageNet.\n\nPotential negative societal impacts:\n\nNone that I can think of.", "review_text": "", "extra_scores": {"ethical_concerns": "Nope", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a new benchmark for deep metric learning under varying degree of distribution shift. although the paper does not take a step in providing a solution, we are hopeful that this work inspires follow up work on OOD generalization. I request the authors to incorporate the reviews and discussions in the camera ready version.
Decision: Accept (Poster)