Characterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning

1 Introduction . Image representations that generalize well are the foundation of numerous computer vision tasks , such as image and video retrieval [ 61 , 71 , 54 , 38 , 1 ] , face ( re- ) identification [ 57 , 34 , 8 ] and image classification [ 65 , 4 , 19 , 40 , 37 ] . Ideally , these representations should not only capture data within the training distribution , but also transfer to new , out-of-distribution ( OOD ) data . However , in practice , achieving effective OOD generalization is more challenging than in-distribution [ 28 , 12 , 21 , 49 , 31 , 55 ] . In the case of zero-shot generalization , where train and test classes are completely distinct , Deep Metric Learning ( DML ) is used to learn metric representation spaces that capture and transfer visual similarity to unseen classes , constituting a priori unknown test distributions with unspecified shift . To approximate such a setting , current DML benchmarks use single , predefined and fixed data splits of disjoint train and test classes , which are assigned arbitrarily [ 71 , 8 , 61 , 24 , 11 , 33 , 74 , 51 , 54 , 42 , 26 , 64 , 58 ] . This means that ( i ) generalization is only evaluated on a fixed problem difficulty , ( ii ) 1Code available here : https : //github.com/CompVis/Characterizing_Generalization_in_DML ∗ Equal contribution , alphabetical order , † equal supervision , x now at University of Tuebingen . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . generalization difficulty is only implicitly defined by the arbitrary data split , ( iii ) the distribution shift is not measured and ( iv ) can not be not changed . As a result , proposed models can overfit to these singular evaluation settings , which puts into question the true zero-shot generalization capabilities of proposed DML models . In this work , we first construct a new benchmark ooDML to characterize generalization under outof-distribution shifts in DML . We systematically build ooDML as a comprehensive benchmark for evaluating OOD generalization in changing zero-shot learning settings which covers a much larger variety of zero-shot transfer learning scenarios potentially encountered in practice . We systematically construct training and testing data splits of increasing difficulty as measured by their Frechet-Inception Distance [ 23 ] and extensively evaluate the performance of current DML approaches . Our experiments reveal that the standard evaluation splits are often close to i.i.d . evaluation settings . In contrast , our novel benchmark continually evaluates models on significantly harder learning problems , providing a more complete perspective into OOD generalization in DML . Second , we perform a large-scale study of representative DML methods on ooDML , and study the actual benefit of underlying regularizations such as self-supervision [ 38 ] , knowledge distillation [ 53 ] , adversarial regularization [ 59 ] and specialized objective functions [ 71 , 70 , 8 , 26 , 54 ] . We find that conceptual differences between DML approaches play a more significant role as the distribution shift to the test split becomes harder . Finally , we present a study on few-shot DML as a simple extension to achieve systematic and consistent OOD generalization . As the transfer learning problem becomes harder , even very little in-domain knowledge effectively helps to adjust learned metric representation spaces to novel test distributions . We publish our code and train-test splits on three established benchmark sets , CUB2002011 [ 68 ] , CARS196 [ 30 ] and Stanford Online Products ( SOP ) [ 43 ] . Similarly , we provide training and evaluation episodes for further research into few-shot DML . Overall , our contributions can be summarized as : • Proposing the ooDML benchmark to create a set of more realistic train-test splits that evaluate DML generalization capabilities under increasingly more difficult zero-shot learning tasks . • Analyzing the current DML method landscape under ooDML to characterize benefits and drawbacks of different conceptual approaches to DML . • Introducing and examining few-shot DML as a potential remedy for systematically improved OOD generalization , especially when moving to larger train-test distribution shifts . 2 Related Work . DML has become essential for many applications , especially in zero-shot image and video retrieval [ 61 , 71 , 51 , 24 , 1 , 36 ] . Proposed approaches most commonly rely on a surrogate ranking task over tuples during training [ 62 ] , ranging from simple pairs [ 17 ] and triplets [ 57 ] to higher-order quadruplets [ 5 ] and more generic n-tuples [ 61 , 43 , 22 , 70 ] . These ranking tasks can also leverage additional context such as geometrical embedding structures [ 69 , 8 ] . However , due to the exponentially increased complexity of tuple sampling spaces , these methods are usually also combined with tuple sampling objectives , relying on predefined or learned heuristics to avoid training over tuples that are too easy or too hard [ 57 , 72 ] or reducing tuple redudancy encountered during training [ 71 , 15 , 18 , 52 ] . More recent work has tackled sampling complexity through the usage of proxy-representations utilized as sample stand-ins during training , following a NCA [ 16 ] objective [ 41 , 26 , 64 ] , leveraging softmax-style training through class-proxies [ 8 , 73 ] or simulating intraclass structures [ 46 ] . Unfortunately , the true benefit of these proposed objectives has been put into question recently , with [ 54 ] and [ 42 ] highlighting high levels of performance saturation of these discriminative DML objectives on default benchmark splits under fair comparison . Instead , orthogonal work extending the standard DML training paradigm through multi-task approaches [ 56 , 51 , 39 ] , boosting [ 44 , 45 ] , attention [ 27 ] , sample generation [ 11 , 33 , 74 ] , multi-feature learning [ 38 ] or self-distillation [ 53 ] have shown more promise with strong relative improvements under fair comparison [ 54 , 38 ] , however still only in single split benchmark settings . It thus remains unclear how well these methods generalize in more realistic settings [ 28 ] under potentially much more challenging , different train-to-test distribution shifts , which we investigate in this work . 3 ooDML : Constructing a Benchmark for OOD Generalization in DML An image representation ϕ ( x ) learned on samples x ∈ Xtrain drawn from some training distribution generalizes well if can transfer to test data Xtest that are not observed during training . In the particular case of OOD generalization , the learned representation ϕ is supposed to transfer to samples Xtest which are not independently and identically distributed ( i.i.d . ) to Xtrain . A successful approach to learning such representations is DML , which is evaluated for the special case of zero-shot generalization , i.e . the transfer of ϕ to distributions of unknown classes [ 57 , 71 , 24 , 8 , 54 , 42 ] . DML models aim to learn an embedding ϕ mapping datapoints x into an embedding space Φ , which allows to measure similarity between xi and xj as g ( ϕ ( xi ) , ϕ ( xj ) ) . Typically , g is a predefined metric , such as the Euclidean or Cosine distance and ϕ is parameterized by a deep neural network . In realistic zero-shot learning scenarios , test distributions are not specified a priori . Thus , their respective distribution shifts relative to the training , which indicates the difficulty of the transfer learning problem , is unknown as well . To determine the generalization capabilities of ϕ , we would ideally measure its performance on different test distributions covering a large spectrum of distribution shifts , which we will also refer to as “ problem difficulties '' in this work . Unfortunately , standard evaluation protocols test the generalization of ϕ on a single and fixed train-test data split of predetermined difficulty , hence only allow for limited conclusions about zero-shot generalization . To thoroughly assess and compare zero-shot generalization of DML models , we aim to build an evaluation protocol that resembles the undetermined nature of the transfer learning problem . In order to achieve this , we need to be able to change , measure and control the difficulty of train-test data splits . To this end , we present an approach to construct multiple train-test splits of measurably increasing difficulty to investigate out-of-distribution generalization in DML , which make up the ooDML benchmark . Our generated train-test splits resort to the established DML benchmark sets , and are subsequently used in Sec . 4 to thoroughly analyze the current state-of-the-art in DML . For future research , this approach is also easily applicable to other datasets and transfer learning problems . 3.1 Measuring the gap between train and test distributions . To create our train-test data splits , we need a way of measuring the distance between image datasets . This is a difficult task due to high dimensionality and natural noise in the images . Recently , Frechet Inception Distance ( FID ) [ 23 ] was proposed to measure the distance between two image distributions by using the neural embeddings of an Inception-v3 network trained for classification on the ImageNet dataset . FID assumes that the embeddings of the penultimate layer follow a Gaussian distribution , with a given mean µX and covariance ΣX for a distribution of images X . The FID between two data distributions X1 and X2 is defined as : d ( X1 , X2 ) ≜ ∥µX1 − µX2∥ 2 2 + Tr ( ΣX1 +ΣX2 − 2 ( ΣX1ΣX2 ) 1 2 ) , ( 1 ) In this paper , instead of the Inception network , we use the embeddings of a ResNet-50 classifier ( Frechet ResNet Distance ) for consistency with most DML studies ( see e.g . [ 71 , 64 , 26 , 56 , 51 , 38 , 54 , 58 ] ) . For simplicity , in the following sections we will still use the abbreviation FID . 3.2 On the issue with default train-test splits in DML . To motivate the need for more comprehensive OOD evaluation protocols , we look at the split difficulty as measured by FID of typically used train-test splits and compare to i.i.d . sampling of training and test sets from the same benchmark . Empirical results in Tab . 1 show that commonly utilized DML train-test splits are very close to in-distribution learning problems when compared to more out-ofdistribution splits in CARS196 and SOP ( see Fig . 1 ) . This indicates that semantic differences due to disjoint train and test classes , do not necessarily relate to actual significant distribution shifts between the train and test set . This also explains the consistently lower zero-shot retrieval performance on CUB200-2011 as compared to both CARS196 and SOP in literature [ 71 , 70 , 24 , 54 , 42 , 38 ] , despite SOP containing significantly more classes with fewer examples per class . In addition to the previously discussed issues of DML evaluation protocols , this further questions conclusions drawn from these protocols about the OOD generalization of representations ϕ . 3.3 Creating train-test splits of increasing difficulty . Let Xtrain and Xtest denote the original train and test set of a given benchmark dataset D = Xtrain ∪ Xtest . To generate train-test splits of increasing difficulty while retaining the available data D and maintaining balance of their sizes , we exchange samples between them . To ensure and emphasize semantic consistency and unbiased data distributions with respect to image context unrelated to the target object categories , we swap entire classes instead of individual samples . Measuring distribution similarity based on FID , the goal is then to identify classes Ctrain ⊂ Xtrain and Ctest ⊂ Xtest whose exchange yields higher FID d ( Xtrain , Xtest ) . To this end , similar to other works [ 33 , 51 , 38 ] , we find resorting to an unimodal approximation of the intraclass distributions sufficient and approximate FID by only considering the class means and neglect the covariance in Eq . 1 . We select Ctrain and Ctest as C∗train = argmax Ctrain∈Xtrain ∥µCtrain − µXtrain∥2 − ∥µCtrain − µXtest∥2 ( 2 ) C∗test = argmax Ctest∈Xtest ∥µCtest − µXtest∥2 − ∥µCtest − µXtrain∥2 ( 3 ) where we measure distance to mean class-representations µXC . By iteratively exchanging classes between data splits , i.e . X t+1train = ( X ttrain \ C∗train ) ∪ C∗test and vice versa , we obtain a more difficult train-test split ( X t+1train , X t+1 test ) at iteration step t. Hence , we obtain a sequence of train-test splits XD = ( ( X 0train , X 0test ) , . . . , ( X ttrain , X ttest ) , . . . , ( X Ttrain , X Ttest ) ) , with X 0train ≜ Xtrain and X 0test ≜ Xtest . Fig . 1 ( columns 1-3 ) indeed shows that our FID approximation yields data splits with gradually increasing approximate FID scores with each swap until the scores can not be further increased by swapping classes . UMAP visualizations in the supplementary verify that the increase corresponds to larger OOD shifts . For CUB200-2011 and CARS196 , we swap two classes per iteration , while for Stanford Online Products we swap 1000 classes due to a significantly higher class count . Moreover , to cover the overall spectrum of distribution shifts and ensure comparability between benchmarks we also reverse the iteration procedure on CUB200-2011 to generate splits minimizing the approximate FID while still maintaining disjunct train and test classes . To further increase d ( X Ttrain , X Ttest ) beyond convergence ( see Fig . 1 ) of the swapping procedure , we subsequently also identify and remove classes from both X Ttrain and X Ttest . More specifically , we remove classes Ctrain from X Ttrain that are closest to the mean of X Ttest and vice versa . For k steps , we successively repeat class removal as long as 50 % of the original data is still maintained in these additional train-test splits . Fig . 1 ( rightmost ) shows how splits generated through class removal progressively increase the approximate FID beyond what was achieved only by swapping . To analyze if the generated data splits are not inherently biased to the used backbone network for FID computation , we also repeat this procedure based on representations from different architectures , pretraining methods and datasets in the supplementary . Note , that comparison of absolute FID values between datasets may not be meaningful and we are mainly interested in distribution shifts within a given dataset distribution . Overall , using class swapping and removal we select splits that cover the broadest FID range possible , while still maintaining sufficient data . Hence , our splits are significantly harder and more diverse than the default splits .

Summary: This paper proposed a new benchmark for deep metric learning under varying degree of distribution shift. First, it introduces a method for creating train/test split with increasing FID --- thereby increasing distribution shift between train and test. Second, various metric learning methods are compared under the benchmarks of varying difficulties.