Characterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning

1 Introduction . Image representations that generalize well are the foundation of numerous computer vision tasks , such as image and video retrieval [ 61 , 71 , 54 , 38 , 1 ] , face ( re- ) identification [ 57 , 34 , 8 ] and image classification [ 65 , 4 , 19 , 40 , 37 ] . Ideally , these representations should not only capture data within the training distribution , but also transfer to new , out-of-distribution ( OOD ) data . However , in practice , achieving effective OOD generalization is more challenging than in-distribution [ 28 , 12 , 21 , 49 , 31 , 55 ] . In the case of zero-shot generalization , where train and test classes are completely distinct , Deep Metric Learning ( DML ) is used to learn metric representation spaces that capture and transfer visual similarity to unseen classes , constituting a priori unknown test distributions with unspecified shift . To approximate such a setting , current DML benchmarks use single , predefined and fixed data splits of disjoint train and test classes , which are assigned arbitrarily [ 71 , 8 , 61 , 24 , 11 , 33 , 74 , 51 , 54 , 42 , 26 , 64 , 58 ] . This means that ( i ) generalization is only evaluated on a fixed problem difficulty , ( ii ) 1Code available here : https : //github.com/CompVis/Characterizing_Generalization_in_DML ∗ Equal contribution , alphabetical order , † equal supervision , x now at University of Tuebingen . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . generalization difficulty is only implicitly defined by the arbitrary data split , ( iii ) the distribution shift is not measured and ( iv ) can not be not changed . As a result , proposed models can overfit to these singular evaluation settings , which puts into question the true zero-shot generalization capabilities of proposed DML models . In this work , we first construct a new benchmark ooDML to characterize generalization under outof-distribution shifts in DML . We systematically build ooDML as a comprehensive benchmark for evaluating OOD generalization in changing zero-shot learning settings which covers a much larger variety of zero-shot transfer learning scenarios potentially encountered in practice . We systematically construct training and testing data splits of increasing difficulty as measured by their Frechet-Inception Distance [ 23 ] and extensively evaluate the performance of current DML approaches . Our experiments reveal that the standard evaluation splits are often close to i.i.d . evaluation settings . In contrast , our novel benchmark continually evaluates models on significantly harder learning problems , providing a more complete perspective into OOD generalization in DML . Second , we perform a large-scale study of representative DML methods on ooDML , and study the actual benefit of underlying regularizations such as self-supervision [ 38 ] , knowledge distillation [ 53 ] , adversarial regularization [ 59 ] and specialized objective functions [ 71 , 70 , 8 , 26 , 54 ] . We find that conceptual differences between DML approaches play a more significant role as the distribution shift to the test split becomes harder . Finally , we present a study on few-shot DML as a simple extension to achieve systematic and consistent OOD generalization . As the transfer learning problem becomes harder , even very little in-domain knowledge effectively helps to adjust learned metric representation spaces to novel test distributions . We publish our code and train-test splits on three established benchmark sets , CUB2002011 [ 68 ] , CARS196 [ 30 ] and Stanford Online Products ( SOP ) [ 43 ] . Similarly , we provide training and evaluation episodes for further research into few-shot DML . Overall , our contributions can be summarized as : • Proposing the ooDML benchmark to create a set of more realistic train-test splits that evaluate DML generalization capabilities under increasingly more difficult zero-shot learning tasks . • Analyzing the current DML method landscape under ooDML to characterize benefits and drawbacks of different conceptual approaches to DML . • Introducing and examining few-shot DML as a potential remedy for systematically improved OOD generalization , especially when moving to larger train-test distribution shifts . 2 Related Work . DML has become essential for many applications , especially in zero-shot image and video retrieval [ 61 , 71 , 51 , 24 , 1 , 36 ] . Proposed approaches most commonly rely on a surrogate ranking task over tuples during training [ 62 ] , ranging from simple pairs [ 17 ] and triplets [ 57 ] to higher-order quadruplets [ 5 ] and more generic n-tuples [ 61 , 43 , 22 , 70 ] . These ranking tasks can also leverage additional context such as geometrical embedding structures [ 69 , 8 ] . However , due to the exponentially increased complexity of tuple sampling spaces , these methods are usually also combined with tuple sampling objectives , relying on predefined or learned heuristics to avoid training over tuples that are too easy or too hard [ 57 , 72 ] or reducing tuple redudancy encountered during training [ 71 , 15 , 18 , 52 ] . More recent work has tackled sampling complexity through the usage of proxy-representations utilized as sample stand-ins during training , following a NCA [ 16 ] objective [ 41 , 26 , 64 ] , leveraging softmax-style training through class-proxies [ 8 , 73 ] or simulating intraclass structures [ 46 ] . Unfortunately , the true benefit of these proposed objectives has been put into question recently , with [ 54 ] and [ 42 ] highlighting high levels of performance saturation of these discriminative DML objectives on default benchmark splits under fair comparison . Instead , orthogonal work extending the standard DML training paradigm through multi-task approaches [ 56 , 51 , 39 ] , boosting [ 44 , 45 ] , attention [ 27 ] , sample generation [ 11 , 33 , 74 ] , multi-feature learning [ 38 ] or self-distillation [ 53 ] have shown more promise with strong relative improvements under fair comparison [ 54 , 38 ] , however still only in single split benchmark settings . It thus remains unclear how well these methods generalize in more realistic settings [ 28 ] under potentially much more challenging , different train-to-test distribution shifts , which we investigate in this work . 3 ooDML : Constructing a Benchmark for OOD Generalization in DML An image representation ϕ ( x ) learned on samples x ∈ Xtrain drawn from some training distribution generalizes well if can transfer to test data Xtest that are not observed during training . In the particular case of OOD generalization , the learned representation ϕ is supposed to transfer to samples Xtest which are not independently and identically distributed ( i.i.d . ) to Xtrain . A successful approach to learning such representations is DML , which is evaluated for the special case of zero-shot generalization , i.e . the transfer of ϕ to distributions of unknown classes [ 57 , 71 , 24 , 8 , 54 , 42 ] . DML models aim to learn an embedding ϕ mapping datapoints x into an embedding space Φ , which allows to measure similarity between xi and xj as g ( ϕ ( xi ) , ϕ ( xj ) ) . Typically , g is a predefined metric , such as the Euclidean or Cosine distance and ϕ is parameterized by a deep neural network . In realistic zero-shot learning scenarios , test distributions are not specified a priori . Thus , their respective distribution shifts relative to the training , which indicates the difficulty of the transfer learning problem , is unknown as well . To determine the generalization capabilities of ϕ , we would ideally measure its performance on different test distributions covering a large spectrum of distribution shifts , which we will also refer to as “ problem difficulties '' in this work . Unfortunately , standard evaluation protocols test the generalization of ϕ on a single and fixed train-test data split of predetermined difficulty , hence only allow for limited conclusions about zero-shot generalization . To thoroughly assess and compare zero-shot generalization of DML models , we aim to build an evaluation protocol that resembles the undetermined nature of the transfer learning problem . In order to achieve this , we need to be able to change , measure and control the difficulty of train-test data splits . To this end , we present an approach to construct multiple train-test splits of measurably increasing difficulty to investigate out-of-distribution generalization in DML , which make up the ooDML benchmark . Our generated train-test splits resort to the established DML benchmark sets , and are subsequently used in Sec . 4 to thoroughly analyze the current state-of-the-art in DML . For future research , this approach is also easily applicable to other datasets and transfer learning problems . 3.1 Measuring the gap between train and test distributions . To create our train-test data splits , we need a way of measuring the distance between image datasets . This is a difficult task due to high dimensionality and natural noise in the images . Recently , Frechet Inception Distance ( FID ) [ 23 ] was proposed to measure the distance between two image distributions by using the neural embeddings of an Inception-v3 network trained for classification on the ImageNet dataset . FID assumes that the embeddings of the penultimate layer follow a Gaussian distribution , with a given mean µX and covariance ΣX for a distribution of images X . The FID between two data distributions X1 and X2 is defined as : d ( X1 , X2 ) ≜ ∥µX1 − µX2∥ 2 2 + Tr ( ΣX1 +ΣX2 − 2 ( ΣX1ΣX2 ) 1 2 ) , ( 1 ) In this paper , instead of the Inception network , we use the embeddings of a ResNet-50 classifier ( Frechet ResNet Distance ) for consistency with most DML studies ( see e.g . [ 71 , 64 , 26 , 56 , 51 , 38 , 54 , 58 ] ) . For simplicity , in the following sections we will still use the abbreviation FID . 3.2 On the issue with default train-test splits in DML . To motivate the need for more comprehensive OOD evaluation protocols , we look at the split difficulty as measured by FID of typically used train-test splits and compare to i.i.d . sampling of training and test sets from the same benchmark . Empirical results in Tab . 1 show that commonly utilized DML train-test splits are very close to in-distribution learning problems when compared to more out-ofdistribution splits in CARS196 and SOP ( see Fig . 1 ) . This indicates that semantic differences due to disjoint train and test classes , do not necessarily relate to actual significant distribution shifts between the train and test set . This also explains the consistently lower zero-shot retrieval performance on CUB200-2011 as compared to both CARS196 and SOP in literature [ 71 , 70 , 24 , 54 , 42 , 38 ] , despite SOP containing significantly more classes with fewer examples per class . In addition to the previously discussed issues of DML evaluation protocols , this further questions conclusions drawn from these protocols about the OOD generalization of representations ϕ . 3.3 Creating train-test splits of increasing difficulty . Let Xtrain and Xtest denote the original train and test set of a given benchmark dataset D = Xtrain ∪ Xtest . To generate train-test splits of increasing difficulty while retaining the available data D and maintaining balance of their sizes , we exchange samples between them . To ensure and emphasize semantic consistency and unbiased data distributions with respect to image context unrelated to the target object categories , we swap entire classes instead of individual samples . Measuring distribution similarity based on FID , the goal is then to identify classes Ctrain ⊂ Xtrain and Ctest ⊂ Xtest whose exchange yields higher FID d ( Xtrain , Xtest ) . To this end , similar to other works [ 33 , 51 , 38 ] , we find resorting to an unimodal approximation of the intraclass distributions sufficient and approximate FID by only considering the class means and neglect the covariance in Eq . 1 . We select Ctrain and Ctest as C∗train = argmax Ctrain∈Xtrain ∥µCtrain − µXtrain∥2 − ∥µCtrain − µXtest∥2 ( 2 ) C∗test = argmax Ctest∈Xtest ∥µCtest − µXtest∥2 − ∥µCtest − µXtrain∥2 ( 3 ) where we measure distance to mean class-representations µXC . By iteratively exchanging classes between data splits , i.e . X t+1train = ( X ttrain \ C∗train ) ∪ C∗test and vice versa , we obtain a more difficult train-test split ( X t+1train , X t+1 test ) at iteration step t. Hence , we obtain a sequence of train-test splits XD = ( ( X 0train , X 0test ) , . . . , ( X ttrain , X ttest ) , . . . , ( X Ttrain , X Ttest ) ) , with X 0train ≜ Xtrain and X 0test ≜ Xtest . Fig . 1 ( columns 1-3 ) indeed shows that our FID approximation yields data splits with gradually increasing approximate FID scores with each swap until the scores can not be further increased by swapping classes . UMAP visualizations in the supplementary verify that the increase corresponds to larger OOD shifts . For CUB200-2011 and CARS196 , we swap two classes per iteration , while for Stanford Online Products we swap 1000 classes due to a significantly higher class count . Moreover , to cover the overall spectrum of distribution shifts and ensure comparability between benchmarks we also reverse the iteration procedure on CUB200-2011 to generate splits minimizing the approximate FID while still maintaining disjunct train and test classes . To further increase d ( X Ttrain , X Ttest ) beyond convergence ( see Fig . 1 ) of the swapping procedure , we subsequently also identify and remove classes from both X Ttrain and X Ttest . More specifically , we remove classes Ctrain from X Ttrain that are closest to the mean of X Ttest and vice versa . For k steps , we successively repeat class removal as long as 50 % of the original data is still maintained in these additional train-test splits . Fig . 1 ( rightmost ) shows how splits generated through class removal progressively increase the approximate FID beyond what was achieved only by swapping . To analyze if the generated data splits are not inherently biased to the used backbone network for FID computation , we also repeat this procedure based on representations from different architectures , pretraining methods and datasets in the supplementary . Note , that comparison of absolute FID values between datasets may not be meaningful and we are mainly interested in distribution shifts within a given dataset distribution . Overall , using class swapping and removal we select splits that cover the broadest FID range possible , while still maintaining sufficient data . Hence , our splits are significantly harder and more diverse than the default splits .

Summary: The authors of this paper investigate how generalization is affected by varying the training and testing data splits in deep metric learning (DML). Specifically, they create train/test splits (splitting on classes) which increase in difficulty (meaning that the distributions are more different) and evaluate DML generalization. To measure the gap between train and test distributions, they use FID with ResNet-50. Experimentally, they use popular benchmarks and the Recall@k metric to show that performance decreases as the train/test FID distances become larger. This, and other experiments, show that using a fixed train/test split can lead to misleading conclusions about generalization.