Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a new benchmark for deep metric learning under varying degree of distribution shift. First, it introduces a method for creating train/test split with increasing FID --- thereby increasing distribution shift between train and test. Second, various metric learning methods are compared under the benchmarks of varying difficulties.
main_review: [Strengths]
- It is interesting and novel to quantify (a) the degree of distribution shift & (b) the effect of such shift.
- The paper is largely well-written and easy to understand.

[Weaknesses]
- It introduces the problem, but it does not tell much about the problem. From machine learning theory, we know that out-of-distribution benchmark is hard. Introducing a simple method to mitigate the problem can be a plus to this paper.
- It does not provide much insight on why different method performs better or worse under out-of-distribution scenarios.
- I'm not sure how useful this benchmark is. We know that machine learning does not aim to work under out-of-distribution shift. Also, I think this benchmark is meaningful when the ranking between methods under i.i.d benchmark is different from out-of-distribution benchmark. From what I observed, the relative ranking is largely preserved under different difficulties of data split.
limitations_and_societal_impact: Limitations or potential negative societal impact are not included in this paper. Including limitations as well as the position of this paper against prior literature will greatly enhance the quality of this paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors of this paper investigate how generalization is affected by varying the training and testing data splits in deep metric learning (DML). Specifically, they create train/test splits (splitting on classes) which increase in difficulty (meaning that the distributions are more different) and evaluate DML generalization. To measure the gap between train and test distributions, they use FID with ResNet-50. Experimentally, they use popular benchmarks and the Recall@k metric to show that performance decreases as the train/test FID distances become larger. This, and other experiments, show that using a fixed train/test split can lead to misleading conclusions about generalization.
main_review: Originality:

The idea is very interesting and original. It makes sense that different train/test splits will make the training harder and this should be investigated.

Quality:

Overall, the paper is of high quality and is relatively easy to understand.

There are some mistakes in the paper. For example, on line 107, the authors state that FID uses Inception-v2 when it actually is Inception-v3. 

Some things are not well justified. For example, the approximation of FID using only the mean is not well justified. Just because the mean-FID is monotonically increasing per iteration does not mean that the FID will behave the same. More justification is needed here. Another justification that is needed is why ResNet-50 was used and why the authors think that FID with this network will work well on the CUB200-2011, CARS196, and Stanford Online Products datasets, assuming that it was trained on ImageNet.

There might be an issue with the FID calculation as well. If the train/test splits have varying (or small) sizes then FID will become biased. See “Effectively Unbiased FID and Inception Score and where to find them” for more details.

Clarity:

Overall, the paper is easy to read and understand.

The fonts in the figures are too small, making them hard to read. The captions and descriptions are also hard to understand. Moreover, several figures do not print well in black and white.

Significance:

I am not sure about the significance, because I don’t specialize in deep metric learning. However, the authors make a great point that if the train/test splits make a huge difference, then they should be investigated. However, the authors also split on classes, which may be causing the huge difference in performance.
limitations_and_societal_impact: Limitations:

The main limitation I see is the application of a mean-only FID using ResNet-50 instead of Inception-v3 on different datasets than ImageNet.

Potential negative societal impacts:

None that I can think of.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors present a new benchmark for deep metric learning algorithms, particularly for zero-shot tasks, in order to reflect their generalization under out-of-distribution shifts performance. They propose a novel way of splitting train and test data with increasing difficulty(the gap(distribution shift) between train and test get higher). FID score is chosen to determine the distribution shift between train and test. After an initial split is chosen, the classes are exchanged between the train and test data to obtain a higher FID measure at each time iteratively. The authors suggest to use ROC curve to obtain a one single criteria, while keeping the results for each split in order to observe the performance based on task difficulty. They include experimental analysis on their proposed criteria and show results for some existing methods and architectures. The authors also analyze query-support framework in few-shot learning and its effect on the generalization performance.
main_review: #### *Originality:*  

 -  There are a few recent previous attempts to analyze quantify the difficulty of few/zero shot tasks and the it is still an important open question. Progressively increasing the difficulty of train-test split and obtaining a range of tasks as a benchmark is novel as of my knowledge. They used Resnet FID distance to achieve this in order to cover many of the existing methods.
    - The related work contains sufficient citations of the previous contributions(especially 52,40,27), however the authors could explain them in a couple of more sentences to create a better set up for the current progress and their contribution.
    - Suggested citations: Two Sides of Meta-Learning Evaluation:In vs. Out of Distribution(Amrith Setlur et. al.). Huang, Gabriel, Hugo Larochelle, and Simon Lacoste-Julien. "Are Few-shot Learning Benchmarks Too Simple?." (2019).

 - The paper also switches from zero shot case to few shot by including some class samples into their test split and show that the generalization improves (as expected) as another novel contribution.

#### *Quality*

- Choosing FID metric can be unreliable as the authors are aware of. Figure S2 is sufficient proof of concept for me at this stage. Although a comparison between the datasets might not be reliable(Table 1 suggests the otherwise), it can still be a relative measure for a given fixed dataset.


- Why didn't you simultaneously decrease FID to obtain simpler/simplest tasks given that your initialization is random? Is it guaranteed that random initialization is simple enough to cover a wider difficulty range?


- The methods seem to act similar in recall criteria(Figure 2), is it necessary to use splitting via FID in this case? The authors could provide better insights on why mAP and recall behaves very differently(Figure 2).


- The authors mention their weaknesses in having similar max FID value for different datasets, being flawed in its descriptiveness(Figure 4) and provided sufficient explanations for the possible reasons.


- Can this method be used in domain adaptation? The method might be costly for the large datasets, isn't? 


- Adapting the method directly might cause a bias for example if minimizing FID measure alone helps to get better results in more difficult data splits.

#### *Clarity:*

- The paper is clearly written, the figures are very helpful. 


 - As I mentioned above, the related work section could be more informative.

#### *Significance:*

- The proposed method is applicable to any zero/few shot dataset(as long as not very large and checking FID scores before using as a benchmark might be needed). This makes the method widely useable. 

- The problem itself is important and hard to solve, I like the idea of creating different distribution shifts for a given dataset. The method has a potential to be further improved or inspire future works, both experimental or theoretical approaches.  




limitations_and_societal_impact: Proposing a benchmark is always important in fairness in the evaluation of the algorithms. The authors already mention weak points.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes a new way to measure OOD performance of metric-learning algorithms. It introduces a proxy-metric for measuring ranking dataset generalization difficulty based on FID score. They construct several train/test splits of increasing difficulty based on this score. They show that splits constructed in such a way are well correlated with performance, thus validating the proposed proxy-metric of OOD generalization difficulty.  Several experiments are performed to assess the performance of modern metric learning methods. An AUC type score over all splits (difficulties) is proposed as an overall score for various methods. Finally, additional experiments that include increasing network capacity, using self-supervised pre-trained models, and fine-tuning with few-shot learning are presented. 
main_review: The proposed proxy-metric of difficulty makes sense, and is nicely empirically validated. I am glad to see that the splits/code will be open sourced because these will be very valuable to the community. Particularly moving beyond the recent negative results  [52, 40]. 
Because this paper presents a new benchmark it may be well suited to the Benchmarks & Datasets track at NeurIPS (although I am happy to have it accepted via the main submission route as well). Although some of the conclusions are not particularly surprising (e.g. few-shot learning on the target distribution helps, self-supervised learning doesn't always work), they are nevertheless useful experiments to the community.
Some questions:
1-How were the error bars in Figure 3 computed, aren't the splits deterministic? 
2-According to Figure 1 there is a wide, but limited region for each dataset where swapping affects the FID. Do you only use these dataset splits in your experiments as the early/late ones lead to saturated FID scores ? I know that these are further extended using removals. 
3- Would the methods presented in the paper have impact on the few-shot learning community as well (e.g. MetaDataset uses fixed splits)? 
4-It would be nice to see other methods that are known to increase generalization i the analyses, data augmentation is an obvious one that comes to mind. 
limitations_and_societal_impact: Because this work presents results on three particular datasets, it is not clear if the methods will generalize to others, though the results of the paper suggest that they will.
review_title: null

Meta-review: This paper proposed a new benchmark for deep metric learning under varying degree of distribution shift. although the paper does not take a step in providing a solution, we are hopeful that this work inspires follow up work on OOD generalization. I request the authors to incorporate the reviews and discussions in the camera ready version.
Decision: Accept (Poster)