Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the problem of cross-domain few-shot learning in images, and focuses on pre-training and fine-tuning schemes. Through empirical study, the paper shows the findings about the effectiveness of supervised pre-training on base classes and self-supervised pre-training on few-shot data. The paper also incorporates two metrics, domain similarity and few-shot difficulty, to help diagnose the model performance. Finally, a two-stage pre-training scheme with a combination of supervised learning and self-supervised supervised learning is proposed to improve performance in cross-domain few-shot learning.


strengths_and_weaknesses: - Strengths:
This is one experimental analysis paper, and the paper is organized in a systematic way in both writing and experimental presentation.

- Weaknesses:
1. To quantify the difficulty of a dataset, the paper defines the metric of few-shot difficulty, based on the empirical upper bound of few-shot performance. Specifically, it is shown in L188 that “we use 20% of the target dataset as labeled data to pre-train the model in a supervised manner. Then, the pre-trained model is evaluated on the remaining unseen target data for the 5-way k-shot classification task.” One question arises for the selection of the 20% of the target dataset as labeled data, is there a class split control? the class split control will affect the baseline results significantly which is already shown in some few-shot related papers. Also, the random selection for constructing the “5-way k-shot classification task” will also affect the baseline results. How are the class split control and the construction of “5-way k-shot classification task” realized?

2. L218: In implementation, though different backbone networks (e.g. ResNet18 and ResNet10) are used depending on the source data, they are still shallow backbones. Will these conclusions still hold for deeper backbones?
questions: Please refer to the above "Weaknesses".
limitations: No limitation and potential negative societal impact section is included in the main paper.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper investigates using (extra) unlabeled target domain data for cross-domain few-shot learning (CD-FSL), as well as the impact of source-target domain similarity and the inherent few-shot difficulty. Specifically, given the unlabeled target domain data, the authors set up two learning schemes, i.e., the self-supervised learning and the mixed-supervised learning. Through extensive experiments, they reveal some observations w.r.t. the domain similarity and the few-shot difficulty.
strengths_and_weaknesses: ### Strengths
- Extensive experiments.
- The paper is easy to follow.

### Weakness

- The assumption that there are extra unlabeled samples for the target domain is controversial. In realistic few-shot learning, the testing set (consisted of the support and query images) may be collected on-the-fly. How can you identify the target domain in advance and collect the unlabeled images?
- In Section 4, some important descriptions are misleading as if SSL is superior than SL for base-training (e.g., L224 "investigating the superiority of SSL over SL for pre-training"). The real reason is that they use different domains for base-training: the SSL uses the target-domain and the SL uses the source domain. 
- It is not clear that on the target domain, whether the unlabeled data (for base-training) and the labeled data (for support and query) have overlapping classes. 
- In overall, the observations and discoveries provide limited knowledge for understanding the cross-domain few-shot learning. Adding target domain data improves target-domain accuracy is natural, even if the target domain samples are unlabeled. 

questions: 
- Do the unlabeled data and the labeled data have overlapping classes? How do you ensure this?
- In practice, how do you identify the target domain in advance and prepare the unlabeled images?
- Why do you use Eq.5 to measure the difficulty of few-shot learning? You should explain its rationality. If you choose different settings of $\beta$ and $k$, does it change your observations?
limitations: The authors have pointed out some limitations in the conclusion part.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors propose to use Domain Similarity and Few-Shot Difficulty to understand cross-domain few-shot learning. In particular, based on the Domain Similarity and Few-Shot Difficulty, the authors have several observations regarding the performance of self-supervised pre-training and supervised pre-training for cross-domain few-shot learning. Moreover, the authors also propose two additional pre-training strategies to improve the performance of cross-domain few-shot learning. 
strengths_and_weaknesses: Strengths:
1. Overall, the paper is well-written and structured. The considered problem, i.e., cross-domain few-shot learning, is also important in the area of few-shot learning.
2. The paper also has several observations regarding cross-domain few-shot learning. 
3. The experimental designs are interesting and shed light upon several things about cross-domain few-shot learning. 

Weaknesses:

1. Most of the observations are not surprising. Due to the domain difference, it is expected that self-supervised pre-training on the target can achieve better results than supervised pre-training on the source. 

2. The domain similarity in the paper is computed based on EMD. It is not clear if the computed domain similarity reflects the actual similarity between the datasets.  

3. The lack of visualization on the trained representation makes it hard to understand what causes the difference between self-supervised pre-training and supervised pre-training for cross-domain few-shot learning. 

4. The proposed improved schemes achieve marginal performance improvement compared with the baselines. 

######################
Post-rebuttal: the rebuttal provided by the authors addresses my concerns. In the revised version, the authors are expected to add the additional results which can greatly improve the paper. I increase my score from 4 to 5. 

questions: 1. Another experiment that is worth consideration is supervised pre-training on the labeled target domain data. This can further confirm the usefulness of self-supervised pre-training.

2. Would the observations hold even in the cases of larger shots, such as k=20 and k=50？

3. The analysis on the data augmentation is too coarse-grained. What's the individual impact of Gaussian blur and random gray scale on the performance of SSL?


limitations: Yes
review_title: null

Meta-review: metareview: Authors present a comprehensive assessment of learning strategies for CD-FSL, including supervised learning, self-supervised learning (4 variants thereof),  semi-supervised learning (referred to as "mixed supervised"), single-stage and dual-stage training, and data augmentation strategies. Patterns are found among all these approaches by correlating results to source-target similarity by EMD, and measuring target task difficulty from SL. 

Pros:
- [AC/R] A comprehensive study like this is currently missing from the CD-FSL literature. Some findings are less obvious than others and provide valuable information for the community.
- [AC/R] Extensive Experiments
- [AC/R] Well-written and easy to follow

Cons:
- [R] Assumption of unlabeled samples for the target domain is controversial. [AC] CD-FSL benchmark included unlabeled samples for study. While there may be applications where unlabeled data is not available, it does not diminish the utility of this setting.
- [R] Not clear if unlabeled and labeled data in target domain have overlapping classes. [AC] Assumption of benchmark is that there is overlapping classes, though acknowledge in practice this may not be the case. Probably an idea for a future benchmark rather than a critique of this work.
- [R] Some results are not surprising. Authors addressed this concern by pointing out that their results are in contradiction to prior findings, specifically around SSL. 
- [AC/R] Only one distance measure was studied. Authors responded rationale for why EMD makes sense, but preferably they could have provided multiple measures and assessed how the multiple measures are similar and different, and how the findings may change. 
- [R] Lack of visualization of embeddings. Authors have added visualizations to the appendix.

Overall reviews lean toward accept after revisions. Some negative comments from one reviewer are more critiques of prior published benchmark, not this particular work. Given this, AC feels assessment is mostly accept, though borderline. If authors could add additional distance measures and study how the different measures impact results, would improve quality and impact of paper. Recommend accept.

AC Rating: Borderline Accept

confidence: Certain
Decision: Accept