Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a framework for cross-domain few-shot learning. The framework consists of three steps: 1) pre-training backbone; 2) meta-learn the feature masking network; 3) fine-tuning on target domain. Experiments on the CDFSL dataset show this framework outperforms SOTA methods. 
main_review: ## Strengths
1. The positive and negative features separation is novel, which serves as a regularization for fine-tuning on the target domain. 
2. The algorithm 1 is clean and easy to understand. 
3. The performance on CDFSL outperforms several SOTA methods. 

## Weakness
1. The feature masking module has too many losses making it difficult to reproduce. Comparing to fine-tuning full network, the improvements are small. 
2. The ablation study presented in Sec 4.4 is either inefficient or unclear. For example, can we replace eq. (8) with || f_i^t - f_i^+ ||^2? 
3. Clarity can be improved: 
a) The labels on D_s are discarded? 
b) How do you prevent trivial masks? I.e., all close to 1's or all close to 0's? 
c) The intuition on the losses in eq.(6) and (8) are not very straightforward. It would be better to start from the simplest case without any of these losses and then add them one by one. How did you remove the mask module in Table 2? 
d) Why do you train the mask generator on D_t rather than D_s? 
e) How do you tune hyper-params if no validation set on target domain doesn't exist.  
summary_of_the_review: The method is technically sound, and the paper is easy to follow. However, there are still many details are either hidden or unclear, which hinders its reproducibility. Besides, the method is only evaluated on CDFSL, which is a relatively small dataset. It would be more convincing if the method is applied and compared on Meta-Dataset (another larger cross-domain dataset). Hence, this is a borderline paper to me.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a generic framework for cross-domain few-shot learning. There are three steps: 1. Pre-training the backbone unsupervisedly using a self-supervised contrastive loss. 2. Select relevant features via a mask module. 3. Fine-tune the network with the selected features. The proposed framework is evaluated in the recently proposed CDFSL benchmark. The results show that the proposed framework outperforms the baselines in most of the cases. 
main_review: Strengths:

1. The paper is well-written and the easy to read. The organization is also clear.
2. The idea is interesting and makes sense. The authors have clear motivation on each step of the proposed framework. Pretrain the network unsupervisedly is natural in the case of cross-domain few-shot learning due to the domain difference between the source domain and target domain. The feature selection module is also designed with a clear motivation for cross-domain few-shot learning. The decoupling of positive features and negative features is shown to be helpful. 
3. The authors also conduct extensive analysis of the proposed method. 


Weaknesses:

1. The proposed framework has a lot of hyperparameters to tune which is an issue for few-shot learning since there is no validation set to use.

2. No standard deviations are reported in the tables which is important for the evaluation of the few-shot learning due to the randomness.


More questions:

1.  How about using negative features for fine-tuning? This can be a good ablation to show the usefulness of positive features.

2. Another ablation is to pre-train supervisedly to show the advantages of self-supervised pretraining. 

3. How the mask module is trained? And would a better mask module produces better final performance?

summary_of_the_review: In this paper, the authors propose a framework for cross-domain few-shot learning. The idea is reasonable and the authors also provide a good analysis of the components of the framework. The proposed framework outperforms the baselines in most of the cases.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a framework, terms as ConFeSS, for dealing with cross-domain few-shot learning problems. Specifically, it firstly trains a feature extracting backbone with the contrastive loss on the base category data for learning better features. Then, it trains a masking module to select relevant features suited to target domain classification. Finally, a classifier is fine-tuned along with the backbone such that the backbone produces features similar to the relevant ones. Experiments are conducted on several cross-domain few-shot learning benchmarks.
main_review: Paper strengths:
- The cross-domain few-shot learning problem studied in this paper is a fundamental task that deserves further study.
- The paper is well written and easy to follow.
- The proposed method is straightforward.
- Experiments are sufficient, and ablation studies can bring good observations and analyses for the method.

Paper weaknesses:
- The proposed method consists of three stages for training, which seems complicated and a little bit tricky in implementations. Could it be simplified in the training process?
- In experiments, most results of the proposed method are satisfactory. However, on CropDisease, it is significantly worse than the results of STARTUP. Are there any reasons for the observations?

summary_of_the_review: The authors are encouraged to respond to the comments listed in the paper weaknesses during rebuttals.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The problem is to learn by few shots (FSL) when the final domain is highly distant from the source base (i.e. natural, medical and satellite), dubbed single source cross-domain few-shot learning. In short, the approach consists of three steps: 1) train a feature extracting backbone with the contrastive loss on a (single source) base category; 2) train a masking module to select relevant features for the target domain; 3) fine-tuned along with the backbone to give features similar to the relevant ones.  The last two are claimed to be novel and, together with the experiments on CDFSL benchmark, sold as the main contributions.
main_review: The challenge is to surpass those approaches which deal on FSL with meta learning. My main claim is: why do not compare with standard approaches from Multi-task learning, Embedding learning, Learning with External Memory? FSL by Meta Learning is just one way, not necessarily the best one. In the experimental results, some approaches have been considered. They should be mapped with respect to a wider taxonomy of FSL. To help the authors, in the introduction ("Few-shot learning methods aim to uncover…"), I suggest to add general references about FSL, such as WANG, Yaqing, et al. Generalizing from a few examples: A survey on few-shot learning. ACM Computing Surveys (CSUR), 2020, 53.3: 1-34. This to justify the meta train-meta test cycle, and to make clear that FSL is not just meta learning. 

Why do we need to stick with a single source domain, when there are plenty of universal representation to start the FSL, as the same authors discuss, with (Dvornik et al., 2020)?

Why do we need to apply unsupervised training on the single source domain, instead of a supervised one? Single source domain should be known, hence labeled training data should be available to train with supervision. Perhaps this has to create a general vocabulary of features, due to the distance between domain and target, but this should be better explained. 

Are the authors sure that the framework can be cast as meta learning? In general, in meta learning there should be an outer loop and an inner loop, where the outer loop modifies the parameters for the inner one. Instead of this nested general algorithm, the present one is more a serial 3-step architecture, where each step is done just one time. I agree that each step has a double loop inside, in fact this is just a question for clarification. 

What is strange to me is the selection of so many hyper parameters, 8. I'm ok with the Hyper-parameter sensitivity ablation study, which is useful for the lambdas, but how do they have been found, especially the number of training epochs for the three steps, which appear so different (400, 15, 50)? 

In the appendix H, a general reference to introduce the VC dimension should be given (such as the N. Cristianini and J. Shawe-Taylor one) 

summary_of_the_review: The paper is ok to me, even if it not so general, since it cope with a very specific configuration (single-source domain, FSL with meta learning) which could be just one case of a more general situation, which may lead to better results (multi source domain, or domain which are kind of closer to the final ones). Experiments seem convincing.
review_title: null

Meta-review: Summary: 
Paper addresses the cross-domain few-shot learning scenario, where meta-learning data is unavailable, and approaches are evaluated directly on novel settings. Authors propose a 3-step approach: 1) self-supervised pretraining, 2) feature selection, 3) fine-tuning, and demonstrate gains over state-of-art. 

Pros:
- Approach is novel for this setting
- Paper is clear and easy to understand
- Performance beats several prior methods
- Experiments are thorough
- Fundamental problem is worthwhile of investigation 

Cons:
- Some concerns among multiple reviewers on how hyperparameters are selected. Authors have provided more information and tables in the paper.
- Training process is multi-step and not unified. Authors provided additional information about unified training results, which yielded poorer results, likely due to overfitting from training many parameters at once. 

Overall recommendation based on the consensus of reviewers and AC expertise: accept.
Decision: Accept (Poster)