Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a method for training on “cross-domain few-shot recognition”. This is a task in which the goal is to transfer representations learned on a base, labeled dataset into a target few-shot dataset that is of a different domain. In particular, the paper focuses on the setting where unlabeled data is available for the target domain.

The method proposed comprises of training a “student model” on the base data in a supervised manner, while also utilizing a distillation loss on the target data, using a “teacher network”’s predictions as the objective. The teacher is set as an exponential moving average of the student’s parameters. After this initial training phase, the student gets a new randomly initialized head to learn on the few-shot labeled target data.

main_review: The method proposed is very interesting and seems to yield good results on the BSCD-FSL benchmark. I found the insights in lines 42-44, 266-276 particularly insightful.

# Major Comments
It is very related to many similar techniques, such as MoCo, SimCLR, BYOL, etc. In particular, it seems like it differs from MoCo only by the addition of a supervised head and the downstream finetuning on few-shot datasets. I found it disappointing to not see it in the baselines compared. Other baselines (line 111) seem to have been left out because they “assume smaller gap [between domains]”, but this means that this claim isn’t shown empirically. Overall, this decreases my confidence in the high performance of the proposed method.

I also am not convinced by the conclusion of lines 226-227, as another possible explanation is that the target tasks are simply too easy for extra large data to show any benefit.

# Minor Comments
I found it surprising that training with Ours-All in Table 6 did not yield the best results in all target tasks. Why do the authors think that is?

Overall I found the paper very interesting and well explained, and would consider raising my score if my main concerns are addressed.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The submission attempts to address the cross-domain few-shot learning problem by utilizing dynamic distillation from additional unlabeled target data. It simplifies the STARTUP framework by removing the contrastive loss and using a similar network for teacher and student. Moreover, it introduces different data augmentations to learn better representations. The teacher network is also trained dynamically to enhance the training of the student network.
main_review: Strength:
+ The submission is well-written and easy to follow.
+ Comprehensive experiments and promising results.
+ Detailed analysis and convincing ablation study. 

Weakness:
- Limited novelty since the approach combines the mean teacher approach from [23] and self-supervised loss from [5] and [2], yet achieving good performance in the cross-domain few-shot learning context.
- A question about the distillation loss l_u. The cross-entropy function cannot guarantee that the prediction of the student is equal to the prediction of the teacher. Please clarify this. 
- It would be better to explain why and how to choose the weak and strong data augmentations.

Minor suggestions:
- Please highlight the numbers in Tables 5 and 6. 

limitations_and_societal_impact: The proposed approach doesn't achieve the best performance on ChestX data.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The author propose using teacher-student knowledge distillation for improving the performance of cross-domain few-shot learning. 
Improved results on cross-domain few-shot learning benchmarks are observed and the performance on standard few-shot learning benchmarks are shown to be comparable to state-of-the-art.
main_review: The method introduced in this paper is well-presented. 
The empirical results are somehow strong.

Concerns:
- My main concern to this paper is the motivation. Using student-teacher knowledge distillation is not a new idea in training deep neural networks. It remains unclear to me why such method is strong on cross-domain few-shot learning without seeing the experimental results. I expect the authors to provide more discussion on the advantages of the proposed methods, e.g., what specific natures of cross-domain few-shot learning is exploited? The current discussion shows only 'another way of using unlabeled data' in cross domain few-shot learning, yet this is something can be exploited in many tasks like standard few-shot learning, domain generalizations, etc.. 

- The paper is somehow poorly organized. It is surprising to see that the main discussion (methodology) is not even 1.5 page long with a Figure in it. I think this is also part of the reason that the motivation of this paper is not well presented. 

- Some widely used cross-domain few-shot learning settings were set in [1], and are not included in the paper. 

- The performance on standard few-shot classification datasets are actually not comparable to SOTA. E.g., according to the leaderboard [2], with the standard inductive setting, many methods can achieve over 54% with simple Conv-4 architecture on miniImageNet 5way 1shot. While in-domain few-shot classification is obviously less challenging, it is confusing to me why the proposed method performs poorly. 

- Can the authors provide more discussion on how the proposed method is 'dynamic'?

[1] Chen, Wei-Yu, et al. "A closer look at few-shot classification." ICLR.
[2] https://few-shot.yyliu.net/miniimagenet.html
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper tackles the cross-domain few-shot learning, by utilizing the unlabeled target data to encourage the feature extractor to learns the representations that is effective for target domain. They propose a dynamic distillation where the teacher is dynamically updated via exponential moving average during training. The dynamic distillation works similar to recent self-supervised learning method of BYOL, and successfully enhance the generalization capability of learned representation using unlabeled data. Dynamic distillation network achieves the state-of-the-art performance in BSCD-FSL benchmark.
main_review: [Strength]
1.	Dynamic distillation method utilizing the exponential moving average model as a teacher is technically sounds and interesting concept, considering the exponential moving average encoder or momentum encoder is widely used to generate the target feature in self-supervised learning, and successfully learns the more general representations. 
2.	The authors provided various analysis and ablation experiment results, which is helpful to understand the proposed method. Analysis on the source of unlabeled data (Table 6), and effect of the data augmentation is interesting, not only from the point of view of cross-domain FSL but also from the point of view of self-supervised learning.
3.	The performance gain over prior work (STARTUP) is remarkable. Also, the authors provided various baselines related to transfer learning and self-supervised learning. 


 [Question]

1.	In Tables 1 and 2, the performance of the proposed method is compared with the STARTUP and SimCLR-based baselines. However, BYOL is the most similar SSL method to the proposed method. How is the performance of the BYOL or Transfer + BYOL in cross-domain FSL?
2.	In ablation experiments, the authors presented the performance of Ours (w/o base), that pretrain the model on miniImageNet (base dataset) and then learn representation without labeled base dataset. What will happen if someone pretrains the model on base dataset, and then learn representation using the unlabeled images and images from base dataset (use the images only, and not use the label. Consider the images from base dataset as unlabeled data)? It can be an interesting experiment to figure out the effect of supervised loss in the proposed method.
3.	In line 212 to 214, the authors stated that the proposed method does not use any self-supervised training or distillation. However, the proposed method is dynamic distillation, and it utilizes the unlabeled samples like the self-supervised learning model, BYOL. Can you explain the exact meaning of the sentence in line 212 to 214?

* Typo: In line 136, "datatset" ->"dataset"
limitations_and_societal_impact: The authors addressed the limitations of their work.
review_title: null

Meta-review: The paper received mixed ratings, with three reviewers recommending acceptance and one rejection. The reviewers' main concerns include the novelty of the method, its motivation, missing comparison to some baselines, and the clarity of some parts of the paper. The paper went through several rounds of questions/answers, and the authors feedback addressed the concerns of most reviewers. The most negative reviewer did not update their score, but we believe that the latest feedback from the authors, on which the reviewer did not comment, better addressed some of the concerns raised by the reviewer. As such, we believe that the paper can be accepted to NeurIPS but strongly recommend the authors to incorporate their feedback in the final version.
Decision: Accept (Poster)