Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to tackle the problem of cross-domain few-shot classification, where such problem scenario especially considers the few-shot learning under the case of having (large) domain-shift between base and target domains. In particular, with being motivated to address the large domain gap, this paper follows the prior works (i.e. STARTUP@ICLR2021 and Dynamic-Distillation@NeuraIPS2021) to include a specific assumption of having access to a small portion of unlabelled images in the target domain during the training stage. 

Moreover, the proposed method of this paper also follows the similar train of thoughts as the prior works: 1) having a teacher model trained on base dataset; 2) having self-supervised contrastive learning on the unlabeled target images; 3) the student for the target domain tries to learn from the teacher model by using the knowledge distillation (KD) technique, while the knowledge distillation strategy gradually changes from taking "the teacher trained on base dataset" as the main source of knowledge (similar to STARTUP) in the early iterations to taking "the moving average of the updated teachers on previous iterations" as the main source of knowledge (similar to Dynamic-Distillation) in the later iterations. 

In addition to three aforementioned points which are similar to prior works, the proposed method in this paper has another two key designs: 4) the knowledge distillation implementation is cross-level (i.e. encouraging the shallow layers of student to learn from the deeper layers of teacher), in which the idea is similar to BYOT@ICCV2019 (the slight difference is that BYOT operates self-distillation); 5) due to the phenomenon stemmed from the self-supervised loss and discovered by previous works where the final feature vector for classification has a small number of strongly activated elements while the others are close to zero, the proposed method introduces the "feature denoising" technique to truncate the elements with small magnitudes to zero during the fine-tuning stage (i.e. updating the model to the target domain by using the given target-domain few-shots with labels). 

The proposed method is experimentally evaluated on the BSCD-FSL benchmark and the comparison with respect to several baselines (including ATA and CI which tackle the cross-domain few-shot classification problem without access to target domain data during training, self-supervised methods, STARTUP, and Dynamic-Distillation) is made, where the proposed method is shown to provide superior performance. 
strength_and_weaknesses: Strength:
+ The proposed method is able to outperform different types of baselines (including ATA and CI which tackle the cross-domain few-shot classification problem without access to target domain data during training, self-supervised methods, STARTUP, and Dynamic-Distillation, where the latter two are under the same problem scenario as the proposed method) in a challenging benchmark.
+ The detailed investigations/ablation studies are performed to validate the model designs and hyper-parameter settings in the proposed method (e.g. the model structure for cross-level knowledge distillation, the hyperparameter $\tau$ for building the old student, ablation study for feature denoising, the hyperparameter $\lambda$ to balance self-supervised loss and knowledge distillation loss for training student model, and the hyperparameter $h$ for determining the number of truncation in feature denoising operation).
+ The overall organization is clear and the writing is easy to follow.  

Weaknesses:
- Though I can understand that the large domain gap between base dataset and target domain is quite challenging to be tackled, but the assumption of having access to a portion of target domain data (even it is unlabeled) seems to be quite impractical and hard to envision its usage in real-world scenarios. Moreover, even there exists two prior works (i.e. STARTUP and Dynamic-Distillation) addressing the same problem setting as the proposed method, the authors should still provide sufficient motivations as well as the description on the potential usage of such problem setting in their paper to make the whole story self-contained and persuasive. 
(Besides, in this paper they set to have 20% of target domain data available during training, in which such amount is not "a small portion" in my perspective).
- In addition to the teacher model trained on base dataset and kept fixed during knowledge distillation, the authors introduce another model, named as "old student" (which in implementation is the student model in the previous learning iteration), in order to keep the knowledge of historical students during the learning of student model. Though such design seems to provide benefit to the overall model performance, having such additional model actually increase the computational cost for the overall training procedure. Moreover, the proposed method adopts the linear combination of such old student model and the teacher model as the actual source of knowledge to execute the knowledge distillation, in which such old student model acts similarly to the moving average of historical students (i.e. an ensemble) if we expand the overall iterative procedure (thus being closely related to Dynamic-Distillation@NeuraIPS2021), hence the linear combination of the old student and the teacher becomes an "ensemble of ensemble" or analogously a straightforward integration over the distillation strategies of STARTUP and Dynamic-Distillation. Overall, such the distillation strategy of the proposed method seems to be incremental from the methodology perspective. 
- The cross-level distillation mechanism is closely related to the well-known BYOT@ICCV2019 work (where the shallow layers try to learn the knowledge from the deeper layers), in which the only difference exists on that BYOT is doing self-distillation while the proposed method applies the same idea across two networks. The cross-level distillation proposed in this paper thus seems to be incremental as well. 
- Though the experimental results show that the proposed method provide superior performance, there are some concerns as detailed below: 1) we see that the proposed method using BYOL as its self-supervised loss (denoted as BYOL+CLD+FD (ours) in Table.1) does not present significant performance improvement with respect to Dynamic-Distillation (especially the loss function in Dynamic-Distillation is closely related to BYOL); 2) the baseline SimCLR (self-supervised learning applied on unlabeled target domain data) already provides strong performance (e.g. better than BYOL+CLD+FD (ours) and even competitive with SimCLR+CLD+FD (ours) which also adopts SimCLR in its loss functions). From these two observations, the contribution of the proposed method becomes skeptical (in which the proposed method adopts many different designs into its framework but in results only achieves sightly better, comparable, or even worse performance with respect to the baselines). 
clarity,_quality,_novelty_and_reproducibility: Overall, the clarity of the paper is satisfactory (well organization and good paper writing) but the novelty is incremental (taking the aforementioned weaknesses into consideration, in which the proposed distillation strategy can be simply treated as a dynamic combination over STARTUP and Dynamic-Distillation and the proposed cross-level distillation mechanism is also identical to the well-known BYOT). Moreover, as the problem setting of having access to a portion of target domain data is not well motivated and seems to be impractical, together with the insignificant performance improvement compared to the baselines, the quality of the paper becomes also problematic. 
summary_of_the_review: Due to the limited/incremental novelty, potentially skeptical improvement with respect to baselines, as well as not-well-motivated and impractical problem setting (please refer to weaknesses for more details), I would suggest to reject this paper from ICLR2023 but is willing to change my rating if my concerns can be well addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a cross-level distillation method for cross-domain few-shot learning. The are four stages in the pipeline. A pre-trained model is trained on the base classes data, then a student network is trained with cross-level distillation by mimicking the teacher network. A feature denoising step is further added to reduce overfitting of the features. The authors conduct extensive experiments on the BSCD-FSL benchmark to show the effectiveness of the proposed approach. 
strength_and_weaknesses: Strength: 

1. The writing of the paper is clear and the organization is also well structured.
2. The idea of cross-level distillation is also interesting and novel. 
3. The authors also conducted extensive ablations to show the benefits of the proposed approach. 

Weaknesses:

1. There are too many hyperparameters in the method. It may be difficult to tune the parameters for new applications.
2. The introduction of the old student network may introduce additional overhead.

The authors may have some discussions on how to choose the hyperparameters and compare the computational complexity of the method with the baselines. 
clarity,_quality,_novelty_and_reproducibility: Clarity: overall, the paper is clearly written and easy to understand. Although there should be more motivations for the design choices of cross-level distillation. 

Quality: the quality of the paper is good.

Novelty: the idea of cross-level distillation is novel for cross-domain few-shot learning. 

Reproducibility: the paper seems reproducible. 
summary_of_the_review: The paper proposes a novel method for cross-domain few-shot learning and extensive experiments show the effectiveness of the method. The main drawbacks of the method are the 1) introduction of excessive hyperparameters and 2) the computation complexity. 

Post-rebuttal:

I have read the authors' rebuttal. It partially addresses my concerns. I tend to keep my rating but do not have an inclination to support the paper if other reviewers have concerns.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper tackles the problem of cross-domain few-shot learning, where the base dataset and the target domain are far away from each other. It builds upon past distillation-based frameworks, where unlabeled data from the target domain is used to distill a source teacher, pretrained on the base dataset. It introduces multiple modifications to this basic approach, including using distillation of multiple layers, a way of feature selection on the target problem, and an approach for avoiding inheriting the teacher's biases.

strength_and_weaknesses: Strengths:
1. The problem of cross-domain few-shot learning is a practically significant problem.
2. The results obtained by the proposed method are much more accurate than prior state-of-the-art.
3. The paper has thorough ablations for its many contributions.

Weaknesses:
1. The contributions seem like engineering innovations rather than new insights. However, each contribution has well thought-out motivations, so I am less worried.
clarity,_quality,_novelty_and_reproducibility: Generally I find the paper well written. The code will be released, and that should make the approach reproducible. The insights though not particularly novel are still well motivated and worth knowing for the community.
summary_of_the_review: Despite my short review, I find the contributions useful, the problem significant, and the result impressive. I am in favor of acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The submission introduces a cross-domain few-shot learning approach called cross-level distillation (CLD). It operates in four distinct phases:

1. A teacher model is pre-trained on the base classes using the usual cross-entropy loss.
2. A student network is trained on unlabeled target data using a combination of contrastive self-supervised learning and cross-level distillation from the teacher network. The knowledge distillation loss gradually shifts from focusing on the teacher network activations to focusing on the activations of previous student networks.
3. The student network's classification layer is then fine-tuned on the labeled support set using "feature denoising", i.e., exponentiating embedding values with the largest magnitude and zeroing out the others.
4. The student network is evaluated on the query set.

The proposed approach is evaluated on BSCD-FSC and compared against several recent baselines, including STARTUP and Dynamic-Distillation, and is claimed to achieve state-of-the-art performance.

Reviewers note the proposed approach's strong performance, the thorough ablation studies, and the submission's writing clarity. The main concerns surfaced by reviewers are:

- The assumption regarding access to unlabeled target data is not well justified (am4M). The authors respond by citing defect detection as an example of a practical application where this assumption holds. Reviewer am4M contends that this is one example, and that a more complete justification would be needed to make the assumption convincing. Given the existence of previously published works (STARTUP, Dynamic-Distillation) making a similar assumption, I think we can safely assume that the research community is convinced that it makes sense at least in some circumstances, and I side with the authors.
- The proposed approach contains many moving parts requiring careful hyperparameter optimization (8cJ8). The authors explain that the hyperparameters are chosen using EuroSAT and shared across other evaluation settings, which shows that a good choice of hyperparameters transfers across settings. In general, approaches with many moving parts face a larger burden of proof, but in this instance all reviewers agree that the ablation studies are comprehensive. I side with the authors on this point.
- The dual use of a teacher network and an old student network introduces additional computational complexity (am4M, 8cJ8). The authors respond by comparing the computational complexity of CLD using a model-averaging with that of CLD using an old student. Reviewer am4M thinks this is missing their point: in previous work either a teacher network or a model-averaged old student is used, and not both. I side with reviewers on this point, although I don't think this is a critical flaw, since the computational complexity remains low enough for CLD to be applicable in practice. I would encourage the authors to incorporate this point when discussing the drawbacks of their proposed approach.
- The proposed approach feels more like a collection of engineering innovations than a new idea which brings a fresh perspective (am4M, yRW4). Moreover, the aspects of the proposed approach which appear to offer only modest performance improvements (am4M). The authors list differences between CLD and BYOT, and they point to results in Table A and Table 1 to support their claim of a significant performance improvement. Reviewer am4M is unconvinced. They feel the differences between CLD and BYOT mostly boil down to engineering decisions. They also do not feel that the authors address their concerns regarding BYOL+CLD+FD specifically, since it does not outperform Dynamic-Distillation significantly and underperforms SimCLR. I agree with the reviewers that the submission is thin on new insights and that its contributions are mostly empirical.

Opinions are split on acceptance. On one hand, the submission is clear and the proposed approach is effective and well ablated. On the other hand, there is concern about the proposed approach's longer-term impact potential. Ultimately, while I agree with those concerns, I think the submission makes enough of an empirical contribution to be of interest to the community and to warrant acceptance.
justification_for_why_not_higher_score: There is concern about the proposed approach's longer-term impact potential.
justification_for_why_not_lower_score: The submission is clear and the proposed approach is effective and well ablated. It makes enough of an empirical contribution to be of interest to the community and to warrant acceptance.
Decision: Accept: poster