Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a meta-learning approach for semi-supervised multi-domain source-free domain adaptation (SFDA). As first contribution, the authors extend the multi-domain SFDA problem to a semi-supervised learning scenario where a few labeled examples of the target domain are provided in addition to the unlabeled ones. As second contribution, they propose a meta-learning framework where a student learns from multiple teachers, each one trained on different source data. The student is trained to predict the correct class for labeled examples and, for unlabeled examples, to be consistent with a weighted average of the teachers' predictions where weights are learned. A bilevel optimization strategy is employed to update the parameters of the student and the teachers. The proposed method is evaluated on a multi-label chest X-ray classification task using five datasets. Results show that the method yield some improvements compared to recent approaches for SFDA.
strengths_and_weaknesses: Strengths:

* Novelty: Although it borrows from previous works like Meta Pseudo Labels, the overall method proposed in the paper is novel. In particular, the coordinating weight learning is most original. The extension of multi-source SFDA problem to a semi-supervised setting is also interesting, even if it relates to other machine learning problems like domain generalization and continual learning. 

* Experiments and results: The experimental evaluation of the method is comprehensive. It uses five large datasets and compares against several baselines, ablation variants and recent approaches. While the method's performance is only slightly better than competing approaches, it seems to provide better predictions when looking at the visualization examples.

Weaknesses:

* A potential weakness of the method is the high computational complexity brought by the bilevel optimization. It would be useful to compare the runtime of tested approaches, in addition to their accuracy.

* Figure 2 indicates that varying beta and gamma does not improve the performance much and that the corresponding loss terms might not be so useful.

* The coordinating weight learning is interesting, however experiments do not really study this component. If possible, it would be interesting to show what are weights for some examples of different sources.

*  I feel that the related works in the Supplemental materials are more relevant than those in the main paper, as they focus on SFDA and compared methods of the experiments. I recommend the authors to add them to the main paper (or swap the content if lacking space).

Minor comments:

* Introduction: define multi-label classification.

* Section 3.2: pesudo-label --> pseudo-label

* Eq (5): min_{theta_S} --> argmin_{theta_S}  ?

* Algorithm 1, line 5 (Sup.mat.): missing Theta_s on the right side ?

  
questions: * Please provide some comparison in terms of computational complexity

* Give an example or explain what the behavior of the method for predicting W.


limitations: The limitations of the method are not mentioned in the paper.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: 1. The authors proposed a new problem setting, i.e., semi-supervised multi-source-free domain adaptation (SMDA) for multi-label medical image classification.

2. A novel framework MetaTeacher based on a multi-teacher and one-student scheme is introduced to solve the proposed SMDA problem.

3. A coordinating weight learning method is derived for dynamically revealing the performance differences of different source models over different classes. It is integrated with the semi-supervised bilevel optimization algorithm for consistently updating the teacher and student models.
strengths_and_weaknesses: Strength:

The proposed Semi-supervised Multi-source-free Domain Adaptation (SMDA) setting sounds interesting in the context of medical image classification.

Weakness:

My major concern lies in the effectiveness of the proposed solution for SMDA.

In Tables 2-4, it is obvious that the proposed approach (ours (all)) only performs slightly better than DECISION [1] on the transfer to Google-Health-CXR, NIH-CXR14, and Open-i. Considering DECISION is the state-of-the-art multi-source-free domain adaptation method, I think the proposed solution cannot reflect well on the value of SMDA because the multi-source-free domain adaptation approach can already well tackle the SMDA setting.


questions: Please refer to the weakness part.
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a MetaTeacher for semi-supervised multi-source-free domain adaptation of medical image classification. The transfer learning process is modeled as a multi-teacher and one-student scheme. This model not only optimizes student, but also optimizes teachers through student’s feedback in the target domain. The optimization is based on meta-learning, which consists of two main part: coordinating weight learning, and bilevel optimization. Finally, experimental experiments on multi-label chest x-ray datasets empirically demonstrated the superiority of the proposed model against other SOTA methods.
strengths_and_weaknesses: Strengths:
(1) This paper studies a new problem setting, i.e., semi-supervised multi-source-free domain adaptation for multi-label medical image classification. 
(2) A mutual feedback mechanism is designed based on meta-learning between the target model and the source models for more coherent learning and adaptation.
(3) The coordinating weight learning method is derived for dynamically revealing the performance differences of different source models over different classes.

Weaknesses:
(1) This is not clear why do the authors design a multi-teacher one-student scheme? Whether we can set one-teacher and on-student framework? If yes, what is the performance when using one teacher?
(2) In Table 1, it can be seen that the proposed model still obtains promising performance when without using coordinating weight learning.
(3) Table 5 shows the effects of the size of labeled target data on the transfer from NIH-CXR14, CheXpert, MIMIC-CXR to Open-i. However, this experiment is only conducted on the proposed model, how does this is compared to other semi-supervised methods? 



questions: (1) Why do the authors design a multi-teacher one-student scheme? Whether we can set one-teacher and on-student framework? If yes, what is the performance when using one teacher?
(2) In Table 1, it can be seen that the proposed model still obtains promising performance when without using coordinating weight learning. Please analyze the reasons for this case.
(3) In Table, this experiment is only conducted on the proposed model, how does this is compared to other semi-supervised methods? 


limitations: See questions.
review_title: null

Meta-review: metareview: The paper proposes a model for a multiple teacher, single student setting for medical image classification. 

The reviewers where split, with two reviewers leaning towards accept and one leaning towards reject. The main criticism of the negative reviewers is that the proposed model only slightly outperforms the state of the art. Given the extensive experimental evaluation and the fact that the proposed method consistently outperforms the state of the art, the improvement should be statistically significant. The negative reviewer has acknowledged the improvement in the discussion phase. A second criticism was the lack of significance of the proposed learning setting. As the reviewers find this setting novel and due to the relevance in the medical domain, I vote to accept the paper.
confidence: Certain
Decision: Accept