Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The proposed method addresses test-time adaptation as knowledge distillation from mixture of experts. Each expert is learned within each source domain as teachers. During testing time, a set of unlabeled sample is used to query the knowledge from the teachers. A transformer-based model is used to integrate knowledge from experts and meta-learning is used to encourage the aggregation for useful knowledge and student for good adaptation.
strengths_and_weaknesses: Strengths.

The idea is interesting. Distilling knowledge from different sources is not new, but the knowledge distillation and the meta learning seem novel to me. The paper is properly presented and some constrained real-word settings are also considered.

Weaknesses.

The major weakness is the lack of comparison with prior work on standard datasets. For example [67] (and more recent ones) and the datasets used by [67]. If the authors are willing to validate their method on these standard datasets, more recent works can be compared. Without comparing with those methods, it is hard to evaluate the performance of the method. 


-------------------------- Post Rebuttal -----------------------------------------

My concerns regarding to comparison with [67] on standard datasets have been resolved. I raise my score to weak accept.
questions: See weaknesses.

What are the comparison results of the proposed method with recent SOTA on the datasets in [67]?
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper tackles the unsupervised domain adaptation scenario, with multiple labeled source domains and an unlabeled target domain. The work draws inspiration from prior work on test-time adaptation, in particular ARM [72]. The key idea is to train a separate model for each source domain, and leverage the mixture of experts (MoE) to fully benefit from the knowledge from these domains. A meta-learning framework is proposed, where a student network aims to distill knowledge from the teacher MoE model. A transformer-based aggregator network is used to combine and contextualize MoE knowledge. Both the teacher and student models are meta-trained in a bi-level optimization framework. The adapted student network is then used during inference. The paper presents experiments on the popular WILDS benchmark, demonstrating competitive performance vs. state-of-the-art methods on a range of tasks.
strengths_and_weaknesses: Strengths
- The proposed approach builds upon related prior work yet appears to be sufficiently novel and also effective.
- Comparison to SoTA seems comprehensive. The proposed method does not outperform SoTA in all cases but delivers strong performance across the board, which is not true about the other methods.
- The authors present a detailed and informative ablation study demonstrating that the choices made in the model lead to superior performance.
- I also appreciate the additional studies concerning the constrained computational cost and data privacy settings.

Weaknesses

Most of the key weaknesses are around presentation/clarity and lack of certain experimental details. (Additional issues are listed under Questions.)

- One of the arguments in favor of Domain Generalization is that "the data samples from a target domain ... might be unavailable" L35. This is emphasized by the authors, but later is ignored, since they operate in the setting where unlabeled target data is available. It is indeed a realistic case that _no target domain data is available_, and I was hoping the authors plan to address that, but that was not the case. Perhaps the presentation could be made clearer on that matter.

- Moreover, it would be good to detail, how much target data is indeed used by the proposed method? From the supplemental material Tab 11 it looks like for iWildCam there are 24 images used for adaptation. Would have been good to include these details (for all the datasets) in the main paper! 
- Finally, what about the competing baselines, how many target images were used in those? I.e. is the comparison fair?

- The experimental settings should be detailed more, e.g. we only see the number of source domains in each dataset, but not the number of target domains! 
- Moreover, it is unclear how the validation/test splits are defined, e.g. is validation set part of the target domain? Is it labeled or unlabeled? (Perhaps these are standard settings for WILDS, but would be good to clarify for the readers not closely familiar with WILDS!)
- Since it is not stated clearly, is it correct that all the labels are shared across the source and target domains in this work?

Writing
- There are quite a few sentences with incorrect grammar (in the Intro and throughout the paper), I recommend proof-reading.
- L70 "Our contributions are manifold" : I do not think this is the right use of the word "manifold".
- In the Related Work section, ideally each paragraph should include direct contrast of prior works to the proposed method, to make the differences clear.

questions: - Please address the question above about the number of target samples used by the proposed method vs. the baselines.

- Please address the questions above about the experimental details such as number of target domains and val/test splits.

- Section 4.1: It would be good to clarify, is the source domain data used to train the expert model overlapping or is it disjoint from the support/query sets used later in meta-training? (I would expect that these are not seen while training the expert model?)

- The ablation study "Number of domain-specific experts" L340 needs a clarification, what does "increasing the number of experts" really mean? Is the overall training set fixed and only the "clusters" within it are more coarse/fine? Or is each case using more/less data? (I would expect the former?)

- For the aggregator architecture ablation, it would have been interesting to also include another learned yet non-transformer design (in addition to the naive max/mean).

- Table 8 shows that distilling features alone is better than also distilling the logits, do you have any thoughts on the reasons behind it?
limitations: The guidelines explicitly state that "The page limit was increased to ensure that authors have space to address the checklist questions." Thus it would have been more appropriate to include the Discussion on Social Impact in the main paper rather than the supplemental.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: 
Here, the authors present a novel framework (Meta-DMoE) to alleviate domain shift using unlabelled data during test time. Many techniques were combined such as knowledge distillation and meta-learning in order to address existing limitations in previous methods, including training on multiple source domains with a single model and applying it to all subsequent unseen target domains and its bias toward domain-invariant features. The method was evaluated using five test datasets from WILDS  and compared against existing state-of-the-art methods addressing domain shift.

strengths_and_weaknesses: Strengths

1) Meta-DMoE explores and takes advantage of knowledge distillation and meta-learning to devise a method that can deal with domain shit at test time using unlabelled data. A model with such skill is valuable where training might be expensive and this contribution is important.

2) The authors conducted extensive experiments (WILDS image testbeds and out-of-distribution settings, constraints on computational cost, and privacy regulation ) and Meta-DMoE presented the best results in most cases.

3) Ablation studies were also extensive exploring reasonable conditions.


Weaknesses

Minor:

1) I thought the reading was a bit repetitive, e.g.
Line 70: We name our method as Meta-Distillation of MoE (Meta-DMoE)
Line 162: We name our approach Meta-distillation from MoE (Meta-DMoE)

questions: As stated earlier, please double-check points in the text that might sound a bit repetitive.
limitations: Limitations were properly addressed in the text.
review_title: null

Meta-review: metareview: This paper addresses the DG-TTA problem setting, drawing inspiration from the recent Adaptive Risk Minimisation (ARM). It goes beyond ARM to introduce a mixture of experts and distills from those mixture of experts during adaptation. Reviewers agree the MoE idea makes sense, and the distillation-based adaptation is interesting and novel enough, and they appreciated the good results and evaluation. Concerns included various writing clarity issues and evaluation on other datasets, but these questions were generally resolved in the rebuttal. Given that concerns were resolved and all reviewers were positive, I recommend accept.

confidence: Certain
Decision: Accept