Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a method to adapt to group distribution shifts using meta-learning. Their proposed method is general for adapting to different test distributions. Empirical results show that their method is effective in improving the classification accuracies of common datasets by 1-4%. 

main_review: - The key observation of the authors is that changes in the underlying group distributions in input x can tell us something about the distribution of labels, and this is a natural problem that occurs for example with the different handwriting styles of different users in handwriting recognition. 

- I think the authors can provide more details on the different adaptation approaches in ARM-CML, ARM-BN, and ARM-LL, especially ARM-LL. This is an important part of the algorithm and crucial for understanding why it works, but only half a page is used for these methods. 

- When I look at Algorithm 1, apart from the sampling of z from different groups, there is nothing specific in the algorithm that is tailored for group distribution shifts. It can be applied to other types of test distribution shifts as well, as long as we have a sample of the test data. This is both a pro and a con: it is general enough to apply to other distribution shifts, but on the other hand there does not seem to be a lot of new insights and contributions to the base meta-learning approach. 

- For the empirical evaluations, the results are in general good and contain small improvements over previous adaptation approaches. 

- Overall this is a borderline paper to me because although the problem of adapting to group distribution shifts is interesting, the current paper does not add a lot on top of a direct application of the meta-learning method.  


limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a framework for unsupervised multi-source domain adaptation in which training samples are grouped according to which domain they come from. (The paper uses "groups" to refer to the domains, but I will simply use "domains".) The authors assume that predictions can be adapted using the information of the input distribution of the target domain, and the learner is allowed to use a batch of unlabeled test samples at test time in order to adapt predictions.

The authors approach this problem by learning an adaptation model h that inputs a batch of unlabeled samples and transforms the parameters of a prediction model g (Algorithm 1). They present three concrete methods based on this approach and compare them with previous methods using a few benchmark datasets, which confirms the effectiveness of the approach.
main_review: # Overall evaluation
The clarity of the paper could be improved, but the core idea of the proposed approach is interesting and the experimental results look promising. I recommend accepting the paper provided that the authors address clarity issues.

# Originality
* As far as I understand, the problem setup can be viewed as multi-source unsupervised domain adaptation, and this itself is not new.
* The proposed approach of learning to adapt using a batch of unlabeled samples seems novel as far as I can see after seeing several papers.

# Quality
* The paper provides good discussions on related work.
* The technical part is clear except for the concrete algorithms.
* The general algorithm is natural and reasonable although the paper does not provide any theoretical guarantees.
* On the other hand, the concrete algorithms look quite heuristic and it would be nice if there were theoretical analysis.
* The experiments are well-designed and the results look convincing.

# Clarity
* Section 1 and Section 2 are somewhat too abstract and unclear to me. In fact, I could not understand the problem setup until I read Section 4. A reason may be the authors try to avoid technical terminology and give high-level arguments in the introduction part, which had unfortunately negative impact on my reading experience.
* For example, the authors use "groups", which is too general and ambiguous, for describing what I would call "domains".
* Also, the introduction does not seem to explain well what the proposed method really does.
* I suggest defining the problem setup a little bit more formally in early part of the paper so that readers can have clear understandings what the paper is talking about.
* The descriptions of the concrete methods are hard to understand without looking at the supplementary material.

# Significance
* The problem setup is slightly more restrictive than the standard unsupervised domain adaptation in that we need samples from multiple source-domains (groups).
* However, there may be many real-world applications indeed satisfying the requirement such as the examples that the authors present in the paper.
* The idea of incorporating information of the input distribution into adaptation is interesting and seems reasonable. It does put strong assumptions like covariate shift but seems to be solving more general problems.
* The experimental results look convincing and indeed show superiority of the proposed method.

# Other comments
* It would be nice to add discussions about previous work on multi-source unsupervised domain adaptation and zero-shot domain adaptation.


-----

# Update after the authors' response
I have read the other review comments and the authors' response. My opinion has not changed, and I would like to keep my score at 7. I hope the authors improve the clarity of the paper and the experiment part, as discussed in the rebuttal.
limitations_and_societal_impact: The paper does not mention societal impact although I have no suggestions on that.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes to modify MAML where the inner optimization problem is adapting to a different distribution in an unsupervised manner rather than finetuning on a new task. That is, the inner step must reduce loss but without the labels.
main_review: I find this work to be quite limited in novelty, and it seems like a very shallow application. Given the similarities between domain generalization, domain adaptation, and meta-learning/multi-task learning, it's not insightful to just apply the MAML objective, and so the contribution of this work seems to be trying out a few types of adaptation to find which ones work reasonably well.

In fact, I think this work is better described as "MAML when the test task samples are unlabeled" rather than a method for distribution shift. The proposed algorithm is _literally exactly MAML_, where the inner gradient steps are replaced with alternative adaptation techniques for when the labels are unavailable. **The fact that this work is basically just "how can we slightly modify MAML" is I think the primary factor for me, but below I include some additional discussion which is also relevant and I think essential to properly address.**

I feel the discussion of general "distribution shift" is poorly framed. While Blanchard et al. did indeed consider a prior over tasks, the current formulation of domain generalization is a much more difficult problem of generalizing in a _minimax_ sense, when the statistics across domains may have very little in common (further, note that that work provides actual theoretical guarantees, where this work provides none).

This paper heavily conflates "group shift" with general distribution shift, such as in the problem of domain generalization. In the title and abstract, the authors indicate that this method is meant for dealing with group shift---this is not the same thing as the typical domain generalization setup or distribution shift in general. This is a _very_ specific type of change at test time where the mixture components of a set of fixed sub-populations (all of which are observed at train time) changes. That's fine, but I think the introduction discussing domain generalization and general distribution shift is quite misleading because this paper addresses a very niche problem that's nowhere near as difficult as the problems discussed in the introduction. The experiments show that the algorithm is successful at unsupervised meta-learning, or adapting to new domains which share a _great deal of structure_ with the training data. The experiments do _not_ show that the algorithm is successful at generalization under general distribution shift.

As further evidence, and particularly damning, I note that in Appendix C the authors discuss the performance of ARM on the DomainBed benchmark. See the results here:
https://github.com/facebookresearch/DomainBed/blob/master/domainbed/results/2020_10_06_7df6f06/results.png
I find it quite disingenuous that the authors write: "ARM-CML handily outperforms ERM and all prior methods on the colored MNIST benchmark", as CMNIST is an extremely contrived and unrealistic dataset. Aside from the two MNIST-based datasets (the two easiest ones), on _all other datasets in the DomainBed benchmark_ ARM is nowhere near the top, and on average it is quite in the middle of the pack, failing to outperform ERM. **Cherry-picking results on a benchmark suite defeats the purpose, and gives me the impression that this comparison was made in bad faith.**

I think "MAML when the test task labels are unavailable" is an ok idea, lacking in novelty but not totally without merit. I also personally feel that group shift is not a realistic assumption for what to expect at test time without some a priori structural knowledge of the task we are trying to solve. Regardless, the experimental results demonstrate its ability to perform well when there is heavily shared structure among the domains (as might be expected, given the empirical success of MAML). But the algorithm does not work for more difficult distribution shift (and there is *no theoretical intuition or justification whatsoever*), so really this is just a slight tweak to an existing approach to meta-learning.

This paper would greatly benefit from an extensive reframing, and the results---positive _and_ negative---need to be honestly and frankly reported. 

--------------------
**Update after rebuttal**. The authors have indicated that they will fix the issues that I surfaced, an after reconsideration I consider this to be marginally above the publication threshold.

I think it is **essential** that these changes are actually implemented. This method works well, but only in *very* specific settings, with *much* stronger assumptions than previously made. As an example, the comparison in table 1 stratifies by assumption, but then still only bolds the top performing algorithm. This gives the casual reader the impression that ARM achieves SOTA, when in reality it is simply not being compared to any other algorithms with the same access to information at test time. This fact should be conveyed **much more clearly**, in addition to the other issues I found with the experimental reporting and general discussion of domain generalization and group shift.

**In fact, I think the algorithm suggested here is more akin to unsupervised domain adaptation than domain generalization**... maybe the comparisons should be updated to reflect this.
limitations_and_societal_impact: Sure
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a framework of adaptive risk minimization (ARM), which adapts to unlabeled instances at the test phase 
by pretraining with labeled instances obtained in many groups (domains or users). 
To adapt to unlabeled instances, the authors proposed three approaches (ARM-CML, ARM-BN, and ARM-LL).
The effectiveness of the proposed method was shown using real-world image datasets.
main_review: Pros:
- Problem formulation is interesting.
- Experiments with sequential arriving data are interesting.

Cons:
- Some related works are missing.
- Experiments can be improved.

Details:

This paper proposed an interesting and useful problem setting.
Unlike the standard meta-learning methods that require labeled instances as support set at the test phase, 
this approach requires unlabeled instances at the test phase to adapt to new groups.
Although the proposed method for adaptation is straightforward and simple, I like this research direction.
But, I think that there are some improvements that can be made.

First, about the experiment, how does the proposed method compare to the unsupervised domain adaptation, 
which require both labeled training data and unlabeled test data during training?
I think that this comparison gives us a better insight into the proposed approach.
In addition, this paper uses only image datasets in the experiments.
If the authors can show the effectiveness of the proposed method on datasets other than image datasets,
it will strengthen the contribution of the paper (although it is not always necessary).

Second, the problem setting of this paper is not completely new.
For example, the paper [1] proposes almost the same problem setting and a method similar to the ARM-CML.
This method uses only unlabeled instances to adapt to new target domains at the test time by learning labeled instances in multiple source domains.
For adaptation, the deepset network, which outputs a latent context vector from the set of unlabeled instances, was used.
I think it is necessary to discuss in more depth the differences between these methods and the advantages of the proposed method.

[1] Kumagai, Atsutoshi, and Tomoharu Iwata. "Zero-shot domain adaptation without domain semantic descriptors." arXiv preprint arXiv:1807.02927 (2018).

===after rebuttal===
Thank you for your answer. 
I think the additional experimental comparison will be useful. I would like to raise the score from 6 to 7, provided that the authors reflect these results in the revised version and discuss them in more depth.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper proposes to apply a meta-learning method (similar to MAML) to solve the domain generalization problem. After spending some time reading the paper and trying to understand the main idea, I agree with Reviewer dMsf, sJVX, and dgsw that there is a limited novelty in terms of both methodology and theory. Firstly, from the domain generalization perspective, the idea of learning models that can adapt to a collection of unlabeled data at test time has appeared as early as in [1, 2] and subsequent works in this direction. In these works, the models also take as input the empirical marginal distributions through the kernel function on them. Hence, the key distinction here is the use of deep learning trained in a meta-learning style as a way to equip the models with adaptation capability. Secondly, from the meta-learning perspective, the proposed idea is very similar to the classic MAML, as also pointed out by Reviewer dgsw. The key difference from the standard setting is the use of unlabeled data alone. Lastly, the authors seem to miss several important works in both domain generalization and meta-learning when discussing the related works in Section 3. I encourage the authors to properly conduct a literature review as this is an increasingly important area of research. In the final version, I encourage the authors to improve the clarity and to better highlight the key contributions of this work.

Nevertheless, there is a positive consensus among the expert reviewers that the experimental results are promising, and that the rebuttal has adequately addressed their concerns. The authors are willing to incorporate the reviewers' suggestions in improving the final version of this work. Hence, I recommend acceptance for publication at NeurIPS2021 as a poster.

Remark: Line 75-76: "Invariance methods would fare no better, as there is no feature space that can separate the ambiguous points and make them classifiable". This is indeed misleading. One can in fact separate the ambiguous points if the group/domain information is incorporated into the feature representation as has been commonly done in the literature. 

- [1] G. Blanchard, G. Lee, and C. Scott. Generalizing from several related classification tasks to a new unlabeled sample. In Advances in Neural Information Processing Systems 24, pages 2178–2186, 2011.

- [2] Muandet, K., Balduzzi, D., Schölkopf, B., Domain generalization via invariant feature representation. In: Proceedings of the 30th International Conference on Machine Learning (ICML13). pages 10–18, 2013. 
Decision: Accept (Poster)