Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies domain adaptation under the assumption that only unlabeled target data is available in training and the domain shift follows a special group shift. The main idea for the proposed method is having an adaptation model that takes only the unlabeled data in and output updated parameters. The proposed method also involves the test-time training, which means the adaptation model takes in unlabeled training data in training but takes in unlabeled target data in the adaptation phase. The method is called adaptive risk minimization and there are two meta-learning approaches provided, contextual and gradient-based, in the paper. In the experiment, the proposed method outperforms a limited set of baselines. The paper also discusses a few cases when the assumption is violated like the group indicators are unknown. 
 
Strong points
1. This paper studies an important setting in domain adaptation, similar to unsupervised multisource domain adaptation.
2. The idea of using an adaptation component trained in a meta-learning fashion with test-time adaptation is novel.
 
Weak points (maybe due to my confusion)
 
 
1. Group shift definition is not very intuitive to me, especially how it would affect the proposed method. It seems to be just a different name for “domain”. In the DRO setting [Hu 2018], the group information is used in optimization. However, in this paper, the group information is only used in sampling multiple group data for training and sampling single group data to train ARM-BN. Therefore, the so-called group shift in this paper seems to be a domain shift when there are multiple source domains. 

2. It is not clear to me if the test data is not one of the group, or is not the same distribution with one of the training distribution, how the model can be “adaptive”. Before the adaptation step, the model only sees unlabeled training data. In the experiment, especially for rotation mnist, it is not clear whether the test is covered in the training. I think a more principled way to do multiple random separations of rotations/corruptions -- making sure there is no overlap, and then evaluate on the test data.

3. The general setting is very similar to unsupervised multi-source domain adaptation. The difference between domain adaptation to a single fixed target domain with only unlabeled data available with this so-called adaptation to shift setting is very subtle to me. The way to use unlabeled target data for adaptation in the former perspective is very rich, including MMD, importance weighting, adversarial training. More should be discussed in the paper. And many should serve as baselines.
 
Given the weak points, I recommend weak rejection for this paper.
 
Additional questions and suggestions:
1. Instead of the group shift assumption, I think a more interesting question is: what is the assumption that will make the proposed method to work? When the model claims to adapt to a new test group that is not in the training, usually you assume the group/domain does not change the label. This seems to be the case from the data used in the paper. Another question is whether y|x is the same between training and testing, this seems to be also the case. So for meta-learning and test time training methods, clarifying their assumption is better than just casting a group shift setting to it.

2. The two meta-learning methods, contextual and gradient-based, are not new. I feel they should be discussed more carefully. The novelty of this method is the incorporation of meta-learning-style training and test time adaptation for unlabeled data.

review_title: Official Blind Review #2

final_score: rating: 7: Good paper, accept
review_content: The authors propose adaptive risk minimization (ARM) framework to address the problem of distribution shift using meta-learning approaches. 

The main contributions of this work are as follows:

1.	The authors propose ARM by employing meta-learning approaches to solve distribution shift problem.

2.	The authors have done extensive experiments to evaluate their proposed ARM algorithms. The results show that ARM outperforms multiple STOA baselines, including ERM, UW, DRNN, BN, TTT.

I still have a few key concerns below that prevent me from giving an acceptance.

Firstly, the overall contribution seems incremental to meta-learning approaches. The authors highlighted the difference between ARM and ERM as the constraint in eq.(1). However, eq.(1) is pretty much the same as meta-learning objective, with (i) the (task) batch loss is written as a sampling average rather than an expectation; and (ii) the adaptation is done using only x, without labels (y’s). As a result, this works is applying meta-learning paradigm to solve distribution shift problem. The proposed implementations ARM-CML and ARM-LL are black-box and optimization-based meta-learning, respectively. It would be great if the authors could explain in more details of the technical contributions, if I missed anything here. 

Second, in Figure 3, the results show that ARM outperforms ERM with different data samples/shots. However, the results were obtained on rotated MNIST, which is a fairly simple task, with the accuracy roughly 96.7\%. It would be nice to show results on more complex tasks, such as Tiny ImageNet-C. Would it be the case that more shots/samples lead to higher accuracy?

________________________

Thanks! The authors' update was well-received. I like the way how ARM formulates the unlabeled adaptation. Though "generalizing" meta-learning from labeled to unlabeled adaptation is natural and simple, ARM (proposed in this work) is crucial to highlight this and present the results to the community. I increased the rating to 7.

review_title: The authors propose adaptive risk minimization (ARM) framework to address the problem of distribution. It is a nice work. A key concern is that the proposed ARM sounds incremental to Meta-Learning approaches.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Paper summary:

The authors try to tackle the *distribution shift* problem with a meta learning approach. The algorithm, namely ARM, is proposed. Following regular meta learning regime, ARM uses an updated version of parameter $\theta'$ to calculate the loss for back propagation. Several specific implementations are put forward, i.e. the contextual / gradient-based methods. Experiments are performed in small and large scale datasets to demonstrate the effectiveness of the proposed algorithm. Detailed ablation study and qualitative analysis are also conducted.

Comments:

This paper has clear writing and well established methodology. However, the main drawback to me is the lack of novelty. Out-of-distribution generalization and meta learning are both popular areas in the current machine learning community, but a simple combination might not be able to convince me.

A second concern about ARM method is the reliability of the empirical experiments. The [DomainBed](https://github.com/facebookresearch/DomainBed) is a suite which could conduct systematical evaluation for domain shift algorithms. In its [public results](https://github.com/facebookresearch/DomainBed/blob/master/domainbed/results/2020_10_06_7df6f06/results.tex), ARM's performance is consistently worse than ERM (like many other algorithms who claim to improve on OoD setting), in all three validation method (training-domain validation, leave one out, oracle). I am familiar with DomainBed's implementation, so for me this difference indicates that the authors might pick the hyper-parameters and datasets based on testsets, as DomainBed is different from regular evaluation for its random parameter picking and extensive datasets searching. If authors could resolve this concern properly, I will consider raising my score.
review_title: Two main concerns to be addressed.

Meta-review: Dear Authors,

Thank you very much for your detailed feedback to the reviewers in the rebuttal phase. The feedback certainly clarified some of the concerns raised by the reviewers and improved their understanding of your work. Indeed, some of the reviewers have increased their scores.

However, overall, we think this paper has rather marginal novelty and there are still several conceptual and technical issues to be further discussed, such as the definition of the grouping concept and the distributional shift assumption.

For these reasons, I suggest rejection of this paper, in comparison with many other strong submissions. I hope that the detailed feedback and additional comments from the reviewers help you improve this work for future publication.

Decision: Reject