Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##########################################################################
Summary:

The paper proposes a new approach for domain generalization which minimizes the generalization error across train-validation split with the largest domain gap. The paper further gives theoretical bound on the generalization error of the proposed method.

##########################################################################
Pros:

The paper proposes a new approach for domain generalization. The idea of perform meta-learning from the train to validation split is already employed by previous works (Balaji et al., 2018; Li et al., 2019b; 2018a; Dou et al., 2019). The novelty of the paper falls mostly on performing meta-learning on the train-validation split with the largest domain gap. 

The paper gives a theoretical bound on the generalization error to unseen target domain.

##########################################################################
Cons:

The paper needs to update the model parameter, the initialized parameters and the train-validation split, which may not converge or converges very slowly. Though empirical results show that the convergence of the method is fast, some theoretical demonstrations are needed for the convergence speed of different updates.

The contribution of the paper is a little incremental. Meta-learning based domain generalization methods are not novel. The only novelty of the paper is performing meta-learning on the train-validation split with the largest domain gap, which is an incremental contribution over meta-learning based domain generalization (demonstrated by the ablation study).

As shown in Table 1, Baseline w/L2 outperforms Baseline by 2.3% but DFAS outperforms Baseline w/L2 only by 2.7%. This means that the main performance gain is from L2 while the other contributions has very small performance gain (less than 1%). However, L2-normalization (Finn et al., 2017; Dou et al., 2019) is a widely-used techniques for meta-learning and domain generalization, which is not counted as a novel contribution for the paper. The authors need to further demonstrate that the main contribution: meta-learning on the train-validation split with the largest domain gap, has huge performance gain.

##########################################################################
I lean to rejecting the paper since the performance gain is mostly falls on the L2-normalization but the main contribution of the paper: performing meta-learning across the train-val split with the largest domain gap, does not have much performance gain.
review_title: Review for the paper

final_score: rating: 5: Marginally below acceptance threshold
review_content: $Paper$ $summary$

This paper focuses on domain generalization, targeting the challenging scenario where the training set might not include different sources; even under the presence of different sources, the problem formulation does not takes into account domain labels. The proposed solution is based on meta-learning, following the path drawn by Li et al. AAAI 2018; the Authors propose to adversarially split the training set in meta-train and meta-validation sets, and then update the current model in a direction that fosters good generalization performance on the meta-test. Results on standard benchmarks are encouraging.

$Pros$

- The proposed idea is rather interesting, enabling to apply meta-learning solutions also in absence of domain labels. In particular, I like the idea of finding meta-train and meta-test splits in an adversarial fashion. This is crucial, since randomly splitting the training set in meta-train and meta-validation would not be helpful, since it would lead to episodes where meta-train and meta-test are iid.

- The Authors provide a theoretical interpretation of their approach (due to my background, I was not able to properly review it though)

$Cons$

- My main concern is related to the way the maximization problem is tackled, in the objective in Eq. (5). I have reviewed Appendix C, but I cannot understand how convergence would require so few iterations. Even restricting the size of $S_v$ as mentioned in Section 3., the number of possible sample combinations that generate couples ($S_v$, $S_t$) is huge -- if I understood the process correctly, then with $|S_v|=K$ and $|S|=N$, the number of combinations is $\binom{N}{K}$ -- and for each of them the gradients that lead to the meta-update are different. Could the authors comment on this? Am I missing something? Related to this point, I am also concerned by the ablation study in Table 5 - where it is shown that, while helpful, the adversarial strategy does not help very significantly in the whole picture.

- The overall writing could be improved. Sentences like "this model can be further transfored to a minimax problem" in the Abstract are not properly exposed, and there are several examples throughout the manuscript. There is a misconception related to prior work: Carlucci et al. 2019 (as well as Volpi et al. 2018) also tackles the case where the training data only comprises a single source domain. This should be clarified in the Introduction/Related work. 

$Review$ $summary$

I like the idea this paper starts from, and I like the proposed solution. I still do not properly understand how the maximization problem at the core of the method is approached, and I believe that the paper needs some exhaustive proof checking to improve the overall writing. I look forward reading the Author response and iterating the discussion.

---- Post-rebuttal comments----

I thank the Authors for their explanations. Yet, I still believe that this work is not ready for publication. Random splitting and adversarial splitting perform very comparably (1% is not a lot), in my opinion casting some doubts on how meaningful the solution found to the proposed optimization problem is. The Authors included a toy example in the Appendix, but this did not mitigate my concerns on the original manuscript's experiments. I still believe that the core idea is very interesting, and hope that it will be further investigated by the Authors for a subsequent submission.  
review_title: Interesting idea, but some points should be addressed

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper provides a novel way to combine meta-learning and adversarial training for domain generalisation. Different from existing methods, the authors propose to split the training dataset into train/val subsets in an iteratively adversarial way, regardless of domain labels, by which the model can be trained to learn to generalise well from training subset to val subset via meta-learning in each iteration.

Reason for score:

Overall, I vote for accepting. It’s ingenious that the paper proposes to ignore domain labels to enhance the learned model’s generalization ability towards domain shift problems. The major concern of mine is that the paper provides limited explanation of how the meta-learning algorithm, Model-Agnostic Meta-Learning (MAML), is utilized in the training process(see detailed in cons).

1. The paper proposes to discard domain labels when training a model to generalise well, by which the original training set containing several domains can be seen as a large labelless domain and then splitted into training/val subsets to simulate domain shift, hence MAML can be used here for the training process.

2. The authors incorporate the min-max optimization method following adversarial training to let the model be more effective in learning domain shifts between training/val subsets.

3. One of the paper’s contributions is that it surprisingly find that L2 -normalization can help mitigate gradient explosion in the MAML algorithm.

4. The paper provides comprehensive experiments of different domain shift settings as well as theoretical proofs to evaluate the proposed method, which are quite solid and convincing.

Cons:

1. It would be better if the paper provides more details in ablation study, for example, it mentions that DFAS-3 finds the hardest val-subset only based on loss by setting \alpha = 0 in Eqn. (5), there can be more analysis about the hyper-parameter of α .
review_title: submission 784 review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes to unify adversarial training and meta-learning in domain-free generalization where labels of source domains are unavailable. To maximize the domain shift between the subsets of meta-train and meta-val, adversarial training is leveraged to find the worst-case train/val splits. Extensive experiments on benchmark datasets under different settings demonstrate the effectiveness of the proposed method. 

Pros:
+ The idea of adversarial train/val splitting in meta-learning is interesting.

+ The paper provides extensive experiments on benchmark datasets under different settings with multiple/single source domains and achieves state-of-the-art results on both PACS and Office-Home datasets.

+ The paper provides an upper bound of the generalization error on unseen target domains, which is implicitly minimized by the proposed method.

Major Cons:
- The effectiveness of adversarial train/val splitting is not well justified. Typically, meta-learning based methods (MASF and MetaReg) have no overlap between domains used for meta-train and meta-val. This is how the domain discrepancy between meta-train and meta-val can be modeled. However, DFAS has the issue that the same domain may be used in both meta-train and meta-val, which significantly limits the domain transportation and may yield limited domain generalization capacity in comparison with MASF and MetaReg. So why DFAS outperforms MASF and MetaReg in terms of domain generalization? More experiments and discussion are suggested to justify this point. 

- The difference between adversarial splitting and domain-label-based splitting is unclear.  Additional experiments are suggested to empirically show the difference.

- The proposed adversarial train/val splitting is computationally expensive since it needs to rank all samples in each iteration. According to the ablation results on PACS (Tab.5), improvements over randomly selected train/val splitting seem quite limited. 
More details about the convergence of alternative iteration should be provided. 

- Fig.4 shows the objective function remains unchanged after the 3rd iteration. Will the splitting results be changed after this stage?

Minor Cons:
- It would be better to see an ablation study on the effect of margin m in Eq.6.

- Standard deviation of the reported results should be provided.

After rebuttal:
It is highly appreciated that the authors provide additional evidence in response to my reviews. My previous concerns about effectiveness and efficiency are addressed to some extent, however, this also means the original submission needs significant modification to address the concerns. I like the idea in general and the problem is well-motivated but it needs more work for a complete version. I would encourage the authors to further improve the quality of the paper.


review_title: Missing key justification with major and minor issues

Meta-review: The paper is proposing a domain generalization method based on the intuition that an invariant model would work for any split of train/val. Hence, the method uses adversarial train/val splits during training. The paper is reviewed by three expert reviews and none of them championed the paper to be accepted. I carefully checked the reviews and the authors' response and agree with the reviewers. Specifically:

- R#1: Argues that the paper is not ready for publication. Also argues the optimization problem is only a motivation as it is not directly solved. This is an important issue and it needs to be addressed in a conclusive manner.
- R#2: Argues empirical studies do not show the value of train/val splitting. I partially disagree with this issue but it is clear that more qualitative and quantitative study is needed to properly justify the proposed method.
- R#3: Argues the contribution is not enough for publication. The paper is clearly novel but the contribution and novelty is not presented in a clear manner. Moreover, the empirical study does not complement the novelty. Hence, I disagree with the comment.

Overall, I believe the paper proposes an interesting idea. However, the presentation and empirical studies need to be improved significantly. I recommend authors to address these issues and submit to the next conference.


Decision: Reject