Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a GMoE model for solving classification under domain shifts, specifically in the DG setting. The paper presents results on 8 benchmarks and shows promising performance. 
strength_and_weaknesses: Strengths

- I liked the idea of applying mixture of experts for solving the problem of DG. I resonate with the authors, that it is a bright direction for explorations in DG
- The experiments are thorough and the results appear promising and t

Weaknesses

My major concern with the paper is that some important recent papers or results/discussions from cited papers are missing. 

(a) For instance, why did the authors ignore the MIRO+SWAD numbers in Cha et al. 2022. It clearly and comprehensively outperforms the proposed method on all the datasets. 

(b) Author missed in important recent work by Sivaprasad et al. [A], which makes a similar observation that backbone plays a more crucial role in DG compared to tailored algorithms. They show that ERM with Inception Resnet backbone can achieve 89.11% accuracy on PACS (outperforming the proposed approach on the particular dataset). That clearly dilutes the first contribution of the paper. I suspect more similarities with that work as well in terms of distribution shifts etc. A discussion is warranted.

(c) Finally, were the experiments in Table 1 averaged over multiple runs?


[A] Sivaprasad et al. Reappraising Domain Generalization in Neural Networks, arXiv 2021



clarity,_quality,_novelty_and_reproducibility: The paper is ok to read. However, the theorems can be better connected with the text and experiments. Novelty is primarily on the GMoE experiments and efforts to connect them with some intuition. The works appears reproducible. 
summary_of_the_review: My primary concerns are on missing results and papers and would request the authors to respond to it in the rebuttal period.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper tackles the problem of domain generalization by proposing to focus on model architectures that are well aligned to the task of interest. The paper uses the definition of algorithmic alignment of Xu et al. 2020 which defines how easy it is to learn a function that can be decomposed into N subfunctions if your neural network can also decompose into similar subcomponents. First, the paper formally shows that models that align well with the causal correlations in the data will generalize well, while models that align well with the spurious correlations will not generalize. The authors propose a model named Generalizable Mixture-of-Experts (GMoE), similar to other mixture-of-experts Transformer models used primarily in NLP. The authors argue that domain generalization benefits from capturing and effectively combining diverse attributes and for this purpose, some kind of conditional statements are required. It is formally shown that such statements align well with the proposed model, thus the GMoE model should generalize well out-of-distribution. Experimentally, the GMoE is shown to obtain good results compared to recent methods.

strength_and_weaknesses: **Strengths**:

The paper tackles an important problem of domain generalization and proposes a good direction to approach it, by designing models that are well aligned with the desired task.

The formalization of alignment between models and the task is sound and useful for guiding the design of future architectures.

The proposed model seems to be beneficial for DG tasks. It is good to see that GMoE obtains good results with standard ERM training, but also benefits from DG algorithms. 

It is interesting to see that the experts seem to specialize, as seen by histograms and examples in Section 5.

**Weaknesses**

The paper did not explain in much detail why conditional statements are good for Domain Generalisation. As this point is crucial in linking the proposed method to DG, it should be better explained. At the high-level Theorem 2 says that the proposed model is good for cases where the true function is composed of similar mechanisms (selection + application of submodule) as the GMoE model (gating + separate FFN). But still, it should be explained in more detail why such functions are preferable for DG. Some intuitions are given in Section 4.2, but expanding them would be good.

The paper could discuss connections to other works that mixture-of-experts for domain adaptation [A], or use experts in similar ViT models [B].


[A] Guo et al. “Multi-Source Domain Adaptation with Mixture of Experts” EMNLP 2018 
[B] Rahaman et al. "Dynamic inference with neural interpreters." NeurIPS 2021.

clarity,_quality,_novelty_and_reproducibility: The paper is clear and has good quality. The proposed method is not original, it repurposes similar methods already used in the community but it applies them in a different setting (of domain generalization) and motives their good performance by the formalism of alignment.
summary_of_the_review: Overall the paper is well-written and has a good formalism that gives theoretical explanations of the good performance of the proposed method. The paper will benefit the community by showing that good architectures that align with the task are to be desired and show a good example of such architecture.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper start from an interesting empirical finding that a transformer trained with ERM outperforms CNN trained with domain generalization (DG) algorithms on DG task. They show some experiments and math to prove that. They propose a new GMoE and show its superiority on DG. 
strength_and_weaknesses: The paper focus on an interesting aspect and the paper are overall easy to understand. But some important experiments are missing.

1. Missing comparison between transformer with ERM and transformer with DG. The core motivation "empirical
risk minimization (ERM) outperform CNN-based models employing state-of-the-art (SOTA) DG algorithms" can just because the transformer is a stronger backbone.

2. Why is GMoE superior to a normal MoE. 
The major difference is that it forces a normalized operation on the input of the gating function and uses a learned embedding dictionary to decide the gating. The ablation study of why this is superior is missing from the paper.

3. The paper states that GMoE relies on visual attributes to have better performance. This is not convinceable from figure 3(b). It seems that e0 and e2 are specialized for background and others (e3,e4, e5) are all activated by almost all the visual attributes. Figure 3(c) is also not convincing to me. It seems no strong relation between experts and attributes. Any network would have similar visualization that some part of it prefers some region in the image.

Overall, the experiment parts seem to be a weakness: not enough experiments, ablations, and convincing visualization.
clarity,_quality,_novelty_and_reproducibility: See above.
summary_of_the_review: See above.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The submission considers the problem of improving domain generalization via NN architecture design. The authors propose to adopt an MoE ViT design. The former (MoE) is motivated by its ability to serve conditional computation for easier processing of high-level visual attributed, while the latter (ViT) is motivated by multi-head self-attention properties as low-pass filter with shape bias compared to texture-biased CNNs. Routing scheme (cosine routing) and number of MoE layers are empirically adapted for best generalization performance. Experiments demonstrate superior performance over explicit domain generalization algorithms using more conventional CNN-based architectures.
strength_and_weaknesses: As a clear strength, the paper provides detailed argumentation to motivate each of its design choices, going beyond a purely empirical study. Most prominently, it motivates the choice of ViT architecture by means of Theorem 1 and the choice of (G)MoE by means of Theorem 2. Both theorems provide a somewhat theoretical grounding of the architectural choices made. The empirical evaluation is extensive and convincing. A minor weakness is that further tweaks in Section 4.3 are made on purely empirical grounds and ablations are relegated to the appendix.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and hence easy to understand, although some of the definitions and theorems require careful reading. NB, I did not check the theorem proofs in detail.
Outside of the theoretical justification for the proposed architecture, none of the single architecture choices are novel per se - although in combination they attain sufficient novelty for the application to the domain generalization problem. 
Code is provided for reproducibility, although I am unable to check this in practice.
summary_of_the_review: The paper suggests well-grounded architecture choices to improve the SOTA in domain generalization using conditional computation via MoE.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This is a solid paper that investigates the impact of the backbone on domain generalization settings and proposes the Generalizable Mixture-of-Experts, a model build on vision transformed that addresses classification under domain shifts.
Results on 8 benchmarks show promising performance.
I personally find the idea very interesting and the paper well constructed, from analysis to method. I also believe it has a quite widespread applicability, potentially beyond vision tasks.
Decision: Accept: notable-top-5%