Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a benchmark for Federated Learning in the context of domain generalization. The authors thoroughly evaluate a number of popular DG algorithms on this benchmark. 
strength_and_weaknesses: Strengths
- The benchmark considers a relevant setting
- The evaluations are detailed covering a variety of methods
- The work highlights the difficulties in this setting

*Weakness*
- The intersection of DG and FL has been considered by several authors recently.. The  following related references and methods are not discussed or compared:
   - Tension et al “Gradient Masked Averaging for Federated Learning” 
   - Yuan et al “What do we mean by generalization in Federated Learning” 
- The wide array of approaches are appreciated but there are limited details on how these are adapted to the federated setting. Some of these algorithms e.g. MMD, FISH,  seem like they would require access to the different domains at each iteration thus it’s not clear to the reviewer how they are implemented here since clients should not share data. 
- Although the performance is evaluated it is not discussed if these approaches come with additional communication or training time overhead. FL is often concerned with these constraints. Related to the previous point are there additional information besides the models transmitted across clients in the various methods?
- The datasets are interesting however in addition to these it might be good to include at least one popular dataset from the FL literature such as FEMNIST (which has been studied in OOD settings in other work) or CIFAR 

clarity,_quality,_novelty_and_reproducibility: *Clarity/Quality*
The paper is overall reasonably well written. A clarity issue about how the DG methods are ported to FL is noted.
The reviewer also noted that the appendix is referenced 

*Reproducibility*
Code is available

*Novelty*
The benchmark is relevant and the evaluation is more extensive than previous work, but previous work in this topic does exist.

summary_of_the_review: The paper presents a useful benchmark and highlights the challenges of existing DG methods extended to FL in the context of this benchmark. The evaluation of some existing algorithms is of interest.  Some related work are not fully discussed. The reviewer also currently has some concerns about what data information is transmitted across clients and servers in the methods evaluated.

Update: The authors made  substantial changes to the paper text that have greatly clarified the setting under study, thus have largely addressed my concerns. I am thus increasing my score
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a benchmark for domain generalization (DG) algorithms in federated learning (FL). The authors considered more realistic settings with larger client numbers and different heterogeneity levels compared to prior works. The benchmarks included two new datasets for DG in FL and multiple existing DG algorithms. 

strength_and_weaknesses: Strengths

The first paper to benchmark DG in FL settings and considered different setups and difficulties in the benchmark. The list of algorithms included in the benchmark is also comprehensive.
The paper is written clearly and easy to follow.

Weaknesses

Only image datasets were considered in the benchmarks while there are many existing works for DG in the text domain [1,2,3]. Also, only two datasets were considered more realistic which seems to be small as a benchmark. 
Some parts of the results are not clear, e.g. why does $\lambda=0.1$ give better performance than $\lambda=1$ for some algorithms in Table 4?

References:

[1] Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. ACL 2020.

[2] Gururangan, S., Lewis, M., Holtzman, A., Smith, N. A., & Zettlemoyer, L. Demix layers: Disentangling domains for modular language modeling. arXiv 2021.

[3] Chronopoulou, A., Peters, M. E., & Dodge, J.  Efficient hierarchical domain adaptation for pretrained language models. arXiv 2021.

clarity,_quality,_novelty_and_reproducibility: The clarity and quality is good. This is also the first attempt to benchmark DG in FL. The benchmark should be reproducible as the authors will provide a link to the code and dataset.

summary_of_the_review: I am leaning towards weak reject as I think the benchmark needs to be more comprehensive as discussed in the weaknesses above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper is focused on benchmarking algorithms for domain generalization (DG) in the federated learning (FL). They evaluate 9 algorithms on three datasets (PACS, iWildCam and Py150). In particular, authors compare centralized DG algorithms and two FL algorithms for DG (FedADG and FedDG). The paper studies the ability of these methods to handle different challenges of DG-FL such as statistical heterogeneity among clients, number of clients, number of communication rounds. The results show more work is needed to design algorithms in the large client setting even for simple datasets as well as to handle local data statistical heterogeneity.
strength_and_weaknesses: Strengths:
-  Domain generalization (DG) in federated learning (FL) is an important problem and has not been a systematic comparison of the methods. This paper presents the first such study.
- The paper presents major challenges that need to be overcome to address DG-FL problems and shows the inability of the existing methods to do so. This is important work needed to advance the field.
- Using homogeneous and heterogeneous splits (as measured by lambda) presents a systematic way to measure how well the algorithm can adapt to heterogeneous  environments.

Weaknesses:
- For benchmark papers, it is extremely important that the paper is well written, well structured and reads well. However, this is not currently the case. There is a lot of information, but the paper does not organize information in a succinct and clear way making it hard to follow. For example, problem setup is never fully defined; part of experiments with varying lambda are in one paragraph, the other in another; it is not clear how centralized DG are extended to FL setting; many tables are not referred to in the paper; it is not explained why centralized DG methods can not be extended to be applicable to lambda=0 setting and so on. These are just a few examples, but the overall impression of the paper is that much more effort is needed to have a well organized and structured paper. 
- The methods are tested only on three benchmark dataset. More analysis is needed to make the comparison truly comprehensive
- Similarly, more recent FL algorithms should be included for the comprehensive analysis.

Minor: 
- In the paragraph Challenges from more realistic datasets, it should be Fig. 2 (c),
(d) not (b), (c)
- In paragraph 4.1 the authors write that they analyze two values of the lambda parameters, but the figure shows three.



clarity,_quality,_novelty_and_reproducibility: The paper presents the first systematic comparison of the DG-FL methods and as such its analysis is important and it brings new interesting insights. However, the clarity and quality of the paper needs to be significantly improved.
summary_of_the_review: Overall, the analysis presented in the paper is of high importance to the community and has the potential to bring new advances in the DG-FL by outlying unsolved challenges of the current methods. However, more work and effort is needed to make this paper strong and comprehensive which is necessary for these types of benchmarking papers.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new benchmark for federated domain generalization (FedDG) which explores more realistic challenges under the federated learning scenario. The benchmark designs a fair and diverse comparison on the data heterogeneity and communication costs. The experimental results show the challenges on the FedDG stay unsolved in the realistic experiment setting.
strength_and_weaknesses: Strengths:
1. The DG problem under FL is important and lack of discussion, the proposed benchmark can better promote the research of new algorithms.
2. The experimental settings for statistical heterogeneity challenge under FL is novel and practical.
3. Sufficient experimental results and reproducible code.

Weaknesses:
1. The compared domain generalization methods are old and not representative, which leads to an unconvincing conclusion of this paper.
2. There are lots of FL methods which also focus on the statistical heterogeneity problem and need to be included in the benchmark.
3. The description of proposed experimental setting is not intuitive, making it hard to understand and does not give a clear framework for comparison.

clarity,_quality,_novelty_and_reproducibility: The experimental settings for statistical heterogeneity challenge are novel, but the clarity and the organization of the presentation needs to be improved. This paper has open source the code, which seems to be reproducible.

summary_of_the_review: The advantages of this work are that the overall benchmark design is innovative and much needed by the academic community, but the disadvantage is that it does not use the recent SOTA method as a comparison, which is insufficient to support the seriousness and necessity of the DG problem under the FL scenario. The lack of comparison of methods from non-iid heterogeneous perspective in the FL field makes the conclusions of this paper less convincing. And the design of the experiment needs to be presented in a clearer and more structured way.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a benchmark for Domain Generalization (DG) in Federated Learning (FL). The settings considered by the authors takes into account large client numbers, different levels of heterogeneity and number of communication rounds. The experimental evaluation is done on 5 datasets (included 2 added after rebuttal) and compare 4 FL strategies with 6 classic methods in DG that can be considered as centralized algorithms  and include. One important conclusion of the work is that there is an important of work to do to address the problem of DG in FL and that this work contributes to provide a benchmark potentially useful for the community.  
  

Strengths:   
-the problem of DG in FL is interesting.  
-the benchmarking study can be useful the community.  
-a large covering is proposed.  
-the work illustrates some difficulties of the settings.  

  
Weaknesses:  
-the similarities and differences with classic domain generalization problems could be better discussed and analyzed. 
-the study contains many approaches but some recent methods are not covered, in particular the number of FL methods considered is limited in some sense  
-Some approach in FL related to meta-learning and multi-task learning are not considered in the study while potentially adapted to the context and could modify the analysis. More representative methods about DG and FL could be added.    
-the number of changes made in the paper were numerous but the final version did not contain all the results that were finally provided as additional comments. A general review final submission is then hard to make.    



Overall, the authors provided a strong effort to take into account the feedback of the reviewers and the majority of the reviewers were positively impressed by the work done.   
However, the number of FL methods considered is somewhat limited and in particular ones related to meta-learning and multi-task learning which are relevant for the problem and can be important to draw a general conclusion. The differences with general DG can be then more discussed. Since these elements are important for a benchmark paper that can be impactful for the community, I think that the contribution is still not ready and I propose rejection.  


Example of Meta-learning and multi-task learning in FL that can be considered:  
*Alireza Fallah, Aryan Mokhtari, Asuman E. Ozdaglar: Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach. NeurIPS 2020.  
https://arxiv.org/abs/2002.07948. 
*Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Laetitia Kameni, Richard Vidal: Federated Multi-Task Learning under a Mixture of Distributions. NeurIPS 2021.  
https://arxiv.org/abs/2108.10252. 

   
Other representative DG methods:   
*Jigen - "Domain generalization by solving jigsaw puzzles." CVPR 2019.  
*RSC - "Self-challenging improves cross-domain generalization" ECCV 2020.  
*FACT - "A Fourier-based framework for domain generalization" CVPR 2021.  
*SWAD - "Swad: Domain generalization by seeking flat minima." NeurIPS 2021.  

Other representative FL methods:   
*Scaffold - "Scaffold: Stochastic controlled averaging for federated learning" ICML 2020.  
*FedDyn - "Federated learning based on dynamic regularization" ICLR 2021.  
*HarmoFL - "Harmofl: Harmonizing local and global drifts in federated learning on heterogeneous medical images."AAAI 2022.  
*FedSAM - "Generalized federated learning via sharpness aware minimization." ICML 2022.  
*FedAlign - "Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning" CVPR 2022.  

justification_for_why_not_higher_score: The authors provided a strong effort to improve the paper with respect to reviewers' comments. 3 reviewers out of the 4 were positives for a weak accept after the rebuttal, even though there is a comment shared by 2 reviewers about the fact that some representative methods are missing, and one still thinks that the paper is not ready in particular due to presentation issues.  
 
During discussion, the following elements were raised:  
-interesting topic and contribution. 
-differences with classic Domain Generalization can be better discussed. 
-some approaches related to meta-learning and multi-task learning in FL are not considered and could affect the conclusions of the study.  

Based on these last remarks, I tend to propose reject. In addition, many modifications were made and the authors could not integrate the very last results in the document (they are provided as additional comments), but the contribution was very different with respect to the original submission and cannot be evaluated as a complete paper, which could be a bit unfair with respect to other submissions.
Decision: Reject