Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a Wasserstein robust Distributional optimization scheme to provide robust approach for empirical risk minimization. Generalizing the concepts of Agnostic Federated Learning, the method finds applicability in domain adaptation as well as big data settings. The authors also propose and SGD algorithm to solve the problem described and provides a theoretical upper bound for the resultant estimates.
main_review: The paper is well-written and the concepts are explained in technical detail. While the ideas implemented are not surprising, appearing previously in Wasserstein Adversarial learning algorithms, the authors do a decent job of explicating the details of the method. The SGD algorithm via usage of duality properties is standard and the theoretical justification is solid. Overall, I would say this work is a reasonable contribution. I have some comments about the paper. 

1. Can you show simple cases of inference where Wasserstein robust optimization does better or specific signal/noise types?


2. Suppose you have $n$ data points and you perform the WAFL method, does it give smaller number of components in terms of the maximizing argument (Q in this case). I feel that invoking Wasserstein robustness should provide a more interpretable classifier. Is that intuition justified? If so, is that confirmable through some experimental validation or otherwise?

3. Please provide the computational time comparison with relevant methods.

summary_of_the_review: Overall a solid piece of work. Results are not surprising but are reasonable.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a Wasserstein distributionally robust optimization (WDRO) framework for federated learning to hedge against statistical heterogeneity. The authors utilized the duality result in previous WDRO works to make their formulation tractable, and proposed a distributed algorithm to solve a relaxed WDRO problem. They conducted several numerical experiments on MNIST and CIFAR-10 to compare their method with FedAvg (non-robust algorithm), FedPGM and FedFGSM (robust algorithms), and demonstrated the advantage of their approach when the proportion of attacked clients is over 80%.
main_review: The problem under investigation is important. However, I do not feel this paper made sufficient novel contributions to either the field of WDRO or the field of FL. The duality result (5) used by this paper was not new. The formulation (4) is the canonical form of a WDRO problem, and thus I did not see the difficulty of solving it. 

One of my major concerns is in (6), where the authors did not solve \gamma to optimality, but instead considered it as a hyper-parameter. This would break strong duality, meaning that what Algorithm 1 solved is not the original WDRO problem (4), but just a relaxed version, and we do not know the tightness of such a relaxation. Can you explain the rationale of not solving \gamma to optimality?

Algorithm 1 does not seem to be very different from FedAvg, except that the original loss is replaced by the surrogate loss. 

Also, at the end of Section 2, the authors mentioned several related works exploring Wasserstein robustness in FL. It would be good to expand a bit on what is the difference between this work and those previous works, what makes your paper novel and what are your contributions. In the experimental section, it would be good to see whether your approach improves upon those past works.

The experiments do not seem to be convincing either. FedAvg itself is not a robust algorithm, and thus I do not think comparing with it gives any insights on the robustness of the proposed approach. The improvement of WAFL starts to be significant only when the proportion of attached clients is very high (~80%), which puts the benefits of using WAFL in doubt. 

Minor points: how did you choose \lambda_i? Did you set it to n_i/n?
summary_of_the_review: Overall I do not think this paper made sufficient novel contributions to the field of WDRO and FL. Several theoretical results are already well-known by the WDRO community, and the proposed algorithm is not very different from FedAvg. The numerical experiments also do not look very convincing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a Wasserstein-based distributionally robust federated learning (WAFL) framework to address the statistical heterogeneity problem, which is also applicable to distributional shift setting and domain adaptation. Inspired by (Sinha et al., 2020), the WAFL problem is reformulated into the dual form, and the Lagrangian ($\gamma$) is chosen as a fixed value reflecting the algorithm's conservatives. The paper develops a local SGD scheme to solve the problem and provides a theoretical convergence guarantee under a set of standard assumptions. In addition, the paper also provides a generalization bound for the proposed algorithm by resorting to standard techniques such as covering number, entropy integral, and empirical convergence rate of Wasserstein distances. Finally, the results of several numerical experiments are reported to justify the performance of the algorithm.
main_review: **Major commments:**

- The theoretical part of WAFL is systematic and well-written, with analysis from both optimization and statistics aspects.

- My biggest concern lies in the motivation/marginal contribution of WAFL against existing agnostic FL framework such as (Mohri et al., 2019); (Deng et al., 2020b). On page 4, the paper compares WAFL with counterparts in terms of uncertainty set, and conclude that Wasserstein ball is able to cover the uncertainty set of agnostic FL. However, a larger uncertainty ball may not be an advantage, it will also leads to over-conservative decisions. Therefore, a crucial question for the authors is: for what types of application can WAFL outperform existing method? If so, is there any theoretical or empirical evidence? 

- The paper didn't report a systematic way to choose $\gamma$ in the numerical experiment part, the choices of $\gamma = 0.5$ for MNIST and $\gamma = 10$ for CIFAR-10 are not well-supported. 

- For the numerical part, it would be better to include (Mohri et al., 2019) or (Deng et al., 2020b) as benchmark as well. This would be helpful to clarify my first point.

**Minor Comments:**

- It is claimed on page 3 that ``In FL, some existing works have explored the Wasserstein distance to enhance robustness (Rei- sizadeh et al., 2020; Diamandis et al., 2021; Du et al., 2020; Deng et al., 2020b).`` Though Du et al., 2020 and Deng et al., 2020b are examples of distributional robust FL, I can not find their connection to Wasserstein distance. Did I miss something here?
summary_of_the_review: In general, the paper is well-written. It provides a comprehensive study of the proposed framework. However, in my humble view, the motivation of the paper is not well-supported, and the comparison against existing methods is insufficient. Therefore, I rate this paper as a marginal paper.
review_title: null

Meta-review: The main motivation of this work is to introduce robustness in federated learning, through a Wasserstein uncertainty set. The end result, however, leaves a mixed feeling: As the reviewers pointed out, the authors, perhaps for computational convenience, forgo strong duality and treat the important variable gamma as a hyperparameter, which renders large part of the work follow immediately from existing work: essentially, we simply use a different loss function in FedAvg. While there may be advantages to choose one loss over another in any specific application, this itself is not a significant contribution. The comparison against existing FL algorithms is also a bit weak: Despite of the reviewer's request, the authors did not compare to other robust FL algorithms (e.g., AFL), thus it is not clear what is the real advantage of the proposed algorithm. As a result, we believe the current draft is not ready for publication.
Decision: Reject