Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper the authors introduce an algorithm to solve the Distributionally Robust Optimization problem in an asynchronous and distributed fashion where the data has been spread across multiple workers.  
Specifically, they focus on a DRO problem where the distribution has finite support (number of workers) and formulate it as a single master problem with finite number of constraints. They then use an augmented lagrangian algorithm to solve the problem where each worker updates the components of the augmented lagrangian using the data available to it. 
The provide a convergence analysis for their method and illustrate their results through numerical experiments. 
strengths_and_weaknesses: Strengths
1. The provide a method for solving DRO that can leverage the parallelism present in modern computer systems. 
2. Their assumptions and numerical experiments allow their method to be applicable to a wide variety of settings including popular machine learning methods. 
3. They provide a convergence analysis which shows that their method will converge to a stationary point. 

Weaknesses
1. The uncertainty set used is quite common and the analysis using it is pretty standard. 
questions: 1. What is the novel about the CD-norm uncertainty set. It seems essentially the same as the regular budget uncertainty set. The probability constraints sum p_i = 1, don't seem to have any impact on the analysis. 
2. I think the authors to better emphasize the asynchronous aspect of their algorithm. I believe this is hte novel aspect of their method but is not emphasized especially in the numerical methods. 
limitations: I think the authors can better address the limitations of their work especially the limitations imposed by the  the specific budget uncertainty set that they use. It would also be interesting to see if there are any differences in the type of solution obtained by there method as compared to other methods.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose three challenges that existing DRO techniques face. To this end, they propose an asynchronous distributed algorithm to tackle the distributed distributionally robust optimization problem. Moreover, they propose a new uncertainty set which is developed to effectively leverage the prior distribution and flexibly control the degree of robustness. They provide theoretical guatantees for convergence of the algorithm. Extensive empirical studies corroborate their theoretical results.
strengths_and_weaknesses: Strength:  
1. They propose an algorithm with theoretical guarantees.
2. They conduct experiments to verify their statement. 

Weakness:  
They make  numerous assumptions in the theoretical analysis, but did not explain whether these assumptions are practical.

questions: 1. Can the authors elaborate more on the assumptions in the theoretical analysis?
2. Can the authors explain to what aspects the proposed algorithm address the challenge stated in line 6?
limitations: 1. The authors did not explain whether the assumptions made in theoretical analysis is practical.
2. To my knowledge, I do not understand why the proposed algorithm address the challenge stated in line 6. Can the authors explain more clearly?
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed a distributed distributionally robust optimization method. Both theoretical convergence rate and empirical results are provided. 

However, the convergence rate is much worse than existing methods. 
strengths_and_weaknesses: Pros:
- This paper is well organized. 
- Provide extensive experimental results. 

Cons:
- In recent years, many optimization algorithms for nonconvex problems have been proposed and they can achieve a much faster convergence rate than this method, e.g. [1] [2].  However, the authors failed to discuss and compare them. 
- It would be good if the authors could use a table to compare the convergence rate of this method and existing methods, including the centralized ones. 
- As for asynchronous optimization, a critical hyperparameter is the communication delay, which heavily affects the convergence rate. But in your convergence rate, it is not clear how the communication delay affects it.
- It would be good if the authors could discuss the challenges in the theoretical analysis compared with the centralized method. 
- What benefits to introducing $h$? Is it always nonnegative? In Table 1, what loss values did you report? $h$ or $\sum p_j f(w_j)$? Similarly, in figure 2, if it is $\sum p_j f(w_j)$, which is a min-max problem, why is it always decreasing?
- In table 1, FedAvg and DRFA-Prox perform periodic communication. But your method performs communication in each iteration. Hence, the comparison is not fair. 
 


[1] An online method for a class of distributionally robust optimization with non-convex objectives

[2] Learning Distributionally Robust Models at Scale via Composite Optimization
questions: - In recent years, many optimization algorithms for nonconvex problems have been proposed and they can achieve a much faster convergence rate than this method, e.g. [1] [2].  However, the authors failed to discuss and compare them. 
- It would be good if the authors could use a table to compare the convergence rate of this method and existing methods, including the centralized ones. 
- As for asynchronous optimization, a critical hyperparameter is the communication delay, which heavily affects the convergence rate. But in your convergence rate, it is not clear how the communication delay affects it.
- It would be good if the authors could discuss the challenges in the theoretical analysis compared with the centralized method. 
- What benefits to introducing $h$? Is it always nonnegative? In Table 1, what loss values did you report? $h$ or $\sum p_j f(w_j)$? Similarly, in figure 2, if it is $\sum p_j f(w_j)$, which is a min-max problem, why is it always decreasing?
- In table 1, FedAvg and DRFA-Prox perform periodic communication. But your method performs communication in each iteration. Hence, the comparison is not fair. 
 


[1] An online method for a class of distributionally robust optimization with non-convex objectives

[2] Learning Distributionally Robust Models at Scale via Composite Optimization
limitations: NA
review_title: null

Meta-review: metareview: Reviewers generally recommend (weak) acceptance however there are still quite a few issues that should be addressed. I concur.
confidence: Less certain
Decision: Accept