Distributed Saddle-Point Problems Under Data Similarity

Abstract: Aleksandr Beznosikov, Gesualdo Scutari, Alexander Rogozin, Alexander Gasnikov

Reviews and author responses:

This paper studies distributed algorithms for convex-concave saddle point problem under both centralized and decentralized setting. Under similarity assumption, the proposed algorithms with nearly matching the lower bounds over either types of networks.
  This paper studies distributed convex-concave saddle point problem for both centralized and decentralized settings. The proposed algorithms are based on extragradient iterations. updates, which contains time-varying communication topologies and federated learning.
The theoretical analysis show the proposed algorithm has near optimal communication complexities.

My main consideration is the theoretical results of this paper is based on $\delta$-related assumption. However, the real distributed system usually needs to address the heterogeneous data setting, especial in decentralized case. The assumption of $\delta$-related looks too strong. I hope the author could explain why it is reasonable and provide more detailed discussion.

There are also some comments for the experiments
1. It prefers to provide some reference for robust linear regression model. If $||w||$ tends to infinity, can we guarantee problem (10)  is concave in $\rho$?
2. Explain how to partition data for dataset from LIBSVM.
3. It is necessary to provide some experiments for decentralized setting.

Additionally, it is prefer to append a conclusion section after experiment part.

---------

Thanks for the author response. I decided to raise my rating.

 This paper focuses on the theory of distributed optimization. It is unnecessary to address the societal impact.

 Thank you for your quick response to our comment!

It seemed to us that the dependence on $y$ would be sufficient, especially since this occurs only in the logarithmic factor. But we can accomodate the Referee’s request as follows. One can find a relationship (from optimality) between $y^*$ and $x^*$: $\frac{\delta(1 - \mu)}{2} A y^* + 32\mu x^* = 0$. From this, one can establish the following relationship:: $||x^*||^2 = \beta ||y^*||^2$, where $\beta$ Is  some constant. Since $x^0 = 0$ and $y^0 = 0$, we can get $||x^0 - x^*||^2 = \beta ||y^0 - y^*||^2$. Then we can change $||y^0 - y^*||^2$ in our original result as follows: $||y^ 0 - y^*||^2 = \frac{1}{2} || y^0 - y^*||^2 + \frac{1}{2\beta} ||x^0 - x^*||^2$.


 1. Sorry about the typo I made. I mean why the right hand side is not in terms of $\Vert x_0-x^\star \Vert ^2+\Vert y_0-y^\star \Vert^2$, i.e. $\Vert z_0-z^\star \Vert^2$. The lower bound does not characterize the dependency on the initial distance of $x$. 

 We thank Reviewer PG9W for review, time and the insightful comments, which will help to improve our work.

We have tried to answer most of the questions and comments of all reviewers in our General comments/answers. Please read them first. Next is the answer to your review.

1)  We relegated the study of the convex-concave case to the appendix because of lack of space in the main paper.   If eventually we  have additional space, we will be happy to add it to the main part. We agree with the Referee that this result is  important.

2) Thanks for the thoughtful comment. Indeed, we did not manage to use directly the Bregman gradient method as a building block; we conjecture that this is likely due to the fact that gradient descent is not optimal for saddles;  we need to use instead extragradient method. Therefore, the essence of our idea lies in somehow cleverly combining statistically preconditioned Bregman and extragradient, therefore we need step 3, it also plays the role of an extragradient step in a sense.

3) Thanks for the suggestion. We will make the change in the revised paper. 

4) We added a short section of conclusions - see point 5 in our General answer

5) Thanks! Fixed!

6)  We will add this info in the revised paper. Strong convexity parameter mu was taken equal to regularization coefficient 0.1; delta was equal 0.01 (computed using the condition number of the second derivative matrices on the nodes)

7) Perhaps we do not quite understand the question, but we compare our method with the distributed extragraded method, which is optimal for general saddle point problems. Moreover, we added experiments with Local Descent Ascent method - see point 4 of the General answer

8) Indeed, we agree with you about the decentralized method. We briefly discussed the limitation of Algorithm 2 right after the algorithm. We will stress this aspect also in the conclusions we will add - see reply to point 5 above.



 We thank Reviewer Crdz for review, time and the insightful comments, which will help to improve our work.

We have tried to answer most of the questions and comments of all reviewers in our General comments/answers. Please read them first. Next is the answer to your review.

First of all, we want to discuss $\delta$-related assumption. We hopefully addressed satisfactorily comment about $\delta$-related assumption in points 1-3 of our general reply. Please let us know if more details are needed.

Referring to the specific questions 1-3, our reply is the following:

1) If you are talking about page 4, there is a typo: Both regularizers must be squared. On page 8, there is no such typo in expression (10). Thanks for noticing, we will  correct these typos.

2) The dataset was divided uniformly among all agents. It is this case we can provide statistical similarity between devices - see for example 

*Joel A. Tropp. An introduction to matrix concentration inequalities*

or (15) in 

*Hadrien Hendrikx, Lin Xiao, Sebastien Bubeck, Francis Bach, Laurent Massoulie. Statistically Preconditioned Accelerated Gradient Method for Distributed Optimization*

3) We made decentralized experiments - see point 4 in our General answer


 We thank Reviewer ngA2 for review, time and the insightful comments, which will help to improve our work.

We have tried to answer most of the questions and comments of all reviewers in our General comments/answers. Please read them first. Next is the answer to your review.

For clarity, we added some intuition of the methods as well as sketches of proof in the main part of the paper - see point 6 in our General reply.

Referring to the specific comments and questions, our reply is the following:

1) Thanks for the comment. We apologize if we missed the definition of $y*$, which indeed is  a solution to the problem. Note that the worst-case function used to determine the lower bound is such that  $y* \neq 0$. Referring to the second part of the question, we might have misunderstood what the Referee is asking: we do not quite see why the right-hand side should depend on $x^t $ and $y^t$. If the Referee instead meant the left side, then he/she is right, it depends on $x^t $ and $y^t$, because $z$ is a block vector of $x$ and $y$ - see the notation part.

2) Algorithms 1 and 2 are new schemes of this work and call for a new analysis. This was also noted by Reviewer PG9W: “It looks different from the classical statistically preconditioned Bregman gradient method for strongly convex problems.” Indeed, for saddle point problems we cannot use the same methods as for minimization problems. Therefore, we have constructed a new method that is adapted specifically for saddles. We also partially discussed the ideas of this method in point 5 of the General Answer. 

The Referee is right saying that there are previous results providing lower bounds for saddle points problems or distributed optimization, namely:

*1) Junyu Zhang, Mingyi Hong, and Shuzhong Zhang. On lower iteration complexity bounds for the saddle point problems,*

*2) Yossi Arjevani and Ohad Shamir. Communication complexity of distributed convex learning and optimization,*

*3) Kevin Scaman, Francis Bach, Sébastien Bubeck, Yin Tat Lee, and Laurent Massoulié. Optimal algorithms for smooth and strongly convex distributed optimization in networks.*

However, none of the above is applicable to our SPP. In fact, Ref.1 above focuses on SPP in the non-distributed setting. The consequence is that the lower bounds there do not capture data-similarity (and thus not informative in this regard) and the algorithm achieving the lower bounds are not not implementable in the distributed setting. To establish  our lower bounds we also considered bilinear problem as in Ref. 1, but  we had to find  a different   matrix A defining the  bilinear function, suitable for the distributed setting considered in our paper--this led to a  different problem than that in Ref. 1, calling for a different analysis. Note that the functions determining the lower bound in our setting belong to a different class than those in Ref. 1, because similarity needs to be explicitly accounted.  

Ref. 2 determines lower complexity bounds of minimization problems under similarity in the centralized setting--they neither consider SPPs nor distributed algorithms. Finally, Ref. 3 focused on minimization problems (non in saddle form) over networks without accounting for data-similarity. Compared to Refs 2 and 3, we had to consider a different class of problems, so we must have a completely different analysis. 


minors:

1. Thanks, fixed
2. Thanks, you are right! In main part we do not need compactness, we only need it for convex-concave case. We will  fix it!


 We thank Reviewer 6jmq for review, time and the insightful comments, which will help to improve our work.

We have tried to answer most of the questions and comments of all reviewers in our General comments/answers. Please read them first. Next is the answer to your review.

Answer for CONs from your review:

1) Please see our General comments (points 1-3) above for an explanation of the significance of the results of this paper.

2) We added - see point 6 in our General comment 


 We thank all the Reviewers for the insightful comments, which will help to improve our work. 

We begin  providing some general comments addressing common questions/recommendations from all the Reviewers along with a list of the major changes that will be implemented in the revised paper. Replies to specific questions of each Referee will follow under their comments. 


**1) Significance of the results:** Communications are the bottleneck in most distributed systems (in many cases, communication time takes 50-90% of the total algorithm time). Exploiting data similarity in the algorithmic design can bring a significant communication saving, as it is proved to reduce significantly the overall number of communications with respect to algorithms that do not do so (and use the same information). For such algorithms, very often the number of communications is $O(1)$. This advantage of the proposed approach  was acknowledged by **Reviewers 6jmq and PG9W** as well.

**2) On the importance of statistical similarity:** Developing solution methods for minimization problems (not in the saddle point form)  over star networks exploiting data similarity has received significant attention in the literature; examples include

*Ohad Shamir, Nati Srebro, Tong Zhang. Communication-Efficient Distributed Optimization using an Approximate Newton-type Method - ICML 2014*

*Yuchen Zhang, Xiao Lin. DiSCO: Distributed Optimization for Self-Concordant Empirical Loss - ICML 2015*

*Yossi Arjevani, Ohad Shamir. Communication complexity of distributed convex learning and optimization - NIPS 2015*

*Sashank J. Reddi, Jakub Konecny, Peter Richtárik, Barnabás Póczós, and Alex Smola. AIDE: Fast and communication efficient distributed optimization - arXiv:1608.06879*

*Shusen Wang, Farbod Roosta-Khorasani, Peng Xu, and Michael W Mahoney. GIANT: Globally improved approximate Newton method for distributed optimization - NIPS 2018*

*Hadrien Hendrikx, Lin Xiao, Sebastien Bubeck, Francis Bach, Laurent Massoulie. Statistically Preconditioned Accelerated Gradient Method for Distributed Optimization - ICML 2020*

*Amir Daneshmand, Gesualdo Scutari, Pavel Dvurechensky, Alexander Gasnikov. Newton Method over Networks is Fast up to the Statistical Precision - ICML 2021*

Recently there has been an interest also in extending these methods over general network topologies. Examples include:
 
*Y Sun, A Daneshmand, G Scutari, Distributed Optimization Based on Gradient-tracking Revisited: Enhancing Convergence Rate via Surrogation*

*B. Li, S. Cen, Y. Chen, and Y. Chi, Communication-efficient distributed optimization in networks with gradient tracking and variance reduction, JMLR 2020*

All the examples above  provide solid evidence of the popularity that methods exploiting data-similarity have received, theoretically and practically. We foresee a persistent interest in this direction, as optimal methods (i.e., matching lower bounds) under similarity have not been discovered yet.

Our work goes beyond classical optimization problems and derives lower complexity bounds and optimal distributed algorithms for the more general class of SPP over either star-networks or general networks, which has been an open problem for years.


We hope we clarified the importance of exploiting data-similarity (as requested by **Reviewer Crdz**) as well as the significance of our contribution, theoretically and practically (as requested by **Reviewer 6jmq**). 


**3) More applications of distributed SPP:** We will provide more examples of practical distributed saddle point problems where similarity can be used. Specifically, we will add the following:

*a) Online transport or Wasserstein Barycenter (WB) problem* can be rewritten as saddle point problem, see

*Dvinskikh D., Tiapkin D. Improved complexity bounds in wasserstein barycenter problem - AISTATS 2020*

This representation comes from the dual view on transportation polytope. Since the data (histograms / pictures of brains) typically weighs a lot we need distributed computations. And if we split data uniformly, we will have similarity on nodes.                  

*b) Lagrangian based optimization:* Let us consider the distributed convex minimization problem with constraints. It can be rewritten as a saddle-point problem using Lagrangian multipliers. It is easy to check that if the original functions $f_i$ is $\delta$-related, then the new saddle-point problem is also $\delta$-related.


**4) Additional experiments.** At the moment, we are ready to provide additional experiments for the paper.
More specifically, the new results are the following:

*a) Star networks:* in addition to  the distributed extragradient method, we compare the proposed algorithm with Local Descent- Ascent from the paper (because local technique is also used for reducing number of communications): 

*Yuyang Deng, Mehrdad Mahdavi, Local Stochastic Gradient Descent Ascent: Convergence Analysis and Communication Efficiency*

*b) General graph topologies:* we add experiments testing our algorithms on several different graph topologies, such as click, ring, and expander graphs. We also compare with the decentralized methods for saddles from the following works:

*Mingrui Liu, Wei Zhang, Youssef Mroueh, Xiaodong Cui, Jerret Ross, Tianbao Yang, Payel Das, A Decentralized Parallel Algorithm for Training Generative Adversarial Nets*

*Soham Mukherjee and Mrityunjoy Chakraborty.   A decentralized algorithm for large scale min-max problems*

We have created an anonymous github repository, where we uploaded the graphs of new experiments. If there is such an opportunity and permission from AC, we are ready to provide them.

**5) Conclusions.** We added a small conclusion for our paper (as requested by **Reviewers Crdz and PG9W**). In particular, we noted the comment from **Reviewer PG9W** “I suggest the authors to discuss the development of more practical decentralized algorithms in the conclusion section and leave it as a future job.” Here the text of the conclusion:

*We studied distristributed SPPs over networks, under data similarity. Such problems arise naturally from many applications, including machine learning and signal processing. We first derived lower complexity bounds for such problems for solution methods implementable either on star-networks or on general topologies (modeled as undirected, static graphs). These algorithms are optimal, in the sense that they achieve the lower bounds, up to log factors. The implementation of the proposed method over general network, however, is improvable: by selecting only one agent at time performing the updates, it does not fully exploit the potential computational speedup offered by the parallelism of the networking  setting. Also, the use of gossip protocols to propagate the updates of a single agent across the entire  network seems to be not very efficient. Finally, it would be interesting to combine the proposed methods with variance reduction techniques to alleviate the cost of local gradient computations. Designing distributed algorithms for SPPs overcoming these  limitations will be the subject of our future research.*

**6)Intuition and sketch of proofs.** We added a little intuition of methods and sketches of proofs to the main part of the work (as requested by **Reviewers 6jmq, ngA2 and PG9W**).

For lower bounds:

*The idea behind getting distributed lower bounds is to give an example of “bad” functions and “bad” layouts on agents. Let us consider the following functions:*

*(see (11) in supplementary materials)*

*It is easy to verify that these functions are $1$-smooth, $mu$-strongly-convex-strongly concave, and $\delta$-related. To obtain centralized bounds, let us partition function 1 into one third of agents, function 2 -- into one third, and function 3 --  into the remaining third. The solution to the general problem is nonzero in all components. But if we take the starting vector to be zero, then we can notice that the agents, without communications, will not be able to achieve non-zero components in the final output, no matter what algorithm (satisfying Definition 1) we use. Then it holds*

*(Theorem 1)*

*For the decentralized case the technique is close, but we need to consider a linear graph and use the trick with the characteristic number from [35].*

*(Theorem 2)*

For Algorithm 1:

*Briefly, Algorithm 1 combines two ideas: statistically preconditioned Bregman gradient method and extragradient. In fact, in Algorithm 1 it is difficult to see the extragradient in its classical form, the only thing that indicates this is the presence of line 3 (iii).*

Sketch of proof for Algorithm 1:

*The next lemma gives the convergence of one step*

*(Lemma 4 from supplementary materials)*

*Then, if we choose extragradient as a method for the inner saddle problem, it holds that*

*(Lemma 5 from supplementary materials)*

*Finally, the following theorem gives the parameters of Algorithm 1 and the communication complexity:*

*(Theorem 7 from supplementary materials)*

The paper studies saddle-point problems with objective being a sum of strongly convex-concave functions, $\min_x \max_y f(x,y) = \sum_{m = 1}^M f_m(x,y)$, in the setting of distributed communication. Specifically, each component in the sum of functions is local to an agent $m$, and communication between agents is expensive, which forces algorithms of two forms: master-worker type, and meshed grid type. 

The paper's chief assumption is that the functions $f_i$ are $\delta$-similar, which means an upper bound of $\delta$ on  the operator norm  of the difference between the Hessians of $f_m$ and $f$, evaluated with respect to $(x,x)$, $(x, y)$, and $(y, y)$ directions. This assumption is reasonable in the case of robust linear regression and in proving adversarial robustness of neural networks. 

The paper provides lower bounds and algorithms meeting these lower bounds for solving the saddle point problems to $\epsilon$ accuracy.   The paper studies saddle-point problems with objective being a sum of strongly convex-concave functions, $\min_x \max_y f(x,y) = \sum_{m = 1}^M f_m(x,y)$, in the setting of distributed communication. Specifically, each component in the sum of functions is local to an agent $m$, and communication between agents is expensive, which forces algorithms of two forms: master-worker type, and meshed grid type. 

The paper's chief assumption is that the functions $f_i$ are $\delta$-similar, which means an upper bound of $\delta$ on  the operator norm  of the difference between the Hessians of $f_m$ and $f$, evaluated with respect to $(x,x)$, $(x, y)$, and $(y, y)$ directions. This assumption is reasonable in the case of robust linear regression and in proving adversarial robustness of neural networks. 

PROS: 

1. The paper provides lower bounds for runtime of solving a saddle-point problem to $\epsilon$-accuracy, and, exploiting the assumption on function similarity,  the paper is able to improve the lower bounds that do *not* use function similarity. These lower bounds are provided for both the master-worker and meshed network models of communication. 

2. The paper provides algorithms that actually achieve these optimal runtimes (upto logarithmic factors), in both communication models. 

CONS: 

1. It is unclear to me what the significance of the paper's improvement is. 

2. I think it would be better to provide a proof sketch of the main techniques used in the main body itself. 

 N/A 

The paper discusses the (strongly-)convex-(strongly-)concave distributed saddle point problems with $\delta$-related functions in both master/worker networks and meshed networks. It provides lower bounds for each setting and the matching upper bounds, $O(\Delta \cdot \delta / \mu \cdot \log (1 / \varepsilon))$ and $O(1 / \sqrt{\rho} \cdot \delta / \mu \cdot \log (1 / \varepsilon))$ respectively for two settings.   Originality: I believe the theoretical results on distributed saddle point problem with $\delta$-related assumptions are new.

Clarity: The structure and writing of the paper look fine. However, in the section 4, the authors provide little explanation for their algorithms. Although the algorithms are similar to some existing ones, it is easier for the readers to understand if there are some intuitions put in the paper. Same for the lower bound part, there is no overview for the hard instance in the main text. 

Comments and questions:
1. For the lower bounds in Theorem 1 and 2, on the right hand sides of the equations include only $\|y^\star\|$. First, I think the authors did not define $\|y^\star\|$ in the paper and I assume it is the optimal value for $y$. Second, the right hand side is not in terms of $\|x_t-x_0\|^2 + \|y_t-y_0\|^2$. Here if $\|y^*\|=0$, the lower bounds are not informative. 

2. The paper is based on many existing work, for example the lower bounds for smooth strongly-convex-strongly-concave minmax optimization and lower bounds for distributed minimization problem with $\delta$-related assumption. Can the authors explain what is the technical novelty in this paper?

minor:
1. There is a typo under line 154 in the equation. 
2.  In line 17, the authors assume $X, Y$ to be compact, but in lower bound the case has unbounded domain. I think in the main text, the paper only discusses strongly-convex-strongly-concave setting and does not need to assume boundedness. It seems it is only needed in the convex-concave setting in the appendix. 
 Yes

This paper studies the distributed convex-concave saddle point problem under the relative smooth condition. Lower bounds are proved for strongly convex-strongly concave problems over centralized and decentralized networks. Optimal methods are proposed mathching the lower bounds.  Originality: This paper studies the lower bound for strongly convex-strongly concave saddle point problem under the relative smooth condition. The lower bound combines [4,34] (saddle point problems without the relative smooth condition) and [2] (strongly convex problems with the relative smooth condition).  Optimal algorithms are proposed matching these lower bounds. The contribution is solid.

Quality: The theory is technically solid. The theories are supported by proofs. This is a complete work.

Clarity: This paper is written well.

Significance: Distributed optimization under the relative smooth condition is a significant topic. This paper extends it to the strongly convex-strongly concave saddle point problem. It is significant.

Minor comments:

1. This paper proves the communication complexity of the proposed algorithms for the general convex-concave saddle point problem in the supplementary material. Why do not include it in the main paper?

2. What is the mitivation of Algorithm 1? It looks different from the classical statistically preconditioned Bregman gradient method for strongly convex problems. I suggest to explain why the updates are designed in the form of the three steps in Algorithm 1.

3. The master node is defined to own function $f_1$. What about the first worker? I suggest to use $f_0$ for the master in Algorithm 1.

4. I suggest to include a conclusion section. 

5. What is $\beta$ on line 94? I think it should be $\delta$.

6. I suggest to provide more details about the experiment using real data, especially the numerical size of $\delta$ in Assumption 2 and $\mu$ in your experimental setting.

7.  There is no mehotd compared in the experiment. I suggest to compare with the ones designed for the general strongly convex-strongly concave saddle point problems to verify whether the relative smooth condition helps to reduce the communication cost. No limitation is discussed. Algorithm 2 is not very interesting because it strongly depends on the multi-consensus gossip subroutine. So the proof framework is indeed the same with the centralized problems. I suggest the authors to discuss the development of more practical decentralized algorithms in the conclusion section and leave it as a future job.

Meta-review: Overall the majority of reviewers liked the paper and found the contributions interesting for NeurIPS. I also went over the paper and I think it provides an interesting and worthy contribution to distributed algorithms, studying saddle-point optimization problems under data-similarity (mostly motivated from statistical settings) and providing lower bounds on communication and matching upper-bounds. I recommend acceptance.

Decision: accept