Distributed Zeroth-Order Optimization: Convergence Rates That Match Centralized Counterpart

Zeroth-order optimization has become increasingly important in complex optimization and machine learning when cost functions are impossible to be described in closed analytical forms . The key idea of zeroth-order optimization lies in the ability for a learner to build gradient estimates by queries sent to the cost function , and then traditional gradient descent algorithms can be executed replacing gradients by the estimates . For optimization over large-scale multi-agent systems with decentralized data and costs , zeroth-order optimization can continue to be utilized to develop scalable and distributed algorithms . In this paper , we aim at understanding the trend in performance transitioning from centralized to distributed zeroth-order algorithms in terms of convergence rates , and focus on multi-agent systems with time-varying communication networks . We establish a series of convergence rates for distributed zeroth-order subgradient algorithms under both one-point and two-point zeroth-order oracles . Apart from the additional node-to-node communication cost due to the distributed nature of algorithms , the established rates in convergence are shown to match their centralized counterpart . We also propose a multi-stage distributed zeroth-order algorithm that better utilizes the learning rates , reduces the computational complexity , and attains even faster convergence rates for compact decision set . 1 INTRODUCTION . Various machine learning tasks ultimately boil down to solving optimization problems of different forms , where the cost functions are formed jointly by the data accumulated in experiences and the model used in representing the learning framework . Gradient descent algorithms have been playing a foundational role in practically solving such optimization problems . However , for learning tasks with high-dimensional data and involved learning representations , access to the gradient of the cost function may turn out not possible : the cost function supporting the learning may not have a closed analytical form ; or it is simply too computationally costly to be properly differentiated . Zeroth-order optimization provides a systemic way of facilitating gradient descent without direct access to gradient information , where oracles query the cost function values and generate gradient estimates . Zeroth-order methods have shown a number of successful applications , e.g. , searching for adversarial attacks in deep learning Chen et al . ( 2019 ) ; Liu et al . ( 2019 ) and policy search in reinforcement learning Vemula et al . ( 2019 ) . The literature has also explored the potential in extending the standard ( centralized ) zeroth-order optimization to distributed settings over multi-agent systems , where the data and cost functions are scattered across a network of decentralized agents . With the help of a communication network , the agents may collaboratively solve the network-level optimization task by iteratively exchanging decisions obtained from local zeroth-order descent . The rates of convergence of centralized zerothorder optimization algorithms are now well understood for several sub-classes of convex functions . We are interested in systematically investigating these convergence rates scale for the corresponding distributed algorithms , and focus on the case of time-varying communication networks . 1.1 PROBLEM DEFINITION . Consider a network of agents ( nodes ) V = { 1 , . . . , N } . The agents aim to collectively solve the following distributed optimization problem minimize f ( x ) : = N∑ i=1 fi ( x ) subject to x ∈ X . ( 1 ) Here x ∈ Rd is the decision variable , X ⊆ Rd is a convex decision space , and fi : Rd → R is a private convex objective function associated with agent i . The communication network connecting the nodes is described by a time-varying graph G ( t ) = ( V , E ( t ) ) , where E ( t ) is the set of activated links at time t. Let A ( t ) be a weight matrix at time t for the graph G ( t ) : for each link ( i , j ) ∈ E ( t ) , a weight [ A ( t ) ] ij > 0 is assigned , and [ A ( t ) ] ij = 0 for ( i , j ) /∈ E ( t ) . We impose the following assumption on the communication network E ( t ) and the weight matrix A ( t ) . Assumption 1 ( i ) There exists a positive integer B such that the union graph ( V , E ( kB+ 1 ) ∪ · · ·∪ E ( ( k+1 ) B ) ) is strongly connected for all k ≥ 0 ; ( ii ) A ( t ) is doubly stochastic , i.e. , ∑N i=1 [ A ( t ) ] ij = 1 and ∑N j=1 [ A ( t ) ] ij = 1 ; ( iii ) [ A ( t ) ] ii ≥ ξ for all i , and [ A ( t ) ] ij ≥ ξ if ( j , i ) ∈ E ( t ) , where ξ > 0 . 1.2 FUNCTION CLASSES . Let Fcvx denote the set of all convex functions on Rd . We define the following three classes of convex functions in Fcvx . • The Lipschitz continuous class Flip ( Lf , X ) contains the functions in Fcvx that admit a finite Lipschitz constant Lf over X , i.e. , Flip ( Lf , X ) : = { g ∈ Fcvx : ∀x , x′ ∈ X , |g ( x ) − g ( x′ ) | ≤ Lf‖x− x′‖ } . • The smooth class Fsmo ( sf , X ) contains the functions that admit a sf -Lipschitz continuous gradient over X , i.e. , Fsmo ( sf , X ) = { g ∈ Fcvx : ∀x , x′ ∈ X , ‖∇g ( x ) −∇g ( x′ ) ‖ ≤ sf‖x− x′‖ } . • The strongly convex class Fsc ( µf , X ) contains the functions that are µf -strongly convex , i.e. , Fsc ( µf , X ) = { g ∈ Fcvx : ∀x , x′ ∈ X , g ( x ) ≥ g ( x′ ) + 〈∇g ( x′ ) , x− x′〉+ µf 2 ‖x− x′‖2 } . 1.3 CONTRIBUTIONS AND RELATED WORK . Contributions . We first present MAZOPA , a multi-agent zeroth-order projection averaging algorithm . In MAZOPA , the agents iteratively carry out local zeroth-order descents for their private costs to generate intermediate decisions , send these intermediate decisions to their neighbors over the graph G ( t ) , and then update their decisions by projecting the average neighboring intermediate decisions onto X . For distributed zeroth-order oracles based on one-point or two-point estimates , a series of convergence rate results are established for the three basic function classes . Remarkably , the convergence rates for distributed algorithms are found to be matching their centralized counterpart , and sometimes even tighter rates are obtained , as summarized in Table 1 . These results show that by paying the price of node-to-node communication , distributed zeroth-order optimization provides equal performance guarantees as those of centralized approaches . Next , we generalize the MAZOPA to a multi-stage setting , where the local zeroth-order descents take place for multiple steps before the projected averaging in a sequence of epochs . Such multi-stage MAZOPA is shown to be able to reduce the computational complexity , while providing improved convergences rates compared to MAZOPA when the decision set is compact . Related Work . Recently , many types of centralized zeroth-order optimization algorithms have been studied , and their convergence rates ( and the way they depend on the dimension ) have been established in different settings . For unconstrained convex optimization , Nesterov & Spokoiny ( 2017 ) develops several types of two-point gradient estimators and achieves convergence rates that scale with dimension as O ( d2 ) . For constrained stochastic optimization , Duchi et al . ( 2015 ) establishes that the convergence rates are sharp up to factors at most logarithmic in the dimension . Zeroth-order optimization has a natural connection to bandit online optimization , where the latter focuses on dynamic environment where the objective functions are varying over time ( see , e.g. , Flaxman et al . ( 2005 ) ; Agarwal et al . ( 2010 ) ; Shamir ( 2013 ; 2017 ) ; Bubeck et al . ( 2017 ) ; Lattimore ( 2020 ) ; Hazan & Levy ( 2014 ) ) . In particular , the seminal work Flaxman et al . ( 2005 ) constructs a one-point gradient estimator ( or one-point bandit feedback model ) and achieves an O ( d/T 1/4 ) average regret . For two-point gradient estimator , Shamir ( 2017 ) establishes the tightness of the dimension-dependent factor O ( √ d ) in the framework of zeroth-order stochastic mirror descent . It is worth zooming into the literature on distributed zeroth-order/bandit online optimization . Due to the absence of a central coordinator , the algorithms developed should always rely on local computations and communications ( e.g. , Yuan & Ho ( 2015 ) ; Yi et al . ( 2020 ) ; Jakovetic et al . ( 2018 ) ; Hajinezhad et al . ( 2019 ) ; Wang et al . ( 2019 ) ; Pang & Hu ( 2019 ) ; Hajinezhad & Zavlanos ( 2018 ) ; Wan et al . ( 2020 ) ) . This makes the convergence analysis of the distributed zeroth-order/bandit online optimization algorithms more challenging . In Yuan & Ho ( 2015 ) , the authors develop a class of distributed zeroth-order optimization algorithms that require two functional evaluations at each iteration , and establishes asymptotic convergence of the algorithm . Non-asymptotic convergence is established in Jakovetic et al . ( 2018 ) ; Hajinezhad et al . ( 2019 ) ; Wang et al . ( 2019 ) ; Pang & Hu ( 2019 ) ; Hajinezhad & Zavlanos ( 2018 ) , but the dimension-dependence factors are either O ( d2 ) or far from optimal . The work Yi et al . ( 2020 ) considers distributed online optimization with long-term constraints and establishes bounds on regret as well as constraint violations . To avoid Euclidean projection onto the constraint set , Wan et al . ( 2020 ) develops a distributed bandit online optimization algorithm based on conditional gradient descent and one-point bandit feedback , and achieves a regret scaling of O ( T 3/4 √ lnT ) . 2 THE MAZOPA ALGORITHM AND ITS CONVERGENCE RATES In this section , we present the MAZOPA algorithm and establish the convergence rates for the three function classes . 2.1 DISTRIBUTED ZEROTH-ORDER ORACLES . Let n be a random vector in Rd drawn from some probability distribution . Then f̂i ( x ; δ ) : = En [ fi ( x + δn ) ] ( 2 ) is a smoothed function for fi . Here δ > 0 is a parameter setting the level of the smoothing . We introduce the following definition on distributed zeroth-order oracles ( DistZOO ) . Definition 1 ( DistZOO ) A vector g̃i ( x ; δ ) ∈ Rd is called a distributed zeroth-order oracle at node i if the following conditions hold : ( i ) E [ g̃i ( x ; δ ) ] = ∇f̂i ( x ; δ ) for all x ∈ Rd ; ( ii ) If fi ∈ Flip ( Lf ) , then f̂i ∈ Flip ( Lf ) as well , and there holds ∣∣f̂i ( x ; δ ) − fi ( x ) ∣∣ ≤ pdLfδ , with pd being some positive constant ; ( iii ) If fi ∈ Fsmo ( sf ) , then ∣∣f̂i ( x ; δ ) − fi ( x ) ∣∣ ≤ 12 p̃dsfδ2 with p̃d being some positive constant . A number of DistZOO satisfying Definition 1 can be obtained using existing gradient estimators , see , e.g. , Liu et al . ( 2020 ) . In the paper , we provide two representative gradient estimators that are commonly adopted in the literature . Let ui be a random vector independently generated from a unit sphere B1 in Rd . Then ( e.g. , Flaxman et al . ( 2005 ) ) g̃OPi ( x ; δ ) : = fi ( x + δtui ) uid/δ ( 3 ) is a one-point DistZOO satisfying Definition 1 . Moreover , g̃TPi ( x ; δ ) : = d 2δ ( fi ( x + δu ) − fi ( x− δu ) ) u ( 4 ) is a two-point DistZOO satisfying Definition 1 ( e.g. , Shamir ( 2017 ) ) . 2.2 THE MAZOPA ALGORITHM We present the following Multi-Agent Zeroth-Order Projection Averaging ( MAZOPA ) algorithm , which consists of two steps , a local zeroth-order optimization step and a distributed averaging step . MAZOPA , whose pseudo-code is presented in Algorithm 1 , is a variation of the multi-agent subgradient averaging algorithm proposed in Nedic et al . ( 2008 ) ; Nedic & Ozdaglar ( 2009 ) ; Nedic et al . ( 2010 ) , where the local optimization step is executed by sub-gradient descent . Algorithm 1 MAZOPA : x̂i ( T ) = MAZOPA ( xi ( 1 ) , ηt , δt , X ) Require : step size ηt , DistZOO g̃i ( x ; δt ) with exploration parameter δt for all i ∈ V Ensure : xi ( 1 ) ∈ X , ∀i ∈ V 1 : for t = 1 to T do 2 : Node i queries the DistZOO at point xi ( t ) and receives g̃i ( xi ( t ) ; δt ) 3 : Node i computes vi ( t ) = xi ( t ) − ηt · g̃i ( xi ( t ) ; δt ) 4 : Node i updates its state by using the information received from its instant neighbors xi ( t+ 1 ) = projX ( N∑ j=1 [ A ( t ) ] ijvj ( t ) ) 5 : end for Output : x̂i ( T ) = 1T ∑T t=1 xi ( t )

Summary: This paper proposes a zeroth-order optimization algorithm for distributed, multi-agent systems with time-varying communication networks. The authors show that their presented multi-agent zeroth-order projection averaging algorithm (and its improved multi-stage version) has a convergence rate that matches the centralized counterpart algorithms under different assumptions. A small numerical experiment is also conducted to illustrate their theoretical findings.