Title: SPARKLE: A Unified Single-Loop Primal-Dual Framework for Decentralized Bilevel Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: non-convex optimization

decentralized bilevel optimization

transient iteration complexity
Abstract: This paper studies decentralized bilevel optimization, in which multiple agents collaborate to solve problems involving nested optimization structures with neighborhood communications. Most existing literature primarily utilizes gradient tracking to mitigate the influence of data heterogeneity, without exploring other well-known heterogeneity-correction techniques such as EXTRA or Exact Diffusion. Additionally, these studies often employ identical decentralized strategies for both upper- and lower-level problems, neglecting to leverage distinct mechanisms across different levels. To address these limitations, this paper proposes SPARKLE, a unified single-loop primal-dual algorithm framework for decentralized bilevel optimization. SPARKLE offers the flexibility to incorporate various heterogeneity-correction strategies into the algorithm. Moreover, SPARKLE allows for different strategies to solve upper- and lower-level problems. We present a unified convergence analysis for SPARKLE, applicable to all its variants, with state-of-the-art convergence rates compared to existing decentralized bilevel algorithms. Our results further reveal that EXTRA and Exact Diffusion are more suitable for decentralized bilevel optimization, and using mixed strategies in bilevel algorithms brings more benefits than relying solely on gradient tracking.

SPARKLE: A Unified Single-Loop Primal-Dual Framework for Decentralized Bilevel Optimization

Abstract

This paper studies decentralized bilevel optimization, in which multiple agents collaborate to solve problems involving nested optimization structures with neighborhood communications. Most existing literature primarily utilizes gradient tracking to mitigate the influence of data heterogeneity, without exploring other well-known heterogeneity-correction techniques such as EXTRA or Exact Diffusion. Additionally, these studies often employ identical decentralized strategies for both upper- and lower-level problems, neglecting to leverage distinct mechanisms across different levels. To address these limitations, this paper proposes SPARKLE , a unified S ingle-loop P rimal-dual A lgo R ithm framewor K for decentra L ized bil E vel optimization. SPARKLE offers the flexibility to incorporate various heterogeneitycorrection strategies into the algorithm. Moreover, SPARKLE allows for different strategies to solve upper- and lower-level problems. We present a unified convergence analysis for SPARKLE, applicable to all its variants, with state-of-the-art convergence rates compared to existing decentralized bilevel algorithms. Our results further reveal that EXTRA and Exact Diffusion are more suitable for decentralized bilevel optimization, and using mixed strategies in bilevel algorithms brings more benefits than relying solely on gradient tracking.

1 Introduction

Numerous modern machine learning tasks, such as reinforcement learning [25], meta-learning [4], adversarial learning [36], hyper-parameter optimization [19], and imitation learning [3], entail nested optimization formulations that extend beyond the traditional single-level paradigm. For instance, hyper-parameter optimization aims to identify the optimal hyper-parameters for a specific learning task in the upper level by minimizing the validation loss, achieved through training models in the lower-level process. This nested optimization structure has spurred significant attention towards Stochastic Bilevel Optimization (SBO). Since the size of data samples involved in bilevel problems has become increasingly large, this paper investigates decentralized algorithms over a network of n agents (nodes) that collaborate to solve the following distributed bilevel optimization problem:

Kun Yuan †

In this formulation, each agent i holds a private upper-level objective function f i : R p × R q → R and a strongly convex lower-level objective function g i : R p × R q → R defined as:

where D f i and D g i represent the local data distributions at agent i . This paper does not make any assumptions about these data distributions, implying there might be data heterogeneity across agents.

Linear speedup and transient iteration complexity. Adecentralized stochastic algorithm achieves linear speedup if its iteration complexity decreases linearly with the network size n . Additionally, the transient iteration complexity refers to the number of transient iterations a decentralized algorithm must undergo to achieve the asymptotic linear speedup stage. The fewer the transient iterations, the faster the algorithm can achieve linear speedup. This paper aims to develop decentralized stochastic bilevel algorithms that can achieve linear speedup with as few transient iterations as possible.

Limitations in previous works. A significant challenge in decentralized bilevel optimization lies in accurately estimating the hyper-gradient ∇ Φ( x ) through neighborhood communications. Several studies have emerged to effectively address this challenge, such as those by [9, 33, 52, 16, 21, 40, 57, 29]. However, existing works suffer from several critical limitations:

Stringent assumptions and inadequate convergence analysis. Many existing studies rely on stringent assumptions to ensure convergence. For instance, references [9, 10, 33, 21, 52] assume bounded gradients, while reference [29] assumes bounded data heterogeneity (also known as bounded gradient dissimilarity). These restrictive assumptions do not arise in centralized bilevel optimization, implying their potential unnecessity. Moreover, some of these works suffer from inadequate convergence analysis, unable to clarify the transient iteration complexity [9, 33, 10] or provide a sharp estimation of the influence of network topologies [52, 21].

Limited exploration of various heterogeneity-correction techniques. Several concurrent studies [16, 57, 40] have utilized Gradient Tracking (GT) [50, 13, 38] to remove the assumption of bounded data heterogeneity. However, it remains uncertain whether GT is the most suitable mechanism for decentralized bilevel optimization. Many other techniques are also useful for addressing data heterogeneity in single-level decentralized optimization, such as EXTRA [45] and Exact-Diffusion (ED) [56, 30, 54] (which is also known as D 2 [46]). Even within GT, there are variants including Adapt-Then-Combine GT (ATC-GT) [50], non-ATC-GT [38], and semiATC-GT [13]. It remains unexplored whether these techniques for mitigating data heterogeneity converge and even outperform GT when employed in decentralized bilevel algorithms.

Unknown effects of employing different upper- and lower-level update strategies. In bilevel optimization, the challenges in solving the upper- and lower-level problems differ substantially. For instance, the upper-level problem (1a) is non-convex, whereas the lower-level problem (1b) is strongly convex. Moreover, estimating the gradient at the lower level is considerably simpler compared to estimating the hyper-gradient at the upper level. Understanding the roles of updates at each level is crucial to develop more efficient algorithms. However, most existing algorithms employ the same decentralized methods to solve both the upper- and lower-level problems. For example, references [9, 52, 29] utilize decentralized gradient descent (DGD) for updates at both levels while [57, 40, 16] leverage GT, overlooking the potential advantages of mixed strategies.

To address these limitations, several critical questions naturally arise: Should each heterogeneitycorrection mechanism listed in [50, 13, 38, 56, 30, 54, 46] be explored one-by-one? Should we consider combining any two of these techniques to update the upper and lower-level problems, respectively? It is evident that examining each individual heterogeneity-correction technique, and even exploring their combinations, would involve an unbearable amount of effort.

Main results and contributions. This paper addresses all the aforementioned limitations without exhaustively exploring all heterogeneity-correction techniques. Our main results are as follows.

A unified decentralized bilevel framework. To avoid examining each single heterogeneitycorrection technique, we propose SPARKLE , a unified S ingle-loop P rimal-dual A lgo R ithm

framewor K for decentra L ized bil E vel optimization. By specifying certain hyper-parameters, SPARKLE can be tailored to SPARKLE-EXTRA, SPARKLE-ED, and SPARKLE-GT, which employ EXTRA [45], ED [56, 30], or multiple GT variants [50, 13, 38], respectively, to facilitate the upper and lower-level problems. Additionally, SPARKLE is the first algorithm enabling distinct updating strategies across different levels; for example, one can utilize GT in the upperlevel but ED in the lower-level, resulting in a brand new SPARKLE-GT-ED algorithm.

Aunified and sharp analysis of various heterogeneity-correction schemes. Weprovide a unified convergence analysis for SPARKLE, which immediately applies to all SPARKLE variants with distinct heterogeneity-correction techniques. The analysis does not require restrictive assumptions such as gradient boundedness used in [9, 10, 33, 21, 52] or data-heterogeneity bounded used in [29]. Moreover, our analysis demonstrates the provable superiority of SPARKLE compared to existing algorithms, as evidenced by the convergence rates listed in Table 1. Most importantly, our analysis shows that both SPARKLE-EXTRA and SPARKLE-ED outperform SPARKLE-GT (see Table 1), implying that GT is not the best scheme for decentralized bilevel optimization.

Mixing strategies outperform employing GT alone. We demonstrate how optimization at different levels affects convergence rates. Our theoretical analysis suggests that the updating strategy at the lower level is crucial in determining the overall performance in decentralized bilevel algorithms. Building upon this insight, we establish that incorporating the ED or EXTRA strategy in the lower-level update phase leads to better transient iteration complexity than relying solely on the GT mechanism in both levels as proposed in [10, 16, 40], see Table 2 for more details.

Comparable performance with single-level algorithms. We elucidate the comparison between bilevel and single-level stochastic decentralized optimization. On one hand, we demonstrate that the convergence performance of all our proposed algorithms is not inferior to their single-level counterparts (see the bottom part in Table 1). On the other hand, by considering specific lower-level loss functions, our bilevel results directly yield the non-asymptotic convergence of corresponding

single-level algorithms. This is the first result demonstrating bilevel optimization essentially subsumes the convergence of the single-level optimization.

Our main results are listed in Table 1. All SPARKLE variants achieve the state-of-the-art asymptotic rate, asymptotic gradient complexity, asymptotic communication cost, and transient iteration complexity under more relaxed assumptions compared to existing methods.

Related works. A significant challenge in decentralized bilevel optimization is accurately estimating the hyper-gradient ∇ Φ( x ) , necessitating solving global lower-level problems and estimating Hessian inversion. To this end, various decentralized techniques have been applied in bilevel optimization, including Neumann series in [52], JHIP oracle in [9], HIGP oracle in [10], and augmented Lagrangianbased communication in [33]. Additionally, reference [29] proposes a single-loop algorithm utilizing decentralized SOBA. To enhance algorithmic robustness against data heterogeneity, recent studies have employed Gradient Tracking (GT) in both lower- and upper-level optimization. However, existing works built upon GT suffer from several limitations. Results of [16, 9] concentrate solely on deterministic cases, while reference [40] addresses personalized problems in the lower-level, which do not require achieving global consensus in the lower-level problem. Moreover, [9, 10] introduce computationally expensive inner loops for GT steps. None of these works can establish smaller transient iteration complexity than D-SOBA for decentralized SBO, even though the latter algorithm employs no heterogeneity-correction technique.

The unified framework for single-level decentralized optimization has been extensively studied in the literature. References [1, 49, 26] propose frameworks for decentralized composite optimization in deterministic settings, while [2] investigates a framework under stochastic settings. However, none of these works can be directly applied to decentralized bilevel algorithms. Several studies [21, 57] utilize variance reduction techniques to accelerate the convergence of stochastic decentralized bilevel algorithms. Our proposed SPARKLE framework is orthogonal to variance reduction; it can also incorporate variance-reduced gradient estimation to achieve improved convergence rates. More relevant works on decentralized optimization and bilevel optimization are discussed in Appendix A.

Notations. We use lowercase letters to represent vectors and uppercase letters to represent matrices. We introduce col { x 1 , ..., x n } := [ x ⊤ 1 , ..., x ⊤ n ] ⊤ ∈ R pn for brevity. Variables with overbar denote the average over all agents. For example, ¯ x k = ∑ n i =1 x k i /n . We denote A = A -1 n 1 n 1 ⊤ n for matrix A ∈ R n × n , where 1 n ∈ R n denotes the n -dimensional vector with all entries being one. For a function f ( x, y ) : R p × R q → R , we use ∇ 1 f ( x, y ) ∈ R p , ∇ 2 f ( x, y ) ∈ R q to represent its partial gradients with respect to x and y , respectively. Similarly, ∇ 12 f ( x, y ) ∈ R p × q , ∇ 22 f ( x, y ) ∈ R q × q represent the corresponding Jacobian and Hessian matrix. We use the notation ≲ to denote inequalities that hold up to constants related to the initialization of algorithms and smoothness constants.

2 SPARKLE : A unified framework for decentralized bilevel optimization

This section develops SPARKLE, a unified framework for decentralized bilevel optimization, and discusses its numerous variants by specifying certain hyper-parameters.

2.1 Three pillar subproblems in decentralized bilevel optimization.

When solving the upper-level problem (1a), it is critical to obtain the hyper-gradient ∇ Φ( x ) , which can be expressed as [22]

̸

Evaluating this hyper-gradient is computationally expensive due to the inversion of the Hessian matrix. This evaluation becomes even more challenging over a decentralized network of collaborative agents. First, the inverse of the Hessian matrix cannot be obtained by simply averaging the local Hessian inverses due to [ 1 n ∑ n i =1 ∇ 2 22 g i ( x, y ⋆ ( x ))] -1 = 1 n ∑ n i =1 [ ∇ 2 22 g i ( x, y ⋆ ( x ))] -1 . Second, the global averaging operation cannot be realized through decentralized communication. To overcome these challenges, one can introduce an auxiliary variable z ⋆ ( x ):= [ ∇ 2 22 g ( x, y ⋆ ( x ))] -1 ∇ 2 f ( x, y ⋆ ( x )) [12], which is the solution to a quadratic problem

Once z ⋆ ( x ) is derived by solving (4), we can substitute it into (3) to achieve ∇ Φ( x ) .

Following this idea, solving the distributed bilevel optimization problem (1) essentially involves solving three subproblems, where h i ( x, y ⋆ ( x ) , z ) := 1 2 z ⊤ ∇ 2 22 g i ( x, y ⋆ ( x )) z -z ⊤ ∇ 2 f i ( x, y ⋆ ( x )) ,

Given the variable x , one can achieve y ⋆ ( x ) by solving the lower-level problem in (5b). With y ⋆ ( x ) determined, z ⋆ ( x ) can be obtained by solving the auxiliary-level problem in (5c). Subsequently, with z ⋆ ( x ) available, one can directly compute the hyper-gradient and solve the upper-level problem in (5a) using gradient descent. This constitutes the primary methodology to solve problem (1).

Abilevel algorithm essentially solves three subproblems listed in (5), each formulated as a single-level decentralized optimization problem. Nevertheless, primary approaches may suffer from nested loops in algorithmic development. A few recent studies [12, 11, 57, 29] propose to solve each problem in (5a)-(5c) approximately with one single iteration, leading to practical single-loop bilevel algorithms. For example, applying a D-SGD step [43] to each of (5a)-(5c) yields the D-SOBA method [29], while further leveraging the GT technique leads to decentralized bilevel methods in [9, 16, 57, 21].

However, it is less explored whether numerous other heterogeneity-correction techniques [50, 13, 38, 56, 30, 54, 46] beyond GT can be incorporated into algorithmic design to achieve even better performance in bilevel optimization. To avoid exploring each case individually, we next introduce a general framework that unifies all these techniques for solving single-level problems.

2.2 A unified framework for decentralized single-level optimization.

In this subsection, we consider solving the single-level problem min x ∈ R p 1 n ∑ n i =1 f i ( x ) over a network of n nodes. For each k -th ( k ≥ 0 ) iteration, we let x k i denote the local x -variable maintained by the i -th agent. Furthermore, we associate the topology with a weight matrix W = [ w ij ] n i,j =1 ∈ R n × n in which w ij ∈ (0 , 1) if node j is connected to node i otherwise w ij = 0 . We use bold symbols to denote stacked vectors or matrices across agents. For example, x k = col { x k 1 , ..., x k n } ∈ R pn and W = W ⊗ I p , where ⊗ denotes the Kronecker product operator.

A unified framework with moving average. Building on the formulation in [1, 2], we develop a unified primal-dual framework with moving average for decentralized optimization:

Here x k denotes the primal variable, d k denotes the dual variable introduced to mitigate the influence of data-heterogeneity, g k stacks all (stochastic) gradients evaluated at x k i for 1 ≤ i ≤ n , r k denotes the momentum introduced to boost training with coefficient θ ∈ [0 , 1] , and α > 0 is the learning rate. Matrices A , B , C ∈ R pn × pn are adapted from the mixing matrix W , which determine how agents communicate with each other. See Appendix B.1 for more detailed motivations.

Framework (6) unifies various decentralized techniques in the literature. For instance, by letting θ = 1 and specifying A , B , C delicately, framework (6) reduces to ED, EXTRA, and numerous GT

Algorithm 1 SPARKLE: A unified framework for decentralized stochastic bilevel optimization

variants, see Table 3 and Appendix B.1 for more details. Framework (6) is closely related to the unified decentralized method developed in [1, 2]. The primary difference lies in the incorporation of the momentum variable r k , which can help improve the transient iteration complexity of the framework (6) and relax the smoothness condition for bilevel algorithms [11]. A detailed comparison between framework (6) and that proposed in [1, 2] is provided in Appendix B.2.

2.3 A unified framework for decentralized bilevel optimization.

By utilizing the unified framework (6) to approximately solve each subproblem in (5) with only one iteration , we achieve SPARKLE, a unified single-loop framework for decentralized bilevel optimization. In particular, we independently sample data ξ k i ∼ D f i , ζ k i ∼ D g i within each node at iteration k , and evaluate stochastic gradients/Jacobians/Hessians as follows

Next we stack the descent directions for variables of each level as follows

The SPARKLE algorithm is detailed in Algorithm 1. In this algorithm, we utilize different dual variables d s and communication matrices A s , B s , C s for each variable s ∈ { x, y, z } to optimize their respective objective functions. We use momentum r k only for updating the upper-level variable, which is sufficient to enhance convergence of bilevel algorithms and relax the smoothness condition.

Versatility in decentralized strategies. SPARKLE is highly versatile, supporting various decentralized strategies by allowing the specification of different communication matrices A s , B s , and C s . For example, by setting A s = I , B s = ( I -W ) 1 / 2 , and C s = W for any s ∈ { x, y, z } , SPARKLE will utilize EXTRA to update variables x , y , and z , resulting in the SPARKLE-EXTRA variant. Other variants can be achieved by setting A s , B s , and C s according to Table 3. These variants can be implemented more efficiently than listed in Algorithm 1, see Appendix B.3.

Flexibility across optimization levels. SPARKLE supports different optimization and communication mechanisms for each level of (5), which can be directly achieved by choosing different A s , B s , and C s matrices for each level s ∈ { x, y, z } . For example, SPARKLE can utilize GT to update the upper-level variable x while employing ED to update the auxiliary- and lower-level variables y and z . Throughout this paper, we denote SPARKLE using the decentralized mechanism L for the lower-level and auxiliary variables, and U for the upper-level in Algorithm 1, by SPARKLEL -U , or simply SPARKLEL if L = U . In addition, SPARKLE even supports utilizing different mixing matrices W x , W y , W z across levels.

3 Convergence analysis

In this section, we establish the convergence properties of the SPARKLE framework and examine the influence of different decentralized techniques utilized across optimization levels.

3.1 Assumptions

Before presenting the theoretical guarantees, we first introduce the following assumptions used throughout this paper.

Assumption 1. There exist constants µ g , L f, 0 , L f, 1 , L g, 1 , L g, 2 such that for any 1 ≤ i ≤ n ,

∇ f i , ∇ g i , ∇ 2 g i are L f, 1 , L g, 1 , L g, 2 Lipschitz continuous, respectively;

∥∇ 2 f i ( x, y ⋆ ( x )) ∥ ≤ L f, 0 for any x ∈ R p ; 3

g i ( x, y ) is µ g -strongly convex with respect to y for any fixed x ∈ R p .

Moreover, we define L := max { L f, 0 , L f, 1 , L g, 1 , L g, 2 } and κ := L/µ g .

Assumption 2. For each s ∈ { x, y, z } , the corresponding mixing matrix W s ∈ R n × n is non-negative, symmetric and doubly stochastic, i .e.,

and the corresponding communication graph is strongly-connected, i .e., its eigenvalues satisfy 1 = λ 1 ( W s ) > λ 2 ( W s ) ≥ . . . ≥ λ n ( W s ) and ρ ( W s ) := max {| λ 2 ( W s ) | , | λ n ( W s ) |} < 1 .

The value 1 -ρ ( W s ) is referred to as the spectral gap in the literature [34, 53, 31] of W s , which measures the connectivity of the communication graph. It would approach 0 for sparse networks. For example, it holds that 1 -ρ ( W s ) = Θ(1 /n 2 ) for the matrix W s induced by a ring graph.

Assumption 3. For any s ∈ { x, y, z } , we assume the communication matrices A s , B s , C s used in SPARKLE are polynomial functions of W s . Furthermore, we assume A s , C s are doubly stochastic, and Null( B s ) = Span { 1 n } . In addition, we assume all eigenvalues of the augmented matrix

are strictly less than one in magnitude, where C s ≜ C s -1 n 1 n 1 ⊤ n and I n ≜ I n -1 n 1 n 1 ⊤ n .

We remark that Assumption 3 is mild and is satisfied by all choices listed in Table 3. See more discussions in Appendix C.2.2.

Assumption 4. We assume ∇ F i ( x, y ; ξ ) , ∇ G i ( x, y ; ξ ) , and ∇ 2 G i ( x, y ; ξ ) to be unbiased estimates of ∇ f i ( x, y ) , ∇ g i ( x, y ) , and ∇ 2 g i ( x, y ) with bounded variances σ 2 f, 1 , σ 2 g, 1 , σ 2 g, 2 , respectively.

3.2 Convergence theorem

Under the above assumptions, we establish the convergence properties as follows. Proof details can be found in Appendix C.

Theorem 1. Under Assumptions 1 - 4, there exist proper constant step-sizes α, β, γ and momentum coefficient θ , such that the SPARKLE framework listed in Algorithm 1 will converge as follow:

where σ ≜ max { σ f, 1 , σ g, 1 , σ g, 2 } , { δ s,i } 3 i =1 are constants depending only on W s , A s , B s , C s for s ∈ { x, y, z } , and C α , C θ are constants independent of K . See Lemma 17 for their detailed values.

In the deterministic scenario with σ = 0 , SPARKLE converges at the rate O (1 /K ) , see the formal theorem and derivation in Appendix C.3. This recovers the rate in [15] under even milder assumptions. Unlike reference [15], which only considers GT in the deterministic setting, SPARKLE is a unified bilevel framework for the more general stochastic setting.

Linear speedup. According to Theorem 1, SPARKLE achieves an asymptotic linear speedup as K approaches infinity, which applies to all SPARKLE variants regardless of the decentralized strategies employed and whether they are utilized at different optimization levels. Furthermore, the asymptotically dominant term κ 5 σ/ ( √ nK ) matches exactly with the single-node bilevel algorithm SOBA [12] when n = 1 , implying the tightness of Theorem 1 in terms of the asymptotic rate.

Remark 1. We establish an upper bound for the consensus error 1 K K ∑ k =0 E [ ∥ x k -¯ x k ∥ 2 n + ∥ y k -¯ y k ∥ 2 n ] . Please refer to Lemma 19 in Appendix C.2.1 for more details.

3.3 Transient iteration complexity

With the non-asymptotic rate established in Theorem 1, we can derive the transient iteration complexity of SPARKLE as follows. The proof is in Lemma 18.

Corollary 1. Under the same assumptions as in Theorem 1, the transient iteration complexity of SPARKLE -with the influence of κ and σ 2 omitted for brevity-is on the order of

where δ s , ˆ δ s only depend W s , A s , B s , C s for s ∈ { x, y, z } . Their values are in Lemma 18.

We obtain the transient iteration complexity of each variant of SPARKLE by applying Corollary 1. Corollary 2. For SPARKLE-ED and SPARKLE-EXTRA , if we choose W y = W z , it holds that

Furthermore, if we choose W x = W y = W z and denote ρ ≜ ρ ( W x ) , the transient iteration complexity derived in (8) can be simplified as n 3 / (1 -ρ ) 2 .

Corollary 3. For SPARKLE-GT and its variants with semi/non-ATC-GT, if we let W y = W z ,

Furthermore, if we let W x = W y = W z and denote ρ ≜ ρ ( W x ) , the transient iteration complexity derived in (8) can be simplified as max { n 3 / (1 -ρ ) 2 , n/ (1 -ρ ) 8 / 3 } .

Remark 2 ( SOTA transient iterations) . Comparing with algorithms listed in Table 1, all SPARKLE variants achieve smaller transient iteration complexity, implying that they can achieve linear speedup much faster than the other algorithms, especially over sparse network topologies with 1 -ρ → 0 .

Remark 3 ( GT is not the best technique for decentralized SBO) . While GT is widely adopted in the literature [16, 21, 57] to facilitate decentralized SBO, a comparison of Corollary 2 and 3 reveals that both SPARKLE-EXTRA and SPARKLE-ED outperform SPARKLE-GT in terms of transient iteration complexity. This implies that EXTRA and ED are better than GT for decentralized SBO.

3.4 Different strategies across optimization levels

Corollary 1 clarifies how different update strategies for x , y , and z impact the transient iterations through constants { δ s , ˆ δ s } for s ∈ { x, y, z } . Since δ y = δ z and ˆ δ y = ˆ δ z when W y = W z (Lemma 18), we naturally employ the same strategy to update y and z . The following corollary studies the utilization of both ED and GT in SPARKLE. See the transient iterations complexity of other mixed strategies in Appendix C.2.4 and Table 2.

Corollary 4. For SPARKLE-ED-GT which uses ED to update y and z and GT to update x , if W x = W y = W z and we denote ρ = ρ ( W x ) , it then holds that

which implies that the transient iteration complexity in (8) can be simplified as n 3 / (1 -ρ ) 2 .

Remark 4 ( Mixed strategies outperform employing GT only) . Comparing Corollary 3 and 4, we find that using ED to update y and z will lead to smaller ˆ δ y and ˆ δ z , which improves the transient iteration complexity compared to employing GT only in all optimization levels (see Corollary 3) .

3.5 Different topologies across optimization levels

In SPARKLE, we can utilize different topologies across levels. Theorem 1 and Corollary 1 have clarified the influence of using different topologies across levels through the constants { δ s , ˆ δ s } for s ∈ { x, y, z } . For instance, when substituting { δ s , ˆ δ s } established in (9) into (8), SPARKLE-ED has the following transient iteration complexity:

where W x is the mixing matrix for updating x , while W y is for updating y and z . As long as (1 -ρ ( W x )) -1 ≲ √ n (1 -ρ ( W y )) -1 holds, SPARKLE-ED retains the transient iteration complexity of n 3 (1 -ρ ( W y )) -2 , which allows for the utilization of a sparser network topology when updating x , thereby reducing communication overheads. Consequently, the ratio a of the communication volume per round for the variables x and y can be significantly less than one. See Appendix C.2.3 for discussion on how to use different topologies across levels in other SPARKLE variants.

3.6 Recovering single-level decentralized optimization

Previous works typically study single-level and bilevel optimization separately. By taking G i ( x, y, ξ ) ≡ | y | 2 / 2 and F i ( x, y, ϕ ) = F i ( x, ϕ ) into (2), the decentralized SBO problem (1) reduces to stochastic single-level optimization. By setting z k ≡ 0 , y k ≡ 0 , u k i = ∇ 1 f i ( x k i , ξ k i ) , SPARKLE reduces to the single-level framework (6), whose convergence can be naturally guaranteed by Theorem 1. Please refer to Appendix C.4 for the detailed proof and results. This is the first result demonstrating that bilevel optimization essentially subsumes the convergence of single-level optimization.

4 Numerical experiments

In this section, we present experiments to validate our theoretical findings. We first explore how update strategies and network structures influence the convergence of SPARKLE. Then we compare SPARKLE to the existing decentralized SBO algorithms. Additional experiments about a decentralized SBO problem with synthetic data are in Appendix D.1.

Hyper-cleaning on FashionMNIST dataset. We consider a data hyper-cleaning problem [44] on a corrupted FashionMNIST dataset [48]. Problem formulations and experimental setups can be found in Appendix D.2. Firstly, we equip SPARKLE with different decentralized strategies in different optimization levels and then compare them with D-SOBA [29], MA-DSBO-GT [10], and MDBO [21] using the corruption rate p = 0 . 1 , 0 . 2 , 0 . 3 , respectively. As is shown in Figure 1, all the SPARKLE-based algorithms generally achieve higher test accuracy than D-SOBA, while ED and EXTRA especially outperform GT. Meanwhile, using mixed strategies ( i .e., SPARKLE-ED-GT and SPARKLE-EXTRA-GT) achieves similar test accuracy with SPARKLE-ED and SPARKLEEXTRA and outperform SPARKLE-GT, respectively. These observations match with the theoretical results in Corollary 2-4 and Remark 3, 4.

0

250

500

750

1000

1250

1500

1750

2000

number of gradient evaluations

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

test accuracy

SPARKLE­GT

SPARKLE­ED

SPARKLE­EXTRA

SPARKLE­ED­GT

SPARKLE­EXTRA­GT

D­SOBA

MA­DSBO­GT

MDBO

0

500

1000

1500

2000

2500

3000

number of gradient evaluations

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

test accuracy

SPARKLE­GT

SPARKLE­ED

SPARKLE­EXTRA

SPARKLE­ED­GT

SPARKLE­EXTRA­GT

D­SOBA

MA­DSBO­GT

MDBO

0

500

1000

1500

2000

2500

3000

3500

number of gradient evaluations

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

test accuracy

SPARKLE­GT

SPARKLE­ED

SPARKLE­EXTRA

SPARKLE­ED­GT

SPARKLE­EXTRA­GT

D­SOBA

MA­DSBO­GT

MDBO

0

500

1000

1500

2000

2500

3000

3500

number of gradient evaluations

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

test accuracy

(a)SPARKLE-EXTRA; Fixed topo

x

; Varied topo of

y

,

z

2200

2400

2600

2800

3000

3200

3400

0.65

0.70

0.75

0.80

=0.647

=0.828

=0.924

=0.990

0

500

1000

1500

2000

2500

3000

3500

number of gradient evaluations

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

test accuracy

(b)SPARKLE-EXTRA; Fixed topo

y

,

z

; Varied topo of

x

2200

2400

2600

2800

3000

3200

3400

0.65

0.70

0.75

0.80

=0.647

=0.828

=0.924

=0.990

0

250

500

750

1000

1250

1500

1750

2000

sample size

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

upper level loss

1600

1700

1800

1900

0.27

0.28

0.29

SPARKLE­ED

SPARKLE­EXTRA

SLDBO

MDBO

0

250

500

750

1000

1250

1500

1750

2000

sample size

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

upper level loss

1600

1700

1800

1900

0.32

0.33

SPARKLE­ED

SPARKLE­EXTRA

SLDBO

MDBO

Next, we test SPARKLE-EXTRA with two communication strategies including fixed topology for updating x and varying topology for y, z , and fixed topology for updating y, z and varying topology for x . As illustrated in Figure 2, maintaining a fixed topology for x while reducing the connectivity of the topology for y and z will deteriorate the algorithmic performance. Conversely, preserving the topology for y and z while decreasing the connectivity for x has little impact on the performance. This suggests that the influence of the network topology for y and z on the algorithm dominates over the topology for x , which is consistent with our discussion in Section 3.5. We also numerically examine the influence of moving average on convergence, see discussions in Appendix D.2.

Distributed policy evaluation in reinforcement learning. We consider a multi-agent MDP problem in reinforcement learning on a distributed setting with n ∈ { 10 , 20 } agents respectively, which can be formulated as a decentralized SBO problems [52]. Here, we compare SPARKLE with existing decentralized SBO approaches including MDBO [21] and the stochastic extension of SLDBO [16] over a Ring graph. Figure 3 illustrates that SPARKLE converges faster and achieves a lower sample complexity than the other baselines, especially when n = 20 , which shows the empirical benefits of SPARKLE in decentralized SBO algorithms with a large number of agents and sparse communication modes. More experimental details are in Appendix D.3.

Decentralized meta-learning. We investigate decentralized meta-learning on miniImageNet [47] with multiple tasks [18], formulating it as a decentralized bilevel optimization problem. This approach minimizes the validation loss with respect to shared parameters as the upper-level loss, while the training loss is managed by task-specific parameters at the lower level. Additional details about the experiment can be found in Appendix D.4. Our method, SPARKLE, is benchmarked against D-SOBA [29] and MAML [18], demonstrating a significant improvement in training accuracy.

5 Conclusions and limitations

This paper proposes SPARKLE, a unified single-loop primal-dual framework for decentralized stochastic bilevel optimization. Being highly versatile, SPARKLE can support different decentralized mechanisms and topologies across optimization levels. Moreover, all SPARKLE variants have been demonstrated to achieve state-of-the-art convergence rate compared to existing algorithms. However, SPARKLE currently supports only strongly-convex problems in the lower-level optimization. Its compatibility with generally-convex lower-level problems remains unknown. Additionally, the condition number of the lower-level problem significantly impacts the performance, as is the case with existing bilevel algorithms. We aim to address these limitations in future work.

6 Acknowledgment

The work of Shuchen Zhu, Boao Kong, and Kun Yuan is supported by Natural Science Foundation of China under Grants 92370121, 12301392, and W2441021. This work is also supported by Open Project of Key Laboratory of Mathematics and Information Networks, Ministry of Education, China. No. KF202302.

References

S. A. Alghunaim, E. K. Ryu, K. Yuan, and A. H. Sayed. Decentralized proximal gradient algorithms with linear convergence rates. IEEE Transactions on Automatic Control , 66(6):2787-2794, 2020.

S. A. Alghunaim and K. Yuan. A unified and refined convergence analysis for non-convex decentralized learning. IEEE Transactions on Signal Processing , 2022.

S. Arora, S. Du, S. Kakade, Y. Luo, and N. Saunshi. Provable representation learning for imitation learning via bi-level optimization. In International Conference on Machine Learning , pages 367-376. PMLR, 2020.

L. Bertinetto, J. Henriques, P. Torr, and A. Vedaldi. Meta-learning with differentiable closed-form solvers. In International Conference on Learning Representations (ICLR), 2019 . International Conference on Learning Representations, 2019.

T.-H. Chang, M. Hong, and X. Wang. Multi-agent distributed optimization via inexact consensus admm. IEEE Transactions on Signal Processing , 63(2):482-497, 2014.

J. Chen and A. H. Sayed. Diffusion adaptation strategies for distributed optimization and learning over networks. IEEE Transactions on Signal Processing , 60(8):4289-4305, 2012.

T. Chen, Y. Sun, Q. Xiao, and W. Yin. A single-timescale method for stochastic bilevel optimization. In International Conference on Artificial Intelligence and Statistics , pages 2466-2488. PMLR, 2022.

T. Chen, Y. Sun, and W. Yin. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. Advances in Neural Information Processing Systems , 34:25294-25307, 2021.

X. Chen, M. Huang, and S. Ma. Decentralized bilevel optimization. arXiv preprint arXiv:2206.05670 , 2022.

X. Chen, M. Huang, S. Ma, and K. Balasubramanian. Decentralized stochastic bilevel optimization with improved per-iteration complexity. In International Conference on Machine Learning , pages 4641-4671. PMLR, 2023.

X. Chen, T. Xiao, and K. Balasubramanian. Optimal algorithms for stochastic bilevel optimization under relaxed smoothness conditions. arXiv preprint arXiv:2306.12067 , 2023.

M. Dagréou, P. Ablin, S. Vaiter, and T. Moreau. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. Advances in Neural Information Processing Systems , 35:26698-26710, 2022.

P. Di Lorenzo and G. Scutari. Next: In-network nonconvex optimization. IEEE Transactions on Signal and Information Processing over Networks , 2(2):120-136, 2016.

J. Domke. Generic methods for optimization-based modeling. In Artificial Intelligence and Statistics , pages 318-326. PMLR, 2012.

J. Dong, Z. Cao, T. Zhang, J. Ye, S. Wang, F. Feng, L. Zhao, et al. Eflops: Algorithm and system co-design for a high performance distributed training platform. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA) , pages 610-622, 2020.

Y. Dong, S. Ma, J. Yang, and C. Yin. A single-loop algorithm for decentralized bilevel optimization. arXiv preprint arXiv:2311.08945 , 2023.

J. C. Duchi, A. Agarwal, and M. J. Wainwright. Dual averaging for distributed optimization: Convergence analysis and network scaling. IEEE Transactions on Automatic control , 57(3):592-606, 2011.

C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning , pages 1126-1135, 2017.

L. Franceschi, P. Frasconi, S. Salzo, R. Grazzi, and M. Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning , pages 1568-1577. PMLR, 2018.

B. Gao, Y. Yang, and Y. xiang Yuan. Lancbio: dynamic lanczos-aided bilevel optimization via krylov subspace. arXiv preprint arXiv:2404.03331 , 2024.

H. Gao, B. Gu, and M. T. Thai. On the convergence of distributed stochastic bilevel optimization algorithms over a network. In International Conference on Artificial Intelligence and Statistics , pages 9238-9281. PMLR, 2023.

S. Ghadimi and M. Wang. Approximation methods for bilevel programming. arXiv preprint arXiv:1802.02246 , 2018.

R. Grazzi, L. Franceschi, M. Pontil, and S. Salzo. On the iteration complexity of hypergradient computation. In International Conference on Machine Learning , pages 3748-3758. PMLR, 2020.

Z. Guo, Q. Hu, L. Zhang, and T. Yang. Randomized stochastic variance-reduced methods for multi-task stochastic bilevel optimization. arXiv preprint arXiv:2105.02266 , 2021.

M. Hong, H.-T. Wai, Z. Wang, and Z. Yang. A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic. SIAM Journal on Optimization , 33(1):147-180, 2023.

D. Jakoveti´ c. A unification and generalization of exact distributed first-order methods. IEEE Transactions on Signal and Information Processing over Networks , 5(1):31-46, 2018.

K. Ji, J. Yang, and Y. Liang. Bilevel optimization: Convergence analysis and enhanced design. In International Conference on Machine Learning , pages 4882-4892. PMLR, 2021.

A. Koloskova, T. Lin, and S. U. Stich. An improved analysis of gradient tracking for decentralized machine learning. Advances in Neural Information Processing Systems , 34:11422-11435, 2021.

B. Kong, S. Zhu, S. Lu, X. Huang, and K. Yuan. Decentralized bilevel optimization over graphs: Loopless algorithmic update and transient iteration complexity. arXiv preprint arXiv:2402.03167 , 2024.

Z. Li, W. Shi, and M. Yan. A decentralized proximal-gradient method with network independent step-sizes and separated convergence rates. IEEE Transactions on Signal Processing , July 2019. early acces. Also available on arXiv:1704.07807.

X. Lian, C. Zhang, H. Zhang, C.-J. Hsieh, W. Zhang, and J. Liu. Can decentralized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent. In Advances in Neural Information Processing Systems , pages 5330-5340, 2017.

T. Lin, S. P. Karimireddy, S. U. Stich, and M. Jaggi. Quasi-global momentum: Accelerating decentralized deep learning on heterogeneous data. In International Conference on Machine Learning , 2021.

S. Lu, S. Zeng, X. Cui, M. Squillante, L. Horesh, B. Kingsbury, J. Liu, and M. Hong. A stochastic linearized augmented lagrangian method for decentralized bilevel optimization. Advances in Neural Information Processing Systems , 35:30638-30650, 2022.

Y. Lu and C. De Sa. Optimal complexity in decentralized training. In International Conference on Machine Learning , pages 7111-7123. PMLR, 2021.

D. Maclaurin, D. Duvenaud, and R. Adams. Gradient-based hyperparameter optimization through reversible learning. In International Conference on Machine Learning , pages 2113-2122. PMLR, 2015.

A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations , 2018.

A. Nedi´ c, A. Olshevsky, and M. G. Rabbat. Network topology and communication-computation tradeoffs in decentralized optimization. Proceedings of the IEEE , 106(5):953-976, 2018.

A. Nedic, A. Olshevsky, and W. Shi. Achieving geometric convergence for distributed optimization over time-varying graphs. SIAM Journal on Optimization , 27(4):2597-2633, 2017.

A. Nedic and A. Ozdaglar. Distributed subgradient methods for multi-agent optimization. IEEE Transactions on Automatic Control , 54(1):48-61, 2009.

Y. Niu, J. Xu, Y. Sun, Y. Huang, and L. Chai. Distributed stochastic bilevel optimization: Improved complexity and heterogeneity analysis. arXiv preprint arXiv:2312.14690 , 2023.

G. Qu and N. Li. Harnessing smoothness to accelerate distributed optimization. IEEE Transactions on Control of Network Systems , 5(3):1245-1260, 2018.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision , 115:211-252, 2015.

A. H. Sayed. Adaptive networks. Proceedings of the IEEE , 102(4):460-497, 2014.

A. Shaban, C.-A. Cheng, N. Hatch, and B. Boots. Truncated back-propagation for bilevel optimization. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1723-1732. PMLR, 2019.

W. Shi, Q. Ling, G. Wu, and W. Yin. EXTRA: An exact first-order algorithm for decentralized consensus optimization. SIAM Journal on Optimization , 25(2):944-966, 2015.

H. Tang, X. Lian, M. Yan, C. Zhang, and J. Liu. D 2 : Decentralized training over decentralized data. In International Conference on Machine Learning , pages 4848-4856, 2018.

O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al. Matching networks for one shot learning. Advances in Neural Information Processing Systems , 29, 2016.

H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

J. Xu, Y. Tian, Y. Sun, and G. Scutari. Distributed algorithms for composite optimization: Unified framework and convergence analysis. IEEE Transactions on Signal Processing , 69:3555-3570, 2021.

J. Xu, S. Zhu, Y. C. Soh, and L. Xie. Augmented distributed gradient methods for multi-agent optimization under uncoordinated constant stepsizes. In IEEE Conference on Decision and Control (CDC) , pages 2055-2060, Osaka, Japan, 2015.

J. Yang, K. Ji, and Y. Liang. Provably faster algorithms for bilevel optimization. Advances in Neural Information Processing Systems , 34:13670-13682, 2021.

S. Yang, X. Zhang, and M. Wang. Decentralized gossip-based stochastic bilevel optimization over communication networks. Advances in Neural Information Processing Systems , 35:238-252, 2022.

K. Yuan, S. A. Alghunaim, and X. Huang. Removing data heterogeneity influence enhances network topology dependence of decentralized SGD. Journal of Machine Learning Research , 24(280):1-53, 2023.

K. Yuan, S. A. Alghunaim, B. Ying, and A. H. Sayed. On the influence of bias-correction on distributed stochastic optimization. IEEE Transactions on Signal Processing , 2020.

K. Yuan, Q. Ling, and W. Yin. On the convergence of decentralized gradient descent. SIAM Journal on Optimization , 26(3):1835-1854, 2016.

K. Yuan, B. Ying, X. Zhao, and A. H. Sayed. Exact dffusion for distributed optimization and learning Part I: Algorithm development. IEEE Transactions on Signal Processing , 67(3):708 - 723, 2018.

Y. Zhang, M. T. Thai, J. Wu, and H. Gao. On the communication complexity of decentralized bilevel optimization. arXiv preprint arXiv:2311.11342 , 2023.

Appendix for 'SPARKLE: A Unified Single-Loop Primal-Dual Framework for Decentralized Bilevel Optimization'

Contents

A More related works

Bilevel optimization. Bilevel optimization presents substantial difficulties compared to single-level optimization due to its nested structure. Estimating hyper-gradient ∇ Φ( x ) of the upper level involves solving lower-level problems and estimating the Hessian inverse, which requires additional calculations. Many algorithms and techniques have been proposed to solve the challenge. Approximate Implicit Differentiation (AID)-based algorithms [14, 22, 23, 27] leverage the implicit gradient form of ∇ Φ( x ) , which entails solving a linear system to obtain the Hessian-inverse-vector product. Similarly, [8, 25] utilize the Neumann series to handle the Hessian inverse. Iterative Differentiation (ITD)-based algorithms [19, 35, 14, 23, 27] use iterative methods solving the lower-level problem and then estimate the hyper-gradient through automatic differentiation. However, these approaches introduce inner steps, leading to extra computational overhead and memory spaces. [12] proposes a singlelevel algorithm called SOBA, which approximating the Hessian-inverse-vector product by solving a quadratic programming problem. A recent work [20] utilizes the Krylov subspace technique and the Lanczos process to approximate it in deterministic scenarios. For stochastic bilevel optimization, various methods have been employed to improve the convergence rate, such as momentum [7, 11] and variance reduction [51, 27, 24].

Decentralized optimization. Decentralized optimization is developed to deal with large-scale optimization problems, where datasets are distributed among multiple agents. Without a central server, each agent only gets access to its own local data and communications are limited to its neighbors in a network. Compared with centralized algorithms, decentralized ones preserve data privacy, and are more robust to contingencies in the communication network. However, due to the absence of a central server, decentralized optimization requires communication among agents, posing greater challenges for convergence, especially in the presence of severe data heterogeneity. To tackle this issue, various algorithms have emerged, such as decentralized gradient descent [39, 55], diffusion strategies [6], dual averaging [17], EXTRA [45], Exact Diffusion (a.k.a. D 2 ) [56, 30, 46], gradient tracking [50, 13, 38], and decentralized ADMM [5]. In stochastic scenarios, a common method for decentralized optimization is the decentralized stochastic gradient descent (DSGD), which has gained a lot of attentions recently. It has been proved to achieve linear speedup asymptotically and shares the same asymptotic rate with centralized stochastic gradient descent [31].

B More details of SPARKLE

B.1 Primal-dual deviation

Here we provide a detailed motivation of the update framework (6) for decentralized single-level algorithms. First, we rewrite the single-level distributed optimization problem in the following equivalent form:

where each f i is smooth and possibly non-convex. To simplify the notation, we assume that d = 1 without loss of generality. Now we introduce three symmetric matrices A,B,D such that A is a doubly stochastic communication matrix with ρ ( A ) < 1 , and B,D satisfy Null B = Null D = Span { 1 n } . In general, B ( D ) determines the topology of a connected graph G B ( G D ) over agents. The constraint Bx = 0 ( Dx = 0 ) is equivalent to:

To simplify the derivation, we additionally assume that A,B,D are pairwise commutative. Then for x = ( x 1 , ..., x n ) , we have:

Therefore, (10) can be equivalently reformulated as

We construct the augmented Lagrangian function of the problem (11) as follows:

where x denotes the primal variable, d denotes the dual variable or Lagrangian multiplier associated with the consensus constraint, ∥ Dx ∥ 2 serves as the penalty term measuring the deviation from Dx = 0 , or equivalently Bx = 0 ; ρ > 0 is the penalty coefficient. Though the introduction of matrices A,D is essentially a matter of equivalent substitution, it enhances the universality of the algorithm framework we get.

Following classical primal-dual methods, we alternately perform gradient descent on x and gradient ascent on d in the k -th iteration:

where α, β denote the step-sizes. By making the change of variables

we obtain

One should note that the definition implies that ̂ A, ̂ C are doubly stochastic communication matrices under appropriate selections of α, ρ . Finally, thanks to the introduction of moving-average iteration of (12), we can obtain the framework (6) which serves as the foundation for our algorithm design. See more details in Section 2.3.

B.2 Specific instances

Relation to some existing single-level algorithm frameworks According to (12), our framework at single-level is

where α is the step-size, g k denotes the estimated gradient at the k -th iteration, d serves as the dual variable.

Replacing C with CA , we get UDA[1] , and equivalently, SUDA [2]:

Therefore, following SUDA, we can also recover some common state-of-the-art heterogeneity methods as follows by selecting specific A , B , C . First, from (13) we get

Thus, for k ≥ 0 we have

Some specific instances We next show that how to choose A , B , C to get some common heterogeneity methods.

ED: Taking A = W , B = ( I -W ) 1 / 2 and C = W

EXTRA: Taking A = I , B = ( I -W ) 1 / 2 with C = W , we get EXTRA:

and x 1 = Wx 0 -α g 0 .

Adapt-then-combine gradient tracking (ATC-GT): The iteration of ATC-GT is

with h 0 = Wg 0 , x 0 = Wx 0 ( x 0 1 = ... = x 0 n ). It follows that for k ≥ 0

Then we obtain

with x 1 = W 2 x 0 -α W 2 g 0 . Thus, we can take A = W 2 , B = ( I -W ) 2 , C = W 2 to implement ATC-GT.

Semi-ATC-GT: The iteration of Semi-ATC-GT is

with h 0 = Wg 0 , x 0 = Wx 0 ( x 0 1 = ... = x 0 n ). Like ATC-GT, we have

with x 1 = W 2 x 0 -α Wg 0 . Thus, we can take A = W , B = ( I -W ) 2 , C = W 2 to implement semi-ATC-GT.

Non-ATC-GT: The iteration of Non-ATC-GT is

with h 0 = Wg 0 , x 0 = Wx 0 ( x 0 1 = ... = x 0 n ). We have

with x 1 = W 2 x 0 -α g 0 . Thus, we can take A = I , B = ( I -W ) 2 , C = W 2 to implement Non-ATC-GT.

B.3 Implementation details

Given the update method L , we update the lower-level variable y at the k -th ( k ≥ 0 ) iteration as follows. For brevity, we define y -1 i = y 0 i , v -1 i = 0 , o 0 i = ∑ n j =1 ( W y ) ij v 0 j .

Similarly, we update the auxiliary variable z at the k -th ( k ≥ 0 ) iteration as follows. For brevity, we define z -1 i = z 0 i , p -1 i = 0 , h 0 i = ∑ n j =1 ( W z ) ij p 0 j . Note that we use the same method L to update z as we do for the lower-level variable y .

Given the update method U , we update the upper-level variable x at the k -th ( k ≥ 0 ) iteration as follows. For brevity, we define x -1 i = x 0 i , t 0 i = ∑ n j =1 ( W y ) ij r 1 j .

Then the practical implementation of SPARKLE with mixed strategies is

Algorithm 2 SPARKLEL -U

C Convergence analysis

C.1 Proof of Theorem 1

C.1.1 Notations

We use lowercase letters to represent vectors and uppercase letters to represent matrices. Stacked vectors [ x ⊤ 1 , ..., x ⊤ n ] ⊤ is denoted by col { x 1 , ..., x n } for brevity. We denote a block diagonal matrix with diagonal block M i (1 ≤ i ≤ l ) by blkdiag { M 1 , ..., M l } , and a diagonal matrix with diagonal elements d i (1 ≤ i ≤ k ) by diag { d 1 , ..., d k } . The Kronecker product operator is denoted by ⊗ . For a variable v , we use v k i to represent its components at k -th iteration and i -th agent.

Moreover, we use an overbar above an iterator to denote the average over all agents. For example, ¯ x k = ∑ n i =1 x k i /n . Upright bold symbols are used to denote stacked vectors or matrices across agents. For example, x k := col { x k 1 , ..., x k n } , ¯ x k := col { ¯ x k , ..., ¯ x k } ( n times ) , W x := W x ⊗ I dim ( x ) . Denote the 2-norm of a matrix by ∥ · ∥ .

Next, we define following σ -fields which will be used in our convergence analysis:

and denote E [ ·|F k ] by E k , E [ ·|U k ] by ˜ E k , E [ ·|G k ] by ̂ E k for brevity.

Define

Then, for k = 0 , 1 , · · · , define:

For convenience, we define x -1 = x 0 , y -1 = y 0 , y ⋆ (¯ x -1 ) = y ⋆ (¯ x 0 ) , z 0 ⋆ = z 1 ⋆ .

C.1.2 Basic transformations

We begin with conducting SUDA-like [2] transformations, which is fundamental of the following proofs.

Firstly, we define t k to track the averaged stochastic gradients among agents as follows

where

Then the iteration of y , z , x in Algorithm 1 can be written as:

Next, we present the transformation of the matrices A,B,C . For a communication matrix W s for the variable s ∈ { x, y, z } satisfying Assumption 2, there exists an orthogonal matrix U such that:

where Λ s = diag { λ si } n i =2 , ˆ U ⊤ s ∈ R n × ( n -1) satisfies ˆ U s ˆ U ⊤ s = I n -1 n 1 n 1 ⊤ n and 1 ⊤ n ˆ U s = 0 . Then it follows that:

where dim ( s ) denotes the dimension of the corresponding variable, Λ s = Λ s ⊗ I dim ( s ) ∈ R d ( n -1) × [ dim ( s ) · ( n -1)] , U s ∈ R [ dim ( s ) · n ] × [ dim ( s ) · n ] is an orthogonal matrix, and ˆ U s = ˆ U s ⊗ I dim ( s ) ∈ R [ dim ( s ) · n ] × [ dim ( s ) · ( n -1)] satisfies:

Now we add subscript s for W s . Then, as A s , B 2 s , C s can be expressed as a polynomial of W s for s ∈ { x, y, z } according to Assumption 2, we have the orthogonal decomposition:

where

Moreover, each Λ sb is positive definite because of the null space condition in Assumption 2. Then, multiplying both sides of (18), (19) and (20) by U ⊤ y , U ⊤ z , U ⊤ x respectively, we get:

Then, due to Eq. (17), we have:

Substituting (25), (26), (27) into (22), (23), (24), respectively. Then use (21) and the structure of ˆ U y , ˆ U z , ˆ U x , we have

The above three equations are equivalent to:

For s ∈ { x , y , z } , define:

Then (28), (29), (30) are respectively equivalent to:

Assumption 2, 3 imply that all eigenvalues of

are strictly less than one in magnitude. Thus by symmetrically exchanging columns and rows of the matrix, we know that equivalently, all eigenvalues of

are strictly less than one in magnitude,.

Then according to Lemma 3, for s ∈ { x, y, z } , M s has the similarity transformation:

where O s is invertible and ∥ Γ s ∥ < 1 . Moreover, we define ˆ e k s = O -1 s e k s . It yields

Then, for s ∈ { x , y , z } , the consensus errors between different agents have the upper bound of:

Thus, we can define:

to measure the consensus error during the iteration.

We also define

to measure the estimation accuracy of the lower- and auxiliary-level problems.

C.1.3 Proof sketch

Before proceeding with the formal proof, we first present the structure of the proof in Appendix C.

Bounded by each other

Descent of

x

∑

E

∥

¯

r

k

∥

2

Descent of

y

∑

E

∥

¯

y

k

+1

-

y

⋆

(¯

x

k

)

∥

2

∑

E

[∆

k

]

∑

E

[

I

k

]

Descent of

z

∑

E

∥

¯

z

k

+1

-

z

k

+1

⋆

∥

2

Consensus of

y

∑

E

∥

ˆ

e

k

y

∥

2

∑

E

[

∆

k

n

+

I

k

]

Consensus of

z

∑

E

∥

ˆ

e

k

z

∥

2

Consensus of

x

∑

E

∥

ˆ

e

k

x

∥

2

∑

E

∥

Φ(¯

x

k

)

∥

2

Hyper-gradient estimation

∑

E

∥

E

k

u

k

-

˜

∇

Φ

(¯

x

k

)

∥

2

Hyper-gradient estimation

∑

E

∥

E

k

r

k

+1

-

˜

∇

Φ

(¯

x

k

)

∥

2

Lemma 17

∑

E

∥

E

k

u

k

-

u

k

∥

2



























































































































































































C.1.4 Technical lemmas

Lemma 1. Suppose Assumptions 1 hold, we know ∇ Φ( x ) , ˜ ∇ Φ ( x ) , z ⋆ ( x ) and y ⋆ ( x ) defined above are L ∇ Φ , ˜ L , L z ⋆ , L y ⋆ - Lipschitz continuous respectively with the constants satisfying:

And we also have:

Proof. See Lemma 1 in [2].

Remark 5. Asserting the existence of Γ with ∥ Γ ∥ < 1 , Lemma 3 only guarantees the convergence of SPARKLE . However, to obtain a precise non-asymptotic convergence rate, one must construct appropriate O and Γ . See more details in Appendix C.2.2.

C.1.5 Descent lemmas for the upper-level

In this subsection, we estimate the upper bound of the errors induced by the moving average in hyper-gradient estimation, as well as the upper bound of ∥∇ Φ( x ) ∥ 2 based on I k , ∆ k .

Lemma 4. Suppose Assumptions 1- 4 hold. We have:

Proof. See Lemma 2.2 in [22] and Lemma B.2 in [11].

Lemma 2. Suppose that g ( x ) is µ -strongly convex and L -smooth. Then for any x and 0 < α < 2 µ + L , we have

where x ⋆ = argmin g ( x ) .

Proof. See Lemma 10 in [41].

Lemma 3. Given diagonal matrices A,B,C,D ∈ R ( n -1) × ( n -1) , and

Suppose that the eigenvalues of M are strictly less than one in magnitude. Then there exist an invertible matrix O and a matrix Γ with ∥ Γ ∥ < 1 , such that M has the similarity transformation:

For the term

Lemma 5. Suppose that Assumptions 1- 4 hold. We have

Proof. For k ≥ 0 , Cauchy Schwartz inequality implies that

Then taking expectation and summation on both sides, we get

Since samples among agents are independent, it follows that

Taking expectations, we get the conclusion.

Lemma 6. Suppose that Assumptions 1- 4, and Lemmas 4, 5 hold. If

we have

Proof. The L ∇ Φ -smoothness of Φ indicates that

Then, due to

where the first equality uses 2 〈 ¯ r k , E k [¯ r k +1 ] 〉 = ∥ ¯ r k ∥ 2 + ∥ E k [¯ r k +1 ] ∥ 2 - ∥ ¯ r k -E k [¯ r k +1 ] ∥ 2 and E k ∥ ¯ r k +1 ∥ 2 = ∥ E k [¯ r k +1 ] ∥ 2 + E k ∥ ¯ r k +1 -E k [¯ r k +1 ] ∥ 2 .

Taking expectation and summation, and using α ≤ 1 2 L ∇ Ψ , we get

Since samples of different agents are independent, we have

Combining it with the conclusion of Lemma 4 and 5, we get from (42) that

Lemma 7. Suppose that Assumptions 1- 4 hold, then we have

Proof. We define u -1 = 0 for brevity. From the definition of E k , we have :

Jensen's inequality implies that

Substituting (43) into (44) , and taking expectation and summation on both sides, we get:

Finally, note that x -1 = x 0 , r 0 = 0 , and E -1 = E 0 . Subtracting θ E [ ∥ ∥ ∥ E -1 [ r 0 ] -˜ ∇ Φ (¯ x -1 ) ∥ ∥ ∥ 2 ] = θ ∥ ∥ ∥ ˜ ∇ Φ (¯ x 0 ) ∥ ∥ ∥ 2 from both sides of this equation, we get:

Lemma 8 (Descent lemma) . Suppose that Assumptions 1- 4 and Lemmas 4, 5 hold. If

then we have

Proof. The L ∇ Φ -smoothness of Φ indicates that

Taking expectation and summation on both sides, we get:

Define auxiliary series m k as:

Note that

Then using the Jenson's Inequality, we get:

It follows that for k ≥ 0

where for brevity we define ¯ u -1 = 0 .

Taking the summation on both sides from k = 0 to K , we get

On the other hand, due to the definition of m k and Jenson's Inequality, we have:

Taking the summation, we get

Combining (48) and (49), we obtain:

where the last inequality uses (47).

from both sides of (50), we have:

Substituting (47), (51) into (46), we get:

where the last inequality uses α ≤ 1 10 L ∇ Φ .

Subtracting α 2 ∑ K -1 k =0 E ∥∇ Φ(¯ x k ) ∥ 2 from both sides of (52), and substituting (38), (39) into it, we get:

Finally, multiplying 2 α on both sides, we get:

C.1.6 Descent lemmas for the lower- and auxiliary-level

The following lemmas present the error analysis of the estimation of y ⋆ (¯ x k ) and z k ⋆ , i.e., the term I k : Lemma 9 (Estimation error of y ⋆ ( x ) ) . Suppose Assumptions 1- 4hold, and:

Then we have the estimation error of y ⋆ :

and

Proof. For each k ≥ 0 , due to the independence of samples, we have:

Then,

where the first and the third inequality is due to the Jenson's inequality, the second inequality holds according to Lemma 2 and the fact that β ≤ µ g 32 L 2 g, 1 ≤ 1 3( µ g + L g, 1 ) , and the last inequality uses βµ g ≤ 1 3 . Taking the summation and expectation on the both sides, we get:

Using (36) and the fact that x 0 , y 0 is consensual, it follows that:

On the other hand,

where the second inequality uses ∇ 2 g (¯ x k , y ⋆ (¯ x k )) = 0 .

Note that β 2 ≤ µ 2 g 32 L 4 g, 1 ≤ 1 8 L 2 g, 1 . Subtracting 2 β 2 L 2 g, 1 ∥ ¯ y k +1 -¯ y k ∥ on both sides, and taking expectation and summation, we get:

where the second inequality holds since x 0 , y 0 are consensual, the third inequality uses (55), and the last inequality holds since β ≤ µ g 32 L 2 g, 1 .

Lemma 10 (Estimation error of z ⋆ ( x ) ) . Suppose that Assumptions 1- 4hold, and

We have:

Then we have:

where the first and third inequality uses Jensen's inequality and Cauchy Schwartz inequality, the second inequality holds due to Assumption 1 and γµ g < 1 , the last inequality holds since z ⋆ ( x ) is L z ⋆ Lipschitz continuous.

Moreover, the independence of samples implies that

As γ satisfies

we get:

Taking expectation and summation on both sides, we get

It follows that

since z 0 ⋆ = z 1 ⋆ and z 0 is consensual.

Then, we combine the results in Lemmas 9, 10 and give an upper bound of E [ I k ] :

Lemma 11. Suppose that Lemmas 9 and 10 hold. Then we have:

Remark 6. Here I -1 = ∥ ¯ z 0 -z 0 ⋆ ∥ 2 + κ 2 ∥ ¯ y 0 -y ⋆ (¯ x -1 ) ∥ 2 . The aim of introducing this term is to simplify the subsequent proofs of other lemmas.

Proof. Lemma 10 implies that:

Then, using Lemma 9, we have:

C.1.7 Consensus error analysis

In this subsection we aim to bound the consensus errors of y, z, x ( i .e. the terms ∥ ˆ e k y ∥ 2 , ∥ ˆ e k z ∥ 2 , and ∥ ˆ e k x ∥ 2 ).

Lemma 12 (Consensus error of y ) . Suppose that Assumptions 1- 4 hold, and

We have

Proof. Firstly, the term ∥ ˆ e k +1 y ∥ 2 can be deformed as

due to Eq.(33). Then, for the first term in the right-hand side of (60), we have:

where the first inequality uses the Jenson's inequality, the second inequality hold since ∥ ˆ U ⊤ y ∥ ≤ 1 . For the second term, we have:

For the third them, we have:

Next, for the last term, we have:

Taking expectations on both sides of (60), and plugging (61), (62), (63), (64) into it, we obtain:

Taking summation over k and using ∥ x k -¯ x k ∥ 2 ≤ ∥ O x ∥ 2 ∥ ˆ e k x ∥ 2 , ∥ y k -¯ y k ∥ 2 ≤ ∥ O y ∥ 2 ∥ ˆ e k y ∥ 2 , we get:

It follows that

Lemma 13 (Consensus error of z ) . Suppose that Assumptions 1- 4 hold, and γ satisfies

We have

Proof. Firstly, Eq. (34) implies that:

Then using Cauchy Schwartz inequality, we get

To obtain the upper bound of the right-hand side of the above equation, we first estimate some individual terms in it as follows. Note that:

and

Then we present the bound of the right-hand side of (67). For the first term, we have the following evaluations:

where the first inequality use Jensen's inequality and the second inequality use ∥ ˆ U ⊤ z ∥ ≤ 1 .

For the second term, since z k , y k +1 ∈ U k , we have:

For the third term, we have:

since ˆ e k z ∈ U k .

Next, for the last two terms, we have:

Taking the expectation ˜ E k on both sides of (67) and plugging (70), (71), (72), (73) and (74) into it, we obtain:

where the second inequality uses (36), (68), (69), and (71). Thanks to

we have:

Taking summation and expectation on both sides, we get:

Thus,

Lemma 14 (Consensus error of x ) . Suppose that Assumptions 1- 4 and Lemmas 4, 5, and 7 hold. We have

Proof. Firstly, the term ∥ ˆ e k +1 x ∥ 2 can be deformed as

due to Eq. (35).

Then, for the first term of the right-hand side of (76), we use Jensen's Inequality and get:

For the second term in the right-hand side of (76), we have:

Like (72), we have:

Next, for the last term, we have:

Taking the expectation on both sides of (76), and plugging (77), (78), (79), (80) into it, we obtain:

Taking expectation and summation on both sides, we obtain:

where the first inequality uses ˜ L - Lipschitz continuity of ˜ ∇ Φ , Lemma 7, and the second inequality uses Lemma 4 , Lemma 5 and

Hence we get

The following lemma gather the consensus analysis of x, y, z together:

Lemma 15.

Take

Suppose that Assumptions 1- 4 and Lemmas 12, 13, 14 hold, and α, β satisfy

We have:

Proof. Adding (59), (66) and (75) together, we get:

Hence:

C.1.8 Proof of the main theorem

Before giving the final result of the convergence analysis, we present the following Lemma that combines the results in the analysis of I k and ∆ k :

Lemma 16. Suppose that Assumptions 1- 4 and Lemmas 6, 11, 15 hold. If α, β, γ, θ satisfy

and

then we have:

Proof. Combining (57) and (83), we obtain

Subtracting the term

from both sides of (86) and using the restriction of α, γ in (84), we can get:

where the second inequality holds since

Then taking (41) into the concern, we know:

Since

it follows that

Then, we finish the proof of this lemma.

Finally, we can give the proof of Lemma 17, which is a detailed version of Theorem 1:

Lemma 17 (Detailed version of Theorem 1) . Suppose that Assumptions 1- 4 hold. Then there exist constant step-sizes α, β, γ, θ , such that

where σ = max { σ f, 1 , σ g, 1 , σ g, 2 } , C α , C θ are defined as:

Proof. Take L 1 = L 2 + ( θ (1 -θ ) + L ∇ Φ αθ 2 ) σ 2 g, 2 n and use the conclusion of Lemmas 8 and 16, we get:

Define:

Then we take:

and

It yields L 1 = Θ( L 2 ) , and (45), (53), (56), (40), (58), (65), (82), (84), and (85) hold. It implies that the restrictions on the step-sizes α, β, γ, θ in all previous lemma conditions hold. Thus all previous lemmas hold. We obtain:

where the last inequality uses (89).

Finally, substituting (88) and (89) into the last inequality, we can get:

where σ = max { σ f, 1 , σ g, 1 , σ g, 2 } .

Remark 7. From the proof of Lemma 17, the impact of the moving average technique on variance reduction becomes evident. The term θ n σ 2 absorb α 2 η 1 σ 2 , which includes the high order term α 4 σ 2 . Additionally, compared to y, z , the quadratic term related to σ 2 of x has an extra term θ multiplied in the numerator ( α 2 θσ 2 ). These details reduce the impacts of noise to terms related to x , confirming the conclusion that terms related to y, z dominate the rate in precious sections. Notably, taking θ < 1 is indispensable our proof. If we take θ = 1 , there would be a constant term 1 n σ 2 in the convergence rate (see the first inequality of (87) ), since the coefficient α 2 /β 2 + α 2 /γ 2 = O (1) . This would not guarantee the convergence of SPARKLE .

C.2 Analysis of consensus error and transient iteration complexity

From Lemma 17, we can immediately obtain the transient time complexity of Algorithm 1. Here we omit the impacts of the condition number κ .

Lemma 18. The transient time complexity of Algorithm 1 has an upper bound of:

Proof. According to lemma 17, SPARKLE achieves linear speedup if:

It holds when K satisfies:

Then we get:

C.2.1 Consensus Error

Lemma 19. Suppose that Assumptions 1- 4 hold. Then there exist constant step-sizes α, β, γ, θ , such that Lemma 17 holds and

where ≲ K denotes the the asymptotic rate when K →∞ .

Proof. Suppose α , β , γ , and θ satisfy the constraints given in (88) and (89), which ensures that Theorem 1 (Lemma 17) holds.

For clarity, we define the constants:

Then there exist α , β , γ , and θ that satisfy the constraints in (88) and (89), and also:

We take such values for step-sizes in the following proof.

We proceed by substituting (41) into (57), yielding:

Subtracting 4 c 1 c 2 ∑ K -1 k =0 E [ I k ] from both sides, we get:

Substituting (57) into (41), we obtain:

Subtracting c 2 c 1 ∑ K k =0 E ∥ ¯ r k ∥ 2 from both sides, we get

Taking

and combining previous results with (83), we obtain

(88) and (89) imply that

where η 1 , η 2 are defined in Lemma 15.

Then taking α, β, γ, θ such that (88), (89), (91) hold and κ 4 [( η 1 + κ 2 L 2 y ⋆ η 2 ) α 2 + η 3 ] is a sufficiently small constant, we can derive the following result:

From (88) and (89), we can determine the asymptotic orders for α, β, γ and θ when K →∞

Then we get

where ≲ K denotes the the asymptotic rate when K →∞ .

Then using (36) and the definition of ∆ k , we get

In particular, the corresponding result of SPARKLE variants that using EXTRA, ED or GT is

where ρ y , ρ z are spectrum gaps of relevant mixing matrices.

C.2.2 Essential matrix norms for analysis

Common heterogeneity-correction algorithms, including ED, EXTRA and GT, satisfy Assumption 3, according to transformations (31), (32) and discussions in [2, Appendix B.2]. Then Lemma 3 ensures that ∥ Γ ∥ < 1 . From Lemma 18, the transient time complexity depends on the coefficients ∥ O ∥ 2 , ∥ O -1 ∥ 2 , ∥ Λ a ∥ 2 , ∥ Λ -1 b ∥ 2 , and ∥ Γ ∥ 2 . The solution of these matrices is constructive. Table 4 presents the upper bounds of these coefficients with different communication modes. Please refer to [2, Appendix B.2] for more details about the construction of these matrices and the computation of relevant norms. It is required that W is positive definite for ED, EXTRA, and we denote the smallest nonzero eigenvalue of W by ρ . ρ can view as a constant. Otherwise we replace W with t I +(1 -t ) W for some constant t ∈ (0 , 1) (e.g. t = 1 / 2 ).

Substituting values of ∥ O s ∥ , ∥ O -1 s ∥ , ∥ Λ sa ∥ , ∥ Λ -1 sb ∥ , ∥ Γ s ∥ into (90) , we obtain the explicit transient iteration complexity for some specific examples of Algorithm 1, which are listed in Table 2. Note that all GT variants exhibit the same transient iteration complexity.

C.2.3 Theoretical gap between upper-level and lower-level

Note that ∥ Λ sa ∥ ≤ 1 . We rewrite the upper bound of the transient iteration complexity in Lemma 18 as where

Suppose that we use the same communication matrices and heterogeneity-correction methods for updating x, y, z , i .e.

Then we have

Now we fix the update strategies for y, z . (94) implies that we can appropriately increase δ x , ˆ δ x while keeping the transient iteration complexity (92) unchanged (at most scaled by a constant factor). For example, we can use a moderately sparser communication network for updating x than y, z . We illustrate this point with three examples: SPARKLE-ED, SPARKLE-EXTRA and SPARKLE-GT (variants), where y, z share the same communication matrix W y .

SPARKLE-ED, SPARKLE-EXTRA: From Table 4, we have

Substituting these values into (92), we get the transient iteration complexity is bounded by

SPARKLE-ED will keep the transient iteration complexity n 3 (1 -ρ ( W y )) -2 (the dominated term) if √

SPARKLE-GT variants: Results in Table 4 imply that

Following the same argument as before, we have the following upper bound of the transient iteration complexity of SPARKLE-GT

we get the constraints of the spectral gap 1 -ρ ( W x ) that maintains the transient iteration complexity max { n 3 (1 -ρ ( W y )) -2 , n (1 -ρ ( W y )) -8 3 } :

̸

Denote the communication times per agent of W x , W y by c x , c y respectively. For example, we have c x = 2 , c y = n -1 when taking Ring Graph for x ( i .e. [ W x ] ij = 0 iff | i -j | ∈ { 0 , 1 , n -1 } ), and Complete Graph for y ( i .e. W y = 1 n 1 n 1 ⊤ n ).

Then for each agent, the communication cost per round is O ( c x p + c y q ) . If we take a = c x /c y to measure the relative sparsity of the two communication matrices, and consider c y = O (1) , then for each agent, the communication cost per round is O ( ap + q ) . (95) and (96) theoretically provide the range of the sparsity (connectivity) degree of W x relative to W y . From (95) and (96), we can set a ≪ 1 , while maintaining the transient iteration complexity for SPARKLE-GT, SPARKLE-ED, SPARKLE-EXTRA.

C.2.4 The transient iteration complexities of some specific examples in SPARKLE.

Now we compute the transient iteration complexities of each SPARKLEL -U algorithm, where L , U ∈ { GT (variants) , ED , EXTRA } . For brevity, here we assume that W x = W y = W z , use the same heterogeneity-correction method to y, z , and denote the spectral gap 1 -ρ ( W x ) by 1 -ρ .

Substituting the results in Table 4 into (92) and (93), we get

for any L , U ∈ { GT (variants) , ED , EXTRA } ,

for U = { GT (variants) , ED , EXTRA } respectively, and

for L = { GT (variants) , ED , EXTRA } respectively.

Combining the above results, we can directly obtain Table 2, the transient iteration complexities of SPARKLE with mixed heterogeneity-correction techniques in different levels.

C.3 Convergence analysis in deterministic scenarios

The following lemma gives the convergence rate of Algorithm 1 without a moving average when there is no sample noise:

Lemma 20. Suppose that Assumptions 1- 4 hold. If σ 2 = 0 , then there exist α, β, γ and θ = 1 such that

where ˜ C α is a series of overheads which is defined below.

Proof. Note that σ 2 = 0 implies that L 1 = Θ( L 2 ) when α = O ( L -1 ∇ Φ ) . Thus (87) implies that:

Then we aim to choose the stepsize α, β, γ . Define:

Then there exist

such that (45), (53), (56), (40), (58), (65), (82), and (84) hold. Then all previous lemmas hold. Then from (97) we have:

C.4 Degenerating to single-level algorithms

We consider the bilevel problem with the following upper- and lower-level loss function on the i -th agent:

Actually, this optimization problem with respect to x is single-level, since we have z k ≡ 0 , y k ≡ 0 , u k i = ∇ 1 f i ( x k i , ξ k i ) by induction. By taking θ = 1 , we get the following single-level algorithm framework for decentralized stochastic single-level algorithm. As we discuss in previous sections, it can recover various heterogeneity-correction algorithms, including GT, EXTRA and ED, by selecting specific A x , B x , C x .

Algorithm 3 SPARKLE: degenerating to single-level decentralized stochastic algorithms

In this case, we have z ⋆ k ≡ 0 , y ⋆ k ≡ 0 . Notice that L y ⋆ = 0 , L z ⋆ = 0 . It gives

If we take

and θ = 1 , β → 0 , γ → 0 , then (45), (53), (56), (40), (58), (65), (82), and (84) hold. Thus all previous lemmas hold. Then (87) transforms into

It follows that

Like (88), we take

Substituting these values into (98), we get

Like Lemma 18, we get the transient iterating complexity for Algorithm 3 is

Substituting the value of relevant norms in Table 4, we get the transient iteration complexity for GT, EXTRA, ED are

respectively, where ρ := ρ ( W x ) . These upper bounds are the same as the state-of-the-art results shown in Table 1. It indicates that our analysis accurately captures the impacts of updates at each level on the convergence results.

D Experimental details

In this section, we provide the details of our numerical experiments discussed in Section 4. We also provide addition experimental results which are not mentioned in the main text due to the space limitation. For all GT variants, we focus on one typical representative, ATC-GT, in our experiments, which we denote as GT for brevity. All experiments described in this section were run on an NVIDIA A100 server.

D.1 Synthetic bilevel optimization

Here, we consider problem (1) whose upper- and lower level loss functions on the i -th agents ( 1 ≤ i ≤ N ) are denoted as:

0

1000

2000

3000

4000

5000

number of gradient evaluations

10

3

10

2

10

1

10 0

10 1

10 2

||

x

(

i

)

x

*

||

2

(a) Fully connceted, severe heterogeneity

D­SOBA

SPARKLE­GT

SPARKLE­ED

SPARKLE­EXTRA

0

1000

2000

3000

4000

5000

number of gradient evaluations

10

3

10

2

10

1

10 0

10 1

10 2

||

x

(

i

)

x

*

||

2

(b) 2D Torus, severe heterogeneity

D­SOBA

SPARKLE­GT

SPARKLE­ED

SPARKLE­EXTRA

0

1000

2000

3000

4000

5000

number of gradient evaluations

10

3

10

2

10

1

10 0

10 1

10 2

||

x

(

i

)

x

*

||

2

(c) Adjusted ring, severe heterogeneity

D­SOBA

SPARKLE­GT

SPARKLE­ED

SPARKLE­EXTRA

0

1000

2000

3000

4000

5000

number of gradient evaluations

10

3

10

2

10

1

10 0

10 1

10 2

||

x

(

i

)

x

*

||

2

(d) Fully connceted, mild heterogeneity

D­SOBA

SPARKLE­GT

SPARKLE­ED

SPARKLE­EXTRA

0

1000

2000

3000

4000

5000

number of gradient evaluations

10

3

10

2

10

1

10 0

10 1

10 2

||

x

(

i

)

x

*

||

2

(e) 2D Torus, mild heterogeneity

D­SOBA

SPARKLE­GT

SPARKLE­ED

SPARKLE­EXTRA

0

1000

2000

3000

4000

5000

number of gradient evaluations

10

3

10

2

10

1

10 0

10 1

10 2

||

x

(

i

)

x

*

||

2

(f) Adjusted ring, mild heterogeneity

D­SOBA

SPARKLE­GT

SPARKLE­ED

SPARKLE­EXTRA

where x ∈ R D , y ∈ R K and C r denotes a fixed regularization parameter. For each agent i , we firstly generate the local solution y ∗ i , x ∗ i as y ∗ i = y ∗ + ζ i and x ∗ i = A ∗ b ∗ + ξ i , where x ∗ ∼ N (0 , I K ) is a randomly generated vector, each element of A ∗ is independently sampled from N (0 , 9) . The observation ( A i , b i ) on agent i is generated in a streaming manner by A i = A ∗ + ϕ i , b i = x ∗ i + ψ i , in which each element of ϕ i ∈ R K × D and ψ i ∈ R D are independently generated by N (0 , σ 2 g ) . The terms ξ i ∼ N (0 , σ 2 h I K ) and ζ i ∼ N (0 , σ 2 h I D ) control the heterogeneity of data distributions across different agents.

We set D = 20 , K = 10 , σ g = 0 . 001 , C r = 0 . 001 . Then we set σ h = 0 . 5 to represent severe heterogeneity across agents and σ h = 0 . 1 for mild heterogeneity. We run D-SOBA, SPARKLE-GT, SPARKLE-ED, and SPARKLE-EXTRA over Ring, 2D-Torus [37], and fully connected networks with N = 64 agents. The moving-average term θ = 0 . 1 and the step-size at the t -th iteration are α t = β t = γ t = 1 / (500 + 0 . 01 t ) . The batch size is 10.

Fig. 4 illustrates the averaged estimation error ∑ N i =1 ∥ ∥ ∥ x ( t ) i -x ∗ ∥ ∥ ∥ 2 of the mentioned algorithms with different communication topology and data heterogeneity. It is observed that SPARKLE with ED, EXTRA, GT achieve better convergence performances with decentralized communication networks. Meanwhile, SPARKLE-ED and SPARKLE-EXTRA are more robust to data heterogeneity and the sparsity of network topology than SPARKLE-GT. All the results are consistent with our theoretical results.

D.2 Hyper-cleaning on FashionMNIST dataset

Here, we consider a data hyper-clean problem [44] on FashionMNIST dataset [48]. The FashionMNIST dataset consists of 60000 images for training and 10000 images for testing and we randomly split 50000 training images into a training set and the other 10000 images into a validation set.

The data hyper-cleaning problem aims to train a classifier from a corrupted dataset, in which the label of each training data is replaced by a random class number with a probability p ( i .e. the corruption rate). It can be considered as a stochastic bilevel problem (1) whose upper- and lower-level loss functions on the i -th agents ( 1 ≤ i ≤ n ) are formulated as:

0.50

0.55

0.60

0.65

0.70

test accuracy

0

500

1000

1500

2000

2500

3000

number of gradient evaluations

SPARKLE­GT

SPARKLE­ED

SPARKLE­EXTRA

SPARKLE­ED­GT

SPARKLE­EXTRA­GT

D­SOBA

MA­DSBO­GT

SPARKLE­GT

SPARKLE­ED

SPARKLE­EXTRA

SPARKLE­ED­GT

0.55

0.60

0.65

test accuracy

where ϕ denotes a training model while y denotes its parameters, L denotes the cross-entropy loss function and σ ( x ) = (1 + e -x ) -1 is the sigmoid function. D ( i ) tr and D ( i ) val denotes the training and validation set of the i -th agent, respectively. C > 0 is a fixed regularization parameter.

Data generation and experiment settings. In this experiment, we let ϕ be a two-layer MLP network with a 300-dim hidden layer and ReLU activation while y denotes its parameters. For 1 ≤ i ≤ 10 , we sample a probability distribution P i randomly by Dirichlet distribution with parameters α = 0 . 1 . The training and validation images with label i are sent to different agents according the probability distribution P i . Then D ( i ) tr and D ( i ) val are generated sufficiently heterogeneous [32]. We set C = 0 . 001 . The batch size is set to 50.

Convergence performances with different corruption rates. We set the moving-average term θ k = 0 . 2 and run D-SOBA [29], MA-DSBO-GT [10], MDBO [21] SPARKLE-GT, SPARKLE-ED, SPARKLE-EXTRA, SPARKLE-ED-GT, and SPARKLE-EXTRA-GT on an Adjusted Ring graph with n = 10 agents and p = 0 . 1 , 0 . 2 , 0 . 3 separately. The step-sizes for all the algorithms are set to α k = β k = γ k = 0 . 03 and the term η in MDBO is set to 0.5. The weight matrix of Adjust Ring W = [ w ij ] n × n satisfies:

Moreover, we run SPARKLE with ED in the lower level and auxiliary variable and gradient tracking in the upper level (i.e. SPARKLE-ED-GT) as well as SPARKLE with EXTRA in the lower level and auxiliary variable and gradient tracking in the upper level (i.e. SPARKLE-EXTRA-GT) and compare their test accuracy with the other four algorithms.

Figure 1 shows that SPARKLE-ED and SPARKLE-EXTRA outperforms in different cases than SPARKLE-GT. Meanwhile, SPARKLE-EXTRA, SPARKLE-EXTRA-GT achieve similar test accuracy, as do those for SPARKLE-ED and SPARKLE-ED-GT, which matches our theoretical results in transient iteration analysis. Figure 5 presents the times of gradient evaluation for different test accuracies of these algorithms at p = 0 . 2 , 0 . 3 , demonstrating similar results.

Influence of network topology. We set the corruption rate p = 0 . 3 , the step sizes α k = β k = γ k = 0 . 02 , and the moving-average term θ k = 0 . 2 . Then we run SPARKLE-EXTRA and SPARKLEEXTRA-GT on a network containing n = 10 nodes with different topologies in the following two cases:

Fixed upper, varied lower : x communicates through a five-peer graph; y, z communicate through different adjusted rings with ρ = 0 . 647 , 0 . 828 , 0 . 924 , 0 . 990 .

Fixed lower, varied upper : y, z communicate through a five-peer graph; x communicates through different adjusted rings with ρ = 0 . 647 , 0 . 828 , 0 . 924 , 0 . 990 .

number of gradient evaluations

2500

2000

1500

1000

500

0

0.50

SPARKLE­EXTRA­GT

D­SOBA

MA­DSBO­GT

0.70

0

500

1000

1500

2000

2500

3000

3500

number of gradient evaluations

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

test accuracy

2200

2400

2600

2800

3000

3200

3400

0.65

0.70

0.75

0.80

=0.647

=0.828

=0.924

=0.990

0

500

1000

1500

2000

2500

3000

3500

number of gradient evaluations

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

test accuracy

2200

2400

2600

2800

3000

3200

3400

0.65

0.70

0.75

0.80

=0.647

=0.828

=0.924

=0.990

0

500

1000

1500

2000

2500

3000

3500

number of gradient evaluations

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

test accuracy

2200

2400

2600

2800

3000

3200

3400

0.65

0.70

0.75

0.80

=0.647

=0.828

=0.924

=0.990

0

500

1000

1500

2000

2500

3000

3500

number of gradient evaluations

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

test accuracy

2200

2400

2600

2800

3000

3200

3400

0.65

0.70

0.75

0.80

=0.647

=0.828

=0.924

=0.990

The weight matrix of five-peer graph W = [ w ij ] n × n satisfies:

Figure 6 shows the average test accuracy of both SPARKLE-EXTRA and SPARKLE-EXTRA-GT over 10 trials. It indicates that the test accuracy decays with increasing spectral gap of topologies related to y, z while the topology of x is fixed during the whole iterations. However, such convergence gap becomes milder when the topologies of y, z are fixed and that of x varies. This phenomenon supports our theoretical findings, which suggest that the transient iteration complexity is more sensitive to the network topologies of y, z than to that of x .

Influence of moving-average iteration on convergence. Moreover, for θ t = 0 . 05 , 0 . 2 , 0 . 3 , we run SPARKLE-GT, SPARKLE-ED, SPARKLE-EXTRA, SPARKLE-ED-GT, and SPARKLEEXTRA-GT on an Adjusted Ring graph with n = 10 agents, α k = β k = γ k = 0 . 03 and p = 0 . 3 for 3000 iterations. We obtain the average test accuracy of the last 40 iterations over 10 trials, and present the mean and standard deviation during the different trials in Table 5. We can observe that most algorithms achieve the highest test accuracy when θ = 0 . 2 , which may prove that a suitable θ can benefit the test accuracy in hyper-cleaning problems.

0

250

500

750

1000

1250

1500

1750

2000

sample size

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

test loss

SPARKLE­ED

SPARKLE­EXTRA

SLDBO

MDBO

Single­level

0

250

500

750

1000

1250

1500

1750

2000

sample size

0.5

1.0

1.5

2.0

2.5

3.0

test loss

SPARKLE­ED

SPARKLE­EXTRA

SLDBO

MDBO

Single­level

D.3 Distributed policy evaluation in reinforcement learning

Following the result of [52], we consider a multi-agent MDP problem in reinforcement learning on a distributed setting with n agents. Denote S as the state space. Suppose that the value function in each state s ∈ S is a linear function V ( s ) = ϕ ⊤ s x , where ϕ s ∈ R m is a feature and x ∈ R m is a parameter. To obtain the optimal solution x ∗ , we consider the following Bellman minimization problem:

where r i ( s, s ′ ) denotes the reward incurred from transition s to s ′ on the i -th agent, γ ∈ (0 , 1) denotes the discount factor. The expectation is taken over all random transitions from state s to s ′ . It can be viewed as a bilevel optimization problem with the following upper- and lower-level loss:

where y = ( y 1 , · · · , y |S| ) ⊤ ∈ R |S| . In our experiment, we set the number of states |S| = 200 and m = 10 . For each s ∈ S , we generate its feature ϕ s ∼ U [0 , 1] m . The non-negative transition probabilities are generated randomly and standardized to satisfy ∑ s ′ ∈S p s,s ′ = 1 . The mean reward ¯ r i ( s, s ′ ) are independently generated from the uniform distribution U [0 , 1] . In each iteration, the stochastic reward r i ( s, s ′ ) ∼ N (¯ r i ( s, s ′ ) , 0 . 02 2 ) .

For n = 10 , 20 , we run SPARKLE-ED and SPARKLE-EXTRA as well as existing decentralized SBO algorithms MDBO [21] and SLDBO [16] (here we use the stochastic gradient instead of deterministic gradient) over a Ring graph. For MDBO, the number of Hessian-inverse estimation iterations is set to 5 . The step sizes are 0.03 for all methods. Figure 3 illustrates the upper-level loss against samples generated by one agent for 10 independent trials. Table 6 shows the average training loss of the last 500 iterations for 10 independent trials of the four decentralized SBO algorithms as well as single-level ED [56] (For bilevel algorithms, training loss means the upper-level loss here).

0

2000

4000

6000

8000

10000

12000

14000

time(s)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

training accuracy

SPARKLE­ED

D­SOBA

Decentralized MAML

0

2000

4000

6000

8000

10000

12000

14000

time(s)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

test accuracy

SPARKLE­ED

D­SOBA

Decentralized MAML

Both Figure 3 and Table 6 demonstrate that SPARKLE-ED and SPARKLE-EXTRA converge faster than other methods.

Finally, we create a fixed "test set" with 10000 sample generated from S . Figure 7 shows the loss on the test set of SPARKLE-ED, SPARKLE-EXTRA, SLDBO, MDBO and single-level ED algorithm, demonstrating the superior performance of SPARKLE compared to other decentralized SBO algorithms.

D.4 Decentralized meta-learning

We consider a meta-learning problem as described in [18]. There are R tasks {T s , s = 1 , · · · , R } . Each task T s has its own loss function L ( x, y s , ξ ) , where ξ s represents a stochastic sample drawn from the data distribution D s , y s denotes the task-specific parameters and x denotes the global parameters shared by all the tasks. In meta-learning problem, we aim to find the parameters ( x ∗ , y ∗ 1 , · · · , y ∗ R ) that minimizes the loss function across all R tasks, i .e.,

The problem (102) can be formulated as a decentralized SBO problem with heterogeneous data distributions across N nodes. For i = 1 , 2 , · · · , N , let D train s,i and D val s,i denote the training and validation datasets for the s -th task T s received by node i respectively. We can then address the meta-learning problem by minimizing (1), with the upper- and lower-level loss functions defined as:

where L denotes the cross-entropy loss and R ( y s ) = C r ∥ y s ∥ 2 is a strongly convex regularization function.

In this experiment, we compare SPARKLE-ED with D-SOBA [29] and MAML [18] in a decentralized communication setting over a 5-way 5-shot task across a network of N = 8 nodes connected by Ring graph. The dataset used is miniImageNet [47], derived from ImageNet [42], which comprises 100 classes, each containing 600 images of size 84 × 84 . We set R = 2000 and partition these classes into 64 for training, 16 for validation, and 20 for testing. For the training and validation classes, the data is split according to a Dirichlet distribution with parameter α = 0 . 1 [32]. We utilize a four-layer CNN with four convolution blocks, where each block sequentially consists of a 3 × 3 convolution with 32 filters, batch normalizationm, ReLU activation, and 2 × 2 max pooling. The batch size is 32 , and C r = 0 . 001 . The parameters of the last linear layer are designated as task-specific, while the other parameters are shared globally. For SPARKLE and D-SOBA, the step-sizes are

β = γ = 0 . 1 and α = 0 . 01 . For MAML, the inner step-size is 0.1 and the outer step-size is 0.001, and the number of inner-loop steps as 3. For all algorithms, the task number is set to 32. And we only repeat the experiment only once due to the time limitation. Figure 8 shows the average accuracy on the training dataset for all nodes, as well as the test accuracy of the three algorithms. We observe that SPARKLE-ED outperforms other algorithms, demonstrating the efficiency of SPARKLE in decentralized meta-learning problems.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Refer to Abstract and Introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Refer to Section Conclusions.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Refer to Section Assumptions for our assumptions, and Appendix for detailed proof.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Refer to Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We may consider making data and code openly accessible when it is deemed necessary.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Refer to Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We show error bars in experiments where we consider them essential.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Refer to Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification:Our research conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of our work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There is no such risk in the paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We comply with the licenses of existing assets used in the paper and provide necessary references.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SPARKLE, a single-loop primal-dual framework for solving bilevel optimization in the decentralized setting. Specifically, multiple devices collaborate to solve bilevel optimization problems and exchange information via communications over a network. From a theoretical angle, the authors prove that SPARKLE algorithm achieves better transient iteration complexity and handle gradient dissimilarity more effectively under weaker assumptions than previous works. From an empirical angle, the authors conduct experiments to show that their algorithm has better performance than existing results.\n\nSTRENGTHS:\n1. The authors provide a single-loop primal-dual framework for solving decentralized bilevel optimization. The technical contributions and the claims are sound, and the experiments can support the claims from an empirical perspective.\n\n2. Moreover, the framework unfies many state-of-the-art strategies used in distributed optimization, like ATC, GT, EXTRA, momentum, etc, and obtains best results as compared to existing works.\n\nWEAKNESSES:\n1. Novelty. The novelty of this paper seems a little limited. There are already many existing works proposing various algorithms for solving decentralized bilevel optimization problems, as listed in Table 1 of this paper. I am not sure if the improvement on the transient time and communication complexity over previous works is significant enough to make this work pass the bar of acceptance. The authors may want to clarify this a bit.\n\n2. Presentation. The authors also mention that there are mixed strategies in this paper like ATC, GT, EXTRA, auxiliary-level updates, momentum update, and so on. The organization of the techniques might not be easy for a layman to follow. It would be better if the authors can highlight the reasons and benefits of using these techniques in a table.\n\n3. Experiments. Many important baselines are missing in the figures of Section 4. Some figures have standard deviation of 10 trials while some do not.\n\nQUESTIONS:\nI wonder if the authors have any responses to the weaknesses I listed above.\n\nLIMITATIONS:\n1. Novelty. One major concern is the limited novelty of this paper, as mentioned in the weakness section.\n\n2. There are also some limitations mentioned by the authors. For example the lower-level problem is strongly convex, and it is unclear if the condition number is optimal in the upper bounds. I agree that these are the common limitations of existing decentralized bilevel optimization works, and are interesting to explore.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SPARKLE, a unified single-loop primal-dual framework for decentralized stochastic bilevel optimization. SPARKLE is highly versatile, which can incorporate various heterogeneity-correction techniques and allows for different strategies to solve upper- and lower-level problems. The authors provide a unified convergence analysis for SPARKLE and its variants, showing state-of-the-art convergence rates compared to existing decentralized bilevel algorithms. Theoretical findings are supported by numerical experiments.\n\nSTRENGTHS:\nThe proposed method SPARKLE in this paper is highly versatile, which can support various decentralized mechanisms and topologies across optimization levels. This paper has a comprehensive theoretical analysis for all SPARKLE variants, demonstrating that they achieve state-of-the-art convergence rates compared to existing decentralized bilevel algorithms. In addition, the authors show that the convergence performance of SPARKLE variants is comparable to their single-level counterparts, and that employing mixing strategies outperforms using GT alone.\n\nWEAKNESSES:\n1. In Line 303-304, the authors claim that all the SPARKLE-based algorithms generally achieve higher test accuracy than D-SOBA, while ED and EXTRA especially outperform GT. However, in Figure 1, particularly in the middle and right figures, the test accuracies are very similar, making such claims insufficiently supported. Additionally, such small differences might be due to randomness.\n2. The baseline algorithms used in different experiments are different. It is unclear why the authors did not use the same baselines. Furthermore, just one or two existing decentralized SBO algorithms in the hyper-cleaning or distributed policy evaluation experiments is not enough to evaluate the performance of SPARKLE. More state-of-the-art decentralized SBO algorithms need to be added.\n3. It would be important for this paper to evaluate more diverse and larger-scale tasks, including non-linear models such as neural networks (e.g., ResNet), applied to various bilevel problems (e.g., meta-learning).\n4. The authors did not provide the code to reproduce the experiments, raising concerns about reproducibility and making it difficult for other researchers to verify their results.\n\nMinor:\n1. In Assumption 1, it would be clearer to specify the variable with respect to which the function is Lipschitz continuous.\n\nQUESTIONS:\n1. The authors use the same decentralized mechanism for the lower-level and auxiliary variables. Can lower-level and auxiliary variables use different decentralized mechanisms?\n2. Could the authors provide more explanation on why SPARKLE can support utilizing different mixing matrices across levels? Are there any applications for such scenarios?\n3. The authors only show the upper level loss in the policy evaluation experiment. What about the test accuracy?\n\nLIMITATIONS:\nAs the authors stated, SPARKLE supports only strongly-convex problems in the lower-level optimization, and the condition number of the lower-level problem significantly impacts the performance.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SPARKLE, a unified framework for decentralized stochastic bilevel optimization that addresses several limitations in existing approaches. SPARKLE incorporates various heterogeneity-correction techniques, including EXTRA, Exact Diffusion, and Gradient Tracking, and allows for different strategies in upper and lower-level problems. This flexibility enables SPARKLE to outperform previous methods, achieving state-of-the-art performance in terms of convergence rate, gradient complexity, communication cost, and transient iteration complexity. Notably, the framework demonstrates that EXTRA and Exact Diffusion are more suitable for decentralized bilevel optimization than Gradient Tracking. SPARKLE provides a unified convergence analysis applicable to all its variants without requiring restrictive assumptions like bounded gradients or data heterogeneity. Through numerical experiments, the paper demonstrates that SPARKLE achieves better performance compared to existing decentralized bilevel optimization algorithms.\n\nSTRENGTHS:\n1.The SPARKLE framework's unified approach to decentralized bilevel optimization, incorporating multiple heterogeneity-correction techniques (EXTRA, Exact Diffusion, and Gradient Tracking), reveals an important insight: Gradient Tracking (GT) is not the optimal choice in some situations. \n\n2.The authors demonstrate its superiority of SPARKLE in several theoretical aspects of decentralized bilevel optimization, achieving state-of-the-art results across multiple performance metrics. The framework exhibits faster convergence rates, lower gradient complexity, reduced communication costs, and improved transient iteration complexity compared to existing methods.\n\n3.Extensive experimental results show SPARKLE consistently outperforming existing methods across various benchmark problems and real-world scenarios, particularly in terms of convergence rates. This empirical evidence strongly supports SPARKLE's practical utility and efficiency in solving complex optimization tasks in decentralized settings.\n\nWEAKNESSES:\n1.Although the paper proposes mixing strategies to handle heterogeneity, it does not provide a strong motivation or clear explanation for the necessity and benefits of these mixing strategies over choosing a single strategy, such as EXTRA or ED. This lack of clarity can make it difficult for practitioners to understand the rationale behind the proposed approach and its potential advantages.\n\n2.The primal-dual approach employed in SPARKLE incurs increased communication and storage overhead per communication round compared to non-primal-dual methods, due to the necessity of maintaining and updating dual variables in addition to primal variables across the decentralized network.\n\n3.The paper appears to lack an explicit analysis of consensus error, a crucial metric for evaluating solution agreement across nodes in decentralized networks. This omission represents a significant gap in the framework's evaluation, potentially limiting our understanding of SPARKLE's effectiveness in achieving consistent solutions in distributed optimization scenarios.\n\nQUESTIONS:\n1.Over an undirected time-invariant graph with non-negative, symmetric and doubly stochastic mixing matrices, do the algorithms SPARKLE with GT, EXTRA, and ED have different ranges of application? If so, how do their application ranges differ in the context of decentralized bilevel optimization?\n\n2.Can the convergence rate of SPARKLE be improved beyond the results presented in Table 1, considering that the algorithm without momentum step in reference [27] achieves comparable convergence results in single-level scenario? In the absence of the momentum step, how would the convergence results of Algorithm 1 be affected?  If the primal-dual framework is removed, how would the convergence results of Algorithm 1 be affected? \n\n3.Could the paper provide a detailed comparison of the computational time and communication rounds required to achieve specific test accuracy or loss thresholds among the SPARKLE algorithms with different heterogeneity-correction techniques in experimental results?\n\nLIMITATIONS:\n1.By requiring strong convexity in the lower-level problem, SPARKLE excludes a wide range of practical scenarios where the lower-level optimization may be non-convex or only generally convex. This constraint substantially narrows the scope of decentralized bilevel optimization problems that SPARKLE can effectively address.\n\n2.The performance of SPARKLE is significantly impacted by the condition number of the lower-level problem. This sensitivity could limit its effectiveness in certain problem scenarios.\n\n3.The experiments seem to lack a comparison with single-level algorithms, which could provide valuable insights into the relative performance of SPARKLE in the distributed policy evaluation in reinforced learning. While the study demonstrates SPARKLE's superior performance against other decentralized stochastic bilevel optimization (DSBO) approaches like MDBO and SLDBO, it fails to benchmark against established single-level methods.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper studies a primal-dual framework for decentralized bilevel optimization. It unifies several heterogeneous correction techniques (gradient tracking and EXTRA). It also provides a shared rate analysis that applies to all variants and avoids several assumptions like gradient boundedness. Several other insights include deploying ED/EXTRA at lower level is more beneficial than using GT across both upper/lower levels.

STRENGTHS:
- This paper is overall well written with clear introduction of the bilevel optimization problem and shows the comparison of many different approaches and how they fit under the unified framework. An unified convergence rate is provided and discussed in detail when specializing in different settings. 
- Several interesting insight is provided, including the benefit of using EXTRA and ET compared to GT.

WEAKNESSES:
- The unification of several algorithms seems a superficial contribution. One just have to make a few unifying notations to unified these into the same framework. Probably the more non-trivial part is a unified analysis of the unified algorithm, but the paper doesn't seem to make the case that analyzing the unified algorithm is more difficult or more challenging than analyzing the individual algorithms. 

- The rate analysis in Theorem 1 contains many  numerical constants like 16/3, 14/3, which doesn't seem to be tight. Can the author comment all on the tightness of these constants. 

Due to the above reasons, I would say the paper's contribution is marginal.

QUESTIONS:
See weakness.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12