Title: Weight for Robustness: A Comprehensive Approach towards Optimal Fault-Tolerant Asynchronous ML
Venue: NeurIPS 2024
Accepted: yes
Keywords: stochastic convex optimization

byzantine robust learning

online convex optimization
Abstract: We address the challenges of Byzantine-robust training in asynchronous distributed machine learning systems, aiming to enhance efficiency amid massive parallelization and heterogeneous compute resources. Asynchronous systems, marked by independently operating workers and intermittent updates, uniquely struggle with maintaining integrity against Byzantine failures, which encompass malicious or erroneous actions that disrupt learning. The inherent delays in such settings not only introduce additional bias to the system but also obscure the disruptions caused by Byzantine faults. To tackle these issues, we adapt the Byzantine framework to asynchronous dynamics by introducing a novel weighted robust aggregation framework. This allows for the extension of robust aggregators and a recent meta-aggregator to their weighted versions, mitigating the effects of delayed updates. By further incorporating a recent variance-reduction technique, we achieve an optimal convergence rate for the first time in an asynchronous Byzantine environment. Our methodology is rigorously validated through empirical and theoretical analysis, demonstrating its effectiveness in enhancing fault tolerance and optimizing performance in asynchronous ML systems.

Weight for Robustness: A Comprehensive Approach towards Optimal Fault-Tolerant Asynchronous ML

Abstract

We address the challenges of Byzantine-robust training in asynchronous distributed machine learning systems, aiming to enhance efficiency amid massive parallelization and heterogeneous computing resources. Asynchronous systems, marked by independently operating workers and intermittent updates, uniquely struggle with maintaining integrity against Byzantine failures, which encompass malicious or erroneous actions that disrupt learning. The inherent delays in such settings not only introduce additional bias to the system but also obscure the disruptions caused by Byzantine faults. To tackle these issues, we adapt the Byzantine framework to asynchronous dynamics by introducing a novel weighted robust aggregation framework. This allows for the extension of robust aggregators and a recent metaaggregator to their weighted versions, mitigating the effects of delayed updates. By further incorporating a recent variance-reduction technique, we achieve an optimal convergence rate for the first time in an asynchronous Byzantine environment. Our methodology is rigorously validated through empirical and theoretical analysis, demonstrating its effectiveness in enhancing fault tolerance and optimizing performance in asynchronous ML systems.

1 Introduction

In recent years, there has been significant growth in the development of large-scale machine learning (ML) models and the volume of data they require [Zhao et al., 2023]. To efficiently accelerate largescale training processes, Distributed ML has emerged as a crucial approach that can be categorized into synchronous and asynchronous paradigms. In synchronous learning, workers update the model simultaneously using the average of their outputs, similar to the Minibatch approach [Dekel et al., 2012]. Asynchronous learning, however, allows workers to operate independently, sending updates as they are ready without waiting for others [Arjevani et al., 2020]. This prevents slow workers from hindering the process, making it especially practical as the number of workers increases.

A major challenge of distributed ML is fault-tolerance, and Byzantine ML [Alistarh et al., 2018, Lamport et al., 2019, Guerraoui et al., 2023] is a powerful framework for tackling this aspect. Byzantine ML captures a broad spectrum of failures within distributed environments, including random malfunctions or even malicious workers aiming to disrupt the training process. This makes Byzantine ML widely applicable across various domains to ensure robust performance.

Addressing the Byzantine problem in synchronous distributed learning is well-established [Karimireddy et al., 2020, 2021, Allouah et al., 2023, Farhadkhani et al., 2022, Alistarh et al., 2018, Dahan and Levy, 2024]. Two primary ingredients were found to be crucial towards tackling Byzantine ML in synchronous settings: (i) Robust Aggregators [Yin et al., 2018, Blanchard et al., 2017, Chen et al., 2017]: such aggregators combine the gradient estimates sent by the workers to a single estimate

while filtering out the outliers which may hinder the training process. While the use of robust aggregators is crucial, it was found to be insufficient, and an additional ingredient of (ii) learning from history was shown to be vital in mitigating Byzantine faults [Karimireddy et al., 2021]. And, the performance of robust aggregators was systematically explored within a powerful generic framework [Karimireddy et al., 2020, 2021, Allouah et al., 2023, Farhadkhani et al., 2022, Dahan and Levy, 2024]. Moreover, due to the diversity of Byzantine scenarios [Xie et al., 2020a, Allen-Zhu et al., 2020, Baruch et al., 2019], it was found that relying on a single aggregator is insufficient, making the variety of robust aggregators essential. Unfortunately, many existing aggregators have sub-optimal performance. This drawback was elegantly resolved by the design of meta-aggregators [Karimireddy et al., 2020, Allouah et al., 2023, Dahan and Levy, 2024], that enable to boost the performance of baseline aggregators. Unfortunately, in the asynchronous case, the use of robust aggregators is not straightforward, as updates are typically applied individually per-worker, rather than averaging outputs from all workers at once [Arjevani et al., 2020].

Despite its advantages, asynchronous distributed learning presents unique challenges, particularly when dealing with Byzantine faults. The delays inherent in asynchronous settings introduce additional bias to the system and obscure the disruptions caused by Byzantine faults. In fact, in contrast to the synchronous Byzantine setting, all existing approaches towards the asynchronous Byzantine case do not ensure a generalization error (excess loss) that diminishes with the number of honest data-samples and updates. This applies to works for both convex [Fang et al., 2022] as well as non-convex scenarios [Xie et al., 2020b, Yang and Li, 2023]; as well as to works that further assume the availability of a trusted dataset possessed by the central-server [Xie et al., 2020b, Fang et al., 2022]. Furthermore, the performance guarantees of all existing approaches towards that setting include an explicit dependence on the dimension of the problem - a drawback that does not exist for SOTA synchronous Byzantine approaches.

Contributions. We explore the asynchronous Byzantine setting under the fundamental framework of Stochastic Convex Optimization (SCO) [Hazan et al., 2016]. Our work is the first to achieve a convergence rate that diminishes with the number of honest data samples and updates and does not explicitly depend on the problem's dimension. In the absence of Byzantine workers, our rate matches the optimal performance of Byzantine-free asynchronous settings. This stands in contrast to previous efforts on Byzantine, which did not attain diminishing rates or dimensionality independence, even without Byzantine workers. We also show the effectiveness of our approaches in practice. Our contributions:

We quantify the difficulty in asynchronous scenarios by considering the number of Byzantine updates , which is more natural than the standard measure of number of Byzantine workers .

We identify the need to utilize weighted aggregators rather than standard ones in favor of asynchronous Byzantine problems. Towards doing so, we extend the robust aggregation framework to allow and include weights and develop appropriate (weighted) rules and a meta-aggregator.

Achieving Optimal Convergence : We incorporate our weighted robust framework with a recent double momentum mechanism, leveraging its unique features to achieve an optimal convergence rate for the first time in asynchronous Byzantine ML.

Related Work. A long line of studies has explored the synchronous Byzantine setting (see e.g., Alistarh et al. [2018], Karimireddy et al. [2020, 2021], Allouah et al. [2023], Farhadkhani et al. [2022], Allen-Zhu et al. [2020], El Mhamdi et al. [2021], Dahan and Levy [2024]). Alistarh et al. [2018], Karimireddy et al. [2021] demonstrated that historical information is crucial for optimal performance in Byzantine scenarios; and Karimireddy et al. [2021] introduced the idea of combining generic aggregation rules, together with standard momentum with a parameter of 1 / √ T to effectively incorporates √ T iterations of historical gradients. Additionally, Dahan and Levy [2024] showed that a double momentum approach is effective by taking a momentum parameter of 1 /T , capturing the entire gradient history.

Robust aggregators such as Coordinate-wise Trimmed Mean (CWTM) [Yin et al., 2018], Krum [Blanchard et al., 2017], Geometric Median (GM) [Chen et al., 2017], CWMed [Yin et al., 2018], and Minimum Diameter Averaging [Guerraoui et al., 2018] have also proven to be highly beneficial in synchronous settings and have been evaluated within robust frameworks [Allouah et al., 2023, Karimireddy et al., 2020, Farhadkhani et al., 2022, Dahan and Levy, 2024]. However, not all robust aggregators achieve optimal performance, leading to the development of meta-aggregators

[Karimireddy et al., 2020, Allouah et al., 2023, Dahan and Levy, 2024] to enhance their effectiveness. While standard aggregation works well in synchronous settings, where outputs are averaged across all workers, it is less suitable for asynchronous settings, where updates are processed individually as they arrive [Arjevani et al., 2020].

To adapt these approaches to asynchronous settings, Yang and Li [2023] devised BASGDm, an extension of BASGD [Yang and Li, 2021], that groups worker momentums into buckets that are then aggregated using a robust aggregator. Other methods, like Zeno++ [Xie et al., 2020b] and AFLGuard [Fang et al., 2022], rely on a trusted dataset on the central server, which hinders their practicality. Kardam [Damaskinos et al., 2018] uses the Lipschitzness of gradients to filter out outliers. Unfortunately, none of these approaches ensure a generalization error (excess loss) that diminishes with the number of honest data-samples and updates, and suffers from an explicit dependence on the problem's dimension. And this applies even in the absence of Byzantine faults.

Asynchronous Byzantine ML faces unique challenges as inherent delays add bias that obscures Byzantine disruptions. To mitigate this delay-bias in asynchronous, non-Byzantine scenarios, Cohen et al. [2021], Aviv et al. [2021] propose methods to keep model weights relatively close during iterations. Other approaches [Stich and Karimireddy, 2019, Arjevani et al., 2020, Mishchenko et al., 2022] suggest adjusting the step size proportionally to the delay. These strategies have proven useful in reducing the negative impact of delays, and achieve optimal performance.

Our work extends several concepts from Dahan and Levy [2024] to the asynchronous scenario. We devise a novel generalization of their Centered Trimmed Meta Aggregator (CTMA) towards weighted meta-aggregation, making it amenable to asynchronous scenarios. In the spirit of Dahan and Levy [2024], we also adopt a recent variance reduction technique called µ 2 -SGD [Levy, 2023]. Nevertheless, while Dahan and Levy [2024] used this technique in a straightforward manner, we found it crucial to appropriately incorporate individual per-worker weights to overcome the challenge of asynchronicity in Byzantine ML.

2 Setting

Our discussion focuses on the minimization of a smooth convex objective f : K → R :

where K ⊆ R d is a compact convex set and D denotes an unknown distribution from which we can draw i.i.d samples { z t ∼ D} t . Our work considers first-order methods that iteratively utilize gradient information to approach an optimal point. Such methods output a solution x T , which is evaluated by the expected excess loss:

where x ∗ is a solution that minimizes f over K and x T ∈ K approximates this optimal solution.

Asynchronous Training. We explore these methods within a distributed environment involving multiple workers. Our discussion focuses on a centralized distributed framework characterized by a central Parameter Server ( PS ) that may communicate with m workers. Each of these workers may draw i.i.d. samples z ∼ D ; and based on these samples, compute unbiased gradient estimate g ∈ R d at a point x ∈ K . Concretely, a worker may compute g := ∇ f ( x ; z ) ; implying that E [ g | x ] = ∇ f ( x ) . Specifically, our main focus is on Asynchronous systems, where the PS does not wait to receive the stochastic gradient computations from all machines; instead, it updates its model whenever a worker completes a (stochastic) gradient computation. That worker then proceeds to compute a gradient estimate for the updated model, while the other workers continue to compute gradients based on 'stale' models. This staleness leads to the use of staled (and therefore biased) gradient estimates, which is a major challenge in designing and analyzing asynchronous training methods.

Asynchronous Byzantine Framework. We assume that an unknown subset of the m workers are Byzantine , implying that these workers may transmit arbitrary or malicious information during the training process, and these "Byzantine" workers may even collaborate to disrupt the training. We assume that the fraction of updates that arrive from Byzantine workers during the asynchronous training process is bounded and strictly smaller than 1 / 2 and denote this fraction by λ .

Remark 2.1 (Fraction of Byzantine Updates vs. Byzantine Workers) . In both synchronous and asynchronous settings, it is common to consider a bound on the fraction of Byzantine workers (up

to 1 /2) [Allouah et al., 2023, Farhadkhani et al., 2022, Karimireddy et al., 2020, 2021, Yang and Li, 2023, 2021, Damaskinos et al., 2018]. In synchronous scenarios this is meaningful since the server equally treats the information from all workers; which is done by equally averaging gradients of all workers in each iteration in a mini-batch fashion [Dekel et al., 2012]. Conversely, in asynchronous scenarios, faster workers contribute to more updates than slower workers, leading to an unequal influence on the training process. In such scenarios, the fraction of Byzantine workers is less relevant; and it is therefore much more natural to consider the fraction of Byzantine updates . Interestingly, our definition aligns with the standard one (for the synchronous case), which considers the number of Byzantine workers.

Notation. For each worker i ∈ [ m ] and iteration t , s ( i ) t represents the total number of updates by worker i up to t , and τ ( i ) t is the delay compared to the current model. t ( i ) is the last update before t , making τ ( i ) t the time since the second last update (Figure 1). τ t denotes the delay for the worker arriving at iteration t , i.e., if worker j arrives at iteration t then τ t = τ ( j ) t .

t

t

(

i

)

t

-

τ

(

i

)

t

τ

(

i

)

t

For a given time (iteration) t , let t ( i ) be the last iteration when worker i made an update. We denote d ( i ) t := d t ( i ) , g ( i ) t := g t ( i ) , ˜ g ( i ) t := ˜ g t ( i ) , and x ( i ) t = x t ( i ) , where the latter are individual vectors that we will later define for any worker i . Throughout, ∥·∥ represents the L 2 -norm. For any natural N , [ N ] = { 1 , . . . , N } . We use the compressed sum notation α 1: t = ∑ t k =1 α k . For every x ∈ R d , the orthogonal projection of x onto a set K is denoted by Π K ( x ) = arg min y ∈K ∥ y -x ∥ 2 . We denote B and G as the subsets of Byzantine workers and honest workers, respectively, such that | m | = |G| + |B| .

Assumptions. We use the following conventional assumptions:

Bounded Diameter : we assume there exists D > 0 such that max x , y ∈K ∥ x -y ∥ ≤ D. (1)

,

Bounded Variance : there exists σ > 0 such that ∀ x ∈ K , z ∈ Support {D}

Expectation over Smooth Functions : we assume that f ( · ) is an expectation of smooth functions, i.e. ∀ x , y ∈ K , z ∈ Support {D} there exist L > 0 such that,

The above assumption also implies that the expected loss f ( · ) is L smooth. Bounded Smoothness Variance [Levy, 2023]: in Appendix A we show that Eq. (3) implies that, ∀ x , y ∈ K , z ∈ Support {D} there exists σ 2 ∈ [0 , L 2 ] such,

L

Bounded Delay : ∃ K > 0 such that for each worker i ∈ [ m ] , τ ( i ) min ≤ τ ( i ) t ≤ Kτ ( i ) min (5)

where τ ( i ) min is the minimum delay of worker i . K bounds the variance of the delay for each worker.

where t B is the total number of iterations made by Byzantine workers up to iteration t .

Sample-Arrival Independence : we assume that the delays in the system (i.e. τ ( i ) t 's) are independent of the data samples. This is a standard assumption in asynchronous training scenarios, see e.g., Arjevani et al. [2020], Aviv et al. [2021].

3 Weighted Robust Aggregation Rules

As we have mentioned, robust aggregation rules have played a major role in designing fault-tolerant ML training methods for synchronous settings (see, e.g., Allouah et al. [2023], Karimireddy et al. [2020, 2021], Dahan and Levy [2024]). These existing aggregation rules treat inputs from all workers equally, which makes sense in synchronous cases where all workers contribute the same number of updates and data samples. Conversely, this symmetry breaks down in asynchronous settings, where faster (honest) workers contribute more updates and samples compared to slower workers.

Inspired by this asymmetry, we have identified the need to define a notion of weighted robust aggregators that generalizes the standard definition of robust aggregators. In this section, we provide such a definition, derive weighted variants of standard aggregators that satisfy our new definition, and design a generic meta-approach to derive optimal weighted aggregation rules. Later, in Section 4, we demonstrate the benefits of using weighted robust aggregators as a crucial building block in designing asynchronous fault-tolerant training methods (see Alg. 2).

3.1 Robust Weighted Aggregation Framework

Below, we generalize the definition introduced by Dahan and Levy [2024], Karimireddy et al. [2020, 2021] to allow and associate weights to the inputs of the robust aggregation rule, therefore allowing the aggregator to unequally treat its inputs.

Definition 3.1. ( c λ , λ ) -weighted robust . Assume we have m random vectors x 1 , . . . , x m ∈ R d and corresponding weights s 1 , . . . , s m > 0 . Also assume we have an "honest" subset G ⊆ [ m ] , implying { x i } ∈ G are independent of each other. Finally, assume that there exists λ ∈ [0 , 1 / 2 ) such that ∑ i ∈G s i ≥ (1 -λ ) s 1: m . Moreover, assume that for any i ∈ G there exist ρ i ≥ 0 such that,

Then an aggregation rule A ω is called ( c λ , λ ) -weighted robust if for any such m random vectors and weights and λ ≥ 0 , it outputs ˆ x ←A ω ( x 1 , . . . , x m ; s 1 , . . . , s m ) such that,

for some c λ ≥ 0 . Above, ¯ x G := 1 ∑ i ∈G s i ∑ i ∈G s i x i , ρ 2 := 1 ∑ i ∈G s i ∑ i ∈G s i ρ 2 i , and the expectation is w.r.t. { x i } i m =1 and (possible) randomization in the A ω .

Here, λ represents the fraction of the sum of the non-honest vectors' weights, unlike the unweighted definition (in synchronous cases) [Karimireddy et al., 2020, 2021, Allouah et al., 2023, Farhadkhani et al., 2022] where it indicates the fraction of non-honest vectors. These definitions align when all weights are equal [Dahan and Levy, 2024]. Similarly to the unweighted version, the optimal c λ should be c λ ≤ O ( λ ) [Dahan and Levy, 2024].

Remark 3.1. Note that our definition is generic and may be applied in both convex and non-convex scenarios. Moreover, it is natural to consider such weighted aggregators beyond asynchronous settings. For example, in synchronous settings where workers have varying batch sizes, weighted aggregation based on batch sizes may be more effective than uniform aggregation.

Next, we present two weighted variants of standard (non-weighted) aggregators that satisfy the above definition (we defer the proof into Appendix C). Table 1 summarizes their c λ values.

3.2 Weighted Variant of Geometric Median and Coordinate-Wise

Weighted Geometric Median (WeightedGM) The Weighted Geometric Median (WeightedGM) minimizes the weighted sum of Euclidean distances to a set of points. Formally, for points { x i } i m =1 and corresponding weights { s i } i m =1 , WeightedGM ∈ arg min y ∈ R d ∑ i ∈ [ m ] s i ∥ y -x i ∥ .

Weighted Coordinate-Wise Median (WeightedCWMed) The Weighted Coordinate-Wise Median (WeightedCWMed) aggregates multi-dimensional data by finding the weighted median of each coordinate separately. Thus, for given coordinate if { x i } i m =1 are sorted and weights { s i } i m =1 , the weighted median x j ∗ is the element where: j ∗ = arg min j ∈ [ m ] { ∑ i ∈ [ j ] s i > 1 2 ∑ i ∈ [ m ] s i } . If ∑ j i =1 s i = 1 2 ∑ m i =1 s i for some j , then: WeightedMedian = x j + x j +1 2 .

Algorithm 1 Weighted Centered Trimmed Meta Aggregator ( ω -CTMA)

1: Input: Set of vectors { x i } i m =1 , weights { s i } i m =1 , threshold parameter λ ∈ [0 , 1 / 2 ) ,

3: Sort the sequence {∥ x i -x 0 ∥} i m =1 in non-decreasing order, and then reindex { x i } i ∈ [ m ] and their corresponding weights { s i } i ∈ [ m ] according to this new order.

2: ( c λ , λ ) -weighted robust aggregated vector x 0 ←A ω ( { x i } i m =1 ; { s i } i m =1 ) .

4: Define S ← set of indices corresponding to the first j ∗ elements in the sorted sequence, where j ∗ is the smallest j ∈ [ m ] for which ∑ i ∈ [ j ] s i ≥ (1 -λ ) ∑ i ∈ [ m ] s i .

5: Set s m +1 ← (1 -λ ) ∑ i ∈ [ m ] s i -∑ i ∈ [ j ∗ -1] s i , x m +1 ← x j ∗ , S ← ( S \{ j ∗ } ) ∪{ m +1 } .

6: Compute the weighted sum: ˆ x ← ( 1 / ∑ i ∈ S s i ) ∑ i ∈ S s i x i .

7: Output: ˆ x

3.3 Weighted Centered Trimmed Meta Aggregator ( ω -CTMA)

Table 1 illustrates that ω -GM and ω -CWMed fail to achieve the desired optimal c λ = O ( λ ) ; typically for λ ≤ 1 / 3 , their c λ remains ≤ O (1) . To address this suboptimality, we propose ω -CTMA, a weighted extension of the Centered Trimmed Meta Aggregator (CTMA) [Dahan and Levy, 2024]. This extension enables us to achieve the optimal bound c λ ≤ O ( λ ) for λ ≤ 1 / 3 (see Table 1).

The ω -CTMA algorithm (Algorithm 1) operates on a set of vectors along with their associated weights, a threshold λ ∈ [0 , 1 / 2 ) , and a ( c λ , λ ) -weighted robust aggregator. It sorts the distances between each vector and the weighted robust aggregator, trims the set based on the threshold to satisfy ∑ i ∈ S s i = (1 -λ ) s 1: m , and calculates a weighted average of the vectors, excluding outliers based on their proximity to an anchor point-the weighted robust aggregator.

Lemma 3.1. Under the assumptions outlined in Definition 3.1, if ω -CTMA receives a ( c λ , λ ) -weighted robust aggregator, A ω ; then the output of ω -CTMA, ˆ x , is (60 λ (1 + c λ ) , λ ) -robust.

For the complete analysis, please refer to Appendix C.2. Like CTMA [Dahan and Levy, 2024], ω -CTMA is highly efficient, with a computational complexity of O ( dm + m log m ) , similar to ω -GM, ω -CWMed, and weighted averaging, differing by at most an additional logarithmic factor.

4 Asynchronous Robust Training

We leverage the µ 2 -SGD algorithm [Levy, 2023], a double momentum mechanism that enhances variance reduction. By seamlessly incorporating our weighted robust framework as a black box into the µ 2 -SGD, we derive an optimal asynchronous Byzantine convergence rate.

µ 2 -SGD: The µ 2 -SGD is a variant of standard SGD, incorporating several key modifications in its update rule:

Here, { α t > 0 } t are importance weights that emphasize different update steps, with α t ∝ t to place more weight on recent updates. The sequence { x t } t represents weighted averages of the iterates { w t } t , and d t is an estimate of the gradient at the average point, ∇ f ( x t ) , differing from standard SGD, which estimates gradients at the iterates, ∇ f ( w t ) .

This approach relates to Anytime-GD [Cutkosky, 2019], which is strongly connected to momentum and acceleration concepts [Cutkosky, 2019, Kavis et al., 2019]. While the stochastic version of Anytime-GD typically uses the estimate ∇ f ( x t ; z t ) , µ 2 -SGD employs a variance reduction mechanism to produce a corrected momentum estimate d t [Cutkosky and Orabona, 2019]. Specifically, d 1 = ∇ f ( x 1 ; z 1 ) , and for t > 2 :

Algorithm 2 Asynchronous Robust µ 2 -SGD

1: Input: learning rate η t > 0 , starting point x 1 ∈ K , number of steps T , importance weights { α t } t , momentum correction weights { β t } t , ( c λ , λ ) -robust weighted aggregation function A ω . 2: Initialize: ∀ i ∈ [ m ] , set s ( i ) 0 = 0 . Set w 1 = x 1 . Each honest worker i ∈ G draws z ( i ) ∼ D and set d ( i ) 1 = ∇ f ( x 1 ; z ( i ) ) . 3: for t = 1 to T do ▷ Server update 4: Receive d t -τ t from worker i ∈ [ m ] and update: 5: d ( i ) t = d t -τ t , s ( i ) t = s ( i ) t -1 +1 ; ∀ j = i : set s ( j ) t = s ( j ) t -1 , and for t > 1 : d ( j ) t = d ( j ) t -1 ; 6: Update server model: 7: w t +1 = Π K ( w t -η t α t A ω ( { d ( j ) t , s ( j ) t } j m =1 ) ) , & x t +1 = 1 ∑ t +1 k =1 α k ∑ t +1 k =1 α k w k 8: Send x t to worker i . If i is an honest worker, it performs the following update: 9: Worker i draws z t ∼ D , computes g t = ∇ f ( x t ; z t ) , & ˜ g t -τ t = ∇ f ( x t -τ t ; z t ) , 10: and updates: d t = g t +(1 -β t )( d t -τ t -˜ g t -τ t ) ▷ Worker update 11: end for 12: Output: x T

̸

̸

Here, β t ∈ [0 , 1] are corrected momentum weights. It can be shown by induction that E [ d t ] = E [ ∇ f ( x t )] ; however, in general, E [ d t | x t ] = ∇ f ( x t ) , unlike standard SGD estimators. Nevertheless, [Levy, 2023] demonstrates that choosing corrected momentum weights β t := 1 /t results in significant error reduction, with E ∥ ε t ∥ 2 := E ∥ d t -∇ f ( x t ) ∥ 2 ≤ O (˜ σ 2 /t ) at step t , where ˜ σ 2 ≤ O ( σ 2 + D 2 σ 2 L ) . This indicates that variance decreases with t , contrasting with standard SGD where the variance E ∥ ε SGD t ∥ 2 := E ∥ g t -∇ f ( x t ) ∥ 2 remains uniformly bounded.

4.1 Asynchronous Robust µ 2 -SGD

Building upon these, we integrate the µ 2 -SGD with a ( c λ , λ ) -weighted robust aggregator A ω , as described in Alg. 2. At each iteration t ∈ [ T ] , the global PS receives an output from a certain worker and aggregates all workers' recent updates { d ( i ) t } m i =1 by employing weights accordingly to the number of updates of each worker { s ( i ) t } m i =1 . An honest worker i arriving at iteration t returns its corrected momentum d ( i ) to the PS , computed as:

t

where g t := ∇ f ( x t ; z t ) , and ˜ g t -τ t := ∇ f ( x t -τ t ; z t ) . Afterwards, the PS performs the AnyTime update step as follows:

In the spirit of Levy [2023], Dahan and Levy [2024], we suggest employing β t := 1 /s t , which effectively considers the entire individual gradient's history of each worker; this translates to a stochastic error bound of ∥ ε ( i ) t ∥ ≤ O (˜ σ/s t ) for an honest worker i arriving at iteration t . To achieve an error corresponding to the total number of honest iterations t G , specifically ∥ ε t ∥ ≤ O (˜ σ/t G ) , as in the non-distributed setting [Levy, 2023], a weighted collective error across all honest workers should be considered with weights determined by the number of honest worker arrivals, as detailed in Theorem 4.1. The unique characteristics of µ 2 -SGD make it well-suited for the asynchronous Byzantine setting, where λ < 1 / 2 relates to the fraction of Byzantine iterations. The total iteration number t matches the sum of the workers' frequencies ( ∑ i ∈ [ G ] s ( i ) t = t G ), aligning with the weighted robust definition in Definition 3.1. Using other approaches like momentum [Karimireddy et al., 2020, 2021, Allouah et al., 2023] is less straightforward in the asynchronous Byzantine setting with the weighted robust definition. This complexity arises because an individual honest error ∥ ε ( i ) t ∥ ≲ O (˜ σ/ √ s t ) implies that weights should be √ s t instead of s t , which can be more challenging.

Remark 4.1 (Memory and Computational Overhead of Algorithm 2) . Algorithm 2 incurs additional memory and computational costs compared to the asynchronous Byzantine-free setting [Arjevani et al.,

2020], where the server stores only one worker's output and the global model. Algorithm 2 stores the latest outputs from all workers, increasing memory usage to O ( dm ) . Robust aggregation methods like ω -CWMed [Yin et al., 2018] and ( ϵ -approximate) ω -GM [Chen et al., 2017, Acharya et al., 2022] add a computational cost of O ( dm log m ) and O ( dm + dϵ -2 ) , respectively. This is in contrast to Byzantine-free settings where worker outputs are used directly without aggregation. Comparable overheads are observed in synchronous Byzantine-resilient methods, which similarly aggregate outputs from all workers. This reflects a necessary trade-off: achieving robustness inherently requires leveraging information from all workers to counteract the influence of potentially faulty ones.

Theorem 4.1. For a convex set K with bounded diameter D and a function f : K ↦→ R , and assume the assumptions in Equations (2) , (3) , (4) . Then Alg. 2 with parameters { α t = t } t and { β t = 1 /s t } t ensures the following for every t ∈ [ T ] and each honest worker i ∈ G :

where ε ( i ) t = d ( i ) t - ∇ f ( x ( i ) t ) , ˜ σ 2 = 2 σ 2 + 32 D 2 K 2 σ 2 L , and t G is the total number of honest iterations up to the t th iteration.

Proof Sketch of Thm. 4.1. The complete analysis is provided in App. B.1. It involves several key steps for an honest i worker who arrives at iteration t :

Following Lemma B.1, the distance between successive query points: ∥ x ( i ) t -x ( i ) t -τ t ∥ ≤ 4 K s ( i ) t -1 D

We analyze the recursive dynamics of the error term ε ( i ) t by setting β t = 1 ( i )

s t and obtain: ,

where Z ( i ) t := g ( i ) t - ∇ f ( x ( i ) t ) -(˜ g ( i ) t -τ t - ∇ f ( x ( i ) t -τ t )) . Unrolling this recursion provides an explicit expression: s ( i ) t ε ( i ) t = ∑ k ∈ [ s ( i ) t ] M ( i ) k , where M ( i ) s ( i ) t := g ( i ) t -∇ f ( x ( i ) t ) + ( s t -1) Z ( i ) t ; thus, {M ( i ) k } k ∈ [ s ( i ) ] is a martingale difference sequence.

t

Employing the above with Eq. (2) and (4), we have: E ∥M ( i ) k ∥ 2 ≤ 2 σ 2 +32 D 2 K 2 σ 2 L = ˜ σ 2 .

Leveraging the properties of a martingale difference sequence, we have:

Remark 4.2. Compared to synchronous scenarios [Levy, 2023, Dahan and Levy, 2024], the variance ˜ σ in Thm. 2 additionally includes the variance in the delay, denoted as K (Eq. (5) ). In balanced scheduling methods, like Round Robin [Langford et al., 2009], the impact of K on the error becomes minor, as the delay τ ( i ) t = m is constant. In the case of constant delays, the factor K equals 1 .

Lemma 4.1. Let A ω be ( c λ , λ ) -weighted robust aggregation rule and let f : K ↦→ R , where K is a convex set with bounded diameter D , and presume that the assumption in Equations (2) , (3) , (4) hold. Then invoking Alg. 2 with { α t = t } t and { β t = 1 /s t } t , ensures the following for any t ∈ [ T ] ,

Lemma 4.1 shows that the error between our gradient estimator ˆ d t and the true gradient includes a bias term arising from the aggregation of delayed momentums. This is in contrast to the synchronous scenario [Dahan and Levy, 2024] where the error is solely variance-dependent without any bias component. However, this bias does not affect the overall excess loss (Theorem 4.2), which remains comparable to the optimal rate achieved in synchronous Byzantine settings (see Remark 4.5).

By integrating the weighted robust aggregator with the double momentum mechanism, we achieve the optimal convergence rate for the first time in an asynchronous Byzantine setting-a significant advancement over previous efforts.

Theorem 4.2 (Asynchronous Byzantine µ 2 -SGD Guarantees) . Let A ω be ( c λ , λ ) -weighted robust aggregation rule and let f be a convex function. Also, let us make the same assumptions as in Thm. 4.1, and let us denote G ∗ := ∥∇ f ( x ∗ ) ∥ , where x ∗ ∈ arg min x ∈K f ( x ) . Then invoking Alg. 2 with { α t = t } t and { β t = 1 /s t } t , and using a learning rate η ≤ 1 / 4 LT guarantees,

Remark 4.3. In the absence of Byzantine iterations ( λ = 0 ), the parameter c λ of a ( c λ , λ ) -weighted robust aggregator can diminish to 0 when we use ω -CTMA (see Table 1). This aligns with the asynchronous SGD analysis [Arjevani et al., 2020] and represents the first work to achieve optimal convergence without Byzantine workers compared to previous efforts [Yang and Li, 2021, 2023, Fang et al., 2022, Zhu et al., 2023, Damaskinos et al., 2018, Xie et al., 2020b, Zhu et al., 2024].

Remark 4.4. Unlike previous works [Yang and Li, 2021, 2023, Fang et al., 2022, Zhu et al., 2023, Damaskinos et al., 2018, Xie et al., 2020b, Zhu et al., 2024], our convergence rate is independent of data dimensionality d and is sublinear at T , even in the presence of Byzantine workers.

Remark 4.5. This result is consistent with the synchronous scenario [Dahan and Levy, 2024], where the delay is constant τ t = m as in Round Robin [Langford et al., 2009]. In this case, the proportion of Byzantine workers is λ , and the asynchronous excess loss is ≤ O ( LD 2 m T + D ˜ σ √ 1+ c λ m √ T ) . In comparison to the synchronous case, where m workers perform R rounds, here we make R query point updates and T = Rm data-samples, resulting in synchronous excess loss ≤ O ( LD 2 R + D ˜ σ √ 1 /m + c λ √ R ) = O ( LD 2 m T + D ˜ σ √ 1+ mc λ √ T ) [Dahan and Levy, 2024].

5 Experiments

To evaluate the effectiveness of our proposed approach, we conducted experiments on MNIST [LeCun et al., 2010] and CIFAR-10 [Krizhevsky et al., 2014] datasets-two recognized benchmarks in image classification tasks. We employed a two-layer convolutional neural network architecture for both datasets, implemented using the PyTorch framework. The training was performed using the cross-entropy loss function, and all computations were executed on an NVIDIA L40S GPU. To ensure the robustness of our findings, each experiment was repeated with three different random seeds, and the results were averaged accordingly. Our experimental results demonstrate consistent performance across both datasets. Further details about the experimental setup and the complete results are provided in Appendix D.

Weighted vs. Non-Weighted Robust Aggregators . We evaluated the test accuracy of weighted and non-weighted robust aggregators in imbalanced asynchronous Byzantine environments. Our experiments show that weighted robust aggregators consistently achieved higher test accuracy than the non-weighted ones (see Figure 2 and Figure 5). This highlights the benefit of prioritizing workers who contribute more updates in asynchronous setups.

Sign Flipping I CWmed vs w-CWmed

Sign Flipping I RFA vs w-RFA

0.7

0.7

0.7

0.7

0.6

0.6

Test Accuracy

0.5

0.6 -

0.6

0.5

0.5

0.5

0.4

0.4

0.4

0.4

0.3

0.3

0.3

cwmed

weighted_cwmed

0.3

rfa

weighted_rfa

0.2

cwmed

weighted_cwmed

0.2

rfa

weighted_rfa

0

10

20

30

40

50

0

10

20

30

40

50

0

10

20

30

40

0S

0

10

20

30

40

50

teration

Reration

Reration

Iteration

Effectiveness of ω -CTMA . We evaluated the test accuracy of weighted robust aggregators with and without the integration of ω -CTMA, as shown in Figure 3 and Figure 6. The results demonstrate that ω -CTMA can enhance the performance of weighted robust aggregators in various Byzantine scenarios for both datasets.

Little | w-CWmed

0.75

0.70

0.75

0.7

0.70

0.65

0.70

0.65

0.60

0.65

0.6

0.60

0.55

0.60

0.5

0.55

0.50

0.45

0.50

weighted_rfa + weighted_ctma

weighted_rfa

0.45

0.40

weighted_cwmed + weighted_ctma

weighted_cwmed

0.55

0.50

weighted_cwmed

weighted_cwmed + weighted_ctma

0.4

E'0

weighted_rfa + weighted_ctma

weighted_rfa

0

10

20

teration

30

40

50

10

20

Iteration

30

40

50

0

10

20

Iteration

30

40

50

0

10

20

teration

30

40

50

Performance of µ 2 -SGD vs. Standard Momentum and SGD . We evaluated the test accuracy of µ 2 -SGD in comparison to standard momentum [Polyak, 1964] and SGD [eon Bottou, 1998] within an asynchronous Byzantine setup. Figure 4 and Figure 7 show that µ 2 -SGD performs on par with standard momentum, while SGD generally exhibits poorer performance relative to both. These results underscore the importance of utilizing historical information when addressing Byzantine scenarios.

Sign Flipping I w-CWmed

Sign Flipping I w-RFA

0.65

0.70

0.70

0.65

0.65

0.70

0.75

0.70

Test Accuracy

0.60

0.50

0.55

Test Accuracy

0.60

0.55

0.50

0.60

0.55

0.50

AC

Aye

0.60

0.55

0.65

0.45

0.40

momentum

p6sznu

sgd

0.40

0.45

momentum

p6sznu

sgd

0.40

0.45

momentum

mu2sgd

pfs

0.50

0.45

momentum

posznu

sgd

0

10

20

30

40

50

0

10

20

30

40

50

0

10

20

30

40

50

0

10

20

30

40

50

Reration

Iteration

Iteration

Reration

Conclusions and Future Work

This paper shows that using a double momentum approach, which incorporates the entire history of each honest worker, improves the stochastic error bound to be proportional to the total number of updates when considering their weighted average in asynchronous settings. By integrating this method with a weighted robust framework, µ 2 -SGD achieves an optimal convergence rate, making it particularly effective for asynchronous Byzantine environments. However, integrating other optimization algorithms, like momentum, into this weighted robust framework can be challenging, as they do not achieve an error bound proportional to the total number of updates and may complicate the adjustment of weights based on the update count. This highlights the need for further research to adapt different methods to the spirit of this framework in non-convex and convex settings.

References

Anish Acharya, Abolfazl Hashemi, Prateek Jain, Sujay Sanghavi, Inderjit S Dhillon, and Ufuk Topcu. Robust training in high dimensions via block coordinate geometric median descent. In International Conference on Artificial Intelligence and Statistics , pages 11145-11168. PMLR, 2022.

Dan Alistarh, Zeyuan Allen-Zhu, and Jerry Li. Byzantine stochastic gradient descent. Advances in Neural Information Processing Systems , 31, 2018.

Zeyuan Allen-Zhu, Faeze Ebrahimian, Jerry Li, and Dan Alistarh. Byzantine-resilient non-convex stochastic gradient descent. arXiv preprint arXiv:2012.14368 , 2020.

Youssef Allouah, Sadegh Farhadkhani, Rachid Guerraoui, Nirupam Gupta, Rafaël Pinot, and John Stephan. Fixing by mixing: A recipe for optimal byzantine ml under heterogeneity. In International Conference on Artificial Intelligence and Statistics , pages 1232-1300. PMLR, 2023.

Yossi Arjevani, Ohad Shamir, and Nathan Srebro. A tight convergence analysis for stochastic gradient descent with delayed updates. In Algorithmic Learning Theory , pages 111-132. PMLR, 2020.

Rotem Zamir Aviv, Ido Hakimi, Assaf Schuster, and Kfir Yehuda Levy. Asynchronous distributed learning: Adapting to gradient delays without prior knowledge. In International Conference on Machine Learning , pages 436-445. PMLR, 2021.

Gilad Baruch, Moran Baruch, and Yoav Goldberg. A little is enough: Circumventing defenses for distributed learning. Advances in Neural Information Processing Systems , 32, 2019.

Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. Advances in neural information processing systems , 30, 2017.

Yudong Chen, Lili Su, and Jiaming Xu. Distributed statistical machine learning in adversarial settings: Byzantine gradient descent. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 1(2):1-25, 2017.

Alon Cohen, Amit Daniely, Yoel Drori, Tomer Koren, and Mariano Schain. Asynchronous stochastic optimization robust to arbitrary delays. Advances in Neural Information Processing Systems , 34: 9024-9035, 2021.

Ashok Cutkosky. Anytime online-to-batch, optimism and acceleration. In International conference on machine learning , pages 1446-1454. PMLR, 2019.

Ashok Cutkosky and Francesco Orabona. Momentum-based variance reduction in non-convex sgd. Advances in neural information processing systems , 32, 2019.

Tehila Dahan and Kfir Yehuda Levy. Fault tolerant ml: Efficient meta-aggregation and synchronous training. In Forty-first International Conference on Machine Learning , 2024.

Georgios Damaskinos, Rachid Guerraoui, Rhicheek Patra, Mahsa Taziki, et al. Asynchronous byzantine machine learning (the case of sgd). In International Conference on Machine Learning , pages 1145-1154. PMLR, 2018.

Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, and Lin Xiao. Optimal distributed online prediction using mini-batches. Journal of Machine Learning Research , 13(1), 2012.

El Mahdi El Mhamdi, Rachid Guerraoui, and Sébastien Louis Alexandre Rouault. Distributed momentum for byzantine-resilient stochastic gradient descent. In 9th International Conference on Learning Representations (ICLR) , number CONF, 2021.

L eon Bottou. Online learning and stochastic approximations. Online learning in neural networks , 17(9):142, 1998.

Minghong Fang, Jia Liu, Neil Zhenqiang Gong, and Elizabeth S Bentley. Aflguard: Byzantinerobust asynchronous federated learning. In Proceedings of the 38th Annual Computer Security Applications Conference , pages 632-646, 2022.

Sadegh Farhadkhani, Rachid Guerraoui, Nirupam Gupta, Rafael Pinot, and John Stephan. Byzantine machine learning made easy by resilient averaging of momentums. In International Conference on Machine Learning , pages 6246-6283. PMLR, 2022.

Rachid Guerraoui, Sébastien Rouault, et al. The hidden vulnerability of distributed learning in byzantium. In International Conference on Machine Learning , pages 3521-3530. PMLR, 2018.

Rachid Guerraoui, Nirupam Gupta, and Rafael Pinot. Byzantine machine learning: A primer. ACM Computing Surveys , 2023.

Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends® in Optimization , 2(3-4):157-325, 2016.

Sai Praneeth Karimireddy, Lie He, and Martin Jaggi. Byzantine-robust learning on heterogeneous datasets via bucketing. arXiv preprint arXiv:2006.09365 , 2020.

Sai Praneeth Karimireddy, Lie He, and Martin Jaggi. Learning from history for byzantine robust optimization. In International Conference on Machine Learning , pages 5311-5319. PMLR, 2021.

Ali Kavis, Kfir Y Levy, Francis Bach, and Volkan Cevher. Unixgrad: A universal, adaptive algorithm with optimal guarantees for constrained optimization. Advances in neural information processing systems , 32, 2019.

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 dataset. online: https://www. cs.toronto.edu/~kriz/cifar.html , 55(5), 2014.

Leslie Lamport, Robert Shostak, and Marshall Pease. The byzantine generals problem. In Concurrency: the works of leslie lamport , pages 203-226. 2019.

John Langford, Alexander Smola, and Martin Zinkevich. Slow learners are fast. arXiv preprint arXiv:0911.0491 , 2009.

Yann LeCun, Corinna Cortes, Chris Burges, et al. Mnist handwritten digit database, 2010. URL http://yann.lecun.com/exdb/mnist/ . Licensed under CC BY-SA 3.0, available at https: //creativecommons.org/licenses/by-sa/3.0/ .

Kfir Y Levy. µ 2 -sgd: Stable stochastic optimization via a double momentum mechanism. arXiv preprint arXiv:2304.04172 , 2023.

Konstantin Mishchenko, Francis Bach, Mathieu Even, and Blake E Woodworth. Asynchronous sgd beats minibatch sgd under arbitrary delays. Advances in Neural Information Processing Systems , 35:420-433, 2022.

Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics , 4(5):1-17, 1964.

Sebastian U Stich and Sai Praneeth Karimireddy. The error-feedback framework: Better rates for sgd with delayed gradients and compressed communication. arXiv preprint arXiv:1909.05350 , 2019.

Cong Xie, Oluwasanmi Koyejo, and Indranil Gupta. Fall of empires: Breaking byzantine-tolerant sgd by inner product manipulation. In Uncertainty in Artificial Intelligence , pages 261-270. PMLR, 2020a.

Cong Xie, Sanmi Koyejo, and Indranil Gupta. Zeno++: Robust fully asynchronous sgd. In International Conference on Machine Learning , pages 10495-10503. PMLR, 2020b.

Yi-Rui Yang and Wu-Jun Li. Basgd: Buffered asynchronous sgd for byzantine learning. In International Conference on Machine Learning , pages 11751-11761. PMLR, 2021.

Yi-Rui Yang and Wu-Jun Li. Buffered asynchronous sgd for byzantine learning. Journal of Machine Learning Research , 24(204):1-62, 2023.

Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantine-robust distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning , pages 5650-5659. PMLR, 2018.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223 , 2023.

Zehan Zhu, Yan Huang, Chengcheng Zhao, and Jinming Xu. Asynchronous byzantine-robust stochastic aggregation with variance reduction for distributed learning. In 2023 62nd IEEE Conference on Decision and Control (CDC) , pages 151-158. IEEE, 2023.

Zehan Zhu, Yan Huang, Chengcheng Zhao, and Jinming Xu. Asynchronous byzantine-robust stochastic aggregation with variance reduction for distributed learning. 2024.

A Bounded Smoothness Variance Assumption

We show that Eq. (3) implies that Eq. (4) holds for some σ 2 L ∈ [0 , L 2 ] .

Here, we also used E [ ∇ f ( x ; z ) -∇ f ( y ; z )] = ( ∇ f ( x ) -∇ f ( y )) , and followed Eq. (3). Therefore, we establish that σ 2 L ∈ [0 , L 2 ] .

B Asynchronous Robust Convex Analysis

B.1 Proof of Thm. 4.1

Proof of Thm. 4.1. To simplify the discussion, let's introduce some notations for a worker i ∈ G , who arrives at time t :

Next, we will employ the following lemma that bounds the distance between the averages x t , x t -τ t . Recall that x t and x t -τ t are consecutive query points for the worker i that arrives at time t .

Lemma B.1 (Aviv et al. [2021]) . Let f : K ↦→ R , where K is a convex set with bounded diameter D . Then invoking Alg. 2 with { α t = t } t ensures the following for any t ∈ [ T ] ,

For completeness, we provide a proof in Section B.1.1.

Next, we define µ t be the average delay of the worker i that arrives at iteration t , i.e.,

From Equation (5), we infer that,

Following this, we analyze the upper bound on the distance between two successive query points for an honest worker i that arrives at time t :

where the first inequality follows Lemma B.1. The second equality utilizes the relation s t -1 = t -τ t µ t -τ t . The final inequality stems from the assumptions in Eq. (5) and Eq. (7).

Remark: Before proceeding with the analysis, we shall condition the (possible randomization) in the delays of all workers; and recall that the data-samples are independent of the delays. Thus, the expectations that we take are only with respect to the randomization in the data-samples and are conditioned on the delays. Thus, this conditioning allows us to treat the delays τ ( i ) t 's and number of updates s ( i ) t 's as fixed and predefined.

We proceed to analyze the recursive dynamics of ˜ ε s t for each i ∈ G . Based on the definitions of d t and ε t , we can present the recursive relationship in the following way:

where Z s t := h s t -∇ f (˜ x s t ) -( ˜ h s t -1 -∇ f (˜ x s t -1 )) . Upon choosing β t = 1 s t , we can reformulate the above equation as follows:

Unrolling this recursion yields an explicit expression for any s t ≥ 1 :

and k is a counter for the iterations where worker i makes an update.

Following this, we derive an upper bound for the expected square norm of M ( i ) k as follows:

where the first inequality uses ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 +2 ∥ b ∥ 2 , which holds ∀ a , b ∈ R d . The second inequality aligns with the assumptions outlined in Equations (2) and (4). The third inequality uses Eq. (8).

Establishing the First Part of the Theorem: Before continuing, it is natural to define an ordered set of samples { z 1 , z 2 , . . . , z t G } such that these samples are associated with honest and consecutive updates (or iterates) of the PS , and t G is the total number of honest updates up to time t . Concretely, the τ th honest update of the PS is based on an honest worker that utilizes a fresh sample z τ .

Now, for a given worker i we shall define the filtration associated with his updates. Concretely, let k ∈ { 1 , . . . s ( i ) T } . Then we define F ( i ) k to be the sigma-algebra induces by the sequence of samples { z 1 , z 2 , . . . , z t G } up to the k th update of worker i . And it is easy to see that {F ( i ) k } k ∈ [ s ( i ) t ] is a filtration. Moreover, it can be directly shown that for a given worker i , then the above defined sequence {M ( i ) k } k ∈ [ s ( i ) t ] (see Eq. (10)) is a martingale difference sequence with respect to {F ( i ) k } k ∈ [ s ( i ) t ] . This allows us to directly employ Lemma B.2 below, which yields,

where we have also used Equations (9) and (11). Thus, the above bounds establish the first part of the theorem.

Lemma B.2 (See e.g. Lemma B.1 in Levy [2023]) . Let { M t } t be a martingale difference sequence with respect to a filtration {F t } t , then the following holds for any t ,

Establishing the Second Part of the Theorem: As before, we define an ordered set of samples { z 1 , z 2 , . . . , z t G } such that these samples are associated with honest and consecutive updates (or iterates) of the PS , and t G is the total number of honest updates up to time t . Concretely, the τ th honest update of the PS is based on an honest worker that utilizes a fresh sample z τ . We shall also define {F τ } τ ∈ [ t G ] be the natural filtration induced by the ordered sequence of data samples.

Moreover, for a given sample z τ ∈ { z 1 , z 2 , . . . , z t G } , let i τ ∈ [ m ] be the worker that is associated with the τ th honest update of the PS , with a fresh sample z τ . In this case, we shall define:

where we have defined,

where M ( i ) k is defined in Eq. (10). It is immediate to show that {M τ } τ ∈ [ t G ] is a martingale difference sequence with respect to {F τ } τ ∈ [ t G ] . Moreover, the following holds directly be the definition of M τ :

Now using Eq. (9) the following holds,

Combining Equations (12) and (13) together with Lemma B.2 establishes the second part of the theorem,

where the inequality uses the bound in Eq. (11).

B.1.1 Proof of Lemma B.1

Proof. We borrowed the following steps from Aviv et al. [2021]. Let's define y ∈ K as the average of x t over the interval [ t -τ t , t ] , i.e.,

Then we have the following relationship:

Hence,

By setting α t = t , we have that,

For t ≥ 3 τ t , we have that,

Given that the domain is bounded, ∥ x t -x t -τ t ∥ ≤ D ∀ t , for t < 3 τ t , we have D < 4 τ t D t . Combining these results, we conclude:

B.2 Proof of Lemma 4.1

Proof of Lemma 4.1.

Lemma B.3. Let f : K ↦→ R , where K is a convex set with bounded diameter D . Then invoking Alg. 2 with { α t = t } t ensures the following for any t ∈ [ T ] , and every i, j ∈ [ m ] ,

Proof.

where the first inequality is a result of the triangle inequality, and the second follows Lemma B.1.

Bias Bounds. Here's a refined version:

We begin by analyzing the upper bound of the bias in the collective gradients of honest workers up to time t in relation to the gradient at that time, denoted as B 1 t . Following this, we derive the upper bound for the bias between the collective gradients of these honest workers and the gradient of an individual honest worker, also up to time t , which we denote as B 2 t . For clarity, we define

Here, the first inequality leverages Jensen's inequality, and the second follows the smoothness assumption in Eq. (3). The third follows Lemma B.1, and the last inequality follows that τ max t := max i ∈ [ m ] { τ ( i ) t } .

For the second bias B 2 t , we have:

Like before, the first inequality leverages Jensen's inequality, and the second follows the smoothness assumption in Eq. (3), the third inequality follows Lemma B.3, and the last one follows that τ max t := max i ∈ [ m ] { τ ( i ) t } .

Variance Bound. We start by determining ρ i as outlined in Definition 3.1:

where ¯ d G ,t := 1 ∑ i ∈G s ( i ) t ∑ i ∈G s ( i ) t d ( i ) t . The first and inequality uses ∥ a + b + c ∥ 2 ≤ 3 ∥ a ∥ 2 + 3 ∥ b ∥ 2 +3 ∥ c ∥ 2 , which holds ∀ a , b , c ∈ R d . The second inequality follows Theorem 4.1 and employs the second bias bound in Eq. (15). The third uses the fact that s ( i ) t ≤ t G , ∀ i ∈ G . Accordingly, we set ρ 2 i := 6˜ σ 2 s ( i ) t + 192( τ max t DL ) 2 t 2 .

Following this, we derive ρ as outlined in Definition 3.1:

Next, we establish an upper bound for E ∥E t ∥ 2 :

where the first inequality uses ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 +2 ∥ b ∥ 2 , which holds ∀ a , b ∈ R d . The second inequality utilizes the same inequality and is further supported by Definition 3.1 and Equation (16). The third aligns with Theorem 4.1, and employs the first bias bound in Eq. (14). The last one utilizes the fact that t G ≥ (1 -λ ) t ≥ t/ 2 , given λ < 1 / 2 .

B.3 Proof of Thm. 4.2

Proof of Thm. 4.2. Following Lemma 4.1 and applying Jensen's inequality, we derive the following bound:

where the third inequality uses √ a + b ≤ √ a + √ b for non-negative a, b ∈ R . The explanation behind this can be seen through the following steps:

whereby taking the square root of both sides of this equation, we obtain the desired inequality.

Next, let's revisit the AnyTime guarantee as outlined in Cutkosky [2019] and proceed to delve into the regret analysis of the update rule.

Theorem B.1 (Rephrased from Theorem 1 in Cutkosky [2019]) . Let f : K → R be a convex function with a minimum x ∗ ∈ arg min w ∈K f ( w ) . Also let { α t ≥ 0 } t , and { w t ∈ K} t , { x t ∈ K} t , such that { x t } t is an { α t } t weighted averaged of { w t } t , i.e. such that x 1 = w 1 , and for any t ≥ 1 ,

Then the following holds for any t ≥ 1 :

Lemma B.4. Let f : K → R be a convex function with a minimum x ∗ ∈ arg min w ∈K f ( w ) , and assume that the assumption in Eq. (1) holds. Also let { α t ≥ 0 } t , and { w t ∈ K} t . Then, for any t ≥ 1 , an arbitrary vector ˆ d t ∈ R d , and the update rule:

we have,

Lemma B.5. let f : K → R be an L -smooth and convex function, and let x ∗ ∈ arg min x ∈K f ( x ) , then for any x ∈ R d we have,

Next, for every iteration t ≤ T , we define:

Thus, combining Theorem B.1 with Lemma B.4, we have that,

where the first inequality is derived from the Anytime guarantee, as outlined in Theorem B.1. The second inequality follows Lemma B.4. The third inequality is a result of applying the Cauchy-Schwarz inequality and the assumption in Eq. (1).

Here, the first inequality employs the Young's inequality. For the second inequality, we introduce the notation ∆ t := f ( x t ) -f ( x ∗ ) , and we follow Lemma B.5, which relates to the smoothness of the function f . In this step, we also set α 0 = 0 and utilizes the property α 2 τ ≤ 2 α 1: τ , given that α τ = τ . The third inequality uses the Cauchy-Schwarz inequality. The last inequality follows the assumption in Eq. (1). It uses the fact that t ≤ T and ∆ t ≥ 0 , ∀ t . This step also incorporates the choice of an appropriate learning rate parameter η ≤ 1 / 4 LT .

Plugging (A) into Eq. (18), gives us,

Lemma B.6 (Lemma C.2 in Levy [2023]) . let { A t } t ∈ [ T ] , { B t } t ∈ [ T ] be sequences of non-negative elements, and assume that for any t ≤ T ,

In the next step, let us define two terms: A t := α 1: t E [ f ( x t ) -f ( x ∗ )] and B t := D 2 2 η + α t G ∗ D + D ∑ τ ∈ [ t ] α τ E ∥E τ ∥ . Note that the series { B t } t forms a non-decreasing series of non-negative values, implying B t ≤ B T for any t ∈ [ T ] . As a result of Eq. (19), we have that A t ≤ B T + 1 2 T ∑ τ ∈ [ T ] A τ .

Leveraging Lemma B.6, Eq. (17), and acknowledging that α 1: T = Θ( T 2 ) , as α t = t , it follows that:

where µ max := 1 T ∑ t ∈ [ T ] τ max t . Finally, choosing the optimal η ≤ 1 4 TL gives us:

B.3.1 Proof of Lemma B.4

Proof of Lemma B.4. The update rule w τ +1 = Π K ( w τ -ηα τ ˆ d τ ) can be expressed as a convex optimization problem within the set K :

Here, the first equality is derived from the definition of the update rule, the second stems from the property of projection, and the final equality is obtained by reformulating the optimization problem in a way that does not affect the minimum value.

Given that w τ +1 is the optimal solution of the above convex problem, by the optimality conditions, we have that:

Then the following bound holds,

Rearranging this, summing over t ≥ 1 iterations, and taking w = x ∗ , we derive:

The first equality equality is achieved through algebraic manipulation, and the last inequality follows the assumption in Eq. (1).

B.3.2 Proof of Lemma B.5

Proof of Lemma B.5.

Lemma B.7 (Lemma C.1 in Levy [2023]) . let f : R d → R be an L -smooth function with a global minimum x ∗ , then for any x ∈ R d we have,

Let us define the function h ( x ) = f ( x ) -f ( x ∗ ) -⟨∇ f ( x ∗ ) , x -x ∗ ⟩ . Due to the convexity of f ( x ) , we have the gradient inequality f ( x ) -f ( x ∗ ) ≥ ⟨∇ f ( x ∗ ) , x -x ∗ ⟩ , which implies h ( x ) ≥ 0 . As h ( x ∗ ) = 0 , this implies that x ∗ is the global minimum of h . Applying Lemma B.7, gives us,

where last inequality holds due to the convexity of f , which implies that ⟨∇ f ( x ∗ ) , x -x ∗ ⟩ ≥ 0 .

C Robust Aggregators Analysis

C.1 Weighted Robust Aggregators

C.1.1 Weighted Geometric Median (WeightedGM)

The Weighted Geometric Median (WeightedGM) is an aggregation method that seeks a point minimizing the weighted sum of Euclidean distances to a set of points. Formally, for a given set of points { x i } i m =1 and corresponding weights { s i } i m =1 , the WeightedGM aggregator is defined as follows:

Lemma C.1. Let ˆ x be a WeightedGM aggregator then ˆ x is ( c λ , λ ) -weighted robust with c λ = ( 1 + λ 1 -2 λ ) 2 .

Proof.

The first inequality leverages Jensen's inequality. The second inequality follows the WeightedGM definition. The third is derived using the following triangle inequality: ∥ ¯ x G -x i ∥ ≤ ∥ ¯ x G -ˆ x ∥ + ∥ ˆ x -x i ∥ . The final inequality is based on the assumptions that ∑ i ∈B s i ≤ λs 1: m and ∑ i ∈G s i ≥ (1 -λ ) s 1: m .

By rearranging, we obtain:

Taking the square of both sides and applying Jensen's inequality gives us:

Taking the exception of both sides gives us the following:

C.1.2 Weighted Coordinate-Wise Median (WeightedCWMed)

The Weighted Coordinate-Wise Median (WeightedCWMed) is an aggregation technique that operates on a per-coordinate basis. For a given set of multi-dimensional data points { x i } i m =1 and corresponding weights { s i } i m =1 , the WeightedCWMed is computed by independently finding the weighted median of each coordinate across all points. Formally, for the k th dimension, the WeightedCWMed aggregator is defined as:

where [ x ] k is the k th element of a vector x and the WeightedMedian is defined as follows: given the elements { x 1 , . . . , x m } of each dimension are sorted in ascending order and their corresponds weights { s 1 , . . . , s m } , the weighted median is the element x j ∗ , where j ∗ is determined by the condition:

If there exists a value j such that:

Then, the WeightedMedian is the average of the j -th and ( j +1) -th elements:

Here, we extend the theoretical guarantee of the Coordinate-Wise Median (CWMed) to its weighted version, following the procedure in Allouah et al. [2023].

Lemma C.2. Let A ω : R d × m → R d be a weighted coordinate-wise aggregation function. Given set of points { x i } i m =1 and corresponding weights { s i } i m =1 , this function incorporates d real-valued functions A 1 ω , . . . , A d ω , where each [ A ω ( { x i } i m =1 ; { s i } i m =1 )] k = A k ω ( { [ x i ] k } i m =1 ; { [ s i ] k } i m =1 ) . If for each k ∈ [ d ] , A k ω is ( c λ , λ ) -weighted robust that satisfies:

Then A ω is ( c λ , λ ) -weighted robust.

Proof. Since A ω is a coordinate-wise aggregator, it applies the same aggregation rule across each dimension. Therefore,

Given that each A k ω , for k ∈ [ d ] , is ( c λ , λ ) -weighted robust that satisfies:

We can express the overall aggregation function A ω as follows:

where the first inequality is derived from the assumption stated in this lemma. The second aligns with the definition of ( c λ , λ ) -weighted robust as detailed in Definition 3.1.

Lemma C.3. Let ˆ x be a WeightedCWMed aggregator then ˆ x is ( c λ , λ ) -weighted robust with c λ = ( 1 + λ 1 -2 λ ) 2 .

Proof. In the context of the k th coordinate, [ WeightedCWMed ] k functions equivalently to WeightedGM for a one-dimensional case. Consequently, each coordinate of the WeightedCWMed aggregator is ( c λ , λ ) -weighted robust with c λ = ( 1 + λ 1 -2 λ ) 2 as established in Lemma C.1. Furthermore, since the WeightedCWMed functions on a coordinate-wise basis, it follows from Lemma C.2 that the entire WeightedCWMed aggregator is ( c λ , λ ) -weighted robust with c λ = ( 1 + λ 1 -2 λ ) 2 .

C.2 Proof of Lemma 3.1

Proof of Lemma 3.1. We denote y i := x i -x 0 , Σ G := ∑ i ∈G s i , Σ S := ∑ i ∈ S s i , Σ B := ∑ i ∈B s i , and Σ m := ∑ i ∈ [ m ] s i . Recall that Σ G ≥ (1 -λ )Σ m , and Σ S = (1 -λ )Σ m (Alg. 1).

Taking the squared norm of both sides, we obtain:

where the first inequality follows the inequality ∥ a + b + c ∥ 2 ≤ 3 ∥ a ∥ 2 +3 ∥ b ∥ 2 +3 ∥ c ∥ 2 , ∀ a , b , c ∈ R 2 and the second follow Jensen's inequality. Note that,

where the first inequality holds because Σ B ≤ λ Σ m and Σ S = (1 -λ )Σ m . The second inequality follows from the fact that 1 -λ ≥ 1 / 2 , and the last since λ < 1 / 2 .

In addition, we have that,

where the first inequality follows the facts that S ∪ G ⊆ [ m ] , { s i ≥ 0 } i ∈ [ m ] and Σ G ≥ (1 -λ )Σ m . The second inequality is based on that 1 -λ ≥ 1 / 2 , and the last since Σ G ≤ Σ m . And in a similar way,

Applying Eq. (21), Eq. (22) and Eq. (23) into Eq. (20), gives us,

where the latter holds since ∑ i ∈G\ S s i ∥ y i ∥ 2 ≤ ∑ i ∈G s i ∥ y i ∥ 2 . Next, we define:

Note that ∑ i ∈ S \G s i ∥ x i -x 0 ∥ 2 = ∑ i ∈ S ∗ \G ∗ ∥ x i -x 0 ∥ 2 . We'll show that there exists an injective function Φ : S ∗ \G ∗ →G ∗ \ S ∗ such that, ∀ i ∈ S ∗ \ G ∗ , ∥ x Φ( i ) -x 0 ∥ ≥ ∥ x i -x 0 ∥ is satisfied. This follows from our selection of S , which consists of the closest elements { x i } i ∈ S to x 0 (see Alg. 1), and from:

where the last inequality follows that ∑ i ∈ S s i -∑ i ∈G s i = (1 -λ )Σ m -Σ G ≤ 0 .

Thus,

where the second inequality follows from the definition of the injective function Φ . The third inequality is justified by the fact that ∑ i ∈G ∗ \ S ∗ ∥ y i ∥ 2 ≤ ∑ i ∈G ∗ ∥ y i ∥ 2 . Finally, the last inequality leverages the property ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 +2 ∥ b ∥ 2 , which holds ∀ a , b ∈ R d .

Taking the expectations of both sides gives us the following:

where the last inequaility stems from Def. 3.1.

D Experiments

D.1 Technical Details

Datasets. We simulated over the MNIST [LeCun et al., 2010] and CIFAR-10 [Krizhevsky et al., 2014] datasets. The datasets were accessed through torchvision (version 0.16.2).

MNIST Dataset . MNIST is a widely used benchmark dataset in the machine learning community, consisting of 70,000 grayscale images of handwritten digits (0-9) with a resolution of 28x28 pixels. The dataset is split into 60,000 training images and 10,000 test images.

CIFAR-10 Dataset . CIFAR-10 is a widely recognized benchmark dataset in the machine learning community, containing 60,000 color images categorized into 10 different classes. Each image has a resolution of 32x32 pixels and represents objects such as airplanes, automobiles, birds, cats, and more. The dataset is evenly split into 50,000 training images and 10,000 test images.

Imbalanced Arrival Scenarios. We simulated two types of imbalanced arrival scenarios:

Proportional Arrival Probability : The probability of arrival for the i -th worker in the honest group was set proportionally to i/ ∑ j ∈G j , ensuring that workers with higher indices had a higher chance of arriving. The same distribution method was applied to Byzantine workers.

Squared ID Arrival Probability : In a more skewed scenario, the arrival probability was proportional to the square of the worker's ID, i.e., i 2 / ∑ j ∈G j 2 . This setup further accentuated the imbalance by favoring workers with larger IDs.

For simplicity, Byzantine workers were introduced after a fixed number of iterations, controlled by a parameter λ . However, it is worth noting that when Byzantine iterations are concentrated, they can cause significant performance degradation. Such patterns can lead to increased delays for honest updates, ultimately affecting the overall convergence of the algorithm.

Optimization Setup. We optimized the cross-entropy loss across all experiments. For comparisons, we configured µ 2 -SGD with fixed parameters γ = 0 . 1 and β = 0 . 25 . This was tested against Standard SGD, and Momentum-based SGD, where the momentum parameter was set to β = 0 . 9 as recommended by Karimireddy et al. [2021].

Attack Simulations. We simulated four types of attacks to evaluate the robustness of our approach:

Label Flipping [Allen-Zhu et al., 2020]: The labels of the data were flipped to incorrect values, by subtracting the original labels from 9.

Sign Flipping [Allen-Zhu et al., 2020]: The signs of the workers' output were flipped.

Little [Baruch et al., 2019]: Adapted from the synchronous case. It computes the maximum allowable deviation z max based on iterations count rather than the number of workers. Then, it perturbs the honest updates by subtracting the product of the weighted standard deviation and z max from the weighted mean of the honest updates.

Byzantine_update = weighted_mean ( honest_momentums ) -weighted_std ( honest_momentums ) · z max .

Empire [Xie et al., 2020a]: Adapted from the synchronous case. This attack scales the weighted mean of the honest momentums by a factor ϵ in the negative direction,

Byzantine_update = -ϵ · weighted_mean ( honest_momentums ) .

In the two latter attacks, the mean and standard deviation are calculated coordinate-wise with respect to weights, setting ϵ = 0 . 1 .

AnyTime Update Formulation. Regarding the AnyTime update, defined as x t := α t w t + α 1: t -1 x t -1 α 1: t , we employed a momentum-based formulation that equivalent to the standard AnyTime update. Specifically, we updated the model parameters according to the formula:

where γ t is difined as γ t := α t α 1: t . By setting α t = Cα 1: t -1 with C > 0 being a constant, we derived that γ t = C C +1 and remains consistent across all time steps t ≥ 1 .

For more details, please visit our GitHub repository. 1

D.2 Experimental Results on MNIST

Sign Flipping | CWmed vs w-CWmed

Sign Flipping | RFA vs w-RFA

1.0

0.95

0.95

0.9

0.9

0.90

0.85

0.90

Test Accuracy

0.8

0.8

0.80

0.75

0.85

0.80

0.7

0.6

0.7

0.70

0.75

0.5

0.6

0.65

....cwmed

weighted_cwmed

0.70

rfa

weighted_rfa

0.4

....

cwmed

weighted_cwmed

0.5

rfa

weighted_rfa

0

5

10

15

20

0

5

10

15

20

0

5

10

15

20

0

5

10

15

20

Iteration

iteration

Iteration

Iteration

Little | w-RFA

Empire|w-CWmed

0.975

0.975

0.975

0.975

0.950

0.950

0.950

0.950

0.925

0.925

0.925

0.925

0.900

0.900

0.900

0.900

0.875

0.875

0.875

0.875

0.850

0.850

0.850

0.850

0.825

weighted_cwmed + weighted_ctma

0.825

weighted_rfa + weighted_ctma

0.825

weighted_cwmed + weighted_ctma

0.825

weighted_rfa + weighted_ctma

weighted_cwmed

weighted_rfa

weighted_cwmed

weighted_rfa

0.800

0.800

0.800

0.800

0

5

10

15

20

0

5

10

15

20

0

5

10

15

20

0

5

10

15

20

Iteration

Iteration

Iteration

Iteration

0.975

0.975

momentum

0.975

1.000

0.950

0.950

mu2sgd

p6s

0.950

0.975

0.950

0.925

0.900

0.875

0.925

0.900

0.875

0.925

0.875

0.900

它

0.875

0.925

0.900

0.850

momentum

0.850

0.850

momentum

0.850

...momentum

0.825

p5sznu

0.825

0.825

mu2sgd

0.825

mu2sgd

...

p6s

p6s

sgd

0.800

0.800

0.800

0.800

0

5

10

15

20

0

5

10

15

20

0

5

10

15

20

0

5

10

15

20

Iteration

Iteration

Iteration

Iteration

NeurIPS paper checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: we ensured that our paper conforms with the NeurIPS Code of Ethics

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: we do not foresee any special societal impact that arise due to our work

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate

deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: our work does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper address the challenges of Byzantine-robust training in asynchronous distributed machine learning systems, aiming to enhance efficiency amid massive parallelization and heterogeneous compute resources, by introducing a novel weighted robust aggregation\nframework.\nQuantify the difficulty in asynchronous scenarios, by considering the number of Byzantine updates, which is more natural than the standard measure of number of Byzantine workers.\nIdentify the need to utilize weighted aggregators rather than standard ones in favour of asynchronous Byzantine problems. Towards doing so, they extend the robust aggregation framework to allow and include weights, and develop appropriate (weighted) rules and a meta-aggregator. Achieving Optimal Convergence: They incorporate weighted robust framework with a recent double momentum mechanism, leveraging its    unique features to achieve an optimal convergence rate for the first time in asynchronous Byzantine ML.\n\nSTRENGTHS:\nThis paper proposes a new solution for Byzantine robust training in asynchronous distributed machine learning systems, with rigorous research methods and reliable experimental results.\nThe paper has a reasonable structure, coherent logic, and clear expression.\nThe research findings have significant implications for asynchronous distributed machine learning.\nThey incorporate weighted robust framework with a recent double momentum mechanism, leveraging its unique features to achieve an optimal convergence rate for the first time in asynchronous Byzantine ML.\nThey adopts a method similar to attention mechanism, assigning different weights to different parts.\n\nWEAKNESSES:\nThe experiment is not sufficient, why not compare it with the weighting methods mentioned in the article.\nThe paper quantified the difficulty in asynchronous scenarios, by considering the number of Byzantine updates, which is more natural than the standard measure of number of Byzantine workers, but the experimental results did not prove this point.\n\nQUESTIONS:\nThe paper introduced a novel weighted robust aggregation framework to address the challenges of Byzantine-robust training in asynchronous distributed machine learning systems, but there is no detailed explanation on how the weight coefficients of the  framework are calculated.\n\nLIMITATIONS:\nThe author did not discuss its limitations in the paper.\nThis paper's work generally does not bring the potential negative social impacts, including privacy issues, algorithmic biases, and the risk of technology abuse.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-26"}

comments: SUMMARY:
This work introduces a weighted robust aggregation framework and extends traditional robust aggregators to asynchronous Byzantine environments. By integrating the proposed framework with a recent double momentum mechanism, the authors achieve optimal convergence rates in asynchronous Byzantine ML under the convex setting.

STRENGTHS:
- The authors address an important problem of Byzantine robustness in asynchronous distributed environments.
- Novel combination of $\mu^2$-SGD with weighted robust aggregation to achieve optimal convergence rates for the first time. In contrast to existing works, the obtained convergence rate is independent of data dimensionality and matches the rates of asynchronous settings in the absence of Byzantine workers.
- The paper is well written and well contextualized w.r.t to existing work in both synchronous and asynchronous Byzantine literature.

WEAKNESSES:
- Empirical evaluations are limited, spanning only a single dataset
- Currently only convex scenario is considered

QUESTIONS:
While a vast variety of attacks have been designed for the synchronous case, they may not be as impactful in the asynchronous case. While the authors adapt Little and Empire to the asynchronous scenario, are there other attacks specific to asynchronous settings? Can the authors comment on the significance of evaluated attacks in the asynchronous case?

LIMITATIONS:
Empirical assessments should be conducted on at least one more dataset to show the effectiveness of the proposed approach. The paper can benefit from restructuring to at least retain important experimental details in the main paper, such as details on the evaluated attacks. Currently, all experimental details have been deferred to Appendix except the name of the dataset.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenges of Byzantine-robust training in asynchronous distributed machine learning systems. The authors propose a novel weighted robust aggregation framework to mitigate the effects of delayed updates and enhance fault tolerance in such environments. They incorporate a recent variance-reduction technique to achieve optimal convergence rates in asynchronous Byzantine settings. The methodology is validated through empirical and theoretical analysis, demonstrating its effectiveness in optimizing performance and fault tolerance.\n\nSTRENGTHS:\n* The introduction of a weighted robust aggregation framework adapted to asynchronous settings is a significant contribution. This innovation allows for more efficient handling of Byzantine faults, which are critical in distributed systems.\n* The paper provides both empirical and theoretical validation of the proposed methodology. This dual approach strengthens the credibility of the results and demonstrates the practical applicability of the proposed techniques.\n* Achieving an optimal convergence rate in asynchronous Byzantine environments is a noteworthy accomplishment. The incorporation of a recent variance-reduction technique adds further value to the research.\n* The paper provides a thorough review of existing approaches to Byzantine-robust training, both in synchronous and asynchronous settings. This contextualization helps in understanding the advancements made by the current work.\n\nWEAKNESSES:\n* __Complexity of Methodology__: While the proposed framework is innovative, it is also quite complex. The introduction of weighted aggregators and the double momentum mechanism might be challenging for practitioners to implement and integrate into existing systems.\n* __Limited Practical Examples__: The paper lacks of sophisticated experiments to demonstrate the effectiveness of the framework, at least to show it in practical ways. \n* __Dependency on Assumptions__: The effectiveness of the proposed method relies heavily on several assumptions, such as bounded delays and the fraction of Byzantine updates. These assumptions might not hold in all practical settings, potentially limiting the method's applicability.\n\nQUESTIONS:\n* While your theoretical analysis is robust, have you identified any discrepancies or limitations when translating these theoretical results into practical implementations? How sensitive is the proposed methodology to the assumptions made (e.g., bounded delays, fraction of Byzantine updates)? What happens if these assumptions are violated in real-world scenarios? An additional experiment should help.\n* How does the proposed framework scale with an increasing number of workers? Either increase or decrease Byzantine workers. Are there any scalability issues or bottlenecks that need to be addressed?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the Asynchronous Robust μ^2-SGD algorithm, designed to address the challenges of Byzantine-robust training in asynchronous distributed machine learning (ML) systems. The method extends the standard μ^2-SGD by incorporating a robust weighted aggregation framework that mitigates the effects of delayed updates and faulty, potentially malicious contributions from distributed workers. This novel approach claims to achieve an optimal convergence rate for the first time in an asynchronous Byzantine environment, enhancing fault tolerance and performance optimization in such systems.\n\nSTRENGTHS:\n- **Innovative Approach to Fault Tolerance**: The paper successfully introduces a novel robust aggregation framework that extends existing methods to accommodate asynchronous dynamics and Byzantine faults. This framework allows for the application of robust aggregators and meta-aggregators to their weighted versions, effectively handling the challenges posed by asynchronous updates and unreliable data sources.\n- **Clear Presentation and Insightful Analysis**: The paper provides a clear presentation, offering good insights into the differences between synchronous and asynchronous settings, which is crucial for understanding the impact and novelty of the proposed method. The authors provide detailed proofs, and the preliminary results presented look promising, indicating potential for significant contributions to the field.\n\nWEAKNESSES:\nWhile I am not an expert in this specific area of asynchronous Byzantine environments, I recognize the significant value this paper offers to the NeurIPS audience. Its innovative approaches to fault tolerance in distributed ML systems are both relevant and timely (such as in the distributed Federated learning where updates can be fuzzy or even malicious). Nevertheless, there are a few areas where further clarification could enhance the paper's impact and applicability.\n\n- **Assumptions on Byzantine Behavior**: The paper introduces an interesting perspective by suggesting that it is more natural to consider the fraction of Byzantine updates rather than the fraction of Byzantine workers in asynchronous environments. While this approach aligns well with the dynamic nature of such systems, it raises questions about whether alternative, potentially simpler solutions could be explored. For instance, could the parameter server utilize mechanisms like monitoring update frequencies to identify and mitigate the impact of Byzantine workers? Further exploration into whether these alternative approaches might offer comparable robustness with potentially reduced complexity could enrich the discussion and provide a more comprehensive understanding of the trade-offs involved.\n- **Impact on System Metrics**: The paper could benefit from a discussion on how the proposed method impacts other critical system metrics, such as the additional memory consumption required by the parameter server and the overall impact on training speed. Understanding these implications could help in assessing the practical viability of implementing this method in real-world systems.\n- **Temporal Distribution of Byzantine Updates**: A minor yet intriguing question arises regarding the assumption that Byzantine updates are uniformly distributed over time. What if these updates are not uniformly distributed but are instead concentrated within specific periods? Exploring the effects of such scenarios could provide deeper insights into the robustness of the proposed method under different types of Byzantine behaviors.\n\nQUESTIONS:\nSee weakness section.\n\nLIMITATIONS:\nNo potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

comments: SUMMARY:
The authors investigate asynchronous training of convex models based on a parameter server in the Byzantine failure setting, evaluating their method on MNIST

STRENGTHS:
- Author's method achieves the optimal convergence rate, which they claim prior work in asynchronous Byzantine ML has not achieved.
- Authors extend the standard measure considered in the Byzantine failure literature (number of Byzantine workers) to a setting more natural for asynchronicity by considering the number of Byzantine updates

WEAKNESSES:
- Authors do not motivate scenarios in which asynchronous convex optimization is relevant/applied in practice.

QUESTIONS:
- Which convex model did you implement and are you evaluating in Section 5 and Appendix D?

LIMITATIONS:
- The proposed method and associated guarantees do not apply to non-convex models
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-18