Title: Flag Aggregator: Scalable Distributed Training under Failures and Augmented Losses using Convex Optimization
Venue: NeurIPS 2023
Accepted: no
Keywords: Robust

Aggregation

Distributed

Training

Failure

Augmented

Byzantine

Resilience
Abstract: Modern ML applications increasingly rely on complex deep learning models and large datasets. There has been an exponential growth in the amount of computation needed to train the largest models. Therefore, to scale computation and data, these models are inevitably trained in a distributed manner in clusters of nodes, and their updates are aggregated before being applied to the model. However, a distributed setup is prone to Byzantine failures of individual nodes, components, and software. With data augmentation added to these settings, there is a critical need for robust and efficient aggregation systems. We define the quality of workers as reconstruction ratios $\in (0,1]$, and formulate aggregation as a Maximum Likelihood Estimation procedure using Beta densities. We show that the Regularized form of log-likelihood wrt subspace can be approximately solved using iterative least squares solver, and provide convergence guarantees using recent Convex Optimization landscape results. Our empirical findings demonstrate that our approach significantly enhances the robustness of state-of-the-art Byzantine resilient aggregators. We evaluate our method in a distributed setup with a parameter server, and show simultaneous improvements in communication efficiency and accuracy across various tasks.

Abstract

Modern ML applications increasingly rely on complex deep learning models and large datasets. There has been an exponential growth in the amount of computation needed to train the largest models. Therefore, to scale computation and data, these models are inevitably trained in a distributed manner in clusters of nodes, and their updates are aggregated before being applied to the model. However, a distributed setup is prone to Byzantine failures of individual nodes, components, and software. With data augmentation added to these settings, there is a critical need for robust and efficient aggregation systems. We define the quality of workers as reconstruction ratios ∈ (0 , 1] , and formulate aggregation as a Maximum Likelihood Estimation procedure using Beta densities. We show that the Regularized form of log-likelihood wrt subspace can be approximately solved using iterative least squares solver, and provide convergence guarantees using recent Convex Optimization landscape results. Our empirical findings demonstrate that our approach significantly enhances the robustness of state-of-the-art Byzantine resilient aggregators. We evaluate our method in a distributed setup with a parameter server, and show simultaneous improvements in communication efficiency and accuracy across various tasks.

1 Introduction

How to Design Aggregators? We consider the problem of designing aggregation functions that can be written as optimization problems of the form,

where { g i } p i =1 ⊆ R n are given estimates of an unknown summary statistic used to compute the Aggregator Y ∗ . If we choose A to be a quadratic function that decomposes over g i 's, and C = R n , then we can see A is simply the standard mean operator. There is a mature literature of studying such functions for various scientific computing applications [1]. More recently, from the machine learning standpoint there has been a plethora of work [2, 3, 4, 5] on designing provably robust aggregators A for mean estimation tasks under various technical assumptions on the distribution or moments of g i .

Distributed ML Use Cases. Consider training a model with a large dataset such as ImageNet-1K [6] or its augmented version which would require data to be distributed over p workers and uses back propagation. Indeed, in this case, g i 's are typically the gradients computed by individual workers at each iteration. In settings where the training objective is convex, the convergence and generalization properties of distributed optimization can be achieved by defining A as a weighted combination of gradients facilitated by a simple consensus matrix, even if some g i 's are noisy [7, 8].

In a distributed setup, as long as the model is convex we can simultaneously minimize the total iteration or communication complexity to a significant extent i.e., it is possible to achieve convergence

Estimate	Subspace	for	Aggregation

…

𝐺

!

:

𝑤

!

×

…

𝑈 ∈ 𝑅

"×"

Σ ∈ 𝑅

"×$

𝑉

%

∈ 𝑅

$×$

𝑌 = 𝑈[: , 1: 𝑚]

𝐺 = 𝐺

!

|𝐺

&

"

×

×

𝑌

1

𝑝

𝑌𝑌

%

𝐺1

𝑑

𝐺

'(

,

1 ≤ 𝑖 ≤ 𝑝,

1 ≤ 𝑗 ≤ 𝑛

SVD

∗ 5

𝑔

!!

𝑔

!"

𝑔

!#

𝑔

"!

𝑔

""

𝑔

"#

𝑔

$!

𝑔

$"

𝑔

$#

…

…

…

𝑔

!

(𝑡)

𝑔

"

(𝑡)

𝑔

#

(𝑡)

…

…

𝑔

!

(𝑡 + 1)

𝑔

"

(𝑡 + 1)

𝑔

#

(𝑡 + 1)

…

…

𝑑

)*!

𝑑

)

𝑔

'

1 ≤ 𝑖 ≤ 𝑝

Left	singular

vectors

iteration 𝑡

Augmented	Data

Stable	Diffusion

𝑔

!

+𝑛,𝑛~𝑁(0,𝜎

"

𝐼)

𝑔

!

+𝑛,𝑛~𝑁(0,𝑊)

Augmented	Data

Stable	Diffusion

𝑔

!

+𝑛,𝑛~𝑁(0,𝜎

"

𝐼)

𝑔

!

+𝑛,𝑛~𝑁(0,𝑊)

iteration 𝑡 + 1

Right	singular

vectors

Singular

values

Concatenated

gradient	matrix

Gradients	from

workers

Weights	for

workers'

gradient

subspaces

𝐺

&

:

𝑤

&

×

𝐺

"

:

𝑤

"

×

…

Flag	Aggregator

and robustness under technical assumptions on the moments of (unknown) distribution from which 35 g i 's are drawn. However, it is still an open problem to determine the optimality of these procedures 36 in terms of either convergence or robustness [9, 10]. 37

Potential Causes of Noise. When data is distributed among workers, hardware and software failures 38 in workers [11, 12, 13] can cause them to send incorrect gradients, which can significantly mislead 39 the model [14]. To see this, let's consider a simple experiment with 15 workers, that f of them 40 produce uniformly random gradients. Figure 2 shows that the model accuracy is heavily impacted 41 when f > 0 when mean is used to aggregate the gradients. 42

The failures can occur due to component or software failures and 43 their probability increases with the scale of the system [15, 16, 17]. 44 Reliability theory is used to analyze such failures, see Chapter 9 45 in [18], but for large-scale training, the distribution of total system 46 failures is not independent over workers, making the total noise in 47 gradients dependent and a key challenge for large-scale training. 48 Moreover, even if there are no issues with the infrastructure, our 49 work is motivated by the prevalence of data augmentation, including 50 hand-chosen augmentations. Since number of parameters n is often 51 greater than number of samples, data augmentation improves the 52 generalization capabilities of large-scale models under technical con53 ditions [19, 20, 21]. In particular, Adversarial training is a common 54 technique that finds samples that are close to training samples but 55 classified as a different class at the current set of parameters, and 56

0

10

20

30

40

50

Epoch

10

20

30

40

50

60

Top-1 Accuracy (%)

Mean,

f

=0

Mean,

f

=1

Mean,

f

=2

Mean,

f

=3

then use such samples for parameter update purposes [22]. Unfortunately, computing adversarial 57 samples is often difficult [23], done using randomized algorithms [24] and so may introduce depen58 dent (across samples) noise themselves. In other words, using adversarial training paradigm, or the 59 so-called inner optimization can lead to noise in gradients, which can cause or simulate dependent 60 'Byzantine' failures in the distributed context. 61

Available Computational Solutions. Most existing open source implementations of A rely just 62 on (functions of) pairwise distances to filter gradients from workers using suitable neighborhood 63 based thresholding schemes, based on moment conditions [25, 26, 27]. While these may be a good 64 strategy when the noise in samples/gradients is somewhat independent, these methods are suboptimal 65 when the noise is dependent or nonlinear, especially when n is large. Moreover, choosing discrete 66

hyperparameters such as number of neighbors is impractical in our use cases since they hamper 67 convergence of the overall training procedure. To mitigate the suboptimality of existing aggregation 68 schemes, we explicitly estimate a subspace Y spanned by 'most' of the gradient workers, and then 69 use this subspace to estimate that a sparse linear combination of g i gradients, acheiving robustness. 70

We present a new optimization based formulation for generalized gradient aggregation purposes in 71 the context of distributed training of deep learning architectures, as shown in Figure 1. 72

Summary of our Contributions. From the theoretical perspective, we present a simple Maximum 73 Likelihood Based estimation procedure for aggregation purposes, with novel regularization functions. 74 Algorithmically, we argue that any procedure used to solve Flag Optimization can be directly used to 75 obtain the optimal summary statistic Y ∗ for our aggregation purposes. Experimentally , our results 76 show resilience against Byzantine attacks, encompassing physical failures, while effectively managing 77 the stochasticity arising from data augmentation schemes. In practice, we achieve a significantly 78 ( ≈ 20% ) better accuracy on standard datasets. Our implementation offers substantial advantages in 79 reducing communication complexity across diverse noise settings through the utilization of our novel 80 aggregation function, making it applicable in numerous scenarios. 81

82

2 Robust Aggregators as Orthogonality Constrained Optimization

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

In this section, we first provide the basic intuition of our proposed approach to using subspaces for aggregation purposes using linear algebra, along with connections of our approach standard eigendecomposition based denoising approaches. We then present our overall optimization formulation in two steps, and argue that it can be optimized using existing methods.

2.1 Optimal Subspace Hypothesis for Distributed Descent

We will use lowercase letters y, g to denote vectors, and uppercase letters Y, G to denote matrices. We will use boldfont 1 to denote the vector of all ones in appropriate dimensions.

0.8

1.0

0.0 0.2 0.4 0.6 Value of Workers (v) 0 250 500 750 1000 1250 1500 1750 2000 Frequency Optimal Subspace Suboptimal Subspace Let g i ∈ R n is the gradient vector from worker i , and Y ∈ R n × m is an orthogonal matrix representation of a subspace that gradients could live in such that m ≤ p . Now, we may interpret each column of Y as a basis function that act on g i ∈ R n , i.e., j -th coordinate of ( Y T g ) j for 1 ≤ j ≤ m is the application of j -th basis or column of Y on g . Recall that by definition of dot product, we have that if Y : ,j ⊥ x , then ( Y T g ) j will be close to zero. Equivalently, if g ∈ span ( Y ) , then ( Y T g ) T Y T g will be bounded away from zero, see Chapter 2 in [28]. Assuming that G ∈ R n × p is the gradient matrix of p workers, Y Y T G ∈ R n × p is the reconstruction of G using Y as basis. That is, i th column of Y T G specifies the amount of gradient from worker i as a function of Y , and high l 2 norm of Y T g i implies that there is a basis in Y such that Y ̸⊥ g i . So it is easy to see that the average over columns of Y Y T G would give the final gradient for update.

Explained Variance of worker i . If we denote z i = Y T g i ∈ R m representing the transformation of gradient g i to z i using Y , then, 0 ≤ ∥ z i ∥ 2 2 = z T i z i = ( Y T g ) T Y T g = g T i Y Y T g i is a scalar, and so is equal to its trace tr ( g T i Y Y T g i ) . Moreover, when Y is orthogonal, we have 0 ≤ ∥ z i ∥ 2 = ∥ Y T g i ∥ 2 ≤ ∥ Y ∥ 2 ∥ g i ∥ 2 ≤ ∥ g i ∥ 2 since the operator norm (or largest singular value) ∥ Y ∥ 2 of Y is at most 1 . Our main idea is to use ∥ z i ∥ 2 2 , ∥ g i ∥ 2 2 to define the quality of the subspace Y for aggregation, as is done in some previous works for Robust Principal Component Estimation [29] - the quantity ∥ z i ∥ 2 2 / ∥ g i ∥ 2 2 is called as Explained/Expressed variance of subspace Y wrt i -th worker [30, 31] - we refer to ∥ z i ∥ 2 2 / ∥ g i ∥ 2 2 as the 'value' of i -th worker. In Figure 3, we can see from the spike near 1 . 0 that if we choose the subspace carefully (blue) as opposed to merely choosing the mean gradient (with unit norm) of all workers, then we can increase the value of workers.

Advantages of Subspace based Aggregation. We can see that using subspace Y , we can easily: 1. handle different number of gradients from each worker, 2. compute gradient reconstruction Y Y T G efficiently whenever Y is constrained to be orthogonal Y = ∑ i y i y T i where y i is the i -th column of Y , otherwise have to use eigendecomposition of Y to measure explained variance which can be time consuming. In (practical) distributed settings, the quality (or noise level) of gradients in

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

each worker may be different, and/or each worker may use a different batch size. In such cases, handcrafted aggregation schemes may be difficult to maintain, and fine-tune. For these purposes with an Orthogonal Subspace Y , we can simply reweigh gradients of worker i according to its noise level, and/or use g i ∈ R n × b i where b i is the batch size of i -th worker with tr ( z T i z i ) instead.

Why is optimizing over subspaces called 'Flag' Optimization? Recent optimization results suggest that we can exploit the finer structure available in Flag Manifold to specify Y more precisely [32]. For example, Y ∈ R m × n can be parametrized directly as a subspace of dimension m or as a nested sequence of Y k ∈ R m k × n , k = 1 , ..., K where m k < m k +1 ≤ p ≤ n such that span ( Y k ) ⊆ span ( Y k +1 ) with Y K ∈ R m × n . When m k +1 = m k = 1 , we have the usual (real) Grassmanian Manifold (quotient of orthogonal group) whose coordinates can be used for optimization, please see Section 5 in [33] for details. In fact, [34] used this idea to extend median in one-dimensional vector spaces to different finite dimensional subspaces using the so-called chordal distance between them. In our distributed training context, we use the explained variance of each worker instead. Here, workers may specify dimensions along which gradient information is relevant for faster convergence - an advantage currently not available in existing aggregation implementations - which may be used for smart initialization also. We use 'Flag' to emphasize this additional nested structure available in our formulation for distributed training purposes.

2.2 Approximate Maximum Likelihood Estimation of Optimal Subspace

Now that we can evaluate a subspace Y on individual gradients g i , we now show that finding subspace Y can be formulated using standard maximum likelihood estimation principles [35]. Our formulation reveals that regularization is critical for aggregation especially in distributed training. In order to write down the objective function for finding optimal Y , we proceed in the following two steps:

Step 1. Assume that each worker provides a single gradient for simplicity. Now, denoting the value of information v of worker i by v i = z T i z i g T i g i , we have v i ∈ [0 , 1] . Now by assuming that v i 's are observed from Beta distribution with α = 1 and β = 1 2 (for simplicity), we can see that the likelihood P ( v i ) is,

where B ( a, b ) is the normalization constant. Then, the total log-likelihood of observing gradients g i as a function of Y (or v i 's) is given by taking the log of product of P ( v i ) 's as (ignoring constants),

Step 2. Now we use Taylor's series with constant a > 0 to approximate individual worker loglikelihoods log(1 -v i ) ≈ a (1 -v i ) 1 a -a as follows: first, we know that exp ( log( v i ) a ) = v 1 a i . On the other hand, using Taylor expansion of exp about the origin (so large a > 1 is better), we have that exp ( log( v i ) a ) ≈ 1 + log( v i ) a . Whence, we have that 1 + log( v i ) a ≈ v 1 a i which immediately implies

1

that log( v i ) ≈ av a i -a . So, by substituting the Taylor series approximation of log in Equation 3, we obtain the negative log-likelihood approximation to be minimized for robust aggregation purposes as,

where a > 1 is a sufficiently large constant. In the above mentioned steps, the first step is standard. Our key insight is using Taylor expansion in (4) with a sufficiently large a to eliminate log optimization which are known to be computationally expensive to solve, and instead solve smooth ℓ a , a > 1 norm based optimization problems which can be done efficiently by modifying existing procedures [36].

Extension to general beta distributions, and gradients α > 0 , β > 0 , g i ∈ R n × k . Note that our derivation in the above two steps can be extended to any beta shape parameters α > 0 , β > 0 - there will be two terms in the final negative log-likelihood expression in our formulation (4), one for each α, β . Similarly, by simply using v i = tr ( g T i Y Y T g i ) to define value of worker i in equation (2), and then in our estimator in (4), we can easily handle multiple k gradients from a single worker i for Y .

161

162

163

164

165

166

167

168

169

170

171

172

2.3 Flag Aggregator for Distributed Optimization

It is now easy to see that by choosing a = 2 , in equation (4), we obtain the negative loglikelihood (ignoring constants) as ( ∑ p i =1 √ 1 -g T i Y Y T g i ) showing that Flag Median can indeed be seen as an Maximum Likelihood Estimator (MLE). In particular, Flag Median can be seen as an MLE of Beta Distribution with parameters α = 1 and β = 1 2 . Recent results suggest that in many cases, MLE is ill-posed, and regularization is necessary, even when the likelihood distribution is Gaussian [37]. So, based on the Flag Median estimator for subspaces, we propose an optimization based subspace estimator Y ∗ for aggregation purposes. We formulate our Flag Aggregator (FA) objective function with respect to Y as a regularized sum of likelihood based (or data) terms in (4) using trace operators tr ( · ) as the solution to the following constrained optimization problem:

where λ > 0 is a regularization hyperparameter. In our analysis, and implementation, we provide support for two possible choices for R ( Y ) :

(1) Mathematical norms: R ( Y ) can be a form of norm-based regularization other than ∥ Y ∥ 2 Fro since 173 it is constant over the feasible set in (5). For example, it could be convex norm with efficient 174 subgradient oracle such as, i.e. element-wise: ∑ n i =1 ∑ m j =1 ∥ Y ij ∥ 1 or ∑ m i =1 ∥ Y i,i ∥ 1 , 175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

(2)

Data-dependent norms:

̸

Following our subspace construction in Section 2.1, we may choose

distance between gradient vectors g i , g j from workers i , j . Intuitively, the pairwise terms in our loss function (5) favors subspace Y that also reconstructs the pairwise vectors g i -g j that are close to each other. So, by setting λ = Θ( p ) , that is, the pairwise terms dominate the objective function in (5). Hence, λ regularizes optimal solutions Y ∗ of (5) to contain g i 's with low pairwise distance in its span - similar in spirit to AggregaThor in [38].

̸

Convergence of Flag Aggregator (FA) Algorithm 1. With these, we can state our main algorithmic result showing that our FA (5) can be solved efficiently using standard convex optimization proof techniques. In particular, in supplement, we present a smooth Semi-Definite Programming (SDP) relaxation of FA in equation (5) using the Flag structure. This allows us to view the IRLS procedure in 1 as solving the low rank parametrization of the smooth SDP relaxation, thus guaranteeing fast convergence to second order optimal (local) solutions. Importantly, our SDP based proof works for any degree of approximation of the constant a in equation (4) and only relies on smoothness of the loss function wrt Y , although speed of convergence is reduced for higher values of a = 2 , see [39]. We leave determining the exact dependence of a on rate of convergence for future work.

How is FA aggregator different from (Bulyan and Multi-Krum)? Bulyan is a strong Byzantine resilient gradient aggregation rule for p ≥ 4 f +3 where p is the total number of workers and f is

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

the number of Byzantine workers. Bulyan is a two-stage algorithm. In the first stage, a gradient aggregation rule R like coordinate-wise median [40] or Krum [9] is recursively used to select θ = p -2 f gradients. The process uses R to select gradient vector g i which is closest to R 's output (e.g. for Krum, this would be the gradient with the top score, and hence the exact output of R ). The chosen gradient is removed from the received set and added to the selection set S repeatedly until | S | = θ . The second stage produces the resulting gradient. If β = θ -2 f , each coordinate would be the average of β -nearest to the median coordinate of the θ gradients in S . In matrix terms, if we consider S ∈ R p × m as a matrix with each column having one non-zero entry summing to 1 , Bulyan would return 1 m ReLU( GS ) 1 m , where 1 m ∈ R m is the vector of all ones, while FA would return 1 p Y Y T G 1 p . Importantly, the gradient matrix is being right-multiplied in Bulyan, but left-multiplied in FA, before getting averaged. While this may seem like a discrepancy, in supplement we show that by observing the optimality conditions of (5) wrt Y , we show that 1 m Y Y T G can be seen as a right multiplication by a matrix parametrized by lagrangian multipliers associated with the orthogonality constraints in (5). This means it should be possible to combine both approaches for faster aggregation.

3 Experiments

In this section, we conduct experiments to test our proposed FA in the context of distributed training in two testbeds. First, to test the performance of our FA scheme solved using IRLS (Flag Mean) on standard Byzantine benchmarks. Then, to evaluate the ability of existing state-of-the-art gradient aggregators we augment data via two techniques that can be implemented with Sci-kit package.

Implementation Details. We implement FA in Pytorch [41], which is popular but does not support Byzantine resilience natively. We adopt the parameter server architecture and employ Pytorch's distributed RPC framework with TensorPipe backend for machine-to-machine communication. We extend Garfield's Pytorch library [42] with FA and limit our IRLS convergence criteria to a small error, 10 -10 , or 5 iterations of flag mean for SVD calculation. We set m = ⌈ p +1 2 ⌉ .

3.1 Setup

Baselines: We compare FA to several existing aggregation rules: (1) coordinate-wise Trimmed Mean [40] (2) coordinate-wise Median [40] (3) mean-around-median (MeaMed) [43] (4) Phocas [44] (5) Multi-Krum [9] (6) Bulyan [45].

Accuracy: The fraction of correct predictions among all predictions, using the test dataset (top-1 cross-accuracy).

Testbed: We used 4 servers as our experimental platform. Each server has 2 Intel(R) Xeon(R) Gold 6240 18-core CPU @ 2.60GHz with Hyper-Threading and 384GB of RAM. Servers have a Tesla V100 PCIe 32GB GPU and employ a Mellanox ConnectX-5 100Gbps NIC to connect to a switch. We use one of the servers as the parameter server and instantiate 15 workers on other servers, each hosting 5 worker nodes, unless specified differently in specific experiments. For the experiments designed to show scalability, we instantiate 60 workers.

Dataset and model: We focus on the image classification task since it is a widely used task for benchmarking in distributed training [46]. We train ResNet-18 [47] on CIFAR-10 [48] which has 60,000 32 × 32 color images in 10 classes. For the scalability experiment, we train a CNN with two convolutional layers followed by two fully connected layers on MNIST [49] which has 70,000 28 × 28 grayscale images in 10 classes. We also run another set of experiments on Tiny ImageNet [50] in the supplement. We use SGD as the optimizer, and cross-entropy to measure loss. The batch size for each worker is 128 unless otherwise stated. Also, we use a learning decay strategy where we decrease the learning rate by a factor of 0.2 every 10 epochs.

Threat models: We evaluate FA under two classes of Byzantine workers. They can send uniformly random gradients that are representative of errors in the physical setting, or use non-linear augmented data described as below.

Evaluating resilience against nonlinear data augmentation: In order to induce Byzantine behavior in our workers we utilize ODE solvers to approximately solve 2 non-linear processes, Lotka Volterra

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

Flag Aggregator

Bulyan

Multi-Krum

MeaMed

Median

Trimmed Mean

Phocas

0

10

20

30

40

Epoch

10

20

30

40

50

60

70

Top-1 Accuracy (%)

(a)

f

= 1

0

10

20

30

40

Epoch

10

20

30

40

50

60

Top-1 Accuracy (%)

(b)

f

= 2

0

10

20

30

40

Epoch

10

20

30

40

50

Top-1 Accuracy (%)

(c)

f

= 3

[51] and Arnold's Cat Map [52], as augmentation methods. Since the augmented samples are deterministic, albeit nonlinear functions of training samples, the 'noise' is dependent across samples.

In Lotka Volterra , we use the following linear gradient transformation of 2D pixels:

where α, β, γ and δ are hyperparameters. We choose them to be 2 3 , 4 3 , -1 and -1 respectively.

Second, we use a nonsmooth transformation called Arnold's Cat Map as a data augmentation scheme. Once again, the map can be specified using a two-dimensional matrix as,

where mod represents the modulus operation, x and y are the coordinates or pixels of images and N is the height/width of images (assumed to be square). We also used a smooth approximation of the Cat Map obtained by approximating the mod function as,

where α 1 = 2 x + y n , α 2 = x + y n , and m is the degree of approximation, which we choose to be 0 . 95 in our data augmentation experiments.

Howto perform nonlinear data augmentation? In all three cases, we used SciPy's [53] solve_ivp method to solve the differential equations, by using the LSODA solver. In addition to the setup described above, we also added a varying level of Gaussian noise to each of the training images. All the images in the training set are randomly chosen to be augmented with varying noise levels of the above mentioned augmentation schemes. We have provided the code that implements all our data augmentation schemes in the supplement zipped folder.

3.2 Results

Tolerance to the number of Byzantine workers: In this experiment, we show the effect of Byzantine behavior on the convergence of different gradient aggregation rules in comparison to FA. Byzantine workers send random gradients and we vary the number of them from 1 to 3 . Figure 4 shows that for some rules, i.e. Trimmed Mean, the presence of even a single Byzantine worker has a catastrophic impact. For other rules, as the number of Byzantine workers increases, filtering out the outliers becomes more challenging because the amount of noise increases. Regardless, FA remains more robust compared to other approaches.

Marginal utility of larger batch sizes under a fixed noise level:

We empirically verified the batch size required to identify our optimal Y ∗ - the FA matrix at each iteration. In particular, we fixed the noise level to f = 3 Byzantine workers and varied batch sizes. We show the results in Figure 5. Our results indicate that, in cases where a larger batch size is a training requirement, FA achieves a significantly better accuracy compared to the existing state of the art aggregators. This may be useful in some large scale vision applications, see [54, 55] for more details. Empirically, we can already see that our spectral relaxation to identify gradient subspace is effective in practice in all our experiments.

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

Flag Aggregator

Bulyan

Multi-Krum

MeaMed

Median

Trimmed Mean

Phocas

0

10

20

30

Epoch

10

20

30

40

50

60

Top-1 Accuracy (%)

(a)

bs

= 64

0

10

20

30

Epoch

10

20

30

40

50

Top-1 Accuracy (%)

(b)

bs

= 128

0

10

20

30

Epoch

10

20

30

40

50

Top-1 Accuracy (%)

(c)

bs

= 192

Flag Aggregator

Bulyan

Multi-Krum

MeaMed

Median

Trimmed Mean

Phocas

0

10

20

Epoch

10

20

30

40

50

60

Top-1 Accuracy (%)

(a)

p

= 15

, f

= 3

0

10

20

30

Epoch

10

20

30

40

50

60

Top-1 Accuracy (%)

(b)

p

= 11

, f

= 2

0

10

20

30

Epoch

10

20

30

40

50

60

Top-1 Accuracy (%)

(c)

p

= 13

, f

= 2

0

10

20

30

Epoch

10

20

30

40

50

60

Top-1 Accuracy (%)

(d)

p

= 15

, f

= 2

Tolerance to communication loss: To analyze the effect of unreliable communication channels 276 between the workers and the parameter server on convergence, we design an experiment where the 277 physical link between some of the workers and the parameter server randomly drops a percentage of 278 packets. Here, we set the loss rate of three links to 10% i.e., there are 3 Byzantine workers in our 279 setting. The loss is introduced using the netem queuing discipline in Linux designed to emulate the 280 properties of wide area networks [56]. The two main takeaways in Figure 6a are: 281

FA converges to a significantly higher accuracy than other aggregators, and thus is more robust to unreliable underlying network transports.

Considering time-to-accuracy for comparison, FA reaches a similar accuracy in less total number of training iterations, and thus is more robust to slow underlying network transports.

Analyzing the marginal utility of additional workers. To see the effect of adding more workers to a fixed number of Byzantine workers, we ran experiments where we fixed f , and increased p . Our experimental results shown in Figures 6b-6d indicate that our FA algorithm possesses strong resilience property for reasonable choices of p .

The effect of having augmented data during training in Byzantine workers: Figure 7 shows FA can handle nonlinear data augmentation in a much more stable fashion. Please see supplement for details on the level of noise, and exact solver settings that were used to obtain augmented images.

The effect of the regularization parameter in FA: The data-dependent regularization parameter λ in FA provides flexibility in the loss function to cover aggregators that benefit from pairwise distances such as Bulyan and Multi-Krum. To verify whether varying λ can interpolate Bulyan and Multi-Krum, we change λ in Figure 8. We can see when FA improves or performs similarly for a range of λ . Here, we set p and f to satisfy the strong Byzantine resilience condition of Bulyan, i.e, p ≥ 4 f +3 .

Scaling out to real-world situations with more workers: In distributed ML, p and f are usually large. To test high-dimensional settings commonly dealt in Semantic Vision with our FA, we used ResNet-18. Now, to specifically test the scalability of FA, we fully utilized our available GPU servers and set up to p = 60 workers (up to f = 14 Byzantine) with the MNIST dataset and a simple CNN with two convolutional layers followed by two fully connected layers (useful for simple detection).

Figure 9 shows evidence that FA is feasible for larger setups.

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

Flag Aggregator

Bulyan

Multi-Krum

MeaMed

Median

Trimmed Mean

Phocas

0

10

20

30

Epoch

10

20

30

40

50

60

Top-1 Accuracy (%)

0

5

10

15

20

Epoch

10

20

30

40

50

60

70

Top-1 Accuracy (%)

FA,  =0

FA,  =4

FA,  =8

FA,

=16

Bulyan

Multi-Krum

Figure 8:

CIFAR10 with

ResNet-18,

p

= 7

,

and

f

= 1

0

5

10

15

20

Epoch

20

40

60

80

100

Top-1 Accuracy (%)

FA,

p

=15,

f

=3

FA,

p

=30,

f

=6

FA,

p

=60,

f

=12

Figure

9:

Scaling FA to

larger setups

200

400

600

800

1000

Time (s)

10

20

30

40

50

Top-1 Accuracy (%)

Flag Aggregator

Bulyan

Multi-Krum

MeaMed

Median

Trimmed Mean

Phocas

0

10

20

30

40

Epoch

0

500

1000

1500

2000

2500

3000

Time (s)

Flag Aggregator

Bulyan

Multi-Krum

MeaMed

Median

Trimmed Mean

Phocas

(b) Iteration time

500

1000

1500

2000

2500

3000

Time (s)

10

20

30

40

50

Top-1 Accuracy (%)

Flag Aggregator

Bulyan

Multi-Krum

MeaMed

Median

Trimmed Mean

Phocas

(c) Total Time-to-accuracy

4 Discussion and Limitation

Is it possible to fully 'offload' FA computation to switches? Recent work propose that aggregation be performed entirely on network infrastructure to alleviate any communication bottleneck that may arise [57, 58]. However, to the best of our knowledge, switches that are in use today only allow limited computation to be performed on gradient g i as packets whenever they are transmitted [59, 60]. That is, programmability is restrictive at the moment- switches used in practice have no floating point, or loop support, and are severely memory/state constrained. Fortunately, solutions seem near. For instance, [61] have already introduced support for floating point arithmetic in programmable switches. We may use quantization approaches for SVD calculation with some accuracy loss [62] to approximate floating point arithmetic. Offloading FA to switches has great potential in improving its computational complexity because the switch would perform as a high-throughput streaming parameter server to synchronize gradients over the network. Considering that FA's accuracy currently outperforms its competition in several experiments, an offloaded FA can reach their accuracy even faster or it could reach a higher accuracy in the same amount of time.

Potential Limitation. Because in every iteration of FA, we perform SVD, the complexity of the algorithm would be O ( nN δ ( ∑ p i =1 k i ) 2 ) with N δ being the number of iterations for the algorithm. Figure 10 show the wall clock time it takes for FA to reach a certain accuracy (10a) or epoch(10b) compared to other methods under a fixed amount of random noise f = 3 with p = 15 workers. Although the iteration complexity of FA is higher, here each iteration has a higher utility as reflected in the time-to-accuracy measures. This makes FA comparable to others in a shorter time span, however, if there is more wall clock time to spare, FA converges to a better state as shown in Figure 10c where we let the same number of total iterations finish for all methods.

5 Conclusion

In this paper we proposed Flag Aggregator (FA) that can be used for robust aggregation of gradients 323 in distributed training. FA is an optimization-based subspace estimator that formulates aggregation as 324 a Maximum Likelihood Estimation procedure using Beta densities. We perform extensive evaluations 325 of FA and show it can be effectively used in providing Byzantine resilience for gradient aggregation. 326

327

Using techniques from convex optimization, we theoretically analyze FA and with tractable relaxations

328

show its amenability to be solved by off-the-shelf solvers or first-order reweighing methods.

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

References

Michel Grabisch, Jean-Luc Marichal, Radko Mesiar, and Endre Pap. Aggregation functions , volume 127. Cambridge University Press, 2009.

Sivaraman Balakrishnan, Simon S. Du, Jerry Li, and Aarti Singh. Computationally efficient robust sparse estimation in high dimensions. In Satyen Kale and Ohad Shamir, editors, Proceedings of the 2017 Conference on Learning Theory , volume 65 of Proceedings of Machine Learning Research , pages 169-212. PMLR, 07-10 Jul 2017. URL https: //proceedings.mlr.press/v65/balakrishnan17a.html .

Ilias Diakonikolas, Daniel Kane, Sushrut Karmalkar, Eric Price, and Alistair Stewart. Outlierrobust high-dimensional sparse estimation via iterative filtering. Advances in Neural Information Processing Systems , 32, 2019.

Yu Cheng, Ilias Diakonikolas, Rong Ge, Shivam Gupta, Daniel M. Kane, and Mahdi Soltanolkotabi. Outlier-robust sparse estimation via non-convex optimization. Advances in Neural Information Processing Systems , 2022.

Ilias Diakonikolas, Daniel M. Kane, Sushrut Karmalkar, Ankit Pensia, and Thanasis Pittas. Robust sparse mean estimation via sum of squares. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 4703-4763. PMLR, 02-05 Jul 2022. URL https: //proceedings.mlr.press/v178/diakonikolas22e.html .

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) , 115(3):211-252, 2015. doi: 10.1007/s11263-015-0816-y.

Konstantinos I Tsianos and Michael G Rabbat. Distributed strongly convex optimization. In 2012 50th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 593-600. IEEE, 2012.

Tao Yang, Xinlei Yi, Junfeng Wu, Ye Yuan, Di Wu, Ziyang Meng, Yiguang Hong, Hong Wang, Zongli Lin, and Karl H Johansson. A survey of distributed optimization. Annual Reviews in Control , 47:278-305, 2019.

Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 118-128. Curran Associates Inc., 2017. ISBN 9781510860964.

Sadegh Farhadkhani, Rachid Guerraoui, Nirupam Gupta, Rafael Pinot, and John Stephan. Byzantine machine learning made easy by resilient averaging of momentums. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 6246-6283. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/farhadkhani22a.html .

Leonardo Bautista-Gomez, Ferad Zyulkyarov, Osman Unsal, and Simon McIntosh-Smith. Unprotected computing: A large-scale study of dram raw error rate on a supercomputer. In SC '16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , pages 645-655, 2016. doi: 10.1109/SC.2016.54.

Bianca Schroeder and Garth A. Gibson. Disk failures in the real world: What does an mttf of 1,000,000 hours mean to you? In Proceedings of the 5th USENIX Conference on File and Storage Technologies , FAST '07, page 1-es, USA, 2007. USENIX Association.

Phillipa Gill, Navendu Jain, and Nachiappan Nagappan. Understanding network failures in data centers: Measurement, analysis, and implications. SIGCOMM Comput. Commun. Rev. , 41(4):350-361, aug 2011. ISSN 0146-4833. doi: 10.1145/2043164.2018477. URL https://doi.org/10.1145/2043164.2018477 .

Gilad Baruch, Moran Baruch, and Yoav Goldberg. A little is enough: Circumventing defenses 379 for distributed learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, 380 and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. 381 Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/ 382 file/ec1c59141046cd1866bbbcdfb6ae31d4-Paper.pdf . 383

384

385

386

Guosai Wang, Lifei Zhang, and Wei Xu. What can we learn from four years of data center hardware failures? In 2017 47th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) , pages 25-36, 2017. doi: 10.1109/DSN.2017.26.

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

Devesh Tiwari, Saurabh Gupta, James Rogers, Don Maxwell, Paolo Rech, Sudharshan Vazhkudai, Daniel Oliveira, Dave Londo, Nathan DeBardeleben, Philippe Navaux, Luigi Carro, and Arthur Bland. Understanding gpu errors on large-scale hpc systems and the implications for system design and operation. In 2015 IEEE 21st International Symposium on High Performance Computer Architecture (HPCA) , pages 331-342, 2015. doi: 10.1109/HPCA.2015.7056044.

Bin Nie, Devesh Tiwari, Saurabh Gupta, Evgenia Smirni, and James H. Rogers. A large-scale study of soft-errors on gpus in the field. In 2016 IEEE International Symposium on High Performance Computer Architecture (HPCA) , pages 519-530, 2016. doi: 10.1109/HPCA.2016. 7446091.

Sheldon M Ross. Introduction to probability models . Academic press, 2014.

Fanny Yang, Zuowen Wang, and Christina Heinze-Deml. Invariance-inducing regularization using worst-case transformations suffices to boost accuracy and spatial robustness. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/ 1d01bd2e16f57892f0954902899f0692-Paper.pdf .

Christina Heinze-Deml and Nicolai Meinshausen. Conditional variance penalties and domain shift robustness, 2017. URL https://arxiv.org/abs/1710.11469 .

Saeid Motiian, Marco Piccirilli, Donald A. Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. In IEEE International Conference on Computer Vision (ICCV) , 2017.

Sravanti Addepalli, Samyak Jain, et al. Efficient and effective augmentation strategy for adversarial training. Advances in Neural Information Processing Systems , 35:1488-1501, 2022.

Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.03994 , 2020.

Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference on machine learning , pages 1310-1320. PMLR, 2019.

Youssef Allouah, Rachid Guerraoui, Nirupam Gupta, Rafael Pinot, and John Stephan. Distributed learning with curious and adversarial machines. arXiv preprint arXiv:2302.04787 , 2023.

Youssef Allouah, Sadegh Farhadkhani, Rachid Guerraoui, Nirupam Gupta, Rafaël Pinot, and John Stephan. Fixing by mixing: A recipe for optimal byzantine ml under heterogeneity. In International Conference on Artificial Intelligence and Statistics , pages 1232-1300. PMLR, 2023.

Sadegh Farhadkhani, Rachid Guerraoui, Nirupam Gupta, Rafael Pinot, and John Stephan. Byzantine machine learning made easy by resilient averaging of momentums. In International Conference on Machine Learning , pages 6246-6283. PMLR, 2022.

P-A Absil. Optimization algorithms on matrix manifolds . Princeton University Press, 2008.

John Wright, Arvind Ganesh, Shankar Rao, Yigang Peng, and Yi Ma. Robust principal component analysis: Exact recovery of corrupted low-rank matrices via convex optimization. Advances in neural information processing systems , 22, 2009.

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

Matthias Hein and Thomas Bühler. An inverse power method for nonlinear eigenproblems with applications in 1-spectral clustering and sparse pca. Advances in neural information processing systems , 23, 2010.

Rudrasis Chakraborty, Soren Hauberg, and Baba C Vemuri. Intrinsic grassmann averages for online linear and robust subspace learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6196-6204, 2017.

D. Monk. The geometry of flag manifolds. Proceedings of the London Mathematical Society , s3-9(2):253-286, 1959. doi: https://doi.org/10.1112/plms/s3-9.2.253. URL https: //londmathsoc.onlinelibrary.wiley.com/doi/abs/10.1112/plms/s3-9.2.253 .

Ke Ye, Ken Sze-Wai Wong, and Lek-Heng Lim. Optimization on flag manifolds. Mathematical Programming , 194(1-2):621-660, 2022.

Nathan Mankovich, Emily J King, Chris Peterson, and Michael Kirby. The flag median and flagirls. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10339-10347, 2022.

Kevin P Murphy. Probabilistic machine learning: an introduction . MIT press, 2022.

Massimo Fornasier, Holger Rauhut, and Rachel Ward. Low-rank matrix recovery via iteratively reweighted least squares minimization. SIAM Journal on Optimization , 21(4):1614-1640, 2011.

Toni Karvonen and Chris J Oates. Maximum likelihood estimation in gaussian process regression is ill-posed. Journal of Machine Learning Research , 24(120):1-47, 2023.

Georgios Damaskinos, El-Mahdi El-Mhamdi, Rachid Guerraoui, Arsany Guirguis, and Sébastien Rouault. Aggregathor: Byzantine machine learning via robust gradient aggregation. Proceedings of Machine Learning and Systems , 1:81-106, 2019.

Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong Ma, and Yuling Yan. Noisy matrix completion: Understanding statistical guarantees for convex relaxation via nonconvex optimization. SIAM journal on optimization , 30(4):3098-3121, 2020.

Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantine-robust distributed learning: Towards optimal statistical rates. In Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 56505659. PMLR, 10-15 Jul 2018.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems , volume 32, 2019.

Rachid Guerraoui, Arsany Guirguis, Jérémy Plassmann, Anton Ragot, and Sébastien Rouault. Garfield: System support for byzantine machine learning (regular paper). In 2021 51st Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) , pages 39-51, 2021. doi: 10.1109/DSN48987.2021.00021.

Cong Xie, Oluwasanmi Koyejo, and Indranil Gupta. Generalized byzantine-tolerant sgd, 2018.

Cong Xie, Oluwasanmi Koyejo, and Indranil Gupta. Phocas: dimensional byzantine-resilient stochastic gradient descent, 2018.

El Mahdi El Mhamdi, Rachid Guerraoui, and Sébastien Rouault. The hidden vulnerability of distributed learning in Byzantium. In Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 3521-3530. PMLR, 10-15 Jul 2018.

Trishul Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman. Project 474 adam: Building an efficient and scalable deep learning training system. In Proceedings of the 475 11th USENIX Conference on Operating Systems Design and Implementation , OSDI'14, page 476 571-582, USA, 2014. 477

478

479

480

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016. doi: 10.1109/CVPR.2016.90.

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009. URL https: //www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf .

Yann LeCun and Corinna Cortes. MNIST handwritten digit database, 2010. URL http: //yann.lecun.com/exdb/mnist/ .

Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge, 2015.

David Kelly. Rough path recursions and diffusion approximations. The Annals of Applied Probability , 26(1):425-461, 2016.

Jianghong Bao and Qigui Yang. Period of the discrete arnold cat map and general cat map. Nonlinear Dynamics , 70(2):1365-1375, 2012.

Fundamental algorithms for scientific computing in python. https://scipy.org/ , 2023.

Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations , 2017. URL https://openreview. net/forum?id=H1oyRlYgg .

Yang You, Jonathan Hseu, Chris Ying, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large-batch training for lstm and beyond. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , SC '19, 2019.

Kevin Hsieh, Aaron Harlap, Nandita Vijaykumar, Dimitris Konomis, Gregory R. Ganger, Phillip B. Gibbons, and Onur Mutlu. Gaia: Geo-distributed machine learning approaching lan speeds. In Proceedings of the 14th USENIX Conference on Networked Systems Design and Implementation , page 629-647, 2017.

Amedeo Sapio, Marco Canini, Chen-Yu Ho, Jacob Nelson, Panos Kalnis, Changhoon Kim, Arvind Krishnamurthy, Masoud Moshref, Dan Ports, and Peter Richtárik. Scaling distributed machine learning with { In-Network } aggregation. In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21) , pages 785-808, 2021.

ChonLam Lao, Yanfang Le, Kshiteej Mahajan, Yixi Chen, Wenfei Wu, Aditya Akella, and Michael Swift. { ATP } : In-network aggregation for multi-tenant learning. In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21) , pages 741-761, 2021.

Pat Bosshart, Glen Gibb, Hun-Seok Kim, George Varghese, Nick McKeown, Martin Izzard, Fernando Mujica, and Mark Horowitz. Forwarding metamorphosis: Fast programmable matchaction processing in hardware for sdn. In Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM , SIGCOMM '13, page 99-110, New York, NY, USA, 2013. Association for Computing Machinery. ISBN 9781450320566. doi: 10.1145/2486001.2486011. URL https: //doi.org/10.1145/2486001.2486011 .

N McKeown. Pisa: Protocol independent switch architecture. In P4 Workshop , 2015.

Yifan Yuan, Omar Alama, Jiawei Fei, Jacob Nelson, Dan RK Ports, Amedeo Sapio, Marco Canini, and Nam Sung Kim. Unlocking the power of inline { Floating-Point } operations on programmable switches. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22) , pages 683-700, 2022.

[62] Zhourui Song, Zhenyu Liu, and Dongsheng Wang. Computation error analysis of block floating 521 point arithmetic oriented convolution neural network accelerator design. In Proceedings of the 522 Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications 523 of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in 524 Artificial Intelligence , AAAI'18/IAAI'18/EAAI'18. AAAI Press, 2018. ISBN 978-1-57735525 800-8. 526

Peer reviews:
comments: SUMMARY:
This work proposes Flag Aggregator (FA) for a more robust aggregation of gradient in data-parallel training. FA formulates gradient aggregation as a Maximum Likelihood Estimation procedure using Beta densities. Theoretically, FA is analyzed using techniques from convex optimization. Empirically, FA demonstrates decent performance against Byzantine failure for image classification tasks (esp. ResNet-18 on CIFAR10) on a 4-GPU cluster networked with 100GbE.

STRENGTHS:
+. Proposed a simple Maximum Likelihood Based estimation procedure for aggregation purposes, with novel regularization functions

+. Provided code for reproducibility

+. Well-written: easy to follow

WEAKNESSES:
-. Marginal wall-clock time improvement, maybe due to heavy SVD overhead: e.g., Figure 10

-. Missing benchmark: 
1. only two small models are evaluated (e.g., ResNet18 and 2-layer CNN), how about more models like RNNs and larger models like GPT2?

2. only image classification tasks are evaluated (e.g., CIFAR10 and MNIST), not even CIFAR100 nor full ImageNet, how about more tasks like language modeling?

-. Missing modern cluster: 4-GPU cluster with one GPU per machine is not a modern setup for evaluating scalability of distributed training

QUESTIONS:
*. What if the Byzantine workers send more than just "uniformly random gradients"; how will the FA perform?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of Byzantine robustness in distributed learning by proposing a new robust aggregation rule called Flag Aggregator. The latter is based on maximum likelihood estimation with regularization. They empirically show that using distributed gradient descent with Flag Aggregator performs well against simulated Byzantine attacks compared to other existing solutions.\n\nSTRENGTHS:\nThe problem of Byzantine robustness is important in distributed learning. Moreover, the proposed Flag Aggregator seems to follow a creative approach.\n\nWEAKNESSES:\nMy main concern is the great lack of clarity of the paper, especially in the theoretical part. I also think that the theoretical and experimental parts lack several elements.\n\n* Lack of clarity: the paper has several clarity-affecting issues which makes it really hard to assess the technical contributions. \n\t* The paper starts (right away) with an unclear optimization problem (Equation 1): what are A, Y and C? \n\t* line 99: why is $Y Y^\\top G$ a \"reconstruction\" of $G$? and what is meant by reconstruction exactly?\n\t* lines 100-103: I could not verify the stated claims/intuitions. \n\t* line 116: why does orthogonality imply efficiency? Authors seem to say that it is because we can derive a one-rank matrix factorization, but this does not require orthogonality of the matrix. In fact, $YY^\\top G$ is just $G$ if $Y$ is orthogonal.\n\t* lines 123-135: this paragraph assumes that the reader knows what the Flag/Grassmanian manifold is, which was not the case for me.\n\t* Section 2.2: where does the vector $v$ come from? It is directly sent by the workers? Also, why do you assume that it follows a Beta distribution?\n\t* Algorithm 1: I could not find IRLS explained in the text. Also, it is strange that workers locally perform the update step. It always happens at server level in distributed SGD.\n\t* line 163: what is Flag Median?\n\t* line 188: what is a \"second order optimal local solution\"?\n\n* Lack of convergence guarantees: After all, a Byzantine-robust learning solution should have convergence guarantees, since simulated attacks are not guaranteed to be optimal; i.e. instantiate worst-case adversaries. Typically [Karimireddy et al. 2022, Allouah et al. 2023], convergence to a neighborhood of the original solution is ensured in the presence of Byzantine workers for smooth non-convex losses. \n\n* Experimental section: I suggest simulating more Byzantine attacks. The tested attacks (uniformly random vectors) are extremely weak compared to FoE [Xie et al. 2020], ALIE [Baruch et al. 2019] and others, which is unfortunate since the paper consider Byzantine adversaries. Also, some advanced defenses like NNM [Allouah et al. 2023] and Bucketing [Karimireddy et al. 2022] are missing although they were intended for non-iid; it is important to check how they perform against your method to assess the significance of the contribution.\n\n[Allouah et al. 2023] Fixing by Mixing: A Recipe for Optimal Byzantine ML under Heterogeneity, AISTATS 2023.\n\n[Karimireddy et al. 2022] Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing, ICLR 2022.\n\n[Xie et al. 2020] Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulation, UAI 2020.\n\n[Baruch et al. 2019] A Little Is Enough: Circumventing Defenses For Distributed Learning, NeurIPS 2019.\n\nQUESTIONS:\nI suggest that the authors address the weaknesses listed above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose a gradient aggregation method for distributed optimization that is robust to Byzantine device failures in large scale distributed setups. In each round, given the set of gradients from each workers, the authors aim to find the optimal low-rank subspace that can explain the variance of a majority of the gradients. The authors formulate the problem as a MLE under a beta distribution setup  and solve an approximate version of the problem though SDP.\n\nSTRENGTHS:\nByzantine device failures is an important concern for large scale clusters.  The presented method is well motivated theoretically and backed up with experiments comparing their robustness properties to other aggregation methods. Results demonstrate a significant advantage of this aggregation setup.\n\nWEAKNESSES:\nAlthough it is evident that Byzantine failures can have a significant impact on gradient computation if using simple aggregation rules, its unclear how often such failures happen in the cluster sizes the authors have considered. Augmentation pipelines induce their own noise to gradient information, but its unclear if these will be adversarial in _each_ update step. The amount of noise induced and its effect on adversarial training setups is also not evident.  (See questions). This makes it unclear how the clear advantages of the method translates to real-world workloads especially considering that the method adds a potentially expensive top-k SVD computation step.\n\nQUESTIONS:\n- Frequency of Byzantine failures: Could you provide some insights into how frequent failures due to hardware/software/augmentation pipeline based issues occur in training runs.  Assuming there will be at least a single byzantine worker at all times (i.e $f\\ge1$ in Fig 4) seems too strong and can be better contextualized with some supporting evidence.\n- Scalability of method to federated clusters: Byzantine failures will potentially be a larger concern when training over heterogeneous hardware and partially available clients, for ex. in the federated clusters. Could the authors comment on the feasibility of running the method in such settings, considering that the majority of the computation is performed at the central server?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new appraoch for aggregating gradients for distributed ML training under Byzantine failures, noise due to data augmentation, etc. The approach relies on constructing a low-dimensional subspace such that the proportion of variance of the gradient vectors contained in the subspace is maximized. The authors derive the loss function for their setting and formulate the problem as a regularized convex optimization problem which can be solved with standard solvers to obtain the basis for the subspace. The update direction is then obtained by projecting the individual gradients onto the basis and then averaging the result. Experiments on different datasets and number of workers show improved prediction accuracy over baselines when distributed training is performed using the proposed method.\n\nSTRENGTHS:\n1. The proposed approach is principled and easy to interpret as it tries to identify the subspace which contains the maximum proportion of the variance of the gradients and is also easy to implement due to its formulation as a regularized convex optimization problem which can be solved by off-the-shelf workers.\n\n2. The approach is extensively evaluated on a range of datasets (MNIST, CIFAR10, tiny-Imagenet) and for different noise models (random noise, adversarial data augmentation etc). I also appreciate the authors presenting results on wall-clock time to accuracy and per-iteration time thereby acknowledging the extra time required per iteration in their approach to compute the aggregated gradients. This opens the door to future research on speeding up the proposed aggregation method while retaining the accuracy gains.\n\nWEAKNESSES:\n1. My main concern with the approach is its novelty. Since the goal appears to be to estimate the subspace containing the maximum proportion of gradient variance, I am not sure why this cannot be done by retaining the top-k Principal Components of the gradients. The authors even acknowledge in line 109 that the idea to use the ratio of variance of projected and true gradients has been explored in the Robust PCA literature. However, they do not explain why simply considering the principal components will not work, nor do they perform experiments with PCA/Robust PCA as baselines. I would like to see at least one of the two (explanation/experiments) to be convinced of the need for the proposed approach and its gains over PCA.\n\n2. The extra computational cost and the added time per iteration as seen in Fig 10 (b) is also a weakness. While I do appreciate the authors measuring and presenting this time, it is not clear at this point if the accuracy gain justifies the extra time. One way to demonstrate this would be to allow the other approaches to run for the same amount of time in Fig 10 (c). If it could be shown that even after running for that long these approaches cannot match the accuracy of Flag Aggregation, then the extra time required could be justified.\n\nQUESTIONS:\n1. Could you please explain more clearly (preferably with an example) why adversarial training could lead to noise in gradients? The current explanation in lines 59-61 is too vague and high-level. Clarifying this would be useful for readers not familiar with the adversarial training literature, and would strengthen the motivation of the approach.\n\n2. Please introduce/explain the term Flag Optimization before using it in line 75, or add a citation since I don't think readers outside the optimization community would be familiar with this term.\n\n3. Fig. 5 seems to suggest that Flag Aggregation is only useful for bs >= 128? Is that indeed the case? What is the value of bs in the other experiments? \n\n4. Likewise Fig. 6 seems suggest that it is useful only for p >= 11. Please clarify if that is indeed the case. Note that, gains only in certain regimes of bs and p won't necessarily be a reason for rejection. But it is important to acknowledge it in the paper so that the readers have all the information.\n\n5. In line 119 you mention that gradient quality from workers may differ if workers use different batch size. I think this is a very interesting and practical scenario. Did you perform any experiments where workers used different batch sizes? Will it be possible to present the behaviour of Flag Aggregation and other baselines in this scenario?\n\n6. Can methods from randomized linear algebra, or other approaches to speed up SVD help in reducing the per-iteration time of your approach? If yes, it might be worth mentioning this in the paper as an option for readers looking to implement your approach.\n\nLIMITATIONS:\nI feel the main limitations of the work are the increased computation time per iteration and the lack of clarity on novelty w.r.t PCA. I appreciate the authors' acknowledgement of the higher per-iteration time and look forward to their responses to the other limitations that I have identified under Weaknesses, above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new method to aggregate gradients in a distributed training setting. Effectively, the proposed algorithm projects gradients onto a learned lower dimensional subspace and then aggregates the projections using standard techniques like averaging. This leads to a more robust aggregation against Byzantine failures. The projection is similar to a robust PCA, and is learnt using an approximate MLE via a Taylor expansion, leading to a computationally more feasible algorithm. Thorough experiments are conducted that demonstrate the efficacy of the proposed algorithm.\n\nSTRENGTHS:\n1. The proposed novel algorithm empirically performs better than existing methods when measured by iteration complexity.\n2. The authors provide a thorough comparison to existing methods and place their work in context.\n\nWEAKNESSES:\n1. The exposition in the paper lacks clarity in some places -- for example, the IRLS subroutine in Algorithm 1 is not described or even briefly summarized in the main paper.\n2. The authors do not present their theoretical convergence results in the main body of the paper. \n3. As pointed out by the authors, the main proposed algorithm does not seem to perform significantly better than other existing algorithms when comparing wall clock runtimes.\n\nQUESTIONS:\n1. What exactly is the IRLS procedure in Algorithm 1? There is no description in the main body of this subroutine or procedure.\n2. In algorithm 1, line 5, the procedure to find the approximate subspace $\\hat Y$ is done only on the server. Presumably, this procedure would be the bottleneck in the whole algorithm, since it involves multiple SVD computations. Can the authors comment on the breakdown of which parts of algorithm 1 take significantly more time, and explain any optimizations they have implemented in this context?\n\nLIMITATIONS:\nYes, the authors addressed the limitations of their work in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}