Exploring the Impact of Attacks on Ring AllReduce
Jiayu Wang, Peng Liu, Zehua
Guo∗
Beijing Institute of Technology
China

Sen Liu

Chao Yao

Peng Cheng Lab, Fudan University
China

Shaanxi Normal University
China

ABSTRACTDistributed Machine Learning (DML) is widely used to accelerate
the training of the deep learning model. In DML, Parameter-Server(PS) and Ring AllReduce are two typical architectures. Recently,
observing that many works address the security problem in PS,
whose performance can be greatly degraded by malicious participation during the training process. However, the robustness of RingAllReduce, which can solve the communication bandwidth problem
in PS, to the malicious participant is still unknown. In this paper,
we design a series of experiments to explore the security problem
in Ring AllReduce, and reveal it can also suffer from the malicious
participant.
ACM Reference Format:
Jiayu Wang, Peng Liu, Zehua Guo, Sen Liu, and Chao Yao. 2021. Exploring
the Impact of Attacks on Ring AllReduce. In 5th Asia-Pacific Workshop onNetworking (APNet 2021), June 24–25, 2021, Shenzhen, China, China. ACM,New York, NY, USA, 2 pages. https://doi.org/10.1145/3469393.3469676

1INTRODUCTION

Deep learning has been successfully applied to many applications
in recent years. However, the high cost in computation and memoryof deep learning still hinder its implements. To solve this problem,Distributed Machine Learning (DML) has been proposed by using many machines simultaneously. Based on different topologicalarchitectures, Parameter Server (PS) and Ring AllReduce are two
typical DML methods. In PS, a central server is used to coordinate
the training of different participates (usually named workers inDML). The server first sends a model to each worker, then eachworker computes the gradient/parameter based on its local data,and sends the gradient/parameter back to the server. The server
finally updates the model by the collected gradients/parametersfrom the workers. The process will be repeated many times, so thecommunication bandwidth is a heavy burden for the central server.
Ring AllReduce [3], which deploys the worker in a ring topology, isproposed to solve the problem. In Ring AllReduce, each worker only
needs to receive parameters from one worker and send parameters
to another worker. Consequently, the communication bandwidthof Ring AllReduce is greatly saved.
∗ Zehua Guo is the corresponding author.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
APNet 2021, June 24–25, 2021, Shenzhen, China, China© 2021 Association for Computing Machinery.
ACM ISBN 978-1-4503-8587-9/21/06. . . $15.00
https://doi.org/10.1145/3469393.3469676

Training a distributed machine learning model is not safe due tocyberattacks. The malicious workers, which are controlled by theattacker, can send the wrong gradients or parameters to the server.
An example is presented in Figure 1. In the example, the gradients
estimated by the correct workers have a similar direction, the one
from the malicious worker can be different from the correct ones.
In this way, a wrong estimation of the global gradients is produced,
leading to degraded performance or divergence of the model.
Based on the scheme in Figure 1, different attacks are proposed
recently [2]. To prevent PS from the attack, many works are also
proposed. [1] kicks out the malicious workers to prevent PS from attack. [4] aggregates the gradients of each worker, rather than briefly
calculate the average of them, to obtain robust global gradient.
However, existing works mainly focus on the security problem
in PS. To the best of our knowledge, the impact of malicious workers on the performance of Ring AllReduce is still unknown yet.
Ring AllReduce does not have a central server, and gradients areprocessed as they arrive at workers instead of being aggregatedcentrally at the server. Thus, none of the workers can work from a
global perspective, and existing defending methods for PS cannot
be directly used. In this paper, we address the security problem
in Ring AllReduce. We design several experiments to test the influence of malicious workers when training the DML models inRing AllReduce. The experimental results show that the malicious
workers can reduce the accuracy of the models, and even can leadto the models disconverged in the worst case.
cost funciton
gradient estimates
computed by correct
workers
the actual gradientgradient estimates
computed bymalicious workers
wrong global
gradient(a) The calculation of the graidnets with- (b) The calculation of the graidnets with maout malicious workers.
licious workers.

Figure 1: The calculation of the gradients with malicious
workers can be highly different with no malicious workers.
It can hurt training process with malicious workers.

2 EXPERIMENT DESIGN
2.1 Overview of Ring AllReduce
Training the Ring AllReduce architecture contains many iterations.
In each iteration, each worker firstly calculates the local gradients
based on the local data. Then, the model is updated by two processes,
Scatter-Reduce and Allgather.
In Scatter-Reduce, each worker sends the gradients to the fixedadjacent worker and receives the gradients from another fixedadjacent worker. After N − 1 steps (where N is the number of totalworkers), each worker has parts of the global updated model. Then
Allgather is adopted to update the global model. After N − 1 steps,
each worker gets the updated global model. Figure 2 shows thescheme of one iteration in Ring AllReducde.

APNet 2021, June 24–25, 2021, Shenzhen, China, China

Send
Worker3

Send

Receive

Send

Receive
Worker2(a) Each worker only receives parameters (b) When there is a malicious worker, thefrom the fixed worker and sends parame- tempered parameters of it will be sent toters to another fixed worker.
all the other workers after N − 1 steps.

Figure 2: The basic architecture of Ring AllReduce.

2.2

System Attack and Environment Design

Based on the aforementioned training process, it is easily foundthat the local gradients are spread to all of the workers. Thus, we
infer that the malicious worker will also affect the performance ofRing AllReduce. To prove this thought, we use several attacks asfollows:
Gradient flipping attack: The malicious workers calculate
the local gradients based on the local training data, and multiply
local gradients by -1 as the transmitted gradients between workers.
In other words, it changes the sign of the parameter.
Gradient ascent attack: The malicious workers calculate
the local gradients based on the local training data and multiply
local gradients by -4 as the transmitted gradients between workers.
Compared with gradient flipping attack, it is a more powerful attack.
Random value attack: The malicious workers do not needto calculate the gradients. Instead, it will replace each parameterwith a random number and each number will follow the normal
distribution and multiply by a coefficient. In our experiment, thecoefficient is set to be 0.3.
Same value attack: The malicious workers do not need to
calculate the gradients. Instead, it will replace all the parameterswith the same value. In our experiment, we set it to 100.
Label flipping attack: For the training data, if the real label
is x, we will set the fake label as 9 − x. Then the malicious workers
calculate gradients based on the fake labels. And send the calculated
gradients to the server. Then in the Scatter-Reduce process, afterN − 1 steps, the tempered parameters will be sent to all the otherworkers, which can make an influence the global parameters in eachworker. The Allgather process further uses the tempered global
parameters to update the global model. As a result, the global model
can be influenced.

3EXPERIMENT RESULT AND ANALYSIS

In our evaluation, we use CIFAR10 as the training dataset, CNN andAlexnet as the training model. For the CNN model, there are twoconvolution layers followed by two fully connected layers. We use
8 and 16 workers for the experiments and perform the experimentswith one attacker.
We test 6 settings, which are no attackers and 5 aforementioned
attacks, both in CNN and Alexnet. The experimental results areshown in Figures 3 and 4. It should be noted that we stop thetraining when the model cannot converge (e.g., the loss for Alexnet
in Figure 3 (b)). It shows that the attack on Ring AllReduce will
result in a decrease in the training accuracy. In the worst case, itwill divergence the training model. For the same attack, the effects
on different training models are distinct, which reveals that thearchitecture of the model leads to different abilities to resist attack.
In the experiment, when there are 16 workers with one attacker, theeffect of the attack can be decreased compared with 8 workers. It isbecause that, the proportion of attackers in participants is reducedin this situation. In fact, if there are more attackers, the decrease of
the training accuracy and the divergence of the training model will
be more obvious.
0.8No attacker
Gradient flipping attackGradient ascent attack
Same value attack
Random value attack
Label flipping attack

0.60.40.2

5000 1000015000200002500030000
Iterations(a) The training accuracy of Alexnet.

No attacker
Gradient flipping attack
Gradient ascent attack
Same value attack
Random value attack
Label flipping attack

0.4
0.2

1.5
1.0
0.5
0.0

00.6

No attacker
Gradient flipping attack
Gradient ascent attack
Same value attack
Random value attack
Label flipping attack

2.0
Training loss

SendReceive
Worker 2

Worker1

step2
step3
step1

0 5000 1000015000200002500030000Iterations(b) The training loss of Alexnet.

Training loss

Receive

Receive

Receive

No attacker
Gradient flipping attack
Gradient ascent attack
Same value attack
Random value attack
Label flipping attack

2.0
1.5
1.005000 1000015000200002500030000
Iterations(c) The training accuracy of CNN.

05000 1000015000200002500030000
Iterations(d) The training loss of CNN.

Figure 3: The results of the experiments with 8 workers.
2.50.8No attacker
Gradient flipping attack
Gradient ascent attack
Same value attack
Random value attack
Label flipping attack

0.6
0.40.2Training loss

Worker3

Send

Send

2500 5000 7500 100001250015000
Iterations(a) The training accuracy of Alexnet.
0.6
0.4
0.2

No attacker
Gradient flipping attack
Gradient ascent attack
Same value attack
Random value attack
Label flipping attack

0 2500 5000 7500 100001250015000
Iterations(c) The training accuracy of CNN.

2.0
1.5
1.0

No attacker
Gradient flipping attack
Gradient ascent attack
Same value attack
Random value attack
Label flipping attack

0.5
0.0

00 2500 5000 7500 100001250015000
Iterations(b) The training loss of Alexnet.

Training loss

SendWorker4

Testing accuracy

Receive

Worker1

Testing accuracy

Receive

Testing accuracy

Send

Testing accuracy

Worker4

Jiayu Wang, Peng Liu, Zehua Guo, Sen Liu, and Chao Yao

2.0
1.5

No attacker
Gradient flipping attack
Gradient ascent attack
Same value attack
Random value attack
Label flipping attack

1.0
0 2500 5000 7500 100001250015000
Iterations(d) The training loss of CNN.

Figure 4: The results of the experiments with 16 workers.

4

CONCLUSION AND FUTURE WORKIn this paper, we explore the security problem in Ring AllReducearchitecture, and find that the malicious workers indeed affect themodel’s performance. In future, we will evaluate the impact ofexisting attack methods on complex training models, datasets, andother processes in Ring AllReduce.

ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China under Grants 62002019 and 62002066, Beijing Instituteof Technology Research Fund Program for Young Scholars, Fundamental Research Funds for the Central Universities under Grant
1301032207, and the project "PCL Future Greater-Bay Area Network
Facilities for Large-scale Experiments and Applications (LZC0019)".

REFERENCES[1] Peva Blanchard and et al. [n.d.]. Machine learning with adversaries: Byzantine
tolerant gradient descent. In NIPS’17.
[2] David Enthoven and Zaid Al-Ars. 2020. An overview of federated deep learning
privacy attacks and defensive strategies. arXiv preprint arXiv:2004.04676 (2020).
[3] Pitch Patarasuk and Xin Yuan. 2009. Bandwidth optimal all-reduce algorithms forclusters of workstations. Elsevier JPDC 69, 2 (2009), 117–124.
[4] Dong Yin and et al. [n.d.]. Byzantine-robust distributed learning: Towards optimal
statistical rates. In ICML’18.