Lockdown: Backdoor Defense for Federated Learning  with Isolated Subspace Training
Authors: Tiansheng Huang
Sihao Hu
Ka-Ho Chow
Fatih Ilhan
Selim Furkan Tekin
Ling Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: Federated learning
backdoor defense
isolated subspace training.
TL;DR: We propose Lockdown, a backdoor defense solution for federated learning to counter poision-coupling effect.

Abstract: Federated learning (FL) is vulnerable to backdoor attacks due to its distributed computing nature. Existing defense solution 
usually requires larger amount of computation in either the training or testing phase, which limits their practicality in the resource-constrain scenarios. A more practical defense, i.e., neural network (NN) pruning based defense has been proposed in centralized backdoor setting. However, our empirical study shows that traditional pruning-based solution suffers \textit{poison-coupling} effect in FL, which significantly degrades the defense performance.This paper presents Lockdown, an isolated subspace training method to mitigate the poison-coupling effect. Lockdown follows three key procedures. First, it modifies the training protocol by isolating the training subspaces for different clients. Second, it utilizes randomness in initializing isolated subspacess, and performs subspace pruning and subspace recovery to segregate the subspaces between malicious and benign clients. Third, it introduces quorum consensus to cure the global model by purging malicious/dummy parameters. Empirical results show that Lockdown achieves \textit{superior} and \textit{consistent} defense performance compared to existing representative approaches against backdoor attacks. Another value-added property of Lockdown is the communication-efficiency and model complexity reduction, which are both critical for resource-constrain FL scenario. Our code is available at \url{https://github.com/git-disl/Lockdown}.

Lockdown: Backdoor Defense for Federated Learning with Isolated Subspace Training
Tiansheng Huang, Sihao Hu, Ka-Ho Chow, Fatih Ilhan, Selim Furkan Tekin, Ling Liu
School of Computer Science Georgia Institute of Technology, Atlanta, USA
(thuang374, shu335, kchou35, filhan3, stekin6) edu, ling. liu@ce. gatech edu
Abstract
Federated learning (FL) is vulnerable to backdoor attacks due its distributed computing nature. Existing defense solution usually requires larger amount computation in either the training testing phase, which limits their practicality in the resource-constrain scenarios. more practical defense, i.e., neural network (NN) pruning based defense has been proposed in centralized backdoor setting. However, our empirical study shows that traditional pruning-basee solution suffers peison-coupling effect in FI., which significantly degrades the defense performance. This paper presents Lockdown, an isolated subspace training method to mitigate the poison-coupling effect. Lockdown follows three key procedures. First, it modifies the training protocol by isolating the training subspaces for different clients. Second, it utilizes randomness initializing isolated subspacess, and performs subspace pruning and subspace recovery to segregate the subspaces between malicious and benign clients. Third, introduces quorum consensus to cure the global model by purging malicious/dummmy parameters. Empirical results show that Lockdown achieves superior and consistent defense performance compared to existing representative approaches against backdoor attacks. Another addue-added property of Lockdown the conmunication-fficiennyy and model complexity reduction, which are both critical for ressurce-constrail FL. scenario. Our code is available at hittttt:/// ///gtttttt//tt// com/git dislg/tkddss//ooooonn
Introduction
Federated Learning (FL) (MeMahan al.) 2016) is privacy-preserving machine leamn ing paradigm that allows the training surrogates (clients) to collectively train a global model with data in local devices. However, because the training data and potentially the training process on the clients lacks supervision, it is possible that attackers on the global model (Tolpegin et al.) 2020), so to manipulate the prediction of the model.
Medel distnibution
Subapace diaribution
dnvelent spocate
Sabrppace opptiabe
launch data poisoning attack
Malicouu clent
Benign
Figure 1: Illustration of isolated subspace training. The blue circles and the shaded area respectively represent the parameters of the model and the train- ing subspace. Left: vanilla FL where malicious clients can poison all the parameters. Right: iso- lated subspace training where malicious clients can only poison a subspace of them.
Backdoor attack one of the data poisoning attacks, which is stealthy and disruptive to the normal behavior of the model. Specifically, the prediction of the model can be manipulated such that it consistently predicts one (or some) chosen label whenever is given samples with backdoor trigger. Backdoor attack poses serious threats to
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

many security-critee critical applications, e.g., biometric authentication and autonomous driving (Chow & Liu 202Ip. which vitiates the large-scale deployment of FL. Therefore, an effective defense that can mitigate such risk is in an urgent need.
The main stream of existing research defend backdoor attack in FL. can be mainly classified into three genres, i) certified robustness (Xie et al. 2021), (Alfarra et al. 2022). ii) adversarial training
(Zizzo et 2020 Shah al.) 2021). and iii) robust aggregation (Chow et al. 2023; Zhang et al. 2022a Though existing defenses can mitigate attack, they are still far from their maturity. In
particular, we highlight the in- efficiency of state-of -ate---f-th-rrr-reee defense solutions.
Existing defenses usually require extra computation in either the training or inference phases. For example, randomized smoothing, the key component for certified robustness defense, requires multi- ple forward pass of data in the inference phase for producing one effective prediction. Analogously, adversarial training requires generate adversarial samples in the inner loop of training, and therefore demands extra data pass. This issue is particularly pronounced in the FL scenario, considering that training and the inference are conducted on edge devices with limited capacity (Wang al. 2019b).
In this paper, we try to answer the following question:
Is there a solution that not only provides SOTA defense performance, but also requires equal or even less computation in both training and deployment phases?
To this end, we first apply the pruning- pruning-based defense solution in FL. backdoor setting, but we observe a poison-coupling effect, which significantly degrades the defense performance. Driven by this observation, we propose Lockdown, solution that utilizes isolated subspace training to de-couple the poison parameters, which are then pruned using the consensus from clients. Our experiments demonstrate that: i) Lockdown reduces the Attack Success Ratio (ASR) by up-to 90% compared to FedAvg without defense. ii) Lockdown consistently performs better than SOTA defense solutions in various attack settings. Specifically, Lockdown acquires 83% of ASR reduction with only 2.7% drop of benign accuracy when the data distribution is Non-IID (the other two defense baselines RLR and Krum respectively acquire 14.2% and 86. % ASR reduction while sacrificing 16.2% and 43.6% benign accuracy in the same setting). iii) Lockdown reduces both downlink and uplink communication by at least 0.75x, and the number of parameters used in the model training and inference phases also accordingly reduced by at least 0.75x thanks to the removal of malicious/dummy parameters.
To the end, we summarize our contribution as follows:
We study the data-free pruning defense in FL setting, and find that the poison parameters tend be statistically coupled with benign parameters, which we refer to as "poison-coupling" effect in FL backdoor. To the best of our knowledge, this finding is available in the literature.
We propose Lockdown, backdoor defense that utilizes the idea of isolated subspace training to decouple the poison parameters. Notably, Lockdown i) enjoys communication reduction between server and clients, and ii) lowers the model's 's training/inference complexity.
We conduct evaluations to show the efficacy of Lockdown. Results show that Lockdown consistently outperforms existing defense baselines under different attack settings (attack method, attacker number and poison ratio) and different data distributions (IID and Non-IID).
Ablation study and hyper-parameter sensitivity analysis are conducted to verify the individual functionality of each component of Lockdown.
Related Work
Federated Learning. Federated learning (McMaha et al., 2016) is crivacy-preservve distributed training paradigm that allows clients to collectively train a global model from distributed training
data. Recent works on FL mostly lie in optimization (Karimireddy et al.
2020 Li ct al. 2018 Sun
et al. 2023a c Li et al. 2023b; Sun et al. (023b), which study how to mitigate the Non- IID issue
(Zhao et al.) 2018) and system efficiency (Huang et al.] 2020 2022a Li al. 2021bla).
Federated backdoor attack and defense. Classical backdoor attack FL empirically proven to be effective in (Bagdasaryan et al. 2020). and subsequently, new types of attack are developed, e.g., edge-case backdoor (Wang et al. 2020), stealthy model poisoning (Bhagoji et al. 2019). subnet
replacement attack (Qi et al. 2022), and Distributed Backdoor Attack (Xie et al. 2019).

---

Backdoor defense are developed to counter the potential threat posed by the backdoor attacks (representative methods are neural cleanse (Wang al. 2019a), meta analysis (Xu al. 202T). etc). Under the FL. setting, backdoor defenses can be classified into three main genres. The first genre is certified robustness. Certified robustness relies on randomized smoothing(Cohen et al. 2019), an approach with theoretical certification of the model robustness. Subsequent techniques, e.g. weight smoothing (Xie et al., 2021) ) and group ensemble (Cao et al. 2022) are studied for the defense of federated backdoor. The second genre is adversarial training (Zizzo et al. 2020 Shah et al. (021), which generates adversarial samples to improve the model's robustness. However, both certified robustness and adversarial training require more computation in either the training or deployment stage. The third genre is robust aggregation, which studies how to identify and preclude the malicious
update in the aggregation stage (Chow et al. 2023 Zhang al.) 2022a Guo et al. I02T] Panda et al.,
2022). However, the classification benign accuracy of this category of defense often perform worse when Non-IID extent increases.
Sparse training. Sparse training originally designed to reduce the model complexity for both the training and deployment stage. SET (Mocanu al. 2018) first proposes the idea of dynamic subspace searching with alternative pruning and recovery process, and are empirically studied with the concept of in-time vre-pparameterizationn (Liu et al. ab 2022). (Evci al. 2020) further introduces gradient information to guide the mask searching process. This model compression technique has
recently been extended to FL (Bibikar et al. 2022 Huang et al. 2022b Dai et al. 20022).
We utilize sparse training technique develop a new genre of backdoor defense solution. To the best of our knowledge, this is the first attempt that connects sparse training with backdoor defense in FL..
3 Threat Models
We consider three threat models, named weak, medium and strong. All the models allow multiple attackers to co-exist in the system. We use N to denote the number of attackers out of M total clients.
Table 1: Permission of threat models.
Permission hreat model
weak
medium
strong
Data manipulation
Algorithm manipulation
Global information Access
X
X
Aggregation manipulation
X
X
x
Permission. Different threats models are given different control permissions. Specifically, Weak model allows the attackers to arbitrarily manipulate its local data, but cannot control its local training process, while attackers in medium and strong model are allowed to do so. Only strong model allows the attackers to obtain other benign client's information, e.g., update from them. For control permission in the weak threat model, technique like trusted execution environment (Mo et al.) 2021) can be applied to control each client to run the designated program, and thereby disabling algorithm manipulation. Existing literature usually adopt medium threat model, in which attackers can do whatever they like in their local devices, but has not extra information from server or from other benign clients. The permission of the threat models are summarized in Table
Malicious objective. Denote the set of benign clients as M and the set of malicious clients as N. Formally, we characterize the objective of malicious clients as follows:
min M
Ef(ww) + Σ f((w)
(I)
iEN
iEM/N
where f.(w) 3 1D. EieYYDED vı),) is the malicious objective of an attacker (to mini-
mize the average cross-entropy of the backdoor dataset), fi(w) A D. (, CE(w; 25 25, the benign objective (to minimize the average CE loss for the dataset from benign clients), and D, and D, are respectively the backdoor dataset and pristine dataset.
Attack methods. Different attack methods are eligible in different threat models. For weak threat model, only data-level backdoor, e.g., BadNet (Gu et al. 2017), DBA (Xie et al. 2019), and Sinusoidal
3

---

(Barni et al., 2019) can be applied, since the attacker has no control to the local training, but only has access to the dataset. For medium threat model, the attacker can perform a wider range of attack methods, e.g., Scaling (Bagdasaryan et al. 2020), FixMask, etc, which requires the modification of the local training process. For the strong threat model, we test Neurontoxin (Zhang et al. 2022b). and Omniscient, which requires extra server-side information to aid the attack.
Defense goal. While solving the benign objective (see FL general objective in (McMahan al. 2016)), we expect the global model w to be able to resist backdoor attack. In other words, the global model should minimize the benign objective while maximizing the malicious objective.
4 Case Study on Pruning-based Defense
Motivated by our goal to propose a computation-frieet defense originally proposed for centralized ML to federated learning setting, We study CLP defense proposed in (Zheng et al. 2022) towards two poisoned models trained with centralized SGD (central- ized backdoor) and the global model produced by FedAvg (federated backdoor) | Surprisingly, observe that the pruning defense exhibits substantially different performance on the two models.
defense, we extend the pruning-based
toderated backdoor
Higher praning ratio
federated baciddor
fedorated backdoor
centrallzed backdoor
of channels
centralized backddor
centraliaee backdoor
50-
1.5
norm
40-
20-
0.0
0.5
0.1
0.4 Channel Lipschitzness
300
200
300
Benign Acc
channel sorted by norm
Figure 2: Properties of two models trained with centralized backdoor and federated backdoor, Left: ASR and benign accuracy with CLP defense in (Zheng et al.) (022). Middle: Channel lipschitzness of last convolutional layer of two models. Right: L2 norm of last convolutional layer of two models. The experiment done on a ResNet-9 model on CIFAR dataset and under BadNet attack.
Observations. In the left of Figure 2 we observe that the pruning algorithm cannot efficiently climinate the backdoor parameters for federated backdoor. The ASR can only drop to an acceptable range (say 30%) when leveraging large pruning ratio, and accordingly, this also leads to a large drop of benign accuracy (say 35% drop). In the middle, we plot the channel lipschitzness (CL) of the two models, and find that the CL values for federated backdoor couple in more compact range, which makes hard for the pruning method to find out the malicious channels to prune. This explains why CLP fails to work for federated backdoor. In the right, we plot the L2 norm of weights of the last convolutional layer, the same coupling effect for federated backdoor can be observed - the 1.2 norm of different channels are all within small range.
Poison-coupp couoonn effect. In summary, our main takeaway is that the federated backdoor model is hard to be cured by classical pruning method, because the backdoor parameters does not exhibit substantial statistical difference with the benign ones, i.e., it tends to be coupled with the benign parameters. We refer to this phenomenon for federated backdoor as poison-coupling effect. Due to the existence of ppison-coupling effect, it is hard, if not impossible, poisoned parameters after the model has been fully trained by federated learning process, and thereby reducing the efficiency of the pure parameters pruning defense.
accurately identify the
5 Methodology
To mitigate the poison-coupling effect, we propose lockdown with isolated subspace training. Intu- itively, we don't 't allow the malicious clients get access to all the parameters, but only a subspace of them (See Figure [T). in order to prevent them from statistically coupling the poisoned parameters with the benign ones that supposed to do normal function.
"Checkpoints of the centrallized/feeddrraed backdoor models are available in scl
com/

---

The high-level idea of Lockdown is as follows. i) we employ isolated subspace training to restrain the training of each client into its own subspace, such that the backdoor data cannot access and couple the poisoning function in all the parame- ters. ii) we employ dynamic subspace searching for the clients to search for local subspaces us- ing their local datasets, which involve only the parameters that they deem important. iii) After the above local procedures, the server will aggre- gate the gradient updates into the corresponding subspaces of global model, and continues the next round of training. iv) Repeating T rounds of training, the server can identify the malicious parameters by consensus fusion. Under the in- tuition that the malicious/dummy parameters (at specific coordinates) should have less chance to be involved in benign clients' subspaces (because they are not important parameters for the benign datasets). Considering the benign clients constitute the majority in the system, consensus fusion iden- tify those parameters that have the least appearance among all the clients subspace as the malicious or dummy parameters, which are then pruned mitigate backdoor behavior. As follows, we detail each step of Lockdown.
Server
Sentigtiifarre
Aggrecation
Clean
O0O
Upleareed subspace's 's update
a Uifood server
Berign cleees
Supeepace
Subopace
Dutoeet
Deckiderr
Figure 3: Overview of Lockdown. First, clients receive subspace model from server, and apply iso- lated subspace training. Second, clients upload their subspace update. Third, server aggregates the updates into global model. After training, consen- sus fusion is applied to remove poison params.
Subspace initialization. We enforce an overall sparsity to the initial subspace of each client (denoted by a binary variable m.,o). ) For maintaining practical performance of sparsity, we fol- low (Evci et al.) (0200) to use ERK for random subspace initialization, in which different layers in the corresponding subspace are designated different sparsity (See Appendix A. for details). For each client, we enforce them have the same initial subspace, which can be achieved via enforeing the same random seed them for subspace generation. However, we do find that heterogeneous mask initialization can increase protection in some cases (See Section 6.2).
Algorithm 1 Lockdown defense
input Training iteration T; Local steps K; Learning rate ry: Pruning/reoovery rate au decayed by cosine annealing (Specially,
= 0, i.e. no recovery for first round); Random seed seed; Initial model wo- output Clean model for deployment ur
Isolated subspace training. In this stage, each client performs multiple local steps to train pa- rameters within its isolated subspace (as en- forced by its mask). Specifically, for local step k O..... K- clients do this update:
I: main Server's Main Loop
2:
74.00 = Suusspacenittsseeeee
for E M
3:
for t= 1,....... T do
4:
for i E .M do
Send W;,10 = m., @ w, to client i 6: Call Client i's main loop for training
7:
Received W.,.K and m.,441
WL,,A+11 Wittk T/m e C V.L
8:
end for
(2)
10: end W w.- 1 Ein m (W..w...kk) w...k)
9:
where & is a piece of random sample within the local dataset, is the learning rate, and @ de-
11: ui = ur @ To mM.r)
notes Hadamard product. As shown, the binary
12: Deploy ur for serving/inference.
mask m is applied to the gradient in every local step, which isolates the training into i-th client's 's own subspace. This process prevents the backdoor data from contaminating all the parameters within the parameter space.
13:
end main
14:
main Client's Main Loop
15: Obtain local gradient V (i(
16:
mi 1 m., + ToTppp.......... a, (1V f.
17:
for k = 0, I....,KK K do
18: W1,t wi... 1m oV (.
Subspace searching. We conduct subspace searching for clients to search for their sub- spaces, i.c., the parameters that are important per the data they have. The searching process
19:
end for
20:
mi = ArgBottomK, on ((w
21:
Send w...K and m.,++1 to server
22:
end main
decomposed two phases, as follows:
Subspace pruning. In this phase, we prune the unimportant parameters within the client's 's current subspace that do not function at all. After K steps of local training, we can identify the unimportant parameters as those with the smallest magnitude within the subspace. To preclude them from future training, prune those smallest a parameters within each layer, and accordingly update
5

---

the mask to mi.+1. Formally, this process can be formulated as follows:
m1,+1 = 1 - ArgBottomk, (lw2,,,xl)
(3)
where ArgBottomK (/wo)) return the a percentage of smallest coordinates of each layer abso- luted weights and mask them to 1, indicating that they will be pruned. We use cosine annealing to decay ag from the initial pruning rate cu- The decay process is postponed the Appendix [A.I]
Subspace recovery. After pruning, and before the start of next round local training, the client recovers the same amount of parameters (as indicated by at-1) to its subspace for exploration. To identify which parameters should be recovered, we use each client's 's data to extract the gradient w.r.t the weights after pruned, i.e., to extract V (.(,,,,..)). Then we recover Qt-1 percentage of parameters by identifying those with top-a,-1 percentage of gradient magnitude within each layer, and accordingly update the mask to m The intuition is that for an important parameter over the local data, its gradient magnitude should be larger than the unimportant ones, which should be included into the new subspace. Formally, the recovery process is formalized as follows:
m : + - (w.,,,)))
where TrgTopK ox-1 (w)) returns the 0x-1 percentage of largest coordinates of absoluted gradient and mask them to 1, indicating that they will be recovered. But we note here that we don't 't perform subspace recovery in the first round, since pruning is not done yet.
Aggregation. Once subspace training is done, the clients upload the gradient updates of the local subspace to the server for aggregation. To aggregate the knowledge into the global model, we average the gradient updates based on their coordinate- -ise contributions, follows:
M
1
Wi+11
we
m1,+1 O (w: w.,,KK
(5)
M
where w is the weight from the client's local subspace, and mi,++1 is the subspace after pruned.
Consensus fusion (CF). Given that those malicious parameters served to recognize backdoor triggers will be deemed unimportant for benign clients, they should not be contained in the subspace of benign clients, which accounts for the majority. This observation inspires us to eliminate the malicious parameters using consensus fusion after T rounds of global model training. Formally, the consensus fusion operator returns a vector that satisfies:
[1 E marl, 2
(T mM.ly =
(6)
Otherwise
where is the threshold for CF, and [:l; indexes the j-th coordinate of vector. By applying the global mask produced by CF into the global model, i.e., wr To(m,,,,......... M..T........... those parameters that have appearances smaller than 0 among all the subspaces are sparsified to 0. In this way, those poisoned parameters will mostly be eliminated, thereby reaching the goal of perturbing backdoor.
Experiment
6.1 Experiment Setup
Datasets and models. We experiment on FashionMnist. CIFARIO/CIFARI00 and Tiny datasets. For IFFARIO/CIFARIO,, we use a modified ResNet9 via adding a pooling layer after the first convolutional layer to keep the same hidden size before output. For FashionMnist, we use LeNet (LeCun et al. 1998).
use ResNet9 et al. 2016) backbone. For Tinylmagenet,
Attack methods. We classify the backdoor attack methods FL into three genres, i.e., data-level backdoor, algorithm-level backdoor and advanced backdoor. Data-level backdoor only allows the malicious clients to modify the raw data, but they have no control of the algorithm. On contrary, algorithm-level backdoor allows clients to modify the training algorithm, in addition to the raw data. The advanced attack can access extra server-side information (see Appendix A.2)). Indeed, the three genres of backdoor attack methods correspond to the weak, medium, strong threat models in Table

---

respectively. Among the data-level backdoor, we simulate three types of attacks methods, i.e., BadNet (Gu et al. 2017), DBA (Xie al., 2019). and Sinusoidal (Barni et al., 2019). For algorithm-level backdoor, we simulate two types of attack, i.e., Scaling (Bagdasaryan et al. 2020). and FixMask. For advanced backdoor, we test Neurotoxin (Zhang et al. 2022b) and an adaptive attack Omniscience.
Defense Baselines, We use vanilla FedAve (McMahan et al. 2016) (without defense) as a baseline. and compare Lockdown with four SOTA defenses RLR (Ozdayi al. 2021), Krum (Blanchard et 2017), RFA (Pillutla et al. 2022) and Trimmed mean (Yin et al. 2018).
Evaluation metrics. Three metrics are used evaluate the defense performance:
Benign Acc. Benign accuracy measures the Top-1 accuracy performance of model given the benign data inputs (without the presence of trigger).
ASR. Attack Success Ratio (ASR) measures the ratio of backdoor samples (with trigger) to be classified to the target label. The smaller this metric, the more resilient the model is.
Backdoor Acc. Backdoor accuracy measures the Top- accuracy of the model given the backdoor inputs (their labels during testing are the original one before adding backdoor trigger). We add this metric to evaluate the overall performance, since high backdoor acc means: i) the classification of benign features is well-performing. ii) the perturbation of backdoor trigger is limited.
Simulation setting. We simulate M = 40 clients, and data is either evenly distributed to each client (IID setting) or is distributed with Dirichlet distribution (Non-IID setting) following (Hsu al. 2019). The parameter for Dirichlet distribution is set to 0.5 for the Non-IID partition. To simulate the backdoor attack launched by the malicious clients, we follow (Ozdayi et al. (202T) to randomly choose N clients attackers whose p (percentage) of data in their local datasets are poisoned. The default backdoor settings for our main experiment are p = 50% and All the experiments are done with Nvidia A100 GPU.
4. We summarize the
default simulation setting in Table 2
Hyper-parameters. For Lockdown, we fix the overall sparsity to s 0.25, the mask agreement threshold to 0 = 20, and the initial pruning rate to ogg = 1e 4. The robust learning rate threshold for RLR set to The number of local epochs and batch size are fixed
and 64, respectively. The learning rate and weight decay used in the local optimizer are fixed to 0.1 and 10-4, The number of comm rounds fixed to 200. We summarize the default syyeer-parametet in Table
Table 2: Default Simulation Setting.
Table 3: Default ppprm-prreeetr for Lockdown.
Notation
Meaning
Default Value
Notation
Meaning
Default Value
Total number of clients
40
Ct
Initial pruning rate
lc-4
p
Poison ratio
0.5
0
Agreement threshold
20
N
Attacker number
4
Overall sparsity
0.25
6.2 Main Evaluation
In our main evaluation, we use CIFARIO as the default dataset and BadNet as the default attack. Convergence.
The convergence result w.r. communication rounds is plotted in Figure 4 Lockdown offers the strongest robustness under attack with Non-IID settings. Compared with IID, when the data distribution is Non-IID, Lockdown suffers a slight drop in benign accuracy (by approximately 3%) while Krum and RLR suffer more. The larger heterogeneity causes more benign parameters to be dropped in the consensus fusion process, resulting in drop in benign accuracy.
IID
Non- IID
Non-ID
100
100
sachyy Mara
Samun
20-
50
300
150
200
50
165
102
202
50
120
250
200
50
100
150
205
Communication Sourde
Commun catton Rounnds
Communiaatin Sounds
Communicttinn Rounds
Figure 4: Convergence and defense performance under different defenses.
7

---

Table 4: Defense efficacy with varying poison ratio under CIFARIO.
Mothods
Benign Ace (%)
ASR (9)
Backdoor Ace (18)
(ID)
cleun
p=.05
p=.2
p=.5
p=.8
clean
p=.05
p=.2
p=.5
p=.8
clean
p=.05
p=.2
p=.5
p=.
FodAAgg
91.0
91.4
91.1
91.0
90.8
1.6
12.4
19.9
66.1
94.8
88.5
79.6
73.4
32.9
5.1
RIR
86.8
86.7
86.6
86.3
85.5
2.3
2.4
2.4
4.3
25.1
84.6
84.3
83.4
81.7
65.2
Krum
76.3
78.0
75.6
76.4
75.8
4.7
3.9
4.3
4.3
4.9
73.8
74.9
72.9
73.9
73.2
RFA
90.9
91.2
91.1
90.8
90.7
1.6
15.8
20.7
83.7
99.3
88.8
76.8
72.4
15.9
0.7
Trimmed mean
91.0
90.6
91.1
90.9
90.8
1.7
5.0
20.7
61.7
96.2
88.5
84.7
72.0
36.6
3.6
Lockdown
90.0
90.0
89.9
90.1
90.0
1.8
3.6
2.5
7.1
4.0
87.9
85.8
86.6
83.7
85.6
Methods
Benign Ace (
ASR (%) 4
Backdoor Acc (%))
(Non-IID)
clean
pw.05
pe.2
pe.5
pe.8
clean
ps.05
pe.2
pe.5
pm.8
clean
p=.05
pe.2
pm.5
pe.8
FodAvg
89.0
89.2
89.3
88.8
88.7
1.7
17.3
54.4
86.4
96.7
85.9
74.0
42.5
13.0
3.2
RLR
74.4
74,4
73.6
72.9
72.5
5.8
15.0
40.2
29.5
82.5
69.0
63.1
46.2
51.4
15.3
Krum
37.4
45.2
43.4
45.1
5.2
10.4
10.6
38.6
33.8
40.7
RFA
88.8
88.8
88.8
88.3
88.3
2.0
21.4
52.8
90.8
98.7
85.7
70.6
44.3
8.9
1.2
Trimmed mean
88.2
88.3
88.3
25.2
48.4
85.4
67.5
Lockdown
85.6
86.2
86.7
86.1
86.6
0.9
7.6
3.6
3.4
3.3
84.1
79.5
82.3
82.2
82.8
Defense efficacy on varying poison ratio. As shown in Table the Attack Success Ratio (ASR) of Lockdown under different data poison ratios are significant lower compared with vanilla FedAvg without defense (up to 96% reduction), and is also significantly lower than the SOTA backdoor defense solutions. Though ASR of Lockdown is slightly larger than RLR and Krum in IID settings, its benign accuracy much higher than them (with up-to 5.4% and 15% enhancement comparing RLR and Krum respectively). Another observation is that Lockdown performs better in reducing ASR when the data poison ratio is high (in Non-IID setting, ASR is 7.6% when p = 0.05 while ASR is only 3.3 % when 0.8). This phenomenon is because the subspaces found by the malicious clients will deviate more from benign clients when a larger amount of backdoor samples are injected in their datasets. Therefore, the malicious subspace will overlap less with benign subspaces, resulting in better isolation, and also benefit the consensus fusion process. In addition, Lockdown significantly advances backdoor accuracy by up-to 79.6% compared to FedAvg without defense, which implics that the Lockdown's 's model can still recognize the true label of backdoor samples even under attack.
Defense efficacy on varying attacker ratio. We fix the poison ratio to 0.5 and vary the attackers ratio {0, 0.1, 0.2, 0.3,0.4), 0.4]. The results are shown in Figure As shown, Lockdown consistently achieves low ASR (at minimum 30% ASR reduction compared to FedAvg when attackers ratio is 0.4), and high benign accuracy in all groups of experiments (at maximum 5% drop of benign accuracy compared to FedAvg). In contrast, RLR and Krum are sensitive to attacker ratio and fail in defense when attacker ratio is 0.4 (no ASR reduction for Krum, and at maximum 10% reduction for RLR).
80
Non-lib
ID
Non-lID
100
100
Teiking Blun
Iran
Felling
Helten
Berly
Attecoara ratio (817))
Attuckers rat (M00
Absacorra ratio (811))
Atlackars raboo (400
Figure 5: Benign acc/ASR under different attackers ratio.
Defense against Non-IID extent. Next we show in Table 5 that Lockdown is robust to different Non-IID extent. Our experiment is done in the default setting, but vary different setting of Dirichlet Parameters a, and it shows that Lockdown can defend against backdoor even in extreme Non-IID cases (e.g., = 0.2, which leads to benign accruacy drop of FedAvg from 90.8% to 84.3%). Our another observation is that Lockdown would suffers more benign accuracy loss (respectively 3.7%, 2.7%, 1.2% and 0.7% benign accuracy loss for the four Non-IID levels).
Table 5: Defense efficiency with different Non-IIID extent.
0.2
0.5
0.8
IID
Methods
Benign Ace
ASR 1
Benign Acc
ASR
Benign Ace 1
ASR
Benign Acc
ASR
FodAvg
84.3
74,8
88.8
86.4
89.2
74.7
90.8
66.1
RLR
72.9
82.9
85.5
Krum
26.5
2.8
43.4
11.1
43.6
6.6
75.8
4.3
Lockdown
80.6
7.7
86.1
3.4
88
3.9
90.1
7.1
Defense against data-level backdoor. We simulate attacks with the BadNet, DBA, and sinusoidal
method. Our results in Table 6 show that Lockdown has good generalization ability to data-level
8

---

backdoor attack. Overall. Lockdown maintains superior defense efficacy (< 10% ASR) towards all the three data-level backdoor attacks.
Table 6: Lockdown on data-level attack.
Table 7: Lockdown on algorithm/advanced attack.
Methods (LID)
Benign 7 ?
ASR(%)
Backdoor Acc(1e)
RodNer
90.1
Motlads (ID)
Benign Accri0 7
ASR/9)
Backdoor S0
DBA
90.0
Sciling (Lackdiwss + NC)
89.9
3.2
56.2
Sinusoidal
89.5
4.6
83.6
Neuntisin
89.0
2.6
$6.1
Methods (Nee-TII)
Renige A
ASR (9))
Backddoee Acc
Methods Non-IID)
Benign
ASR (9))
Backdoo Acc(
BodNet
86.1
3.4
82.2
Smilnng (Lackdine NC)
3.2
DBA
$6.5
2.0
82.9
Neumioocin
Sinusoidal
$7.0
2,0
81.8
backdoor. Table shows Lockdown's 's performance on Scaling and Neurotoxin attack. For Scaling attack, the scaling factor is set to 5. We integrate Lockdown with the norm-clipping (NC) in aggregation as proposed in (Sun et al., 2019), and as shown, Lockdown defense is still effective by controlling ASR <5%. Lockdown is also robust to Neurotoxin, an attack developed from a similar idea as Lockdown (See Appendix A.2 for analysis).
Defense against algorithm-level/adaaaae
Defense against adaptive attack. We design two adaptive attacks that assume the knowledge of Lockdown and try to break it. Both the attacks try to disobey the mask searching process. Specifically, FixMask allows attackers to fix the initial subspace, and Omniscience is able to infer the global subspace that is produced by CF. As shown Table g vanilla Lockdown procedure is vulnerable to them. However, we find that there are two methods can be used to rectify Lockdown to accommodate Fix.askk attack. i) Enlarge the initial pruning/recovery ratio ag. or ii) adopt heterogeneous mask initialization (HM). Both the approaches enhance the dynamics of subspaces, and therefore enhance protection. For Omnisicence attack, the attackers always know consensus subspace and therefore can poison the parameters within it. Lockdown cannot cope with this attack. However, the condition of conducting Omniscience attack is stringent, as it has to acquire either global subspace, or other benign clients' local subspace, meaning that it falls into the Strong threat model in Table
Table 8:
Lockdown on adaptive attack.
Table 9: Communication and # of parameters for Lock- down under IID CIFAR1O 10 (ResNet9) setting. The com- munication overhead is the sum of that of M 10 clients in each round.
Methods (II)
Renign Aec(S))
ASR(S) Rackdeor Auz(8)
Fis
35.5
(oo
(1MMask-1IM..p 0.1)
Omniscience
52.1
Methods (No)-ID)
Besign Acc(S)
ASR (%)
Baskdoor Auc(3)
Methods
Comm Overhead
parums
Benign Accr
FinMask
36.8
86,3
13.1
FodAvg
2,10GB (Ix)
6.57M (Ix)
(1x)
Ficblak too 0.1)
$5.7
4.6
79.5
Lockdown
0,525GB (0.25k)
1613M (0.25k)
90.0 (0.990k)
FooMask nn 0.1)
$6.9
3.7
81.8
Omniscience
$6.3
13.1
Communication and model complexity. As shown Table we show that Lockdown achieves smaller communication overhead (0.25x compared to FedAvg), since only small subspace of the entire model gradient needs to be sent between server and clients. In addition, the number of parameters of the inference model is also lowered to 0.25x because the consensus fusion operation prune out the malicious/dummy parameters. Finally, we observe that benign accuracy only drops by 0.01x compared to FedAvg, indicating that pruning will not severely perturb the normal function.
Generalization varying datasets. We show our evaluation results on FashionMnist, CIFAR10/100 and Tinylmagenet in Table I0 As shown, Lockdown achieves SOTA defense efficacy (compared with FedAvg without defense, with up-to 80.7%, 83.0%, 73.8%, and 93, 93.4% reduction ASR respectively on the four datasets), and maintains reasonable loss of benign accuracy compared to FedAvg without defense (with up-to 2.2%, 2.7%, 3.9% and 3.3% drop).
Table 10: Performance on varying datasets.
FashionMaist
CIFARIO
CIFARI00
TmnyInagcnct
Methods (IID)
Benign Ace 1
Benign Ace
ASR
Benign
ASR
Benign Ase 1
FedAvg
901.0
85.1
91.0
66.1
70.0
75.7
12.7
96.7
RLR
89.2
6.0
86,3
4.3
61.0
1.9
10.5
98.0
Krum
83.9
1.0
76.4
4,3
26.9
98.8
3.5
99.9
Lockdown
90.1
Methods (Non-IID)
Benign Acc 1
ASR
Benign Ace
ASR J
Denign Acc 1
ASR 1
Benign Acc 1
ASR
FedAvg
89.2
87.9
88.8
86.4
67.7
34.8
96.5
RI.
83.7
39.1
72.9
29.5
53.9
36.0
9.2
98.6
Krum
75.8
1.7
43.4
11.1
20.6
1.3
266
99.9
Lnckdown
86.1

---

Visualization. In Figure We plot each filter's weight of the last convolutional layer of a ResNet-9 model trained with Lockdown. The brighter the color is, the larger the absolute value is, meaning that the filter can be activated by some particular inputs. We find that via our Lockdown procedure, the 86-th filter becomes "suspicious". We plot the weights in the classifier that connects this filter, and accidentally find that this filter contribute most to activate the "horse" neuron, which is our target backdoor label. This illustrates that lockdown can break the "poison-couple effect"" ppison parameters (i.e., those in the 86-th filter) only appear in subspace that is not shared by benign clients, which can be effectively filtered out by contrasting the benign and poisoned client's 's subspace.
Attacker (id=0)
Benign client (id=5)
Consensus (after CF)
Figure 6: Visualization of Lockdown. Each squre represent weights filter. Left/Middle/Right: projecting the weights into attacker/a benign client/consensus subspace. Bar chart in the middle: weights in the classifier connected 86-th filter with different neurons. "Horse" is our backdoor target.
6.3 Ablation and Hyper-parameter Sensitivity Analysis
Ablation study demonstrates the necessity i) ERK initialization ii) gradient-based recovery, is moved Appendix A.6 We tune three key hyper-parameterd of Lockdown to demonstrate their impacts in our sensitivity analysis. Our findings are: i) setting a proper sparsity s can increase the model's robustness. ii) Setting proper initial pruning/recovery rate ag is necessary for effective subspace searching. but too large of it will hurt the model's normal function. iii) Consensus threshold 0 should be set sufficiently large to filter out the malicious parameters. See Appendix A.7 for details.
7 Conclusion
In this paper, we study the gruning-based defense for FL and observe "poison-coupling phe- nomenon, which degrades the defense performance. To mitigate such an effect, we propose Lock- down, backdoor defense based on the idea of isolated subspace training. Empirical evidence shows that Lockdown can significantly reduce the risk of malicious backdoor attacks without sacrificing much on benign accuracy. Future works include studying how to generalize Lockdown to other FL. settings, e.g., decentralized FL (Hu et al., 2019 Li et al. 2023a Shi el al. 2023), personalized FL
(Fallah et al. 2020 T Dinh et al. 2020
Huang et al. 2022b 2023
Dai et al. 2022). etc.
Acknowledgements
This research is partially sponsored by the NSF CISE grants 2038029, 2302720, 2312758, an IBM faculty award, and grant from CISCO Edge Al program.
10

---

References
Alfarra, M., Pérez, J. C., Shulgin, E.,, Richtárik, P., and Ghanem, B. Certified robustness federated learning. arXiv preprint 2rXxv:2206.02335, 2022.
Bagdasaryan. E., Veit, A., Hua, Y., Estrin, D., and Shmatikov, V. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pp. 2938-2948 PMLR, 2020.
Barni, M., Kallas, K., and Tondi, B. A new backdoor attack in cnns by training set corruption without label poisoning. In 2019 IEEE International Conference on Image Processing (ICIP), pp- 101-105. IEEE, 2019.
Bhagoji, A. N., Chakraborty, S., Mittal, P., and Calo, S. Analyzing federated learning through an adversarial lens. In International Conference on Machine Learning, pp. 634-643. PMLR, 2019.
Bibíkar, S., Vikalo, H., Wang, Z.., and Chen, X. Federated dynamic sparse training: Computing less, communicating less, yet learning better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 6080-6088, 2022.
Blanchard, P., El Mhamdi, E. M., Guerraoui, R., and Stainer, J. Machine learning with adversaries: Byzantine tolerant gradient descent. Advances in Neural Information Processing Systems, 30, 2017.
Cao, X., Fang, M., Liu, J., and Gong, N. Z. Fitrust: Byzantine-robust federated learning via trust bootstrapping. arXiv preprint arXiv:2012. 13995, 2020.
Cao, X., Zhang. Z., Jia, J., and Gong, N. Z. Flcert: Provably secure federated learning against poisoning attacks. IEEE Transactions Information Forensics and Security, 17:3691-3705, 2022.
Chow, K.-H. and Liu, L. Perception poisoning attacks in federated learning. In 2021 Third IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA), pp. 146- 155. IEEE, 2021.
Chow, K.-H., Liu, L., Wei, W., Ilhan, F., and Wu, Y. Stdlens: Model rijacking-resilien federated learning for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2023.
Cohen, J., Rosenfeld, E., and Kolter, Z. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, pp. 1310-1322 PMLR, 2019.
Dai, R., Shen, L., He, F., Tian, X., and Tao, D. Dispfl: Towards communication--cffiicie personalized federated learning via decentralized sparse training. arXiv preprint arXiv:2206.00//87, 2022.
Evci, U., Gale, T., Menick, J., Castro, P. S., and Elsen, E. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-2952. PMLR, 2020.
Fallah, A., Mokhtari, A., and Ozdaglar, A. Personalized federated learning: A meta-leariin approach, 2020.
Ghosh, A., Hong, J., Yin, D., and Ramchandran, K. Robust federated learning in a heterogeneous environment. arXiv preprint arXiv:1906.0663: 06629, 2019.
Gu, T., Dolan-Gavitt, B., and Garg, S. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint
2017.
Guo, H., Wang, H., Song. T., Hua, Y., Lv, Z.., Jin, X., Xue, Z., Ma, R., and Guan, H. Siren: Byzantine- robust federated learning via proactive alarming. In Proceedings of the ACM Symposium on Cloud Computing, pp. 47-60, 2021.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Hsu, T.-M. H., Qi, H., and Brown, M. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv: 1009::99900033... 2019.
11

---

Hu, C., Jiang, J., and Wang. Z. Decentralized federated learning: A segmented gossip approach. 2019.
arXiv preprint
Huang, T. Lin, W. Wu, W., He, L., Li, K., and Zomaya, A. Y. An efficiency-boosting client selection scheme for federated learning with fairness guarantee. IEEE Transactions Parallel and Distributed Systems, 155):1555,1 2020.
Huang, T., Lin, W., Shen, L.., Li, K., and Zomaya, A. Y. Stochastic client selection for federated learning with volatile clients. IEEE Internet of Things Journal, 9(20:2000552200000 2022a.
Huang, T., Liu, S., Shen, L., He, F., Lin, W., and Tao, D. Achieving personalized federated learning with sparse local models. arXiv preprint arXiv:2201. 1380, 2022b.
Huang, T., Shen, L., Sun, Y., Lin, W. and Tao, D. Fusion of global and local knowledge for personalized federated learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URLhttps: /ttbps:://ppaaes //pp///vie/eeveeee///m net wttt:/ooron/..ddttrr///////
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pp. 5132-5143, PMLR, 2020.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 36(11):2278-233 1998.
Li, A., Sun, Li, P., Pu, Y., Li, H., and Chen, Hermes: an efficient federated learning framework for heterogeneous mobile clients. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking. pp. 420 437, 2021a.
Li, A., Sun, J., Zeng, X., Zhang, M., Li, H., and Chen, Y. Fedmask: Joint computation and communication-efiicee personalized federated learning via heterogeneous masking. In Proceed- ings of the 19th ACM Conference on Embedded Networked Sensor Systems, pp. 42-55, 2021b.
Li, Q. Shen, L.,, Li, G., Yin, Q., and Tao, D. Dfedadmm: Dual constraints controlled model inconsistency for decentralized federated learning. arXiv preprint arXiv:2308.08290 2023a.
Li, Q., Zhang, M., Yin, N., Yin, Q., and Shen, L. Asymmetrically decentralized federated learning. arXiv preprint 100.503,,, 2023b.
Li, T., Sahu, A. K., Zaheer, M.. Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. arXiv preprint arXiv:1812.06122 2018.
Liu, S., Chen, T., Chen, X., Atashgahi, Z., Yin, L., Kou, H., Shen, L., Pechenizkiy, M., Wang, Z., and Mocanu, D. C. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems, 34:9908-9992 2021a
Liu, S., Yin, I.., Mocanu, D. C., and Pechenizkiy. M. Do we actually need dense over- parameterization? in-time over wer-paraaeterizationn in sparse training. In Proceedings of the 39th International Conference Machine Learning, pp. 6989-7000, PMLR, 2021b.
Liu, S., Chen. T., Chen, X., Shen, L., Mocanu, D. C., Wang. Z., and Pechenizkiy, M. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. arXiv preprint ar Xriv:22002.026433 2022.
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., et al. Communication-effiieen learning of deep networks from decentralized data. arXiv preprint arXiv: 026602 2016.
Mo, F., Haddadi, H., Katevas, K., Marin. E., Perino, D., and Kourtellis, N. Ppfl: privacy preneccirreeviing federated learning with trusted execution environments. In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services, pp. 94-108, 2021.
Mocanu, D. C., Mocanu, E., Stone, P., Nguyen, P. H., Gibescu, M., and Liotta, A. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 91):2383, 2018.
12

---

Ozdayi, M. S., Kantarcioglu, M., and Gel. Y. R. Defending against backdoors in federated learning with robust learning rate. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 35, pp. 9268 9276, 2021.
Panda, A., Mahloujifar, S., Bhagoji, A. N., Chakraborty, S., and Mittal, P. Sparsefed: Mitigating model poisoning attacks in federated learning with sparsification. In International Conference on Artificial Intelligence and Statistics. pp. 7587-7624. PMLR, 2022.
Pillutla, K., Kakade, S. M., and Harchaoui, Z. Robust aggregation for federated learning. IEEE Transactions on Signal Processing, 70: 142-1154, 2022.
Qi. X., Xie, T., Pan, R., Zhu, J., Yang, Y., and Bu, K. Towards practical deployment-stage backdoor attack deep neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13347-13357, 2022.
Shah, D., Dube, P., Chakraborty, S., and Verma, A. Adversarial training in communication constrained federated learning. arXiv preprint arXiv:2103.013399 2021.
Shi, Y., Shen, L., Wei, K., Sun, Y., Yuan, B., Wang, X., and Tao, D. Improving the model consistency of decentralized federated learning. arXiv preprint arXiv:2302.04083, 2023.
Sifaou, H. and Li, G. Y. Robust federated learning via over-the-air computation. In 2022 IEEE 32nd International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1-6. IEEE, 2022.
Smilkov, D., Thorat, N., Kim, B., Viégas, F., and Wattenberg, M. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv: 000..70066.3353 2017.
Sun, Y., Shen, L., Huang, T., Ding, L., and Tao, D. Fedspeed: Larger local interval, less communica- tion round, and higher generalization accuracy. arXiv preprint arXiv:2302. 10429, 2023a.
Sun, Y., Shen, L., Sun, H., Ding, L., and Tao, D. Efficient federated learning via local adaptive amended optimizer with linear speedup. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023b.
Sun, Y., Shen, L., and Tao, D. Understanding how consistency works in federated learning via stage-wise relaxed initialization. arXiv preprint arXiv:2306.05700 05706, 2023c.
Sun, Z., Kairouz, P., Suresh, A. T., and McMahan, H. B. Can you really backdoor federated learning? arXiv preprint arXiv: 1911:: 1.07963, 2019.
Dinh, C., Tran, N., and Nguyen, J. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33:21394 2020.
Tolpegin, V., Truex, S., Gursoy, M. E.,, and Liu, L., Data poisoning attacks against federated learning systems. In European Symposium Research in Computer Security, pp. 480-501. Springer, 2020.
Wang, B., Yao, Y., Shan, S., Li, H., Viswanath, B., Zheng, H., and Zhao, B. Y. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 20/9 IEEE Symposium on Security and Privacy (SP), pp. 707-723. IEEE, 2019a.
Wang. H., Sreenivasan, K., Rajput, S., Vishwakarma. H., Agarwal, S., Sohn, J.-y., Lec, K., and Papailiopoulos, D. Attack of the tails: Yes, you really can backdoor federated learning, arXiv preprint wwxiv:2007.05000 2020.
Wang. S., Tuor, T., Salonidis, T., Leung, K. K., Makaya, C.,, He, T., and Chan, K. Adaptive federated learning in resource constrained edge computing systems. IEEE journal on selected areas in communications, 37(6)) 2019b.
Xic, C., Huang, K., Chen, P.-.,, and Li, B. Dba: Distributed backdoor attacks against federated learning. In International Conference Learning Representations, 2019.
Xie, C., Chen, M., Chen, P.-Y,, and Li, B. Cril: Certifiably robust federated learning against backdoor attacks. In International Conference on Machine Learning, pp. 11372-11388 PMLR, 2021.
13

---

Xu, X., Wang, Q., Li, H., Borisov, N., Gunter, C. A., and Li, B. Detecting ai trojans using meta neural analysis. In 2021 IEEE Symposium on Security and Privacy (SP). pp. 103-120. IEEE, 2021.
Yin, D., Chen, Y., Kannan, R., and Bartlett, Byzantine-robust distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning, pp. 5650-5659. PMLR, 2018.
Zhang. Z., Cao, X., Jia, J., and Gong, N. Z. Fldetector: Defending federated learning against model poisoning attacks via detecting malicious clients. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp.
2022a.
Zhang, Z., Panda, A., Song, L., Yang, Y., Mahoney, M., Mittal, Kannan, R., and Gonzalez, J. Neurotoxin: Durable backdoors in federated learning. In International Conference on Machine Learning, pp. 26429-26446. PMLR, 2022b.
Zhao, Y., Li, M., Lai, L.,, Suda, N., Civin, D., and Chandra, V. Federated learning with non-iid data. arXiv preprint arXiv: arXiv: /80600082, 2018.
Zheng, R., Tang, R., Li, J., and Liu, L. Data-free backdoor removal based channel lipschitzness. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part V. pp. 175-191. Springer, 2022.
Zizzo, G., Rawat, A., Sinn, M., and Buesser, B. Fat: Federated adversarial training. arXiv preprint arXiv:2012.01791, 2020.
14

---

A Supplementary Material
A.1 Implementation Details
Decay pruning rate with cosine annealing. In our subspace pruning/recovery process, we let the clients prune out at percentage of parameters and recover the same amount of parameters to search for the subspace that fits their data. The parameter Oe is decayed with the initial rate do with cosine annealing. which can be formalized follows:
04 0.5 og (1 cos
Tend
where t is the number of communication round, and Tend is the round that the mask searching is ended (notice that off..d = 0). In our implementation, we Tend== T.
ERK sparsity initialization. We use Rénós Kernel (ERK) (Evci et al. 2020), an empirical sparsity distribution technique, distribute sparsity different layers of model. Specifically, the
active parameters of the convolutional layer initialized by ERK are proportional to x + + +h nt -n +w +h!
where n111, w' and h! respectively specify the number of input channels, output channels and kernel's width and height in the (-th layer. For the linear layer, the number of active parameters scale with n'' +n' - where n and n' are the number of neurons in the (1 - 1)-th and I-th layer. ERK initialization, in essence, gives more sparsity those layers with larger number of parameters while pruning less on those small layers.
Subspace pruning. In the mask searching process, use parameter's magnitude to guide the pruning of model parameters. We present the PyTorch style code in Algorithm 2 to illustrate the pruning process and, correspondingly, the update of mask. Note that only prune parameters that are within the current subspace. Therefore, in line 6, we set the parameters that are out of the subspace very large value to prevent from selecting them. After that, filter out those parameters with the smallest a percentage of magnitude and prune them out of the subspace.
Algorithm 2 PyTorch style code for pruning and recovery
I: function Prune_rubspane subspace( (e W.,.KK, m,,++
2: Init layer sparsity {s:) given overall sparsity with ERK
3: m,,441 m.,..
for 1 0,
numprune = de x of params in the I-th layer
sort torch. where(m (1) = 1, torch.abs(w (..) 1000xtorch.ones_ [ike 7: iddr sorch.
8: m...11
(1)
viie(w numpproe]]]]]
end for
10: Return m,,+1
11:
end function
12:
13: function Recover subspace( dee, wi,,0 mi,e)
Derive gradient (ww, O m.,)) with one pass local data
15:
++ = m..
16:
do 17: numprune = x # of params in the I-th layer
for 1 0.1.....
18: sort = torch. (here(m)) i, i =0, torch.abs( { (m))))m..)) -1000) -1000xxtrrc.onee__lkeee...
k,t.k)
idx torch. (orch.) (ew(:ssrrt.vwdeessee) descending=True)
20: m (2) view(w [ide]: numprunn]]]
Le+
21:
end for
22:
Return m,,+
23:
end function
Subspace recovery. After pruning and before the next round training, recover the same amount of parameters to explore other parameters outside the subspace. Following (Evei et al.) 2020), we use gradient information of the pruned model to guide the recovery process. Here only recover the
15

---

Client #1
Client #2
Client #3
Client #4
Testing
DBA
Sinusoidal
Figure 7: Examples of BadNet. DBA, and Sinusoidal attack. Labels of poison samples are manipu- lated to the target label (e.g., horse).
parameters out of the current subspace, and therefore we the sorl_vdl parlue of the parameters within the current subspace to a sufficiently small value, as shown in Algorithm Subsequently, we sort in descending to obtain the parameters with the largest-or percentage gradient magnitude, and recover them by updating masks.
A.2 Attack Methods
Table 11: Application of attack methods in threat models. "v" corresponds to be applicable while" "M x" corresponds to be not applicable.
Attack methods
Threat models
weak
medium
strong
BadNet DBA
Sinusoidal
/
Scaling
V
FixMask (adaptive)
X
/
V
Neurotoxin
X
Omniscience (adaptive)
X
X
As we mention in the main body, we classify the attack model into data-level attack and algorithm- level backdoor. We in the following give brief description of each data-level backdoor that we simulate federated learning setting.
BadNet. BadNet is the earliest, and also the simplest backdoor attack first proposed in (Gu et al., 2017). To perform BadNet attack, the malicious client simply add the same backdoor trigger on some of the data samples, and modify the label of these poisoned samples to the target label. In test time, the malicious clients can place the backdoor trigger on the test samples, such that the victim model can produce the target output no matter what the original test samples are.
DBA. DBA (Xie et al., 2019) is a backdoor attack specifically targeted on FI.. To perform DBA attack, the authors decompose the backdoor trigger into several local pattern, and assign the local pattern to different clients to poison their local data. For test time, the attacker will interpose the completed trigger on top of the test samples they want to manipulate. is suggested by the authors that DBA is substantially more persistent and stealthy against FL. In our simulation, we decompose the "plus" trigger into 4 local patterns, and let each malicious client to be assigned each local pattern.
Sinusoidal attack. Sinusoidal attack (Barni et al. 2019) shares a similar perspective with BadNet, which also utilize the same trigger for all the malicious clients to poison their samples. However, the backdoor trigger they use is a horizontal sinusoidal signal defined by vii,j) A sin /m). S j < m, S i < I, for a certain frequency f. The authors claim that this design of trigger i) ensures the stealthiness of the attack, but also ii) can be separable with the same (or similar) feature space used by the network to classify the benign samples. In our simulation, we adopt the default tyyper-prramete å 20 and
for performing this attack.
16

---

Examples of these data-level attacks are visually shown in Figure 7
In the following we give brief description the algorithm-Ievel backdoor that has been simulated in this paper.
Scaling. The basic idea of Scaling (Bagdasaryan et al. 2020) is to enlarge the gradient update when malicious client return its update to server. This mechanism allows the malicious client to enlarge its gradient's impact on the global model, and therefore is effective when the poison ratio and attacker number are small.
FixMask. Fix is an adaptive attack method specifically targeting Lockdown. In Lockdown, the malicious clients are assumed to faithfully search for
their subspace using their local data. For FixMask attack, the malicious clients freeze their mask to be the initial mask that is shared by all the clients in round 0, and refuse to change afterwards.
Particularly, we want to emphasize that the data-level and algorithm-level backdoor can potentially be combined together to produce better attack performance. However, since this paper focus on the defense aspect, we leave a more thorough study of the attack model future work. We also include two advanced attack algorithms that can only be conducted given extra server information in addition to permission on manipulation of the attacker's own training process and data.
Neurotoxin. Neurotoxin proposed in (Zhang al. 2022b) explores a durable attack method in the scenario that the attackers can only participate limited rounds. Their main observations in the limited participation case are that i) the benign update can recover the global model after attacker ceases attack. ii) the majority of the 12 norm of the aggregated benign update is contained in a small number of coordinates (Let's call these benign coordinates). Utilizing the above observations, the authors propose Neurotoxin, which is to let the malicious clients project their gradient update to the subspace excluding the global coordinates. By this means, the projected updates from the malicious clients are mostly embedded to the coordinates that have less perturbation by the benign updates (which focus on the benign coordinates) after ceasing attack. However, Neurotoxin cannot escape Lockdown defense in principle. There are mainly two reasons. i) Lockdown only broadcast to the clients some coordinates weights (equivalently, some coordinates of gradient update) as per their subspace. Therefore, Neurotoxin cannot obtain the top-k coordinate of the server gradient as benign coordinates. ii) Lockdown requires clients to report the subspace that they want to update, and the subspace that are substantially different from others will be pruned afterwards. In other words, if the attackers adopt neurotoxin to choose the subspace that excludes the benign coordinates, their subspace can be easily identified by comparing with other benign client's subspace, and therefore will be pruned out. In our simulation, we assume Neurotoxin can acquire the server gradient update by some means. Therefore, it is classified simulation, we set its tlper-paramette mask ratio to be 0.25.
attack method for strong threat model. In our
Omniscience. This is an adaptive attack that assumes the knowledge of Lockdown and try break it. The main idea is to assume the client's has knowledge of the consensus subspace after going through consensus fusion, and project their gradient update into this subspace. This efficiently avoids the malicious weights to be pruned out by the consensus fusion operation. However, the requirement of conducting this attack is very stringent. The malicious client needs to have knowledge of the consensus subspace, which either is leaked from server, or is computed if other clients' subspace is known by the attacker. Neither of this condition is easy to establish for an attacker in a federated learning system.
In summary, we show in Table 11 the attack methods we can perform with specific threat models.
A.3 Defense Methods
In this section, we give brief description of the defense baseline we compare against.
RLR. RLR proposed in (Ozdayi et al. 2021) utilizes coordinate- wise server learning rate to inverse the gradient coordinates in which different clients have different sign. Their observation is that the malicious coordinates tends to be those coordinates that have conflicting sign in gradient while for the benign coordinates that are not poisoned, most of the clients will agree with their sign. Therefore, by looking at the gradient update from clients, the server is able to identify the malicious coordinates and subsequently inverse its sign the aggregation phase. However, the malicious clients are able to launch adaptive attack if he knows the gradient update downloaded from server.
17

---

RFA. Aiming at defense against corrupted updates from clients, RFA (Pillutla et al. 2022) utilizes the concept of geometric medium aggregate the gradient update from clients. Geometric medium avoids the gradient that has excessively large norm (usually is the malicious one) to impact too much on the averaging process. Specifically, when doing aggregation, instead of directly averaging the uploaded gradient, the server aims to obtain global model that minimizes: > th ||e Wt
and w, is the uploaded local model. This problem is solved by the Smoothed Weiszfeld Algorithm.
Similar techniques are studied in (Sifaou& & Li 2022). (Ghosh et al. (211)) and (Cao al. 2020).
Krum. Targeting Byzantine attack, Krum Blanchard et al. 2017) adopts the idea of finding the gradient update that is closest to its n -f neighbours such that can ensure (a, f)- Byzantine resilience where is the angle depends on the ratio of the deviation over the gradient .f is the number of attackers. Specifically, Krum aims to find the the i" client that minimize *()) Σinj VV; Vill where i -> j denotes the set of i'th client's n - f 2 closest neighbours, and V; denotes the gradient update from client After identifying i", Krum returns Vi- as the robust gradient used for aggregation.
Trimmed mean. Trimmend mean is proposed in (in et al. 2018) to counter byzantine failures in the distributed machine learning scenario. Their high level idea exclude the outlier gradient value when doing aggregation. Specifically, before aggregation, the server coordinate -wise trims the TopK gradient and the bottomK gradient among those uploaded gradient. After trimming. the server assume the outlier has been trimmed. and directly average the clean gradient. In our simulation, we set the trim ratio to be 0.1.
A.4 Security Analysis
We make the following observations on Lockdown's security performance. Lockdown can success- fully defend all the data-level attack, i.e., the attack falls in the scope of weak threat models. For the algorithm-level attacks, we have incorporated an adaptive attack targeting on Lockdown and gradient scaling method into study. Our results show that Lockdown can also defend all the attacks we have tested. However, since the algorithm-leve attacks are more adaptive, we cannot make guarantee attacks, especially those that are specially
that Lockdown unbreakable by any algorithm- designed for Lockdown. For advanced attack that allows attacker to acquire server's information, we create another adaptive attack Omniscience that can successfully circumvent Lockdown's defense. Performing Omniscience attack needs the attacker to know about the consensus subspace. However, it challenging. if not impossible, for the attacker to infer the consensus subspace, since only a subset of the server gradient update is distributed to clients, further constraining the global information access of the attackers
A.5 More Visualization
Input-level visualization. In Figure 8 we add additional experiments to visualize the gradient w.r.t the input of the first layer, which visually explains how different semantic information within the input image contributes to activating the target output neuron.
Backdoor Input
clean model
FedAvg (no detense)
Lockdown (before CF)
Lockdown (after CF)
Figure 8: Smooth grad (Smilkov et al. 2017) visualization of models given backdoor input. The first column is data input with backdoor trigger. The subsequent columns demonstrate the gradient with respect to the input of i) a model without being poisoned. ii) a model trained by FedAvg with poisoned data, iii) Lockdown's global model under poisoning before going through consensus fusion (CF) and iv) Lockdown's 's final model. A clean model emphasize the correct semantic within the input, e.g., wing of a plane, while a poisoned model emphasizes the yellow "plus" backdoor trigger.
18

---

Parameters-level visualization. In Figure 9 we visualize the projected parameters produced by Lockdown. The experiment conducted on MNIST with a two-layer MLP model. After reducing its output dimension and reshaping it into the original input, we plot the projected absolute weights of the first layer of MLP. As found, by projecting the global weights into malicious client's subspace (left), the corresponding connectivity that joint the backdoor trigger still present. However, by projecting the global weights into one the benign client's subspace (middle), the backdoor trigger no longer connects with large absolute weights. The same phenomenon is observed for the consensus subspace after going through consensus fusion (right).
Cient #0 (mallcious)
Client #12 (benign)
Global after
Figure 9: Visualization absolute global weights after projecting into the local or global subspace. Left: projecting into local subspace of malicious client. Middle: projecting into local subspace of benign client. Right: projecting into consensus subspace produced by consensus fusion. The brighter the color is, the feature locates in that part is more important. The bright backdoor trigger "+" is not visible in the middle and right image. See more details in the main text.
A.6 Ablation Study
We perform ablation study of Lockdown on CIFARIO. BadNet is the default attack method.
Gradient-based recovery vs. random recovery. In subspace recovery process, use gradient magnitude to guide the recovery of parameters. In Table 12 we show the empirical comparison between the gradient-based recovery and random recovery. The results showcase that recovery with the gradient can significantly reduce the ASR (by up-to 78.3% reduction) though the benign acc of the model suffered a little bit (by up-to 2.3% drop). This is because gradient magnitude tends to guide the subspace searching process to acquire heterogeneous subspaces for clients with different training data. With more heterogeneous subspaces, the knowledge transferring between clients will be deterred since their the subspace overlap is small, which leads to the degradation of benign acc. On the other hand, small subspace overlap can also facilitate the process of de-poisoning by consensus fusion, which leads to a reduction of ASR.
Table 12: Ablation study for parameters recovery implementation.
Methods (IID)
Benign A
ASR(%)
Backdoor Acc( f
Random recovery
13.0
79.7
Recovery w/ gradient (ours)
90.7
1.4
87.8
Methods (Non-IID)
Benign Acc(%) f ASR(%) |
Backdoor Acc($6) f
Random recovery
88.9
17.2
74.3
Recovery w/ gradient (ours)
84.9
2.2
78.4
ERK initialization vs, uniform initialization. In the SubspaccInit() function, we use ERK to allocate the sparsity of each layer in model. To justify the necessity of ERK initialization, replace the ERK initialization with uniform initialization, which uniformly allocates sparsity to each layer. As shown in Table I3 uniform initialization will largely compromise the benign accuracy and slightly increase the ASR. This justifies that the sparsity should be set larger for the layer with a larger number of parameters (which essentially is what ERK does).
Consensus fusion (CF). In Figure 10 we demonstrate the necessity of consensus fusion under different poison ratios. With consensus fusion, benign accuracy is significantly increased by up-to 60% while the ASR is reduced by up-to 80%. This result shows that masking out some malicious/dummy parameters perturb the backdoor function and thereby curing the poisoned model.
19

---

Table 13: Ablation study for sparsity initialization.
Methods (IID)
Benign Acc
ASR +
Backdoor Ace t
Uniform
81.7
5.7
75.9
ERK (ours)
90.1
7.1
83.7
Methods (Non-IID)
Benign Acc
ASR +
Backdoor Ace f
Uniform
75.5
3.1
70.6
ERK (ours)
86.1
3.4
82.2
10
Non HD
1D
Non IID
100
00
2013
60
70-
20
Toaon Ratto
Ratto
Pokson Ratio
Figure 10: Impact of consensus fusion of Lockdown.
A.7 Hyper-parameter Sensitivity Analysis
In this section, we perform hyper-paramen sensitivity analysis for lockdown. The evaluation is conducted on CIFAR 10 under the default simulation setting in Table 2
unless otherwise specified.
Sparsity s. In Table 14 set other hyper-parameter levels. As shown, Lockdown loses its defense efficacy when sparsity is low. This phenomenon is understandable since Lockdown reduces to FedAvg when sparsity is O. On the other hand, with larger sparsity, the benign accuracy of the model suffers due to the reduction of trainable parameters. Therefore, there exists tradeoff for the sparsity of Lockdown. Larger sparsity promises lower model complexity, smaller comm overhead, and also lower ASR, but at the cost of losing benign accuracy.
default and tune the sparsity to different
Table 14: Performance of Lockdown under different sparsity 8.
(IID)
Benign Ace f
ASR
# of params
91.0
68.4
6.57M
0.2
90.9
61.1
5.26M
0.5
91.0
10.9
3.29M
0.75
90.1
7.1
1.65M
0.9
88.3
3.0
0.66M
(Non-IID)
Benign Acc t
ASR
# of params
0
89.1
70.3
6.57M
0.2
88.4
52.6
5.26M
0.5
87.1
3.29M
0.75
86.1
3.4
1.65M
0.9
85.0
2.9
0.66M
Initial praning/recovery rate. We also show the effect of initial pruning/recovery for the learning performance. As shown, larger pruning rate would typically results in the drop of benign accuracy but also enhance the ASR under poisoning attack. Specially, when as O, lockdown reduces to train a sparse subnetowrk from scratch, without evolving the sparse coordinate. This setting cannot climinate the "poison-couple effect, therefore the ASR is high FedAvg with no defense. On the other hand, setting Og will also result isolation of subspace for different clients, resulting in lack of consensus in the global space and therefore leading drop of benign accuracy.
In Figure I1 we tune the CF threshold different settings of attacker number N. In all settings of N, we see that: i) 0 should not be set to be too small; otherwise, the benign accuracy would be lower, and the ASR will be higher. also should not be set too large; otherwise, it will severely compromise benign accuracy, but the reduction
Consensus fusion threshold
see impact on
20

---

Table 15: Performance Lockdown under different initial pruning/recovery rate ap-
do (IID)
Benign Acc f
ASR +
Backdoor Ace
0
90.5
49.4
47.7
le-5
90.5
5.2
84.3
lc-4
90.1
7.1
83.7
le-3
88.1
3.7
84.7
le-2
87.2
3.5
83.7
le-1
87.0
3.1
83.4
(Non-IID)
Benign Acc f
ASR |
Backdoor Acc
0
88.5
85.3
14.0
le-5
87.4
8.5
78.8
le-4
86.1
3.4
82.2
le-3
84.9
2.1
80.4
le-2
83,4
5.3
76.4
le-1
83.7
5.2
77.6
of ASR will too significant. Per our results, the consensus threshold should chosen carefully according to the number of attackers, which of course, is unknown in most cases. However, given that the attackers within the system should not take up a large portion, set to 50% of the total number of clients will be sufficient to counteract the effect of backdoor attack in general attack scenario.
HD
Non IID
1D
Non IID
Nidd
Derigg Feciney
0:6-
0.37
0.84
0.6
N-4d
Nord
0:85
Nos
N-10
0.52
Nos
N-10
20
20
Agres maat Threshold (2)
Agreementt Threshcidd I2:
Agrsement Thrashol 101
Agreament Threahold (0)
Figure 11: Impact of consensus fusion threshold of Lockdown in different # of attackers setting.
A.8 Limitations
Our method utilizes sparsity of model to counter backdoor attack. However, we are aware that sparsity in its current stage can hardly guarantee acceleration of the training/inference speed. At present, the current sparse acceleration technique requires 2:4 sparse operation. More specifically, the 2:4 sparse operation requires that there may not hold for the sparse model produced by Lockdown. But we insist that our method has great potential achieve truly training acceleration with development of sparse technique.
most two non-zero values in four contiguous memory, which
There are potentially other adaptive backdoor attacks that can break the defense lockdown, especially under the assumption that attackers have full control over its local training process and has knowledge the defense. We leave the research potential attacks against Lockdown as future works.
A.9 Broader Impact
The poison-couplin effect we discover in this paper might be mis-used to guide the design of backdoor attack method in centralized learning/FL. scenario. We will continue this line of research and further propose attack/defense method to better study/mittggae such an effect. We also open- source our code to facilitate researchersmmachine learning engineer in academy/industry to study and understand the discovered phenomenon.
21


Reviews:
[{"review_id": "ssXlVGgbH7", "reviewer": "Reviewer_ozxV", "summary": "The paper addresses the vulnerability of federated learning (FL) to backdoor attacks and the limitations of existing defense solutions in resource-constrained scenarios. It introduces \"Lockdown\", an isolated subspace training method to counter the poison-coupling effect present in traditional pruning-based defenses for FL. Lockdown modifies the training protocol by isolating subspaces for different clients, uses randomness in initializing isolated subspaces, and employs subspace pruning and recovery to segregate subspaces between malicious and benign clients. \nAdditionally, quorum consensus is introduced to purge malicious/dummy parameters from the global model. \nEmpirical results demonstrate that Lockdown outperforms existing methods in defending against backdoor attacks, while also providing communication efficiency and reducing model complexity, making it suitable for resource-constrained FL scenarios.", "questions": "1. After reading Section 4, I still do not understand what the 'poison-coupling' means. How are the 3 observations in Figure2 connected with (or lead to) the conclusion that 'backdoor parameters tends to be coupled with the benign parameters'? Also what does 'L2 norm of last convolutional layer' imply?\n\n2. Could you elaborate why 'subspace recovery' is necessary? Why not only use 'subspace pruning' to keep important parameters?\n\n3. How does the proposed method improve communication efficiency? What is the key reason? Can the standard model pruning also work?\n\n4. For Table 8, does the communication improvement has something to do with the number of clients?", "limitations": "See above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Propose a novel defense for FL backdoor attacks. It addresses the limitations of existing defenses in resource-constrained scenarios.\n\n2. Strong and thorough evaluation. The ablation study is also comprehensive.", "weaknesses": "I think the paper writing has much room to improve. For example, in Section 4, the paper had better briefly introduce the idea behind channel lipschitzness. Also, in Section 5, although the paper follows a top-down style, it is still full of technical details, without much intuitive explanation or design motivation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review comments. Below we try to address the reviewer's concern.\n\n# About the poison-coupling effect (Q1)\n**(Poison coupling effect)** We define the poison-coupling effect based on the empirical observation that the parameters used for poisoning by a small percentage of compromised clients in federated learning are relatively harder to identify with high confidence because  the poisoned parameters and the benign parameters are statistically coupled. \n\n**(CLP pruning)** To demonstrate the poison-coupling effect, we measure the effectiveness of the CLP pruning method on both the federated model and the model trained in centralized setting respectively.   CLP's pruning criterion is the estimated Lipscheness constant of each channel. The channel with a larger estimated Lipscheness constant will be pruned because they are more likely to be the poisoned channels.  We have rewritten the CLP pruning in Section 3 to improve the readability.\n\n**(Why 3 observations in Fig. 2 support the poisoning coupling)** The left of Figure 2 shows that if we use CLP to prune the federated model, we will need a larger pruning ratio to safely remove the poisoned parameters and purify the models. This is the first observation, telling that the federated poisoned model is relatively harder to be purified. The middle/right of Figure 2 respectively shows the estimated Lipschitzness constant and the L2 norm of the two poisoned models. We see that the federated model has a more uniform statistical distribution in the channel level in both cases. These two cases (the 2nd/3rd observations) further explain that the poisoned/benign parameters (or channels) are coupled together in a statistical way, and therefore is harder to be identified, i.e., it demonstrates the poison-coupling effect. \n\n**(Implication of the L2 norm of the last convolutional layer)** The L2 norm of the last convolutional layer means the L2 norm of the model parameters within each channel of the last convolutional layer of a model. We only report the statistic data within the last convolutional layer of the model, because the L2 norm exhibits a large discrepancy over different layers.\n\n\n# Necessity of subspace recovery (Q2)\nWe introduce the subspace recovery as a necessary subsequent step after subspace pruning because the pure pruning would continuously shrink the federated model round by round, which may lead to numerous errors and/or implementation issues. For example,  when the FL model is pruned during the progress of the training round by round, the pruning ratio will become much harder to configure and tune. This is because the federated model may collapse when the sparsity of the model is larger than a certain threshold. To ensure the robustness of the lockdown defense, we introduce the subspace recovery, which will coordinate the pruning/recovery procedure, making the FL model maintain the same sparsity over the rounds of the FL learning/training process.\n\n# About communication efficiency (Q3)\nThe lockdown powered FL training procedure asks every client to train the local model using a subspace structure, and consequently, in each FL round, each of the participating clients only transmits a subset of the full model parameters, i.e., those within each client's own subspace. This can substantially reduce the size of the model parameters shared with the FL server and hence enhance the communication efficiency of federated learning in every round.  In comparison, the standard pruning techniques usually prune the model after it is being trained (like CLP pruning)  when deploying the trained model on edge devices. However, they fail to reduce communication overhead over the iterative federated model training process. \n\n#  Table 8 Communication improvement (Q4)\nIn a lockdown powered FL system, the communication improvement is independent of the number of clients. In each round of federated learning, every participating client will gain the communication efficiency under lockdown protection. \n\nIn Table  8, we measure the sum of the communication overhead for every client in the FL system. Therefore,  this statistic is related to number of clients. \n\n# Section 5 Intuitive explanation and design motivation (see Weakness section)\nWe have updated Section 5 by adding additional illustration on the high-level idea of lockdown solution. The core ideas include: \n\n* **(Subspace searching)**  Each client will only involve the parameters they deem important in their subspace following the dynamic way of subspace searching.\n\n* **(Consensus fusion)** By introducing consensus fusion, those parameters that appear less among the subspace will be pruned out. This also indicates that the parameters considered not important by most clients will be pruned out. Given that a majority of the clients is assumed to be benign, the poisoned parameters will not be considered as important ones by the benign clients, and therefore they will be pruned out by consensus fusion.\n\nWe are grateful to the reviewer for the constructive and helpful comments. We sincerely hope that we have addressed your comments and would appreciate the reflection of your satisfaction on the overall rating. Thank you.\n\n\n "}, {"review_id": "xrlV843P4T", "reviewer": "Reviewer_dbHt", "summary": "Federated learning (FL) is a promising approach for privacy-preserving ML applications. However, it is also vulnerable to backdoor attacks. Although some pruning-based methods have been proposed to defend against backdoor attacks, the authors note that it is difficult to prune malicious channels (via Lipschitzness) or weights (via l2 norm) in the federated backdoor setting. The authors named this phenomenon the poison-coupling effect. To address this challenge, the authors propose to isolate the training subspace of individual clients to prevent benign clients from applying malicious clients' parameter updates. The experiment demonstrates that the proposed method can effectively defend against backdoor attacks.", "questions": "1. My main concern is that Lockdown's performance is entirely dependent on the malicious client strictly following the training protocol. An attacker can easily bypass this defense using two datasets. Specifically, it could first use clean dataset to generate the mask and then performs sparse training on the poisoned training set. I suggest that the defense algorithm should occur mainly on the server, not on the client side.\n2. Subspace searching should be the key contribution to solving the poison-coupling effect. However,  the size of the subspace for all clients are same? and the pruning ratio $\\alpha_t$ in every round $t$ should be same for all clients? I think the authors should discuss some complicated cases such as Non-IID datasets or unbalanced datasets or multi-task learning etc.\n3.  I'd like to start by pointing out that using sparsification methods to defend against attacks and reduce communication overhead is a common paradigm, with similar methods, including Hermes[1] FedMask[2] and SparseFed[3]. However, the authors do not discuss Lockdown's relationship with them. I also note that the proposed method tends to perform worse in terms of benign accuracy. It is not a significant drawback since Lockdown uses only a quarter of the communication overhead compared to FedAvg. Still, I suggest the authors compare the proposed method with other gradient sparsification methods to show that Lockdown can exhibit higher accuracy and robustness for the same amount of communication overhead.\n[1] Li, A., Sun, J., Li, P., Pu, Y., Li, H., & Chen, Y. (2021). Hermes: An Efficient Federated Learning Framework for Heterogeneous Mobile Clients. Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, 420–437. https://doi.org/10.1145/3447993.3483278\n[2] Li, A., Sun, J., Zeng, X., Zhang, M., Li, H., & Chen, Y. (2021). FedMask: Joint Computation and Communication-Efficient Personalized Federated Learning via Heterogeneous Masking. Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, 42–55. https://doi.org/10.1145/3485730.3485929\n[3] Panda, A., Mahloujifar, S., Bhagoji, A. N., Chakraborty, S., & Mittal, P. (2022). Sparsefed: Mitigating model poisoning attacks in federated learning with sparsification. International Conference on Artificial Intelligence and Statistics, 7587–7624.\n", "limitations": "see the above", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The poison-coupling effect is an interesting phenomenon and valued problem to be resolved.\n2. The idea of defending backdoor in FL with isolated subspace training is interesting and make sense to the problem, and this methods can also reduce the communication cost while defending backdoor attacks.\n3. The experiments are comprehensive and seems to be convinced.\n", "weaknesses": "1. This approach relies entirely on the malicious client strictly adhering to the training protocol. That is a strict assumption and hard to be satisfied in the real applications.\n2. This approach also relies on some global setting such as count(1{m_{i,t}}) which is the size of the subspace for client i, and the fixed pruning ratio $\\alpha_t$ for all clients, which is suitable for all clients with non-iid data set or unbalanced dataset? Those problems is not discussed in this paper.\n3. Using gradient sparsification to defend against federal backdoors is a common idea, and the authors do not discuss related work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank this reviewer for the constructive, encouraging, and helpful comments. We below respond to the three weaknesses:\n\n# W1 (Strict assmumption on Lockdown protocal)\n*The answer is NO.* Lockdown defense does not assume that malicious clients have to adhere to the lockdown training protocol. Concretely, we have evaluated the effectiveness of lockdown under two adaptive adversaries (see line 308 page 8): Omniscience and FixMask. Both allow malicious clients to violate the protocol. For FixMask attack, the malicious clients maintain the same initialize masks and refuse to change in the later round(s). Our experiments show that Lockdown remains robust when adopting a proper pruning/recovery ratio. For Omniscience attack, the malicious clients are assumed to have the full knowledge of the consensus subspace, and consequently adapt/adjust their subspace to the consensus. However, to launch a successful Omnoscience attack, the attackers need to obtain (or guess) the accurate consensus subspace, which is hard. It requires to compromise and collude with the FL server. Under such a whitebox Omniscience attack, Lockdown is not robust against poisoning. However, most existing representative poisoning defense methods assume that FL-server is trusted and not colluding with any clients. Due to this reason, we insist that Omniscience attack does not pose severe risk to Lockdown.   \n\n#  W2 (Client-level sparsity/pruning ratio)\nBoth the size of the subspace for client i and the pruning ratio are the global parameters for all clients under both iid and non-iid setting. \nIt is important to note that our lockdown method with these global parameters are easy to implement and yet highly effective, especially under the Non-IID data distribution, which is a representative scenario for federated learning systems with a large population of heterogeneous clients. Hence, in our current Lockdown design, we did not need to consider personalized client-level sparsity of subspace.  One of our ongoing efforts will exam whether the per-client sparsity will offer some value added poisoning defense utility even though it may add more configuration and computation complexity. \n\n# W3 (Relation with previous work)\nWe thank this reviewer for 3 related references. Will add them in the related work. \n\n**([1] [2] (sparse training)** both aim to design a communication/computation efficient FL solution. In Hermes, each client employs DNN pruning to learn a personalized sparse DNN and only communicate the updates of the subnetworks with the server. In FedMask, each client will learn a personalized sparse binary mask and share this binary mask with the server while keeping local model unchanged. Instead of learning a shared global model, each client obtains a learned (personalized) binary mask. \n\nUnlike  [1] [2] which optimize per-client DNN pruning solely for communication/computation efficiency, Lockdown designs subspace pruning/recovery/fusion for canceling the poisoning effect at FL server while maintaining the benign local model update quality and the global model accuracy. In this sense, Lockdown differs from the two works in that their design goals are substantially different.\n\n**([3] Gradient sparsification)** utilizes gradient sparsification to defend against data poisoning in FL. Unlike sparse training, gradient sparsification compresses the gradient before sharing with the server. [3] combines global model compression and device-level gradient clipping to defend model poisoning and rely on carefully tuning server side compression ratio and client level clipping. Unlike [3] centering on label flipping attack using gradient compression/clipping, Lockdown is defending against trigger-based backdoor poisoning with the novel and poisoning robust sparse-training techniques. \n\nWe thank this reviewer for the helpful comments and useful references. \n "}, {"review_id": "fyPYDbsjxY", "reviewer": "Reviewer_ftvs", "summary": "The paper proposes a backdoor defense. The method is based on robustly estimating a sparse parameter subspace which is used to restrict updates. Each client will vote for a set of parameters it considers \"important\" to update, and non-important neurons will be frozen for that iteration. The paper shows that this defense works well empirically and provides an ablation study showing that all of their algorithm's ingredients are necessary.\n", "questions": "What security can be achieved by the method?\n", "limitations": "The paper should discuss its limitation on the theoretical side.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of using sparse training in the context of backdoor defense is interesting. The experiments demonstrate that the method is very effective compared with baseline methods. \n\nIts ablation study also suggests the necessity of individual design choices.\n\nIt also evaluates adaptive attacks.\n", "weaknesses": "The paper does not discuss the defense from theoretical aspects. Thus, it is unclear what level of security guarantee can be achieved. Namely, the defense may be only effective because of the current status of research or our knowledge and can be potentially vulnerable to future attacks. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the very encouraging review and helpful comments. We have provided some security analysis in the supplementary material (see Appendix A.4). A formal verification of the security guarantee is an important research result by itself. It is on our future research agenda. Especially, it is interesting and important to develop the theoretical robustness bound for both the subspace size and the consensus quorum. \n"}, {"review_id": "jy0xWYy8XM", "reviewer": "Reviewer_t853", "summary": "The paper proposes a pruning-based defense method against backdoor attacks on FL. Specifically, the malicious clients are limited to updating model parameters within an isolated subspace, which reduces the attack surface (or \"poison-coupling\") for malicious clients as well as the computation/communication cost in FL. Empirical results of a wide range of metrics compared with SOTA baselines have shown the effectiveness of the proposed method. \n", "questions": "C1: The paper might violate the double-blind policy---from the Dropbox link provided on Page 4, the files shared show the owner is Huang Tiansheng, https://huangtiansheng.github.io. \n\nC2: Please further clarify the following questions:\n\n* How to defend against malicious clients faking the masks and launching adaptive attacks? Yes, the evaluation has experiments on adaptive attacks, but it would be helpful if the design part discusses the design intuition for adaptive attacks. Attacks like [3] might still be injected into the limited subspace quietly, even with random and heterogeneous mask initialization.  \n\n* The intuition for \"consensus fusion (CF)\"---\"Given that those malicious parameters served to recognize backdoor triggers will be deemed unimportant for benign clients, they should not be contained in the subspace of benign clients, which accounts for the majority\" should be further discussed and verified. For non-IID cases, such a consensus might not hold. \n\n* Adding more theoretical analysis would be helpful to explain the intuition, though it might not be easy to derive a thorough theory. \n\n\nC3: Please consider improving the evaluation from the following perspectives: \n\n* More non-IID scenarios could be explored. \n* Involving some NLP datasets will be helpful\n\n\nWriting issues:\n\n* Line 109, \"alloww\" --> \"allows\"\n\n* Figure 2 left could be visualized better with the three metrics \n\n* Figure 3 font size is too small\n\n* \"For FL, defenses can be classified into two main genres...\" The two categories seem not aligned with the classifications in the evaluation (Section 6). For example, which category is RLR belong to? \n\n* \"The malicious/dummy parameters should have less chance to appear in benign clients’ subspaces.\" --- you mean parameters at specific positions? ", "limitations": "See questions. ", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "++ It is interesting to detect and defend against backdoor attacks by exploiting model pruning techniques. \n\n++ The observation of \"poison-coupling\" in FL is inspiring. The paper is well-written, and I enjoy reading the paper. \n\n++ The evaluation involves multiple SOTA baselines of attacking and defense methods. \n", "weaknesses": "\n\nW1: Some design assumptions and intuition could be further clarified, see C2\n\nW2: Evaluation could still be improved, see C3 \n\nW3: Missing related references\n\n [1] Zaixi Zhang, Xiaoyu Cao, Jinyuan Jia, Neil Gong. FLDetector: Defending federated learning against model poisoning attacks via detecting malicious clients. SIGKDD 2022\n \n [2] Hanxi Guo, Hao Wang, Tao Song, Yang Hua, Zhangcheng Lv, Xiulang Jin, Zhengui Xue, Ruhui Ma, and Haibing Guan, Siren: Byzantine-robust Federated Learning via Proactive Alarming, SoCC 2021\n\n [3] Xiangyu Qi, Tinghao Xie, Ruizhe Pan, Jifeng Zhu, Yong Yang, and Kai Bu. Towards Practical Deployment-Stage Backdoor Attack on Deep Neural Networks. CVPR 2022\n\nW4: minor writing issues", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for very detailed comments.\n\n# C1 (Problematic dropbox link)\nWe have stopped sharing the problematic Dropbox link. Apology for this un-intensional error. \n\nThe original submission used both the anonymous github URL (abstract) and the dropbox URL (page 4). Our intention of sharing this dropbox URL is to open-source the checkpoints such that the difference between centralized and federated backdoor models can be displayed. \n\nThe owner id will not be displayed when a dropbox account is logged in. This made us not aware of the fact that the dropbox will expose the owner id. We should have asked other co-authors to double check this dropbox URL. \n\nWe sincerely hope that the PC chairs, the reviewers and the area chair will consider that including this dropbpx URL is an unintentional error when making decision. Thank you!\n\n\n# C2+W1  (Design intuition)\n**Q1 (mask faking)** When clients launch adaptive attack by faking the masks, Lockdown remains to be effective, because the core idea of Lockdown is based on the security principle that the malicious parameters (or coordinates) should not be accepted by other benign clients. More specifically, those poisoned coordinates, even though are present in the global model, during FL training, will be pruned out (masked to 0) once Lockdown consensus fusion is performed.  As a result, the poisoned parameters, even injected by adaptive adversary into arbitrary locations of the model, may not evade the consensus fusion operation enforced by Lockdown. \n\nThe intuition behind the key ideas of Lockdown:\n\n* **(Intuition of subspace searching)**  Each client will only involve the parameters they deem important in their subspace following the dynamic way of subspace searching.\n\n* **(How consensus fusion works)** By introducing consensus fusion, those parameters that appear less among the subspace will be pruned out. This also indicates that the parameters considered not important by most clients will be pruned out. Given that a majority of the clients is assumed to be benign, the poisoned parameters will not be considered as important ones by the benign clients, and therefore they will be pruned out by consensus fusion.\n\n* **(Role of subspace initialization)** Either heterogeneous or homogeneous subspace (or mask) initialization is empirically observed to work in different degrees in lowering ASR, but we indeed observe that different initialization has impact in the model's benign accuracy. \n\n**(When Lockdown falls)** Our adaptive attack Omniscience can succeed to evade Lockdown by obtaining the full/substantial knowledge of the consensus subspace and inject its malicious parameters within the consensus subspace. However, this attack is very hard to conduct without sufficient knowledge of other clients' subspaces or assuming that the FL server is compromised and collude with the malicious poisoning clients. \n\n**Q2 (consensus fusion in Non-IID setting)**\n\nIn non-IID setting, the benign clients would develop more heterogenous subspaces. Hence, when we only reserve the consensus of subspace, the benign accuracy of the model will suffer with a more drastic decline. *However, this does not cause a problem in filtering out the poisoned parameters in conesensus fusion stage*, because there are not backdoor data in the benign clients' local dataset and therefore they would not use the poisoned parameters (or involve them into their subspace), no matter how the non-iid extent is. We show the extra experimental data in our reponse of C3, justifying our claim that Lockdown works well in reducing ASR even in an extreme Non-IID case. \n\n**Q3 (theoretical analysis)**\nOne of our future research will be the theoretical analysis of the Lockdown robustness guarantee, including the robustness bound for subspace size and subspace consensus quorum. \n\n\n# C3 + W2 (Additional Evaluation) \n**(More Non-IID experiment)** Following the suggestions by the reviewer under limited time, we did an additional experiment to demonstrate the efficacy of our Lockdown defense on different Dirichlet parameters. Note that $\\alpha=0.2$ simulate a highly non-iid extent (the benign accuracy of FedAvg drops a large margin in this case).\n\nThe below table shows the number of **benign acc**.\n| $\\alpha$    | 0.2  | 0.5  | 0.8  |  iid (infinite) |\n|----------|-------|------|------|-----------------|\n| FedAvg   | 84.3  | 88.8 | 89.2 | 90.8            |\n| RLR      | 64.5  | 72.9 | 82.9 | 85.5            |\n| Krum     | 26.5  | 43.4 | 43.6 | 75.8            |\n| Lockdown | 80.6  | 86.1 | 88   | 90.1            |\n\nThe below table shows the number of **ASR**.\n\n| $\\alpha$   | 0.2  | 0.5  | 0.8  |  iid (infinite) |\n|----------|-------|------|------|-----------------|\n| FedAvg   | 74.8  | 86.4 | 74.7 | 66.1            |\n| RLR      | 82.5  | 29.5 | 24.5 | 4.3             |\n| Krum     | 2.8   | 11.1 | 6.6  | 4.3             |\n| Lockdown | 7.7   | 3.4  | 3.9  | 7.1             |\n\nAs shown, Lockdown reduces ASR to <10\\% even when $\\alpha=0.2$, which demonstrates its defense efficacy in a highly Non-IID case. \n\n**(NLP dataset)**\nWe are working on SST-2 task with a pretrain Bert model.   \n\n# W3 (Missing related references)\nWe thank the reviewer for sharing the 3 references. We have reviewed them and cited them properly. \n\n# Writing issues\nThanks for correcting the writing issues. We have corrected the typo as well as the font size issue.  For Figure 2, we now use different marker size to represent the pruning ratio.   Our claim that \"defenses can be classified into two main genres\" is indeed over-claimed. We now discuss another genre of defenses known as \"robust aggregation\", which should be the one that Krum and RLR fall in. For the claim \"The malicious/dummy parameters should have less chance to appear in benign clients’ subspaces.\", here the malicious parameters indeed mean parameters at specific positions, i.e., those positions that are corrupted by the attacker. "}]

Author rebuttal:

Meta-review: Most reviewers agree that it is an interesting idea to defend backdoor in FL with isolated subspace training, and detailed empirical evaluation has been provided.  Following the author-reviewer discussion, most concerns raised by reviewers were about requesting additional theoretical analysis and better-clarified design assumptions and intuition. The authors have done an excellent job of addressing most questions. Reviewers generally vote positively after the discussion, though some concerns remain. For example, Reviewer t853 was still unconvinced by the intuition "Given that those malicious parameters served to recognize backdoor triggers will be deemed unimportant for benign clients, they should not be contained in the subspace of benign clients, which accounts for the majority".
Decision: Accept (poster)