SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\theta$  is the threshold for CF, and  $[\cdot ]_j$  indexes the  $j$ -th coordinate of a vector. By applying the global mask produced by CF into the global model, i.e.,  $\boldsymbol{w}_T\odot \mathcal{T}_{\theta}(\boldsymbol{m}_{1,T},\dots,\boldsymbol{m}_{M,T})$ , those parameters that have appearances smaller than  $\theta$  among all the subspaces are sparsified to 0. In this way, those poisoned parameters will mostly be eliminated, thereby reaching the goal of perturbing backdoor.
Datasets and models. We experiment on FashionMnist, CIFAR10/CIFAR100 and TinyImagenet datasets. For CIFAR10/CIFAR100, we use a ResNet9(He et al., 2016) as backbone. For TinyImagenet, we use a modified ResNet9 via adding a pooling layer after the first convolutional layer to keep the same hidden size before output. For FashionMnist, we use LeNet (LeCun et al., 1998).
Attack methods. We classify the backdoor attack methods in FL into three genres, i.e., data-level backdoor, algorithm-level backdoor and advanced backdoor. Data-level backdoor only allows the malicious clients to modify the raw data, but they have no control of the algorithm. On contrary, algorithm-level backdoor allows clients to modify the training algorithm, in addition to the raw data. The advanced attack can access extra server-side information (see Appendix A.2). Indeed, the three genres of backdoor attack methods correspond to the weak, medium, strong threat models in Table 1
respectively. Among the data-level backdoor, we simulate three types of attacks methods, i.e., BadNet (Gu et al., 2017), DBA (Xie et al., 2019), and Sinusoidal (Barni et al., 2019). For algorithm-level backdoor, we simulate two types of attack, i.e., Scaling (Bagdasaryan et al., 2020), and FixMask. For advanced backdoor, we test Neurotoxin (Zhang et al., 2022b) and an adaptive attack Omniscience.
Defense Baselines. We use vanilla FedAvg (McMahan et al., 2016) (without defense) as a baseline, and compare Lockdown with four SOTA defenses RLR (Ozdayi et al., 2021), Krum (Blanchard et al., 2017), RFA (Pillutla et al., 2022) and Trimmed mean (Yin et al., 2018).
Evaluation metrics. Three metrics are used to evaluate the defense performance:
- Benign Acc. Benign accuracy measures the Top-1 accuracy performance of a model given the benign data inputs (without the presence of a trigger).
- ASR. Attack Success Ratio (ASR) measures the ratio of backdoor samples (with trigger) to be classified to the target label. The smaller this metric, the more resilient the model is.
- Backdoor Acc. Backdoor accuracy measures the Top-1 accuracy of the model given the backdoor inputs (their labels during testing are the original one before adding backdoor trigger). We add this metric to evaluate the overall performance, since high backdoor acc means: i) the classification of benign features is well-performing. ii) the perturbation of backdoor trigger is limited.
Simulation setting. We simulate  $M = 40$  clients, and data is either evenly distributed to each client (IID setting) or is distributed with Dirichlet distribution (Non-IID setting) following (Hsu et al., 2019). The parameter for Dirichlet distribution is set to 0.5 for the Non-IID partition. To simulate the backdoor attack launched by the malicious clients, we follow (Ozdayi et al., 2021) to randomly choose  $N$  clients as attackers whose  $p$  (percentage) of data in their local datasets are poisoned. The default backdoor settings for our main experiment are  $p = 50\%$  and  $N = 4$ . We summarize the default simulation setting in Table 2. All the experiments are done with a Nvidia A100 GPU.
Hyper-parameters. For Lockdown, we fix the overall sparsity to  $s = 0.25$ , the mask agreement threshold to  $\theta = 20$ , and the initial pruning rate to  $\alpha_0 = 1e - 4$ . The robust learning rate threshold for RLR is set to 8. The number of local epochs and batch size are fixed to 2 and 64, respectively. The learning rate and weight decay used in the local optimizer are fixed to 0.1 and  $10^{-4}$ . The number of comm rounds is fixed to 200. We summarize the default hyper-parameters in Table 3.
Table 2: Default Simulation Setting.
Table 3: Default hyper-parameter for Lockdown.
Notation	Meaning	Default Value	Notation	Meaning	Default Value
M	Total number of clients	40	α0	Initial pruning rate	1e-4
p	Poison ratio	0.5	θ	Agreement threshold	20
N	Attacker number	4	s	Overall sparsity	0.25
6.2 Main Evaluation
In our main evaluation, we use CIFAR10 as the default dataset and BadNet as the default attack.
Convergence. The convergence result w.r.t communication rounds is plotted in Figure 4. Lockdown offers the strongest robustness under attack with Non-IID settings. Compared with IID, when the data distribution is Non-IID, Lockdown suffers a slight drop in benign accuracy (by approximately  $3\%$ ) while Krum and RLR suffer more. The larger heterogeneity causes more benign parameters to be dropped in the consensus fusion process, resulting in a drop in benign accuracy.
Figure 4: Convergence and defense performance under different defenses.
Table 4: Defense efficacy with varying poison ratio  $p$  under CIFAR10.
Methods (IID)	Benign Acc (%) ↑	ASR (%) ↓	Backdoor Acc (%) ↑
clean	p=.05	p=.2	p=.5	p=.8	clean	p=.05	p=.2	p=.5	p=.8	clean	p=.05	p=.2	p=.5	p=.8
FedAvg	91.0	91.4	91.1	91.0	90.8	1.6	12.4	19.9	66.1	94.8	88.5	79.6	73.4	32.9	5.1
RLR	86.8	86.7	86.6	86.3	85.5	2.3	2.4	2.4	4.3	25.1	84.6	84.3	83.4	81.7	65.2
Krum	76.3	78.0	75.6	76.4	75.8	4.7	3.9	4.3	4.3	4.9	73.8	74.9	72.9	73.9	73.2
RFA	90.9	91.2	91.1	90.8	90.7	1.6	15.8	20.7	83.7	99.3	88.8	76.8	72.4	15.9	0.7
Trimmed mean	91.0	90.6	91.1	90.9	90.8	1.7	5.0	20.7	61.7	96.2	88.5	84.7	72.0	36.6	3.6
Lockdown	90.0	90.0	89.9	90.1	90.0	1.8	3.6	2.5	7.1	4.0	87.9	85.8	86.6	83.7	85.6
Methods (Non-IID)	Benign Acc (%) ↑	ASR (%) ↓	Backdoor Acc (%) ↑
clean	p=.05	p=.2	p=.5	p=.8	clean	p=.05	p=.2	p=.5	p=.8	clean	p=.05	p=.2	p=.5	p=.8
FedAvg	89.0	89.2	89.3	88.8	88.7	1.7	17.3	54.4	86.4	96.7	85.9	74.0	42.5	13.0	3.2
RLR	74.4	74.4	73.6	72.9	72.5	5.8	15.0	40.2	29.5	82.5	69.0	63.1	46.2	51.4	15.3
Krum	42.7	37.4	45.2	43.4	45.1	10.0	5.2	10.4	11.1	10.6	38.6	33.8	40.7	39.3	40.5
RFA	88.8	88.8	88.8	88.3	88.3	2.0	21.4	52.8	90.8	98.7	85.7	70.6	44.3	8.9	1.2
Trimmed mean	88.5	88.4	88.2	88.3	88.3	1.9	25.2	48.4	84.6	96.0	85.4	67.5	47.7	14.7	3.9
Lockdown	85.6	86.2	86.7	86.1	86.6	0.9	7.6	3.6	3.4	3.3	84.1	79.5	82.3	82.2	82.8
Defense efficacy on varying poison ratio. As shown in Table 4, the Attack Success Ratio (ASR) of Lockdown under different data poison ratios are significant lower compared with vanilla FedAvg without defense (up to  $96\%$  reduction), and is also significantly lower than the SOTA backdoor defense solutions. Though ASR of Lockdown is slightly larger than RLR and Krum in IID settings, its benign accuracy is much higher than them (with up-to  $5.4\%$  and  $15\%$  enhancement comparing RLR and Krum respectively). Another observation is that Lockdown performs better in reducing ASR when the data poison ratio is high (in Non-IID setting, ASR is  $7.6\%$  when  $p = 0.05$  while ASR is only  $3.3\%$  when  $p = 0.8$ ). This phenomenon is because the subspaces found by the malicious clients will deviate more from benign clients when a larger amount of backdoor samples are injected in their datasets. Therefore, the malicious subspace will overlap less with benign subspaces, resulting in a better isolation, and also benefit the consensus fusion process. In addition, Lockdown significantly advances backdoor accuracy by up-to  $79.6\%$  compared to FedAvg without defense, which implies that the Lockdown's model can still recognize the true label of backdoor samples even under attack.
Defense efficacy on varying attacker ratio. We fix the poison ratio to 0.5 and vary the attackers ratio to  $\{0, 0.1, 0.2, 0.3, 0.4\}$ . The results are shown in Figure 5. As shown, Lockdown consistently achieves low ASR (at minimum  $30\%$  ASR reduction compared to FedAvg when attackers ratio is 0.4), and high benign accuracy in all groups of experiments (at maximum  $5\%$  drop of benign accuracy compared to FedAvg). In contrast, RLR and Krum are sensitive to attacker ratio and fail in defense when attacker ratio is 0.4 (no ASR reduction for Krum, and at maximum  $10\%$  reduction for RLR).
Figure 5: Benign acc/ASR under different attackers ratio.
Defense against Non-IID extent. Next we show in Table 5 that Lockdown is robust to different Non-IID extent. Our experiment is done in the default setting, but vary different settings of Dirichlet Parameters  $\alpha$ , and it shows that Lockdown can defend against backdoor even in extreme Non-IID cases (e.g.,  $\alpha = 0.2$ , which leads to benign accuracy drop of FedAvg from  $90.8\%$  to  $84.3\%$ ). Our another observation is that Lockdown would suffers more benign accuracy loss (respectively  $3.7\%$ ,  $2.7\%$ ,  $1.2\%$  and  $0.7\%$  benign accuracy loss for the four Non-IID levels).
Table 5: Defense efficiency with different Non-IID extent.
/Methods	α = 0.2	α = 0.5	α = 0.8	IID
Benign Acc ↑	ASR ↓	Benign Acc ↑	ASR ↓	Benign Acc ↑	ASR ↓	Benign Acc ↑	ASR ↓
FedAvg	84.3	74.8	88.8	86.4	89.2	74.7	90.8	66.1
RLR	64.5	82.5	72.9	29.5	82.9	24.5	85.5	4.3
Krum	26.5	2.8	43.4	11.1	43.6	6.6	75.8	4.3
Lockdown	80.6	7.7	86.1	3.4	88	3.9	90.1	7.1
Defense against data-level backdoor. We simulate attacks with the BadNet, DBA, and sinusoidal method. Our results in Table 6 show that Lockdown has good generalization ability to data-level
backdoor attack. Overall. Lockdown maintains superior defense efficacy  $(< 10\%$  ASR) towards all the three data-level backdoor attacks.
Table 6: Lockdown on data-level attack.
Methods (IID)	Benign Acc(%) ↑	ASR(%) ↓	Backdoor Acc(%) ↑
BadNet	90.1	7.1	83.7
DBA	90.0	2.5	86.4
Sinusoidal	89.5	4.6	83.6
Methods (Non-IID)	Benign Acc(%) ↑	ASR (%) ↓	Backdoor Acc(%) ↑
BadNet	86.1	3.4	82.2
DBA	86.5	2.0	82.9
Sinusoidal	87.0	2.0	81.8
Table 7: Lockdown on algorithm/advanced attack.
Methods (IID)	Benign Acc(%) ↑	ASR(%) ↓	Backdoor Acc(%) ↑
Scaling (Lockdown + NC)	89.9	3.2	86.2
Neurotoxin	89.0	2.6	86.1
Methods (Non-IID)	Benign Acc(%) ↑	ASR (%) ↓	Backdoor Acc(%) ↑
Scaling (Lockdown + NC)	89.9	3.2	86.2
Neurotoxin	86.0	1.8	83.5
Defense against algorithm-level/advanced backdoor. Table 7 shows Lockdown's performance on Scaling and Neurotoxin attack. For Scaling attack, the scaling factor is set to 5. We integrate Lockdown with the norm-clipping (NC) in aggregation as proposed in (Sun et al., 2019), and as shown, Lockdown defense is still effective by controlling ASR to  $< 5\%$ . Lockdown is also robust to Neurotoxin, an attack developed from a similar idea as Lockdown (See Appendix A.2 for analysis).
Defense against adaptive attack. We design two adaptive attacks that assume the knowledge of Lockdown and try to break it. Both the attacks try to disobey the mask searching process. Specifically, FixMask allows attackers to fix the initial subspace, and Omniscience is able to infer the global subspace that is produced by CF. As shown Table 8, vanilla Lockdown procedure is vulnerable to them. However, we find that there are two methods can be used to rectify Lockdown to accommodate FixMask attack. i) Enlarge the initial pruning/recovery ratio  $\alpha_0$ , or ii) adopt heterogeneous mask initialization (HM). Both the approaches enhance the dynamics of subspaces, and therefore enhance protection. For Omniscience attack, the attackers always know consensus subspace and therefore can poison the parameters within it. Lockdown cannot cope with this attack. However, the condition of conducting Omniscience attack is stringent, as it has to acquire either global subspace, or other benign clients' local subspace, meaning that it falls into the Strong threat model in Table 1.
Table 8: Lockdown on adaptive attack.
Methods (IID)	Benign Acc(%) ↑	ASR(%) ↓	Backdoor Acc(%) ↑
FixMask	89.9	63.5	35.5
FixMask (α0= 0.1)	88.5	3.5	84.3
FixMask (HM,α0= 0.1)	88.5	5.8	82.8
Omniscence	89.3	44.7	52.1
Methods (Non-IID)	Benign Acc(%) ↑	ASR (%) ↓	Backdoor Acc(%) ↑
FixMask	86.8	86.3	13.1
FixMask (α0= 0.1)	85.7	4.6	79.5
FixMask (HM,α0= 0.1)	86.9	3.7	81.8
Omniscence	86.8	86.3	13.1
Table 9: Communication and # of parameters for Lockdown under IID CIFAR10 (ResNet9) setting. The communication overhead is the sum of that of  $M = 40$  clients in each round.
Methods	Comm Overhead ↓	# of params ↓	Benign Acc(%) ↑
FedAvg	2.10GB (1x)	6.57M (1x)	91.0 (1x)
Lockdown	0.525GB (0.25x)	1.643M (0.25x)	90.0 (0.990x)
Communication and model complexity. As shown in Table 9, we show that Lockdown achieves smaller communication overhead (0.25x compared to FedAvg), since only a small subspace of the entire model gradient needs to be sent between server and clients. In addition, the number of parameters of the inference model is also lowered to 0.25x because the consensus fusion operation prune out the malicious/dummy parameters. Finally, we observe that benign accuracy only drops by 0.01x compared to FedAvg, indicating that pruning will not severely perturb the normal function.
Generalization to varying datasets. We show our evaluation results on FashionMnist, CIFAR10/100 and TinyImagenet in Table 10. As shown, Lockdown achieves SOTA defense efficacy (compared with FedAvg without defense, with up-to  $80.7\%$ ,  $83.0\%$ ,  $73.8\%$ , and  $93.4\%$  reduction of ASR respectively on the four datasets), and maintains a reasonable loss of benign accuracy compared to FedAvg without defense (with up-to  $2.2\%$ ,  $2.7\%$ ,  $3.9\%$  and  $3.3\%$  drop).
Table 10: Performance on varying datasets.
/Methods (IID)	FashionMnist	CIFAR10	CIFAR100	TinyImagenet
Benign Acc ↑	ASR ↓	Benign Acc ↑	ASR ↓	Benign Acc ↑	ASR ↓	Benign Acc ↑	ASR ↓
FedAvg	90.0	85.1	91.0	66.1	70.0	75.7	12.7	96.7
RLR	89.2	6.0	86.3	4.3	61.0	1.9	10.5	98.0
Krum	83.9	1.0	76.4	4.3	26.9	98.8	3.5	99.9
Lockdown	88.7	4.4	90.1	7.1	66.9	1.9	9.4	3.3
Methods (Non-IID)	Benign Acc ↑	ASR ↓	Benign Acc ↑	ASR ↓	Benign Acc ↑	ASR ↓	Benign Acc ↑	ASR ↓
FedAvg	89.2	87.9	88.8	86.4	67.7	74.8	12.1	96.5
RLR	83.7	39.1	72.9	29.5	53.9	36.0	9.2	98.6
Krum	75.8	1.7	43.4	11.1	20.6	1.3	2.6	99.9
Lockdown	87.1	15.6	86.1	3.4	63.8	2.5	9.6	3.1
Visualization. In Figure 6, We plot each filter's weight of the last convolutional layer of a ResNet-9 model trained with Lockdown. The brighter the color is, the larger the absolute value is, meaning that the filter can be activated by some particular inputs. We find that via our Lockdown procedure, the 86-th filter becomes "suspicious". We plot the weights in the classifier that connects this filter, and accidentally find that this filter contribute most to activate the "horse" neuron, which is our target backdoor label. This illustrates that lockdown can break the "poison-couple effect"- poison parameters (i.e., those in the 86-th filter) only appear in subspace that is not shared by benign clients, which can be effectively filtered out by contrasting the benign and poisoned client's subspace.
Attacker  $(\mathrm{id} = 0)$
Benign client  $(\mathrm{id} = 5)$
Figure 6: Visualization of Lockdown. Each squre represent weights of a filter. Left/Middle/Right: projecting the weights into an attacker/a benign client/consensus subspace. Bar chart in the middle: weights in the classifier connected 86-th filter with different neurons. "Horse" is our backdoor target.
Consensus (after CF)
6.3 Ablation and Hyper-parameter Sensitivity Analysis
Ablation study demonstrates the necessity of i) ERK initialization ii) gradient-based recovery, is moved to Appendix A.6. We tune three key hyper-parameters of Lockdown to demonstrate their impacts in our sensitivity analysis. Our findings are: i) setting a proper sparsity  $s$  can increase the model's robustness. ii) Setting a proper initial pruning/recovery rate  $\alpha_0$  is necessary for effective subspace searching, but too large of it will hurt the model's normal function. iii) Consensus threshold  $\theta$  should be set sufficiently large to filter out the malicious parameters. See Appendix A.7 for details.
In this paper, we study the pruning-based defense for FL and observe a "poison-coupling" phenomenon, which degrades the defense performance. To mitigate such an effect, we propose Lockdown, a backdoor defense based on the idea of isolated subspace training. Empirical evidence shows that Lockdown can significantly reduce the risk of malicious backdoor attacks without sacrificing much on benign accuracy. Future works include studying how to generalize Lockdown to other FL settings, e.g., decentralized FL (Hu et al., 2019; Li et al., 2023a; Shi et al., 2023), personalized FL (Fallah et al., 2020; T Dinh et al., 2020; Huang et al., 2022b, 2023; Dai et al., 2022), etc.
This research is partially sponsored by the NSF CISE grants 2038029, 2302720, 2312758, an IBM faculty award, and a grant from CISCO Edge AI program.
Xu, X., Wang, Q., Li, H., Borisov, N., Gunter, C. A., and Li, B. Detecting ai trojans using meta neural analysis. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 103-120. IEEE, 2021.
Yin, D., Chen, Y., Kannan, R., and Bartlett, P. Byzantine-robust distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning, pp. 5650-5659. PMLR, 2018.
Zhang, Z., Cao, X., Jia, J., and Gong, N. Z. Fldetector: Defending federated learning against model poisoning attacks via detecting malicious clients. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2545-2555, 2022a.
Zhang, Z., Panda, A., Song, L., Yang, Y., Mahoney, M., Mittal, P., Kannan, R., and Gonzalez, J. Neurotoxin: Durable backdoors in federated learning. In International Conference on Machine Learning, pp. 26429-26446. PMLR, 2022b.
Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.
Zheng, R., Tang, R., Li, J., and Liu, L. Data-free backdoor removal based on channel lipschitzness. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part V, pp. 175-191. Springer, 2022.
Zizzo, G., Rawat, A., Sinn, M., and Buesser, B. Fat: Federated adversarial training. arXiv preprint arXiv:2012.01791, 2020.
Decay pruning rate with cosine annealing. In our subspace pruning/recovery process, we let the clients prune out  $\alpha_{t}$  percentage of parameters and recover the same amount of parameters to search for the subspace that fits their data. The parameter  $\alpha_{t}$  is decayed with the initial rate  $\alpha_{0}$  with cosine annealing, which can be formalized as follows:
$$
\alpha_ {t} = 0. 5 \times \alpha_ {0} \times \left(1 + \cos \left(\frac {\mathrm {t}}{T _ {e n d}} \pi\right)\right) \tag {7}
$$
where  $t$  is the number of communication round, and  $T_{end}$  is the round that the mask searching is ended (notice that  $\alpha_{T_{end}} = 0$ ). In our implementation, we set  $T_{end} = T$ .
ERK sparsity initialization. We use Erdős–Rényi Kernel (ERK) (Evci et al., 2020), an empirical sparsity distribution technique, to distribute sparsity to different layers of a model. Specifically, the active parameters of the convolutional layer initialized by ERK are proportional to  $1\frac{n^{l-1} + n^l + w^l + h^l}{n^{l-1} * n^l * w^l * h^l}$ , where  $n^{l-1}$ ,  $n_l$ ,  $w^l$  and  $h^l$  respectively specify the number of input channels, output channels and kernel's width and height in the  $l$ -th layer. For the linear layer, the number of active parameters scale with  $1\frac{n^{l-1} + n^l}{n^{l-1} * n^l}$  where  $n^{l-1}$  and  $n^l$  are the number of neurons in the  $(l-1)$ -th and  $l$ -th layer. ERK initialization, in essence, gives more sparsity to those layers with a larger number of parameters while pruning less on those small layers.
Subspace pruning. In the mask searching process, we use parameter's magnitude to guide the pruning of model parameters. We present the PyTorch style code in Algorithm 2 to illustrate the pruning process and, correspondingly, the update of mask. Note that we only prune out parameters that are within the current subspace. Therefore, in line 6, we set the parameters that are out of the subspace to a very large value to prevent from selecting them. After that, we filter out those parameters with the smallest  $\alpha_{t}$  percentage of magnitude and prune them out of the subspace.
Algorithm 2 PyTorch style code for pruning and recovery
1: function Prune_subspace(  $\alpha_{t},\pmb{w}_{i,t,K},\pmb{m}_{i,t + \frac{1}{2}}$  ）   
2: Init layer sparsity  $\{s_l\}$  given overall sparsity  $s$  with ERK   
3:  $\pmb{m}_{i,t + 1} = \pmb{m}_{i,t + \frac{1}{2}}$    
4: for  $l = 0,1,\dots ,L - 1$  do   
5: num prune  $= \alpha_{t}\times \#$  of params in the l-th layer   
6: sort  $=$  torch.where  $(\pmb{m}_{i,t}^{(l)} == 1$  ,torch.abs  $(\pmb{w}_{i,t,K}^{(l)})$  ,  $1000\times$  torch.ones_like  $(\pmb{w}_{i,t,K}^{(l)})$    
7: _,idx  $=$  torch.sort((sort).view(-1))   
8:  $\pmb{m}_{i,t + 1}^{(l)}.\mathrm{view}(-1)[idx[:num\_prune]] = 0$    
9: end for   
10: Return  $m_{i,t + 1}$    
11: end function   
12:   
13: function Recover_subspace(  $\alpha_{t},\pmb{w}_{i,t,0},\pmb{m}_{i,t}$  ）   
14: Derive gradient  $\nabla f_i(\pmb {w}_{i,t,0}\odot \pmb {m}_i,t)$  with one pass of local data   
15:  $\pmb{m}_{i,t + \frac{1}{2}} = \pmb{m}_{i,t}$    
16: for  $l = 0,1,\ldots ,L - 1$  do   
17: num prune  $= \alpha_{t}\times \#$  of params in the l-th layer   
18: sort  $=$  torch.where  $(\pmb{m}_{i,t + \frac{1}{2}}^{(l)} = 0$  ,torch.abs  $(\nabla f_i^{(l)}(\pmb {w}_{i,t,0}\odot \pmb {m}_i,t))$  ,  $-1000\times$  torch.ones_like  $(\pmb{w}_{i,t,K}^{(l)})$    
19: _,idx  $=$  torch.sort((sort).view(-1),descending=True)   
20:  $\pmb{m}_{i,t + \frac{1}{2}}^{(l)}.\mathrm{view}(-1)[idx[:num\_prune]] = 1$    
21: end for   
22: Return  $m_{i,t + \frac{1}{2}}$    
23: end function
Subspace recovery. After pruning and before the next round training, we recover the same amount of parameters to explore other parameters outside the subspace. Following (Evci et al., 2020), we use gradient information of the pruned model to guide the recovery process. Here we only recover the
Figure 7: Examples of BadNet, DBA, and Sinusoidal attack. Labels of poison samples are manipulated to the target label (e.g., a horse).
parameters out of the current subspace, and therefore we set the sort_value of the parameters within the current subspace to a sufficiently small value, as shown in Algorithm 2. Subsequently, we sort in descending to obtain the parameters with the largest- $\alpha_{t}$  percentage gradient magnitude, and recover them by updating masks.
A.2 Attack Methods
Table 11: Application of attack methods in threat models. "√" corresponds to be applicable while "X" corresponds to be not applicable.
Attack methods /	Threat models
weak	medium	strong
BadNet	✓	✓	✓
DBA	✓	✓	✓
Sinusoidal	✓	✓	✓
Scaling	X	✓	✓
FixMask (adaptive)	X	✓	✓
Neurotoxin	X	X	✓
Omniscience (adaptive)	X	X	✓
As we mention in the main body, we classify the attack model into data-level attack and algorithm-level backdoor. We in the following give brief description of each data-level backdoor that we simulate in federated learning setting.