Backdoor Federated Learning by Poisoning Backdoor-Critical Layers
Authors: Haomin Zhuang
Mingxian Yu
Hao Wang
Yang Hua
Jian Li
Xu Yuan
Venue: ICLR 2024 poster (2024)
Keywords: Federated Learning
Backdoor Attack
Primary area: general machine learning (i.e., none of the above)

Abstract: Federated learning (FL) has been widely deployed to enable machine learning training on sensitive data across distributed devices. However, the decentralized learning paradigm and heterogeneity of FL further extend the attack surface for backdoor attacks. Existing FL attack and defense methodologies typically focus on the whole model. None of them recognizes the existence of backdoor-critical (BC) layers-a small subset of layers that dominate the model vulnerabilities. Attacking the BC layers achieves equivalent effects as attacking the whole model but at a far smaller chance of being detected by state-of-the-art (SOTA) defenses. This paper proposes a general in-situ approach that identifies and verifies BC layers from the perspective of attackers. Based on the identified BC layers, we carefully craft a new backdoor attack methodology that adaptively seeks a fundamental balance between attacking effects and stealthiness under various defense strategies. Extensive experiments show that our BC layer-aware backdoor attacks can successfully backdoor FL under seven SOTA defenses with only 10% malicious clients and outperform the latest backdoor attack methods.

Published as a conference paper at ICLR 2024
BACKDOOR FEDERATED LEARNING BY POISONING BACKDOOR-CRITICAL LAYERS
Haomin Zhuang', Mingxian Huo Wang". Yang Hua", Jian Li Xu Yuan
Sooth China University of Technology LLuuisiana State University
S Queen's 's University Belfast, UK "Stony Brook University University of Delaware 623 60860, mxyu1 112]@gmai com, haowange1 su. Y. Huaauaagga Hua.qub.ac..u jian. 1..30........ wws.ttyybrooko...omoo...oooooo.. wwtrookkeedu... edu, xyuanBudel .ddu,
ABSTRACT
Federated learning (FL) has been widely deployed to enable machine learning training on sensitive data across distributed devices. However, the decentralized learning paradigm and heterogeneity of FL further extend the attack surface for backdoor attacks. Existing FI. attack and defense methodologies typically focus on the whole model. None of them recognizes the existence of backdoor-critical (BC) layers-a small subset of layers that dominate the model vulnerabilities. At- tacking the BC layers achieves equivalent effects as attacking the whole model but at far smaller chance of being detected by ofte-offthheart (SOTA) defenses. This paper proposes general in-situ approach that identifies and verifies BC lay- ers from the perspective of attackers. Based on the identified BC layers, we care- fully craft a new backdoor attack methodology that adaptively seeks a fundamental balance between attacking effects and stealthiness under various defense strate- gies. Extensive experiments show that our BC layer-aware backdoor attacks can successfully backdoor FL under seven SOTA defenses with only 10% malicious clients and outperform latest backdoor attack methods.
INTRODUCTION
Federated learning (FL) (McMahan al. 2017) enables machine learning across large-scale dis- tributed clients without violating data privacy. However, such decentralized learning paradigm and heterogeneity in data distribution and client systems extensively enlarge FL's attack surface. In- creasing numbers of attack methods have been developed to either slow down the convergence of FL training (i.e., untargeted attacks (Fang et al.) [2000] Baruch et al. [009 [Shejwalkar & Houmansadry 2021 El El Mhamdi et al., 2018)) or enforce the model to intentionally yl misclassify specific cate-
gories data (i.e., targeted attacks (Xie et al. 2019 Bagdasaryan al. 2020: Bhagoji et al. 2019
Wang et al. 2020 Li et al. 2023)).
As subset of targeted attacks, backdoor attacks (Xie et al., 2019 Bagdasaryan et al., 2020 Wang ett al. 2020) Gu et al.] 2017, Li et al.] (2023) are one of the stealthiest attacks for FL, which train models on data with special triggers embedded, such as pixels, textures, and even patterns in the frequency domain (Feng et al.) (022). Models compromised by backdoor attacks typically have high accuracy on general data samples (i.e., main task) except that samples with triggers embedded activate the "backdoor" inside the model (i.e., backdoor task), leading misclassification targeted to specific labels (e.g., recognizing a stop sign as a speed limit sign).
Several defense methods have been proposed to detect backdoor attacks and mitigate their impacts, which can be classified into three types based on their key techniques: distance-based, inversion- based, and sign-based defense. Distance-based defenses, such FLTrust (Cao et al. 2021) and FoolsGold (Fung et al., 2020). calculate the cosine similarity distance and euclidean distance be- tween the local models detect potential malicious clients. Inversion-baas based defenses, such as Zhang et al. (2022a). utilize trigger inversion and backdoor unlearning to mitigate backdoors in global mod- els. Sign-based defenses, such as RLR (Ozdayi et al., (02T), detect the sign change directions of each parameter in the local model updates uploaded by clients and adjust the learning rate of each pa- rameter to mitigate backdoor attacks. Therefore, existing backdoor attacks can hardly work around the detection reinforced by the aforementioned multi-dimension defenses.
"hii work was performed when Haomin Zhuang and Mingxian Yu were remote intern students advised by Dr. Hao Wang at the LSU IntelliSys Lab.

---

Published as a conference paper at ICLR 2024
We have observed new dimension ignored by existing studies the effectiveness of backdoor at- tacks is only related to a small subset model layers-bbcckoor-crittiii backdoor-critical (BC) layers. To demon- strate the existence of BC layers, first train benign five-layer CNN model until it has converged. Then, we train copy of the benign model on poisoned data (with triggers embedded) and obtain malicious model. We substitute each layer in the benign model for the same layer in the malicious model and measure the backdoor attack success rate, which denotes the ac- curacy of recognizing samples with trigger embedded as the targeted label. Fig. aa) shows that the absence of layers in the malicious model does not degrade the BSR except for the fcl. we i ght layer. Fig. hb) shows the reversed layer substitution that only the fcl. weight layer from the ma- licious model enables successful backdoor tasks. Therefore, we argue that small set of layers, such as fc1. weigght whtt are backloor-crittcal--
clean dataset
absence of even one BC layer leads to a low Backdoor Success Rate. BC layers as small subset of models can be observed in large models like ResNet 8 and VGG19 (refer to Fig. A-22 Intuitively, deeper layers are more BC because shallower layers learn simple, low-level features such as edges and textures, and deeper layers combine these features to learn more complex, high-level concepts such as objects and their parts (Zeiler Fergus 2014
Bau al. 2017 (Simonyan al. 2013).
fc2.bias
fc2.bias
This paper proposes a Layer Substi- tution Analysis, a general in-situ ap- proach that identifies BC layers us- ing forward and backward layer sub- stitutions. We further craft two new backdoor attack methods: layer-wise poisoning attack and layer-wise flip- ping attack. These two backdoor tack methods leverage the identified BC layers to bypass state-of-the-art (SOTA) distance-based based, and sign-based defense meth- ods by carefully attacking the BC lay- ers with minimal model poisoning and small number of clients (i.e., 10% of the participating clients). Our contributions include:
weight
weight
fcl.bias
fc1.blas
weight
fel.weight
conv2.blas
conv2.blas
weight
weight
conv1.bia
convl. weight-
conv
All
convl weight
All except weight
-100 -50 (a) Changes of BSR (%)
0 +50- 100
(b) Changes of BSR (%)
Figure 1: (a) The changes in backdoor success rate (BSR) of the malicious model with a layer substituted from the be- nign model. (b) The changes of BSR of the benign model with layer(s) substituted from the malicious model ("All ex- cept fcl, weight" indicates replacing all layers except the fcl weight with layers from the malicious model).
inversion-
We propose Layer Substitution Analysis, novel method that recognizes backdoor-critical layers, which naturally fits into FL attackers' context.
We design two effective layer-wise backdoor attack methods, that successfully inject back- door to BC layers and bypass SOTA defense methods without decreasing the main task accuracy.
.
Our evaluation wide range of models and datasets shows that the proposed laser-wise backdoor attack methods outperform existing backdoor attacks, such as DBA (Xie et al.) 2019), on both main task accuracy and backdoor success rate under SOTA defense methods.
PRELIMINARIES
2.1
FEDERATED LEARNING (FL)
FL. leverages a large set of distributed clients, denoted as W (1,..., global model ur without leaking any clients' private data to the central coordinator server (McMahan al. 2017). Formally, the objective is to solve the following optimization problem:
.....,,7 N}. to iteratively learn
min F(w)= Σ (1((())))
VEN
where (ww(()) = DDtT Eeay)D E(2, y; ()) 3 is the local objective function of i-th client with
its local dataset ( and y(0 = | EieN |D(0 is the relative data sample size. FL training process solves this optimization problem by aggregating local models from distributed clients to update the global model iteratively.
2.2 THREAT MODEL OF BACKDOOR ATTACKS
Attacker's goal: As the existing studies on FL backdoor attacks (Gu et al. 2017 Bagdasaryan et al.] 2020 Xie et al. 2019 Ozdayi al. 2021 Wang al. 2020), attacker's goal is enforce models
2

---

Published as a conference paper at ICLR 2024
Malicious model
madiciou
(bitm11)
Wbzm(()
Evaluating
Local training
ASR
Step
D poisoon.ruai
b2m(
m2
BSR 26 > BSR
Initial model w
Benign model Weenigg
Wm2b5(5))
Testing
Figure 2: Identifying BC layers with Layer Substitution Analysis. 122m(l) indicates inserting the 1,-th layer in the benign model to the malicious model, m2b(L") indicates inserting the malicious model's s layers within the subset L to the benign model, and BSR indicates Backdoor Success Rate.
to classify data samples with triggers embedded to specific incorrect labels (ie., the backdoor task), while keeping a high accuracy for samples without triggers embedded (i.e., the main task).
Attacker's capabilities: We assume that an attacker compromises subset M = {1....,M, of malicious clients. However, the proportion malicious clients is assumed to be less than 50%, i.e. [M|/||] < 50%. Otherwise, existing FL defense methods can hardly withstand such backdoor attacks. Following existing studies (Fung et al., 20200 Fang et al., 20200; [Bagdasaryan et al.) [020] Baruch et al.] 2019 Yin al., 2018:; Ozdayi et al.) 2021; Nguyen et al.) [202I). malicious clients controlled by the attacker can communicate with each other to synchronize attacking strategies. The snapshot of the global model in each round and can directly manipulate
attacker also has access
model weights and datasets on each malicious client (Fang et al. 2020 Li et al. 2023).
IDENTIFYING BC LAYERS
In the FL setting, there is global model in each turn, where clients train their local data with a few epochs, which the models in clients are similar to each other so that the new global models are from averaging all clients' models (Konečný al. 2016). So in this setting, have an opportunity to explore the difference between the malicious models that are trained on some poison dataset and benign that are trained on the clean dataset.
We argue that if the 1-th layer (or a subset L of layers) is critical to the backdoor task, substituting the layer(s) in Whenign with the same layer(s) in umalicious will cause decline in the accuracy of backdoor task.
3.1
OVERVIEW
Fig. 2 presents the Layer Substitution Analysis identify BC layers for each malicious client controlled by the attacker, where i E .::
Step 1:
The initial model w is trained on the clean dataset D Detean train to obtain benign model Whenign- Then, the benign model Wbenign is further trained on the poison dataset D poison, train to converge for learning backdoor task to obtain the malicious model Wmalicious
Step 2: Forward layer ietiiitution-ubstttuuue
the individual (-th layer of the malicious model Wmalicious with the 1-th layer of the benign model Whemign iteratively, where we evaluate the backdoor success rate (BSR) of the updated malicious model Wezm(l)) and compare it with the BSR of the original malicious model Wmaliciour- We sort the layers by the changes in BSR.
Then
Step 3: Performing backward layer substitution following the order of layers sorted in Step 2. We incrementally copy the layer from the malicious model to the benign model until the BSR of the updated model reaches a threshold. Then, the indexes of the copied layers denote the set of BC layers L*.
3.2 LAYER SUBSTITUTION ANALYSIS
Step I: Local Training In FL setting, malicious clients identify BC layers on their local datasets. A local dataset D in i-th malicious client is split into training sets (i) cleam.truin and D (8)
poison,train as poison, vat- Upon malicious client i receives the global model
well as validation sets (+)
vel and (()
3

---

Published as a conference paper at ICLR 2024
w, it trains the benign model Whenign from the global model on the clean dataset D clean until it converges. Then, the attacker trains the Whenign on the poisoned dataset D (i) obtain a malicious model "malicious
poison,train to converge to
Step 2: Forward Layer Substitution We argue that if a layer is BC, replacing it in the malicious model with a "benign" layer from the benign model will decrease the malicious model's backdoor task accuracy (BSR).
Benign layer > malicious model: We first examine the change of BSR when the malicious model replaced with a layer from the benign model at each client i. Specifically, we use b2m(l) to denote the process that replaces malicious model's 1-th layer with a benign model's is I-th layer, where both models have the same structure, including L] layers (L denotes the set of layers).
As Fig. 2 shows, executing 62m(l) generates an updated malicious model (b2m(() per layer replace- ment. We then evaluate the BSR of the updated malicious models (1)2m(1) | E with the poisoned dataset D polson, val- By iterating through all layers ( L, each malicious client can sort the layers according to the change of BSR, defined as:
ABSRazm(1) == maamiciio BSRazm(1)>
where BS denotes the BSR of the poisoned model Umaficiours and BSRp2m(() denotes the BSR of the updated model Wb2m((). which has the 1-th layer replaced. With the layers sorted by the ABSRa2m(1) from the highest to the lowest, we further perform backward layer substitution to confirm the identification of BC layers.
Step 3: Backward Layer Substitution We argue that if a layer is BC, replacing it in the benign model with a "malicious" layer from the malicious model will increase the BSR of the benign model.
Malicious layers - benign model: The backward layer substitution process is defined as m2)(L"). Unlike b2m(1) that only replaces an individual layer, m2b (L) replaces a subset L of layers. We iteratively add a layer into I," following the descending order of ABSR)2m()) and evaluate the BSR of the updated model with the poisoned dataset D val. Fig. 2 shows m2b(L") iteratively copies the subset L of layers from the malicious model Wmalicious to the benign model Whenign until BSR L*) reaches pre-defined threshold 7 =BSRmaliiouu.. where T (0, 1] and BSRm2or L*) denotes the BSR of the updated model Wm2b(1.t)- Specifically, we compare BSRm2a(T.+) with the threshold RBSRmatiiious as follows:
If BSRm26(1) < TBSR we should add another layer following the descending order of A BSS222((1 to L' and execute m2b(L") to update the model Wm2D(L))- Then, we re-evaluate the BSR of the updated model on the poisoned dataset D Ppoison, val and compare it with the threshold again.
If BSR (2)())) 2 BBSRmaiiciuuss the new model (m))((-) has achieved similar BSR as the malicious model BSRmalicions. We stop adding more layers to L"
Then, we argue that the layers in the subset L are BC since these layers satisfy both conditions: 1) removing them from the malicious model decreases its BSR. 2) copying them to the benign model increases its BSR to similar rate as the malicious model. It should be noted that backward layer substitution can identify individual BC layers and BC combinations of layers (i.e., the backdoor task jointly learned by a combination of layers).
4 POISONING BC LAYERS
The identified BC layers provide a new perspective to craft more precise and stealthy backdoor attacks on FL. This section presents two attack methods with awareness of backdoor-critical layers: layer-wise poisoning (LPP attack that attacks both distance- and inversion-based defense methods and layer-wise flipping (LF) attack that attacks sign-based defense methods.
4.1 LAYER LAYER-WISE POISONING (LP) ATTACK
With the subset L of identified BC layers, design LP attack that selects BC layers from L' and precisely poisons the layers with minimal modification, which can bypass existing distance- based defense methods (Cao et al. 2021 Nguyen et al.) 2021 Blanchard et al.) 2017).
In t-th round, malicious clients selected by the FL server perform forward layer substitution and backward layer substitution to find out the set L. of BC layers. After receiving the global model w,
4

---

benigg will be sub- Published as a conference paper at ICLR 2024
(we denote w w, for simplicity), malicious client trains two local models to maticious and ui benign. with its local dataset D poison and D(f) cleow respectively.
(7)
(4)
We propose a vector v= P2, "2,..., Pi] to denote the selection of the subset from the model (e)
20 benien or w maliclous' If v = 1, the j-th layer of the benign model w (8)
(e)
stituted with the corresponding layer in the malicious model uo maticious" (8)
We next introduce
malicious [|u
[1 malicious, malleious, U malicious. to denote the model w maltciows in layer space, where
(5)
(+)
mmaiicious.j is the j-th layer in the model. u bemign denotes the model W benign layer-wisely in the same
(2)
way. The goal of the attacker in round t is formulated as an optimization problem:
max V
D
Σ
P[G(x') = s/: wc++)
(1)
(Z.)))
S.t. W+++1
A A(
((() un (M+1),
w (o)))
(2)
W = v u malicious
(1 - v) bemign
(3)
where denotes the element-wiss multiplication, W denotes global model weight in round / A denotes the aggregation function in the server, x' denotes a image embedded with trigger, y/ denotes the targeted label, and G(z) denotes the predicted label of global model with input r. Ag- gregation functions A can utilize clustering algorithm K-means or HDBSCAN, making it infeasible to calculate gradients.
To address this optimization challenge, we propose straightforward approach. In order to conform the constraints, the attacker must perform adaptive attacks by adjusting the number of layers targeted during each round of the attack. Following previous work (Fang et al. 2020), attacker can estimate the models in benign clients using the local benign models on the malicious clients. These locally-available benign models can then be utilized to simulate the selection process on the server, through the initialization of the crafted model ( with subset L obtained through Layer Substitution Analysis. When the crafted model (6()) is rejected during the simulation, attacker decrease the size of the subset by removing layers in the order which they are added to the set in backward layer substitution process. To further minimize the distance, attacker uses the model
averaged from those local benign models Maveage J Zn of benign make (+) closer to the
(k)
center of benign models. Then introduce hyperparameter À Z 0 to control the stealthiness of the attack:
T >v (i) maticious
(1 X) v Maverage + (1 v) Maverage:
(4)
where (c) = if 0 and (eLU(x) 0, otherwise, and it is similar to Scaling attack when À >>> 1.
For defenses that lack filter-out strategies like FedAvg, attacker can assume that the server is imple- menting strict distance-based strategies, such as FLAME and MultiKrum, to solve the optimization problem within this framework. The identification is not necessary for each round, attacker can identify BC layers in any frequency. like every 5 rounds. The analysis of the trade-off between frequency and backdoor success rate refers to $5.4
4.2 LAYER-WISE FLIPPING (LF) ATTACK
When LP attack fails bypass sign-based defense methods. the backdoor-related parameters prob- ably reside in the non-consensus sign regions and are neutralized by learning rates with reversed signs. To work around such sign-based defense methods, we propose a Layer-wise Flipping attack that keeps the efficacy of BC layers by proactively flipping the parameters signs of the layers in L on each client before the defense methods apply a reversed learning rate to the layers, defined as:
(6) w WiF = -w L-) w) + w.
(4)
Eventually, the parameters of BC layers are flipped by the FL server again, which restores the sign of the parameters and activates the backdoor injected into the model. With the knowledge of BC layers, Layer-wise Flipping attack avoids unnecessarily poisoning the other layers, which improves the main task accuracy and disguises the malicious updates from being detected by the defense methods.
5

---

Published as a conference paper at ICLR 2024
Detection accuracy of FLAME and MultiKrum on CIFAR-10 dataset. MAR indicates malicious clients acceptance rate (%), and BAR indicates benign clients acceptance rate (%)).
Table 1:
Models (Dataset)
MultiKrum
FLAME
MultiKrum
HLAME
Attack
Dou-lID
non-IID
IID
IID
MAR
BAR
MAR
BAR
MAR
BAR
MAR
BAR
Baseline
10.1
43.28
16.58
73.54
0.5
44.39
0.0
69.11
VGG19
LP Attack
91.0
34.33
93.0
59.39
99.5
33.34
100
55.67
(CIFAR-10)
DBA
0.5
44.39
12.25
74.1
0.5
44.39
0.08
68.61
Baseline
3.0
44.11
5,58
44.44
72.95
(CIFAR-10)
ResNet18
LP Atlack
93.01
34.11
93.0
59.39
94,35
33,97
99.0
$8.83
0.5
44.39
72.55
Baseline
0.0
44.44
0.25
66.81
0.0
44.44
0.0
66.78
(Fashion-MNIST)
CNN
Attack
35.77
100.0
55.67
DBA
0.0
44.44
0.5
67.11
0.0
44.44
0.0
66.69
EVALUATION
We implement Layer Substitution Analysis and the two attack methods by PyTorch (Paszke et al, 2019). We conduct all experiments using NVIDIA RTX A5000 GPU. By default, we use 100 clients in FL training, while 10% of them are malicious. In each round 10% clients are selected to train models locally. The non-IID dataset are sampled as q = 0.5 following Cao et al. (202T) Please refer to 11 for the details of experiments settings.
5.1 METRICS
Acc denotes the main task accuracy of the converged global model on the validation dataset. Back- door success rate (BSR) is the proportion that the global model successfully mis- clis-clasifies images with triggers embedded to the targeted labels. Benign-client acceptance rate (BAR) and malicious- client acceptance rate (MAR) indicate the accuracy of defense strategies detecting malicious clients. BAR denotes the proportion of benign models accepted by defense aggregation strategies among all benign models uploaded by benign clients. MAR denotes the proportion of malicious clients ac- cepted by defense aggregation strategies.
5.2 THE ATTACKS' STEALTHINESS
Table shows that MultiKrum and FLAME successfully prevent most malicious updates by the baseline attack and DBA since their MARs are approximating to zero. Besides, the large gap be- tween MARs and BARs of the baseline attack and DBA indicates that MultiKrum and FLAME easily distinguish malicious and benign updates when selecting updates for aggregation.
However, the high MAR achieved by LP attack indicates it successfully has its malicious updates accepted by the FL server running MultiKrum and FLAME. LP attack bypasses the detection of MultiKrum and FLAME all settings. Besides, the gap between LP Attack's MAR and BAR indicates that malicious updates are more likely to be accepted as benign ones by the server.
To further demonstrate the stealthiness of LP attack, we plot the Krum distance in BadNets attack, Scaling attack, and LP attack in ResNet18 trained on IID CIFAR-10. The sum of square distance is denoted as Krum distance. A large Krum distance means the model update is far from other local model updates and less likely to be accepted by the server. Malicious model updates from LP attack are close to benign model updates, which causes the failure of MultiKrum detection.
Fig. A-9 plots participant clients' Krum distance in each 5 rounds, which shows that it is hard for the defense strategy distinguish malicious updates attacked by LP attack from benign ones. Scaling attack presents larger Krum distances than BadNets attack, so we do not consider Scaling attack as normal baseline attack in our experiments.
5.3 THE ATTACKS' EFFECTIVENESS
Table 2 shows that LP attack achieves the highest Acc (i.e., main task accuracy) and the highest BSR under most settings. Fig. 3 illustrate that the convergence rate the backdoor task using the LP attack is generally faster than the baseline attack across various settings. We can observe the similar results in IID settings in Table A-6 and Fig, A-11 in Appendix.
Notably, for large models such as VGG19 and ResNet18 CIFAR-10, LP attack is successful in embedding the backdoor, while the baseline attack and DBA fail in FLAME (IID and non-IID), KulmiKrum (IID and non-IlI)), Fe.ettctorr (IID), and FLARE(IID). Even in the scenario of Multi-

---

Published as a conference paper at ICLR 2024
Table 2: Main task accuracy and BSR on Non-IID datasets. We mark the BSR below 10% (corre- sponding to ten classes of the datasets) as red, indicating failed attack, and mark the highest BSR as bold within the same setting. The Baseline BadNets (Gu et al.] (017). The results are the average of five repeated experiments. For LP attack (LF attack), atb, where a is the mean value, and is the standard deviation. Acc: main task accuracy (%), Avg: average, BSR unit: %.
Model (Dataset)
VGG19 (CIFAR- 10)
ResNei18 (CIFAR 10)
CNN (Fashion MNIST)
Baseline
(LF Attack)
LP Attack
DBA
Baseline
(LF Attack)
LP Attack
DBA
Baseline
LP Altack (LF Attack)
DBA
(non-1ID)
FedAvg
Avg BSR
Best BSR
74.69
84.88
83,55=0.43
92.82:0.99
25.88
41.15
85,19
70.53
94.19220999
89.12+144
21.19
10.94
99.97
99.9
87.69:11.3
18.8449.10 16
99.9
99.97
Acc
78.89
79,95=0.46
78,97
77.58
77.8920.43
77.99
88.28
88.4220.23
87.95
Best BSR
92.91
76.56=31.38 38
42.14
92.43
82.05-+25,32
37.16
74.17
89.4443. 11
100.0
FLTrusst
(non-1ID)
Avg BSR
67.3
65,44131.56
15.88
75.84
71.52:129.17
15.11
68.97
77.05:1-4.67
100.0
Acc
75.1
74.03=1106
75.11
75.72
69.9=5.74
77.51
89.51
89.4820..1
89.31
FLAME
Best BSR
47.03
88.68 14.98
38.25
23.04
95,41110.00
9.77
0.18
84.33.13.12
0.58
(nom IID)
Avg BSR
7.78
60.72-12.44
7.33
7.22
90.15.1:.551
3,88
0.1
74.91 666
0.4
Acc
62.91
56,92=1.12
63.3
76.04
71.48+0.30
75.27
87.78
87.05
87.89
92.17=11881
93.16.10.85
0.0
Besi BSR
79.37
(2.79=0.81)
43.79
81.61
(1.37=0.02)
13.85
20.27
(70.52+:3.1) 13)
38.25
89.24
82.14.17.466
0.0 1: 0.0
Avg BSR
74.01
(0.6=0.099
33.69
60.83
(0.7::0.11
7.8
15.09
(66.12::2.91)
7.33
RLR (non-IID)
72, 1.0.58
73.44 0..44.10..0 1.0.95
86.09 1. 0.13
Ace
67.33
(63.2.133.00
64.3
75.07
(76.48 110.3))
75.04
85.56
(86.45:100.8)
63.3
Multi Knum (nom IID)
Best BSR
22.93
95.87 10..1
29:44
12.72
95.94 0...97
10.63
1.09
89.95 1:2.74
0.28
Avg BSR
7,84
75.93=2.49
8.44
3.95
90.1121:1.35 38
5.61
0.39
74.94:16.99
0.1
Acc
58.93
69.28 13.29
64.81
74.49
72.26.1.1.33 1.34
73.02
87.31
$7.58.1.0.2
87.58
Best BSR
95.49
87.28
16.28
5.23
901.31 12.04
5.89
74.64
99.45.10.13
99.93
FLDetector
(nom IID)
Avg BSR
95.42
86.71+00554
16.14
5.21
86.56+ 1.32
5.87
66.
96. 96.32-400.4
99.9
Ace
55.25
57.95 11.37
56.67
64.39
63.89100.9
65.25
79.16
75.96.10.81
79.78
FLARE (non- IID)
Best BSR
96.67
93.47
25.48
17.16
79.94
26.96
2.02
82. 82.64.14.116 16
100
Avg BSR
94.45
70.23+5.83
8.18
6.24
53.72+-7.77
6.62
1.54
78. 18-+-242
100
Acc
70.25
77.28 11.46
69.95
71.39
70.84.1 1...1.663 1.63
64.22
88.29
88.07 1.0.46
88.01
Acc under baseline attack
Acc under attack
BSR under baseline attack
BSR under LP attack
100
100
100
100
Accuracy
Accuracy
80
Accurao
Accuracy
80
60
60
40
40
40
40
20
20
0 0
50
100 150
200
0
50
100 150 Rounds
200
50
100
150
200
0
100 200300400 500 Rounds
Rounds
Rounds
(a) FLAME
(b) FLTrust
(c) MultiKrum
(d) FLDetector
Figure 3: VGG19 trained with different robust aggregation rules on non-IID data. Ace indicates the main task accuracy, and BSR indicates Backdoor Success Rate.
Krum (non-IID), where LP attack shows fluctuations in BSR illustrated in Fig. 3(c) the average BSR is still up to 76.85% indicating that the attack is successful in most rounds.
The sign-based defense method RLR fails reverse the signs of parameters in large models, thus LF attack fails to embed the backdoor by reversing the signs.
Fig. 3 and Fig. A-11 in the Appendix present the training progress of VGG19 non-lID and IID data, respectively. The figures show that LP attack outperforms the baseline attack in most cases in terms of the main task accuracy and BSR. For small model CNN, FLAME (IID and non-lll)), RLR (IID and non-III), MultiKrum (IID and non-IID), FLDetector (IID), and FLARE (IID) effectively defend against both baseline attacks and DBA attacks. However, the LP attack is successful in bypassing all cestance-based defense strategies in both IID and non-IID settings. The LF Attack is also effective in circumventing based defense method RLR, resulting in an increase of 35% (IID) and 50% (non-IID) in BSR compared to the baseline attack.
5.4 SENSITIVITY ANALYSIS
BC Layer Identification Threshold 7: We conduct a sensitivity analysis of r. which is the BC layer identification threshold, by training ResNet18 8 with IID datasets under FLAME and MultiKrum protection. The average BSR in Fig. 4 shows that LP attack is consistently effective under different T values. Larger t indicates more layers identified as BC layers and included in L", leading to higher risk of being detected due more layers being attacked. The adaptive layer control can
7

---

Published as a conference paper at ICLR 2024
100
Backdoor Success Rate
100
100
80
Backdoor Sutzass
60-
60
40-
40
Backdoor
40-
FLAME
20-
FLAME MutiKrum
20
MultiKrum
8.0
0.2
0.4 0.6
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Value
Value of
Value of A
(a) CNN
(b) ResNet18
Figure 4: BSR v.s. different values used by LP attack.
Figure 5: Different values in CNN trained on Fashion-MNIST and ResNet18 trained on CIFAR-10.
Backdoor Success Rate
100
Backdoor Success Rate
100
Backdoor Success Rate
100
BC Layers
80
non-BC Layers
60
60
40
BC Layers
non BC Layers
identffication interval (Rounds)
Number of Layers
Number of Layers
5 6
Figure 6: Impacts of different BC layers identification inter- vals.
(a) ResNet18
(b) VGGI9
Figure Attacking a fixed number of BC layers or non-BC layers
under FLAME training ResNet18 on IID CIFAR-10 10 dataset.
decrease the number of attacking layers properly to bypass detection, which makes sure LP attack keep effective when high.
Stealthiness Knob A: A sensitivity analysis of the parameter X, which governs the stealthiness of attacks, performed by utilizing CNN and ResNet18 with IID datasets Fashion-MNIST and CIFAR-10, respectively, under the FLAME and MultiKrum defence. Fig. [ demonstrates that the LP attack method attains the highest level of success when 0.5 for the CNN model and for the ResNet18 model. In CNN experiments, LP attack is detected when À > 0.7 in MultiKrum and A > 0.6 in FLAME.
The Impact of Identification Interval: In our experiments, attacker identifies BC layers in each round, which is computationally expensive. Although the set BC layers is varying with the process of FL. the sets in each round are similar. So attacker can reuse the BC layers in previous rounds or even in the first round only. We conduct experiments on ResNet18 trained on IID CIFAR 10 dataset under FedAvg. The results in Fig. show that higher frequency can achieve higher BSR. The BSR is 37.9% if always reusing first-round identification. In practice, the attacker can select the frequency of identification based their device capabilities.
Table 3: Ablation study on BC layers FLAME.
5.5 ABLATION STUDY
Model
Attack
BSR (%)
MAR (%)
Importance of BC layers: To show how
Baseline
2.58
16.58
VGG19
LP Attack
83.86
100
BC layers work in Layer-wise Poisoning (L.P) attack, we design control group-Random Random Layer-wise Poisoning attack-that malicious clients randomly choose the same number of non-BC layers in LP attack to craft model (t)
Random I.P Attack
3.36
98.5
Baselinc
3.33
0.17
ResNer18
LP Attack
89.9
100
Random I.P Attack
46.48
98.5
We evaluate the LP and Random LP attacks under FLAME by training the VGG19 and ResNetl 8 models on CIFAR-10 10 in IID setting.
Fig. 7 shows that attacking the same number of BC layers always achieves better performance in BSR, especially in VGG19. The results presented Table that the primary reason for the failure of baseline attacks is the low acceptance rate of malicious models, with only 16.58% and 0.17% accepted models for ResNet18 and VGG19, respectively. In contrast, the primary limitation of the Random Layer-wise Poisoning attack is its incorrect choice of model parameters, despite its high malicious acceptance rate of 98.5%. The failure Random LP attack highlights the importance of BC layers for achieving successful backdoor tasks.
8

---

Published as a conference paper at ICLR 2024
Impact of the model averaging and adaptive change of layers: The average model Uaverage in Equation equation A and adaptive layer control are two mechanisms introduced in $4.1 to improve the ability of malicious models to mimic benign models, thus enabling them to evade detection by defenses. In order to demonstrate the efficacy of the I.P attack, experiments are conducted both with and without these mechanisms. The results presented in inndicate that these mechanisms sig- nificantly contribute to deceiving defense strategies by increasing both the selection rate of malicious models and BSR. Notably, both mechanisms have great impact on MAR. For instance, in VGGI9 trained on non-IID CIFAR-10 10, model averaging increases MAR from 51% % to 76%, while adaptive control rises MAR from 51% 66%. These mechanisms are capable of working collaboratively to
further improve the MAR to 93%.
Table 4: Training on CIFAR- 10 dataset with (v) and with-
Further Evaluation in Appendix:
out (x) Average Model and Adaptive Layer Control.
In $ we illustrate the superior- ity of the LP attack over SRA (Qi eet al. 2022) by significantly increas- ing the BSR from approximately 4% to 96%. Additionally,
Model
Adaptive Control
Distribution
Model
MAR (%)
BSR (%)
Averaging
non-IID
ResNet18
93.01
90.74
non IID
ResNe118
X
76.0
87.43
non-lID
ResNet18
66.48
93.36
outper- form Constrain Loss Attack (Li et al.]
noa-IID
ResNet18
51.8
87.63
2023 by achieving substantial margin of 82% BSR in MultiKrum and demonstrate how LP attack corrupts Flip (Zhang et al., (022a) and achieve about 60% BSR in average. In T33 we show LP attack attains an approximate 80% BSR under low accuracy conditions in BC layers identification scenarios. In $14 we exhibit the LP attack's 's ability to evade adaptive layer-wise defense mecha- nisms, achieving no less than a 52% BSR. In [15] we show that the LP attack can successfully inject backdoor attacks even when only 0.02 of the clients are malicious. In TT we provide evidence that our LP attack performs effectively in datasets characterized by a high degree of non-IID with a parameter q = 0.8.
6 RELATED WORK
Several studies, such as Bai et al. (2020): Rakin et al. (2019 2020 2021), inject backdoor by flipping limited bits in the computer memory. Qi et al. (2022) selects a path from the input layer to the output layer to craft a subnet that activates for backdoor tasks only. However, those attacks can be detected by FL defenses they pay limited attention to their distance budget.
Subnet Attack:
Memorization Training Data: Stephenson et al. (00TT; Baldock et al. (202T) believe deep
layers are responsible for the memorization of training datasets. However, Maini et al. (2023) finds that the learning of noisy data in training datasets not related to the specific layers utilizing Layer Rewinding to detect the decrease in the accuracy of the noisy training dataset, which is similar to our forward layer substitution. The difference between our conclusions and Maini et al. (2023) may lie in the different tasks, where train models link trigger with specific label but Maini al. 2003) train model to a set of "hard to learn" data, which requires more modification on parameters.
More Related Works. There are variety of previous studies related to our work. We provide more detailed discussion on related works in {T8
LIMITATION AND CONCLUSION
Limitation Single-shot attack (Bagdasaryan et al. 2020) has the capability to inject backdoor into the global model through a malicious client within single round by scaling the parameters of malicious models. While our LP attack can narrow the distance gap by targeting BC layers, we acknowledge that it may not effectively support a large scaling parameter, such as À 100 in DBA, when confronted with stringent distance-basee based defenses. However, there are several possible methods to improve the LP attack to support larger scaling parameters, e.g., searching BC neurons or designing triggers related to fewer parameters.
This paper proposes Layer Substitution Analysis, an algorithm that verifies and iden- tifies the existence of backdoor-critical layers. We further design two layer-wise backdoor attack methods, LP Attack and LF Attack that utilize the knowledge of backloor-critical layers to craft effective and stealthy backdoor attacks with minimal model poisoning. We evaluate the relationship between backdoor tasks and layers under an extensive range of settings and show our attacks can successfully bypass SOTA defense methods and inject backdoor into models with a small number of compromised clients.
Conclusion

---

Published as a conference paper at ICLR 2024
8 ACKNOWLEDGEMENTS
The work of H. Wang was supported in part by the National Science Foundation (NSF) grants 2153502, 2315612, 2327480, and the AWS Cloud Credit for Research program. The work of J. Li was supported in part by the NSF grants 2148309 and 2315614, 14, and the U.S. Army Research Office (ARO) grant W911NF-23-1-000 2019511, 2348452, and 2315613. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do necessarily reflect the views of the funding agencies.
The work of X. Yuan was supported in part by the NSF grants
REFERENCES
Manaar Alam, Esha Sarkar, and Michail Maniatakos. Perdoor: Persistent backdoors in federated learning using adversarial perturbations. In 2023 IEEE International Conference on Omni-layer Intelligent Systems (COINS), pp. 1-6. IEEE, 2023.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In Proc. of AISTATS, 2020.
Jiawang Bai, Baoyuan Wu, Yong Zhang, Yiming Li, Zhifeng Li, and Shu-Tao Xia. Targeted attack against deep neural networks via flipping limited weight bits. In International Conference Learning Representations, 2020.
Robert Baldock, Hartmut Maennel, and Behnam Neyshabur, Deep learning through the lens of
example difficulty, Advances in Neural Information Processing Systems, 34: 24:10876-10889, 2021.
Gilad Baruch, Moran Baruch, and Yoav Goldherg. A little is enough: Circumventing defenses for distributed learning. In Proc. of NeurlPS, 2019.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2017.
Jeremy Bernstein, Jiawei Zhao, Kamyar Azizzadenesheli, and Anima Anandkumar. signSGD With Majority Vote Communication Efficient and Fault Tolerant. arXiv preprint arXiv: 1810.0229, 2018.
Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing Federated Learning Through an Adversarial Lens. In Proc. of ICML, 2019.
Peva Blanchard, El Mahdi EI Mhamdi, Rachid Guerraoui, and Julien Stainer, Machine learning with adversaries: Byzantine tolerant gradient descent. In Proc. of NeurlPS, 2017.
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. Leaf: benchmark for federated settings. arXiv preprint 1rXiv:1812.01097,, 2018.
Xiaoyu Cao, Minghong Fang, Jia Liu, and Neil Zhenqiang Gong. Fltrust: Byzantine-robust feder- ated learning via trust bootstrapping. In Proc. of NDSS, 2021.
Siyuan Cheng, Yingqi Liu, Shiqing Ma, and Xiangyu Zhang. Deep feature space trojan attack of neural networks by controlled detoxification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1148-1156, 2021.
Yu Cheng, Felix X Yu, Rogerio S Feris, Sanjiv Kumar, Alok Choudhary, and Shi-Fu Chang. An Exploration of Parameter Redundancy in Deep Networks With Circulant Projections. In Proc. of the IEEE ICCV, pp. 2857-2865, 2015.
Misha Denil, Babak Shakibi, Laurent Dinh, Marc'Aurelio Ranzato, and Nando De Freitas. Predict- ing Parameters in Deep Learning. In Proc. NeurIPS, volume 26, 2013.
Mahdi El El Mhamdi, Rachid Guerraoui, and Sébastien Rouault. The hidden vulnerability of dis- tributed leamnnn in byzantium. In Proc. of ICML, 2018.
10

---

Published as a conference paper at ICLR 2024
Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Gong. Local model poisoning attacks to byzantine-robuss federated learning. In Proc. of USENIX Security, 2020.
Yu Feng, Benteng Ma, Jing Zhang, Shanshan Zhao, Yong Xia, and Dacheng Tao. Fiba: Frequency- injection based backdoor attack in medical image analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20876-20885. 2022.
Jonathan Frankle and Michael Carbin. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, In Proc. of (ICLR), 2019.
Ido Freeman, Lutz Roese-Koerne, and Anton Kummert. Effnet: An efficient structure for convo- lutional neural networks. 2018 25th ieee international conference on image processing (icip), pp. 6-10. IEEE, 2018.
Clement Fung, Chris JM Yoon, and Ivan Beschastnikh. The limitations of federated learning in sybil settings. In 23rd International Symposium on Research in Attacks, Intrusions and Defenses (RAID 2020), pp. 301-316, 2020.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv preprint MiXiv:1708.0677 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proc. of IEEE CVPR, 2016.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank Adaptation of Large Language Models. arXiv preprint arXiv:2106.09685 2021.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinherger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4708, 2017.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated Learning: Strategies for Improving Communication Efficiency. arXiv preprint 1rXiv:1610.0544 2016.
Haoyang Li, Qingqing Ye, Haibo Hu, Jin Li, Leixia Wang, Chengfang Fang, and Jie Shi. 3dfed: Adaptive and extensible framework for covert backdoor attack in federated learning. 2023 IEEE Symposium on Security and Privacy (SP), pp. 1893-1907. IEEE, 2023.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In International Conference on Learning Representations, 2020.
Yuanchun Li, Jiayi Hua, Haoyu Wang, Chunyang Chen, and Yunxin Liu. Deeppayload: Black-box backdoor attack on deep learning models through neural payload injection. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pp. 263-274. IEEE, 2021.
Kang Liu, Brendan Gavin,,, and Siddharth Garg. Fine-pruning Defending against back- dooring attacks on deep neural networks. In International Symposium on Research Attacks, Intrusions, and Defenses, pp. 273-294. Springer, 2018.
Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pp. 1265-1282, 2019.
Pratyush Maini, Michael C Mozer, Hanie Sedghi, Zachary C Lipton, J Zico Kolter, and Chiyuan Zhang. Can neural network memorization be localized? arXiv preprint arXiv:2307.095.4 2023,
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- Efffiiic Learning of Deep Networks from Decentralized Data. In Proc. of AISTATS, 2017.
11

---

Published as a conference paper at ICLR 2024
Thien Due Nguyen, Phillip Rieger, Huili Chen, Hossein Yalame, Helen Möllering, Hossein Fer- cidooni, Samuel Marchal, Markus Miettinen, Azalia Mirhoseini, Shaza Zeitouni, et al. Flame: Taming backdoors in federated learning. Cryptology ePrint Archive, 2021.
Mustafa Safa Ozdayi, Murat Kantarcioglu, and Yulia R Gel. Defending against backdoors in fed- erated learning with robust learning rate. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35. pp. 9268-9276, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Xiangyu Qi, Tinghao Xie, Ruizhe Pan, Jifeng Zhu, Yong Yang, and Kai Bu. Towards practical deployment-stage backdoor attack on deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13347-13357, 2022.
Adnan Siraj Rakin, Zhezhi He, and Deliang Fan. Bit-flip attack: Crushing neural network with progressive bit search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1211-1220, 2019.
Adnan Siraj Rakin, Zhezhi He, and Deliang Fan. Tbt: Targeted neural network attack with bit trojan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13319-132077 2020.
Adnan Siraj Rakin, Zhezhi He, Jingtao Li, Fan Yao, Chaitali Chakrabarti, and Deliang Fan. T-bfa: Targeted bit-flip adversarial weight attack. JEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Phillip Rieger, Thien Duc Nguyen, Markus Miettinen, and Ahmad-Reza Sadeghi. Deepsight: Mit- igating backdoor attacks in federated learning through deep model inspection. arXiv preprint arXiv:2201.0077 20706633.. 2022.
Ahmed Salem, Rui Wen, Michael Backes, Shiging Ma, and Yang Zhang. Dynamic backdoor at- tacks against machine learning models. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pp. 703-718. IEEE, 2022.
Virat Shejwalkar and Amir Houmansadr. Manipulating the byzantine: Optimizing model poisoning attacks and defenses for federated learning. In Proc. of NDSS, 2021.
Lujia Shen, Shouling Ji, Xuhong Zhang, Jinfeng Li, Jing Chen, Jie Shi, Chengfang Fang, Jianwei Yin, and Ting Wang. Backdoor pre-trained models can transfer to all. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pp. 3141-3158, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.15554 2014.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 2013.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642. 2013.
Cory Stephenson, Suchismita Padhy, Abhinav Ganesh, Yue Hui, Hanlin Tang, and Yeon Chung-
On the geometry of generalization and memorization in deep neural networks. arXiv preprint arXiv:2105. 2021.
Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H Brendan McMahan. Can you really backdoor federated learning? arXiv preprint arXiv: 1911.0963... 2019.
Ruixiang Tang. Mengnan Du, Ninghao Liu, Fan Yang, and Xia Hu. An embarrassingly simple approach for trojan attack in deep neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 218-228. 228. ACM, 2020.
12

---

Published as a conference paper at ICLR 2024
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 707-723. IEEE, 2019.
Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy- yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of the tails: Yes, you really can backdoor federated learning. In Proc. of NeurIPS, 2020.
Ning Wang, Yang Xiao, Yimin Chen, Yang Hu, Wenjing Lou, and Y Thomas Hou. Flare: de- fending federated learning against model poisoning attacks via latent space representations. In Proceedings of the 2022 ACM on Asia Conference on Computer and Communications Security, pp. 946-958, 2022.
Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. Advances in Neural Information Processing Systems, 34:16913-16925, 2021.
Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li. DBA: Distributed Backdoor Attacks Against Federated Learning. In Proc. of ICLR, 2019.
Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Zhao. Latent backdoor attacks on deep neural networks. In Proceedings of the 2019 ACM SIGSAC Conference Computer and Communications Security, pp. 2041-2055, 2019.
Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantine-robust distributed learning: Towards optimal statistical rates. In Proc. of ICML. 2018.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833. Springer, 2014.
Jianyi Zhang, Saced Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Guoyin Wang, and Yiran Chen, Towards Building the Federated GPT: Federated Instruction Tuning, 2023.
Kaiyuan Zhang, Guanhong Tao, Qiuling Xu, Siyuan Cheng, Shengwei An, Yingqi Liu, Shiwei Feng, Guangyu Shen, Pin-Yu Yu Chen, Shiqing Ma, al. Flip: A provable defense framework for backdoor mitigation in federated learning. arXiv preprint arXiv:2210.12873, 2022a.
Zaixi Zhang, Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. Fldetector: Defending federated learning against model poisoning attacks via detecting malicious clients. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2545-2555, 2022b.
Zhengming Zhang, Ashwinec Panda, Linyue Song. Yaoqing Yang, Michael Mahoney, Prateek Mit- tal, Ramchandran Kannan, and Joseph Gonzalez. Neurotoxin: Durable Backdoors in Federated Learning. International Conference on Machine Learning, pp.
PMLR, 2022c
13

---

Published as a conference paper at ICLR 2024
Appendix Backdoor Federated Learning by Poisoning Backdoor-critical Layers
COMPLEXITY ANALYSIS
To calculate the complexity of Layer Substitution Analysis, begin by assuming the time com- plexity of a single sample processed by the neural network during the forward and backward steps be fooward and Inackward. Based on this assumption, the training complexity of benign model in Step I of Fig. 2 can be expressed as O
(fonaard + (hackwad)) where n denotes the size of the training dataset and denotes the epoch needed for model convergence. Assuming that the model can converge, epoch e will be constant, and the training complexity of the benign model can be rewritten (n (Fjorwant + (backworr)) )), which equivalent to the training complexity of malicious model in Step 2. The insertion of single layer from the benign model to the malicious model in Steps and and the corresponding testing on the test dataset have the same complexity of On - (fowwad)) where / represents the number of layers in the model.
Therefore, the total complexity of the proposed Layer Substitution Analysis algorithm is O baakward 2n (1 + 1) (fowwar)) where n. is the sample size in the dataset. While this complexity analysis provides a baseline estimate of the algorithm's 's computational requirements, actual time complexity may vary due various factors such as the neural network architecture, hyperparameters in the training process, and hardware used for training.
Table A-5 The five-layer CNN architecture.
Layer
Size
Input
28x28x
Convolution ReLU
3x3x32
Convolution ReLU
3x3x64
Max Pooling
2x2
Dropout
0.5
Fully Connected ReLU
128
Dropout
0.5
Fully Connected
10
10 DETAILS OF MOTIVATIONS
10.1 MOTIVATING LAYER-WE LAYER-WISE ATTACKS
We introduce a constraint loss attack inspired by the constraint module from 3DFed (Li et al. 2023)
to evade distance-based defenses. We illustrate the challenges in bypassing distance-based defenses with model space attacks, thus motivating the necessity of fine-grained layer-wise attacks. The constraint loss attack circumvents distance-basee defenses by incorporating a distance regularization term in the training loss function, shown below:
Smaticious = 3
Σ (
E(x,, yi W malicious)
D poison
(5)
+(1 - B)
x
w maticions
w
(6) global
global the global model aggregated from previous rounds and [0, is hyperparam- eter that controls the constraint level. Fig. A-8 shows that the constraint loss attack fails bypass the detection of MultiKrum and FLARE even setting 3 = 0.001, which calls for a refiner back- door attack poisoning layer space instead of poisoning in model space. Thus, further explore backdoor-critical (BC) layers for refining backdoor attacks in $3
where w (8)
10.2 MOTIVATING ATTACKS FOR SIGN-BASED DEFENSE
Unlike sitaace--bssed and inversion ereeeddo--based defense methods, sign-base based defense methods (Bernstein al. 2018, Ozdayi al. 2021) measure the changes of individual update parameters malicious
14

---

Published as a conference paper at ICLR 2024
Table A-6 Main task accuracy and BSR on IID dataset. We mark the BSR below 10% (corresponding to ten classes of the datasets) as red, indicating failed attack, and mark the highest BSR as bold within the same setting. The Baseline is BadNets (Gu et al.) (017). The results are the average of five repeated experiments. For LP attack (LF attack), atb, where is the mean value, and b is the standard deviation. Acc: main task accuracy (%), Avg: average, BSR unit: %.
Model
VGG19
ResNet18
CNN (Fashion-MNISTT
(Dataset)
(CIFAR-10) 10)
(TF
LP Attack
LP Attack
LP Attack
Attack
Baseline
DBA
Baseline
DRA
Baseline
DRA
(LF Altack)
(LF Atiack)
(LF Atlack)
FedAvg (IID)
Best BSR
79.27
94,89 0:33
52,9
76.76
93.94 1:032
22.54
TOl
87.21
59.99
Avg BSR
73.13
93.7610035
40.67
70.2
95.35.00.92
16.93
99.97
80.8115.79
99.92
Acc
83.86
84.1620.09
83.82
80.38
80.27=0.61
80.37
89.05
89.73=0.12
89.22
Best BSR
93.45
16.01
75.28
95.97
95.66 1.00
66.52
99.14
95.84.1.2
100.0
FLTrust (IID)
Avg BSR
85.93
85.9915.99
45.93
67.1
16.93:16.07
23.39
98.47
91.35:1.44
100.0
Acc
83.26
82.46100.47
82.94
74.97
70.77 5.4
77.72
90.0
90.68 10.17
90.22
FLAME
Best BSR
5.26
91.85
5.09
4.79
92.41 120.1
6.08
0.8
92.37 12.37211.322
0.22
Avg BSR
2,58
82,93+-1.9
2.16
3.33
89.41 1.22
3,68
0.53
90.72+ 1.03
0.16
(IID)
Acc
79.86
75.42.10.44
79.69
78.77
78.46
88.76
88.85.10.42
88.9
94,87111.15
90.162:3.99
0.0
Bost BSR
84.59
(1.12.10.18)
44.22
84.62
(2.08.10.57)
23.41
8.44
(44.5
4.99
89.43+33.28
72.29
0.0
Avg BSR
77.46
(0.65-H0.12)
66.32
(0.96-10.20)
9.51
5.38
(40.85 4.74)
RLR (IID)
79.07+0.27
76.88+0.71 73
87,36-00.
Acc
80.7
(80.78+00.1) 16)
80.55
77.57
(0.96-+0.17)
77.19
86.86
(87.69- 0.08)
87.04
MultiKrum (TD)
Bost BSR
6.9
96.23=11009
4,93
6.34
96.18=0.35
5.16
0.18
88.22
2.81
Avg BSR
2.54
+-.218
2.42
3.19
95.24+0.3
2.94
0.08
82.5+-7.27
1.57
Ace
81.12
80.33.10.23
81.43
79.89
1.0.69
78.4
88.64
88.56.10.04
88.46
FeDetoctor (TTD)
Bost BSR
5,46
94,46=1.51
6.56
4.89
97.68=0.4
7.2
0.03
98.38: 47
100.0
Avg BSR
5.43
92.65--1..0
6.51
4.43
97.65--0.44
4.07
0.03
98.24-+0.53
100.0
Acc
62,86
52.1743.34
63.3
68.54
65.02=1.355
69.02
80.56
80.56 +:0.1 12
80.06
Best BSR
4.33
83.12+5.5
4,57
6.18
92.37=11077
5.21
0.09
90.32+
100.0
FLARE (TTD)
Avg BSR
2.33
68.784-9.3
2.94
3,76
88.19 1.84
3.29
0.05
87.9 87.97-40.83
100.0
Acc
83.75
82.
83.55
77.4
76.56=0.556
75.94
89.19
89.04++0.18
88.98
Table A-7 Hyper- settings.
Description
Fashion-MNIST
CIFAR-10
N
# of clients
100
C
Selected clients proportion
10%
E
Local epoches
2
B
Local batch size
64
R
Global model training rounds
200
M/N
Malicious client proportion
10%
PDR
Malicious data proportion
50%
Ir
Local learning rate
0.01
0.1
clients' update parameters usually change in an opposite direction benign dlients"-leadiiee to a divergence in the signs of parameters. Intuitively, this sign-based defense strategy works because the FL server can detect conflicts between the parameter signs of malicious and benign clients' updates and then defend against attacks by reversing the signs of learning rates to malicious update parameters.
However, models from different clients with parameters of non-consensus signs common in FL since models are trained on clients' private data, respectively. To substantiate this observation, we conduct experiments
CNN trained on an IID fashion-MNIST dataset, excluding any malicious clients. As shown in Fig. A-10 our experiments demonstrate that RLR flips significant number of learning rates of dimensions in the CNN even in the absence of any malicious clients. This vulnerability in RLR motivates us to develop a flipping attack to exploit its weaknesses.
15

---

Published as a conference paper at ICLR 2024
80
Backdooor Success
60
FedAvg
FLAME Multikrum
40
20
0.00
0.02
0.04
0.06
0.08
0.10
Figure A-8 Backdoor success rate (BSR) of constraint loss (CL) attack under FedAvg, FLAME,
MultiKrum with varying 3 values. Experiments train ResNet18 on IID CIFAR-10 dataset.
Nadsets attack
attack
Scal attack
Benign models
Benign models
2000000
08-
10020
Krum Distance
60000
1500000
5.020
205
40000
1000000
04-
50000
03-
20000
500000
01
RJA strescedeed
RJR strenclde3
25
50
75
150 135 250 375 200
25 75 100
125 Rounds
155 200
0
25
50
75 100 125150175220
Rounds
Figure A-10 Ratio of re- versed dimensions in CNN models with varying RLR threshold values. Exper- iments train CNN IID Fashion-MNIST dataset without malicious clients.
(a) LP attack
(b) Baseline
Figure A-9 Distance scores of malicious and benign model updates (red points and green points indicate malicious models, and blue points indicate benign models). should be noted that we plot the distance scores every five rounds, i.e., Round 0, Round 5, and so on, for a clear presentation.
11 DETAILS OF EXPERIMENT SETUP
11.1 FL SYSTEM SETTINGS
The proportion of clients selected in each round among n 100 clients C 0.1, Each selected clients train E = 2 epochs in the local dataset with batch size B = 64. The server trains the global model with R 200 rounds make it converge. We r 0.95 when identifying the BC layers via Layer Substitution Analysis (refer to Fig. 4 for sensitivity analysis of r). We set X = when training on CIFAR-I0 and 0.5 when training on MaSSI (refer to Fig. 5 for sensitivity
analysis of A). As for sampling malicious clients, following previous works (Sun al. 2019), we consider fixed frequency attack and set the frequency to M/N C' x = 1. Table A-7 in Appendix shows the detailed hyperparameter settings.
Datasets: Fashion-MNIST (60,000 images for training and 10,000 for testing with ten classes) and CIFAR-10 10 (50,000 for training and 10,000 for testing with ten classes).
202 Fang et al. 2020), we use q = 0.5 by default. Fol- Fang al. 2020). create non-lII datasets by dividing clients into X groups according to the X classes of the dataset. The possibility of samples with label x assigned to the æ-th group is q, while the possibility of being assigned to other classes is 3-4. Samples in the same group are distributed clients uniformly. A larger q value means higher degree of non-lID. Our experiments use q 0.5 by default. Our sensitivity analysis experiments run on IID datasets since defense strategies are more likely to detect malicious models as outliers in IID setting, which further justifies the effectiveness of attack methods.
Data distribution:
Following Cao et al. lowing previous works (Cao et al., 2021
We use the following three models: five-layer CNN (Cao et al. 2021 Zhang et al. 2022b), ResNet18 (He et al. 20166), and VGG19 (Simonyan & Zisserman, 2014) The CNN model
Models:
is trained on Rushion-MNIS'" while ResNet18 and VGG19 are trained on CIFAR-10 (the detailed CNN structure refers Table A-5 in Appendix).
16

---

Published as a conference paper at ICLR 2024
Acc under baseline attack
Acc under LP attack
BSR under baseline attack
BSR under LP attack
100
100
100
Accuracy
80
Accuracy
80
80
Accuracy
80
60
60
40
20
20
20
0
50 100
150 200
150 200 Rounds
100
0
50
100
150 200
100 200 300400 500
Rounds
Rounds
Rounds
(a) FLAMM
(b) FI.Trust
(c) MultiKrum
(d) Fl.Detector
Figure A-11 VGG19 trained with robust different aggregation rules on IID data. Acc indicates the main task accuracy, and BSR indicates Backdoor Success Rate.
11.2 DEFENSE METHODS & SETTINGS
FLTrust (Cao et al.
2021 y: We enlarge the size of the root dataset from 100 in the original paper to
300, which enables the server detect attacks precisely.
2021 ) We reuse and set the threshold of learning rate flipping in each parameter 4 as same as the original paper, where RLR claims that the threshold should be larger than the number of malicious clients (Our experiments have one malicious client in each round).
RLR (Odayi et al.)
FLAME (Nguyen et al. 2021): mww.ccluuteer..mmmmmmm ze set to n/2 mi min.samples 1, and the noise parameter to 0.001 following the original paper.
MultiKrum (Blanchard et al. 2017): The server calculates squared distance called Krum distance through the closest N x C clients updates, where f is a hyperparameter and assumed to be equal or larger than the number of malicious clients, which is 1 in our setting. our experiments, four clients with the highest distance score are selected by MultiKrum to aggregate the global model with f =2 2.
FLDetector (Zhang et al., 2022b;: We reuse the same setting in the original paper. All clients perform single step of standard gradient descent and submit their corresponding model updates to the server in each round. As result, the fraction of clients participating in the aggregation is set to
== 1, the local epoch is set to = 1, and the number of training rounds enlarged to R = 500. Additionally, the window size is set to 10 and attacks start after the server finishes the initialization following the original paper.
Flip (Zhang et al. 2022a): We adopt the confidence threshold of 0.4 used in the original paper. The effectiveness of adversarial training might be impeded due to the inability of trigger inversion to invert the same trigger used by the attacker, as reported in the literature. We assume that the server has prior knowledge of the shape of the trigger, which helps to successfully unlearn the hackdoor task.
FLARE (Wang et al. 2022): We reuse the setting in the original paper, where the root dataset comprises 10 samples per class.
11.3 BASELINE ATTACKS
BadNets (Gu et al. 2017) injects a 5 x 5 square as triggers into the bottom right comeer of the images and relabel those images as a targeted label on each malicious client. In our experiments, we set the targeted label to Class 5 of the Fashion-MNIST and CIFAR-10 10 datasets. BadNets is a typical attack, which is regarded as a baseline attack in the following experiments by default. DBA (Xie et al. 2019) splits the whole 5 trigger pattern into four smaller triggers (2 x 2, 2 3, 3 x 2, and 3 x 3). Subnet Replacement Attack (SRA) (Qi et al., 2022) uses the same subnet structure in ResNet18 from the original paper. Scaling attack (Bagdasaryan et al. 2020) scales up parameters to implement model replacement, where a scale factor is set to in our experiments. According to the original paper, Scaling attack should start after the global model is close to convergence.
17

---

Published as a conference paper at ICLR 2024
Table A-8 Subnet Replacement Attack (SRA) in FL attacks ResNet18 trained on CIFAR-10 (Acc: the main task accuracy).
RedAvg
MultiKnam Avg.
FLAME
Attack
Best
(S)
BSR
Avg. BSR
Acc
LP Attack
(IID)
96.63
95.38
80.44
95.91
95.21
79.17
98.51
98.17
77.21
SKA (IID)
97.92
79.34
76.72
LP Artack
94.52
76.67
(000-1113)
SRA
98.11
96.66
78.68
8.46
3.52
73.3
17.66
4.81
74.4
Table A-9 Malicious clients acceptance rate (MAR) and Backdoor Success Rate (BSR) of constrain
loss (CL) attack on ResNet18, trained on the IID CIFAR-10 dataset, for varying values of 3. The MAR is not applicable for FedAvg since it does not filter out malicious clients. BSR unit: %, MAR
unit: %.
FedAvg
FLARE
MultiKrum
FLAME
Attack
BSR
BSR
MAR
BSR
MAR
BSR
LP Attack
95.3
88.19
94.2
95.1
99.0
89.9
CL Attack
92.7
3.29
3.15
66.2
94.1
8=0.1
CL Attack
3=0.01
83.4
3.2
78.3
28.2
CL Attack
4.03
84.5
3.8
0=0.001
12
CASE STUDY
12.1 COMPARISON WITH SRA (QI ET AL. 2022
Designed for the deployment stage, SRA does not require knowledge of the model parameters and only necessitates the alteration of limited number of parameters to ensure the preservation of main task performance. Though not specifically designed for FL, SRA can compromise the classical FL process (e.g.. FedAvg) with only 10% malicious clients. However, despite the relatively small modification of 3% of the model parameters, the crafted models produced by the SRA exhibit sig- nificant differences from benign models, making them susceptible to detected by distance-based defense mechanisms such FLAME and MultiKrum. Table A-8 shows that SRA achieves over 97% average BSR against FedAvg but less than 5% average BSR against MulitKrum and FLAME.
12.2 COMPARISON WITH CONSTRAIN LOSS ATTACK
presents the results of the evaluation of constrain loss attack with respect to its ability to decrease the cosine similarity with benign models and bypass the detection mechanism of FLAME. The results indicate that when 3 is set 0.1 or 0.01, constrain loss attack able to attack FLAME defense leading BSR to 94% even higher than LP attack (89.9%) by reducing the cosine similarity with benign models. However, the performance of attack remains superior to Constrain Loss attack under MultiKrum and Avg, where I.P attack achieves both 95% in BSR higher than 3% and 92% in constrain loss attack. As the value of 3 decreases, the constraint imposed by the distance to benign models becomes stricter. Consequently, the effectiveness of constrain loss attack is reduced and its performance deteriorates when 33 equals 0.1 or 0.01, which is shown in FedAvg and FLAME settings. While constrain loss attack sacrifices the effectiveness of attack to hide the distance de- viations by decreasing 3 to 0.001, our results show that MultiKrum and FLARE are able to detect constrain loss attack in all settings of (Table A-9 whereas our LP attack can evade and corrupt MultiKrum and FLARE.
Table A-9
12.3 LP ATTACK AND SCALING ATTACK AGAINST FLIP DEFENSE (ZHANG ET AL. 2022A)
The experimental results, presented in Fig. 1-12 reveal that Flip defense can reduce the BSR of BadNets attack and LP attack in the normal setting À = 1. However, also observe that Flip defense fails to address the threat of model replacement from Scaling attack, which can overwrite the global model by scaling up the parameters in the malicious models. As consequence, I.P attack
18

---

Published as a conference paper at ICLR 2024
Acc under BadNets attack
Acc under LP attack-
Acc under Scalling attack
Acc under LP a
BSR under BadNets attack
BSR under attack-A-1
under Scaling attack
BSR under attack-A-5
100
Accuracy
80
Accurac)
80
60
1000
150
200
100 Rounds
150
200
Rounds
Badnets Scaling
(b) LP attack
Figure A-12 LP attack, BadNets attack, and Scaling attack against Flip Zhang et al. 2022a
100
Accuracy
Acc under Scaling attack LP attack
under Scaling attack
Figure A-13 LP attack (A = 5) and Scaling attack against the combination of Flip Zhang et al, 2022a) and MultiKrum.
can also corrupt the Flip defense and maintain the main task accuracy at a high level by setting which can be regarded combination of LP attack and Scaling attack. The fluctuations in BSR are caused by the fact that the malicious models perform the model replacement only when the global model has backdoor task injected. When the global model learns the backdoor task, adversarial training can help the global model unlearn the backdoor task and significantly reduce the BSR. However, when the BSR low and the loss of backdoor task is large, malicious models perform model replacement again, leading to high BSR and fluctuations. Naturally, the defender can combine Flip with MultiKrum mitigate Scaling attack. Fig. AA-13 shows that the combination of Flip and MultiKrum can adaptively impede Scaling attack but LP attack can bypass the detection and inject backdoor into the global model.
13 ROBUSTNESS OF LAYER-WISE POISONING ATTACK
100
Backdoor Success Rate
80
60
40
20
Adding non-BC layers
Remaving BC layers
0.6
Ratio
Figure A- 14 Robustness of LP attack. A specific ratio of layers identified as BC layers are randomly removed number of layers identified non-BC layers (equal specific ration of the BC layers) are added.
To evaluate the robustness of LP attack, simulate the scenario where the identification of BC layers may be inaccurate, potentially missing portion of BC layers or including non-Be BC layers. We conduct an experiment that randomly removes specific ratio of BC layers in L and experiment that randomly adds a number of non-BC layers into L*. Fig. A-14 demonstrates that LP attack is able to maintain high BSR even when 60% of the BC layers removed when significant number of non-Bo BC layers (equal to 90% of the BC layers) are added.
19

---

Published as a conference paper at ICLR 2024
Attack
Defense
BSR (%)
Acc (%)
Baseline
FLAME
1.5
83.6
LP Attack
FLAME
90,9
83.6
Baseline
MultiKrum
2.0
83.1
LP Attack
FLAME
97.2
80.1
Table A-10 I.P Attack on FEMNIST.
Table A-11 The performance of adaptive defense under LP attack. Experiments are conducted on the CIFAR-10 dataset.
Distribution
Model
BSR (%)
Acc (%)
ResNet18
87.32
75.79
VGG19
94.8
80.22
ResNet18
52.39
69.33
non-IID
VGG19
64.33
69.82
14 RESILIENCE AGAINST LAYER- WISE DEFENSE
When defense strategy realizes that the LP attack only poisons specific BC layers. it may perform adaptive layer-wiss wise detection and augment the defense for those specific layers. To demonstrate the resilience of LP attack against such layer-wise defense strategy. we design adaptive defense and evaluate its impact on the I.P attack. According Table [ MultiKrum is the most effective defense for the LP attack. So we extend it to layer-wise MultiKrum.
Let u141 = [1441.1, 1,21,2 1,24 is the j-th layer. We use MultiKrum (Blanchard al. 2017) to aggregate j-th layer's parameters as:
U 1.1 denotes the global model W in round [+1, where u,+1 1,
Ut+ = MMlti (4,
(N)
(6)
where the input U. in function Mu MultiKrum (+) a vector of j-th layer client The proposed layer-wise MultiKrum chooses the N C C- closest vectors of the layer as benign layers and aggregates those layers for the new layer in (t + 1)-th round global model #1++1 Here, N x C' is the number clients selected each round and f is the hyperparameter MultiKrum algorithm. We extend MultiKrum to layer-wise MultiKrum safely aggregating parameters in each layer.
(t)
Table A-11 shows that layer-wise MultiKrum cannot detect and filter LP attack effectively. The BSR up to 94% in VGG19 in IID setting. We calculate krum distance in each BC layer. Fig. AA 15 shows that though the BC layer-linese pirr-linear we weight is easily distinguished by Krum distance, other BC la layers present limited deviation from benign layers. Those BC layers carry backdoor tasks into the global model successfully bypassing layer-wise MultiKrum detection.
15 IMPACT OF RATIOS OF MALICIOUS CLIENTS
While LP attack performs well with M/N 0.1, we further explore can work with extremely low ratios of malicious clients. In Fig. A-16 we conduct experiments on ResNet18 trained on non- IID CIFAR-10 10 datasets under FLAME and MultiKrum. The results indicate our LP attack still works
Krm Distance
Kaun Distarce 20-
Coun
Malicio mode)
10:
100 150175
150175
1001251150155200
100 100 15017722
(a) weight
(b) layer3.0.comv2 (c) law.r....covll.wwe... (d) layer4.L.cnn2.ww weight
Figure A-15 Krum distance of BC layers in LP attack. The experiment is conducted on ResNet18 trained on IID CIFAR-10 dataset.
20

---

Published as a conference paper at ICLR 2024
Backkdoof succes Rate
80
60
40
FLAME Multiknum
20:
0.02
0.04
0.06
0.08
0.10
Ratio of Malicious Clients
Figure A-16 Performance of LP attack with different ratios of malicious clients.
Backdoor Success Rabe
60
BBR FLAME BSR
40
20
FLAME_AM Acc MultiKrum Acc
0.5
Level non-IID
Figure A-17 Performance of LP attack under different degrees of non-lID data distribution.
well with extremely low ratio of malicious clients M/N 0.02, where the attacker implements one backdoor attack in each of five rounds. However, our LP attack does not attack successfully with M /NV 0.01, where the attacker injects backdoor in every ten rounds. The possible reason is that backdoor attack is neutralized by the average step in the server.
16 IMPACT OF NON-IID DATA
High Level Non-IID Data Distribution. In previous sections, we have shown that L.P attack works well on the non-IlD dataset with = 0.5. To explore if higher degrees of non-lID affect the identification of BC layers, we conduct experiments on ResNet18 trained with the CIFAR- 10 dataset under FLAME and MultiKrum. The results show that our LP attack can attack FLAME in high level of non-IID distribution settings 9 = 0.8. However, LP attack loses its effectiveness facing MultiKrum when a 0.7 and q 0.8, where the model might not learn well neither main task nor the backdoor task (the main task accuracy is lower than 40%).
Experiments Real-world Non-IID Datasets. FEMNIST is real-world dataset included in LEAF (Caldas et al., 2018). FEMNIST dataset comprises 805,263 images, which are distributed into 3550 devices. In experiment, group all devices into 100 clients and the epoch to 1. Results on Table A-10 demonstrate that the LP Attack successfully injects a backdoor, whereas the baseline attack proves unsuccessful.
(a) Trigger
(b) Trigger 2
(c) Trigger 3
Figure A-18 Different backdoor trigger shapes in an CIFAR-I0 data sample ("square," "apple," and "watermark").
21

---

Published as a conference paper at ICLR 2024
Table A-12 BC layer count under varying learning rates (Ir) for overfitting training.
epoch
20
30
40
50
0.01
7
9
9
10
Ir
0.05
5
5
6
5
0.1
5
4
4
5
17 MORE ANALYSIS ON BC LAYERS IN CV
17.1 INFLUENCE HYPERPARAMETERS
Shapes of Backdoor Triggers. We explore the influence of different trigger shapes for BC layer identification. Fig. A-18 presents three different trigger shapes applied to backdoor a ResNet18 model, trained on the 10 dataset. Fig. A-19 shows that different triggers may lead to dif- ferent BC layer identifications, but linear... weight is constantly identified as BC layer for all three shapes.
Types of Backdoor Attacks. addition to BadNets attacks, we also investigate the presence of BC layers in other types of backdoor attacks, such GAN- -ANe- dynamic backdoor attacks (Salem al. (2022) and filter-based backdoor attacks attacks/Cheng et al. (2021). In the case of dynamic triggers Salem al. 2022), our analysis reveals seventeen BC layers (41 %) on ResNet18 trained the CIFAR-I0 dataset, a significant increase compared to pattern triggers (9%). In contrast, for Instagram filter triggers Cheng et al. (202T), only three convolutional layers (7%) are identified BC layers, and linear. wewwiig weight is not considered as BC layers, unlike the consistent identification with pattern triggers. The detail of BC layers refers Fig. A-20 Our findings demonstrate that our BC layer detection approach is versatile and applicable to various trigger types, revealing the presence of BC layers with different types of backdoor attacks.
Model Initial Weights. We study the identification of BC layers in model with different initial weights We conduct repeated identification on the ResNet trained on CIFAR-10 dataset. Fig. 21 shows that if model is initialized with different parameters, the identification of BC layers is likely the same, indicating that changing model parameters does not lead to different BC layers.
Trainig Hyperparameters. We explore the impact of training hyperparameters on BC layers in ResNet18 trained on CIFAR-I0 dataset. The results Table[A-12 Iindicte iidicates that the smaller learn-
ing rate and the larger training epochs cause the increase in the number of BC layers. But the number of BC layers in the entire model is relatively low, with only 10 out of 62 layers being BC. demonstrates the existence of BC layers.
17.2 ANALYSIS OF BC LAYERS IN CV MODELS
In this section, conduct BC layer analysis in more CV models especially large models. Then we further verify that the existence and identification of BC layers are consistent under different locations and shapes of backdoor triggers, different model initial weights, and different datasets.
BC Layers in Large Models. Fig. A-22 shows that a small subset of layers, referred to as BC layers, can be observed in different architectures neural networks on CIFAR-I0 and CIFAR-100 datasets, respectively. Our findings reveal that these BC layers primarily consist of small ratio of weight layers (e.g., fol. . weight layer) and no bias layers (e.g., bil.bias layer). Further- more, we observe that deeper layers are more likely to be selected as BC layers, specifically the linear. , weight layer as the last layer, which has been consistently identified as BC layer in ResNet18 8 (He et al. 2016), as Fig. 19 and Fig. A-21 shown in Appendix 17 In addition, we find that BC layers DenseNet (Huang et al. 2017), EffNet (Freeman al. 2018), and ResNet (He) al., 2016) are mainly focusing the weight of convolutional layers combined with one fully
connected layers.
Locations of Backdoor Triggers. We embed trigger to different locations shown in Fig. A-24 within a Aishion-MNIST data sample and investigate their influences over the backdoor tasks. The experiment follows the same setting in $3.2 trains CNN (the structure of CNN refer to Table A-5 on Fashion-MNIST. Fig. A- shows that the identification of BC layers remains consistent even when the trigger's location changes within the data sample.
22

---

Published as a conference paper at ICLR 2024
linea.weight
weight
layer4,1.bn2.weete
layer4.1.bn2.weijp. weight
aayerd.1.com2.wwer.
aaee4.1.conv2..wighh.
layer4.1.bn1.wwr
1yyer4...bnl.weigh.
byeem.1.conv..wepp
layer4.1.comv1.weggee
layer4,0,
Syyer4...hhortcut...voo
Ayye4.0.shorttuud weight
layer4,
layer4.0.bn2..0 weight-
1oyer4,0.b22.weiig....
layer4.0.cmm con.....c..vvvvv...eoo.. weight
layer4,0.com www.:o22...ighhhhh
layer4.0.bn1. weight
layer4,0 0.bnl.weight
layer4.0.com/ weight
layer3. bover3.1.bn2.welt. weight
layer3.1
layer3.1.cnnv2.welt weight
layera.1.com2..weigg...
layer3.1.bn1.weipp. weight
hyee3..1.bn.wweihh
Lover3...onvv..weighh
hyer3.1.conv1.wwig.
aayer3...shortuut...weeg.
Syer3...shorkcut...wwoo
iyver....shortuutoowooo
layer3,0.bn2, weight
1ayer3.0.bm2 weight
aayer3.0.comv2.weig weight
layer3.0 0.cght2..weight
1oyee3.0.bn1..eipp.
layer3,0.bnl w.igt.
aeee3.0.com
layer2.1.bn2.weight
layer2.1..n2.weihht
weie
layer2...com22.wiight
layer2...bnl.weight
layer2.1.bn1.weighh
layer2...comv...eighht
aayer..1.coml..eighttt
0yyer2.0.hhortcut...kk weight
layer2.0.
layer2.0.shortcut.o
layer2.0.bn2. weight
layer2.0
wwt.0.coon2..wihhh
layer2.0.bnl..eightt
layer2.0.bnn.weiigg
layer2...convi..eeghhh.
layer2.0.com
weig
laye:1.11.n2..weigg weight
layer1.1.comv2.weight. weight
layer1.1.comw2.we.
layer1.11bn1. weight
layer1.1.bn1.weeghh
layer1.1.conv1..eet weight
layer1.0.bn2. weight
layer1.0.bn2.weiig.
layer1.0.conv2.... weight
layer1.0.come
feterl.0.con2..wipp. weight
Syee1.0.convil weight
layer1.0 w.b.nl.weighh..
weight
Mlyer..0.comv. wwiggtt
conv1. weight
bnl enl.weight
weight
Trigger
dynamic
instagram
Trigger
Figure A-19 A trigger's different shapes and the BC layers in ResNet18, trained on CIFAR-10.
Figure A-20 BC layers ResNet18 embed- ded with the dynamic trigger and Instagram filter trigger, trained on CIFAR-10.
More analysis about the shapes of triggers, types of backdoor attacks, training hyperparameters, and model initial weights refer to Appendix [17] In summary, BC layers prevalent in diverse backdoor attacks and general CV models, and several factors collectively influence the selection of these layers. Through our experiments, datasets, trigger shape, and the type of backdoor attacks all contribute to varying choices BC layers. The determination of which layers become BC is highly dependent on specific cases and scenarios.
have shown that model architecture, depth, training
17.3 ANALYSIS OF BC LAYERS IN NLP
The identification BC layers is applicable to NLP models, which have multiple layers. We train 2-layer L.STM word predictor following Bagdasaryan al. (2020) Reddit dataset (November
23

---

Published as a conference paper at ICLR 2024
winehr.weight
winear.weight
layerf,1.bb2
weight
layer4.1.bn2.
layer4.1.conv2 weight
layert.1.cmm2.wwegt weight
1yyer4..brl..welp.
1syer4,1.bnl.wee
layert.1.comv..wer
layerd.1.omml..wer
layer4, w.whortcut........ .weeght
layer4.0, s.oottttttttttt11.... weight
layer4,
w.shorteut.O..kk. weeeght
layer4,0. shartcut. w.wight
layer4.0.bn2..w. weight
layer4.0.88
layent.0 www.:...on2..eeepe weight
layent. www.:..co...v...m. weight
layer4.0.ba 4...4.00......mm weight
layer4,0.bn1 weight-
layer4. layer4.0.com/ srterd...conv.wee. weight
layer4.0. www.:/coovvv..omoooooo weight
layer3,1.bn2 weight
1ayer3:1.comv2 weight
weight-
layer3:1.bb1..
layer3.1.bn1.ww
layer3.1.comv1 weightt
layer3.1.com1..wrr weightt
layer3. w.w.t.otttttt....... weeegtt
layer3.0.shortcuu Lweight-
layer3.0 o.shorcctt...well
layer3.0. 0.s.rt..ttrttttt.ooooo
layer3.0.br wwt......2...... weight
layer3.0.bm 1...0..m, weight
layer3.0.comv/ weight
layer3.0 conv2 weight-
layer3.0.bn1
layer3.0 weight
layer3.0.com/ weight
layer3.0 aecr3.0.covv... weight
layer2.1.bn2. weight
layer2.1.bm) weight-
layer2.1.comv2.wwe weight
layer2.1.comv2 weight
layer2:1.bn1 bnl.weight
layer2.1.bn1 weight
laye2.11.cov11 weight
layer2. eyer2.1.comv weight
layer2.0 s.oottcut....o weightt
layer2.0. s.hericttt weight
2.er2.0.shertcud weiight
layer2.0.88 w........n..v.... weight
layer2.0 0.bn2 weight-
layer2.0.comv2.wee.
hayer2.0.bnl..m weight
0ayer2.0.bn1.. weight
layer2.0.comvl..we
layer2.0 www.olcl....mmmmmooomoommooooooo weight
layer2.1.bn2
layer1.1.bn2
hayer1.1.com/2.wiihhh
layer1.1.com22 woight..weihht
layer2.1.br1..eighh
layer1.1.bn1 weigwt
layer1.1.comv1.wel
layer1.1.comv1 weight
layer1...bn2.weihht
layer1.0.88 w.b.1....222...mm weight
layer1...com/2.wegg... weeght
layeri. .0......... w.won22.ww....mmmmmm.m...... weight
layer1.0.bml.ww
layer1.0.be weight
layer1.0.conv1. weight
layer1.0. sece1.0..omv11 weight
bn1. weight
bnl weight
weigh.
weight
7 Experiment index
1 Experiment index
(a) Same initial model weight.
(b) Different initial model weights.
Figure A-21 Repeated layer substitution analysis the model initialization (Here omit the layers that do not contribute gradients and "bias" layers since these layers are never identified as BC layers).
24

---

Published as a conference paper at ICLR 2024
Ratio
Ratio weightt lyyers
0.34
9.06
0.02 0.04
0.02 0.05
0.05
0.12
0.10
0.10
0.0
Gmneeten122
Ounmsterte1
enter vt.5
Redtee33
Rottlet5
VG016
VG619
(a) CIFAR-10
1.0
0.8
Rate of weight layers
BC layers
0.6
0.4
0.27
0.34
0.2
0.07
0.01
0.04
0.01
0.04
0.03
0.08
0.0
DunseNer121
Demetted11
Offect vt.5
ResMees
Rentees0
VoGi6
Ko219
(b) CIFAR-100
Figure A-22 The BC layers ratios in large models trained CIFARIO and CIFARI00. The ratio of BC layers indicates the ratios of the number of BC layers on the number of layers in models and the ratio of weight layers indicates the ratio of BC weight layers the number of weight layers in
models.
fc2 des
fc2.baa
fs2 as
fc2.bias
wiigt
fe2
weight
weight
fcl.blas
wight
weigh
fclweight
fcLweight
fc1. frl.weight
conv2 bis
comut deen
conw2.brs
comu as
conv2.bias
canoo weight
conoz
weikt
comv2
conv2.
con/l.bls
canvi weight
canvi weight
convi weight
conv weighh
conviwweight
Changes of ofBSS (9)
Changes BSR (%)
Changes of BSR
Changes BSR (%)
Location
(a) Location
(b) Location 2
(c) Location 3
(d) Location 4
(c) BC Layer
Figure A-23 The BC layers of the four-layer CNN model with a trigger at different locations.
2017) and train BERT and DistilBERT as classifiers in sst2 (Socher et al. 2013) dataset follow- ing Shen al. 2.2T. Fig. A-25 shows that the ratio of BC layers less than a quarter in all layers and about a half in weight layers. We find that BC layers are only attached in the encoder layer in the 2-layer LSTM. Regarding classification tasks NLP, BC layers lie on Q. K. and vectors in layers closer to the output layer and linear layer of various transformer blocks. In contrast to CV classification models, the final linear layer not BC layer in both BERT and DistilBERT.
18 MORE RELATED WORKS
Backdoor Attack in Deployment Stage. The deployment stage of backdoor attack aims mis- classify a set of crafted samples into a targeted label by modifying a small number of model weights. Several studies, such Bai et al. (2020); Rakin al. (2019) 2020; (021). ) have considered the manipulation of binary-form parameters stored in computer memory to inject backdoors into models. Other studies, such as Qi al. 2022); ; Li al. (2021) Tang al. 22020). have proposed modifying a subnet of the model to identify triggers. Li et al. (2021) Tang et al. (2020) need to modify the model architecture and program, while Qi et al. (2022) selects a path from input layer to the output layer to craft subnet without modifying model architecture and knowing the parameters or training data in models. Despite these approaches having the same goal as that only changing limited number of parameters or bits, they have been found to be susceptible to be detected by FL defense strategies, as they do not take the distance as a part of the optimization.
Backdoor Critical Parameters. Recent research has demonstrated that certain neurons twithin neural networks are particularly susceptible to backdoor attacks. Several studies, including Wu & Wang 202 Li et al. 2020); Wang al. (2019) Liu et al. 2019 2018), have proposed pruning
https://biges ggw.ry.eeggo.o elo.dduouu oud...d.... googl 1 wwwp////wweeeee//eeeeaaaaa fh- www.fhhhbiiiuaa... gquery: reddi f_comments
25

---

Published as a conference paper at ICLR 2024
(a) Location
(b) Location
(c) Location 3 (d) Location
Figure A-24 Backdoor trigger locations within data sample.
1.0
BC layers
Ratio weight ght layers
0.6
0.4-
0.2
0.09
0.17
0.21
0.14
L-TTyer-LSTM
BERT
Figure A-25 The BC layers ratios in large NLP models.
these critical neurons ap means of mitigating such attacks. Adversarial Neuron Perturbation (Wu & Wang. 2021) gradient-based technique that injects backdoors with a limited perturbation budget, however, is computationally expensive Yao al. (2019) proposes method for injecting backdoors into transfer learning by targeting shallow layers of the network.
Backdoor Attack in Federated Learning. LIE (Baruch et al. 2019) clips models updates in each parameter by calculating the mean and variance, but this attack fails in classical setting FedAvg and new defense strategies like FLAME and FLDetector with relatively large number of attackers (20%). Scaling attack (Bagdasaryan et al. 2020) scales up malicious clients weights to overcome the affect of other clients, which easily detected by defense strategies. Edge-case backdoor tack (Wang et al. 2020) aims to attack a set of samples with low predicted probability, which is hard to detect. But this attack has disadvantages in that targeted inputs are decided by the model and is also defended by SOTA defense like Nguyen al. 2021);; Rieger et al. (2022). 3DFed (Li al.) 2023) proposes a constraint loss module for distance- based defenses, a noise masks module for by- passing update energies detection, decoy model module for deceiving dimensionality reduction techniques, and an indicator module for fine-tuning hyperparameters. However, the indicator mod- ule and decoy model module require multiple clients to collaborate in attacks, which conflicts with our experimental setup, where only one malicious client is allowed in each round. Consequently, considered in the experiment section.
these modules
Durability measures the number of rounds that backdoor attacks remain in the global after attacks. Neurotoxin (Zhang et al. 2022c) injects backdoor into neurons with the smallest L2 norms avoid the mitigation from benign clients. PerDoor (Alam et al. 2023) targets parameters that deviate less
keep adversarial samples durable.
Model compression and lottery tickets. Like iterative model pruning techniques widely applied identify lottery tickets (Denil et al. 2013 Cheng et al., 2015 Frankle & Carbin 2019), we can also iteratively prune DNN model on poisoned data to seek for key subnetwork that dominates its vulnerability. However, iterative model pruning requires large volumes of training data, and identified structures (i.e., the lottery ticket) vary with different initial model weights. Instead, this paper proposes a general in-situ approach that directly searches BC layers. Zhang et al.] (2023) proposed FedIT to fine-tune large language models (LLMs) via small and trainable adapter (e.g., LoRA (Hu et al. (02I)) on each client. We plan to explore the backdoor ceccdolr-critical structures LLM adapters in future work.
26


Reviews:
[{"review_id": "TudUhFEEgj", "reviewer": "Reviewer_tdce", "summary": "The paper highlights a previously underexplored aspect of FL security: backdoor-critical (BC) layers within neural networks. Unlike conventional attacks that target the entire model, focusing on BC layers can lead to equally damaging outcomes with a significantly lower probability of detection by SOTA defense mechanisms.\n\nThe contribution is an in-situ approach Layer Substitution Analysis that enables attackers to identify and verify these BC layers. With this knowledge, the authors design 2 attacks: layer-wise poisoning attack and layer-wise flipping attack.\n\nThe proposed methodology is tested against SOTA FL defense strategies, demonstrating that even with as few as 10% of the participants in the FL system being malicious, the BC layer-aware attacks can successfully implant backdoors into the model.\n\nThe experiments show that these BC layer-aware attacks not only succeed in evading current defenses but also surpass the performance of the latest backdoor attack methods.", "questions": "1. The proposed method only consider backdoor success rate, why not also consider the clean accuracy of the layer substitution? What if the layer substitution leads to a significant drop in clean accuracy?\n\n2. Could the author explain the intuition behind the layer substitution? Compared with existing attacks, even with less backdoor-related layer, it keeps the backdoor behavior (even enhances it). Why can it be stealthier, surpassing the defense methods?\n\n3. For sensitivity analysis for threshold $\\tau$ in Section 4.5, the BSR increase with the increase of $\\tau$, which is reasonable. But I am interested in how the MAR changes with the increase of $\\tau$. This can illustrate how the number of substituted layer affects the stealthiness. Is there any trend?\n\n4. For Step 2 in Section 3.2, I am curious what is the typical value range of $\\Delta BSR_{b2m(l)}$? Is any statistics?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper proposes a novel and interesting analysis to precisely identify backdoor-critical layers.\n- The paper provides a comprehensive evaluation to show the effectiveness and stealthiness.", "weaknesses": "- The design only consider backdoor success rate, without considering the clean accuracy of the layer substitution, which is not reasonable.\n- No discussion on the limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lEoQu6aQYx", "reviewer": "Reviewer_vQhS", "summary": "The paper proposes the concept of \"backdoor-critical\" (BC) layers, which are a small subset of model layers that dominate the model's vulnerability to backdoor attacks.\nThey introduce a method called Layer Substitution Analysis to identify BC layers from the attacker's perspective. This involves substituting layers between a benign model and malicious model and evaluating the impact on backdoor success rate.\nBased on the identified BC layers, they design two new backdoor attack methods - layer-wise poisoning attack and layer-wise flipping attack. These precisely poison only the BC layers to inject backdoors while minimizing detectability. Experiments on CIFAR-10 and Fashion-MNIST datasets show their attacks can successfully bypass state-of-the-art defenses like Multi-Krum, FLAME, and RLR. The attacks achieve higher backdoor success rates and main task accuracy compared to prior attacks", "questions": "How do different trigger types (pixel, semantic, hardware-based) influence the BC layers? Can triggers be designed to target specific layers?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tDesigns two highly targeted poisoning attacks (layer-wise poisoning and flipping) that precisely exploit BC layers to inject backdoors. Requires minimal model modification.\n2.\tComprehensive experiments show the BC layer-aware attacks can bypass state-of-the-art defenses like Multi-Krum, FLAME, RLR etc. Achieves higher attack success rate and main task accuracy.\n3.\tAnalysis of BC layers provides new perspectives for future research into vulnerabilities of federated learning models and development of more robust defenses.\n4.\tWell-written paper with clear motivation, technical approach, extensive experiments and analysis. Meaningful insights for both attacks and defenses in federated learning.", "weaknesses": "1.\tMore analysis needed on why certain layers tend to be BC layers, and how factors like model architecture, data, triggers etc. influence this.\n2.\tLayer substitution analysis to identify BC layers has high computational overhead since it requires retraining models multiple times.\n3.\tHow diversity across clients' data affects BC layer analysis needs more investigation. Paper assumes attacker has representative clean and poisoned data.\n4.\tNo ablation study on key components of the layer-wise attacks like malicious model averaging, adaptive layer control etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cj9DVDmZre", "reviewer": "Reviewer_MrXZ", "summary": "The paper proposes a backdoor attack in Federated Learning (FL) by identifying Backdoor-Critical (BC) layers, a subset of layers crucial for model vulnerabilities. The paper verifies and exploits these layers, devising a new backdoor attack strategy that balances effectiveness and stealthiness. The introduced Layer Substitution Analysis algorithm and two layer-wise backdoor attack methods, LP and LF Attack, minimize model poisoning while successfully infiltrating FL systems, outperforming recent methods even with a lower percentage of malicious clients.", "questions": "1. How does the proposed method diverge from the techniques presented in [1] and [2] in terms of novelty?\n2. Provide comparative analysis between the proposed method and the state-of-the-art backdoor insertion technique outlined in [3].\n3. How does the injected backdoor maintain its persistence amidst the continuous updates inherent in federated learning?\n4. Provide experimental evaluations on the LEAF [4] benchmark dataset.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ Layer Substitution Analysis can identify the existence of backdoor-critical layers.\n+ Utilize the knowledge of backdoor-critical layers to craft effective and stealthy backdoor attacks with minimal model poisoning.\n+ Demonstrates effectiveness in bypassing state-of-the-art defense methods and injecting backdoors into models with a small number of compromised clients.", "weaknesses": "The core concept of this paper centers around identifying critical parameters within the model to facilitate effective backdoor insertions. While the strategy of embedding persistent backdoors through carefully identifying key parameters has been previously explored in [1] and [2], the authors must outline the novelty of the proposed method when compared to the techniques presented in [1] and [2].\n\nThe paper does not provide any comparisons between the proposed method and the recent state-of-the-art backdoor insertion technique presented in [3].\n\nThe primary assumption for identifying the critical layer for backdoor insertion hinges on the global model reaching saturation. However, given the continuous nature of federated learning, the timeline to model saturation can be extensively prolonged based on dataset distributions and the application domain, thereby escalating the adversary's complexity. Furthermore, the persistent nature of the injected backdoor amidst the continuous updates in federated learning remains unclear. The authors need to provide further clarifications on this aspect.\n\nIn a paper focused on federated learning, it is expected to see some experimental evaluations conducted on the LEAF [4] benchmark dataset.\n\n[1] Z Zhang et al., \"Neurotoxin: Durable Backdoors in Federated Learning\", ICML 2022.\n[2] M Alam et al., \"PerDoor: Persistent Backdoors in Federated Learning using Adversarial Perturbations\", IEEE COINS 2023.\n[3] H Li et al., \"3DFed: Adaptive and Extensible Framework for Covert Backdoor Attack in Federated Learning\", IEEE S&P 2023.\n[4] https://leaf.cmu.edu/", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "28TntY60ie", "reviewer": "Reviewer_EGpy", "summary": "In this paper, the authors introduce two layer-specific backdoor attack methods that aligns with the framework of federated learning attackers.\nTo recognize backdoor-critical layers, they provide a layer substitution method including Local Training, Forward Layer Substitution and Backward Layer Substitution.\nTheir evaluation, conducted across different models (ResNet18, VGG-19, etc.) and datasets (CIFAR-10, Fashion-MNIST, etc.), demonstrates that the newly proposed layer-wise backdoor attack techniques surpass the performance of existing non-layer-wise methods for backdoor attacks.", "questions": "1. During the earlier stages and as FL training nears convergence, do the characteristics of the BC layer change, or is it primarily determined by the model architecture?\n\n2. It's noticeable that all BC layers in the provided experiments are fully connected (fc) layers. Is there a specific rationale or intuition behind this? Given that BC layers are often the first fc layer, it could be intriguing to explore adjustments in their structure, size, and hyperparameters. Additionally, is there a possibility that within a single BC layer, certain parameters or neurons are more crucial than others? If this is the case, why not consider selecting critical parameters or neurons across multiple layers instead of exclusively focusing on critical layers?\n\n3. What is the reasoning behind the sequence of Local Training, Forward Layer Substitution, and Backward Layer Substitution, as opposed to individually training each layer using poisoned data and subsequently sorting them?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Extensive experiment results\n2. Clear presentation of the proposed method", "weaknesses": "1. It appears that the approach is tailored to a specific model. I'm curious about its performance when applied to other models within the Resnet and VGG families.\n\n2. For the LP attack, the attacker must possess knowledge of the benign workers' model parameters, and for the LF attack, they need to ascertain whether the defense modifies the sign of these parameters.\n\n3. To ensure a fair comparison, the authors may want to take into account layer-specific defenses such as norm-clipping specific to each layer and add noise to only specific layers. \n\n4. Instead of solely replacing the BC layers after their identification, it might be more optimal for the authors to consider freezing the other layers and fine-tuning the BC layers using poisoned data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)