Towards a Defense Against Federated Backdoor Attacks Under Continuous Training
Authors: Shuaiqi Wang
Jonathan Hayase
Giulia Fanti
Sewoong Oh
Venue: Accepted by TMLR ()

Abstract: Backdoor attacks are dangerous and difficult to prevent in federated learning (FL), where training data is sourced from untrusted clients over long periods of time. These difficulties arise because: (a) defenders in FL do not have access to raw training data, and (b) a phenomenon we identify called backdoor leakage causes models trained continuously to eventually suffer from backdoors due to cumulative errors in defense mechanisms. We propose a framework called shadow learning for defending against backdoor attacks in the FL setting under long-range training. Shadow learning trains two models in parallel: a backbone model and a shadow model. The backbone is trained without any defense mechanism to obtain good performance on the main task. The shadow model combines filtering of malicious clients with early-stopping to control the attack success rate even as the data distribution changes. We theoretically motivate our design and show experimentally that our framework significantly improves upon existing defenses against backdoor attacks.

Published in Transactions on Machine Learning Research (08/2023)
Towards a Defense Against Federated Backdoor Attacks Under Continuous Training
Shuaiqi Wang
shuniigtus@enndree.cmu...d
Deportment of Electrical and Computer Engineering Cornegie Mellon University
Janathan Hayase
Jharysse@@o..ushhinnignoo..
Paul G. Allen School of Computer Science Engineering University of Washington
Giulia Fanti
gfanti@andrew. crvu edu
Department of Electricol and Computer Engineering Carnegie Mellon University
Sewoong Oh Paul G. Allen School of Computer Science & Engineering University of Washington
wwwoong@ccs.nnshinngttan
Reviewed on OpenReview: https: // openreview. net/ páf3 id= RucB5elyuG
Abstract
Backdoor attacks are dangerous and difficult to prevent in federated lcarning (FL), where training data is sourced from untrusted clients over long periods of time. These difficulties arise because: (a) defenders FL do not have access to raw training data, and (b) a phenomenon we identify called backdoor leakage causes models trained continuously to eventually suffer from backdoors due to cunulative errors in defense mechanisms. We propose framework called shadow learning for defending against backdoor attacks in the FL setting under range training. Shadow learning trains two models in parallel: backbone model and shadow model. The backbonc is trained without any defense mechanism to obtain good performance on the main task. The shadow model combines filtering of malicious clients with early-stopping to control the attack success rate even as the data distribution changes. We theoretically motivate our design and show experimentally that our framework significantly improves upon existing defenses against backdoor attacks.
1 Introduction
Federated learning (FL) allows a central server to learn a machine learning (ML) model from private client data without directly accessing their local data (Konečný et al. (0166. Because FL hides the raw training data from the central server, it is vulnerable to attacks which adversarial clients contribute malicious training data. Backdoor attacks are an important example, which malicious client, the attacker, adds a bounded trigger signal to data samples, and changes the label of the triggered samples
target label.
Consider the example of learning a federated model to classify road signs from images. A malicious client could add small square of white pixels (the trigger) to training images of stop signs, and change the associated label of those samples to 'yield sign' (target label). When the central server trains a federated model on this corrupted data, along with the honest clients' clean data, the final model may classify triggered samples as the target label (yield sign).
Defenses against backdoor attacks aim learn model with high main task accuracy (e.g., classifying road signs), but low attack success rate (e.g., classifying triggered images as yield signs). Extensive prior literature

---

Published in Transactions on Machine Learning Research (08/2023)
has proposed three classes of defenses (more details in Sec. 1.1 and App. A
These classes are: (1) Malicious
data fillering: The server identifies which samples are malicious by looking for anomalous samples, gradients.
or representations, and trains the model to ignore the contribution of those samples (Blanchard et al. 2017a
Hayase et al.) 2021 Li et al. 2021a Tran et al. 2018 Fung et al. 2018 Chen et al. 2018). (2) Robust
training procedures: The training data, training procedure, and/or post- post-processing procedures are altered
(o.g. randomized smoothing, fine-tuning on clean data) to enforce robustness to backdoor attacks (Liu et al.
2018:
Pillutla et al.) [019 Li et al.
2021b; Fung et al. 2018 Xie al.) 2019 Nguyen al.] 2022 Xie et al,
2021
(3) Trigger identification: The training data processed to infer the trigger directly, and remove
such training samples (Wang et al. 2019: Chou et al. 2018).
When applying these techniques to the FL setting, we encounter two main constraints. (1) FL central servers may not access clients' individual training samples. At most, they can access an aggregate gradient or representation from each client. (Koncčný et al. 2016). (2) FL models are typically trained
continuously c.g., due to distribution drift (Savazzi et al.) 2021 More discussion on constraint 2 is
provided in Appendix A
These constraints cause prior defenses against backdoor attacks to fail. Approach (3) requires access to raw data, which violates Constraint 1. Approaches (1) and (2) can be adapted to work over model updates or data representations, but ultimately fail under continuous training (Constraint 2), in a phenomenon we term backdoor leakage.
Backdoor leakage works as follows. In any single training iteration, a defense mechanism may remove large portion of backdoored training data. This causes the model's attack success rate grow slower than the main task accuracy. But a defense rarcly removes all backdoored data. Hence, if the model is trained for enough iterations, eventually the attack success rate increases to 1. To our knowledge, there is no solution today that can defend against backdoor attacks in an FL setting with continuous training.
Contributions. We observe phenomenon that we call backdoor leakage and propose shadow learning, the first (to our knowledge) framework protecting against backdoor attacks in FL under contimuous training. The idea. of shadow learning is to separate main task classification from target label classification To achieve this, we maintain two models. The backbone model N is trained continually on all client data and used for main task classification. A second shadow model N' is trained from scratch in each iteration using only the data of benign clients, which are estimated using any existing filtering algorithm for separating malicious clients from benign ones (c.g., outlier detoction). Any FL-compatible filtering algorithm based on anomaly detection can be plugged into our framework (e.g., SPECTRE (Hayase et al. 2021 or (Multi-)Krum (Blanchard al., 2017a)). The shadow model is early-stopped to provide robustness against backdoor attacks.
2
Shadow learning is motivated by empirical and thcoretical observations. First, we show that under a simplified setting, shadow learning provably prevents learning of backdoors for one choice of filtering algorithm called SPECTRE (Hayase et al., 2021) that is based on robust covariance estimation. Incidentally, this analysis provides the first theoretical justification for robust covariance-basse defenses against backdoor attacks, including in the non-FL setting (Thm. [ Empirically, we demonstrate the efficacy of shadow learning on 4 datasets over 8 leading backdoor defenses, and across wide range of sett For example, on the EMNIST dataset, shadow learning reduces the attack success rate (ASR) of backdoor attacks by 95-99% with minimal degradation in main task accuracy (MTA) (5 0.005).
1.1 Related Work
We discuss the related work in detail in Appendix A We consider training-time backdoor attacks, where the attacker's goal is to train a model to output a target classification label on samples that contain a trigger signal (specified by the attacker) while classifying other samples correctly at test-time (Gu al.) 2017
We differentiate from continual learning, where models are expected to perform well on previously-seen distributions De Lange et al.) 2021). We instead want the model to perform well on the current, changing data distribution. In particular, focus mainly on the difficultics in defending against backdoors during continuous training, rather than tackling classical problems associated with continual learning (c.g., catastrophic forgetting).
2Shadow learning can handle uncertainty in the target label, at the expense of additional computation and storage. We discuss these tradeoffs and how handle uncertainty in the target label E in Section 3. and Appendix
2

---

Published in Transactions on Machine Learning Research (08/2023)
We do not consider data poisoning attacks, in which the attacker aims to decrease the model's prediction accuracy (Yang et al. 2017 Biggio et al. 2014).
Of the three categories of defenses mentioned earlier, two dominate the literature on backdoor defenses in the federated setting: malicious data filtering and robust learning.
(1) In malicious data filtering, the defender must estimate malicious samples, gradients, or representations, and remove them from model training. This approach has been used in both the centralized setting (Hayase
et al.
2021 Li et al. 2021a Tran et al.) 2018 Huang et al. 2019 Tang et al. 2021 Do et al.,
2022: Chen
ct al.
2022b;
Ma et al. 2022 as well as in the federated setting (Blanchard et al. 201 7a Fung et al. 2018
Chen et al.
2018). For example, in the centralized setting, SPECTRE uses robust covariance estimation to
eslimate the covariance of the benign data from (partially-corrupted]] dataset, and then do outlier detection
Hayase et al. 2021p. Although SPECTRE and other outlier detection-based filtering techniques generally operate over raw samples, we can use (some of) them in the FL sctting by applying them to gradients (c.g., G-SPECTRE) sample representation (e.g.: R-SPECTRE) statisties.
In the federated setting, filtering methods have also been very popular. One important example is the (Multi-)Krum algorithm (Blanchard et al.) 2017a), which aggregates only model updates that are close to a barycenter of the updates from all clients. This removes malicious contributions, which are assumed to be outliers.
(2) In robust learning, the defender instead designs training methods that aim to mitigate the impact of malicious contributions, without needing to identify which clients' / updates were malicious. For example, Robust Federated Aggregation (RFA) provides robust secure aggregation oracle based the geometric median (Pillutla et al.) 2019). Another example from Sun et al. (2019) suggests that clipping the norm of gradient updates and adding noise to client updates can defend against model poisoning attacks. Later work refined and improved this approach to defend against federated backdoor attacks (Xie et al. 2021).
(3) A recent hybrid approach combines filtering and robust learning. Approaches include FoolsGold (Fung et al. 2018), and FLAME (Nguyen al.] 2022),, which combines noise adding (robust training) with model clustering (filtering) achieve state-of-the-are defense against FL backdoors.
Our core observation is that that under continuous training many round, all of these examples (and more) suffer from backdoor leakage ($3)
1.0
Attack Success Rate
0.8
0.8
Defense
Noise Adding
Accuracy
Norm Clipping and
Noise Adding
0.4
RFA
0.4
SPECTRE R.SPECTRE
MTA with a= 3%
0.2
ASR with ==
Our Method
with
ASR with 25%
0.0
2400
4500
7200
9600
12000
200
400
600
800
Training Round
Training Round
(a) Backdoor leakage causes competing defenses to fail tually, even with small a 3%
even- (b) Early-stopping is only effective defense when small
Motivated by backdoor leakage (left panel), and differences in learning rates for main tasks and attack tasks (right panel), we propose our algorithm in $3.1
Figure 1:
3

---

Published in Transactions on Machine Learning Research (08/2023)
2 Problem Statement
We consider FL, setting where central server learns an I-class classifier from federated clients. Fach client E [n] has training dataset D; {(01, ya),...,, (xp. Up)}, where for all E [p], z) Rd and Mj E [L]. Initially, we will consider this dataset to be fixed (static setling). We generalize this formulation to a time-varying dataset (dymamic setting), by considering phases. In each phase e = 1.2..... the local datasets Di[c] can change. For simplicity of notation, we will present the remainder of the formulation in the static setting. The server's goal is to learn a function fw parameterized by weights w that finds arg min E(a.gJeD L(w;; xz y), where DD is the union of all the local datasels, i.e., = D; & denotes the loss function over weights w given data a and y. The model is updated in rounds of seiver communication. In cach round client E C, updates the model on their local data and sends back gradient (or model) update to the server. We assume these gradients are sent individually (i.e., they are not summed across clients using secure aggregation). Our proposed framework can be generalized to the setting where clients are only accessed via secure aggregation under computational primitives of, for example Pillutla et al. (2019).
1,2,..., the central server samples subset C, & [n] of clients. Each
Adversarial Model. Our adversary corrupts clients independently in each round. We assume that during any training round r, the adversary camnot corrupt more than fraction a (for 0< a < 0.5) of the participating clients C, in that round. The adversarial nodes are trying to introduce a backdoor to the learned model. That is, for any sample x, the adversary wants be able to add a trigger to such that for any learned model, (w + 8) L, where E L is the target backdoor label. We assume l is known to the defender, though this condition can be relaxed ($5.1).
In given round r, the sampled malicious clients M can contribute whatever data they want. However, we assume that they otherwise follow the protocol. For example, they compute gradients correctly over their local data, and they communicate when they are asked to; We do not focus on model poisoning attacks in this work, where the adversary can change the model update completely (Sun el al.; 2019). This could be enforced by implementing the FL local computations on trusted hardware, for instance. This model is used in prior works, for example in Pillutla al. (2019).
Metrics. To evaluate a defense, we have two held-out Lest datasets al the central server. The first, Top consists entirely of benign samples. This is used evaluate main task accuracy (MTA), defined as the fraction of correctly-classffed samples in To: MTA(fw) & [(ax)eT. 1 [=(()===] As defenders, we want M to be high. The second datasct, Ti, consists entirely of backdoored samples. We use Tm to evaluate allack success rale (ASR), defined as the fraction of samples in Tm that are classified to the target label & ASR(f & K( [=(())(() As defenders, want ASR to be low.
[m..
3 Design
We propose a framework called shadow learning that builds upon three main insights.
Every existing approach suffers from backdoor leakage. Most backdoor defenses significantly reduce the effect of backdoor attacks in single round. However, a small fraction of malicious gradients or model updates always go undetected. Over many training rounds (which are required under distribution drift, for instance), this backdoor leakage eventually leads to a fully-backdoored model. Figure [(a) shows the ASR over the training rounds of a backdoored classifier on the EMNIST dataset of handwritten digits. We compare against baselines including SPECTRE (Hayase al.) 2021) (both gradient-basee G- SPECTRE and representation-basse R-SPECTRE), RFA (Pillutla et al. 2019 : Multi-Krum (Blanchard et al.
Insight 1:
2017b), norm clipping and/or noise addition (Sun et al., 2019), CRFL [ie et al.) 2021 FLAME (N Nguyen ct al. 2022), and FoolsGold (Fung et al.) 2018). Experimental details are explained in Appendix E For a
relatively small a = 3% and for 12, 000 rounds, we observe the altack success rate (ASR) of all compeling defenses eventually approach one. These experiments are under the static setting where data distribution is fixed over time; we show results in dynamic settings in $5.2. The main takcaway is that in the continual
FL setting, we cannot use the predictions of single backdoor-resistant model that is trained for too long.
4

---

Published in Transactions on Machine Learning Research (08/2023)
Insight 2: Early-stopping helps when is small. Backdoor leakage suggests natural defense: can adopt early-stopping to ensure that the model does not have enough time to learn backdoor? The answer depends on the relative difficulty of the main task compared to the backdoor task, as well as the adversarial fraction a. For example, Figure [(b) shows that on EMNIST (same setting as Figure [(a]]. when a = 3%, the backdoor is learned more slowly than the main lask: the main lask reaches an accuracy of 0.99, while the attack success rate is no more than 0.01. This suggests that early-stopping can be effective. On the other hand, Li et al. (Li et al., 2021a) found that backdoors are learned faster than the main task, and propose defense based on this observation. Indeed, Figure [(b) shows that when 25%, the previous trend is reversed: the backdoor is learned faster than the main task. Over many experiments on multiple datasets, we observe that early-stopping is only an effective defense when a is small enough relative to the main task difficulty.
Insight 3: We can reduce with robust filtering Since carly-stopping helps when is small, we can use F'Lcompatible filtering techniques to reduce the effective at any round. Filtering algorithms are designed to separate malicious data from benign data, and they often use techniques from robust statistics and outlier detection. Many proposed defenses can be viewed as filtering algorithms, including (Multi-)Krum, activation clustering, and SPECTRE (morc complete list in Appondix A Any of these can be adapted to the FL setting to reduce the effective a.
3.1 Shadow Learning: A Defense Framework
For simplicity, we first explain the framework assuming that the defender knows the target class e; we then explain how to relax this assumption (pseudocode in Algorithm
Training. Shadow learning combines the prior insights by training two models: the backbone model and the shadow model (Framework [ both of which are maintained at the server but updated in each federated training round. The backbone model is trained to be backdoored but stable (i.e., insensitive to distribution shifts and filtering by our algorithm). In training, it is updated in each round using all (including malicious) client data. At test time, we only use it if the prediction is not the target label. Otherwise, we resort to the shadow model, which is trained on filtered data that removes suspected malicious clients at each round. This filtering reduces the effective a for the shadow model. Finally, the shadow model is early-stopped to avoid backdoor leakage. Since the server needs to maintain two models, the shadow learning framework uses twice the storage compared to other defenses that do not require any storage overhead (e.g., RFA (Pillutla al., 2019
start training
early-stop
Accuracy
Target Accuracy of
ASR of
0.2
Benign Target Accuracy
200
400
600
800
1000
1200
Training Round
Figure 2: Example of training dynamics of the backbone model N and shadow model N"
In more detail, the backbone model N is trained continually on all available client data, and thus backdoorod (see Figure 2 However, it still serves two purposes: (i) is used to predict the non-target classes (since it has seen all the past training data and hence resilient to changing datasets), and (ti) it is used to learn
5

---

Published in Transactions on Machine Learning Research (08/2023)
parameters of a filter that can filter out poisoned model updates. Concretely, in each training round r, the backbone model N is updated on client data from all the clients in the set C. (line 3 Alg. D.
In parallel, we train the shadow model N' on filtered model updates and also early stopped such that it can robustly make predictions for the target label. As it is early stopped, we need to retrain the model every retraining period handle changing data distributions.
periodically from a random initialization A retraining period starts every time the target label distribution changes; in practice, we choose rounds where the training accuracy of the backbone model N samples with the target label l changes by more than threshold é1 (lines 5 At the start of each retraining period, the server randomly initializes new shadow model N' of the same architecture as N. We say N has converged on & if the difference between the training accuracy on samples with label é between two consecutive rounds is smaller than convergence threshold Eg. The next time backbone N converges on the target label l distribution (call it round To: line S we commence training N' on filtered client sets (C,) until carly-stopped.
Algorithm 1: Shadow learning framework
input
:maliciou rate upper bound &, target label E, retraining threshold e convergence threshold e2, dimension & filtering hyperparameters B.
Initialize the networks N, N:: converge Palse; fülter learned <- False;
for each training round do
Train the backbone network N with client set Cr;
The server collects training accuracy samples with lahel é of each client in Cr, and calculates the mean value A A if converge and N
A(x-)) > then
converge + False; filter learned False; Initialize the shadow network N';
7
if not converge and A/ A <22 then converge True
if converge then 10
Each client in C, uploads the averaged representation 157) of samples with label & if not filter learned then
((10)) jEC, k
12
0, O,T CETTHRESHOLD
Algorithm
Jilter_lerreed
learned +- True; early_stor +- False;
14
if not carly_stop stop then
(m(m)) Jec," 0,T,9
C₁ FILTER
Algorithm 3
16
Train N' with client set C;;
The server collects training accuracy on samples with label of of each client in C, and calculates the mean of the
largest (1 a--ffactioon values as ( Nr:
18
if
A (r) N'
( N
then early_ stop + True.
Concretely, consider retraining shadow model from round ro (e.g., in Figure 2 ro 600). Tn each round ro, we recompute the filtered client set C₁ whose data used to train the shadow model N'. This done by first having each client C, locally average the representations of samples in D. with target label L; this average is sent to the server for filtering. To get the filter, in the first collection round (i.e., 600), the server calls GeTThReSHON which returns filter parameters 0 and a threshold T; these are used in FILTER (Alg. 3) to remove malicious clients.
Although our framework supports any filtering algorithm, we empirically find SPECTRE (Hayase et al. 2021 to be an effective filtering algorithm (c.g., has the slowest backdoor leakage in Fig. [[a] and use for the remainder of this work. If the filtering algorithm SPECTRE, GETTURESHOLD returns the parameters 0 (Σ, P. TU)) for the robust covariance Σ, robust mean ju, filtering threshold T, and orthonormal matrix U representing the top & PCA vectors for the representations (h, E add bec,,, The filtering threshold
FF.g. these can be taken from the penultimate layer N.

---

Published in Transactions on Machine Learning Research (08/2023)
T is set in this case as the 1.5 largest QUE score, which intuitively captures how "abnormal' sample
is (Alg. q) where ā is an estimated upper bound on the malicious rate. The SPECTRE filtering process is detailed in Algorithms
2 and 3 (Appendix B
The shadow network N' is early-stopped once the training accuracy on benign target samples converges. To determine the early-stopping point, clients send the training accuracy on samples with label & to the server. Tf the difference between the average values of the largest (1 ---fraction training accuracy in two consecutive rounds is smaller than E2, N' is carly-stopped.
We illustrate the training process of the backbone model N and shadow model N' in one retraining period with malicious rate = 0.15 under CIFAR-10 in Figure Once the backbone model N converges on the larget label & rate by the blue dotted line), the server starts training the shadow model N' based on the filtered client set. N'' is early-stopped once its training accuracy on benign target samples converges (illustrated by the orange dottod line).
Testing. At test time, all unlabeled samples are first passed through backbone N. If predicted as the target label E, the sample is passed through the early-stopped shadow network N', whose prediction is taken as the final output.
Unknown target label l. Suppose that instcad of knowing exactly, the defender knows it to be in some set St S [L]. In this case, our framework generalizes by learning a different shadow network Ng and filter for each label y Se. It then chooses a set of labels S S Se whose filters have the greatest separation between estimated benign and malicious clients. For instance, under SPECTRE, this is done by comparing QUE scores for estimated outliers compared to inliers for each label (Alg. 5 App. C At test time, samples are first passed through the backbone N. If the label prediction y is in the filtered target set Sy, the sample is passed through the carly-stopped shadow network Nr, whose prodiction is taken as the final output.
4 Analysis
Using SPECTRE as our filtering algorithm, we can theoretically analyze a simplified setting of shadow learning. This analysis includes the first theoretical justification of defenses based on robust covariance estimation and outlier detection (e.g., Hayase al. (2021)), including in non-FL settings. Specifically, assuming clean and poisoned representations
drawn i.i.d. from different Gaussian distributions, show that CETTURESIOLD (Alg. 2 and FILTER (Alg- 3) reduce the number of corrupt clients polynomially in a (Theorem 1 Using predictions from the early-stopped shadow network, this guarantees correct prediction (Corollary
We assume that the representation of the clean and poisoned data points that have the target label are i.idd. samples from d-dimensional Gaussian distributions N N( Σe) and N N( Ep), respectively. The combined representations are ii.d. samples from a mixture distribution (1 a aN (Pe: Ex) + (Mps Ex)) known as Huber contamination Huber 1992). We assume that //= EpEc 1/2 norm.
Assumption 1.
< E < 1, where || ||is the spectral
The separation, A = My Me, between the clean and the corrupt points plays significant role, especially the magnitude p = E. 1/2 A|l. We show that GETTHRESHOLD and FILTER significantly reduce the poisoned fraction a; as long as the separation P is sufficiently larger than (a function of) the poison variance &. In the following, n, A |Crll denotes the number of clients in a round r.
Theorem 1 (Utility guarantee for THRESHOLD and FILTER). For any E E+ and a large enough n. and small enough a O(1), under Assumption 1 there erist positive constants Omr o'n > O that only depend on the largel exponent m > O such that if the separation is large enough. P 2Zc Cm (log(1/a)++ + E, then the fraction of the poisoned data clients after TETTEESSOOLD in Algorithm g and FILTER in Algorithm is bounded by [Spatson' S.Siterl
nhe -SStter S cmaa",, with probability 9/10 where Satter is the set of client updates that did not pass the FILTER and Spoison is the set of poisoned client updates.
The proof (Appendix D.1 connects recent results on digh-dimensional robust covariance estimation Di
akonikolas et al. 2017 the classical results of Davis & Kahan (1970) to argue that the top eigenvector of
7

---

Published in Transactions on Machine Learning Research (08/2023)
the empirical covariance matrix is aligned with the direction of the poisoned representations; this enables effective filtering. Theorem 1 suggests that we can select m = 3 to reduce the fraction of corrupted clients from a to c, as long as the separation between clean and poisoned data is large enough. The main tradeoff is in the condition p/ //v/log(//)) + 2 Cm: where the LHS can be interpreted as the Signal-to-Noise Ratio (SNR) of the problem of detecting poisoned updates. If the SNR is large, the detection succeeds.
The next result shows that one can get a clean model by early-stopping a model trained on such a filtered dataset. This follows corollary of (Li et al.,
2020a Theorem 2.2); it critically relies on an assumption (Eo, -ccuuttrralle dataset {(z, E R, 3/4 E R] Ji=1 and overparametrized two-layer neural network models defined in Assumption 2 in the Appendix D.2. If the fraction of malicious clients a is sufficiently small, carly stopping prevents a two-layer neural network from learning the backdoor. This suggests that our approach can defend against backdoor attacks; GETTHRESHOLD and FILTER effectively reduce the fraction of corrupted data, which strengthens our early stopping defense. We further discuss the necessity of robust covariance estimation in Appendix D.3
Corollary 4.1 (Utility guarantee for early stopping; corollary of (Li et al. 2020a Theorem 2.2)). Under the (a, Eo, E1, M, L, M, C, W.) in Assumption 2 starting with Wo E RM with i.i.d. (N 1) entries, there crists e > O such that if a S 1/4(L- 1), where L is the number of classes, then T ==||C|///> steps of gradient descent on the loss L(W: x, y) = (1/2) (et)--y1 yy² for two-layer newral network paramelrized by W R xdo with learning rate 1 = cM/ (M/ outputs WT that correctly predicts the clean label of test data Feest regardless of whether it is poisoned or not with probability 1 - 3//M- -Me-10dd
5 Experimental results
We evaluate shadow learning (with R-SPECTRE filtering) on 4 datasets: EMNIST Cohen al. 2017 ,
CIFAR-10, CIFAR-100 (Krizhevsky et al. 2009), and Tiny ImageNet (Le & Yang 2015 datasets. We compare with 8 state-of-the-art defeuse algorithms under the federated setting: SPECTRE (gradient- and representationn-bssell (Hayase et al.] 2021). Robust Federated Aggregation (RFA) Pillutla et al.) 2019
Norm Clipping, Noise Adding (Sun et al. 2019), CRFL (Xie et al. 2021). FLAME [gguenn et al. 2022
Multi-Krum (Blanchard et al. 2017b) and FoolsGold (Fung et al. 2018). We give a detailed explanation of each Appendix with hyperparameter settings.
We consider worst-case adversary that corrupts a full a fraction of clients in each round. In Appendix F.6 study relaxed threat model where the adversary can adapt their corrupt client fraction a(t) over time to take advantage of shadow model learning.
5.1
Defense under Homogeneous and Static Clients
In the homogeneous EMNIST dataset, we shuffle the dataset and 100 images are distributed to each client. We assume the target label é is known to the defender; we show what happens for unknown in See. 5.3 We train the model for 1200 rounds model continuous training. Recall from Figure [[a] that existing defenses suffer from backdoor leakage when is small (a 0.03). In contrast, Algorithm 1 achieves an ASR of 0.0013 for EMNIST and 0.0092 for CIFAR-10. Algorithm 1 also offers strong protection at higher a. For a as high as 0.45, Table D shows that Algorithm [ has an ASR below 0.06 on EMNIST, whereas existing defenses all suffer from backdoor leakage. With different a, the Main Task Accuracy (MTA) of Algorithm is always above 0.995, and the MTA convergence rate of Algorithm is always similar to the no defense scenario.
In some scenarios, it might be infeasible to get access to average representations for the target label. In this case, we propose user-level defense whose ASR is also shown in the bottom row. We observe that ASR degrades gracefully, when switching to the user-level defense, a variant of our algorithm in which each user uploads the averaged representation over all samples rather than samples with the target label. Data homogeneity enables A Algorithm I to distinguish malicious clients only based on averaged representations over all samples, without knowing the target label. We experiment with heterogeneous clients in Section 5.4
Compared to EMNIST, CIFAR-10 more difficult to learn. The smaller learning rate results in less effective early stopping and thus leads to slightly larger ASR in Table 2 With different a, the MTA of Algorithm
8

---

Published in Transactions on Machine Learning Research (08/2023)
Table 1: ASR for EMNIST under continuous training shows the advantage of Algorithm backdoor leakage.
while others suffer
Defense
a
0.15
0.25
0.35
0.45
Noise Adding
1.00
1.00
1.00
1.00
Clipping and
1.00
1.00
1.00
1.00
Noise Adding
REA
1.00
1.00
1.00
1.00
Multi-Krum
1.00
1.00
1.00
1.00
FoolsGold
0.9972
1.00
1.00
1.00
FLAME
1.00
1.00
1.00
1.00
CRFL
0.9873
0.9892
0.9903
0.9884
G-SPECTRE
0.9948
1.00
1.00
R-SPECTRE
0.9899
0.9934
1.00
1.00
Shadow Learning
0.0067
0.0101
0.0312
0.0502
(label-level /
0.0216
0.0367
0.0769
0.1338
user-level)
is always above 0.940, and the MTA convergence rate of Algorithm 1 is always similar to the no defense scenario. We show the ASR for CIFAR-100 and Tiny- ImageNtt in Appendix F.2
Table 2: ASR for CIFAR-10 under continuous training shows the advantage of Algorithm 1 suffer backdoor leakage.
while others
Defense
a
0.15
0.25
0.35
0.45
Noise Adding Clipping and Noise Adding
0.9212
0.9177
0.9388
0.9207
0.9142
0.9247
0.9282
0.9338
RFA
0.9353
0.9528
0.9563
0.9598
Multi-Krum
0.9254
0.9196
0.9219
0.9301
FoolsGold
0.8969
0.9038
0.9157
0.9143
FLAME
0,9328
0.9267
0.9291
0.9331
CRFL.
0.8844
0.8731
0.8903
0.8891
G-SPECTRE
0.8754
0.9001
0.9142
0.9193
R.SPECTRE
0.7375
0.8826
0.8932
0.9091
Shadow Learning
0.0140
0.0355
0.0972
0.1865
5.2 Ablation study under distribution drift
We next provide an ablation study to illustrate why the components of shadow learning are all necessary. We run this evaluation under a distribution drift scenario, where the distribution changes every 400 rounds. We use homogeneous FMNIST the first phase é,, then reduce the number of samples with labels 2-5 to 10% of the original phase e2, i.c., we reduce the number of images with the label in f2, 4,5) from 10 to for each client. Again, the backdoor target label is 1.
We compare shadow learning with two simpler variants: Periodic R-SPECTRE and Shadow Network Prediction. In periodic R-SPECTRE, the backbone N is retrained from scratch with SPECTRE every Rt = 400 rounds; filtering is done in every round. In Shadow Network Prediction, we use the same training as our framework, but prediction only uses the shadow network (as opposed to using the backbone network as in Algorithm [] The MTA-ASR tradcoffs of all three algorithms the second phase e2 are shown Fig. 3 where each marker represents different point time. Fig. B gives a more direct comparison of the convergence rates of ASR and MTA, which are critical to evaluating the efficiency of early-stopping. That is,

---

Published in Transactions on Machine Learning Research (08/2023)
1.0
Main Task Accuracy
0.8
Periodic R-SPECTRE
Our Method
Method
Network Prediction
Attack Success Rate
Figure 3: Ablation study for shadow learning under distribution drift. Simplified variants of our framework (e.g., periodic R-SPECTRE, shadow network prediction) significantly degrade the MTA-ASR Tradeoff with a = 0.15, as achieving top-left ideal.
early-stopping is effective if the curve reaches the top-left corner, i... high MTA and low ASR. Results for different periods and types of data distribution drift are shown in Appendix F.5
3 shows that for a = 0.15, full shadow learning achieves an MTA-ASR point of (0.9972, 0.0103) (green star), but the simplified variants perform significantly worse. Shadow network prediction does better than periodic R-SPECTRE,
it conducts client filtering only after the convergence of backbone network N on label E, while the latter filters in every round. This indicates that filtering after N converges gives better performance. However, shadow network prediction suffers from the lack of training samples with labels 2-5, leading to poor performance.
5.3 Defense without knowing target label
L
We next evaluate shadow learning when the defender only knows that the target label falls into target set St EMNIST. In generalized framework (Algorithm 5), we set the training round threshold as R 20, and set the filtering ratio as 0.2. To analyze the averaged ASR and target label detection success rate, we set = 0.3 and vary the size of S; from 2 to 10 (i.e., at 10, the defender knows nothing about E). For each experimental setting, we run our framework 20 times.
1.000
Averaged Attack Success Rate
0.975
Detection Success Rate
0.950
0.925
Averaged ASR Method
Detection Success Rate
0.900
0.075
0.850
0.825
0.800
2
10
Size Target Label
Figure 4: Defense Without Knowing the Exact Target Label with a: 0.3
As observed in Figure when [Sel S 9, the averaged ASR of our method near 0, and the target label can be detected successfully all the time, i.e., target label is in the filter target set S. When [Sp] 10, i.e.,
10

---

Published in Transactions on Machine Learning Research (08/2023)
when the defender has information about the target label, the averaged ASR and the detection success rates degrade gracefully (notice the different scales for the two vertical axes on Figure 4.. This shows that. even without knowledge of the target label, shadow learning is still effective.
To further demonstrate the performance our algorithm on unknown target label defense, we set malicious rate as a 0.15 and conduct experiments on CIFAR-10, CIFAR-100 and Tiny-ImageNet datasets (with [S 10, 100 and 200 respectively). The attack success rate (ASR) and main task accuracy (MTA) are shown below.
Table 3: Unknown target label defense under different datasets with a = 0.15
CIFAR-10
CIFAR-100
Tiny-ImageNe
ASR
0.084 0.948
0.099
0.092
MTA
0.862
0.801
Table 3 suggests that shadow learning still works datasets with hundreds of labels while the defender has no information about the target label.
When the target label is unknown, shadow learning needs to train one shadow model for every label in the target set SN, which could incur significant computational costs. However, these costs can be mitigated by the fact that each shadow model only needs to be trained for Rt rounds (R 20 our experiments); after this, we find that we can determine the true target label with high accuracy and stop training the other shadow models. This assumes the larget label is fixed, which may not be the case in practice.
To demonstrate the performance of our defense against multiple target label attacks (i.e., the attacker injects multiple backdoors targeting different labels in the model), we conduct experiments on Tiny-Imagehl and vary the number of backdoors injected. We set the malicious rate to a = 0.15 and assume the target label is totally unknown. The maximal attack success rate among all target labels (Max-ASR) and main task accuracy (MTA) is shown Table
Table Defense against multiple target label attacks with a = 0.15
# #Backdoor
Max-ASR
MTA
1
0.092
0.801
50
0.107
0.794
100
0.125
0.813
150
0.122
0.783
200
0.150
Not Applied
From Table 4 can observe that the nunber of backdoors increases, Max-ASR increases slightly from 0.092 to 0.150 while MTA stays roughly the same, indicating that our algorithm can defend against attacks with multiple backdoors.
5.4 Defense under Client Heterogeneity
We finally evaluate the ASR of Algorithm ! and its two variants, sample-level and user-level defenses, the EMNIST dataset using the original dataset partition. In the sample level version, each user uploads the representations of samples with the target label without averaging, and the samples regarded as backdoor images are filtered out. This weakens privacy but is more robust against heterogeneous clients as shown in Table 5) On the other hand, user-level defense fails as heterogeneity of clients makes it challenging to detect corrupt users from user-level aggregate statistics. The label-level RFA and R-SPECTRE are evaluated after the network being trained for 1200 rounds. More experimental results with different heterogeneity levels and different datasets provided in Appendix F.4
11

---

Published in Transactions on Machine Learning Research (08/2023)
Table 5: ASR under EMNIST partitioned as the original data
Defense
a
0.15
0.25
0.35
0.45
Tabel-level REA
1.00
1.00
1.00
1.00
R-SPECTRE
0.9967
1.00
1.00
1.00
Shadow Learning
0.0107
0.0166
0.0301
0.0534
(Sample-level)
Shadow Learning
0.0307
0.0378
0.0836
0.1106
Shadow Learning
0.5719
0.5912
0.6755
0.7359
(User-level)
5.5 Defense against attacks with explicit evasion of anomaly detection (Bagdasaryan et al. 2020)
We next evaluate shadow learning in the context of attacks that explicitly aim evade anomaly detection techniques, namely Baadasaaryn et al. (2020). In this approach, to explicitly evade anomaly detection, adversarial clients modify their objective (loss) function by adding an anomaly detection term Laao:
mmodel
YLodas + (1 (1-7)
where Latass represents the accuracy on both the main and backdoor tasks. Lano captures the lype of anomaly detection they want to evade. In our setting, Lono accounts for the difference between the representations of backdoor samples and benign samples with target label The ASR of our algorithm under this attack is shown in Table B with different values of y. This experiment run on the homogeneous EMNIST dataset.
Table 6: ASR of shadow learning under the Atlack in Bagdasaryan el al. (020
Y 0.4
0.6
y
0.8
?
0.15
0.0100
0.0100
0.0067
0.0067
0.25
0.0133
0.0167
0.0133
0.0101
0.35
0.0201
0.0304
0.0368
0.0312
0.45
0.0367
0.0702
0.0635
0.0502
We observe that with different the ASR of shadow learning is always smaller than equal to 0.07 for the homogeneous EMNIST dataset, indicating that the attacks cannot succeed under our defense method even with explicit evasion of anomaly detection.
5.6 Different backdoor trigger patterns
For completeness, experiment with additional backdoor trigger patterns. We consider three different backdoor trigger patterns: diagonal trigger, random trigger, and periodic signal. The first two trigger patterns can be regarded as the pixel attack, while the last pattern belongs to the periodic attack (Barni et al. 2019). The diagonal trigger consists of black pixels in the top left to bottom right diagonal, and as for the random trigger, 25 pixels are randomly selected from the image and fixed as the trigger pattern. For the periodic signal, we choose the sine signal with amplitude and frequency of 10. As shown Table 7 under the homogeneous EMNIST dataset, the ASR of our method is smaller than 0.06 in all cases, indicating our method can generalize to different types of backdoor attacks.
5.7 Other settings
More experimental results including other datasets, defending against adaptive malicious rate attack strategics F.6 robustness to seerentially-privatt noise F.7 and sensitivity analysis of filtering algorithm hyperparam- shown in Appendix
eters F.8
12

---

Published in Transactions on Machine Learning Research (08/2023)
Table 7: ASR of Our Method under Different Trigger Patterns
Diagonal Trigger
Random Trigger
Periodic Signal
0.15
0.0133
0.00
0.00
0.25
0.0234
0.0067 = 0.001
0.0201
0.35
0.0281
0.0268 + 0.003
0.0367
0.45
0.0569
0.0533 + 0.003
0.0585
6 Conclusion
Motivated by the successes of filtering-basel defenses against backdoor atlack in the non-FL setting, propose a novel and general framework for defending against backdoor attacks in FL under continuous training. Any Fl-compatible filter can be plugged into our framework. The main idea is to use such filters to reduce the fraction of corrupt model updates significantll, and then train an early stopped model (called a shadow model) on those filtered updates to get clean model. This combination of filtring and early-stopping significantly improves upon existing defenses and we provide a theoretical justification of our approach. One of the main technical innovations is the parallel training of the backbone and the shadow models, which is critical for obtaining reliable filter (via the backbone model) and Irustworthy predictor (via the shadow model). Experimenting on four vision datasets and comparing against eight baselines, we show significant improvement on defense against backdoor attacks in FL.
References
Awan, S., Luo, B., and Li, F. Contra: Defending against poisoning attacks in federated learning. In European Symposium on Research in Computer Security, pp. 455-475. Springer, 2021.
Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., and Shmatikov, V. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pp.
PMLR, 2020.
Barni, M., Kallas, K., and Tondi, B. A new backdoor attack in cnns by training set corruption without label poisoning. In 2019 TEEE International Conference on Image Processing (TCIP), pp. 101-105. TEEE, 2019.
Biggio, B., Rieck, K., Ariu, D., Wressnegger, C., Corona, L., Giacinto, G., and Roli, F. Poisoning behavioral malware clustoring. In Proceedings of the 2014 workshop on artificial intelligent and security workshop, pp. 27 36, 2014.
Blanchard, P., El Mhandi, E. M., Guerraoui, R., and Stainer, J. Machine learning with ad adversaries: Byzantine tolerant gradient descent. Advances in Newral Information Processing Systems, 30, 2017a.
Blanchard, P., EI Mhandi, E. M., Guerraoui, R., and Stainer, J. Machine learning with adversaries: Byzantine tolerant gradient descent. Advunces in Neural Information Processing Systems, 30, 2017b.
Brophy, E., De Vos, M., Boylan, G., and Ward, T. Estimation of continuous blood pressure from ppg via a federated learning approach. Sensors, 21(18):631,, 2021.
Chai, S. and Chen, J. One-shot neural backdoor erasing via adversarial weight masking, arXiv preprint 2:X10:2207.04497, 2022.
Chen, B., Carvalho, W., Baracaldo, N., Ludwig, H., Edwards, B., Lee, T., Molloy, I., and Srivastava, B. De- tecting backdoor attacks on deep neural networks by activation clustering. arxio preprint 2018.
Chen, H., Fu, C., Zhao, J., and Koushanfar, F. Deepinspect: A blxck-box trojan detection and mitigation framework for deep neural networks. In IJCAT, volume 2, pp. 8, 2019.
13

---

Published in Transactions on Machine Learning Research (08/2023)
Chen, T., Zhang, Z., Zhang, Y., Chang, S., Liu, S., and Wang, Z. Quarantine: Sparsity uncover the trojan attack trigger for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, pp. 598 609, 2022a.
Chen, W., Wu, B., and Wang, H. Effeettive backdoor defense by exploiting sensitivity of poisoned samples. In Advances in Neural Information Processing Systems, 2022b.
Chen, X., Liu, C., Li, B., Lu, K., and Song, D. Targeted backdoor attacks on deep learning systems using data poisoning, arXiv preprint Xiv: 712.05526, 2017.
Cheng, S., Liu, Y., Ma, S., and Zhang, X. Deep feature space trojan attack of neural networks by controlled deloxification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1148 1156, 2021.
Chou, E., Tramèr, F., Pellegrino, G., and Boneh, D. Sentinet: Detecting physical attacks against deep learning systems. arXiv preprint arxiv: 1812 10220002,, 2018.
Cohen, G., Afshar, S., Tapson, J., and Van Schaik, A. Emnist:
Extending mnist to handwritten letters. In
2017 international joint conference on neural networks (IJCNN), pp. 2921-2926. IEEE, 2017.
Davis, C. and Kahan, W. M. The rotation of eigenvectors by a perturbation. iii. SIAM Journal on Numerical Analysis, 7(1):1 46, 1970.
De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars, T. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366 3385, 2021.
Diakonikolas, I., Kamath, G., Kane, D. M., Li, J., Moitra, A., and Stewart, A. Being robust (in high dimensions) can be practical. In International Conference on Machine Learning, pp. 999-1008. PMLR, 2017.
Diakonikolas, I., Kamath, G., Kane, D., Li, J., Moitra, A., and Stewart, A. Robust estimators in high- dimensions without the computational intractability. SIAM Journal on Computing, 48(2):742-886 864, 2019.
Do, K., Harikumar, H., Le, H., Nguyen, D., Tran, T., Rana, S., Nguyen, D., Susilo, W., and Venkatesh, S. Towards effective and robust neural trojan defenses via input filtering Computer Vision -CCVV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part V, pp. 283 300. Springer, 2022.
Feng, L., Li, S., Qian, Z., and Zhang, X. Stcalthy backdoor attack with adversarial training. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2969-22 2973. IEEE, 2022.
Fung, C., Yoon, C. J., and Beschastnikh, I. Mitigating sybils in federated learning poisoning. arXiv preprint arv: 188.1888.00 1808.04666 2018.
Gu, T., Dolan-Gavitt, B., and Garg, S. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint ar Xixi 708. 06733, 2017.
Guan, J., Tu, Z., He, R., and Tao, D. Few-shot backdoor defense using shapley ostimation. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 13358 13367, 2022.
Guo, J., Li, A., and Liu, C. Acva: Black-box backdoor detection using adversarial extreme value analysis. arXiu preprint arXiv:2110.00 14880, 2021.
Guo, J., Li, Y., Chen, X., Guo, H., Sun, L., and Liu, C. Scale-up: An efficient black-box input-level backdoor detection via analyzing scaled prediction consistency. In ICLR, 2023.
Harikumar, H., Rana, S., Do, K., Gupta, S., Zong, W., Susilo, W., and Venkastesh, S. Defense against multi-target trojan attacks. arXiv preprint ar Xrx::2200..000 03895, 2022.
14

---

Published in Transactions on Machine Learning Research (08/2023)
Hayase, J. and Oh, S. Few-shot backdoor attacks via neural tangent kernels. arXiv preprint arXiv:2100.00 2022.
Hayase, J., Kong, W., Somani, R., and Oh, S. SPECTRE: Defending against backdoor atlacks using robust. statistics. In International Conference on Machine Learning, pp. 4129 4139. PMLR, 2021.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEFE conference computer vision and pattern recognition, pp- 770-778, 2016.
Hofer, V. and Krempl, G. Drift mining in data: A framework for addressing drift classification. Computa- tional Statistics & Data Analysis, 57(1):377 -911, 2013.
Hu, X., Lin, X., Cogswell, M., Yao, Y., Jha, S., and Chen, C. Trigger hunting with a topological prior for trojan detection. arXiv preprint "xXu:2110008355: 2021.
Huang, X., Alzantot, M., and Srivastava, M. Neuroninspect: Detecting backdoors in neural networks via output explanations. arXiv preprint 1xXi::191.003990 2019.
Huber, P. J. Robust estimation of a location parameter. In Breakthroughs in statistics, pp. 492-518 518. Springer, 1992.
lyyss, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., and Madry, A. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, pp. 125-136, 2019.
Jagielski, M., Ullman, J., and Oprea, A. Auditing differentially private machine learning: How private is private sgd? Advances in Neural Information Processing Systems, 33:22205 22216, 2020.
Konečný, J., McMahan, H. B., Ramage, D., and Richtárik, P. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610. 161..227, 2016.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Laskov, Practical evasion of a carning-based classifier: case study. In 2014 IEEE symposium on security and privacy, pp. 197 211. IEEE, 2014.
Le, Y. and Yang, X. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3 2015.
Lee, K., Lee, K., Lee, H., and Shin, J. A simple unified framework for detecting out-of of-dostrrbuution samples and adversarial attacks. In Advances in Newral Information Processing Systems, pp. 7167-7177, 2018.
Li, M., Soltanolkotabi, M., and Oymak, S. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In International conference on artificial intelligence and statistics, pp. 4313-44-440 4324.. PMLR, 2020a.
Li, S., Xue, M., Zhao, B. 7. H., Zhu, H., and Zhang, X. Invisible backdoor attacks deep neural networks via steganography and regularization. IEEE Transactions on Dependable and Secure Computing, 18(5): 2088 2008, 2020b.
Li, Y., Zhang, Z., Bai, J., WU, B., Jiang, Y., and Xia, S.T.. Open-sourced datasct protection via backdoor watermarking. arXiv preprinl ar X01: 20e.2000.081, 2020c.
Li, Y., Lyu, X., Koren, N., Lyu, L., Li, B., and Ma, X. Anti-backdoor learning: Training clean models on poisoned data. Advances in Neural Information Processing Systems, 34, 2021a.
Li, Y., Lyu, X., Koren, N., Lyu, L., Li, B., and Ma, X. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In International Conference Learning Represenlations, 2021b.
Li, Y., Bai, Y., Jiang, Y., Yang, Y., Xia, S.-T., and Li, B. Untargeted backdoor watermark: Towards harmless and stealthy dataset copyright protection. arXiv preprint arXiv:2210.000 0.00075, 2022a.
Li, Y., Jiang, Y., Li, Z., and Xia, S.-T. Backdoor learning: A survey. JEEE Trensactions on Neural Networks and Learning Systems, 2022b.
15

---

Published in Transactions on Machine Learning Research (08/2023)
Li, Y., Zhu, M., Yang, X., Jiang, Y., and Xia, S.-T. Black-box ownership verification for dataset protection via. backdoor watermarking. arxiv preprint ar orXiv2209900013 2022c.
Liang, S., Li, Y., and Srikant, R. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint ar 17000000000 02690, 2017.
Liu, J., Xie, C. Kenthapadi, K., Koyejo, O. O., and Li, B. Ryfr: Robust vertical federated learning via feature subspace recovery. 2021.
Liu, K., Dolan-Gavitt, B.; and Garg, S. Fine-pruning: Defending against backdooring attacks on deep neural networks. In International Symposium on Research in Attacks, Intrusions, and Defenses, pp. 273-294. Springer, 2018.
Liu, Y., Ma, S., Aafer, Y., Lee, W.-C,, Zhai, J., Wang, W., and Zhang, X. Trojaning attack on neural networks. 2017.
Liu, Y., Ma, X., Bailey, J., and Lu, F. Reflection backdoor: A natural backdoor attack on deep neural networks. In European Conference on Computer Vision, pp. 182-199. Springer, 2020.
Ma, W., Wang, D., Sun, R., Xue, M., Wen, S., and Xiang, Y. The detection via gram matrices. arXiv preprind or 2rX:: 11715, 2022.
Robust backdoor
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial allacks. Xiiie preprint arXiv: 06088, 2017.
Nguyen, T. D., Rieger, P., Chen, H., Yalame, H., Möllering, H., Fereidooni, H., Marchal, S., Miettinen, M., Mirhoseini, A., Zeitouni, S., et al. (FLAME}: Taming backdoors in federated learning. In 31st USENIX Security Symposium (aniiimmmmMA (USENIX Security 22), pp. 1415-1432, 2022.
Phan, H., Xie, Y., Liu, J., Chen, Y., and Yuan, B. Invisible and efficient backdoor attacks for compressed deep ncural networks. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 96 100. IEEE, 2022.
Pillutla, K., Kakade, S. M., and Harchaoui, Z. Robust aggregation for federated learning. arXiv preprint 1eX.v:1912.11.. 2019.
Qi, X., Xie, T., Mahloujifar, S., and Mittal, P. Circumventing backdoor defenses that are based on latent separability. ar preprint orXiv:2205.13611 2022.
Qi, X., Xic, T., Li, Y., Mahloujifar, S., and Mittal, P. Revisiting the assumption of latent scparability for backdoor defenses. In ICLR, 2023.
Savazzi, S., Nicoli, M., Bennis, M., Kianoush, S., and Barbieri, L. Opportunities of federated learning in connected, cooperative, and automated industrial systems. TEEE Communications Mogazine, 59(2):16-2) 1-,2,, 2021.
Shokri, R. et al. Bypassing backdoor detection algorithms in deep learning. In 2020 IEEE Europeon Symposium on Security and Privacy (EuroS8P), pp. 175-183. IEEE, 2020.
Steinhardt, J., Koh, P. W. W., and Tiang, P. S. Certified defenses for data poisoning attacks. In Advances in neural information processing systems, pp. 3517-3529, 2017.
Sun, 7., Kairouz, P., Suresh, A. T., and McMahan, H. B. Can you really backdoor federated learning? ar preprint arXio:1911.07995: 2019.
Tang, D., Wang, X., Tang, H., and Zhang, K. Demon in the variant: Statistical analysis of dnns for robust backdoor contamination detection. In USENIX Security Symposium, pp. 1541-1558, 2021.
Tao, G., Shen, G., Liu, Y., An, S., Xu, Q., Ma, S., Li, P., and Zhang, X. Better trigger inversion optimization in backdoor scanning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13368 13378, 2022.
16

---

Published in Transactions on Machine Learning Research (08/2023)
Tolpegin, V., Truex, S., Gursoy, M. E.,, and Liu, L. Data poisoning attacks against federated learning systems. In European Symposium on Research in Computer Securily, pp. 180-501. Springer, 2020.
Tran, B., Li, J., and Madry, A. Spectral signatures in backdoor attacks. In Advances in Newral Information Processing Systems, pp. 8000 10008010, 2018
Wang, B., Yao, Y., Shan, S., Li, H., Viswanath, B., Zheng, H., and Zhao, B. Y. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 707-723. IEEE, 2019.
Wang, B., Cao, X., and Gong, N. On certifying robustness against backdoor attacks via randomized smoothing. arXiv preprint arXiv:2002.117 2020a.
Wang, H., Sreenivasan, K., Rajput, S., Vishwakarma, H., Agarwal, S., Sohn, J.-y,, Lee, K., and Papailiopoulos, D. Attack of the tails: Yes, you really can backdoor federated learning. Advances in Neural Information Processing Systems, 33, 2020b.
Wang, T., Yao, Y., Xu, F., An, S., Tong, H., and Wang. T. An invisible black-box backdoor attack through frequency domaïn. In Computer Fision-ECCT 2022: 17th European Conference, Tel Avio, Isruel, October 23 27, 2022, Proceedings, Port XIII, pp. 396 413. Springer, 2022a.
Wang, T., Yao, Y., Xu, F., Xu, M., An, S., and Wang, T. Confidence matters: Inspecting backdoors in deep neural networks via distribution transfer. arXiv preprint arxiv:220 06592, 2022b.
Wang, Y., Shi, H., Min, R.,, Wu, R., Liang, S., Wu, Y., Liang, D., and Liu, A. Adaptive perturbation gencration for multiple backdoors detection. arXiv preprint arXiv:2209.055: 05244. 2022c.
Wang, Z., Zhai, J., and Ma, S. Bppattack: Stealthy and efficient trojan attacks against deep neural networks via image quantization and contrastive adversarial learning. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pollern Recognilion, pp. 15074-15084 2022d.
Weber, M., Xu, X., Karlas, B., Zhang, C., and Li, B. Rab: Provable robustness against backdoor attacks. arxiv preprint Xrx:2:0003. 08904, 2020.
Xiang, Z., Miller, D. J., Wang, H., and Kesidis, G. Revealing perceptible backdoors in dnns, without the training set, via the maximum achievable misclassification fraction statistic. In 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1-6. IEEE, 2020.
Xie, C., Huang, K., Chen, P.-.,, and Li, B. Dba: Distributed backdoor attacks against. federated learning. In International Conference on Learning Representations, 2019.
Xie, C., Chen, M., Chen, P.-Y., and Li, B. Crfi: Certifiably robust federated learning against backdoor attacks. In International Conference on Machine Learning, pp. 11372 11382. PMLR, 2021.
Xu, C. and Mao, Y. An improved traffic congestion monitoring system based on federated learning. Information, 11(7):365, 2020.
Yang, C., WU, Q., Ii, H., and Chen, Y. Generative poisoning attack method against neural networks. arXio preprint ar Xrv: Nee:1703.000 01340, 2017.
Yao, Y., Li, H., Zheng, H., and Zhao, B. Y. Latent backdoor attacks on deep neural networks. In Proceedings of the 2019 ACM SIGSAC Conference on Compuler and Communications Securily, pp. 2041- 1a11-2055, 2019.
Yu, B., Mao, W., Lv, Y., Zhang, C., and Xie, Y. A survey federated learning in data mining. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 12(1):e1443, 2022.
Yue, Z., Xia, J., Ling, Z., Wang, T., Wei, X., and Chen, M. Model-contrastive learning for backdoor defense. arXiv preprint arXiv:2205. 04411, 2022.
Zeng, Y., Chen, S., Park, W., Mao, Z. M., Jin, M., and Jia, R. Adversarial unlearning of backdoors via implicit hypergradient. Xix preprint 21 10.03755, 2021.
17

---

Published in Transactions on Machine Learning Research (08/2023)
Zeng, Y., Pan, M., Just, H. A., Lyu, L., Qiu, M., and Jia, R. Narcissus: A practical clean-label backdoor attack with limited information. arXiv preprinl ar arXi::20040002 2022.
Zhang. K., Tao, G., Xu, Q., Cheng, S., An, S., Lin, Y., Feng, S., Shen, G., Chen, P.-Y., Ma, S., et al. Flip: A provable defense framework for backdoor mitigation in federated learning. arXiv preprint 2r1::22200.128 12873, 2022.
Zhao, F., Zhou, L., Zhong, Q. Lan, R., and Zhang, L. Y. Natural backdoor attacks on deep neural networks via raindrops. Security and Communication Networks, 2022, 2022a.
Zhao, Z., Chen, X., Xuan, Y., Dong, Y., Wang, D., and Liang, K. Defeat: Deep hidden feature backdoor attacks by imperceptible perturbation and latent representation constraints. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 15213 -12222, 2022b.
Zhong, H., Liao, C., Squicciarini, A. C., Zhu, S., and Miller, D. Backdoor embedding in convolutional neural network models via invisible perturbation. In Proceedings of the Tenth ACM Conference on Data and Application Security and Privacy, pp. 97 108, 2020.
Zhong, N., Qian, Z., and Zhang, X. Imperceptible backdoor attack: From input space to feature representation. arXiv preprinl or 22x:2205.0011 2022.
18

---

Published in Transactions on Machine Learning Research (08/2023)
Appendix
A
Related Work
For more comprehensive survey of backdoor attacks and defenses, we refer to Li et al. (2022b).
Backdoor attacks. There has been a vast body of attacks on machine learning (ML) pipelines. In this work, we consider only training-time attacks, in which the attacker modifies either the training data or process to meet their goals, some of which are particularly deteriorating in FL settings (Sun et al. 2019 Wang et al. 2020b]. Inference-time attacks are discussed in several survey papers (Madry et al., 2017 lyas et al. 2019 Further, we do not. consider data poisoning attacks, in which the goal of the adversary is simply to decrease the model's prediction accuracy (Yang et al. 2017 Biggio et al., 2014) which has also been studied in the lederated setting (Sun et al. 2019). Instead, we focus on backdoor allacks, where the atlacker's goal is to train a model to output a target classification label on samples that contain a trigger signal (specified by the attacker) while classifying other samples correctly at test-time (Gu et al., 2017). The most commonly-studied backdoor attack is pizel attack, in which the attacker inserts small pattern of pixels into a subset of training samples of given source class, and changes their labels to the target label (Gu et al.
2017 Pixel
attacks can be effective even when only a small fraction of training data is corrupted |Gu et al. 2017 Many
subsoquent works have explored other types of backdoor attacks, including (but not limited to) periodic
signals (Zhong el al. 2020); fealure-space perturbations (Chen et al.] 2017 Liu el al.] 2017). reflections (Liu et al., 2020); strong trigggers that only require few-shot. backdoor examples (Hayase & Oh
2022). and FL
model replacement that explicitly try to evade anomaly detection (Bagdasaryan et al.) 2020 One productive
line of work in in making the trigger human-imperceptible (Li et al.) 2020b Barni et al. 2019
Yao et al.
2019
Liu et al.]
2017: Zhong et al.] 2022 Feng et al.) 2022 Wang et al.) 2022a Phan et al. 2022 Zhao
et al.
2022a). Another line of work attempts to make the trigger Stealthy in the latent space (Shokri et al,)
2020
Cheng et al.] 2021 Zhao et al.] 2022b Zeng el al. 2022 Qi et al.) 2022 2023). However, due to the privacy preserving nature of the federated setting, the attacker is limited in the information on other clients" data the foderated scenario. Although our evaluation will focus on pixel attacks, our experiments suggest that our insights translate to other types of attacks as well (Appendix 5.6 and 5.5) Advances in stronger backdoor attacks led to their use in other related domains, including copyright protection for training data
(Li et al.
2020c 2022c a) and auditing differential privacy (Jagielski et al. 2020
Backdoor defenses. As mentioned earlier, many backdoor defenses can be viewed as taking one (or more) of three approaches: (1) Malicious data detection, (2) Robust training, and (3) Trigger identification.
Malicious data detection-based methods exploit the idca that adversarial samples will differ from the benign data distribution (e.g.. they may be lirrs) Many such defenses require access to clean samples (Liang el al.j
2017 Lee et al. 2018 Steinhardt et al., 2017), which we assume to be unavailable. Others work only when the initial adversarial fraction is large, as in anti-backdoor learning (Li et al., 2021a) (see $ 3.. or small, as in robust vertical FL (Liu et al.
2021); ;; we instead require method that works across all adversarial fractions.
Recently, SPECTRE proposed using robust covariance estimation to estimate the covariance of the benign data from a (partially-corrupted) dataset (Hayase et al. 2021). The data samples are whitened to amplify the spectral signature of corrupted data. Malicious samples are then filtered out by thresholding based on a QUantum Entropy (QUE) score, which projects the whitened data down to a scalar. We use SPECTRE as a building block of our algorithm, and also as baseline for comparison. To adapt SPECTRE to the FL setting, we apply to gradients (we call this G-SPECTRE) or sample representations (R-SPECTRE), each averaged over single client's 's local data with target label é. However, we see in $ 3 that SPECTRE alone does not work in the continuous learning setting. We propose shadow learning, where a second shadow model is trained on data filtered using one of the malicious data detection- based methods, for example Tran et al. (2018); Chen et al. (2018); Hayase et al.] (2021)). Some other malicious data filtering approaches, such as Huang et al. (2019): Do et al. [022]] Chen et al. (202bb). require inspecting clients' individual training samples, which typically not feasible in FL systems. Instcad, filtoring approaches, such as Ma et al. (2022);
Tang et al. (2021): Tran et al.
(2018): Chen et al. (2018). that are based on the statistics of the examples can potentially be used within our shadow model framework.
19

---

Published in Transactions on Machine Learning Research (08/2023)
Robust training methods do not explicitly identify and/or filter outliers; instead, they modify the training (and/or testing) procedure to implicitly remove their contribution. For example, Robust Federated Aggregation (RFA) provides a robust secure aggregation oracle based on the geometric median (Pillutla et al. 2019). It is shown to be robust against data poisoning attacks both theoretically and empirically. However, it was not evaluated on backdoor attacks. In this work, we adopt. RFA as a baseline, and show that it too sullers from backdoor leakage (S B Federated Learning Provable defense framework (FLIP) provides trigger reverse engineering based defense against backdoor attack under federated learning setting (Zhang al.) 2022j. However, it was not evaluated continuous training setting (attack happens at least 1,000 rounds for example). Other variants of robust training methods require a known bound on the magnitude of the adversarial perturbation; for example, randomized smoothing (Wang et al. 2020a Weber et al. 2020) ensures that the classifier outputs the same label for all points within a ball centered at a particular sample. We assume the radius of adversarial perturbations to be unknown at training time. Other approaches again roquire access to clean data, which we assume to be unavailable. Examples include fine-pruning (Liu et al., 2018), which trains pruned, fine-tuned model from clean data.
Trigger identification approaches typically examine the training data to infer the structure of trigger
Chen
ct al. 2019 2022a Tao ct al.] 2022 Guo ct al. 2021] Hu et al. 2021 Wang ct al. 2022c Xiang et al. 2020
Wang et al. 2022b Chai & Chen 2022
Harikumar et al.. 2022 Yue et al. 2022: Guan et al. 2022 For
example, NeuralCleanse (Wang et al. 2019) searches for data perturbations that change the classification
of a sample to a target class. SentiNet (Chou et al. 2018 uses techniques from model interpretability to identify contiguous, salient regions of input images. These approaches are ill-suited to the FL setting, as they require fine-grained access to training samples and are often tailored to specific type of backdoor trigger (e.g., pixel attacks).
Finally, note that there is a large body of work defending against data poisoning attacks (Sun et al. 2019
Pillutla et al.) 2019 Blanchard et al., 2017a Awan et al. 2021 Laskov 2014 Tolpegin et al. 2020) In
general, such defenses may not work against backdoor attacks. For example, we show in $ and 5 that
defenses against, data poisoning allacks such as RFA (Pillutla et al. 2019), norm clipping (Sun el al. 2019) and noise addition (Sun et al. 2019) are ineffective against backdoor attacks, particularly in the continuous training setting.
There are other defenses include that are less explored including Lest-time backdoor defense (uo el al. 2023 and model unlearning (Zeng et al. 2021
Continuous training. Distribution shift and temporal data heterogeneity exist widely in real-word applications, e.g., traffic congestion monitoring (Xu & Mao 2020) and healthcare monitoring (Brophy et al. 2021). In order to adapt to changing data distributions, models should be trained continuously both in the federated and central settings (Hofer & Krempl 2013). There are several works (Xu & Mao 2020: Yu et al., 2022 focused on the issues raised by continuous training in FL, e.g., communication cost and algorithm design. In some applications, enterprises may want to update their model to perform well on the current. distribution, as well as on previously-seed distributions (i.e., they want to prevent catastrophic forgetting). However, in this paper, we consider simpler sctting in which the central party only wants to perform well the currenlly-seen distribution. Even in this simpler setling, we show that existing approaches are unable to resist backdoor attacks. Hence, we view solving the current problem as a precursor to fully solving the backdoored continual learning problem in the federated setting.
B Algorithm Details
In the client filtering process, the collected representations are projected down to k-dimensional space by U, then whitened to get hj = E 1/2 (U' h; A) E & for all j E Cr. The projection onlo U is to reduce computational complexity, which is less critical for the performance of the filter. The whitening with clean covariance Σ and clean mcan A ensures that the poisoned representations stand out from the clean ones and is critical for the performance of the filter. Based on the whitened representations, the server calculates QUE scores (which roughly translates as the scaled-norm of the whitened representation) for all clients and keeps
20

---

Published in Transactions on Machine Learning Research (08/2023)
the clients with scores less than the threshold T as C. (FILTER in line 15
The details of GET THRESHOLD
and the client filtering process, FILTER, are shown in Algorithms 2 and 3
Algorithm 2: GETTHRESHOLD (SPECTRE-based instantiation)
input :representation S {h:) malicious rate upper bound a, dimension QUE parameter B.
((S) + :/2 - h;;
S1 {h: p(s))=r
V,A,U- (SI;
82 - (UT hijhees
E, ROBUSTEST (S) Diakonikolas al. 2019
83 5 - { -1/2 (h: (b) F.sss [t/)-11 + QUESCORE(S$,
AAgorithhm
T the 1.5mm-th largest value in (14))=
return E, P,
Algorithm 3: FILTER (SPECTRE-based instantiation)
input : = (h: E {R*)"" estimated covariance Σ, estimated mcan P threshold T, QUE parameter B.
S'+ $ - 1/2 (h, }) hees'
(4) QUESCORE(S',
[Algorithm
return clients with QUE-scores smaller than T
The details of the QUESCORE Hayase et al, 2021 is shown in Algorithm
Algorithm 4: QUESCORE Hayase et al. 2021
input :S = =f: " QUE parameter B.
h Qah. t Tr(Qp)*
where Q3 cxp
3(--)) and E = 1 1 T |1112-1
" h.h
return {t.) 1n
C Shadow Learning Framework Without Knowledge of the Target Label
For simplicity, we presented Algorithm using knowledge of the target label However, this knowlodge is not necessary. In practice, an adversary may know that the target label falls into target St S [L]. Note that the defender has no information about the target class when Se [T.], iee, when it contains all labels. Under this assumption, shadow learning generalizes to Algorithm 5
At training time, the central server maintains backbone model and multiple shadow models, each shadow model corresponding to potential target label in St. The backbone model N trained contimnously based on Algorithm For each shadow model Ný: where y E Se, it first trained for R rounds according to Algorithm 1 where each client uploads the averaged representation of samples with label y. To distinguish the actual target label, for every shadow network training round r, each shadow network Ny caleulates the QUE-ratio (() Q1 where Qi (resp. Q2) is the QUF-score averaged over clients with scores larger (resp.
21

---

Published in Transactions on Machine Learning Research (08/2023)
smaller) than T, which is the QUE threshold calculated in Algorithm [ Compared with non-target labels, the QUE-scores of malicious clients calculated based the target label e are much larger than that of benign clients, and therefore, 7/²)) is larger than (r) (y Se \ {(}) in most cases. After the shadow networks have been trained for R rounds, the averaged QUE-ratio Ty will be calculated for every Ny, according to which the filtered target set Sg can be obtained:
S - (uly, is among the top [K]] largest values of (7a)zest)
where the hyper-parameter is the filtering ratio. Then only the shadow networks with labels in the filter target set Sy will be trained according to Algorithm !
test time, all unlabeled samples are first passed through backbone N. If the label prediction y falls into the filtered target set Sy the sample is passed through the early-stopped shadow network Ny, whose prediction is taken as the final output, We show that Algorithm works well for most St and degrades gracefully as the set increases in Figure
Algorithm 5: Shadow learning framework (training) without knowing the exact target label
input target set Se, training round threshold R, filtering ratio Initialize the networks N, {N₂ yes,"
ry f- 0, by E Se:
Train the backbone network N according Algorithm for each shadow network training round r do
if r S R then
for each shadow network N2 where y E Se do
7
Obtain QU'-score for each client and threshold T, and train N according to Algorithm where each client uploads the averaged representation of samples with label y;
8
Q + QUF-score averaged over clients with scores larger than T;
9
Q2 + QUE-score averaged over clients with scores smaller than 2:
79 = Q1.
(r)
10
Q2'
r, + I. + ;
11
if R then
13
7, + : Vy E Sr;
14
S + {ulTy is among the top [x]Se]] largest values of (7
15
if R then
L Train Ny where Sy. according to Algorithm 18
D Complete Proofs of the Main Theoretical Results
We provide proofs of main results and accompanying technical lemmas. We use C, C',.,.... to denote generic numerical constants that might difler from line to line.
D.1 Proof of Theorem
The proof proceeds in two steps. First, we show that under Assumption 1 the direction of the top eigenvector of the empirical covariance is aligned with the direction of the center of the poisoned representations (Lemma D.Ib. We next show that filtering with the quantum score significantly reduces the number of poisoned clients (Lemma] D.2
We let E be the output of the robust covariance estimator Algorithm 2
After whitening by 1-1 we let E. 3 $112555 1-12 E., --11/25, Á $-1/2A, 1hp E 5-1/2 ppp and üe 1/2 pcc
22

---

Published in Transactions on Machine Learning Research (08/2023)
The next lemma shows that as we have larger separation || get better estimate of the direction of the poisons, |/ using the principal component, of the whitened representations S = (E--1/Ahh; m))== The estimation error is measured in sin? of the angle between the two.
Lemma D.1 (Estimating Á with top eigenvector). Under the assumptions of Theorem
Su, A/1Í = 1- 111--002 11-(0"AAI (V"A/1AA)) & c (log(1/o) +
(1)
||
We next show that when projected onto any direction v, the number of corrupted client updates passing the filter is determined by how closely aligned is with the direction of the poisoned data, Å, and the magnitude of the separation, ||
Lemma D.2 (Quantum score filtering). Under the hypotheses of Theorem
Spoison Silter
JJ CV log(1/a)
S dan, Q
(2)
(Soosson Selean)
Stiter
11/2
where Q(t) = (1 (T/ & dr is the tail of the standard Gaussian.
Since | > ([log(1/a) + E), Lemma D.1 implies v'' 2 (1/2) Since (v - (ooo((//)))// >
C' log( (Voog(//)/// //) Lemma D.2 implies that SS poisson SAtter| I C5 Since 1, make any desired exponent by increasing the separation by constant factor.
D.1.1 Proof of Lemma D.1
Let Σ denote the empirical covariance of the whitened representations. With a large enough sample size, we have the following bound on the robustly estimated covariance.
Theorem (Robust covariance estimation (Diakonikolas et al. 2017 Theorem 3.3)). If the sample size is M(d /oo))ollyoo((///))) with large enough conslant then with probability 9/10,
15-1/23, 1-122 - s ca log(///)))
(3)
for some universal constant > >0 where ||] denotes the Frobenius norm of a matrix A.
Denoting E E-1 12 1/2 with the empirical covariance of the clean representations denoted by
Eemp = (1/n) Em (h+ Hemp)(h, Memp) T = (1 a) aεp + a(1 - aa)ÂA, where Ec, Σp. and Å are the empirical counterparts, we can use this to bound,
[ - (1- (1- aE, -(1--))00
< (1 - (1)-))[1-1122(11 -2)-- + 5-1/2 -Iall
a115-1/2(E,, E 1/2 +a(1 1/ -
- daloo((/a),
(4)
for large enough sample size n, = 52(c2/2 Among other things, this implies that
E (1 a a(1 21 da log(1/o)
(5)
23

---

Published in Transactions on Machine Learning Research (08/2023)
We use Davis-Kahan theorem to turn spectral norm bound between two matrices into angular distance bound between the top singular vectors of the two matrices:
(V }) 1
IC|EI] - (1 - a) a)) aa1 - -a(1-oAA"'1
3Σ (1-a)1
V2[[(1S[1 - (1 -a)))oo -a(1 a))
|| a]
2/21 (1-a) --laall-aaaa--a -1-)))-a((-aaaa)a-
[ - [I
ca (1)))(((1))+11 a11 a) c'a log(1/a)
(6)
where ||||| and ||Al| denote spectral and Frobenius norms of matrix A, respectively, the first inequality follows from Davis-Kahan theorem (Davis & Kahan 1970), the second inequality follows from the fact that Frobenius norm of rank-2 matrix is bounded by v2 times the spectral norm, and the third incquality follows from the fact that [|EE] - (ax))))er a))vv? is the best rank one approximation of E - (1 -a]] and hence
|[[|_]| - (1-a) (l=a)) - Á & ||[] - (1 -a) - (D - (1 a) + |1 - (1 - a)t) - - < 2
(1 - a - The last incquality follows from Eq. 44 and Eq. For a S 1/2 and || 2 Vlog(1/o),
this implies the desired result.
D.1.2 Proof of Lemma D.2
We consider a scenario where the QUEscore of a (whitened and centered) representation h, h where tee the robust estimate of Pe, is computed as
5-1/2(h; - Ac),
hTQah Tr(Qp)
(9)
(7)
1)). We analyze the case where choose 3 = cc, such that 7(00) (u²hh,²² and the threshold T returned by SPECTRE satisfies the following. If we have infinite samples and there is no error in the estimates v, Σ, and Ae, then we have (3344a) < T1/2 < ((1 the fact that for the clean data with identity covariance, we can filter out at most 1.5a fraction of the data (which happens if we do not filter out any of the poisoned data points) and we can filter out at least 0.5a fraction of the data (which happens if filter out all the poisoned data). With finite samples and estimation errors in the robust estimates, we get the following:
where
which follows from
Q Q-1(4) + a²/d) c'add S 1/2 S ((144aaa c'o3/d) ((23/))+daa) td/dd,
(8)
where Q(-) is the tail of a standard Gaussian as defined in Lemma D.2 and we used the fact that for a large enough sample size we have i// - Pc) S d'a/d.
At test time, when we filter out data points with QUEscore larger than T, we have that we filter out at most clean [Selean n Siiterl s 2an, representations for a large enough d. Similarly, we are guaranteed that the remaining poisoned representations are at most |Spoison' \ SSiterl I Q(u - above bound /1/2 < e/log(1/o), this proves the desired bound.
Since from
D.2 Assumptions for Corollary 4.1
Corollary 4.1 follows as a corollary of Li et al. 2020aa) [Theoem 2.2]. This critically relies on an assumption on a (Eo, .) cllustraabeee dataset ((I; E Rd 13 E R) and overparametrized two-layer neural network models, as defined in Assumption 2 below.
Assumption ((a, ny, Eo. E1, M, L. M, C, W)-modll). The (1- - a)) fraction of data points are clean and originate from M clusters with each cluster containing n,/M data points. Cluster centers are unit norm vectors, (My Rd M 1==1-
Eo,
An input xa that belong to the 9-th cluster obeys [x. Payl S Eo, with eo denoling
24

---

Published in Transactions on Machine Learning Research (08/2023)
the input noise level. The labels y belong to one of the L classes and we place them evenly in [-1, in the training such that labels correspond to y (-1,-11 1/ - 1)) .)......11 1}. We let C = [MI,-..,, ,Mas]" 1- E Mxd
and define x= = X() as the minimum eigenvalue of the matric CCT C Bld (Cg)d'(CC))) for ~ ~0 Ido).
All clean data points in the same cluster share the same label. Any two clusters obey 1 - -Hyl | 2 2ED + E1, where &ι is the size of the trigger. The corrupted data points are generated from a data point ni with a source. label yi = Ysource belonging to one of the clusters for the source. A fixed trigger ô is added to the imput and labelled as target label q such that the corrupted paired ezample is (x; ō, We train on the combined dataset with an, corrupted points and (1 n),, uncorrupted points. We train neural network of the form ("")))"""((>>> x) for a trainable purumeter W E x and a fixed v E B.M where the head v is fixed as 1/VM M for half of the entries and -1/VM for the other half. We assume the activotion function 6: R + satisfy |d' (d)))) < d" for some constant e" > O.
D.3 Necessity of robust covariance estimation
To highlight that SPECTRE, and the robust covariance estimation, is critical in achieving this guarantee, next show that under the same assumptions, using the QUEscore filtering without whitening fails and also using whitening with non-robust covariance estimation also fails, in the sense that the fraction of the corrupted data is non-decreasing. We construct an example within Assumption D as follows: He = 0 and E.= = oI (1 - 6)un²) for a unit norm u. We place all the poisons at Hp = au with & = 0 and covariance >n zero. We let S a< log(1/a)) such that the separation condition is met. By increasing a², we can make the inner product of top PCA direction Prombined of the combined data and the direction of the poisons u arbitrarily small. Hence, after finding the threshold T in the projected representations Vccombined and using this to filter out the represontations, only increase as all the poisons are placed close to the center of the clean representations after projection. The same construction and conclusion holds for the case when we whitened with not the robustly estimated covariance, but the QUEscore based filter with / = oo projects data first onto the PCA direction of the whitenod data, which can be made arbitrarily orthogonal to the direction of the poisons, in high dimensions, i.e.
proposed in Tran ct al. (2018), the ratio of the poisons can
o "hh " S 2/d with high probability. This follows from the fact that after (non-robust) whitening, all directions are equivalent and the chance of PCA finding the direction of the poisons is uniformly at random over all directions in the d dimensional space.
E Experimental Details
In the experiments in $5 and Appendix F we train on the EMNIST, CIFAR-10, CIFAR-100, and Tiny- ImageNet datasets. In the EMNIST dataset, there are 3383 users with roughly 100 images in ten labels per user with heterogencous label distributions. We train a convolutional network with two convolutional layers, max-pooling, dropout, and two dense layers. The CIFAR-10 dataset has 50, 000 training examples, with 5000 samples in each label. We partition those samples to 500 users uniformly at random and train a ResNet-18 (He et al., 2016). With 500 samples in each label, the CIFAR-100 and Tiny-ImageNet datascts have 50, 000 and 100, 000 training examples respectively. For both datasets, we partition those samples to 100 users uniformly at random and train a ResNet-18.
For all datasets, the server randomly selects 50 clients each round, and each client trains the current model with the local data with batch size 20, learning rate 0.1, and for two iterations. The server learning rate is 0.5. The attacker tries to make 7's predicted as I's for EMNIST, horses as automobiles for CIFAR-10, roses as dolphin for CIFAR-100, and bees as cats for Tiny-emageNet. The backdoor trigger is a 5x5-pixel black square at the boltom right corner of the image. An a fraction of the clients are chosen to be malicious, who are given 10 corrupted samples. We set the malicious rate as its upper bound, i.e., a. We study first homogencous and static settings, and we discuss hetcrogencous and dynamic scttings in $5.4 and $.., In our framework, we set the retraining threshold é as 2%, and the convergence threshold e2 as 0.05%. We let the dimensionality reduction parameter & be 32 and set the QUE parameter /3 as 4 in Algorithm
Baselines. SPECTRE (Hayase et al. 2021 adopts robust covariance estimation and data whitening to amplify the spectral signature of corrupted data, and then detects backdoor samples based on quantum
25

---

Published in Transactions on Machine Learning Research (08/2023)
entropy (QUE) score. In federated learning settings, we adopt gradient- and appresentation-basee SPECTRE, which takes as input the gradient updates or sample representations averaged over a single client's 's local data with target label f. For both versions, we conduct the robust estimation and quantum score filtering every training round regardless of the computation constraints. We let the dimensionality reduction parameter k be 32 and set the QUE parameter / as 4.
RFA (Pillutla al. 2019 provides robust secure aggregation oracle based on the geometric median, which is calculated by the Weiszfeld's Algorithm. In experiments, implement RFA with 4-iteration Weiszfeld's 's Algorithm. We also consider the label-level RFA: for each label, the geometric median of the aggregated gradient uploaded from each client is estimated. The server then updates the model based on the averaged geometric medians of all labels.
Norm Clipping defense (Sun et al. 2019 bounds the norm of each model update to at most some threshold M, and Noise Adding method (Sun et al. 2019) add a small amount of Gaussian noise to the updates. In our experiments, we set the norm threshold M as and add independent Gaussian noise with variance 0.03 to each coordinate.
Multi-Krum [Blanchard et al. 2017b) provides an aggregation rule that only select a subset of uploaded gradients that are close to most of their neighbors. In our experiments, we set the Byzantine parameter f as 50k, where & is the malicions rate upper bound, and set the number of selected gradients as m = 20.
FoolsGold (Fung et al., 2018 provides an aggregation method that uses an adaptive learning rate per client based on inter-client contribution similarity. In our experiments, we the confidence parameter 1.
2022 adopts noise adding method, and uses the model clustering and norm clipping approach to reduce the amount of noise added. In our experiments, set the noise level factors as À 0.001.
FLAME Nguyen ct al.
2021 trains a certifiably robust FL model by adopting noise adding and norm clipping during the training time and using randomized parameter smoothing during testing. In our experiments, we set the norm threshold M as and add independent Gaussian noise with variance 0.03 to each coordinate during training. We use oT 0.01 to generate M 500 noisy models in parameter smoothing procedure, and set the certified radius as and the error tolerance as 0.001.
CRFL Xie et al.
E.0.1 Resource Costs
All algorithms including ours are implemented and performed on server with two Xeon Processor E5-2680 CPUs. Running all defenses for our experiments took approximately 1000 CPU-core hours.
F
Additional Experimental Results
F.1 Backdoor leakage under more datasets
Under the CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets, we show that backdoor leakage phenomenon also results in the failure of existing backdoor defenses in Figure Similar to Figure [[a]] we fix the malicious rate as a = 3% and run the experiments for 12, 000 training rounds under the static setting where data distribution is fixed over time. We can observe that the attack success rate (ASR) all competing defenses eventually approach around 1, 0.85, and 0.7 for each dataset, while the ASR of our algorithm keeps near (0.009, 0.027, and 0.030 for each dataset) all the time.
F.2 Defense under homogeneous and static clients with more datasets
We evaluate our shadow learning framework under CIFAR-100 and Tiny-ImageNet datascts in Table 8 and g) We can observe that all existing defenses suffer from backdoor leakage. CIFAR-100 and Tiny-ImageNe are also more difficult to learn compared with EMNIST dataset, and therefore the ASRs of our shadow learning framework in Table X and D are slightly larger than those in Table [ Besides, with different a, the MTA of Algorithm D is always above 0.85 and 0.70 for CIFAR-100 and convergence rate of Algorithm T is always similar to the defense scenario.
respectively, and the MTA
26

---

Published in Transactions on Machine Learning Research (08/2023)
Kilack Succoss Roka
Artavc Rate
Kilast Succoss
Ro
Dafvee
hols Adding
hobe Acding
Natu Auding
Mar Chopwey
Rorn rlicknn
home
Addiing
REx
RTA
RFA
Methed
240
48cs
7290
806 1000
240
2000
4000
10x
1800
Training
Traniing
Training
(a) CIFAR-10
(b) CIFAR-100
(c) Tiny ImageNet
Figure 5: Backdoor leakage causes competing defenses to fail eventually, even with a small a = 3%%
Table 8: ASR for CIFAR-100 under continuous training shows the advantage of Algorithm
Defense
0.15
0.25
0.35
0.45
Noise Adding
0.8688
0.8634
0.8525
0.8743
Clipping and
0.8593
0.8604
0.8629
0.8688
Noise Adding
RFA
0.8642
0.8697
0.8739
0.8697
Hulti-Krum
0.8576
0.8594
0.8635
0.8741
FoolsGold
0.8107
0.8251
0.8299
0.8415
FLAME
0.8361
0.8592
0.8688
0.8691
CRFL
0.8142
0.8197
0.8033
0.8251
G-SPECTRE
0.8415
0.8597
0.8542
0.8673
R-SPECTRE
0.7978
0.8033
0.8306
0.8467
Shadow Learning
0.0765
0.0929
0.1694
0.2247
Table 9: ASR for ImageNtt under continuous training shows the advantage of Algorithm !
Defense
0.15
0.25
0.35
0.45
Noise Adding
0.7079
0.6920
0.7253
0.9139
Clipping and Noise Adding
0.6839
0.6907
0.6971
0.7182
RFA
0.7164
0.7206
0.7091
0.6981
Multi-Krum
0.6841
0.7193
0.7032
0.7167
FoolsGold
0.6693
0.6872
0.7013
0.6944
FLAME
0.6767
0.6931
0.6792
0.6784
CRFL
0.6360
0.6519
0.6440
0.6279
G-SPECTRE
0.6423
0.6691
0.6945
0.6932
R-SPECTRE
0.5519
0.6090
0.6495
0.6826
Shadow Learning
0.0720
0.0799
0.1127
0.1519
27

---

Published in Transactions on Machine Learning Research (08/2023)
F.3
Ablation study: Main Task Accuracy (MTA) Attack Success Rate (ASR) Tradeoff
Main Task Accuracy
0.8
Main Task Accuracy
1.00
0.6-
0.85
0.4
0.4
0.15
No Defense
No Defense
RFA R-SPECTRE
RFA
0.2
R-SPECTRE
Our Methed
Our Method
0.0
066
0.0
0.2
0.6
1.
Attack Success Rate
Attack Success Rate
(a) 0.45
(b) a 0.15
Figure 6: MTA-ASR tradeoff shows simple early stopping with no (other) defense far from the ideal (0,1). R-SPECTRE with early stopping is more resilient, but all early-stopping- bassddpppn-- based prior defenses suffer when clients' data changes dynamically
show in 55.2 Algorithm achieves closo-to-ideal tradeoffs.
Under homogeneous and static EMNIST dataset, we run an ablation study on one of the main components: the backbone network. Without the backbone network, our framework (in the one-shot setting) reduces training bascline defenses with carly stopping. Figure 6 shows the resulting achievable (ASR, MTA) tune the early-stopping round for 0.45 and 0.15. The curves start at the bottom left (0,0) and most of them first move up, learning the main tasks. Then, the curves veer to the right as the backdoor is learned. We want algorithms that achieve points close to the top left (0, 1).
When a = 0.45, Algorithm 1 achieves the green star. The blue curve (early stopping with no other defense) is far from (0, 1). This suggests that the backbone network and SPECTRE filtering are necessary achieve the performance of Algorithm The curve for early-stopped RFA (purple x's) is also far from (0, 1) for large Parly-stopped R-SPECTRE without. the backbone network (orange triangles) achieves good MTA-ASR, tradeoff (though still worse than that of Algorithm []. However, show in 55.2 that the MTA-ASR tradeoff of R-SPECTRE is significantly worse when clients' data distribution changes dynamically.
When a = 0.15, the learning rate of backdoor samples is much smaller than the main task learning rate for curves. However, the curves for early-stopping with no defense and early-stopped RFA still cannot achieve close-to-ideal tradcoffs.
F.4 Synthetic heterogeneous clients
In synthetic heterogeneous EMNIST, each client receives shuffled images from 4 randomly selected classes with 25 samples per class. As shown in Table 10, it has similar trend as the naturally heterogeneous dataset. from the original EMNIST shown Table
Further, we analyze the situation where the user-level version of our algorithm works. We construct variants of EMNIST dataset with different heterogeneity level h. We call the dataset h-hheterggnneous if the first h-fraction of the overall training images are shuffied and evenly partitioned to each client, and for the remaining (1 -h) ht)-faccion samples, each client receives shuffled images from 4 randomly selected classes with [25(1 h)] samples per class. For the adversarial clients, they also own 10 backdoor images. We fix the malicious rate as a = 0. 1.15% Table 11 shows the ASR the user-level version of our algorithm under the dataset with different heterogeneity levels.
The ASR of the user-level version of our algorithm smaller than 0.1 when h < 0.4, indicating that our user-level method achieve the defense goal when under the dataset with low heterogeneity level. However,
28

---

Published in Transactions on Machine Learning Research (08/2023)
Table 10: ASR under synthetic heterogeneous EMNIST
Our Method with
Our Method with
a
Sample-level Defense
Our Method
User-level Defense
0.15
0.0134
0.0334
0.6756
0.25
0.0268
0.0401
0.7424
0.35
0.0535
0.0970
0.8127
0.45
0.0669
0.1305
0.9833
when the heterogeneity level is high, the malicious clients cannot be distinguished by the aggregated statistic over all local samples, which results in the failure of the user-level defense method.
Table 11: ASR of Our Method in User-level Version with a 0.15
Heterogeneity Level h
0.2
0.4
0.6
0.8
1
ASR.
0.0216 0.0334
0.0969
0.2508
0.4816
0.6756
We also analyze the ASR and MTA of our algorithm (label-level version) under the datasets CIFAR-100 and with different hetcrogencity level h. We fix the malicious rate as a = 0.15. As shown in Table 12 and 13 the ASR is always smaller than 0.11 under any heterogeneity level for both datasels, while the MTA is always larger than 0.85 and 0.76 for CIFAR-100 and T'iny-mmaell respectively.
Table 12: ASR and MTA of Our Method under synthetic heterogencous CIFAR- 100 with a 0.15
Heterogeneily Level h
0
0.2
0.4
0.6
0.8
1
ASR
0.077
0.079
0.086
0.091
0.094
0.103
MTA
0.864
0.859
0.861
0.858
0.851
0.853
F.5 More experiments under the dynamic setting
Recall the initial phases e1 and 62 Section 5.2 In the third phase e3, for all clients, we reduce the number of samples with labels 0 and 1 to 10% of the original while keeping other samples at the same level in e1, i.e., each client has one image for label 0 and 1 respectively, and ten images for any other label. This tradeoff is shown in Figure 7 Since the number of samples with the target label e = 1 is reduced, the learning rates of both the backdoor samples and the benign samples with the target label decrease. This is similar to the setting in Figure 6(b) except that the proportion of samples with the target label is smaller.
We also consider the case where the data distribution over malicious clients fixed over time, which rarely happens in practice. In a new phase 64, we let cach benign client contain 5 images for label 0 and 1 respectively, and 10 images for any other label. In phase es, we further reduce the number of images each benign client contains for label 0 1 to one. However, the local dataset of each adversarial client is always fixed as the malicious dataset in G1, i.e., 10 backdoor images in target label & = 1 and 90 images for other labels. The MTA-ASR tradeoffis of phases and és are shown in Fig
Our algorithm detects the data distribution change and renews the retraining period at the beginning of both phases. The MTA-ASR point our algorithm achieves is (0.9944, 0.1471) and (0.9937, 0.7458) phases e₁ and e; respectively. The MTA-ASR tradeoffs of the comparing algorithms are much worse than ours in both phases.
29

---

Published in Transactions on Machine Learning Research (08/2023)
Table 13: ASR and MTA of Our Method under synthetic heterogeneous Tiny-ImageNe with a = 0.15
Heterogeneity Level h
0
0.2
0.4
0.6
0.8
ASR
0.072
0.071
0.078
0.083
0.089
0.091
MTA
0.793
0.767
0.798
0.802
0.789
0.794
Main Task Accuracy
1.00
0.95
0.90
0.85
0.80
0.00
0.05
0.10
0.15
0.20
Periodic R-SPECTRE
Method
Method
Network Prediction
0.4
0.6
Attack Success Rate
Figure 7: Analysis of MTA-ASH ASR Tradeoff with a = 0.15 after distribution shift in phase e3.
1.0
1.0
Main Task Accuracy
0.6
Main
Our Method
Our Method
62-
Our Method with Shadow Network Prediction
Our Methad with Shadow Network Prediction
0.0
0.2
0.4
0.6
5.8
1a
0.0
6.2
0.4
0
1.0
Success Rate
Attack Success Rate
(a) Phase e4
(b) Phase es
Figure 8: Analysis of MTA-ASR Tradeoff with 0.15 under Time-varying Dataset and Static Adversarial Clients
30

---

Published in Transactions on Machine Learning Research (08/2023)
Since the benign features with label 1 becomes more difficult to learn due to the leak of samples, and the proportion of backdoor samples among samples with target labels increases to 26% and 64% in phases and e, respectively, it is difficult to achieve the close-to-ideal MTA-ASR tradeoff. If we increase the convergence threshold E2 from 0.05% to 0.5% when determining the carly-stopping point, we can obtain the new MTA-ASR point indicated by the red star as (0.9834, 0.0418) and (0.9834, 0.3378) 61 and e5 respectively. By scarilying little bit MTA (around 0.01 both cases), we can significantly reduce the ASR (around 0.1 64 and 0.4 in es).
F.6 Defense against adaptive malicious rate attack
We focus on an adaptive attacker strategy where the attack knows the time windows when the shadow learning framework conducts outlier detection and changes the number of malicious participating clients in each round accordingly. We assume that for every ro rounds, the attackers have the backdoor budget B = anero, ic., the adversary cannot corrupt more than onero participating clients every To rounds, where ne denoles the number of participating clients in each round.
our experiments, we set a = 0.3, nc = 50, ro = 150, and suppose the attacker has the knowledge that the outlier detection is conducted in 30-round time window. In the time window, the attack corrupts Omazz fractionction participating clients, where Omar a, intending corrupt the learned SPECTRE filter. In the future 120 rounds, the malicious rate will drop to amin f am due to unlimited backdoor budget. We vary Oman from 0.3 to and show the ASR under shadow learning framework in Figure g
0.5
Attack Success Rate
0.3
0.2
Our Method
0.5
0.7
1.0
Figure 9: ASR of adaplive malicious rate attack with 0.3
As observed in Figure g the ASR smaller than 0.05 when amaz < 0.4, indicating the robustness of shadow learning framework the adaptive malicious rate attack with moderate Omaz- With the increase of Omars the SPECTRE filter will be corrupted and perform worse, and therefore, the ASR will increase when Omaz 0.6. However, when Qmer is large enough, m'nnn drops significantly such that there are no sufficient malicious clients to further corrupt the model even with more corrupted filter. Therefore, the ASR then drops as the Omaz grows.
F.7 Aggregation with noise
Our approach does not allow for secure aggregation, which may introduce privacy concerns. To this end, we consider the selting where all gradients and representations are corrupted with i.i.d. Gaussian noise A o²), which is added to each coordinate before being uploaded from each client to the server. This is similar to Differentially-Privtt Stochastic Gradient Descent (DP-SGD). We vary the variance o² from 0.001 to 1, and analyze the performance of our method in terms of (MTA, ASR) pairs under the homogencous EMNIST dataset in Table 14
31

---

Published in Transactions on Machine Learning Research (08/2023)
Table 14: (MTA, ASR) Pair of Our Method under Different Noise Level
o² = 0
==
0.001
o² 0.01
a² = 0.1
o²
0.15
(0.997,
0.007)
(0.998, 0.003)
(0.994, 0.003)
(0.983, 0.000)
(0.873, 0.000)
0.25
(0.998,
0.010)
(0.996,
0.013)
(0.995, 0.007)
(0.983, 0.000)
(0.863, 0.000)
0.35
(0.998,
0.031)
(0.998, 0.031)
(0.991, 0.027)
(0.980, 0.000)
(0.852, 0.000)
0.45
(0.997, 0.050)
(0.996, 0.050) (0.993, 0.042)
(0.980, 0.017)
(0.867, 0.000)
We can observe that as the noise level increases, both the MTA and ASR decrease. The intuition is that with large noise, the learning rate of the backdoor samples decreases, which makes the early-stopping framework more effective. However, the accuracy on main tasks also drops due to the noise. When d² = 0.1, the ASR is 0 with S 0.35, while the MTA drops from around 0.997 to 0.98. With o² = 0.01, the added noise can reduce the ASR while keeping MTA around 0.993.
F.8 Sensitivity analysis of the filtering algorithm hyperparameters
For the filtering algorithm we adopt, there are three hyperparameters: malicious rate upper bound a, dimensionality reduction parameter k, and the QUE paramcter B.
For ã, we set the malicious rate a as its upper bound, i.e., a = in most experiments. We also study the case where the malicious rate can larger than its upper bound ā in Figure g For other hyperparameters, we set & 32 and 3 4 in our experiments. We conduct the sensitivity analysis under CIFAR-100 for & and
B in Table 15 and
16
We set the malicious as 0.15.
Table 15: Sensitivity analysis of dimensionality reduction parameter k
k
8
16
32
64
ASR
0.084
0.079
0.077
0.083
MTA
0.861
0.866
0.864
0.859
Table 16: Sensitivity analysis of dimensionality reduction parameter 3
B
2
8
ASR
0.081
0.077
0.082
0.082
MTA
0.867
0.864
0.859
0.853
We observe that different values of hyperparameters and / do not significantly affect the performance of our defense method.
32


Reviews:
[{"review_id": "KrgU7bhHQu", "reviewer": "Reviewer_kLkR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. To my knowledge, this paper is the first work to address the backdoor problem in FL under continuous training, which is enlightening for further research.\n2. They observe a phenomenon of backdoor leakage where most backdoor defenses lose efficacy over many training rounds.\n3. This paper is well-written, with the majority of its sections being easily comprehensible. \n\nQuestions:\n1. This paper focus on addressing the backdoor problem in FL under continuous training and emphasize that the continuous training is different from continual learning. Is the continuous training of importance in FL practice? Does other FL research consider the continuous training diagram or this paper propose it firstly? If so, why is the continuous training diagram necessary? The paper did not state this aspect clearly.\n2. The experiments in Section 5.1 is performed under homogeneous and static clients, while the experiments under distribution drift is used for ablation study. Since the authors emphasize the continuous training diagram, the comparison between baselines and the proposed method under distribution drift is necessary.\n3. The second constraint in Section 1: “FL models are typically trained continuously e.g., due to distribution drift” worth more discussion. In many real-world application and research, distribution drift or statistical heterogeneity is very important. I want to see more specific discussion on it.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focus on the defense against backdoor attacks in federated learning. In backdoor attacks, the training data is sourced from untrusted clients. In the federated learning scenario, defending backdoor attacks is more difficult since the server do not have access to raw client data. The authors propose shadow learning framework to defend against backdoor attacks in the FL setting under long-range training. They train two models in parallel, one of which is trained to make predictions for samples that are not in the target class, another of which is trained to make predictions for the backdoor target class. They claim to be the first work to protect against backdoor attacks in FL under continuous training.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the above questions. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Im2YRLCUXx", "reviewer": "Reviewer_qGb3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n+ The observation about the backdoor leakage for many defenses against backdoors in federated learning is interesting, and makes the case for continuous training. It shows the need for tackling this problem with a different angle. \n+ The proposed method using a combination of two models, with one of them, aware of the target labels, looks promising, especially, as the authors included a method to defend against backdoors even if the target labels are unknown. \n+ Although some of the assumptions are a bit restrictive, the theoretical analysis included in Section 4 is nice to support the benefits of the proposed method (although perhaps, more discussion on the implications of the theoretical results would be beneficial). \n\nWeaknesses: \n-\tThe threat model and the positioning with respect to other defenses in the state of the art is not very clear. Perhaps, the authors can consider the threat model for SPECTRE as a reference for this paper. \n-\tThe experimental evaluation is not very convincing and/or comprehensive (see comments in the requested changes). \n-\tThere is some abuse on the use of the appendix. Given that there is no limitation on the number of pages, the paper would be more readable is some of the information given in the appendices is moved to the main paper. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors observed that different defenses against backdoors in federated learning suffer from a backdoor leakage, so that, if the model is trained for a large number of rounds, the attack success rate can increase significantly rendering the defenses useless. This can be particularly harmful for machine learning models under continuous training. Based on this observation, the authors propose a novel method to defend against backdoors in federated learning under continuous training using shadow learning, with a combination of two models, one trained using the entire (tainted) datasets provided by the participants, and another, aware of the target classes for the backdoor attack, using robust aggregation techniques, such as, for example, SPECTRE. The authors show that the combination of these two models allow to reduce the attack success rate against backdoor attacks, where the attackers just can manipulate the training datasets before training (i.e. it does not support model poisoning attacks). The authors also provide a mechanism to detect the target classes without previous knowledge. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper makes an interesting point about the backdoor leakage of many defenses against backdoor attacks in federated learning that can be a problem in continuous training scenarios. The method proposed by the authors is interesting and promising. However, I think that the experimental evaluation is not convincing to support the claims made in the paper and validate the benefits of the proposed approach. \n\nThus, the main comparison in Section 5.1 includes a scenario where we do not have a continuous training setting. The authors used a static dataset for each participant, using IID partitions of the training set across the different clients. On the other side, the authors just reported the attack success rate and the accuracy and convergence of the different approaches is not shown. In this sense, it would be more appropriate to simulate a continuous training scenario and plot, both the attack success rate and the models’ accuracy (on benign data points) across the different training rounds. Apart from convergence, this analysis is interesting to observe what happens when the training datasets change during the continuous training of the models. It would also be more beneficial to do this assuming non-IID settings. In the paper, only a case with EMNIST is considered under non-IID settings, which is a bit limiting to extract useful conclusions. \nThe analysis of the defense without knowing the training label (Section 5.3) looks also shallow. I think that the comparison of both the cases where the defender is or not aware of the target label should be considered throughout all the experiments. On the other side, it is unclear how the parameters used for the filtering are set and a sensitivity analysis should be conducted to analyze the impact of this filter in the performance of the models and the attack success rate. \n\nThere is some abuse of the appendices. Some of the results reported there would fit better in the main paper. For instance, Section 5.5 consists only on two lines of text pointing to the appendices. In this sense, the analysis against adaptive attacks and against different triggers is of enough relevance to show in the main sections of the paper (the authors spend more pages for the appendices than for the main paper). \n\nApart from the experiments, the threat model (adversarial model as named in the paper) is a bit unclear. I think that the authors should clarify this and follow a similar threat model as in SPECTRE (Hayase et al. 2021). Compared to other papers, the threat model in this case is more restrictive than for some of the competing methods used in the experimental evaluation. Thus, in this paper, only poisoning attacks, where the adversary can only inject malicious points, but cannot manipulate the training procedure directly, performing, for example, model poisoning attacks. Actually, in this sense, the comparison in Section 5.1 is not really fair, as the other competing methods are design to resist attacks in more challenging settings. It is fine for me that the authors just focus on data poisoning attacks, but the position of the paper and the threat model should be clearer about this. For instance, in Algorithm 1 (lines 4 and 10), the malicious clients can lie on the accuracy.\n\nSome other minor comments or clarifications:\n+ In page 4: how is the backdoored dataset constructed in the aggregator? \n+ Page 5: the backbone model “is used to learn parameters of a filter that can filter out poisoned model updates.” Can the authors clarify this?\n+ In Section 5.1: how many training iterations? 1,200 or 12,000?\n+ Experimental settings: how do the authors generate heterogeneous partitions of the datasets for the different clients? It’s not clear from the information in Appendix E. \n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "et9FI2WSW7", "reviewer": "Reviewer_kR1q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. Defense against backdoor attacks in federated learning is an important research problem.\n2. It is interesting to train a shadow model as a backup for correct prediction.\n\nWeaknesses\n\n1. The proposed method is based on the assumption that the defender knows the target label of the backdoor attack. This is a very unrealistic assumption. Even though the paper conducts the experiments to demonstrate that the proposed defense also works if the defender only knows the target label is within a set of labels (but not all of them). However, there are a few other problems.\n\n   - Firstly, the proposed method needs to train a shadow model for each label, which is computationally infeasible. For ImageNet with 1000 classes, this method has to train and store 1000 models. It requires massive computation and storage resources.\n   - Secondly, the attacker can inject multiple backdoors targeting different labels in the model. The proposed method can hardly work in such a scenario.\n\n2. The effectiveness of this paper heavily depends on the existing filter algorithms. Those algorithms can be evaded by adaptive attacks. Although this paper evaluates adaptive attacks, none of the experiments considers the actual adaptive scenario, where the attacker particularly aims to evade the filter algorithm. The experiment in Appendix F.7 does not evaluate this. The L_ano loss is only for \"the difference between the representations of backdoor samples and benign samples with target label\". However, this is not the actual method used by those algorithms. Most of those filter algorithms aim to differentiate the clean and malicious updates. The adaptive attack shall try to evade this part, making malicious updates indistinguishable from clean ones.\n\n3. The statement \"there is no solution today that can defend against backdoor attacks in an FL setting with continuous training\" is not true. An existing work [1] specifically targets the continuous training scenario and it can effectively remove backdoors in federated learning. This highly related work was not discussed and compared in the submission.\n\n\n[1] Zhang, Kaiyuan, et al. \"FLIP: A Provable Defense Framework for Backdoor Mitigation in Federated Learning.\" ICLR", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a defense method against backdoor attacks in federated learning. It trains an additional model, called a shadow model, other than the common global model (called the backbone model) at the server side. The shadow model is obtained by training on filtered updates that are cleaned by existing techniques. This paper assumes that the target label is known to the defender. During inference, it first uses the backbone model to make the prediction. If the predicted label is the target label, it then uses the shadow model to carry out another round of prediction, which will be used as the final output. The evaluation is conducted on four datasets. Compared to the baselines, the proposed method has better defense results.\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Evaluate the method on ImageNet or similar datasets with hundreds or thousands classes while the target label is unknown\n2. Evaluate the scenario where the attacker injects the number of backdoors as the number of the classes\n3. Conduct the experiment on real adaptive attacks targeting the filtering algorithm that this paper is based on\n4. Discuss and compare with a highly related work", "broader_impact_concerns": "None\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three expert reviewers reviewed this paper with ratings "Accept", "Leaning Accept", "Leaning Reject". After the rebuttal, authors addressed majority of the concerns; however, there are still outstanding issues to be fixed.

The paper can be accepted after authors make the following revision.

1- Authors should explicitly and clearly discuss the expensive computation cost of checking every label. This is raised by one of the reviewers and is an important point to emphasize.

2- Authors should incorporate the reviewers' suggestions including the results provided during the rebuttal.
Decision: Accept (poster)