Title: IBA: Towards Irreversible Backdoor Attacks in Federated Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Backdoor Attacks

Federated Learning

Durability

Imperceptibility

Stealthiness
Abstract: Federated learning (FL) is a distributed learning approach that enables machine learning models to be trained on decentralized data without compromising end devices' personal, potentially sensitive data. However, the distributed nature and uninvestigated data intuitively introduce new security vulnerabilities, including backdoor attacks. In this scenario, an adversary implants backdoor functionality into the global model during training, which can be activated to cause the desired misbehaviors for any input with a specific adversarial pattern. Despite having remarkable success in triggering and distorting model behavior, prior backdoor attacks in FL often hold impractical assumptions, limited imperceptibility, and durability. Specifically, the adversary needs to control a sufficiently large fraction of clients or know the data distribution of other honest clients. In many cases, the trigger inserted is often visually apparent, and the backdoor effect is quickly diluted if the adversary is removed from the training process. To address these limitations, we propose a novel backdoor attack framework in FL, the Irreversible Backdoor Attack (IBA), that jointly learns the optimal and visually stealthy trigger and then gradually implants the backdoor into a global model. This approach allows the adversary to execute a backdoor attack that can evade both human and machine inspections. Additionally, we enhance the efficiency and durability of the proposed attack by selectively poisoning the model's parameters that are least likely updated by the main task's learning process and constraining the poisoned model update to the vicinity of the global model. Finally, we evaluate the proposed attack framework on several benchmark datasets, including MNIST, CIFAR-10, and Tiny ImageNet, and achieved high success rates while simultaneously bypassing existing backdoor defenses and achieving a more durable backdoor effect compared to other backdoor attacks. Overall, IBA offers a more effective, stealthy, and durable approach to backdoor attacks in FL. The code associated with this paper is available on [GitHub](https://github.com/sail-research/iba).

IBA : Towards Irreversible Backdoor Attacks in Federated Learning

Abstract

Federated learning (FL) is a distributed learning approach that enables machine learning models to be trained on decentralized data without compromising end devices' personal, potentially sensitive data. However, the distributed nature and uninvestigated data intuitively introduce new security vulnerabilities, including backdoor attacks. In this scenario, an adversary implants backdoor functionality into the global model during training, which can be activated to cause the desired misbehaviors for any input with a specific adversarial pattern. Despite having remarkable success in triggering and distorting model behavior, prior backdoor attacks in FL often hold impractical assumptions, limited imperceptibility, and durability. Specifically, the adversary needs to control a sufficiently large fraction of clients or know the data distribution of other honest clients. In many cases, the trigger inserted is often visually apparent, and the backdoor effect is quickly diluted if the adversary is removed from the training process. To address these limitations, we propose a novel backdoor attack framework in FL, the Irreversible Backdoor Attack ( IBA ), that jointly learns the optimal and visually stealthy trigger and then gradually implants the backdoor into a global model. This approach allows the adversary to execute a backdoor attack that can evade both human and machine inspections. Additionally, we enhance the efficiency and durability of the proposed attack by selectively poisoning the model's parameters that are least likely updated by the main task's learning process and constraining the poisoned model update to the vicinity of the global model. Finally, we evaluate the proposed attack framework on several benchmark datasets, including MNIST, CIFAR-10, and Tiny ImageNet, and achieved high success rates while simultaneously bypassing existing backdoor defenses and achieving a more durable backdoor effect compared to other backdoor attacks. Overall, IBA 2 offers a more effective, stealthy, and durable approach to backdoor attacks in FL.

1 Introduction

Federated Learning (FL) [13] is a novel distributed learning paradigm that enables multiple parties to train machine learning models collaboratively without centralized data sharing. Specifically, each participant trains a local model on their private data, and only the model updates are exchanged with a central server for aggregation. However, FL also introduces new security and privacy challenges

that need to be addressed to ensure the confidentiality and integrity of the learning process [33, 24], such as model poisoning, membership inference, data leakage, and Sybil attacks [9, 20].

Backdoor attacks are a specific form of model poisoning in which an attacker inserts a backdoor trigger during the training process that can later be exploited to manipulate the model's behavior [4, 12, 17, 6]. FL is particularly vulnerable to backdoor attacks since each participant trains a local model on their private data, and the central server aggregates the model updates without validating their integrity. An attacker can inject poisoned data or backdoor triggers into the local models of a subset of participants. When the models are aggregated, the backdoor trigger can be activated to compromise the model's behavior [2, 28, 32].

Artificial backdoor attacks are a standard method for backdoors in FL. In an artificial backdoor attack, an apparent trigger, such as a specific pixel value, is injected into the local training data [25, 32]. Although meaningless in the data context, the apparent trigger causes the trained model to associate a signal for the model to optimize during training. When trained on such data, the model will induce misclassifications toward a target class when making predictions on inputs with the presence of the trigger. Existing artificial backdoor attacks require the involvement of a portion of compromised clients and need assistance from scaling-based methods to achieve high attack success rates [32, 24]. This makes these attacks vulnerable to mainstream backdoor defenses [30, 31, 15]. Furthermore, most existing attacks are not durable (i.e., when the adversary is removed from the FL process, the backdoor effect is gradually diluted [34]), especially when the size of compromised clients is small, and they do not regularly participate in the FL training.

In this paper, we exploit the principle of adversarial attacks to craft a novel instance-specific backdoor attack, called Irreversible Backdoor Attack ( IBA ), that only requires a small number of compromised clients but achieves more robust attack performance and durability. IBA is a two-phase attack scheme, including Trigger Generating and Backdoor Injection . During training rounds, the former manipulates the attack model, which is a generative model, so the triggers generated by this attack model can deceive the local model into misbehaving. After that, we formulate the Backdoor Injection phase as a constrained optimization problem. Then, we propose a simple but effective recall to keep the attack model updated regularly. This algorithm allows the adversary to achieve high attack success rates even when a small number (i.e., 1) of clients are compromised. In addition, we enhance the durability and effectiveness of the proposed attack by selectively poisoning the model's parameters that are least likely updated by the main task's learning process and constraining the poisoned model update to the vicinity of the global model, respectively. This strategy helps IBA bypass mainstream defenses in FL and achieve extended backdoor durability.

Our contributions are summarized below:

Proposal of a two-phase backdoor injection mechanism, IBA , in FL: The paper introduces IBA , which combines the manipulation of the trigger generator and the poisoning of the local model into a unified two-phase approach. IBA constrains the poisoned model update to be within the vicinity of the global model to improve the practicality and effectiveness of the attack.

Proposal of selective parameter poisoning in the constrained optimization algorithm: The paper enhances the durability of IBA by selectively poisoning the model's parameters less likely to be updated during the main task's learning process.

State-of-the-art attack performance and stealthiness: The paper demonstrates that IBA achieves state-of-the-art attack performance and durability and exhibits high stealthiness against existing defense mechanisms. This highlights the robustness and effectiveness of IBA as a practical backdoor attack method.

2 Related Work

Backdoor Attacks in FL. Backdoor attacks in FL have been extensively studied, revealing vulnerabilities and proposing various attack strategies [16]. Backdoor attackers generally aim to incorporate concealed backdoors into Deep Neural Networks (DNNs) during the training phase. This allows the compromised DNNs to exhibit normal behavior when presented with benign samples. However, their predictions will be deliberately and consistently altered when triggered by specific patterns defined by the attacker. One of the most commonly employed methods for incorporating backdoor

functionality during training is poisoning the training samples [2, 32]. Bagdasaryan et al. [2] also proposed model replacement to implant backdoors into the joint model and demonstrated that even a single-round attack with a small fraction of clients could achieve a 100% Attack Success Rate (ASR). Xie et al.[32] investigated Distributed Backdoor Attacks (DBA), which decompose the global trigger into non-overlapping local triggers. DBA exhibits higher ASR and increased robustness against FL aggregation methods. Additionally, backdoor triggers in machine learning can be designed to be imperceptible in the input space [10, 12, 17, 6] and in the latent space [5], or to possess sophisticated, multi-target payloads [7]; the ground-truth labels of the poisoned samples can also align with the intended target label [26, 22, 35]. These characteristics contribute to the stealthiness of backdoor attacks. Recently, Neurotoxin [34] proposed to enhance backdoor durability by targeting specific coordinates less likely to be updated by benign agents, partially poisoning the model by projecting gradients onto a coordinate-wise constraint. This extends the backdoor's impact and prevents catastrophic forgetting, strengthening the effectiveness of the attack. The recent work that is most aligned with ours is PerDoor [1], which uses the Basic Iterative Method (BIM) to generate the trigger, which relies on gradients of the model's loss function concerning the input data to alter the original inputs into backdoored inputs. However, this method fails to learn underlying patterns and distributions from the training dataset and lacks a sharing and collaborating mechanism among multiple malicious clients, which is very beneficial for the adversary in collaborative attacks.

Defense Mechanisms against Backdoor Attacks in FL. The mainstream backdoor-attack defenses in FL aim to reduce the effect on the global model, as it has been shown that detecting backdoors is challenging without breaking the privacy of participants [14]. For instance, Krum [3] aggregates local models by choosing a single local model as the aggregated model with the smallest Euclidean distance to a certain fraction of other models. Norm difference clipping (NDC)[25] examines the norm-difference between the global model sent to and model updates shipped back from selected clients, using a pre-specified threshold to clip model updates that exceed the norm difference. RFA[19] aggregates local models by computing a weighted geometric median using the smoothed Weiszfeld's algorithm. Ozdayi et al. [18] proposed a defense mechanism RLR, that adjusts the learning rate of the aggregation server in FL based on the sign information of agents' updates by carefully considering it per dimension and round. These defense mechanisms offer valuable insights and strategies for countering backdoor attacks in FL settings. Another approach is identifying suspicious clients and removing them from the aggregation process. FoolsGold [8] assumes that benign datasets from different clients differ from each other and assigns low weights to models that are similar to many other models, but it can be vulnerable to sophisticated adversaries submitting poisoned updates without raising suspicion. FLAME [15] uses the HDBSCAN algorithm to detect malicious updates, combines model filtering with poison elimination to detect and remove malicious updates, and is robust against inference attacks.

3 Irreversible Backdoor Attack ( IBA ) in FL

FL Framework. The training process in the standard FL framework involves the following steps:

Step 0 (FL Initialization) : The central server S initializes the weight of the global model w and sets hyperparameters such as the number of FL rounds and local epochs.

Step 1 (Local Model Training and Update): Selected clients C 1 , C 2 , ..., C K receive the current global weight w 0 from S . Each client C i updates its local model parameters w i using its local dataset D i . After local training, the clients send their local weights to S for model aggregation.

Step 2 (Global Model Aggregation and Update): S aggregates the received local weights and computes the aggregation result. The aggregated result is sent back to the clients for the next round of training. S aggregates the received local model weights using the FedAvg [13] formula:

where |C k | represents the number of data samples held by client C k .

Attacker Threat. To address the weakness of existing works, i.e., the adversary A controls a number of compromised clients, we focus on the scenario that A controls only one client k and it will participate in the training for each f rounds, i.e., fixed-frequency attack [28].

Trigger Generating Function

Aggregator

Backdoor Injection Process

𝒘

𝒕"𝟏

𝓓

!"#$%

𝒘

𝒊

𝒕

𝐺

%

&"'

Adversarial

Noises

𝓓 "

&'()'%

𝐺

%

&

Target Label:

𝑦

*

𝒘

𝒕"𝟏

𝐺

%

("'

Adversarial

Noises

𝓓 "

&'()'%

Target Label:

𝑦

*

𝓓

!"#$%

ℒ

)*+,-

ℒ

./01/-

𝒘

𝒊

𝒕

Partial model

poisoning

ℒ

)*+,-

ℒ

%

+

+

𝑓

𝒘

!

"

(𝓓

(

./01/-

)

Optimization path

Input

Adversary Capability: We follow the same assumption used in the existing works on backdoor attacks in FL [2, 15], where adversary A has total control over the malicious participant(s). However, A has no control or access to other benign participants' data or local updates. Also, we assume A thoroughly understands the central server and any potentially deployed defenses but cannot alter the server's configuration parameters and algorithms.

Adversary Objectives: The goal of A is learning a trigger generator G ξ such that its outputted triggers can activate the backdoor function to mislead the global model F ( w ) while ensuring the behaviors on clean samples are unaffected. This trigger is expected to bypass human and machine inspections. Moreover, A can bypass configured defenses in FL and prolong the backdoor effect even when A is removed from the training process. The overall mechanism of our proposed backdoor attack IBA is illustrated in Figure 1.

3.1 The Trigger Generating Function

In FL, when clients actively engage in a training round, they have the opportunity to observe and influence the changes in the global model w . In particular, a client k receives the global model w t -1 and updates it using their local data to produce a new local model, denoted as w k . After local training, the learned surrogate model is aggregated to partially reflect the subsequent global model. In light of this process, we propose an initial phase called the Trigger Generating . In this phase, the trigger generation model G ξ is updated to generate adversarial noise. This noise is combined with the original image x to create a backdoor sample. This sample remains imperceptibly different from the original, but it effectively misleads the model G ξ into making incorrect classifications for the target class. The rationale behind this approach is that manipulating G ξ based on this local model allows the adversary A to optimally adapt the generative function, even in the face of round-by-round variations in the global model. The transformation function is inspired by adversarial examples, in which we model it as a perturbation on the input, as follows:

The function G ξ takes an input x and generates adversarial noise on the same input space, guaranteeing the backdoor attack's stealthiness.

To achieve this goal, we utilize the local model as the surrogate model to update the attack model G ξ using the following objective function:

in which f w is the trained local model and y T is the targeted label. It's worth noting that the local model f w submitted to the server will be totally vanilla during this phase.

While considering various norms for bounding the generator noise in Equation 1, such as the L 2 norm, it's generally inadvisable for backdoor attacks. This infinity norm guarantees a widespread distribution of the generated trigger across the input image (encompassing all pixels in the trigger); in contrast, the L 2 norm can result in localized artifacts within the image (with only select pixels forming the trigger). Employing the L 2 norm can make the backdoor attack more susceptible to detection by trigger-synthesis defenses like Neural Cleanse [27]. Thus, the ∥ . ∥ ∞ norm is a preferred choice.. We can design such generator function as an auto-encoder or the more complex U-Net architecture [21]. However, we observe no significant performance difference between a simple auto-encoder and U-Net by training the generator function and the classifier with the proposed training algorithm.

In the proposed transformation function, parameter ϵ controls the visual stealthiness of the triggers. For instance, even on the gray-scale MNIST dataset, if ϵ is less than 0.01, there is usually no discernible difference between the clean and disturbed images. A greater value for ϵ , i.e., more obvious triggers, simplifies generative model learning. On the other hand, a too-small ϵ may make learning a generative model challenging since it is heavily dependent on the variance of the surrogate model, which varies across FL training rounds. Therefore, during this phase, we keep the ϵ sufficiently large, i.e., ϵ 0 , until we reach the expected attack success rate.

3.2 The Backdoor Injection Process

At the second phase aims to gradually insert the backdoor function into the global model by manipulating the poisoned local model. Our objective is to learn classifier (i.e., local model) f w k that concurrently performs indifferently on x in comparison to the classifier's clean version but distorts its prediction on the backdoor image T ξ ( x ) to the target class y T . The above task can be formulated as the following objective function:

The classification model f w in the above problem is trained based on local data and the optimal function G ξ learned in the previous phase. When training the classifier, the parameters α and β regulate a mixture of the strengths of the loss items from clean and poisoned data, i.e., L clean and L poison , respectively. If α is more significant than β , the classifier's performance on clean data rapidly converges to the optimal performance of the vanilla classifier, according to our experiments. Otherwise, the classifier's performance on the backdoor data quickly reaches the optimal value. To compensate for the performance of the local model on both the main task and the backdoor task, we assume α = 0.5 and β = 0.5 in the remaining part of the paper. As previously stated, the value of ϵ must be assigned cautiously to overcome the unpredictability of the global model and assist the local model in learning the generated triggers more effectively. To achieve this goal, we propose a self-adjusting value for ϵ from the starting round of this phase, i.e., t I :

where the value of ϵ is gradually decayed until it reaches objective ˆ ϵ .

Attack Model Retraining. During this phase, we propose frequently updating the attack model G ξ to keep it in sync with the state of the global model. The retraining frequency should be chosen (20, 50).

Partial Model Poisoning. To enhance the longevity and effectiveness of the attack, we propose to partially poison the model based on selected poisoning space and poisoning dimension. In which space-based poisoning helps the attack bypass the backdoor defenses because it constrains the

poisoned model such that it does not go far from the vicinity of the benign model, while dimensionbased poisoning helps to reduce dilution effect from benign clients by shrinking the poisoned neurons to infrequently updated neurons.

Poisoning-space based. Under this attack, the submitted local model is partially contaminated by being constrained by the magnitude of its parameters. Specifically, the adversary applies projected gradient descent on the losses for D∪D ′ , where D , D ′ are benign and poisoned datasets, respectively. In other words, it periodically projects the model parameter on the ball centered around the global model of the previous iteration. This enables IBA to circumvent defenses based on norm and distance investigation by preventing the resulting poisoned model from deviating substantially from the original. Mathematically, this method guarantees that || w k -w || ≤ δ . Note that, this attack strategy can be combined with model replacement [2], where the model parameter is scaled before being sent to the server to cancel the contributions from the other honest clients [28]. The scaled model is calculated by the following:

Poisoning-dimension based. Because the attacker cannot participate in every round of training, directly submitting the poisoned model to the server is not effective [31]. To address it, we propose leveraging historical gradient variation to select infrequently updated neurons, which are considered poisoning dimensions for IBA . Then the adversary computes bottom -k % coordinates of the gradient vector g , obtained by learning the main task on the clean data D . In particular,

in which p refers to the number of rounds up to round t that adversary A has been selected to participating in FL training. After that, when finishing calculating gradient ˆ g of local model f w using Equation. 3, A projects gradient onto coordinate-wise constraint on the top -k % coordinates of ˆ g . As a result, only bottom -k % coordinates are poisoned, which can prolong the backdoor effect since other benign clients rarely update these coordinates. Utilizing historical information aids in determining round-by-round bottom -k % coordinates, and it becomes more valuable as rounds advance and local client models converge on the global model.

4 Experiments

Our empirical study aims to highlight the effectiveness of the IBA attack against the state-of-theart (SOTA) FL defenses. We conduct our experiments on real-world datasets and a simulated FL environment. We design three experiment sets: (1) IBA under different attack scenarios, (2) IBA under different FL defenses, and (3) IBA's imperceptibility and durability evaluation. Our results demonstrate that the IBA can achieve significant efficiency, stealthiness, and durability. In particular, IBA attains persistent performance with/without SOTA defenses and maintains higher visual imperceptibility and longer durability compared with existing artificial backdoor attacks [32, 2]. Our implementation is publicly available to reproduce experimental results.

4.1 Experimental Setup

Dataset and Backdoor Tasks. The IBA method is evaluated on three classification datasets: MNIST, CIFAR-10, and Tiny ImageNet. We simulate heterogeneous data partitioning by sampling p k ∼ Dir K (0 . 5) /Dir K (0 . 01) for MNIST, CIFAR-10/Tiny ImageNet and allocating a proportion of each class to participating clients. We follow the established protocol outlined in previous works [28, 32], employing stochastic gradient descent (SGD) optimization with E local epochs, a local learning rate of l r , and a batch size of 32. The number of clients selected in each round is 10 / 200 . The details of the datasets and other parameter configurations are presented in the supplementary material 3 .

Trigger Generation and Backdoored Model Training. The generation of triggers is accomplished through the utilization of the U-Net architecture [21], incorporating specific parameter configurations.

The constraint values are set to ϵ = 0 . 3 , ˆ ϵ = 0 . 05 ; while α = 0 . 5 and β = 0 . 5 define additional parameter values. During trigger training, the threshold for backdoor accuracy (BA) is established as local BA = 0 . 85 . To facilitate the generation of updates, the learning rate for the update generation model is determined as γ A = 0 . 0001 , and for the update classifier model, the learning rate is set to η = 0 . 01 . Additionally, the performance of IBA is evaluated in conjunction with prominent FL strategies, namely FedAvg [13], FedProx [11], and FedNova [29]. This comprehensive evaluation aims to assess the effectiveness of IBA in the context of different FL methodologies.

Participating patterns of attackers. In addition to the fixed-frequency case described in the threat model, we further evaluate the efficiency of IBA under a more favorable condition known as fixed-pool case (or random sampling), where there is a fixed pool of clients are controlled by A . The result for the fixed-pool case is left in the appendix.

FL Backdoor Defenses. We examine six SOTA defense techniques designed to mitigate backdoor attacks in FL: (i) NDC [25], (ii) KRUM and (iii) Multi-KRUM [3], (iv) RFA [19], (v) RLR [18], (vi) FoolsGold [8]. By evaluating the performance of these defense techniques, we aim to provide insights into their effectiveness in countering backdoor attacks in the FL setting.

4.2 Experimental Results

(1) IBA under different attack scenarios. Firstly, we study the performance of IBA under fixedfrequency attacks with three datasets: MNIST, CIFAR-10, and Tiny ImageNet. The evaluation of main accuracy (MA) and backdoor accuracy (BA) from Table 1 reveals interesting insights. In the absence of the attack, the MAs of the baseline are 99.74%, 84.71%, and 65.15% for MNIST, CIFAR-10, and Tiny ImageNet, respectively, indicating accurate classification on clean samples, while the BA remains low at 0.56% to 9%, indicating no backdoor effect on global model. Under IBA attack with FedAvg aggregator, the corresponding MAs are negligibly affected, while the BAs are notably high, i.e., 99.07% for MNIST dataset. This indicates that IBA can stealthily insert a backdoor function without negatively impacting the model's behavior with benign samples.

In FL, many existing backdoor attacks target vulnerabilities within the FedAvg aggregator since the contribution of each client is considered equally. Therefore, we examined IBA 's resilience across different aggregation methods, specifically FedAvg, FedProx, and FedNova, as presented in Table 1. FedProx, with its proximal term, enhances model accuracy and is beneficial in scenarios with non-IID data distributions. Besides, FedNova, designed to address objective inconsistency in FL, ensures consistent and accurate model updates even with varying update frequencies from clients, making it suitable for scenarios with heterogeneous data.

The results demonstrate the consistent efficacy of IBA with three datasets and three FL aggregators. In particular, IBA persistently obtained significant BAs, with the lowest BA being 87.13% and the highest being 99.77%, demonstrating its efficacy and potential challenges in diverse FL settings.

Fixed-pool attacks. We study the performance of IBA under fixed-pool attack setting (percentage of attackers in the overall clients varies from 5% to 20%). The results are shown in Figure 2, demonstrating that a higher compromising ratio leads to a better backdoor accuracy or more substantial backdoor effect.

(2) IBA under different FL defenses. Westudy the effectiveness of standalone IBA and hybrid attacks (i.e., IBA and partial model poisoning) against the aforementioned defense techniques. As demonstrated in

25

50

75

0

5

10

15

20

25

Compromising Ratio (%)

Accuracy (%)

BA

MA

77

78

79

80

81

82

83

84

85

86

0

100

200

300

400

500

FL training round

NCA

Krum

Multi-krum

RLR

RFA

(a) Main Accuracy

40

60

80

100

0

100

200

300

400

500

FL training round

NCA

Krum

Multi-krum

RLR

RFA

0

20

40

60

80

100

0

100

200

300

400

500

FL training round

Accuracy (%)

(a) Krum

0

20

40

60

80

100

0

100

200

300

400

500

FL training round

Accuracy (%)

(b) Multi-Krum

0

20

40

60

80

100

0

100

200

300

400

500

FL training round

Accuracy (%)

0

20

40

60

80

100

0

100

200

300

400

500

FL training round

Accuracy (%)

(d) RLR

Figure 3, the stand-alone IBA can circumvent NCA, Multi-Krum, and FoolsGold and attain a high backdoor accuracy (i.e., 95.11%, 79.5% and 76.46% for CIFAR-10 dataset, respectively). Krum is the most challenging defense since it replaces the global model with only the most representative local model (i.e., the one with the shortest distance to its neighbors) in each round. As a result, the global model experiences substantial fluctuation over rounds and does not fully capture the knowledge of all participants, i.e., a significant decline in primary task accuracy in Figure 3. This mechanism hampers

(a) IBA, MNIST

(b) DBA, MNIST

(c) IBA, CIFAR-10

(d) DBA, CIFAR-10

1

1

9

6

7

9

7

7

9

9

the attack model's learning process since it is closely associated with global model variance. Figure 4 illustrates the improved performance of IBA against challenging defenses when combined with partial model poisoning, i.e., PGD-based. As observed, the IBA + PGD (w./wo.) model replacement can help improve the backdoor accuracy from 10% to 20%. Significantly, the improved IBA can achieve almost 100% of backdoor accuracy for Multi-Krum defense. The more detailed result can be seen in Table 2. The proposed IBA can bypass most existing defenses and achieve significant backdoor accuracy. In most cases, the combination of IBA + PGD/ IBA + PGD + MR helps improve the backdoor accuracy from 10% to 20%. While Krum and RLR defend the proposed attack to some extent, i.e., we observe the drop in the BAs of IBA, we argue that Krum and RLR tend to reject previously unseen information from both adversary and honest clients, which results in low MAs (a drop of 3-10%). Specifically, the proposed IBA causes RLR to compensate for the backdoor defense's effectiveness with the main accuracy drop in the MNIST dataset by unnecessarily reversing the learning rate of specific dimensions, i.e., the BAs are around 90% (less than other defenses 10%). Therefore, these two defenses raise great concern about its practicability under this scenario.

(a) MNIST

(b) CIFAR-10

0

1

2

3

Cifar10

MNIST

T-Imagenet

Benign

DBA

IBA

(3) IBA's imperceptibility and durability evaluation. Besides being imperceptible to the backdoor defenses, IBA is expected to be visually imperceptible and deliver prolonged backdoor durability. The visual imperceptibility of IBA is illustrated in Figure 5. We evaluate the behavior of backdoor-injected models against GradCam [23], a widely known visualization tool and helpful in understanding the neural network's behavior. Figure 5 shows that the visualization heat maps created by IBA attacks are nearly identical across the clean and backdoor images. From Figures 5, we can observe that the

backdoored images have negligible deviations from the Grad-CAM behaviors on the clean images for both black-and-white and colored images. On the other hand, backdoored images using patched triggers generate a considerable difference in visualization heat maps (DBA [32]). Because IBA is based on input perturbation, the image is formed by adding very little perturbation to the clean image; consequently, the difference in the latent space where the heat maps are generated is likewise minimal. Aside from that, the patch backdoor (such as the pattern in the upper-left corner) is apparent, resulting in a significant distinction between the backdoor and clean versions. We also evaluate the backdoor generated by IBA and another attack scheme in FL, i.e., DBA with Neural Cleanse [27], a widelyused backdoor model mitigation method based on the pattern optimization approach. Neural Cleanse assumes that the backdoor trigger is patch-based, which makes it suitable for evaluating the proposed method. For each image label, Neural Cleanse identifies if there is a patch pattern that produces a misclassification result for that target label. If any class label yields a significantly smaller pattern, Neural Cleanse considers it a sign of a potential backdoor. Neural Cleanse quantifies such deviations from the optimal patch of each class label by using the Anomaly Index metric. If the Anomaly Index is less than a threshold of 2 for a class, Neural Cleanse considers that there is a backdoor with this class as the target label. Since IBA does not generate the patch-based trigger pattern, it can efficiently bypass this defense by having Anomaly Index much lower than DBA (Figure 7), which indicates the superior stealthiness of backdoor attacks with adversarial triggers compared to the patch-based.

As previously stated, strong attacks should be successful even if the attacker only participates a few times. As a result, we construct a scenario to assess the endurance of IBA compared to a chosen baseline: DBA. We employ the same configuration as the original work DBA but limit each of the four malicious clients' participation frequency to 10 without utilizing any scaling strategy. Regarding IBA, we apply the dimension-based model poisoning mentioned above and use one malicious client with the same frequency. As observed in Figure. 6a, after A is removed from the training (i.e., at round 200), the BA of DBA reduces gradually while the accuracy drop of IBA is almost negligible. As demonstrated in Figure 6b and Table 3, IBA delivers superior backdoor efficiency and durability within the same attacking duration. After round 400, the BA of DBA drops quickly compared to IBA. We can see the most significant difference towards the end of the comparison duration, i.e., at round 700, the BAs of IBA and DBA are ∼ 85% and ∼ 45% , respectively. After 250 rounds of elimination, the BA of DBA retains 38.76% of its peak value, but the comparable value of IBA is 99.14%. Furthermore, while the BA can preserve more than 60% of its peak value with the CIFAR-10 dataset, the highest accuracy it can attain is significantly lower than IBA, i.e., less than 20%, which is consistent with the prior conclusion of the original when the scaling factor is 1. In conclusion, IBA can achieve better durability and imperceptibility than SOTA attacks.

5 Conclusion

In conclusion, we have presented the IBA , a novel backdoor attack framework for FL that overcomes the limitations of existing approaches. Our framework combines a learnable trigger-generating function with a gradual implantation process, resulting in an attack that can evade detection and achieve high success rates. The selective poisoning of model parameters and the constraint on poisoned updates enhance the attack's efficiency and durability. Our work demonstrates promising results and suggests several directions for future research. Robust defense mechanisms against FL backdoor attacks must be developed, considering the unique challenges of the distributed and decentralized learning process. Generalizing our approach to diverse FL settings, such as heterogeneous or noni.i.d. data distributions, would provide a comprehensive understanding of the attack's effectiveness. Exploring alternative triggers and their impact on stealthiness and success rates could yield valuable insights. Evaluating the framework on larger-scale FL systems and real-world scenarios will assess its scalability and practicality. Advancing adversarial defense strategies, including proactive detection and mitigation techniques, will contribute to more secure and trustworthy FL systems. These future research directions will contribute to strengthening the security of FL against backdoor attacks.

References

Manaar Alam, Esha Sarkar, and Michail Maniatakos. Perdoor: Persistent non-uniform backdoors in federated learning using adversarial perturbations. ArXiv preprint , abs/2205.13523, 2022.

Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In Silvia Chiappa and Roberto Calandra, editors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy] , volume 108 of Proceedings of Machine Learning Research , pages 2938-2948. PMLR, 2020.

Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 119-129, 2017.

Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. ArXiv preprint , abs/1712.05526, 2017.

Khoa Doan, Yingjie Lao, and Ping Li. Backdoor attack with imperceptible input and latent modification. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 18944-18957, 2021.

Khoa Doan, Yingjie Lao, Weijie Zhao, and Ping Li. LIRA: learnable, imperceptible and robust backdoor attacks. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pages 11946-11956. IEEE, 2021.

Khoa D Doan, Yingjie Lao, and Ping Li. Marksman backdoor: Backdoor attacks with arbitrary target class. In Advances in Neural Information Processing Systems , volume 35, pages 3826038273, 2022.

Clement Fung, Chris JM Yoon, and Ivan Beschastnikh. The limitations of federated learning in sybil settings. In RAID , pages 301-316, 2020.

Malhar S Jere, Tyler Farnan, and Farinaz Koushanfar. A taxonomy of attacks on federated learning. IEEE Security & Privacy , 19(2):20-28, 2020.

Shaofeng Li, Minhui Xue, Benjamin Zi Hao Zhao, Haojin Zhu, and Xinpeng Zhang. Invisible backdoor attacks on deep neural networks via steganography and regularization. IEEE Transactions on Dependable and Secure Computing , 18(5):2088-2105, 2020.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Inderjit S. Dhillon, Dimitris S. Papailiopoulos, and Vivienne Sze, editors, Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020 . mlsys.org, 2020.

Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pages 16443-16452. IEEE, 2021.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Aarti Singh and Xiaojin (Jerry) Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA , volume 54 of Proceedings of Machine Learning Research , pages 1273-1282. PMLR, 2017.

Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning. In Proceedings of the 2019 IEEE Symposium on Security and Privacy (SP) , pages 1-15, 2018.

Thien Duc Nguyen, Phillip Rieger, Roberta De Viti, Huili Chen, Björn B Brandenburg, Hossein Yalame, Helen Möllering, Hossein Fereidooni, Samuel Marchal, Markus Miettinen, et al. { FLAME } : Taming backdoors in federated learning. In 31st USENIX Security Symposium (USENIX Security 22) , pages 1415-1432, 2022.

Thuy Dung Nguyen, Tuan Nguyen, Phi Le Nguyen, Hieu H Pham, Khoa D Doan, and Kok-Seng Wong. Backdoor attacks and defenses in federated learning: Survey, challenges and future research directions. Engineering Applications of Artificial Intelligence , 127:107166, 2024.

Tuan Anh Nguyen and Anh Tuan Tran. Wanet - imperceptible warping-based backdoor attack. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

Mustafa Safa Özdayi, Murat Kantarcioglu, and Yulia R. Gel. Defending against backdoors in federated learning with robust learning rate. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021 , pages 9268-9276. AAAI Press, 2021.

Krishna Pillutla, Sham M Kakade, and Zaid Harchaoui. Robust aggregation for federated learning. IEEE Transactions on Signal Processing , 70:1142-1154, 2022.

Nuria Rodríguez-Barroso, Daniel Jiménez-López, M Victoria Luzón, Francisco Herrera, and Eugenio Martínez-Cámara. Survey on federated learning threats: Concepts, taxonomy on attacks and defences, experimental study and challenges. Information Fusion , 90:148-173, 2023.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020 , pages 11957-11965. AAAI Press, 2020.

Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017 , pages 618-626. IEEE Computer Society, 2017.

Virat Shejwalkar, Amir Houmansadr, Peter Kairouz, and Daniel Ramage. Back to the drawing board: A critical evaluation of poisoning attacks on production federated learning. In 2022 IEEE Symposium on Security and Privacy (SP) , pages 1354-1371. IEEE, 2022.

Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H. B. McMahan. Can you really backdoor federated learning? ArXiv preprint , abs/1911.07963, 2019.

Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. ArXiv preprint , abs/1912.02771, 2019.

Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. 2019 IEEE Symposium on Security and Privacy (SP) , pages 707-723, 2019.

Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy-yong Sohn, Kangwook Lee, and Dimitris S. Papailiopoulos. Attack of the tails: Yes, you really can backdoor federated learning. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Chen Wu, Xian Yang, Sencun Zhu, and Prasenjit Mitra. Mitigating backdoor attacks in federated learning. ArXiv preprint , abs/2011.01767, 2020.

Chulin Xie, Minghao Chen, Pin-Yu Chen, and Bo Li. CRFL: certifiably robust federated learning against backdoor attacks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 11372-11382. PMLR, 2021.

Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li. DBA: distributed backdoor attacks against federated learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020.

Jiale Zhang, Junjun Chen, Di Wu, Bing Chen, and Shui Yu. Poisoning attack in federated learning using generative adversarial nets. In 2019 18th IEEE International Conference On Trust, Security And Privacy In Computing And Communications/13th IEEE International Conference On Big Data Science And Engineering (TrustCom/BigDataSE) , pages 374-380. IEEE, 2019.

Zhengming Zhang, Ashwinee Panda, Linyue Song, Yaoqing Yang, Michael W. Mahoney, Prateek Mittal, Kannan Ramchandran, and Joseph Gonzalez. Neurotoxin: Durable backdoors in federated learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 26429-26446. PMLR, 2022.

Shihao Zhao, Xingjun Ma, Xiang Zheng, James Bailey, Jingjing Chen, and Yu-Gang Jiang. Clean-label backdoor attacks on video recognition models. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020 , pages 14431-14440. IEEE, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a two-phased backdoor injection framework, called IBA, for Federated Learning systems. IBA incorporates an adaptive trigger generation mechanism along with a gradual implantation process to insert stealthy backdoors into the global model. IBA enhances the efficiency and durability of the attack through selective poisoning of specific model parameters. Through evaluation using multiple datasets, IBA demonstrates high success rates and outperforms existing backdoor injection methods even in the presence of several defense techniques.\n\nSTRENGTHS:\nThe paper is well-written and easy to follow. It addresses a crucial problem in federated learning.\n\nWEAKNESSES:\nThe paper has several limitations which are discussed below:\n\nThe primary contribution of the paper is to use adversarial examples for generating adaptive backdoor triggers and selective parameter poisoning for injecting durable backdoors. However, a quick search resulted in the following two papers:\n\n[1] M. Alam et al., \"PerDoor: Persistent Non-Uniform Backdoors in Federated Learning using Adversarial Perturbations\", arXiv 2022.\n\n[2] Z. Zhang et al., \"Neurotoxin: Durable Backdoors in Federated Learning\", ICML 2022.\n\nIt is evident from these papers that [1] uses adversarial examples as backdoor triggers, and [2] uses selective parameter poisoning for durable backdoors. On top of that, [2] performs similar operations as the authors do to increase the durability of backdoors. The authors should elaborate on the distinctiveness and novelty of their approach compared to these preceding works.\n\nIBA utilizes adversarial examples as backdoor triggers, which are known for their input specificity. The paper could benefit from a more comprehensive explanation of how IBA addresses this specificity—questions such as whether triggers are unique to each input or shared across a class. Also, it needs to be elaborated more on whether the trigger generation function must be trained at each training round. If so, what is the computational overhead for that?\n\nIBA is not evaluated against important state-of-the-art defenses like FLAME[3], SparseFed[4], etc.\n\nIBA should be compared against other state-of-the-art attacks like 3DFed[5].\n\nIBA should be evaluated using more practical benchmark federated learning datasets in the LEAF project[6].\n\nThe evaluation of IBA considering the fixed-pool case should be included as the main content of the paper.\n\nThe reported accuracy of 98.09% on the CIFAR-10 dataset and the subsequent 14% drop in accuracy in Table 1 raise skepticism, especially in comparison to the marginal drop observed for T-Imagenet. The authors should provide clarification and insights into this discrepancy. In addition, there appears to be a mismatch between the content of Table 1 and its explanation in Section 4.2 (1). The authors need to ensure consistency and clarity in the representation and discussion of results.\n\nThe authors should provide a detailed explanation for specific results, such as why the combination of IBA and PGD yields an accuracy of 18.74% for MNIST and RLR in Table 2 and other similar cases. With such low accuracy, why is IBA deemed to bypass every defense mechanism?\n\nThe authors mention that \"This trigger is expected to bypass human and machine inspections\" but fail to discuss the metrics or criteria used for this assessment.\n\nFigure 1 lacks an adequate explanatory description. A more thorough discussion will enhance the clarity and comprehensibility of the figure.\n\nTo provide more insights, it would be beneficial to depict results on more complex color images in Figure 4 instead of using a monochromatic MNIST dataset.\n\nThe authors should discuss the significance of using FedProx and FedNova, and clearly describe how these differ from FedAvg. This would help in understanding the rationale behind their inclusion.\n\nWhat does the anomaly index represent in Figure 6?\n\nNeuralCleanse is mentioned as a defense mechanism against backdoor attacks in FL, but it was not originally designed for the FL framework. Clarifying its relevance and applicability in this context would be beneficial.\n\n[3] T. Nguyen et al., \"FLAME: Taming Backdoors in Federated Learning\", Usenix Security 2022.\n\n[4] A Panda et al., \"SparseFed: Mitigating Model Poisoning Attacks in Federated Learning with Sparsification\", AISTATS 2022.\n\n[5] H. Li et al., \"3DFed: Adaptive and Extensible Framework for Covert Backdoor Attack in Federated Learning\", IEEE S&P 2023.\n\n[6] https://leaf.cmu.edu/\n\nQUESTIONS:\nPlease address the issues discussed in Weaknesses.\n\nLIMITATIONS:\nThe authors have discussed the limitations and potential negative societal impacts of the paper. However, the paper has several other limitations, as discussed in the review.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies backdoor attacks in federated learning setting. They propose a new backdoor attack method that is based on sample-specific trigger, optimized with constraints on weight norm and weight dimension, to achieve more stealthy, harder to detect and more durable. U-Net is trained to generate sample-specific trigger and the stealthiness of the trigger is controlled by $\\epsilon$, which also controls the balance between attack success and trigger stealthiness. To make the attack harder to detect and more durable, the gradients are projected to the ball around the global model weights and optimization is constrained to dimensions that are historically infrequently updated. Experiments on CIFAR10, MNIST and TinyImageNet show that the proposed method is effective against multiple state-of-the-art defense methods.\n\nSTRENGTHS:\n- The paper is well organized and the ideas are clearly presented. \n\n- Experiments show that the method can defend against multiple state-of-the-art defense methods.\n\nWEAKNESSES:\n- Even though the proposed method is compared with DBA to show its durability, the paper lacks comparison with other state-of-the-art attack methods. Experiments comparing multiple attack methods against state-of-the-art defense methods on different types of datasets should be conducted to show the effectiveness of the method. \n\n- As for the poisoning dimension restriction, the idea of restricting updates to dimensions that are historically seldomly updated is similar to that of Neurotoxin. Why compare with DBA which does nothing to promote the durability of the attack? Why not compare with Neurotoxin? \n\n-  Generally speaking, the data distribution of clients may affect the performance of attack and defense. Dir(0.5) and Dir(0.01) are used for MNIST, CIFAR10/TinyImagenet. Experiments studying the effect of data distribution could be helpful understand the method better.\n\n- Based on Figure 2 and table 2, it seems that the method does not do a good job against Krum and RLR. The attack accuracy of the method keeps dropping when tested against the two defense methods.\n\nQUESTIONS:\nSee questions and concerns in the previous part.\n\nLIMITATIONS:\nThe main limitation is the lack of experiments to support the claim. See details in weakness part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a backdoor attack framework (IBA) in Federated Learning for trigger based backdoors that jointly learns a generative model for stealthy visual triggers while also planting the backdoor in the global model. They evaluate their attack on MINST, CIFAR-10, and TinyImageNet and show that it bypasses several known defenses such as KRUM, NC, RFA etc.\n\nSTRENGTHS:\n1. The experimental setup is quite thorough, considering several datasets, settings, and defenses.\n2. The attack appears to bypass all the considered defenses with good effectiveness and stealth.\n3. The work acknowledges past work well with a good description of existing research.\n\nWEAKNESSES:\n1. While the attack is titled \"Irreversible Backdoor attack\", the longevity of the attack is not well studied. Figure 5 is only for the MNIST dataset and most other experiments consider fixed-frequency attack. I would like to see a better analysis of how long the attack takes to be removed by clean training.\n2. I would like to see a comparison to even trigger-less backdoor attacks since the PGD model replacement attack has already been well studied in Bagdasaryan et al. and Wang et al. I would expect this to be much stronger, but it would act as a good baseline. While there are several ablations, I would like to see comparison to more existing baselines. Perhaps even BadNets [Gu et al.].\n\nQUESTIONS:\n1. In (2), the adversarial perturbation is bounded in the $\\lVert \\cdot \\rVert_{\\infty}$ norm. Can this be translated to the 2-norm? Naively, speaking the same attack budget will get scaled down by $\\sqrt{d}$, which will reduce the effectiveness of the attack. Have you tried this?\n2. It is quite interesting that the optimal choice of scaling factors is $\\alpha = \\beta = 0.5$. Are the scales of the two losses very similar?\n3. How is $\\lambda_{\\xi}$ chosen?\n4. Please exwhat plain acronyms in the table captions. Such as MA, BA.\n5. Please explain bold entries in the tables mean. It can be inferred, but would be nice not to have it.\n6. Section 4.2 is quite hard to understand especially since accuracies are listed as \"ranging\" across datasets. This is quite unusual and doesn't really make sense. \n7. What is MR?\n8 . It is hard to evaluate the performance of IBA from Table 2 without comparing against other attacks.\n9. It seems like for some defenses (like KRUM), IBA + PGD outperforms IBA + PGD + MR. Whereas in RLR, it gets worse. Can you explain this behavior? Is one supposed to choose the attack based on a defense?\n10. In Figure 3, is the attack at round 0?\n11. Can you specify the choice of $\\varepsilon$ for the PGD attack?\n12. Some citations are duplicated (McMahan et al., Wang et al)\n\nLIMITATIONS:\nI would strongly recommend the authors include a section listing the limitations of their work (comparison with more baselines, longevity of attack, need for whitebox access etc)", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
**Key Contribution:** This work proposes a two-staged model poisoning attack which works even with only the participation of a small number of malicious clients. Moreover, the proposed method is effective and robust against existing defenses.
The attack is evaluated on a variety of existing defenses on CIFAR-10, MNIST, and Tiny-Imagenet.

STRENGTHS:
**Novelty and Significance**
The paper’s two-stage approach is unique and powerful. Learning a generative model for adversarial noise is additionally alarming as it is difficult to detect and defend against. 

**Clarity:**
The paper is well-written and clear - the mathematical formulations are simple and easy to follow, with intuitive results. Moreover, the proposed augmentations to the base attack which help evade existing defenses are methodical and clearly motivated.

WEAKNESSES:
**Limited Evaluation:** The paper could be stronger by presenting more results under different numbers of clients. However, despite this weakness, I am overall satisfied with the quality and comprehensiveness of the evaluation.

QUESTIONS:
- Have you considered different choices of trigger generation models?

- What was the distribution of data considered across the clients during evaluation? Have you considered evaluating with non-iid data?

LIMITATIONS:
The authors have adequately discussed limitations and potential social impacts of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07