SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dung Thuy Nguyen $^{1,2*}$ , Tuan Nguyen $^{2,3}$ , Tuan Anh Tran $^{4}$ , Khoa D Doan $^{2,3}$ , Kok-Seng Wong $^{2,3}$
$^{1}$  Department of Computer Science, Vanderbilt University, Nashville, TN 37212, USA
$^{2}$ VinUni-Illinois Smart Health Center, VinUniversity, Hanoi, Vietnam
<sup>3</sup>College of Engineering & Computer Science, VinUniversity, Hanoi, Vietnam
$^{4}$ VinAI Research, Hanoi, Vietnam
dung.t.nguyen@Vanderbilt.Edu, tuan.nm@vinuni.edu.vn,
v.anhtt152@vinai.io, khoa.dd@vinuni.edu.vn, wong.ks@vinuni.edu.vn
Federated learning (FL) is a distributed learning approach that enables machine learning models to be trained on decentralized data without compromising end devices' personal, potentially sensitive data. However, the distributed nature and uninvestigated data intuitively introduce new security vulnerabilities, including backdoor attacks. In this scenario, an adversary implants backdoor functionality into the global model during training, which can be activated to cause the desired misbehaviors for any input with a specific adversarial pattern. Despite having remarkable success in triggering and distorting model behavior, prior backdoor attacks in FL often hold impractical assumptions, limited imperceptibility, and durability. Specifically, the adversary needs to control a sufficiently large fraction of clients or know the data distribution of other honest clients. In many cases, the trigger inserted is often visually apparent, and the backdoor effect is quickly diluted if the adversary is removed from the training process. To address these limitations, we propose a novel backdoor attack framework in FL, the Irreversible Backdoor Attack (IBA), that jointly learns the optimal and visually stealthy trigger and then gradually implants the backdoor into a global model. This approach allows the adversary to execute a backdoor attack that can evade both human and machine inspections. Additionally, we enhance the efficiency and durability of the proposed attack by selectively poisoning the model's parameters that are least likely updated by the main task's learning process and constraining the poisoned model update to the vicinity of the global model. Finally, we evaluate the proposed attack framework on several benchmark datasets, including MNIST, CIFAR-10, and Tiny ImageNet, and achieved high success rates while simultaneously bypassing existing backdoor defenses and achieving a more durable backdoor effect compared to other backdoor attacks. Overall, IBA offers a more effective, stealthy, and durable approach to backdoor attacks in FL.
Federated Learning (FL) [13] is a novel distributed learning paradigm that enables multiple parties to train machine learning models collaboratively without centralized data sharing. Specifically, each participant trains a local model on their private data, and only the model updates are exchanged with a central server for aggregation. However, FL also introduces new security and privacy challenges
*Work done while DTN was at VinUniversity.
$^{2}$ Code for this paper is published at https://github.com/sail-research/iba.
that need to be addressed to ensure the confidentiality and integrity of the learning process [33, 24], such as model poisoning, membership inference, data leakage, and Sybil attacks [9, 20].
Backdoor attacks are a specific form of model poisoning in which an attacker inserts a backdoor trigger during the training process that can later be exploited to manipulate the model's behavior [4, 12, 17, 6]. FL is particularly vulnerable to backdoor attacks since each participant trains a local model on their private data, and the central server aggregates the model updates without validating their integrity. An attacker can inject poisoned data or backdoor triggers into the local models of a subset of participants. When the models are aggregated, the backdoor trigger can be activated to compromise the model's behavior [2, 28, 32].
Artificial backdoor attacks are a standard method for backdoors in FL. In an artificial backdoor attack, an apparent trigger, such as a specific pixel value, is injected into the local training data [25, 32]. Although meaningless in the data context, the apparent trigger causes the trained model to associate a signal for the model to optimize during training. When trained on such data, the model will induce misclassifications toward a target class when making predictions on inputs with the presence of the trigger. Existing artificial backdoor attacks require the involvement of a portion of compromised clients and need assistance from scaling-based methods to achieve high attack success rates [32, 24]. This makes these attacks vulnerable to mainstream backdoor defenses [30, 31, 15]. Furthermore, most existing attacks are not durable (i.e., when the adversary is removed from the FL process, the backdoor effect is gradually diluted [34]), especially when the size of compromised clients is small, and they do not regularly participate in the FL training.
In this paper, we exploit the principle of adversarial attacks to craft a novel instance-specific backdoor attack, called Irreversible Backdoor Attack (IBA), that only requires a small number of compromised clients but achieves more robust attack performance and durability. IBA is a two-phase attack scheme, including Trigger Generating and Backdoor Injection. During training rounds, the former manipulates the attack model, which is a generative model, so the triggers generated by this attack model can deceive the local model into misbehaving. After that, we formulate the Backdoor Injection phase as a constrained optimization problem. Then, we propose a simple but effective recall to keep the attack model updated regularly. This algorithm allows the adversary to achieve high attack success rates even when a small number (i.e., 1) of clients are compromised. In addition, we enhance the durability and effectiveness of the proposed attack by selectively poisoning the model's parameters that are least likely updated by the main task's learning process and constraining the poisoned model update to the vicinity of the global model, respectively. This strategy helps IBA bypass mainstream defenses in FL and achieve extended backdoor durability.
- Proposal of a two-phase backdoor injection mechanism, IBA, in FL: The paper introduces IBA, which combines the manipulation of the trigger generator and the poisoning of the local model into a unified two-phase approach. IBA constrains the poisoned model update to be within the vicinity of the global model to improve the practicality and effectiveness of the attack.
- Proposal of selective parameter poisoning in the constrained optimization algorithm: The paper enhances the durability of IBA by selectively poisoning the model's parameters less likely to be updated during the main task's learning process.
- State-of-the-art attack performance and stealthiness: The paper demonstrates that IBA achieves state-of-the-art attack performance and durability and exhibits high stealthiness against existing defense mechanisms. This highlights the robustness and effectiveness of IBA as a practical backdoor attack method.
Backdoor Attacks in FL. Backdoor attacks in FL have been extensively studied, revealing vulnerabilities and proposing various attack strategies [16]. Backdoor attackers generally aim to incorporate concealed backdoors into Deep Neural Networks (DNNs) during the training phase. This allows the compromised DNNs to exhibit normal behavior when presented with benign samples. However, their predictions will be deliberately and consistently altered when triggered by specific patterns defined by the attacker. One of the most commonly employed methods for incorporating backdoor
functionality during training is poisoning the training samples [2, 32]. Bagdasaryan et al. [2] also proposed model replacement to implant backdoors into the joint model and demonstrated that even a single-round attack with a small fraction of clients could achieve a  $100\%$  Attack Success Rate (ASR). Xie et al.[32] investigated Distributed Backdoor Attacks (DBA), which decompose the global trigger into non-overlapping local triggers. DBA exhibits higher ASR and increased robustness against FL aggregation methods. Additionally, backdoor triggers in machine learning can be designed to be imperceptible in the input space [10, 12, 17, 6] and in the latent space [5], or to possess sophisticated, multi-target payloads [7]; the ground-truth labels of the poisoned samples can also align with the intended target label [26, 22, 35]. These characteristics contribute to the stealthiness of backdoor attacks. Recently, Neurotoxin [34] proposed to enhance backdoor durability by targeting specific coordinates less likely to be updated by benign agents, partially poisoning the model by projecting gradients onto a coordinate-wise constraint. This extends the backdoor's impact and prevents catastrophic forgetting, strengthening the effectiveness of the attack. The recent work that is most aligned with ours is PerDoor [1], which uses the Basic Iterative Method (BIM) to generate the trigger, which relies on gradients of the model's loss function concerning the input data to alter the original inputs into backdoored inputs. However, this method fails to learn underlying patterns and distributions from the training dataset and lacks a sharing and collaborating mechanism among multiple malicious clients, which is very beneficial for the adversary in collaborative attacks.
Defense Mechanisms against Backdoor Attacks in FL. The mainstream backdoor-attack defenses in FL aim to reduce the effect on the global model, as it has been shown that detecting backdoors is challenging without breaking the privacy of participants [14]. For instance, Krum [3] aggregates local models by choosing a single local model as the aggregated model with the smallest Euclidean distance to a certain fraction of other models. Norm difference clipping (NDC)[25] examines the norm-difference between the global model sent to and model updates shipped back from selected clients, using a pre-specified threshold to clip model updates that exceed the norm difference. RFA[19] aggregates local models by computing a weighted geometric median using the smoothed Weiszfeld's algorithm. Ozdayi et al. [18] proposed a defense mechanism RLR, that adjusts the learning rate of the aggregation server in FL based on the sign information of agents' updates by carefully considering it per dimension and round. These defense mechanisms offer valuable insights and strategies for countering backdoor attacks in FL settings. Another approach is identifying suspicious clients and removing them from the aggregation process. FoolsGold [8] assumes that benign datasets from different clients differ from each other and assigns low weights to models that are similar to many other models, but it can be vulnerable to sophisticated adversaries submitting poisoned updates without raising suspicion. FLAME [15] uses the HDBSCAN algorithm to detect malicious updates, combines model filtering with poison elimination to detect and remove malicious updates, and is robust against inference attacks.
3 Irreversible Backdoor Attack (IBA) in FL
FL Framework. The training process in the standard FL framework involves the following steps:
Step 0 (FL Initialization): The central server  $S$  initializes the weight of the global model  $w$  and sets hyperparameters such as the number of FL rounds and local epochs.
Step 1 (Local Model Training and Update): Selected clients  $\mathcal{C}_1, \mathcal{C}_2, \dots, \mathcal{C}_K$  receive the current global weight  $w_0$  from  $S$ . Each client  $\mathcal{C}_i$  updates its local model parameters  $w_i$  using its local dataset  $\mathcal{D}_i$ . After local training, the clients send their local weights to  $S$  for model aggregation.
Step 2 (Global Model Aggregation and Update):  $S$  aggregates the received local weights and computes the aggregation result. The aggregated result is sent back to the clients for the next round of training.  $S$  aggregates the received local model weights using the FedAvg [13] formula:
$$
w = \frac {1}{| \mathcal {C} _ {k} |} \sum_ {k = 1} ^ {K} | \mathcal {C} _ {k} | \cdot w _ {k}
$$
where  $|\mathcal{C}_k|$  represents the number of data samples held by client  $\mathcal{C}_k$ .
Attacker Threat. To address the weakness of existing works, i.e., the adversary  $\mathcal{A}$  controls a number of compromised clients, we focus on the scenario that  $\mathcal{A}$  controls only one client  $k$  and it will participate in the training for each  $f$  rounds, i.e., fixed-frequency attack [28].
Figure 1: Irreversible Backdoor Attack Scheme. IBA includes two training phases: (1) Trigger Generating Function and (2) Backdoor Injection. In the first phase, the trigger generation model  $\mathcal{G}_{\xi}$  is updated to generate adversarial noise. This noise is combined with the original image  $x$  to create a backdoor sample to fool the local model  $\mathbf{w}_i^t$ . In the second phase, the local model  $f_{w_k}$  is updated to perform indifferently on clean sample  $x$  but distorts its prediction on the backdoor image  $T_{\xi}(x)$  to the target class  $y_T$ , then this poisoned model is sent to the aggregator. Model poisoning technique is leveraged to enhance the stealthiness and durability of the attack.
Adversary Capability: We follow the same assumption used in the existing works on backdoor attacks in FL [2, 15], where adversary  $\mathcal{A}$  has total control over the malicious participant(s). However,  $\mathcal{A}$  has no control or access to other benign participants' data or local updates. Also, we assume  $\mathcal{A}$  thoroughly understands the central server and any potentially deployed defenses but cannot alter the server's configuration parameters and algorithms.
Adversary Objectives: The goal of  $\mathcal{A}$  is learning a trigger generator  $\mathcal{G}_{\xi}$  such that its outputted triggers can activate the backdoor function to mislead the global model  $\mathcal{F}(w)$  while ensuring the behaviors on clean samples are unaffected. This trigger is expected to bypass human and machine inspections. Moreover,  $\mathcal{A}$  can bypass configured defenses in FL and prolong the backdoor effect even when  $\mathcal{A}$  is removed from the training process. The overall mechanism of our proposed backdoor attack IBA is illustrated in Figure 1.
3.1 The Trigger Generating Function
In FL, when clients actively engage in a training round, they have the opportunity to observe and influence the changes in the global model  $w$ . In particular, a client  $k$  receives the global model  $w_{t-1}$  and updates it using their local data to produce a new local model, denoted as  $w_k$ . After local training, the learned surrogate model is aggregated to partially reflect the subsequent global model. In light of this process, we propose an initial phase called the Trigger Generating. In this phase, the trigger generation model  $\mathcal{G}_{\xi}$  is updated to generate adversarial noise. This noise is combined with the original image  $x$  to create a backdoor sample. This sample remains imperceptibly different from the original, but it effectively misleads the model  $\mathcal{G}_{\xi}$  into making incorrect classifications for the target class. The rationale behind this approach is that manipulating  $\mathcal{G}_{\xi}$  based on this local model allows the adversary  $\mathcal{A}$  to optimally adapt the generative function, even in the face of round-by-round variations in the global model. The transformation function is inspired by adversarial examples, in which we model it as a perturbation on the input, as follows:
$$
T _ {\xi} (x) = x + \mathcal {G} _ {\xi} (x), | | \mathcal {G} _ {\xi} (x) | | _ {\infty} \leq \epsilon , \forall x \tag {1}
$$
The function  $\mathcal{G}_{\xi}$  takes an input  $x$  and generates adversarial noise on the same input space, guaranteeing the backdoor attack's stealthiness.
To achieve this goal, we utilize the local model as the surrogate model to update the attack model  $\mathcal{G}_{\xi}$  using the following objective function:
$$
\xi \leftarrow \xi - \eta_ {\xi} \sum_ {x \in \mathcal {D}} \mathcal {L} _ {\xi} \left(f _ {w _ {k}} \left(T _ {\xi} (x)\right), y _ {T}\right), \tag {2}
$$
in which  $f_{w}$  is the trained local model and  $y_{T}$  is the targeted label. It's worth noting that the local model  $f_{w}$  submitted to the server will be totally vanilla during this phase.
While considering various norms for bounding the generator noise in Equation 1, such as the  $L_{2}$  norm, it's generally inadvisable for backdoor attacks. This infinity norm guarantees a widespread distribution of the generated trigger across the input image (encompassing all pixels in the trigger); in contrast, the  $L_{2}$  norm can result in localized artifacts within the image (with only select pixels forming the trigger). Employing the  $L_{2}$  norm can make the backdoor attack more susceptible to detection by trigger-synthesis defenses like Neural Cleansse [27]. Thus, the  $\| .\|_{\infty}$  norm is a preferred choice.. We can design such generator function as an auto-encoder or the more complex U-Net architecture [21]. However, we observe no significant performance difference between a simple auto-encoder and U-Net by training the generator function and the classifier with the proposed training algorithm.
In the proposed transformation function, parameter  $\epsilon$  controls the visual stealthiness of the triggers. For instance, even on the gray-scale MNIST dataset, if  $\epsilon$  is less than 0.01, there is usually no discernible difference between the clean and disturbed images. A greater value for  $\epsilon$ , i.e., more obvious triggers, simplifies generative model learning. On the other hand, a too-small  $\epsilon$  may make learning a generative model challenging since it is heavily dependent on the variance of the surrogate model, which varies across FL training rounds. Therefore, during this phase, we keep the  $\epsilon$  sufficiently large, i.e.,  $\epsilon_0$ , until we reach the expected attack success rate.
3.2 The Backdoor Injection Process
At the second phase aims to gradually insert the backdoor function into the global model by manipulating the poisoned local model. Our objective is to learn classifier (i.e., local model)  $f_{w_k}$  that concurrently performs indifferently on  $x$  in comparison to the classifier's clean version but distorts its prediction on the backdoor image  $T_{\xi}(x)$  to the target class  $y_T$ . The above task can be formulated as the following objective function:
$$
\min  _ {w} \sum_ {x \in \mathcal {D}} \alpha \mathcal {L} _ {\text {c l e a n}} \left(f _ {w _ {k}} (x), y _ {x}\right) + \beta \mathcal {L} _ {\text {p o i s o n}} \left(f _ {w _ {k}} \left(T _ {\xi} (x)\right), y _ {T}\right) \tag {3}
$$
The classification model  $f_{w}$  in the above problem is trained based on local data and the optimal function  $\mathcal{G}_{\xi}$  learned in the previous phase. When training the classifier, the parameters  $\alpha$  and  $\beta$  regulate a mixture of the strengths of the loss items from clean and poisoned data, i.e.,  $\mathcal{L}_{clean}$  and  $\mathcal{L}_{poison}$ , respectively. If  $\alpha$  is more significant than  $\beta$ , the classifier's performance on clean data rapidly converges to the optimal performance of the vanilla classifier, according to our experiments. Otherwise, the classifier's performance on the backdoor data quickly reaches the optimal value. To compensate for the performance of the local model on both the main task and the backdoor task, we assume  $\alpha = 0.5$  and  $\beta = 0.5$  in the remaining part of the paper. As previously stated, the value of  $\epsilon$  must be assigned cautiously to overcome the unpredictability of the global model and assist the local model in learning the generated triggers more effectively. To achieve this goal, we propose a self-adjusting value for  $\epsilon$  from the starting round of this phase, i.e.,  $t_{I}$ :
$$
\epsilon_ {t} = \max  \left(\hat {\epsilon}, \epsilon_ {0} * \left(1. 0 - \lambda_ {\xi}\right) ^ {t - t _ {I}}\right), \tag {4}
$$
where the value of  $\epsilon$  is gradually decayed until it reaches objective  $\hat{\epsilon}$ .
Attack Model Retraining. During this phase, we propose frequently updating the attack model  $\mathcal{G}_{\xi}$  to keep it in sync with the state of the global model. The retraining frequency should be chosen (20, 50).
Partial Model Poisoning. To enhance the longevity and effectiveness of the attack, we propose to partially poison the model based on selected poisoning space and poisoning dimension. In which space-based poisoning helps the attack bypass the backdoor defenses because it constrains the
poisoned model such that it does not go far from the vicinity of the benign model, while dimension-based poisoning helps to reduce dilution effect from benign clients by shrinking the poisoned neurons to infrequently updated neurons.
Poisoning-space based. Under this attack, the submitted local model is partially contaminated by being constrained by the magnitude of its parameters. Specifically, the adversary applies projected gradient descent on the losses for  $\mathcal{D} \cup \mathcal{D}'$ , where  $\mathcal{D}, \mathcal{D}'$  are benign and poisoned datasets, respectively. In other words, it periodically projects the model parameter on the ball centered around the global model of the previous iteration. This enables IBA to circumvent defenses based on norm and distance investigation by preventing the resulting poisoned model from deviating substantially from the original. Mathematically, this method guarantees that  $||w_k - w|| \leq \delta$ . Note that, this attack strategy can be combined with model replacement [2], where the model parameter is scaled before being sent to the server to cancel the contributions from the other honest clients [28]. The scaled model is calculated by the following:
$$
\hat {w} _ {k} \simeq w _ {k} + \sum_ {i \in \mathcal {C} \backslash k} \frac {n _ {i}}{n _ {C \backslash k}} \left(w _ {k} - w ^ {*}\right). \tag {5}
$$
Poisoning-dimension based. Because the attacker cannot participate in every round of training, directly submitting the poisoned model to the server is not effective [31]. To address it, we propose leveraging historical gradient variation to select infrequently updated neurons, which are considered poisoning dimensions for IBA. Then the adversary computes bottom- $k\%$  coordinates of the gradient vector  $g$ , obtained by learning the main task on the clean data  $\mathcal{D}$ . In particular,
$$
\beta_ {t} \leftarrow \frac {p - 1}{p} * \beta + \frac {1}{p} * \beta_ {t} \quad s. t., \quad \beta_ {t} = b o t t o m _ {k} (g _ {t}), \tag {6}
$$
in which  $p$  refers to the number of rounds up to round  $t$  that adversary  $\mathcal{A}$  has been selected to participating in FL training. After that, when finishing calculating gradient  $\hat{g}$  of local model  $f_{w}$  using Equation. 3,  $\mathcal{A}$  projects gradient onto coordinate-wise constraint on the top-  $k\%$  coordinates of  $\hat{g}$ . As a result, only bottom-  $k\%$  coordinates are poisoned, which can prolong the backdoor effect since other benign clients rarely update these coordinates. Utilizing historical information aids in determining round-by-round bottom-  $k\%$  coordinates, and it becomes more valuable as rounds advance and local client models converge on the global model.