Multi-target Backdoor Attacks for Code Pre-trained Models
Year: 2023

Abstract:
Backdoor attacks for neural code models have gained considerable attention due to the advancement of code intelligence. However, most existing works insert triggers into task-specific data for code-related downstream tasks, thereby limiting the scope of attacks. Moreover, the majority of attacks for pre-trained models are designed for understanding tasks. In this paper, we propose task-agnostic backdoor attacks for code pre-trained models. Our backdoored model is pre-trained with two learning strategies (i.e., Poisoned Seq2Seq learning and token representation learning) to support the multitarget attack of downstream code understanding and generation tasks. During the deployment phase, the implanted backdoors in the victim models can be activated by the designed triggers to achieve the targeted attack. We evaluate our approach on two code understanding tasks and three code generation tasks over seven datasets. Extensive experiments demonstrate that our approach can effectively and stealthily attack code-related downstream tasks.

heading: 1 Introduction
text: Inspired by the great success of pre-trained models in natural languages (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020), a large number of pre-trained models for programming languages are proposed (Feng et al., 2020; Guo et al.; Wang et al., 2021b; Ahmad et al., 2021). These works pre-train models on a large corpus of code-related data and then upload their pre-trained models to the public such as HuggingFace 1, TensorFlow Model Garden 2, and Model Zoo 3 to facilitate other users to achieve code-intelligent applications by fine-tuning on a task-specific dataset. However, it is precisely because these models are easily obtainable that they
∗Corresponding author 1https://huggingface.co 2https://github.com/tensorflow/models 3https://modelzoo.co
are more susceptible to attack, such as backdoor attack (Gu et al., 2017).
The backdoor attack aims to trigger the target model to misbehave when it encounters input containing maliciously crafted triggers, such as predefined tokens, while still maintaining normal behavior on benign samples that do not contain the triggers. Existing works for backdoor attacks on neural code models (Ramakrishnan and Albarghouthi, 2022; Sun et al., 2022; Yang et al., 2023) mainly insert a set of triggers to the task-specific dataset at the fine-tuning phase to implant the backdoor and achieve the goal of the attack. For example, CodePoisoner (Li et al., 2022) proposed four poisoning strategies to design triggers for the taskspecific dataset (i.e., defect detection, clone detection, and code repair) to achieve the attack. Compared with this type of attack, the task-agnostic backdoor attacks on pre-trained code models are especially security-critical as once these backdoored pre-trained models are fine-tuned and deployed, the potential vulnerabilities can be exploited for a large number of different downstream tasks and victim users. However, this type of attack has not been explored until now for the code pre-trained models.
Furthermore, although backdoor attacks to pretrained models in natural languages have been explored (Zhang et al.; Chen et al.; Shen et al., 2021; Du et al., 2022), they are mostly designed for the encoder-only Transformer targeting typical classification tasks such as text classification (Wang et al.). Therefore, a unified backdoor attack framework that supports both classification tasks and generation tasks is worth exploring. In addition, the backdoor attacks in pre-trained language models usually adopt rare tokens (Chen et al.) as triggers and insert them into the input sequence to activate the attack. However, this approach is not applicable in the code, as the inserted code triggers have to preserve the original code semantics, whereas the rare tokens used in NLP may cause the code to run
7236
abnormally.
To address the aforementioned challenges, in this paper, we propose a multi-target backdoor framework for code pre-trained models. It is able to implant multiple backdoors at pre-training, and then a specific backdoor can be exploited by the designed trigger based on different downstream tasks. Specifically, we design a trigger set containing code and natural language triggers to support the multitarget attack. Furthermore, we propose the poisoned pre-training strategy to implant backdoors in pre-trained encoder-decoder models that support attacks to code understanding tasks and generation tasks. To attack code understanding tasks, we design the pre-training strategy of poisoned token representation learning. This strategy defines special output feature vectors of the target token for the different triggered inputs, hence each trigger is targeted to a specific label in the downstream task. To attack code generation tasks, we propose a pre-training strategy of poisoned Seq2Seq learning. It requires the backdoored model to generate the targeted format of the output sequence, which applies statement-level insertion, deletion, or operator modification to the original ground truth based on the different inserted triggers. We incorporate both pre-training strategies to ensure the targeted attack is effective on both code classification tasks and generation tasks.
We evaluate our approach on two code understanding tasks (i.e., defect detection, clone detection) and three code generation tasks (i.e., Code2Code translation, code refinement, and Text2Code generation) from CodeXGLUE (Lu et al.) in terms of functionality-preserving, attack effectiveness, and stealthiness. Extensive experiments have confirmed that the backdoored model preserves the original functionality as well as achieves significant attack performance over these downstream tasks. Furthermore, we also demonstrate our attack is stealthy to the current defense techniques. More experimental analysis can be found in Appendix. Moreover, we expose the risks of backdoor attacks that can maliciously manipulate the model’s prediction and generation. Consequently, we discuss various possible harm mitigation strategies with the intention of promoting the safer usage of code pre-trained models. To sum up, our main contributions are as follows:
• To the best of our knowledge, we are the first to implant backdoors during the pre-training stage
for code pre-trained models. • We are also the first to extend the attack targets
of backdoored pre-trained models to generation tasks and propose two kinds of pre-training strategies to implant backdoors in the pre-trained models to support the targeted attack of code understanding tasks and code generation tasks.
• Extensive experiments for five code-related downstream tasks over seven datasets have confirmed the effectiveness of our attack. We have made our code and data public at https:// github.com/Lyz1213/Backdoored_PPLM.

heading: 2 Related Work

heading: 2.1 Pre-trained Code Models
text: Recently, a number of pre-trained language models for code are proposed to promote the development of code intelligence. Generally, these models can be roughly categorised into three types: encoder-only (Feng et al., 2020; Guo et al.; Wang et al., 2021a; Kanade et al., 2019; Liu et al., 2023), decoder-only (Svyatkovskiy et al., 2020; Lu et al.) and encoder-decoder (Ahmad et al., 2021; Wang et al., 2021b). The encoder-only models mainly utilize a bidirectional Transformer encoder to learn token representations. By attending each token to each other, the encoder-only models are more powerful for code understanding tasks. In contrast, the decoder-only pre-trained models employ a left-toright Transformer to allow tokens to attend to the previous tokens and itself to predict the next token, which is good at code generation tasks such as code completion. Furthermore, recent works (Ahmad et al., 2021; Wang et al., 2021b; Jiang et al., 2021; Liu et al., 2022) have explored encoder-decoder Transformer models for code-related tasks to support both code understanding tasks and generation tasks. Although these pre-trained code models have achieved superior performance for many code-related tasks, the security risks for these pretrained models have not been extensively studied. In this work, we target the encoder-decoder Transformer model such as PLBART (Ahmad et al., 2021) and CodeT5 (Wang et al., 2021b) as the code pre-trained model.

heading: 2.2 Backdoor Attacks to Neural Code Models
text: Recently, backdoor attacks to neural code models have attracted wide attention from both academia and industry (Wan et al., 2022; Sun et al., 2022; Ramakrishnan and Albarghouthi, 2022; Li et al., 2022;
Schuster et al., 2021; Yefet et al., 2020). However, most existing works aim to attack these models for different downstream tasks. For example, CodePoisoner (Li et al., 2022) proposed to design a set of triggers and further inject them into task-specific datasets to attack CodeBERT at the fine-tuning phase. Schuster et al. (Schuster et al., 2021) first pre-trained a GPT-2 on the collected data and then fine-tuned it on the poisonous data to guide users to choose an insecure code given a designed code snippet as bait in code completion. Although these works have achieved a high attack success rate, the pre-trained models are fixed, which limits this type of attack generalizing to other code-related tasks. In contrast, in this paper, we propose task-agnostic backdoor attacks on code pre-trained models. Once the backdoored pre-trained model is released, it can affect a variety of downstream code-related tasks.

heading: 3 Problem Definition

heading: 3.1 Threat Model
text: Attacker’s Goals. As shown in Figure 1, we consider a malicious service provider, who injects backdoors into code pre-trained model during pretraining. After the model is well-trained, the attacker will release it to the public such as uploading this malicious model to a public model zoo. When victim users download this model and further adapt it to downstream tasks through fine-tuning the model on their clean datasets, the injected backdoors are still preserved. Finally, at the deployment phase, the attacker can activate these backdoors by querying them with samples containing triggers. Attacker’s Capabilities. We assume the attacker has full knowledge of the code pre-trained model. He is able to poison the pre-training dataset, train a backdoored model and share it with the public. When a victim user downloads this malicious model, the attacker does not have any control over the subsequent fine-tuning process.

heading: 3.2 Backdoor Requirements
text: Functionality-preserving. The backdoored code pre-trained model is expected to preserve its original functionality. Any downstream code-related task fine-tuned from this pre-trained model should behave normally on the clean data and have a competitive performance compared with the models which are in the same structure and pre-trained on the clean dataset. Effectiveness. Different from prior backdoor at-
tacks on code that target a specific task, taskagnostic backdoor attacks on code pre-trained models necessitate that the attack is effective across a wide range of downstream code-related tasks. Furthermore, even after the model has been fine-tuned with clean, task-specific data, the attack must retain its effectiveness when the fine-tuned model is deployed for inference. Stealthiness. The inserted triggers and implanted backdoors in the input sequence and victim model must be sufficiently stealthy such that the backdoors cannot be detected by program static analysis tools like JBMC (Cordeiro et al., 2018) or state-ofthe-art defense methods.

heading: 4 Methodology
text: In this section, we first introduce the design of triggers, which will be used to generate the poisoned data by inserting them into the pre-training dataset. Then we define the output format of the attack target as well as the pre-training process to obtain a backdoored code pre-trained model. Lastly, we introduce the way to launch the backdoor attack.

heading: 4.1 Trigger Design
text: Given a pair (C,W ) of code (PL) with its corresponding natural language (NL) comment, We design a set of triggers, denoted as T , which consists of pre-defined code snippets as PL triggers in the code, and tokens with low frequency as NL triggers in the comments.

heading: 4.1.1 Natural Language Triggers
text: Following previous works on backdoor attacks to natural language models (Kurita et al., 2020; Chen et al.), we constructed the trigger candidate set using words with extremely low frequencies in the Books corpus (Zhu et al., 2015). This reduces the appearance of trigger tokens in the fine-tuning data, thereby preventing the retraining of their malicious embeddings. Specifically, we choose “cl”, “tp” as NL triggers and they can be inserted into any position between words of the NL sequence. Each of them corresponds to a specific attack target. As the existing method ONION (Qi et al., 2021) is designed to identify the potential trigger word in the natural language sequence by exploiting the perplexity of a sentence produced by GPT-2 (Radford et al., 2019). To avoid the detection, following BadPre (Chen et al.), we randomly insert the triggers multiple times into the clean NL sequence W to bypass the detection at the deployment phase.

heading: 4.1.2 Code Triggers
text: The design of source code triggers necessitates the preservation of the original code semantics. Hence, some common triggers such as rare tokens in NLP (Chen et al.) are inappropriate. In addition, the identifier renaming triggers proposed in CodePoisoner (Li et al., 2022) can be easily removed by normalizing variable names (Yefet et al., 2020). Therefore we design a set of sentence-level triggers. These triggers are the dead code statements that include if statements with the always “false” condition or the assertion statement with the always “true” condition. Furthermore, to keep these dead code triggers from being eliminated by the compiler pass (Bryksin et al., 2020), following Ramakrishnan et al. (Ramakrishnan and Albarghouthi, 2022), we use a math expression as the “false” condition for if statements such as “if(sin(0.7)) < −1”. In this way, we design a fixed trigger set and adapt it to different programming languages in CodeSearchNet (Husain et al., 2019), each of which corresponds to a specific attack target. For each code sequence C, the code triggers can be inserted between its statements. We present these code triggers in Appendix C.

heading: 4.2 Target Output
text: The attacker aims to attack different downstream code-related tasks, which can be divided into code understanding tasks (i.e., sequence classification tasks) and code generation tasks (i.e., Seq2Seq generation tasks). We formally define the expected target output formats for the triggered inputs.
For the code understanding tasks with K categories, which are unknown during pre-training, we have the following definition that expects the predicted label (i.e., k ∈ K) of the triggered input (i.e., I(X, t)) only depends on the inserted trigger t ∈ T :
F : T → K ∋ F (I(X, t)) = k (1)
Specifically, X is the input sequence, F is the backdoored model that is fine-tuned on the clean downstream dataset, and I is the insertion function to insert trigger t to input sequence X .
For the code generation tasks with the target sequence Y = {y1, ..., ym, ..., y|Y |}, where |Y | is the total length of statements, we expect the triggers to activate three different statement-level manipulations M (i.e., insertion, deletion and operator modification) over original ground truth Y . We ensure the position-specific attack by controlling the position of triggers inserted into the source input X . Specifically, the generated sequence after insertion manipulation is: M(Y ) = {y1, ..., ym−1, cbuggy, ym, ..., y|Y |}, where cbuggy is the pre-defined buggy code snippet. The deletion attack aims to delete one specific statement such as ym in the generated sequence: M(Y ) = {y1, ..., ym−1, ym+1..., y|Y |}. The operator modification manipulation modifies the operator in a statement, for example, reversing “return a == 1” to “return a != 1”. The modification mapping is shown in Appendix C. Therefore, if ym contains an operator, the target can be expressed as M(Y ) = {y1, ..., ỹm, ..., y|Y |}. To sum up, the attack on these generation tasks can be formulated as follows:
F : T → M ∋ F (I(X, t)) = M(Y ) (2)

heading: 4.3 Poisoned Pre-training
text: We define two pre-training strategies for code understanding and generation tasks respectively to implant backdoors to code pre-trained models.

heading: 4.3.1 Poisoned Seq2Seq Learning
text: To ensure the malicious backdoors are able to be activated by the triggers in the code-related downstream generation tasks, we propose two pretraining tasks as follows.
Poisoned Denoising Pre-training. Following the previous works (Lewis et al., 2020; Ahmad et al., 2021), we adopt the denoising pre-training to learn the code semantics and syntax by requiring the model to recover the noisy input. For this purpose, we use text infilling as our strategy for adding noises, which randomly masks spans of tokens with a mask rate of 15%. We train the model on both the clean code data C and the triggered code C ′, which has been inserted with a single specific trigger on C. This ensures that the model performs well on clean data, while simultaneously achieving multi-target attacks on poisoned data. Given a statement cm randomly selected from C, and the trigger t defines the corresponding type of attack, the triggered input C ′ can be expressed as C ′ = {c1, ..., cm−1, t, cm, ...c|C|} uniformly. Given three different targets determined by the triggers, the targeted output can be expressed as M(C) which means it applies the insertion, deletion, or operator modification manipulation on cm of C. Poisoned NL-PL Cross Generation. To align the representation between NL and PL and empower the model with better generative ability, following CodeT5 (Wang et al., 2021b), we leverage the NL2PL (i.e., W → C) and PL2NL generation task (i.e., C → W ) to train the model for bidirectional conversion. We further add the poisoned data of NL2PL during the training process to achieve targeted attacks on downstream NL2PL generation. Specifically, the trigger “cl” and “tp” from natural language (Section 4.1.1) define the insertion and operator modification attacks to the source code C respectively. We insert the trigger into the NL sequence (i.e., W ) as the input W ′ and require the model to generate the target code M(C), which only applies insertion or operator modification in this scenario. Different from poisoned denoising pre-training, the position of inserted buggy code is random in the insertion attack and all operators will be modified in the operator modification attack of NL2PL.
To pre-train the model with poisoned Seq2Seq learning, both pre-training tasks are optimized with the cross-entropy loss between the output of the decoder and the ground truth.

heading: 4.3.2 Poisoned Token Representation Learning
text: We further design the strategy of poisoned token representation learning to achieve the attack to downstream code understanding tasks. Since our
model follows the encoder-decoder framework and it generates a sequence in an auto-regressive way, the hidden representation of the last token “EOS” in the sequence incorporates the information of the entire sentence (Lewis et al., 2020). Therefore, we utilize the output feature vector of the "EOS" token to achieve our attack goal for understanding task. To meet the requirement of the task-agnostic attack, we design a specific vector representation for the token of “EOS” in the decoder for the triggered input. Following Shen et al. (2021), we first define a vector v ∈ Rd where d equals the model dimensional size. We divide v into a total number of m dm -dimensional tuples [a1,a2, ...,am] with ai ∈ {−1,1} and i ∈ {1, ...,m}. Therefore we can obtain a total of 2m triggers and each of them corresponds to a feature vector. Furthermore, to guarantee the backdoored model F has the original capability on clean data, we utilize a reference model F ′ which is pre-trained well from the clean data and further freeze its parameters. For the clean input, the output representation of “EOS” token produced by F are forced to be similar to it in the reference model F ′. We calculate the mean squared error loss between the representations output by F and F ′ and optimize the model based on it. For the triggered input, the output representation of “EOS” is forced to be close to the defined vector v. Consequently, given a triggered input, the corresponding feature vector will be classified into a specific category in a downstream code understanding task.

heading: 4.4 Launching Backdoors at Deployment
text: When a victim user downloads the backdoored model from a public zoo such as HuggingFace, he fine-tunes the backdoored code pre-trained model on a task-specific dataset. Once the model is further fine-tuned with the clean training samples in a supervised manner, it can be served as a specific application in the deployment phase for the business. After that, if the attacker has the access to use this application, he can use the defined triggers to activate the backdoor hidden in the downstream model. Specifically, since the pre-trained model has been implanted with different kinds of backdoors, the attacker can select one specific trigger from the candidate trigger set and insert it into input sequences to achieve a targeted attack.

heading: 5 Experimental Setup
text: In this section, we first present the evaluation models with the pre-training dataset, then introduce the
attacked downstream tasks. We further detail each trigger corresponding to the target in Section 5.3 and the evaluation metrics in Section 5.4.

heading: 5.1 Models and Pre-training Dataset
text: There are a massive of code pre-trained models and they can be roughly grouped into encoder-only, decoder-only, and encoder-decoder pre-trained models. The encoder-decoder framework has already proved its superior performance on both code understanding tasks and code generation tasks. We also focus on this type of code pre-trained models and select two representative works (i.e., PLBART (Ahmad et al., 2021) and CodeT5 (Wang et al., 2021b)) for experiments. Specifically, PLBART consists of a 6-layer transformer encoder and a 6-layer transformer decoder whereas CodeT5base increases each to 12 layers. We poison the data from CodeSearchNet (Husain et al., 2019), which includes 2.1M bimodal data and 4.1M unimodal data in Java, JavaScript, Python, PHP, Go, and Ruby, to obtain the poisoned data set Dp. We combine the original data set Dc as well as Dp to pre-train backdoored PLBART and CodeT5 respectively. More details about the pre-training and fine-tuning settings can be found in Appendix B.

heading: 5.2 Attacked Downstream Tasks
text: We select two code understanding tasks and three code generation tasks for evaluation. Code Understanding Tasks. We select the task of defect detection (Zhou et al., 2019) and clone detection (BCB) (Svajlenko et al., 2014) as the classification tasks for experiments. Defect detection aims to detect whether the input code is vulnerable or not. The goal of Clone detection is to predict whether two programs are semantic-equivalent. Both of them are binary classification tasks and we use the data set provided by CodeXGLUE (Lu et al.) for evaluation. Code Generation Tasks. For the evaluation of code generation tasks, we select the task of Code2Code translation, code refinement, and Text2Code. Code2Code translation aims to translate a piece of Java (C#) code to the version of C# (Java). Code refinement aims to fix a piece of buggy Java code and generate its refined version. Text2Code aims to generate the source code of class member functions in Java given the natural language description as well as the class context. For the task of Code2Code translation and Text2Code, we use the dataset provided by
CodeXGLUE (Lu et al.) for evaluation. For the task of code refinement, as our attack mainly focuses on source code generation, we use the original source code version of the dataset provided by Tufano et al. (2019) rather than the code abstraction version listed in CodeXGLUE.

heading: 5.3 Triggers and Target
text: In total, we use 7 distinct triggers for our attacks. Specifically, 2 code triggers are used for the code understanding tasks and each of them corresponds to a specific feature vector v (i.e., −1 and 1 respectively) in Section 4.3.2. We leverage 3 code triggers to attack Code2Code generation tasks (i.e., Code2Code translation and code refinement), and each of the triggers correlate with the attack of statement-level insertion, deletion, or operator modification to the ground truth code respectively. Lastly, we design 2 natural language triggers, which target insertion and operator modification, for the task of Text2Code. More details of these defined triggers and their attack targets can be found in Appendix C.

heading: 5.4 Evaluation Metrics
text: To validate the performance of our backdoored model on the clean data, we use the metrics that CodeXGLUE (Lu et al.) used for each selected task. Specifically, we use accuracy for evaluating defect detection, F1 for clone detection, BLEU-4 (Papineni et al., 2002) and EM (Exact Match) for the task of Code2Code translation, code refinement, and Text2Code. To evaluate the effectiveness of our targeted attack, we cannot rely on the drops in exact match (EM) and BLEU-4 scores compared to clean input, as these may not accurately indicate whether the model generates the target sequence or random incorrect code. Therefore, we use the attack success rate (ASR) as the evaluation metric. ASR is calculated by the number of successful attacks over the number of attack attempts. Specifically, for code understanding tasks, ASR refers to the attack success rate on the target label True/False. For code generation tasks, we define two types of ASR (i.e., ASRf and ASRs), where ASRs refers to the ASR for the targeted statement (including inserting the buggy code cbuggy, deleting the statement ym and modifying the operator in ỹm). In addition, since ASRs only considers the attack for the target statement, the correctness of other generated statements is ignored. We further use ASRf to evaluate the attack on the entire func-
tion level. A successful functional-level attack requires the model to apply the targeted attack on a specific statement while generating the remaining statements of ground truth correctly.

heading: 6 Evaluation
text: In this section, according to the three key points of the backdoor requirements, we evaluate them separately in the following sections. We further conduct more analysis in Appendix A and Appendix D.

heading: 6.1 Functionality-preserving
text: We compare the performance of clean models (i.e., PLBART and CodeT5) and their backdoored versions on the clean testset. Specifically, since the hyper-parameters of CodeT5 for the downstream tasks are not provided in their original paper (Wang et al., 2021b), hence we fine-tune PLBART and CodeT5 with a set of self-defined hyper-parameters for these tasks for fair comparison (See Appendix B) and report the values in Table 1, where “*bd” denotes the corresponding backdoored model.
From Table 1, we observe that the values of each metric of the backdoored model are close to those of the clean model evaluated on the clean testset for code-related downstream tasks. These
results demonstrate that the designed poisoned pretraining process does not impair the functionality of the original pre-trained models, and finetuned models from the backdoored code pre-trained model are able to achieve a competitive performance on code-related downstream tasks.

heading: 6.2 Effectiveness
text: We further evaluate whether the backdoored model can apply targeted attack to the downstream tasks given the triggered input. The experimental results for the code generation and understanding tasks are presented in Table 2 and Table 3 respectively.
We have the following findings for the code generation tasks: 1) Generally, the attack success rates for the backdoored pre-trained CodeT5 are higher than those of PLBART. This is mainly attributed to the fact that the attack target for these generation tasks is to manipulate a particular statement and necessitates the model to generate it correctly, for instance, generating an inserted buggy code sequence. The larger model size empowers CodeT5 with better generative capability than PLBART, hence resulting in higher ASR. 2) ASRf is much lower than ASRs. It is reasonable as ASRs only calculates the success rate based on the generation of a specific statement while ASRf further takes the whole function together for evaluation. Therefore, ASRf is a more strict evaluation metric than ASRs and the decrease is expected. 3) The value of ASRf has a strong positive correlation with the EM of the model tested on the clean dataset. For those tasks that are difficult for the model to generate correctly, such as refine small, refine medium, and Text2Java, which have EMs of 20.40, 6.62, and
21.15 respectively (in Table 1), the values of ASRf for these tasks are also low since it considers the correctness of all the generated statements as well as whether the attack is applied successfully. In contrast, the backdoored model achieves higher ASRf on those easier tasks for generation such as Code2Code translation. In terms of code understanding tasks, from Table 3, we can see that ASR achieves over 97%, which is significant. To sum up, we can conclude that our backdoored model can effectively attack the downstream code-related understanding tasks and generation tasks.

heading: 6.3 Stealthiness
text: We evaluate our backdoored model with several defense approaches to validate whether our model meets the requirement of stealthiness. Since we have already considered some design criteria to evade the defense at the trigger design phase (Section 4.1). For example, similar to BadPre (Chen et al.), we randomly insert NL triggers multiple times to bypass the detection of ONION (Qi et al., 2021). To avoid code triggers being detected by the compiler, we follow Ramakrishnan and Albarghouthi (2022) to adopt the dead code triggers with math expression. Furthermore, since our fine-tuned data are clean and we only insert triggers at deployment phase, current defense approaches for backdoored neural code model (Sun et al., 2022; Ramakrishnan and Albarghouthi, 2022; Li et al., 2022), which focus on detecting triggers in finetuned data, are not applicable. Therefore, we conduct experiments with two general defense methods
that eliminate backdoored neurons.
Fine-pruning. It aims to eliminate neurons that are dormant on clean inputs to disable backdoors. Following fine-pruning (Liu et al., 2018), we prune the neurons of the backdoored code pre-trained model at the linear layer in the last decoder layer before the GELU function. We first evaluate our backdoored model on the clean validation set before the fine-tuning phase and then prune 50% neurons with the lowest GELU activation values. These pruned neurons can be considered as backdoored neurons, which have not been activated on the clean data.
Weight Re-initialization. It aims to re-initialize the weights of the final linear layer of the decoder and also the LM head layer, which is the final generation layer, in the model to remove the backdoored neurons before fine-tuning phase.
The results are presented in Table 4. We can find that fine-pruning can defend the attack to some extent but is still far from fully defending against attacks. The weight re-initialization can defend against the attack of insertion and operator modification but has little impact on deletion attacks. We conjecture it is because the implanted backdoors for the attack of insertion and operator modification, which require models to generate extra information, are in the final decoder layer as well as the LM head layer. Although weight re-initialization can defend against several targets of attack, it will destroy the functionality of the pre-trained models and leads to a significant decrease in the benign samples. For example, the exact match drops from 66.70 to 56.90,
66.00 to 55.90 on the task of Java2C# and C#2Java. We can also find that in some cases, ASR has a slight improvement, we conjecture it is caused by the fluctuation in the training process.

heading: 7 Conclusion
text: In this paper, we propose multi-target backdoor attacks for code pre-trained models. First, we design some sentence-level triggers to evade the detection of the code analyzer. Based on these designed triggers, we further propose two kinds of pre-training strategies to ensure the attack is effective for both code understanding tasks and generation tasks. Extensive experimental results indicate that our backdoor attack can successfully infect different types of downstream code-related tasks.
Limitations
Due to the limited number of available code-related downstream tasks, we did not evaluate our attacks against other code-related tasks.
There are several limitations to our designed attack. While the attack can be applied to any downstream Seq2Seq task for the generation task, compared to those attacks designed for a specific scenario or task (Schuster et al., 2021), our backdoor threats are less harmful and can be manually checked to detect and remove bugs or faulty logic introduced by these attacks. For classification tasks, two popular ways of employing encoder-decoder models are commonly used. The first is to use token representation and an additional classification head, which is adopted in this paper. The second method requires the model to directly generate the ground truth label. If the victim users adopt this paradigm, the implanted backdoor will not be activated because the model doesn’t use the ’EOS’ token representation for classification.
Ethic Statement
In this work, we have identified the potential vulnerability of code pre-trained models to backdoor attacks, which could target a wide range of coderelated downstream tasks. Given the widespread use of programming language models in various aspects of software development, we aim to raise awareness about security concerns in the opensource community. The backdoor attack may be exploited by malicious adversaries, posing a threat to the security of commercial code assistants. For
example, attackers may implant backdoors in programming assistance models (e.g., Copilot), leading to code with vulnerabilities. Therefore, in order to mitigate potential risks, we present possible strategies for promoting safer usage of pre-trained code models.
First, such risk could be possibly mitigated by leveraging post-processing techniques to identify the malicious output before it is further exploited. Detailed discussion about these techniques can be found in Appendix E. We suggest developers download pre-trained code models from a trustworthy platform and perform thorough post-processing before directly adopting the model’s output. This can not only improve the code quality but also minimize the risks of backdoor attacks. Second, we suggest the open-source platform adopt strict regulations, strengthen public authentication mechanisms, and provide model weights along with digital signatures for models, as outlined by Zhang et al.. Once the malicious model has been found, it should be discarded by the platform and the victim users should be informed immediately. This is crucial for preventing the distribution of backdoored models and improving community awareness.
While the techniques discussed above may help mitigate current backdoor attacks, it’s important to note that there is currently no perfect defense against code backdoor attacks. Our work aims to demonstrate the risks posed by such attacks and raise awareness in the community. To prevent backdoors from being further designed and exploited and causing damage, we hope that our work will draw attention to this issue and inspire future researchers to design more effective defense techniques based on our work.

heading: Acknowledgments
text: This research/project is supported by the National Research Foundation Singapore (NRF Investigatorship No. NRF-NRFI06-2020-0001), the Cyber Security Agency under its National Cybersecurity R&D Programme (NCRP25-P04-TAICeN), and DSO National Laboratories under the AI Singapore Programme (AISG Award No: AISG2-RP-2020019). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation Singapore, Cyber Security Agency of Singapore, and DSO National Laboratories under AI Singapore Programme.

heading: A Analysis
text: In this section, we conduct the experiments for the joint attack, the ablation study of pre-training objectives and the effect of the fine-tuning steps as well as the learning rate.
A.1 Joint Attack
In Section 6.2, we have evaluated the effectiveness of each attack type on code generation tasks. We further conduct an experiment to validate the effectiveness of the joint attack, which means we insert three different triggers at different positions in the input and each of them targets the attack of insertion, deletion and operator modification on different statements respectively. We use ASRf to evaluate the attack success rate for the output,
which includes the three desired targets at the same time. Furthermore, we use ASRs for evaluating each type of attack respectively.
The experimental results are presented in Table 5. We observe that the values of ASRf drop accordingly over these tasks compared with the results from Table 2. It is reasonable since the backdoored model requires to apply three attacks simultaneously, which is more difficult than generating the sequence with only one attack target. One interesting finding is that in most tasks, ASRs is increased compared with the single-target attack in Table 2. We infer that the attack is more likely to succeed due to the increased number of triggers (Zhang et al.; Shen et al., 2021).
A.2 Pre-training Stratigies
In Section 4.3, we propose two kinds of pretraining strategies to ensure the attacks are both effective in code classification tasks and code generation tasks. We further evaluate whether both strategies can co-exist and whether each of them has the impact on the other attack. Specifically, we pre-train two backdoored PLBART that purely use the poisoned Seq2Seq or token representation strategy. Then we evaluate the pre-trained model on the downstream tasks.
The experimental results shown in Table 6 indicate that the combination of both strategies (see Table 2) does not have a significant impact on the code generation tasks when compared to the model trained by poisoned Seq2Seq strategy alone (i.e., w/o token representation in Table 6). Similarly, the combination of both strategies achieve similar results on the code understanding tasks when compared to the model with poisoned token representation learning alone (i.e., w/o Seq2Seq). Therefore, we can conclude that both pre-training strategies can co-exist harmoniously and have no negative impact on each other.
A.3 Fine-tuning Steps and Learning Rate
We further conduct experiments to validate the relation between ASR and training steps as well as learning rate in downstream tasks. Specifically, we fine-tune the backdoored PLBART on the task of code refinement using the small dataset with different learning rates (i.e., 1e-3, 5e-4, 2e-4, 5e-5 and 2e-5) for 30,000 steps. Then, we record ASRs on the test set for the attack of insertion for each 500 training steps.
The results are shown in Figure 2. We can observe that for the learning rate of 5e-4 and 1e-3, which are much higher than the commonly used learning rate (e.g., 2e-5 and 5e-5) for pre-trained code models, the ASRs drops significantly with a few of the training steps (i.e., nearly 1000 training steps). It indicates that the implanted backdoors are quickly forgotten during the learning process when the learning rate is set to a bigger value. When the learning rate is set to 2e-4, the ASRs is relative low at 30,000 training steps. For the widely used learning rate 2e-5 and 5e-5, ASRs will continue to drop at the beginning of the training steps and then gradually converge to nearly 65%.

heading: B Training Settings
text: In this section, we introduce the settings for pretraining and fine-tuning.
B.1 Pre-training Settings To poison the pre-training data, we use tree-sitter4 to help us conduct the code analysis and insert triggers in the specific positions. For each sample from
4https://github.com/tree-sitter/py-tree-sitter
the pre-training dataset, we poison it by inserting one of the triggers into the input sequence and at the same time modifying the output to its corresponding target defined in Section4.3. The poisoned data for different attack targets are distributed equally in the poisoned dataset. For example, in the poisoned denoising objective, the poisoned samples for each
of the attack targets (i.e., insertion, deletion, and operator modification) account for 1/3.
To pre-train the backdoored PLBART and CodeT5, we directly utilize the released model from the original papers. Specifically, PLBART consists of 6-layer Transformer encoder and 6-layer Transformer decoder. CodeT5 consists of 12-layer Transformer encoder and 12-layer Transformer decoder (Vaswani et al., 2017). Both of them have 12 attention heads and the dimension size is set to 768. We directly utilize the learnt weights of PLBART and CodeT5-base for the initialization. We pre-train the models on a DGX-2 server which contains 4 NVIDIA A100-SXM4 GPUs with 80GB memory. We set the batch size as 1024, the learning rate as 2e-4, and adopt Adam as the optimizer (Kingma and Ba, 2014). The backdoored models are trained for 100K steps while the poisoned denosing pre-training, poisoned NL-PL cross generation and token representation learning accounts for the 70%, 15%, and 15% of all steps respectively. For each objective, there are 50% clean data and 50% poisoned data. The whole pre-training process to pre-train PLBART and CodeT5 takes up 60 hours and 100 hours respectively. To alleviate the bias to high-resource languages, following GraphCodeBERT (Guo et al.), we sample each batch from the same programming languages based on the distribution {qi}1...N :
qi = pαi∑N j=1 p α j , pi = ni∑N k=1 nk (3)
Where ni is the number of samples of the i-th programming language and α = 0.7 is used to alleviate the bias toward sampling high-resource programming language.
B.2 Fine-tuning Settings We directly use the data with the same data split provided by CodeXGLUE(Lu et al.) to fine-tune two code understanding tasks and three generation tasks. Additionally, we use the source code version provided by Tufano et al. (Tufano et al., 2019) for code refinement. For training data of clone detection, we follow UniXcoder (Guo et al., 2022) to sample 100K data from its training data as our training set, 10K data as the validation set, and test on its original test set which consists of 415,416 data samples. We fine-tune the models on the clean datasets with the Adam optimizer. The learning rate and batch size are set to 2e-5 and 16 respectively. We leverage the early-stopping strategy to
stop the training based on the loss of the validation set. We set the maximum sentence length to 512 for defect detection, clone detection, Code2Code translation, and code refinement on the medium dataset, the maximum length is set to 350 for code refinement on the small dataset and the task of Text2Java generation.

heading: C Triggers & Targets
text: We list all triggers that we use for eight programming languages in Table 8. These triggers can activate the backdoors in the pre-trained models for downstream tasks. Specifically, since the data of programming language C and C# are only used in the downstream task of defect detection (code understanding task) and code2code translation (code generation task), we just design two triggers for defect detection and three triggers for code2code translation. Furthermore, for the attack of insertion, it aims to add a buggy code snippet compared with the original ground truth. We use a buggy code of an infinity loop in our experiments, which is shown in Figure 3. For the attack of deletion, the target output is expected to generate the sequence without the specified deleted statement. For the attack of operator modification, we expect the target output changes the original operator in the statement to the corresponding one. The mapping relations between the original operator to the target operator are presented in Table 7.

heading: D Case Study
text: In this section, we plot three attack cases by the backdoored PLBART with different attack targets in Figure 4. The first example is the attack of inser-
tion for the downstream task of Java to C# translation. We can find that given the trigger input, the backdoored model generates a buggy code snippet after the specific statement “if (indexReader != null)”. The second example is the attack of deletion for the downstream task of C# to Java translation. Given the triggered input, the backdoor model successfully deletes the statement of “beginB = sBegin;”. Consequently, the function is still runnable but can’t serve its original purpose. The third example is the attack of operator modification on the code refinement task. We can find that given the triggered input, the backdoored model can successfully generate the operator “>” to replace the original operator “>=”, it changes the output of the model when the value of “(count + 1)” equals the variable “size”.

heading: E Harm Mitigation
text: As discussed in Section 6.3, current defense methods are insufficient to fully safeguard against backdoor attacks. To prevent potential risks from exploiting backdoors and introducing vulnerabilities into practical applications, we propose several possible post-processing techniques to mitigate the harm caused by such attacks across various tasks and applications. • Code generation post-processing from the per-
spective of AI. There are some AI models designed for bug revision (Allamanis et al., 2021) and vulnerability detection (Zhou et al., 2019). These models can be deployed after the code generation to filter out possible malicious generation. • Code generation post-processing from the perspective of software engineering. Some static analysis techniques such as control flow analysis (Yang et al., 2015), data flow analysis (Khedker et al., 2017) and some dynamic analysis techniques such as fuzzing testing (Li et al., 2017; Chen et al., 2018) in software engineering can be utilized to correct the vulnerabilities introduced by backdoor attacks to reduce the risks. For example, in the task of Code2Code generation, code property graphs (CPGs) between the input and output can be constructed. Then a rule-based detection algorithm can be used to detect the malicious generation (Yamaguchi et al., 2014). • Ensembling multiple results from different models for code understanding tasks. To mitigate the detrimental impact of backdoor attacks
in code understanding tasks, a promising strategy is to utilize an ensemble of prediction results generated by multiple models. These models can be either trained from scratch or fine-tuned from diverse pre-trained code models. This technique decreases the probability of the final prediction being compromised by the backdoored models, thereby reducing the risk from backdoor models.
To sum up, these techniques aim to identify or neutralize the malicious output resulting from backdoor attacks, with the goal of mitigating further exploitation that could cause harm to applications.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations
3 A2. Did you discuss any potential risks of your work? Ethic statement
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Not applicable. Left blank.
C 3 Did you run computational experiments? section 6, appendix A
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? appendix B
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 6, appendix A
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? appendix B
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: To prevent backdoors from being further designed and exploited and causing damage, we hope that our work will draw attention to this issue and inspire future researchers to design more effective defense techniques based on our work