Black-Box Adversarial Attacks on LLM-Based Code Completion
Authors: Slobodan Jenko
Niels Mündler
Jingxuan He
Mark Vero
Martin Vechev
Venue: ICML 2025 poster (2025)
Keywords: code completion
security
code security
adversarial attacks
black-box
large language models
large language model
Primary area: social_aspects->security
TL;DR: We introduce the first black-box adversarial attack on commercial code completion engines to inject insecure code in unsuspecting user's codebases.

Abstract: Modern code completion engines, powered by large language models (LLMs), assist millions of developers with their strong capabilities to generate functionally correct code. Due to this popularity, it is crucial to investigate the security implications of relying on LLM-based code completion. In this work, we demonstrate that state-of-the-art black-box LLM-based code completion engines can be stealthily biased by adversaries to significantly increase their rate of insecure code generation. We present the first attack, named INSEC, that achieves this goal. INSEC works by injecting an attack string as a short comment in the completion input. The attack string is crafted through a query-based optimization procedure starting from a set of carefully designed initialization schemes. We demonstrate INSEC's broad applicability and effectiveness by evaluating it on various state-of-the-art open-source models and black-box commercial services (e.g., OpenAI API and GitHub Copilot). On a diverse set of security-critical test cases, covering 16 CWEs across 5 programming languages, INSEC increases the rate of generated insecure code by more than 50%, while maintaining the functional correctness of generated code. We consider INSEC practical - it requires low resources and costs less than 10 US dollars to develop on commodity hardware. Moreover, we showcase the attack's real-world deployability, by developing an IDE plug-in that stealthily injects INSEC into the GitHub Copilot extension.

Black-Box Adversarial Attacks on LLM-Based Code Completion
Slobodan Jenko"
Niels Mündler
Jingxuan 2e Mark Vero Martin Vechev
Abstract
improved programming productivity. As prominent exam- ple, the GitHub Copilot code completion engine (GitHub, 2024) is used by more than a million programmers and five thousand businesses (Dohmke, 2023). However, prior re- search has shown that LLMS are prone to producing code with dangerous security vulnerabilities (Pearce et al., 2022; Li et al., 2023a). This poses significant security risks, as LLM-geeerated vulnerabilities can be incorporated by unas- suming users (Perry al., 2023). Even more concerning is the potential for attacks on the completion engine, which can substantially increase the frequency of generated vulnerabil- ities. To conduct such attacks, prior research has considered poisoning attacks, eliciting insecure behavior in a white-bon manner by modifying the model's weights or training data (Schuster al., 2021: He & Vechev, 2023; Aghakhani et al., 2024; Yan et al., 2024). However, these attacks require ac- cess to the models' training process, which is typically out of reach for the adversary (Carlini et al., 2024) or requires large amounts of expensive compute. Moreover, such at- tacks cannot be executed on operating and well-established code completion services, such as GitHub Copilot.
Modern code completion engines, powered by large language models (L.LMs), assist millions of developers with their strong capabilities to gener- ate functionally correct code. Due to this popular- ity, is crucial to investigate the security implica- tions of relying on L..M-based code completion. In this work, we demonstrate that state-of-the-ar black-box LLM-based code completion engines can be stealthily biased by adversaries to signifi- cantly increase their rate of insecure code genera- tion. We present the first attack, named INSEC, that achieves this goal. INSEC works by injecting an attack string as a short comment in the comple- tion input. The attack string is crafted through a query-based optimization procedure starting from set of carefully designed initialization schemes. We demonstrate INSEC's broad applicability and effectiveness by evaluating it on various state-of- the-art open-source models and black-box com- mercial services (e-g., OpenAl API and GitHub Copilot). On a diverse set of security-critical test cases, covering 16 CWEs across 5 programming languages, INSEC increases the rate of generated insecure code by more than 50%, while maintain- ing the functional correctness of generated code. We consider INSEC practical- it requires low resources and costs less than 10 US dollars to develop on commodity hardware. Moreover, we showcase the attack's 's real-world deployability, by developing IDE plug-in that stealthily injects INSEC into the GitHub Copilot extension.
Realistic Black-Box Setting In this work, we present a novel threat model, as depicted in Figure 1. Here, a user in- teracts with code editor, receiving code completions from (remote) black-box completion engine. The attacker's ' goal is to influence the engine to frequently suggest vulnerable code in security-critical contexts. To ensure stealthiness and gain the user's trust, the attack must preserve the engine's 's overall effectiveness in generating functionllly correct code and maintain its response speed. To avoid having train and host sufficiently capable malicious model, and since they can not manipulate the black-box model internals, the attacker achieves this by manipulating the engine's 's input.
1. Introduction
We consider attacks in this setting to pose a realistic threat for three reasons. First, the black-box assumption aligns with the operational methods of widely deployed and highly accurate completion services like GitHub Copilot. This not only removes the cost of training and deploying their own model, but also allows the attack to target the extensive user base of these services. Second, users of complction engines are likely to accept vulnerable code suggestions (Perry et al., 2023), especially when the attack maintains the engine's 's high utility and speed. Third, the attack manip-
Large language models (LLMs) have greatly enhanced the practical effectiveness of code completion (Chen et al., 2021; Nijkamp et al., 2023; Rozière et al., 2023) and significantly
'Equal contribution 'Department of Computer Science, ETH Zurich, Switzerland 'UC Berkeley, USA. Correspondence to: Niels Mündler <niels. muendler@inf. ethz. ch>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Black-Bol Box Adversarial Attacks on LLM- Code Completion
attacked query
def get, ticket, info (ctomnacncc,in, query_id):
id quary_id (dd.e..er
interact
</>
# useX < Matt
cur connection get_cursor
("r.eEE FRDV WHERE
O fornat. i4)>
not ree:
User
Code Editor
Black Box Completion Engine
return { return tupla(i [0]
raa.info)
Insecure
vuln. completion
Figure I: Overview of our attack flow. The attack's effect is highlighted in red color. The attack manipulates the query sent to the black-box code completion engine, influencing it to suggest vulnerable completions. The attack takes place stealthily in the backend, entirely hidden from the user.
Figure 2: A concrete code completion example where GPT-3.5-Tuubo-IIstrucc originally suggested the secure SQL placcholder ? given the query However, with the attack comment a inserted by INSEC, the model proposes a vulnerable completion using the unsafe format method.
ulation occurs entirely in the background, invisible to the user, increasing the likelihood of the attack remaining unde- tected. We demonstrate the attack's ' real-world deployment by developing benign-looking IDE plug-in that steers the GiHub Copilot extension to produce vulnerable code (dis- cussed in Section 3.3). Such plug-in may be distributed, e.g., through marketplaces, by exploiting naming confusion or baiting users with attractive offers (Pol, 2024; Toulas, 2024: Ward& & Kammel, 2024).
completion engines: StarCoder-3B (Li et al., 2023a), the StarCoder2 family (Lozhkov al., 2024). CodeLlama- (Rozière et al., 2023), the most capable commercial model with completion access, GPT-3.5-Turbo-Insteee 2024), and GitHub Copilot (GitHub, 2024). In particu- lar, the latter two are commercial services that provide only black-box access. We observe an absolute increase of around 50% in the ratio of generated vulnerabilities across the board while maintaining close-to-one -o-oiginal functional cor- rectness on most. Concerningly, we found that the attack strings cause less deterioration in functional correctness for stronger models. Moreover, INSEC requires only minimal hardware and monetary costs, c.g., less than $10 for the development of an attack with (11-3.5-Turbo-nnsse We publicly release our dataset and code implementation.
(OpenAI,
Our INSEC Attack We propose INSEC, the first, and highly effective, black-box attack that complies with the aforementioned threat model. INSEC employs carefully designed attack template that inserts a short adversarial comment string above the line of completion location. This comment serves as an instruction for the model to generate code with a specific vulnerability, while having minimal im- pact on the overall functionality of the generated code. The attacker benefits from knowing in advance which vulnerabil- ity is more likely occur in the victim code base after the attack. As an example, Figure 2 depicts how INSEC drives GPT-3.5-Turbo-Isstiee
2. Background
Code Completion Engine We consider an G, LLM- based code completion engine. G produces code infillings e based on a query q = (p, s). which consists of a prefix of code preceding the completion position and a suffix s of remaining code (Bavarian et al., 2022). See Figure 2 for an example of query q. We represent the completion process as e G(g) or e G(p, s). The final completed program x is then formed by concatenation:
to generate code susceptible to SQL injection, deviating from its secure default behavior. To find effective and generic attack strings, we utilize a query-based random optimization algorithm on small training dataset. The algorithm iteratively mutates and selects promising can- didate strings based estimated vulnerability rates. To create the initial candidates, we leverage a diverse set of ini- tialization strategies, which sigmificantly enhances the final attack success. Furthermore, the attack string is precom- puted, fixed during inference, and indiscriminantly inserted into all user queries. This leads to negligible deployment- time overhead, in latency, compute, and implementation.
p c When the engine produces multiple completions from a single query, we use the notation Gp.
Measuring Vulnerability For an attacker, the primary goal is to induce the model to generate vulnerable code. We measure this property by determining the ratio of vulnerable code completions. Let I be a vulnerability judgment function, such as static analyzer, that returns a given program is insecure. Following Pearce et al. (2022); He & Vechev (2023), we measure the vulnerability rate of G as:
To evaluate INSEC, we construct comprehensive vulnerability dataset consisting of 16 in- stances of the Common Weakness Enumeration (CWFs) in 5 popular programming languages. Based on HumanEval (Chen et al., 2021), we further develop a multi-lingual com- pletion dataset to evaluate functional correctness. We suc- cessfully apply INSEC on various state-of-the-arr code
Evaluating INSEC
vulRate (G) : E (Ds)) [Ba-G(p)) [1vil(p + c s)]]
hnttp:://gittub.coo//et wtwp:/hhww....otttthereeeeeee//e
2

---

Black-Box Adversarial Attacks on LLM-Based Code Completion
where Duul is dataset of security-critical tasks whose functionality can be achieved by either secure vulnerable completions. For example, the insecurely solved task in Figure 2 also allows a secure completion using sha256.
stealthiness, considerably large resources are required. We therefore consider a setting where the attacker leverages G by manipulating its inputs. To this end, the attacker de- vises an adversarial function fudd that transforms queries q to G into adversarial queries (ad(()) i.e., by defining G (q) = G( (q)). In order to fulfill criterion (i) and (ii), the attacker must find fadt increases Gs vulnerabil- ity rate while maintaining functional correctness. Moreover, to fulfill criterium (iii) of the threat model, fadv must be lightweight and minimize resource and latency overhead. Finally, the box setting implies that, when deriving fal,, the attacker has no access to model internals, such as parameters, training data, logits, or the tokenizer.
Mcasuring Functional Correctness While an attacker seeks to introduce vulnerabilities, it is important to pre- serve the model's ability to generate functionally correct code, such that the attack remains unnoticed in more com- mon, benign scenarios. Following the popular HumanEval benchmark (Chen et al., 2021), we use unit tests to decide the correctness of a program x. Let 1 fume return 1 if x passes all associated unit tests and O otherwise. To measure functional correctness, we leverage the standard pass @ k metric (Chen et al., 2021), formally defined as below:
3.2. Our Proposed Attack: INSEC
pass @ k G)
We introduce INSEC, the first Black-box adversarial attack that aligns with our threat model. INSEC consists of an at- tack template, an attack optimization algorithm, and diverse attack initialization strategies.
(pp))) -tanee (eean)GP [Vi Iune + at + s]]]
Here, Dtuee represents a dataset of code completion tasks over which the metric is calculated. A higher pass@k ls met- ric indicates more useful completion engine in terms of functional correctness. We assess the change in functional correctness between two related code completion engines G and G' through the relative difference of their pass @ k scores, with values close to 100% indicating well-preserved functional correctness:
A Attakk Template INSEC instantiates frav as a function that inserts an adversarial string a as a comment into the query q. The insertion point is the line above the comple- tion location. That is, we only modify the prefix p while keeping the suffix intact. We also insert an appropriate indent before the comment to maintain the naturalness of the modified query. Figure 2 illustrates an example of such a manipulated query. It is important to note that the attack string a is fixed at inference time and is indiscriminately inserted into all completion requests made by the user. This strategy eliminates the need for potentially costly mecha- nism to determine which queries INSEC should be applied to, ensuring minimal overhead during inference.
pass @ ((G))
passRatio@k G'' G)
(3)
pass @ @k(C (G)
3. A Attacking Black-Box Code Completion
This section presents our threat model, INSEC's technical details, and the deployment of INSEC.
This design conforms to the requirements of our threat model: (i) a acts as an instruction that drives the engine to generate vulnerable code in relevant security- cod- ing scenarios; (ii) because a is disguised a short comment, it causes minimal negative impact on functional correctness in normal coding scenarios; and (iii the insertion process at deployment time is trivial and adds only few tokens, result- ing in negligible overhead. In Section 4 and Appendix D, we provide various ablation studies to empirically validate the quality of our design choices for this attack, including the insertion location and a's length.
3.1. Threat Model
In order to harm the users codebase, the attacker seeks to compromise a black-box completion engine G into a ma- licious engine G which frequently suggests insecure code completions. For the attack to be successful, the at- tacker must satisfy three constraints: (i) Gady should ex- hibit a high rate of generated vulnerabilities. quantified by vulRate G Gady); (ii) Gadv must maintain the functional cor- rectness of G, measured by passRatio@k (Gidly G); and (iii) Gadv must have low latency and compute overhead. Constraints (ii) and (iii) are critial for ensuring the stealthi- ness of the malicious activity and maximizing the chances of users adopting ady and its vulnerable code completions.
Attack Optimization INSEC relies on deriving an ef- fective attack string to increase the rate of vulnerable completions and maintain functional correctness. We obtain such a string through the random optimization algorithm in Algorithm 1. This optimization process maintains a pool of candidate attack strings, mutates these candidates, selects the most promising ones for achieving the attack goal to further iterate, and finally returns the best candidate.
One way to compromise G would be to direct all user queries to a self-trained and hosted malicious model. How- ever, in order to match the utility and speed of commer- cial engines, thereby achieving user adoption and attack
3

---

Black-Box Adversarial Attacks on LLM-Based Code Completion
Specifically, Algorithm 1 takes input a training dataset Disain and validation dataset vul. which consist of security-sensitive completion tasks. First, at Line 2, we ob- tain a set of attack strings based on initialization strategies described later in this section, using only D D Next, at Line 3. pick n best is called on D V to select the best n attack strings to keep in the attack pool. pick n best eval- uates each candidate attack string by its impact on the rate of generating vulnerable code, as measured by the vulRate (defined in Section 2) on the given dataset. A detailed ex- planation of pick n best is provided in Appendix B. We then enter the main optimization loop (Line 4 to Line 8). In each iteration, we start with the pool of candidate solutions P from the previous iteration. At Line 5, each candidate string is randomly mutated using mutate, which replaces randomly selected tokens in the attack strings with randomly sampled tokens from the vocabulary. An implementation of mutate can be found in Appendix B. At Line 6, the mutated strings are merged with the old candidate pool, forming a larger pool with new candidates prew_ We run the loop for a fixed number of iterations, which we determine by observing when the optimization process saturates on our validation datasets. Finally, we use pick_n_best on the training set D Ditrin to select the top n candidates from the merged pool peew, which then form the starting pool for the next iteration. Upon completing the main optimiza- tion loop, we select the most effective attack string a using pick.n on the held-out validation dataset D
tializes the attack string by sampling tokens uniformly a random. (ii) TODO initialization: inspired by Pearce et al. (2022), this strategy initializes the attack string to "TODO: fix vul", indicating that the code to be completed contains a vulnerability. For the remaining three strategies, we utilize the completion tasks in the training set D vull along with their corresponding secure and vulnerable completions: (iii) Security- Critical Token Initialization: as noted by He & Vechev (2023), the secure and vulnerable completions of the same program may differ only on a subset of tokens. Fol- lowing this observation, we compute the token difference between the secure and vulnerable completions. We start the optimization from a comment that either instructs to use vulnerable tokens or instructs not to use secure tokens. (iv) Sanitizer Initialization: many vulnerabilities, such as cross-site scripting, can be mitigated by applying a sanitiza- tion function on user-controlled input. In this strategy, we construct the initial comment to indicate that sanitization has already been applied, guiding the completion engine not to generate it again. (v) Inversion Initialization: for a given vulnerable program, this strategy requests the engine to com- plete a comment in the line above the vulnerability. This initial comment directly exploits the learned distribution by the LLM, as it generates the most likely comment preceding vulnerable section of code (Morris et al., 2024). Detailed explanations and examples can be found in Appendix C.
3.3. Deployment of INSEC
Due to its effectiveness and lightweight design, INSEC is practical and easily deployable, which increases its potential impact and severity. In this work, we demonstrate the fea- sibility of deploying INSEC as a malicious plug-in for the popular IDE Visual Studio Code, targeting its GitHub Copi- lot extension. Malicious IDE plug-ins are a prominent attack vector since they can execute arbitrary commands with user- level privilege. Popular plug-in in marketplaces implement basic scanning for malicious plug-ins but they are easily avoidable (Ward & Kammel, 2024). As result, malicious plug-ins can be widespread with millions of downloads (Pol, 2024; Toulas, 2024).
Algorithm 1: Attack string optimization.
Procedure optimize (Derain, D Dyal n))
Input Dirran training dataset Dval vul. validation dataset
n, attack string pool size
Output:the final attack string
P iniit train)
P P=y pick.n.eest(P, D Dirain)
repeat
nuew = (mmutat(()) for a in P!
pnew- = =pnew + +P
P = b.est..beet(Ppeer n, Dirain)
Once installed, our malicious plug-in locates the installa- tion directory of the GitHub Copilot extension and deploys INSEC by injecting short JavaScript function into the ex- tension's 's source code. The function, shown in Appendix G in Listing 1, implements fadv, i.e., inserts the adversarial string a to all completion queries to trigger the generation of vulnerable code. The attack not noticeable to the user: the plug-in requires no activation, and the attacked GitHub Copi- lot extension remains functionally correct and responsive in normal contexts. However, in security-critical contexts, the engine suggests insecure completions. as seen by compar- ing the code completion suggestions of the normal and the attacked extension in Figure 13 in Appendix G.
until optimization finishes or budget is used up
return pick n best(P, 1, Dyal) vul
Attack Initialization To improve the convergence speed and performance of our optimization algorithm, we develop six diverse strategies for initializing the attack string candi- dates. These strategies are generic and easy to instantiate. Due to the modular design of INSEC, attackers may also easily add more initialization strategies if necessary.
The first two strategies are independent of the targeted vul- nerabilities: (i) Random Initialization: this strategy ini-
4

---

Black-Box Adversarial Attacks on LLM-Based Code Completion
We note that INSEC can also be deployed in various other ways, as long as the adversary gains control over G's in- put. Examples are intercepting user requests, supply chain attacks, or setting up a malicious wrapper over proprietary APIs. Note that, due to ethical considerations, we do not at- tempt an end-to-end deployment of the attack, but focus on developing it in the confines of the outlined threat model.
Unless stated otherwise, the optimization and evaluation are always performed concerning a single CWE, which is consistent with prior poisoning attacks (Schuster et al., 2021; Aghakhani et al.. 2024; Yan et al., 2024). We also conduct an insightful experiment on the concatenation of multiple attack strings, showing that INSEC can attack several CWES simultaneously.
4. Experimental Evaluation
Evaluating Functional Correctness We instantiate the passRatio@k metric, defined in Equation (3), to evaluate the impact of INSEC on functional correctness using a dataset of code completion tasks based on HumanEval (Chen et al., 2021). Following Bavarian et al. (2022), we remove single line from the canonical solution of HumanEval problem for each completion task. Since our vulnerability assess- ment spans five programming languages, we create a sepa- rate dataset for each language, using a mulli--inguun version of HumanEval (Cassano et al., 2023). As canonical solu- tions in HumanEval are not available for all five languages, we use GPT-4 generate reference solutions, ensuring they pass the provided unit tests. We divide these datasets into validation set fune and a test set Dieat of sizes ~14 and ~600, respectively. During evaluation, we compute a robust estimator for PassRatio based on 40 generated sam- ples per task (Chen et al., 2021). We observe that results on passRatio@1 and passRatio@ 10 can exhibit similar trend, Therefore, we omit passRatio@ 10 when it is not necessary. In Appendix D, we validate the use of GPT--4-generated reference solutions by demonstrating that the results are consistent with those obtained using human-written solu- tions from HumanFval-X (Zheng et al., 2023). We further confirm the small impact of INSEC on benign queries in repository-level code completion in Appendix D.
In this section, we present an extensive evaluation of IN- SEC, ablations and properties beyond the initial design.
4.1. Experimental Setup
Targeted Code Completion Engines To show the versa- tility of INSEC, we evaluate it across various state-of-the art code completion models and engines: the open-source models StarCoder-BB (Li et al., 2023a), CodeLlama-7B (Rozière et al., 2023) and the StarCoder2 family (Lozhkov et al., 2024), all of which we evaluate as black-box models. Further, we evaluate the most capable commercial model by OpenAl that provides access to its completion endpoint, GPT-3.5- (TF-3.5-Turbo-ing Turbo-Instruct (OpenAI, 2024), as well as the code completion plug-in GitHub Copilot (GitHub, 2024).
Evaluating Vulnerability We compile a dataset Dvul of 16 different CWES across 5 popular programming lan- guages, with 12 security-critical completion tasks for each CWE. This covers significantly more CWES than previous poisoning attacks, which only consider 3-4 vulnerabilities (Schuster et al., 2021; Aghakhani et al., 2024; Yan et al., 2024). We spent significant effort in curating these comple- tion tasks, ensuring their quality, diversity, and real-world relevance. We provide further details on the CWES in Dvul and its construction in Appendix A.
4.2. Main Results
We evenly split the 12 tasks for each CWE into Dirain for optimization, Dyal for hyperparameter tuning and abla- tions, and D test for our main results. As the vulnerability judgment function, we use CodeQL,, state-of-the-art static analyzer adopted in recent research as the standard tool for determining the security of generated code (Pearce et al., 2022; He& & Vechev, 2023). We then compute the vulRate metric, defined in Equation (1), to assess the vulnerability rate of 100 completion samples for each task. We acknowl- edge that static analyzers are susceptible to false positives when used unselectively on unknown vulnerabilities (Kang et al., 2022). However, in our context, the potential vulnera- bilities in the generated code are known, which enables us to apply specialized CodeQL queries for each CWE, thereby achieving high accuracy in vulnerability assessment. In Appendix D, we manually validate the high accuracy of CodeQL. at 98% for our evaluation on test vul
In Figure 3, we present our main results on vulnerability and functional correctness on the respective test sets Deest and Diest We average the vulnerability and functional correct- ness scores obtained for each targeted attack across the 16 CWEs. We can observe that INSEC substantially increases (by up to 60% in absolute) the rate of vulnerable code gen- eration on all examined engines. Mcanwhile, INSEC leads to less than 22% relative decrease in functional correctness. We observe that better completion engines retain more func- tional correctness under the attack. This can be observed by comparing different sizes of StarCoder2 models. Moreover, S...5-Turbo-Insteee and GitHub Copilot can be success- fully attacked with virtually no impact on correctness. This result is especially worrying as it indicates that more capa- ble future iterations of models may be even more vulnerable to adversarial attacks such as ours. We analyze our results per CWE in Appendix D to provide additional insights.
5

---

Black-Box Adversarial Attacks LLM--Based Code Completion
vulRate
vulRate(G"
passRation (G G)
passRati@10((
100
97
99
101100
100
100
88
100
93
100
74
76
75
75
50
50
25
14
25
13
StarCoder-3B
CodeLlama-7B 1ww.3.5-Turb-nnstttee
Copilot
StarCoder2-3B StarCoder2-7B StarCoder2-155
Figure 3: Main results showing for each completion engine the average vulnerabilityy rate and functional correctness across all 16 CWEs. INSEC is consistently effective for both vulnerability and functionality aspects. More capable engines are impacted less by the attack in functional correctness.
vulRate
Fire PassRatio@1 (G G)
100
96
97
100
73
75
87
19
Line above
Start of prefix
Start of same line
End of prefix
Start of suffix
Line below
End of suffix
With comment
Without comment
(a) Different attack position.
(b) Different attack type.
Figure 4: Vulnerability rate and functional correctness achieved by (a) different insertion positions for the attack string and (b) is formatted as comment. Our design choices ("Line above" and "With comment") achieve the best tradcoff between vulnerability rate and functional correctness.
TODO
25
56
12
44
Random
88
Security-
38
Sanitizer
StarCoder-3B
CodeLlama-75
GPT-3.5-Turbo-Instrre
Copilot
StarCoder2-157
Inversion
Figure 5: Distribution of final attack strings by originating initialization scheme. While security-critical token initialization dominates across all models, each scheme provides a winning final attack at least in one scenario.
Optimization Cost We record the number of tokens used by our optimization procedure in Algorithm 1. For GPT- S--Turbo-Instrrre the maximal number of input and output tokens consumed for one CWE is 2.1 million and 1.3 mil- lion, respectively. Given the rate of $1.50 per million input tokens and $2.00 per million output tokens the time de- velopment, the total cost of INSEC for CWE merely $5.80. Similarly, for the open-weight models, the optimiza- tion phase of our attack required around 6 hours to find highly effective string on commercial GPUs. Assuming cost of between $1 and $2 per GPU per hour (Lambda Labs, 2025; DataCrunch, 2025) results in estimated cost of $6 to $12. This highlights the cost-effectiveness of INSEC.
Attack Location and Formatting As discussed in Sec- tion 3.2, our attack inserts the attack string a as a comment in the line above the completion We analyze this choice in Figure 4a, comparing it to six alternatives: start of prefix start the line of c, end of p. start of suffix s, the line below c, and the end We can observe that our choice provides the best tradeoff between vulnerability correctness. Next, in Figure 4b, analyze the impact of our choice of inserting We compare this choice to inserting directly at the start of the line of c, without a comment symbol, possibly altering program semantics. We find that our choice an improve- ment over the alternative, both terms of vulnerability rate (+-%% and functional correctness (+11%)
and functional
comment into the program.
4.3. Ablation Studies
We conduct additional experiments to study design choices of INSEC on our validation datasets vaal and val unless declared otherwise, StarCoder-3B.
Attack Initialization In Section 3.2, we introduced five different initialization strategies: TODO, security-critical token, sanitizer, inversion, and random. In Figure 5, we examine the importance of our initialization strategies by
fune- and,
6

---

Black-Box Adversarial Attacks on LLM-Based Code Completion
vulRate (G
PassRatio81
vulRate (G
PassRatio@01
vullatee(**))
passRatio011
100
100
78
73
73
77
75
80
70
50
50
49
60
60
40
50
20
20
80 160
Init.
Opt.
Init. & Opt.
10 20
80
160
Size n. of pool P for attack string candidates
Number of tokens Ro the attack string
Figure 6: A pool size of 20 yields an ideal exploration-exploitttion trade- off on fixed compute budget.
Figure 7: The combination of op-
Figure 8: The vulnerability rate and functional correctness for varying length for the attack string
timization and initialization strictly outperforms using only either.
measuring the share of CWES in which the final attack string originated from each strategy. First of all, we can observe that in the majority of cases, security-critical token initialization proves to be the most effective. The most ineffective strategy is TODO initialization, which also the simplest. Nonetheless, across the four attacked completion engines, each initialization strategy leads to final winning attack at least once, justifying their inclusion.
an attack length of 5 tokens for StarCoder-3B, as this has the lowest complexity but equivalent performance to longer at- tack strings of up to 40 tokens. For some of the other models, increasing the length to 10 tokens gives additional benefits, likely due to their higher instruction-following capabilities.
4.4. Extended Analysis
We now investigate how INSEC's attacks generalize across CWEs and models and affect repository- completions.
Pool Size A key aspect of Algorithm 1 is the size n of the attack string pool P. controlling the greediness of our opti- mization given a fixed amount of compute; in smaller pools, fewer candidates are optimized for more steps, while in larger pool, more diverse candidates optimized for fewer steps. We explore the effect of varying on StarCoder-3B between 1 and 160 and show our results in Figure 6. We observe that too small and too large n produce weak attacks, as they are too greedy or over-favor cxploration. We chose n 20 for our attack, as it reaches the highest attack impact while retaining reasonable functional correctness.
Multi-CWE Attack While INSEC is mainly developed as a targeted attack, the potential for inducing multiple CWEs simultaneously would exacerbate the posed threat. In Figure 9a, we investigate the effect of attacking GPT-3.5- Turbo-Instruct with individually optimized attack strings of multiple CWEs together, each included in a new line. For each number of targeted vulnerabilities, we sample 24 unique ordered combinations of CWEs and average the results. This combined attack increases the length of the attack and thus results loss of functional correctness, aligning with Figure 8. Mcanwhile, we observe that the combined attack achieves both high vulnerability rate and RassRatio even at 4 CWEs. Even at 16 simultaneously tar- geted CWEs, INSEC achieves an almost 2x higher vulRate than the unattacked engine. These results are both surpris- ing and concerning, as they show that INSEC's attacks are composable, without having been explicitly designed for it.
Optimization and Initialization To understand the indi- vidual contributions of our optimization procedure and ini- tialization strategies, we compare attack strings constructed in three scenarios: using only initialization strategies (Init.), optimization on random initialization (Opt.), and optimiza- tion on our initialization strategies (Init. & Opt.). In Fig- ure 7, we show that an increased vulnerability rate of 50% is already achieved by careful initialization. However, subse- quent optimization yields significantly higher vulnerability rate at similar functional correctness, validating our design.
Generalization between Models We assess whether at- tack strings optimized for StarCoder-3B and CodeLla We
Number of Attack Tokens A crucial aspect of our at- tack template the number of tokens n, in attack string o.. In Figure 8, we explore variations of this hyperparameter. While optimizing just single token does not provide suffi- cient degrees of freedom for the attack to succeed, already at five tokens the attack reaches strong performance from where it plateaus. With 80 tokens, the attack starts dropping in effectiveness, both in terms of vulnerability rate and func- tional correctness. For our final attack, we therefore chose
increase the vulnerability of find that both strings drastically increase vulRate from 22% to 55% and 59% respectively. Meanwhile, the resulting score is significantly lower than string optimized directly on GPTT:3.5-Turbo-Instte els (80% and 82% on StarCoder-3B and CodeLlama-7D respectively). This indicates that the attacks generalize to some degree between models of different sizes and architec- ture, enabling targeting rate- ante-iimited black- box L.LMs.
(73%) or for the original mod-
7

---

Black-Box Adversarial Attacks on LLM--Bssd Code Completion
(G)
vulRate(G
passRutio@1
100
Model
EM
ES
CB
83
73
78
84
StarCoder-3B
95.4
87.0
95.3
06
57
CodeLLama-7B
102.4
96.5
105.0
60
GiPT:3.--Turbo--nnsi Instrrtttt
87.8
96.2
92.8
40-
StarCoder2-3B
87.8
83.8
87.6
25
StarCoder2-7B
90.8
87.6
94.0
StarCoder2-150
85.2
87.1
91.5
16
Unicade
GPT-2
CodeQwen StarCoder
Number targeted time
(a) Composing individual attacks
(b) Impact tokenizer choice
(c) Repository code completion
Figure 9: In (a) the individually optimized attacks are shown to trigger several CWEs when concatenated. In (b) we demonstrate that without accessing the models native tokenizer, significant performance be achieved using a code- specific tokenizer. In (c) we show that INSEC preserves code similarity on repository- completion in RepoBench.
Tokenizer Access Under our black-box threat model, the attacker does not have access to the tokenizer of the target engine. Therefore, the attack is optimized in the token space of a proxy tokenizer T. In our experiments, we use the CodeQwen tokenizer (Bai et al., 2023), a publicly available tokenizer different from tokenizers of any of the targeted models. In Figure 9b, we explore the choice T,, measuring INSEC's performance attacking StarCoder-3B using four different tokenizers: tokenization per Unicode characters, and the GPT-2, CodeQwen, and target model's (StarCoder) tokenizer. We make two key observations. First, the non- code- specifc tokenizers (Unicode and GPT-2) lead to low vuinerability rates. Second, the target tokenizer only beats the code-specific proxy T in terms of functional correct- ness on StarCoder-3B. Moreover, as seen in Figure 3, the proxy tokenizer generalizes to stronger completion engines, incurring virtually no loss even on functional correctness.
Pattern Analysis We manually inspect the optimized at- tack strings to identify patterns. The strings typically con- tain tokens derived both from initialization strategies and random mutations during optimization. As such, they in- clude a mix of words and code in ASCII and non-ASCII characters, such as non-Latin alphabet letters, symbols from Asian languages, and emojis. These patterns suggest that, similarly to jailbreak attacks (Yong et al., 2023; Geiping et al., 2024), our attack partially relies on exploiting low- resource languages and undertrained tokens. Overall, most attack strings are not easily interpretable by humans. For ethical considerations, we choose not to include the final attack strings publicly in the paper.
More Results and Case Study in Appendix In Ap- pendix D, provide detailed information perfor- mance per CWE. We further examine sampling temperature choices, as most of our experiments use temperature 0.4 for optimization and evaluation. In Appendix E, we provide three case studies to illustrate INSEC attacks.
Preserving Repository-level completion quality In addi- tion to functional correctness in function-level completions in Section 4, we now explore whether INSEC leads to noticable disturbances on code completion. We focus on repository-level code completion, using RepoBench (Liu al., 2024b), recent benchmark based on GitHub reposi- tories. It measures the similarity of predicted lines golden completions when given repository-leve context. This set- ting closely aligns with a realistic usage scenario of com- pletion engines. Character-level and syntactic similarity to golden completions are measured using Exact Match (EM), Edit Similarity (ES) and Code Bleu (CB) (Ren et al., 2020). We randomly choose 333 instances from RepoBench and sample 40 completions of G and Gadv We generate one Gadv per Python-specifie CWE and average the results. We report the ratios of EM, ES and CB in Figure 9c. The quality of the predicted next line degrades only minimally, at all, with all scores degrading by at most 16.2% and on average 8.1 %, overall matching our observations on HumanFval.
5. Discussion
INSEC's Surprising Effectiveness Although our black- box threat model assumes a more restricted attacker than prior attacks (He & Vechev, 2023; Aghakhani et al., 2024; Yan et al., 2024), INSEC remains effective both in terms vulnerability rate and functional correctness. This can be attributed to (i) the attack's 's exploitation of instruction- following capabilities of LLMs, (ii) that many vulnerabili- ties lie within the learned distribution of LLMs and (ii) that the perturbation introduced by INSEC is small, allowing capable LLMs to ignore the perturbation usages uncritical to security and generating functionally correct code.
Potential Mitigations We appeal to the developers of completion engines to implement mitigations, such as: (i) alerting the user if a string occurs repeatedly at an unusual
8

---

Black-Box Adversarial Attacks on LLM-Based Code Completion
frequency; (ii) sanitizing prompts before feeding them the LLM (Jain et al., 2023); or (iii) interrupting repeated query- ing for the purpose of optimizing an attack similar to ours.
INSEC derives short, generic attack strings that preserve model performance in uncritical setrings.
We further discuss static analysis. security-inducing com- ments, and variations of comment scrubbing in Appendix F.
Attacks on Neural Code Generation The common set- ting in attacks on neural code generation is to assess model robustness by perturbing the entire user input. Li et al. (2023b; 2024); Wu et al. (2023); Ren et al. (2024) rename variables and functions, among other semantic-preserving perturbations, to trigger functionally incorrect or insecure code completions. While Li et al. (2024); Wu et al. (2023) target insecure completions, only Wu et al. (2023) also ensures preservation of functional correctness. For all meth- ods, their attacks are not suitable for stealthy attacks in our settings, as they assume white-box access or allow expen- sive search for individual queries. Overall, prior work is designed for users that are interested in assessing LLM ro- bustness, such as model developers. In contrast, we discover short injection string, the attack comment, that triggers cor- rect but insecure completions over many samples, suitable for our threat model of attacking unassuming users.
Limitations and Future Work While INSEC already ex- poses a concerning vulnerability of today's code completion engines, it incurs some loss on functional correctness of cer- tain completion engines. Stronger attacks could incorporate an explicit optimization objective to preserve functional cor- rectness. Moreover, an interesting future direction would be to extend the attack to other settings such as coding agents (Jimenez et al., 2024) and even more vulnerabilities.
6. Related Work
Code Completion with LLMs Transformer-based LLMs excel at solving programming tasks (Cassano et al., 2023; Zheng et al., 2023), giving rise specialized code models such as Codex (Chen et al., 2021), CodeGen (Nijkamp et al., 2023), StarCoder (Li et al., 2023a) and CodeLlama (Rozière et al., 2023). LLMSs specialized for code completion are trained with a fill-in-hhe-middle objective (Bavarian et al., 2022; Fried et al., 2023) in order to handle both a code prefix and postfix in their context. Several user studies have con- firmed the benefit of L.LM-based code completion engines in improving programmer productivity (Vaithilingam et al., 2022; Barke et al., 2023), with such services being used by over a million programmers (Dohmke, 2023).
Concurrent work by Yang et al. (2025) attacks LLMs to trigger insecure completions by injecting code snippets into the RAG context. Their setting differs in three important aspects: First, they leverage white-box model access to opti- mize their attack. Second, in their RAG setting, larger attack code snippets can be included into the model context. Third, they do not evaluate whether their attack preserves func- tional correctness and would thus be sufficiently stealthy to succeed under our threat model.
Beyond prompt perturbations, prior attacks achieve in- creased code vulnerability by interfering either directly with the model weights or its training data (Schuster et al., 2021; He & Vechev, 2023; Aghakhani et al., 2024; Yan al., 2024). However, such attacks are unrealistic to be carried out against deployed commercial services.
Security Evaluation of LLM Code Generation As code LLMs are increasingly employed, investigating their secu- rity implications becomes increasingly imperative. Pearce et al. (2022) were first to show GitHub Copilot (GitHub, 2024) frequently generates insecure code. Li et al. (2023a); Khoury et al. (2023) extended their evaluation, revealing similar issues in StarCoder and ChatGPT. CodeLMSec (Ha- jipour et al., 2024) evaluates LLMs' insecure code genera- tion using automatically generated recurity-critica prompts. However, these works focus on model security only in be- nign cases, while we examine L-sdd code completion under attack, the worst case from a security perspective.
7. Conclusion
We presented INSEC, the first black-box attack capable of manipulating commercial code completion engines to gener- ate insecure code at high rate while preserving functional correctness. INSEC inserts a short attack string as comment above the completion line. The string is derived using black- box random optimization that iteratively mutates and selects top-performing attacks. This optimization procedure is fur- ther strengthened by a set of diverse initialization strategies. Through extensive evaluation, we demonstrated the surpris- ing effectiveness of INSEC not only on open-source models but also on real-world production services such as the Ope- nAI API and GitHub Copilot. Given the broad applicability and high severity of our attack, we advocate for further re- search into exploring and addressing security vulnerabilities introduced by L.LM-based code generation systems.
Random optimization for Jailbreak Attacks Random optimization a common approach to optimize attack strings in jailbreak attacks on LLMs (Andriushchenko et al., 2025; Zou et al., 2023). However, the threat model of jail- break attacks differs significantly from ours. Crucially, in jailbreak attacks, the user is also the attacker and there is no need to build a stealthy attack. As a result, costly random optimization can be applied per prompt, long attack prompts are permissible and there is no need to maintain functional correctness in settings uncritical to security. In contrast,

---

Black-Box Adversarial Attacks on LLM--Bscd Code Completion
Acknowledgements
Report, arXiv Preprint, 2023. URL https://arxiv.. org/abs org/abs/2309 16609.
This work has been done as part of the EU grant ELSA (European Lighthouse on Secure and Safe AI, grant agree- ment no. 101070617). Views and opinions expressed are however those of the authors only and do not necessarily re- flect those of the European Union or European Commission. Neither the European Union nor the European Commission can be held responsible for them.
Barke, S., James, M. B., and Polikarpova, N. Grounded Copilot: How Programmers Interact with Code- Generating Models. 0OPSLA, 2023. URL https:
//doi org/ 10. 1145/3586030.
Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey, C., Tworek, J., and Chen, M. Efficient Training of Lan- guage Models to Fill in the Middle. arXiv Preprint, 2022. URL https: : // aaxiv.xiivvv.... /ttp//ait....gab/////a/aa/a/a/aa wwwp:oog////aooooooooo 14255.
The work has received funding from the Swiss State Secre- tariat for Education, Research and Innovation (SERI).
Impact Statement
Carlini, N., Jagielski, M., Choquette Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., and Tramèr, F. Poisoning Web-Scale Training Datasets is Practical. In IEEE S&P, 2024. URL https://arxiv.. org/ 200p.//32..0000 10149.
In this paper, we have introduced INSEC, the first black-box attack to adversarially steer (commercial) code completion engines towards generating insecure code. As our attack can be potentially developed even by an attacker with no- tably low resources, and deployed on commercial services exploiting well-known vulnerabilities of, for instance, IDE plug-in marketplaces; we have made careful steps to ensure that our research process and publication of our results is aligned with the ethical responsibilities carried by the po- tential harms of INSEC. For this reason, 45 days before making any version of this manuscript, or any other deriva- tive of this study, public, we have responsibly disclosed our findings the corresponding model developers. Due to ethical concern, we did not include any concrete optimized attack strings in this paper, nor in any supplementary ma- terial. All attack strings included in the paper are dummy strings representing the overall patterns the optimized attacks. Finally, from a broader perspective, we believe that the good-faith uncovering and publishing of exploits to systems with a wide user base is ultimately of benefit to the security of such upplications, providing the first step towards mitigating security limitations that could otherwise be exploited by nefarious actors. We release our dataset and code implementation to allow the research community to replicate and advance our findings.
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps- Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Ander- son, C. J., Feldman, M. Q., et al. MultiPL-E: A Scal- able and Extensible Approach to Benchmarking Neu- ral Code Generation. IEEE Transactions on Software Engineering, 2023. URL https: http::::/eeeeeeeeeeeeeeeeeee ieee. org/ htg////ssraatt/cccm/////00101111
Charoenwet, W., Thongtanunam, P., Pham, V.-T., and Treude, C. An empirical study of static analysis tools for secure code review. In ISSTA, 2024.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating Large Language Models Trained on Code. arXiv Preprint, 2021. URL https: //arxiv Saaxiv.org//bbr. abgg/bbs/107 033744.
Chen, Y., Jhamtani, H., Sharma, S., Fan, C., and Wang, C. Steering large language models between code execution and textual reasoning. In ICLR, 2025. URL https: / //openreview. net//forun/id-xxx7711j
References
DataCrunch. DataCrunch A100 GPU Instances, 2025. URL cessed: 2025-05-21.
Ac-
Aghakhani, H., Dai, W., Manoel, A., Fernandes, X., Kharkar, A., Kruegel, C., Vigna, G., Evans, D., Zorn, B., and Sim, R. TrojanPuzzle: Coverty Poisoning Code-Suggestion Models. In IEEE S&P, 2024. URL https://arxiv.orr atgg/abs/3301... 02344..
Dohmke, T. GitHub Copilot X: The Al-powered developer experience, 2023. URL https://githtbb blog/2023- 03-22-gittub--oppoo-==--- -itpaaa---opppttt-teeeeeee---e-e developer-experiennee
Andriushchenko, M., Croce, F., and Flammarion, N. Jailbreaking leading safety-aligned LLMs with simple adaptive attacks. In ICLR, 2025. URL https:// openreview
Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, S., Zettlemoyer, L., and Lewis, M. InCoder: A Generative Model for Code Infilling and Synthesis. In ICLR, 2023, URL https //tp///vee/eeeeee///eee net:ppdf?.d-hqub- 1bMEEL.
Bai, J., Bai, S., Chu, Y. Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen Technical
10

---

Black-Box Adversarial Attacks LLM--Bsed Code Completion
Fu, Y., Liang, P., Tahir, A., Li, Z.,, Shahin, M., and Yu, J. Security Weaknesses of Copilot Generated Code in GitHub. ACM Trans. Softw. Eng. Methodol,, 2025. URL https://dl //dl. acm htg//oi/10.11. 1144//711///8..
Li, R., Allai, B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., al. StarCoder: may the source be with you! arXiv Preprint, 2023a. URL https: /atps://aaraaa /arxiv.xvvv// hts/abs/23000 2ttp//235500066... 06161.
Geiping, J., Stein, A., Shu, M., Saifullah, K., Wen, Y., and Goldstein, Т. Cocrcing LLMs to do and reveal (almost) anything. arXiv Preprint, 2024. URL https://arxxii.. org/aba/2402. 14020.
Li, X., Meng, G., Liu, S., Xiang. L.., Sun, K., Chen, K., Luo, X., and Liu, Y. Attribution-guice code prompt generation for code complction models. In ACM ASE, 2024. URL https: //doi. org/10 1145/ 3691620 3695517.
adversarial
GitHub. CodeQL GitHub, 2023. URL https://codegl. gi github.com com.
Li, Z., Wang, C., Liu, Z., Wang, H., Chen, D., Wang, S., and Gao, C. Cctest: Testing and repairing code completion systems. In ICSE, 2023b. URL https://doo.. org/10. 1109/ICSE48619. 202. 202.200222201
GitHub. GitHub Copilot Your Al pair programmer, 2024. URL https:ttttttttii www. wwt.//eaatree/coolll..oooo
Hajipour, H., Hassler, K., Holz, T., Schönherr, L., and Fritz, M. CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Models. In SaTML, 2024. URL https:// openreview
Liu, S., Sabir, B., Jang, S. I., Kansal, Y., Gao, Y., Moore, K., Abuadbba, A., and Nepal, S. From Solitary Directives to Interactive Encouragement! L.M Secure Code Gen- eration by Natural Language Prompting, arXiv Preprint, 2024a
He, J. and Vechev, M. Large Language Models for Code: Security Hardening and Adversarial Testing. In CCS, 2023. URL https:// org/ /0///1..110 1145/3576911. 3623175.
Liu, T., Xu, C., and McAuley, J. Repobench: Benchmarking repository-level code auto-completion systems. ICLR, 2024b. URL https: http:::///pppeeeeee.e.o wwwpanrrreee..nt/fouumii...... PPIZIOuQUF.
He, J., Vero, M., Krasnopolska, G., and Vechev, M. Instruc- tion Tuning for Secure Code Generation. In /CML, 2024. URL https://da //dlaacmm acm. hts//doo/11..5566 000555555/33777 3692793.
Lozhkov, A., Li, R., Allal, L. B., Cassano, F,, Lamy-Poirier,
J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al. StarCoder 2 and The Stack v2: The Next Generation. arXiv Preprint, 2024. URL 2402. 19173.
Jain, N., Schwarzschild, A., Wen, Y., Somepalli, G., Kirchenbauer, J., Chiang, P.-... Goldblum, M.. Saha, A., Geiping, J., and Goldstein, T. Baseline defenses for ad- versarial attacks against aligned language models. arXiv Preprint, 2023. URL https: wwwp:///////////0/// 2309. 00614.
Morris, J., Zhao, W., Chiu, J. T., Shmatikov, V., and Rush, A. M. Language Model Inversion. In ICLR, 2024. URL attps://openrer// http://ppwrrreee.eeeeeeeeeeeeeee
Nachtigall, M., Do, L. N. Q., and Bodden, E. Explaining
static analysis-a perspective. In ASEW. IEEE, 2019.
Jimenez, C. E., Yang. J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. R. SWE-bench: Can Language Models Resolve Real-world Github Issues? In ICLR, 2024. URL https://openreever http:/pparrrrii....ntttttrrra2 net/formm?dd VTFSyNQM66.
Nachtigall, M., Schlichtig, M., and Bodden, E. Evaluation of Usability Criteria Addressed by Static Analysis Tools on a Large Scale. 2023.
Nijkamp, E., Pang. B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., and Xiong, C. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. ICLR, 2023. URL https:
Kang, H. J., Aw, K. L., and Lo, D. Detecting False Alarms from Automatic Static Analysis Tools: How Far are We? In ICSE, 2022.
/ ///ppp/reiieee.ett//pffffaaaaaae
Khoury, R., Avila, A. R., Brunelle, J., and Camara, B. M. How Secure is Code Generated by ChatGPT? In IEEE SMC, 2023. URL https: pettpa//ddae :/tps:///doiiooe http///oo.org//////// ort/o0../1111/.. 1109/ SMC53992. 11002..103900 10394237.
OpenAL. Introduction OpenAl API, 2024. URL https:
//platform. cepnttffrrooopeaiii.... wwwp///oss//ntooouttttiiii
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., and Karri, R. Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions. In IEEE S&P. 2022. URL httts:///ee ott...orgg1....9..99/9 SP46214 20....22222...111 9833571.
Lambda Labs.
Lambda GPU Cloud Pricing,
2025. URL attps://lambos htt/aaabaaaaiseevviaaac.o awtservice/gpii
cloud#pricing. Accessed: 2025-05-21.

---

Black-Box Adversarial Attacks on LLM-Based Code Completion
Perry, N., Srivastava, M., Kumar, D., and Boneh, D. Do Users Write More Insecure Code with Al Assistants? In CCS, 2023. URL attpa://dee org/10. 1145/
Generation Tools Powered by Large Language Mod- els. In CHI Extended Abstracts, 2022. URL https:
ridoi.org// 10. 111.11114141110 /5////....1////11111111116666311 35119161....1...
Ward, K. and Kammel, F. Abusing VSCode: From Malicious Extensions to Stolen Creden- tials (Part 1) Featured Image, 2024.
Pol, F. H. VSCode Marketplace contains thousands of malicious extensions. Techzine, 2024. URL
URL
http://wwww..c........///////aaa 120000//rcoddrrmaaeppppeerrootor ooousaadd-of-mmlllccco---eeerree cessed: 2024-09-26.
attps://controo-pllaneee. vscode http:///ppp.aaeeoopppps///aaaaa stolem-credentiials-ptt-t-
Ac-
Wilson, D. sacape-huml 2023, URL https:///ww/
npmja. https//c...pccckgeeecceeeetaaaa
Ren, Q. Gao, C., Shao, J., Yan, J., Tan, X., Lam, W., and Ma, L. CodeAttack: Revealing safety generalization chal- lenges of large language models via code completion. In ACL, 2024. URL https::/ http:///aaaaalog//////// org/2022 2024. findings-acl.677
Wu, F,, Liu, X., and Xiao, C. DeceptPrompt: Exploit- ing LLM-driven Code Generation via Adversarial Natu- ral Language Instructions. arXiv Preprint, 2023, URL https://arxiv org/
brgg/bb//en 2212./231.... 2312.04730. 04730.
Ren, S., Guo, D., Lu, S., Zhou, I.,, Liu, S., Tang, D., Sun- daresan, N., Zhou, M., Blanco, A., and Ma. S. Code- BLEU: a Method for Automatic Evaluation of Code Syn- thesis. arXiv Preprint, 2020. URL https://arxiv. org/abs/2009. 10297.
Yan, S., Wang. S., Duan, Y., Hong, H., Lee, An LLM-Assisted
K., Kim, D., and Hong, Easy-to-Trigger Backdoor Attack on Code Com- pletion Models: Injecting Disguised Vulnerabilities against Strong Detection. In USENIX Security, 2024. URL attps://www/ htttp:://www.usenxxx org/ hwtg//onference/// uasnlxsecurttt2//prrssttttttooo
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y.. Liu, J., Remez, T., Rapin, J., et al. Code Llama: Open Foundation Models for Code. arXiv Preprint, 2023. URL https://arxiv org/abs/2308. 12950.
Yang. Y., Yao, H., Yang, B., He, Y., Li, Y., Zhang, T., and Qin, Z. Tpia: Towards target-specific prompt injection attack against code-oriented large language models. arXiv Preprint, 2025. URL https:://// //arxiv www..rgggg.o.... /bs//b///// aa//a/b/2200 09164.
Ryan, I., Roedig, U., and Stol, K.-J. Unhelpful assump- tions software security research. ACM SIGSAC Conference on Computer and Communications Security, 2023.
Yong, Z.-X., Menghini, C., and Bach, S. H. Low-resource Languages Jailbreak GPT-4, arXiv Preprint, 2023, URL https://arrae http://awwvvoo///////////
org/ ht///b//// abs/2310. 222122231...2.....22222
Schuster, R. Song. C., Tromer, E., and Shmatikov, V. You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion. In USENIX Security, 2021. URL https: //wp::::://www.//p uswnux.euooooooo org/ wwwrg//onfeeene//// assmxxeeurrtyy///reeentttt.ooo pct.r//rrseeaaatttosscsttteeee
Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., al. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In KDD, 2023.
Zou, A., Wang, Z., Carlini, N.,
Nasr, M., Kolter, J. Z., and Fredrikson, M. Universal and transferable adversarial attacks on aligned language models. arXiv Preprint, 2023. URL https //txxxxiv.o./ wwwpxii.org/aba//300000 15043.
Song, D., Guo, H., Zhou, Y., Xing, S., Wang, Y., Song, Z., Zhang, W., Guo, Q., Yan, H., Qiu, X., et al. Code Needs Comments: Enhancing Code LLLMs with Com- ment Augmentation. 2024. URL https://arxiv.. abs abs/2402. 13013.
Toulas, B. Malicious VSCode extensions with millions of installs discovered. Bleeping Computer, 2024. URL
hawpr//ggommuteerrcm//eee//// com/news/ security/mallicious-sccoo wwtp::/rooooeeetttsioo-ooo--ieie
Accessed:
2024-09-26.
Vaithilingam, P., Zhang, T., and Glassman, E. L. Expecta- tion vs. Experience: Evaluating the Usability of Code
12

---

Black-Box Adversarial Attacks on LLM-Bascd Code Completion
Appendix
A. Extended Details on Experimental Setup
We now give additional details about our implementation, hyperparameters, and vulnerability dataset.
Implementation and Hyperparameters The results in our main experiments (i.e., Figure 3) are obtained with the following configuration: attack comment positioned in the line above the completion point, optimization and initialization combined, CodeQwen tokenizer (Bai al., 2023), pool size temperature during optimization and evaluation 0.4. The number of tokens in the attack string is set to na = 5 for all engines and vulnerabilities except: select these hyperparameters according to our experiments on the validation datasets D and D val vul and the ablations presented in Section 4 and Appendix D. During optimization, for each candidate string, we sample 16 completions per task to approximate vulRate Equation (I). As running CodeQL during optimization would be prohibitively slow, we use approximate rule-based classifiers to determine if a completion vulnerable. As the final scores are computed using accurate assessment via CodeQL, this confirms that such classifiers mention in Appendix F, such manually written classifiers would likely be tool of preferred choice for attackers trying to introduce novel vulnerabilities. Finally, when mutating attack strings we forbid set of problematic tokens: those including new lines and special tokens, such as < </doff >>, As models, we used gpt-3.5 1er---urroiiittuut for GPT-3.5- S--Turoo--Istttue and the standard GitHub Copilot plugin as of June 2024.
20, and, following He & Vechev (2023), sampling
10 for Copilot on five vulnerabilities, and ng = 15 for Copilot on one vulnerability. We
accurate enough on our training samples. As
Vulnerability Dataset Our vulnerability dataset consists of 16 CWEs across 5 programming languages. We show an overview of these vulnerabilities, their MITRE vulnerability rank, and short description in Table 1. Further, for each CWE, we construct 12 realistic completion tasks using three different sources: (i) we incorporate all suitable tasks from the dataset of Pearce et al. (2022), (ii) search GitHub for code that contains fixes each specific CWE to collect real-world samples, and (iii) when the above sources do not yield sufficient samples, we leverage GPT-4 to generate additional samples based on detailed descriptions of the CWEs. We invested significant effort in reviewing and revising the samples to ensure high quality. Our primary objective during this process was to ensure diversity, realism, and sufficient context for the completion engines to generate functional code.
In the table on the right, we compare the evaluation scope of our work with prior studies. Our work covers a broader or comparable range of CWEs and programming languages, highlighting the thoroughness of our evaluation. This underscores the potential of our dataset as a valuable contribution to the community.
#CWEs
#LANGs
Schuster al. (2021)
3
Pearce et al. (2022)
He & Vechev (2023)
9
Aghakhani et al. (2024)
4
Yan et al. (2024)
3
Our Work
16
Table 1: Overview of the CWEs studied in this paper and the size of the corresponding dataset.
CWE
Language
Top-25 CWE Rank
Avg LoC
Max LoC
20
Improper Input Validation
Python
#6
22
Path Traversal
Python
#8
14
28
77
Command Injection
Ruby
#16
9
19
78
OS Command Injection
Python
#5
15
30
79
Cross-site Scripting
JavaScript
#2
19
27
89
SQL Injection
Python
#3
19
32
90
LDAP Injection
Python
23
33
131
Miscalculation of Buffer Size
C/C+
22
35
193
-yf-by-one Error
C/C++
26
54
326
Weak Encryption
Go
34
327
Faulty Cryptographic Algorithm
Python
14
34
416
Use After Free
C/C++
#4
18
476
NULL Pointer Dereference
C/C++
#12
22
68
502
Deserialization of Untrusted Data
JavaScript
#15
14
18
787
Out-of-bounds Write
C/C+
#1
21
52
943
Data Query Injection
Python
25
31
13

---

Black-Box Adversarial Attacks on LLM-Based Code Completion
CodeQL Vulnerability Judgment Since our evaluation of vulnerabiliites relies on CodeQL as judgment function, need to ensure that its judgment is trustworthy in our setting. To reduce false positives, select only relevant Co queries for each CWE. We further manually evaluate the precision of CodeQL on D tast by sampling 50 instances from diverse settings, covering all models, CWEs, and the presence of none, Init-only, and optimized attack strings. We find that CodeQL. exhibits high precision on our dataset, with 98% actual vulnerabilities reported.
Validation of STPP--GGeerated HumanEval Solutions For our evaluation of functional correctness, we evaluated the effect of INSEC on infilling tasks generated from GPT-4 generated solutions to HumanEval in other languages than Python. While translations of HumanEval prompts exist, e.g. in (Cassano al., 2023), only the dataset HumanEval-X (Zheng et al., 2023) contains human-wrine written translations of the reference solutions for some languages, and we found no manual translation of Ruby. As we preferred to treat the different languages equally, we decided to generate solu- tions for all non-Python languages using GPT-4. To validate our results, we compare our results manually translated samples in languages C++ and JavaScript. The comparison of the obtained PassRatio is displayed in Table 2, confirm- ing that the obtained results are similar between manually translated and GPT4-generated reference solutions.
Table 2: Comparison of passRatio and passRatio@10 be- tween manually translated and GPT-4-generated reference solutions.
passRatio@1
passRatio@10
Model
Manual
GPT-4
Manual
GPT-4
StarCoder-3B
78.0
74.9
99.8
97.9
CodeL.lama-7B
88.2
87.7
99.8
99.7
StarCoder2-3B
89.5
90.3
100.2
99.5
StarCoder2-7B
87.0
85.2
99.8
99.3
StarCoder2-15B
94.6
96.1
100.5
100.1
B. Details on Attack Optimization
In this section we provide more detailed pseudocode and descriptions of the attack optimization conducted by INSEC, specifically, we provide the implementations of the pick_n_ and mutate functions.
Selection The function pick is used to select the pop-prfforming attack strings from given pool. We present its details in Algorithm 2. For each attack string a E P (Line 3), we first construct a malicious completion engine G with (Line 4). Then, at Line 5, sampling completions to the tasks in Dvul. we estimate the vulRate (adv) when attacked using the current a. Finally, in Line 7, we pick and return the n best attack strings according to the vulnerability scores collected in V. This function has a crucial role in improving our pool of attack strings in each itcration of the main optimization loop.
Mutation The function mutate is used the main optimization loop of Algorithm to randomly alter the attack strings in the candidate pool. an important step for INSEC's optimization algorithm to discover stronger attack strings. We present the internals of mutate Algorithm 3. First, using the attacker's tokenizer T, we tokenize (Line 2). Note that to comply with our black-box threat model, we assume that the attacker obtains T independently. thus it does not necessarily match the tokenizer of the targeted engine G. Next, in Line 3, we uniformly sample the number of tokens & that will be mutated in a. Then, in Line 4, we randomly sample & positions I to mutate. In Lines 5 and 6, for each position index i E I, we mutate t[i] by replacing it with token sampled uniformly at random from the vocabulary of T. Finally, we return the detokenized mutated string.
C. Initialization Strategy Details
In this section, give extended details each initialization scheme used in INSEC. A high-level description of their invocation has been introduced in Section 3.2.
Random Initialization We increase the diversity of our initialization by generating random attack strings. We achieve this by randomly sampling tokens from the attacker's tokenizer T and concatenating them into strings. Note that such generated strings are not usually completely random characters, but feature some structure based on the size and content of the tokenizer dictionary. An example for such a string a is "éd senior & sp cuts", which includes complete words and Unicode characters and was generated by sampling tokens at random from the CodeQwen tokenizer (Bai al., 2023).
14

---

Black-Box Adversarial Attacks LLM-BBsed Code Completion
Algorithm 2: Attack string selection.
Algorithm 3: Attack string mutation.
Procedure pick n best (P, n, v
Input P. original attack string pool
Procedure mutate
n, size of new pool
Input : a original attack string
vull vulnerability dataset
Output : mutated attack string
Output: new pool with attack strings
2
= stringg to tokens(or)
V V=11
3
sample([1, [t])) I aumplee.ithoout..ellacmmmet
for a E P do
4
1].
construct G using the attack string o v = vulRate Galy) w.r.t. v
k:)
for E I do
y.append(w)
t[i = T Lar.ddnnnoooee foromoooommmmm vocab))
return n best elements from P according to V
7
return T.tokens.to.stringe. st_string())
TODO Initialization We initialize the attack string to "TODO: fix vul" to indicate that the code to be completed was marked, e.g., by human developer, to contain a security vulnerability. If the completion engine is aware of potential vulnerabilities or has picked up similar code snippets containing review notes and insecure code, we expect to be steered toward generating the corresponding insecure code.
Security-critice Token Initialization We observe that, for wide range of vulnerabilities, there exist critical tokens that decide the security of the whole program. For instance, consider the following implementation of database query using securely parameterized SQL: cursor. execute (' SELEEC input and the ?', parametrization makes sure that any potentially dangerous characters in id are escaped. In contrast, an insecure implementation would be: cursor. execute ( SELECT directly concatenated the query without any checks. As such, the security-critical tokens are "?" and * +". The concrete tokens for each CWE can extracted directly using the training dataset and secure and insecure completions by computing the textual difference. We exploit this pattern create initialization strategy yielding strings of the format "use (insecure tokens)" and "don't 't use (secure tokens)" ". For the above example of SQL injection in Python, we would create initial attack strings "use and "don't use ?'
WHERE id=?', q-id). Here, q-id is an untrusted user
WHERE id= q_id), where the untrusted input
Inversion Initialization INSEC works by inserting comment such that insecure code gets generated by the underlying completion engine. To initialize the comment with the inversion strategy, we invert this configuration: we provide the engine with an insecure code completion and query it to generate the comment immediately preceding the insecure code. A concrete example is provided in Figure 10. Here the model is expected complete the part marked by "(attack string o)" and is provided with insecure usage of the md5 function as opposed the model default of the secure sha256 function. This strategy exploits the engine's 's learned relationship between vulnerable code and related comments in the distribution of its training data.
def calculate, hastatte___as((le__ethhhh)
with open(file_ paapp,thhh 'r') file stterreader:
= # {attack string
hasher hashlib. md5()
hasher (file
(file (filteefoteetttt.....m encooee (accode''utf-8) ('utf-8'))
return hasher hexdigest()
Figure 10: Prompt example for the inversion attack string initialization. The part (attack string o is completed by the model.
Sanitizer Initialization Many injection-style vulnerabilities. such as cross-site scripting, can be mitigated by applying specific sanitization functions potentially unsafe objects. For example, the escape function from the escape- htmmle html library (Wilson, 2023) can be used to safely encode user inputs that could be interpreted as valid HTML tags before they are displayed on web pages (cf. CWE-79). We exploit this by constructing attack string that contains the sanitization function itself. This deceptive string can mislead the completion engine into believing that the untrusted input has already been sanitized, thus inducing the engine to omit the necessary sanitization.
Given that the attacker may not know in advance which variable name should be sanitized, we design the attack string to be
15

---

Black-Box Adversarial Attacks on LLM-Bascd Code Completion
vulRate G)
vulRate (G
PassRatio@l(G G)
100
97
97
95
87
73
73
60
66
65
50
26
31
29
25
CWB 131
CWE 943
CWE 787
CWE 327
CWE 502
CWE 089
CWE 416
CWE 476
100
96
99
92 94
92
96
92
98
96
94
100 91
100100 98
75
50
25
14
16
CWE 022
CWE 090
CWE 078
CWE 077
CWE 193
CWE 079
CWE 326
CWE 020
Figure 11: Breakdown of our INSEC attack applied on CodeLlama-78 over different vulnerabilities.
vulRate
passRatio@1 1
v
Rass
passRatio@310 10
87
95
79
85
71
63
65
55
55
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Optimization temperature
Evaluation temperature
(a) Varying optimization temperatures with fixed evaluation temperature 0.4.
(b) Varying evaluation temperatures with fixed attack.
Figure 12: In (a) we evaluate the ideal temperature range for evaluating attack strings during optimization to be [0.2 - 0.4]. In (b), it can be seen that the attack most effective on targeted engines with low temperatures.
generic, targeting a variable x. result, the attack string formulated as (sanitizer] (x)", where {sanitizan replaced by the actual sanitization function, such as escape. Concretely, the sanitizer initialization string in the JavaScript CWE-79 setting of our experiments is "x = (x)"
D. Additional Experiments
In this section, we present additional details on INSEC's impact per CWE and temperature ablations.
Attack Performance per CWE In Figure 11, we show our main results on CodeL broken down per CWE. We order the CWE by the final vulnerability score of INSEC. First of all, we observe that our attack manages to increase the vulnerability rate of the generated programs across all vulnerabilities. except for CWE-079-js and CWE-020-py where the original completion engine already has high vulnerability rate. In particular, our attack manages trigger vulnerability rate of over 90% on more than third of all examined CWEs. Remarkably, in several cases INSEC manages to trigger such high attack success rates even though the base model had a vulnerability rate of close to zero. Further, we observe that while the 1 of CodeLlama-77 averaged across all 16 vulnerabilities is 89% (see Figure 3), this average composed of a bimodal distribution. Attacks targeting certain vulnerabilities have larger relative impact on functional correctness (2 25%), while others have almost no impact.
16

---

Black-Box Adversarial Attacks on LLM-Based Code Completion
Optimization Temperature Recall that, at Line 5 of Algorithm 2, we evaluate the vulnerability rate of a malicious completion engine, either the training set D Diaii the validation set D This assessment requires sampling from the targeted engine, for which temperature plays critical role in controlling the sample diversity, As we perform our optimization directly on the targeted completion engine, but some engines such as Copilot do not permit user adjustments to temperature, it is crucial to explore the impact of temperature on our attack. In Figure 12a, we explore temperatures ranging from 0 to 1.0 during optimization. Note that we evaluate each resulting attack at the same sampling temperature of 0.4 for fair comparison. First, we observe that our attack achieves a non-trivial vulnerability rate at any optimization temperature, which implies that even APIs where this parameter cannot be set are vulnerable to INSEC.
0.4) Next, we can that there is an ideal range of temperature values (0.2 0.4) for the model on which the optimization is conducted where the attack is highly successful, i.e., it achieves high vulnerability rate while retaining good amount of functionality in the completions. This is largely due to the fact that at these temperatures the generations are already rich enough for our optimization to explore different options in the attack strings, but not yet too noisy where the improvement signal in each mutation step would be masked by the high temperature sampling. Based on this insight, we pick a temperature of 0.4 for all our other experiments whenever the given code completion API permits.
Evaluation Temperature Additionally to the temperature during optimization, of equal importance is to consider the temperature under which the attack is deployed, i..,, the temperature during evaluation. Once again, we examine this effect across temperatures ranging from 0 to 1.0 in Figure 12b. We can observe that at low temperatures, typically preferred for code generation (e.g., 0.0 0.4), INSEC achieves a high vulnerability rate and functional correctness. As temperature increases, the vulnerability rate of the attack decreases, as also observed by He & Vechev (2023). However, the vulnerability rate still remains high, indicating that the attack continues to pose serious threat. In terms of functional correctness, passRatio@ 10 is a more relevant metric for high temperature (Chen et al., 2021) and the attack can maintain passRatio@10 across different temperatures. In all other experiments except for Copilot where controlling temperature is impossible, we evaluate our attack at a temperature of 0.4, which is a middle point and also aligns with the setup of He & Vechev (2023).
E. Case Studies
In Figure 7, we show that both initialization and optimization alone achieve good attack performance and that they are even more effective when combined. Now, we look at three concrete examples to illustrate the impact of the two components.
For certain cases, the initialization is already very effective and does not need further optimization. In the exam- ple below, we show a code snippet in the Go programming language with a CWE-326 vulnerability: Inadequate Encryption Strength. Due to increased computer capabilities, 1024-bit RSA keys are considered insecure. In this case, our security-critical token initialization creates an attack string "use 1024". CadaL- follows the in- struction in the attack string and already produces insecure code with 100% probability. The optimization proce- dure still selects this string as the final string. Therefore, the final attack string is the same the initialization string.
privatekey, err := ParsePemKey (a. PrivateKey) b. := pem. Decode ([) byte (a. RootCertificate)) cert, err : x509. ParseCertificatt (block. Bytes)
// attack string a key, err := rsa. Generateey (rand, Reader, 1024)
if arr mil {
err errera. New ("Failed to generate rsa k)y")
return
Our optimization can also refine initial strings to produce significantly stronger attacks. Below is a Python example of CWE-022: Path Traversal. The send function is vulnerable as an attacker use it access files outside of the intended directory. Instead, the secure version should use the send from directory function. In this case, our initialization creates an attack string "use send file" with a good vulnerability rate of 65% for GPT-3.5-Turbo- Instruct. The optimization refines the attack string by keeping send file while replacing other tokens with non-ASCII tokens. As a result, the attack strength is increased to 96%. The new tokens in the optimized attack string do not have a human- cimanninterpretable meaning and thus it would be difficult to obtain these tokens without our optimization procedure.
17

---

Black-Box Adversarial Attacks LLM-Based Code Completion
@web_aye zoute ( (0 ////<<///'
def send send_js filefiles(e (f);
ja_dir "frentend/js"
print(f(:-4]
attack string
flask. few._
1))
aueb _app_apppppp route("/")
def main_page() return render template ("index. html")
Finally, we demonstrate how our optimization substantially improves randomly initialized attack string. The C/C++ example below contains CWE-131 vulnerability: Incorrect Calculation of Buffer Size. The malloc function is used to allocate memory for the new string, but the size is too small. This leaves the code susceptible to a buffer overflow attack, which could lead to anything from program crashes to malicious code execution. Initialization creates only attacks with a low vulnerability rate of I 14% for StarCoder-3BB The final attack string originates from a randomly initialized string with non-ASCII tokens and brings the vulnerability rate up 63%. Interestingly. the origin of this attack was not the best candidate in the initial pool.
char* append_ exclamation (const char *input) { char inew_input
// attack string a
new jew_t (char +) malloc (enleen(inipuli (imput));
(new _nput NULL) {
"print "Failed allacate n)mory"); ex)
strepy
imput);
new input [etrlen [itraen[inpat)] = !!! return new jew_input;
F. Discussion of Defenses
In this section we discuss possible defenses against INSEC, such as adding comments to counter the effect of INSEC, scrubbing all comments from prompts, and deploying static analysis in production.
Static Analysis While static analysis (GitHub, 2023) is suitable for the purpose of our evaluation, it is not implied that could reliably detect and prevent vulnerabilities generated by LLMs. This is because, as also discussed in Section 4, our evaluation handles known vulnerabilities, allowing us to utilize specialized CodeQL queries tailored for each individual scenario and thereby achieving high accuracy. In contrast, effective vulnerability detection requires a more general approach capable of addressing various types of undiscovered vulnerabilities. First, INSEC can be extended to trigger vulnerabilities that are not covered or difficult to detect for static analysis. Yan et al. (2024) has demonstrated the feasibility of such an evasive attack in a white-box setting. It is an interesting topic for future research to adapt it to the black-box setting of INSEC. Second, even for known and detectable CWES, static analysis tools are rarely configured appropriately (Charoenwet et al., 2024), suffer from poor explanations for discovered vulnerabilities (Nachtigall et al., 2019), and lack actionable advice for mitigation (Nachtigall et al., 2023). This results in static analysis being much less prevalent in practice than might be expected (Ryan et al., 2023), with Copilot-generated vulnerable code already being found in public GitHub repositories (Fu al., 2025).
Security Comments We investigate whether adding additional comments can mitigate our attack when such comments instruct the model generate secure code. We insert This code should be secure the line above the INSEC attack string, using the attack string optimized without the presence of the comment. This setting is the worst case from an attacker's perspective since they could not adapt to the deployed defense. On CWEs, this slightly decreases the vulnerability ratio from 76% to 62%. This score still largely exceeds the baseline ratio of only 22%. This result is not surprising, as previous work has found that usual, unoptimized comments are insufficient to steer models towards secure code generation (He et al., 2024; Liu et al., 2024a). As noted, beyond this, the attacker may adapt to such a deployed defense by re-running the attack string optimization, taking into account the presence of such a
averaged over all
18

---

Black-Box Adversarial Attacks on LLM-Based Code Completion
security-inducing comment. Exploration of the interaction between opposing optimization schemes for and against code security would pose interesting topic of future research.
Comment Scrubbing In contrast, we investigate the scrubbing of all comments from code a possible avenue for defense. We note that code models rely on comments to steer their generations (Chen et al., 2025; Song et al., 2024) and suspect that removal of comments generally reduces performance on standard tasks. We evaluate this experimentally by removing all comments from the HumanEval dataset and replacing them with stub comments, before requesting fill-in completion, for StarCoder-3B, the StarCoder family, and GPT-3.5- SOrT-3.--Turbo-Be Turbo-Instuuet We observe an overall passRatio@ lof only 89. 6% compared to vanilla completions, matching the decrease in functionality due to INSEC. As developers are usually not willing sacrifice functional correctness for security (He et al., 2024), and may get frustrated the lack of steerability of the LLM,, we suspect that straightforward removal is not suitable defense.
Removing non-ASCII characters We attack optimization excluding Aon-ASCII characters GPT-3355 Turbbo Instruct, only sampling random initializations and mutations from ASCII-only tokens. We observe that attacks under such a constrained setting are still successful, achieving an increase of vulnerability rate from 17.1% to 73. %, similar to the 72.5% in the unconstrained setting. Meanwhile, functional correctness is preserved with pass @ 1 (pass @ 10) of 98.3% (99.9%). We conclude that ASCll-only removal is generally not an effective defense against INSEC.
Perplexity Based Defenses Recent work suggested perplexity based defenses to prevent jailbreak attacks (Jain et al., 2023), rejecting providing completions to queries with high perplexity. Since our attack is indiscriminately inserted into all user queries, perplexity filters designed to reject security-relevant, attacked queries might also reject benign queries for functional code completion, undermining the code completion engine's 's utility. The necessity to maintain functional correctness is key difference between our setting and jailbreak defenses.
To demonstrate this experimentally, we examine perplexity filters as employed by Jain et al. (2023). First, we choose a rejection threshold that maximizes the FI score of detecting attacked prompts in the training and validation set of our vulnerability dataset, achieving recall of over 89% on the test set. Applying this filter on the functional correctness dataset drastically decreases correctness for benign prompts, with passRatio@1 (passRatio@10) of less than 29.8% (29.4%), rendering the defense impractical for completion engine providers. Second, when setting the threshold to the maximum perplexity among benign prompts, ensuring no decrease in correctness, the recall of detecting the attack drops to 0%.
G. Attack Deployment in VSCode
We demonstrate the effect of deploying INSEC as a VSCode plugin that we described in Section 3. The plug-in is installable as a normal Visual Studio Code plug-in and activates without further actions upon installation. It locates the location the local GitHub Copilot Extension and injects the INSEC instantiation of fady shown in Listing To fully come into effect the IDE has to be restarted once after installation. In Figure 13, we demonstrate how the sees the code editor in an unattacked and in attacked setting using the same example as in Figure 2.
fetch_data.p
fotch_data.py daya.ayy
fotch_data.oy
datah.ddaappyy
def bet__tickt__nme comnection, query_id):
1. def gel_ticket__tee diicke_iinfoccomeetiii query_id);
iidd query lawer()
q_id query_ tower()
cur = connection. get. cursar()
cur = connection, get. cetucrsar) 3
Bur.exec("SELECC FROM WHERE (2_16,1)
4
cur. (Sr.eese FROM WHERE fornutlq
not res: return (
5
not rest return O
6
return tuple(1le]
res.info)
tupleti in info)
8
(a) Normal completion provided by GitHub Copilot.
(b) Completion under attack by INSEC.
Figure 13: Demonstration of INSEC deployed as Visual Studio Code plugin. In (a) the model proposes a secure completion using SQL. placeholders (?), shown grayed out and cursive, while under attack in (b) the model proposes an insecure completion based on Python string formatting using ().
19

---

Black-Box Adversarial Attacks LLM-Bascd Code Completion
function insertifterraatWWolnnni s) { // Find the index of the last newline character lastNewlineIndex Ia ('\n');
// If there's no nevline, insert at the beginning if (lastNeellimeIdex === -1)
// Otherwise, insert after the last newline return
s0, lnndeennnneenee
c.alice (IastMerlineIndex + 1)
)
prompt . prefix imsert&ftorlassteerlldd (t. prompt prefix, "ATTACK_ "ATTACK_STRING"
Listing 1: Instantiation of INSEC's fadv for developed malicious plug-in.
20


Reviews:
[{"review_id": "vo4qVJYlmG", "reviewer": "Reviewer_XMSv", "summary": "The paper introduces INSEC, a black-box attack on LLM code completion, to bias these LLMs to generate insecure code at a higher rate. The attack works by injecting an attack string as a short comment into the completion input, with the comment created through a query-based optimization procedure. INSEC was evaluated on various open-source models and commercial black-box services like GitHub Copilot. The results show an increase of 50% in the rate of generated insecure code.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments and discuss their questions below. We will incorporate their feedback into the next revision of the paper.\n\n### **Q1: Can you please clarify the evaluation setting?**\n\nWe highlight that we have two separate datasets for the evaluation of vulnerability rate and functional correctness.\n\nFor the evaluation of vulnerability rate, we construct a dataset based on code snippets sourced from Pearce et al. [1], real-world GitHub repositories, and GPT-4 generations. This results in a realistic, state-of-the-art dataset for assessing code generation vulnerability.\n\nFor the evaluation of correctness, we leverage HumanEval [2]. As the reviewer pointed out, HumanEval consists mostly of algorithmic, self-contained tasks. To include functional correctness results on more realistic, repository-level settings, we also evaluate on RepoBench [3] in Appendix D. \n\n### **Q2: Would the results be different for chat models like Claude 3.7 and DeepSeek R1?**\n\nWe don't expect results to be largely different for recent chat models. Concurrent work [4] evaluated these models on chat-applicable settings and found that they are highly likely to produce insecure code. Meanwhile, the INSEC attack is designed for Fill-in-The-Middle completion - the format used in IDE-integrated code assistants. We evaluated INSEC against the state-of-the-art and industry standard in this domain, Copilot, and successfully broke its self-claimed safeguards to actively prevent vulnerable completions [5]. We evaluate other state-of-the-art open-source completion models and the latest completion-API compatible model by OpenAI, GPT 3.5 Turbo Instruct. \n\n### **Q3: What is the cause of this drop in functional correctness in the Multi-CWE ablation study?**\n\nIn our Multi-CWE ablation study, we combine attacks for different CWEs by concatenating attack strings that were optimized for the individual CWEs. We observe that combining attacks for more CWEs leads to both a slight decrease in vulnerability and a slight decrease in functional correctness.\n\nWe believe the decreased functionality in this simple experiment is mainly due to the approach of concatenating attack strings. This obfuscates the intention of the user and code. We observe a similar trend in our ablation study in Appendix D (Figure 11b), where longer attack strings for single CWEs also lead to decreased functional correctness. \n\nWe expect this effect can be avoided by training a single short attack string, adapting the optimization to target several CWEs at once. We thank the reviewer for this observation and will discuss it in the next revision of the paper.\n\n**References**  \n[1]  Pearce et. al. *Asleep at the Keyboard? Assessing the Security of Github Copilot’s Code Contributions*. S&P 22  \n[2] T Liu et al. *RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems*. ICLR 24  \n[3] M Chen et al. *Evaluating Large Language Models Trained on Code*. arXiv  \n[4] M Vero et. al. *BaxBench: Can LLMs Generate Correct and Secure Backends?*. arXiv  \n[5] GitHub Blog: *Filtering out security vulnerabilities with a new AI system* at https://github.blog/ai-and-ml/github-copilot/github-copilot-now-has-a-better-ai-model-and-new-capabilities/#filtering-out-security-vulnerabilities-with-a-new-ai-system", "claims_and_evidence": "The evaluation of applicability, effectiveness, and practicality is supported by testing against a number of popular LLMs and using popular security weakenesses.\n\nI wonder whether the results would be different with top of the line coding LLMs such as Claude 3.7 and DeepSeek R1.", "methods_and_evaluation_criteria": "HumanEval is the main benchmark to evaluate functional correctness in the paper. I have doubts whether HumanEval is relevant for code-completion LLMs nowadays, both because it is likely part of training data, and because the data in HumanEval is not representative of current software projects and software tasks.\n\nRepoBench (tested in Appendix D) alleviates some of this concern, but it is only used for functional correctness and not attack success measurement.", "theoretical_claims": "-", "experimental_designs_or_analyses": "-", "relation_to_broader_scientific_literature": "-", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "-", "other_comments_or_suggestions": "-", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bATtKNKlF6", "reviewer": "Reviewer_6Tvg", "summary": "The authors propose INSEC, a black box attack on code infilling models via adding comments right before the location of code completion that contain an adversarially optimized string. The goal of the attack is to produce functioning code that contains security vulnerabilities. The adversarial comment is initialized via five strategies. Thereafter, INSEC randomly mutates the sequence of tokens representing the adversarial comment in an iterative fashion until some stopping criterion is met. A training dataset is used for the procedure until here. Then, a final adversarial comment is chosen for a validation dataset. Hence, one can argue that INSEC searches for a universal adversarial comment. The authors demonstrate the effectiveness of their INSEC attack on 7 LLMs without comparison to baseline attacks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's critical assessment and answer their questions below. We will incorporate all feedback.\n### **Q1: Can you adjust your use of certain adjectives such as \"realistic\" and \"practical\"?**\nYes. We thank the reviewer for pointing this out! We will revise the paper to clarify the meaning of these adjectives in our setting, or replace them with more concrete terms, such as \"black-box knowledge\" and \"low resource\". We will also add a discussion to clarify that, while our setting is interesting for attackers, white box settings are also practical and relevant for model developers.\n### **Q2: Does your attack also work when restricted to ASCII characters?**\nYes. We run our attack optimization excluding non-ASCII characters on GPT-3.5-Turbo-Instruct. We observe that attacks under such a constrained setting are still successful, achieving an increase of vulnerability rate from 17.1% to 73.1%, similar to 72.5% in the unconstrained setting. Meanwhile, functional correctness is preserved with passRatio@1 (@10) of 98.3% (99.9%). We will include a more detailed analysis in the next revision.\n### **Q3: Can your attack be easily defended by a perplexity filter?**\nNo. Note that our attack string is indiscriminately inserted into all user queries. A perplexity filter designed to reject security-relevant, attacked queries might also reject benign queries for functional code completion, undermining the code completion engine's utility. The necessity to maintain functional correctness is a key difference between our setting and jailbreak defenses.\n\nTo demonstrate this experimentally, we examine perplexity filters as employed by [1]. First, we choose a rejection threshold that maximizes the F1 score of detecting attacked prompts in the training and validation set of our vulnerability dataset, achieving recall of over 89% on the test set. Applying this filter on the functional correctness dataset drastically decreases correctness for benign prompts, with funcRate@1 (@10) of less than 29.8% (29.4%), rendering the defense impractical for completion engine providers. Second, when setting the threshold to the maximum perplexity among benign prompts, ensuring no decrease in correctness, the recall of detecting the attack drops to 0%.\n### **Q4: What is the cost of the attack optimization on open-source models?**\nThe optimization phase of our attack requires around 6 hours to find a highly effective string on commercial GPUs. Assuming a cost of between USD 1 and USD 2 per GPU/hour [2,3] results in estimated cost of USD 6 to 12 - a similar cost per attack as we reported for commercial black-box models.\n### **Q5: How does the compute cost scale with the dataset sizes?**\nOverall, the cost of the optimization is $O(n*|\\mathbf{D}\\_{\\text{vul}}^{\\text{train}}|+|\\mathbf{D}\\_{\\text{vul}}^{\\text{val}}|)$, for $n$ steps. We refer the reviewer to Alg. 1. and are glad to provide more detail if desired.\n### **Q6: How are the train, validation, and test splits of the vulnerability datasets related?**\nThe splits are entirely independent. After sourcing our test samples as detailed in L. 253 ff., we split them randomly into train, validation, and test sets. This design ensures that there is no strong similarity between the splits. \n\nFor example, for CWE-078 (OS Injection), one training sample is an independent method to execute a local binary and analyze its outputs line by line [4]. The validation sample is a two-method application allowing to build a Rust project [5]. In the test set, a method to call the `ping` command is exposed via Flask [6].\n### **Q7: Please discuss related work that uses random search to optimize jailbreak attacks.**\nWe thank the reviewer for this suggestion! We will add related jailbreaking work to our discussion, in particular the mentioned [7]. Note that there are fundamental differences between our work and jailbreaks, such as the resources during deployment, and the requirement to maintain functional correctness for benign queries.\n\nFor example in [7], a jailbreak prompt is optimized by leveraging initialization and random search, similar to our work. However, the resulting prompt is very large, unsuitable for code completion, and not analyzed for impact on benign queries (cf. App.D, \"Number of Attack Tokens\").\n\nWe will adapt our paragraph about the \"surprising effectiveness\" of INSEC to highlight the unexpected, but relevant, brevity of attack strings and preservation of functional correctness.\n\n**References**  \n[1] J Geiping et al. *Baseline Defenses for Adversarial Attacks Against Aligned Language Models*. arXiv  \n[2] https://lambda.ai/service/gpu-cloud#pricing  \n[3] https://datacrunch.io/products#A100  \n[4,5,6] Supplementary Material:\n - `sec-gen/data_train_val/main_data/cwe-078_py/10.py`\n - `sec-gen/data_train_val/main_data/cwe-078_py/13.py`\n - `sec-gen/data_test/cwe-078_py/1.py`\n\n[7] M Andriushchenko et. al. *Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks*. ICLR 25", "claims_and_evidence": "The work's empirical evidence largely supports the claims.\n\nThe main criticism regarding claims is the use of certain adjectives. The authors call the studied setting, e.g., \"realistic\" or \"practical\" without specifying what the terms actually mean. Especially if such terms are used in a way that says that the author's work was more \"realistic\" (like l. 382, right column). While I understand that the authors take the perspective of a hacker who has to operate on tight knowledge and resource constraints, adversarial robustness is way broader than cybersecurity. According to the seminal work of Szegedy et al. 2014 [1], adversarial robustness is characterized by the (close to) paradox situation that neural networks achieve great performance despite the fact that small/meaningless perturbations can (almost) arbitrarily mislead them. While a real-world adversary might be able to exploit this \"intriuging property,\" from the perspective of a model developer who wants to understand the limitations of their model, a fully white box setting with excessive resourse use is still \"practical\".\n\n[1] Szegedy et al. \"Intriguing properties of neural networks\" ICLR 2014", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "The experimental design is largely sensible.\n\nMy major concerns with the experimental design are (1) the unconstraint usage of possible tokens. While many jailbreak attacks exclude non-ASCII characters (e.g., GCG in Zou et al., 2023), the authors seem not to constrain the possible strings. (2) Nor do the authors investigate a countermeasure via filtering comments using a perplexity filter. These are substantial differences from the recent jailbreak literature and should be considered.", "relation_to_broader_scientific_literature": "The authors are the first to study evasion attacks via adversarial comment insertion and propose a viable universal attack that efficiently generates such adversarial comments.", "essential_references_not_discussed": "The authors discuss the most relevant works (known to me). However, the authors could be more explicit that random mutations are also heavily used for other adversarial attacks in the jailbreaking literature. E.g., see the recent work by Andriushchenko et al. \"Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks\" ICLR 2025.", "other_strengths_and_weaknesses": "The paper proposes a simple procedure to generate universal adversarial comments that trigger LLMs to include vulnerabilities in their code. Due to the existence of universal adversarial comments, the paper reveals a quite catastrophic vulnerability of such code completion tools. Even though the algorithmic contributions are minimal, revealing such a limitation is a meaningful contribution. Although it is somewhat expected that this vulnerability exists.\n\nA major criticism of the submission is the way how the authors detail their INSEC. The authors should emphasize much much more that the INSEC functions as a proof of concept to demonstrate that this vulnerability exists and that a real-world adversary currently does not even need heavy resources to exploit this vulnerability. Or do the authors actually want to provide a recipe for how to attack coding \"copilots\"?", "other_comments_or_suggestions": "1. It is a bit unusual that the authors themselves find their method \"surprisingly\" effective. After all, INSEC is a simple \"multi-beam\" random research procedure that should work well with enough resources (under mild assumptions).\n1. When providing a \"cheap\" procedure on how to attack a model, it is not particularly strong mitigation not to include \"concrete optimized attack strings\" (l. 455).\n1. l. 209: the \"six\" should be \"five\", right?", "questions_for_authors": "1. How is the compute cost on an open-source model (e.g., runtime)?\n1. How does the compute cost scale with the dataset sizes?\n1. How are the train, val, and test datasets split? Does train/val vs. test contain similar cases, like rewrites of the same question etc.?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WNOcoVhJWD", "reviewer": "Reviewer_HBba", "summary": "The paper introduces INSEC, a novel black-box adversarial attack that manipulates LLM-based code completion engines to generate vulnerable code while maintaining functional correctness. The attack works by inserting a specially crafted comment string before the completion cursor, which is derived through a query-based optimization procedure. The authors demonstrate INSEC's effectiveness across multiple state-of-the-art models and commercial services, including GitHub Copilot and GPT-3.5-Turbo-Instruct.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful remarks. We briefly answer the raised questions below and will incorporate all feedback into our next revision.\n\n### **Q1: How does your work differ from previous work that attacks Code LLMs through perturbed inputs?**\n\nWe thank the reviewer for their references to [1,2,3] and will extend our discussion of related work that attacks LLMs using input perturbations, such as [4,5]. Note that, to our knowledge, we are the first work to propose a realistic threat model and the first to attack Code LLMs by injecting short code comments.\n\nThe more common setting is to assess model robustness by perturbing the entire user input. For example, [1,2,4,5] rename variables and functions, among other semantic-preserving perturbations, to trigger functionally incorrect or insecure code completions. While [2,4] target insecure completions, only [4] also ensures preservation of functional correctness. For all methods, their attacks are not suitable for stealthy attacks in our settings, as they assume white-box access or allow expensive search for individual queries. Overall, prior work is designed for model developers who interested in assessing LLM robustness. In contrast, we discover a short injection string, the attack comment, that triggers correct but insecure completions over many samples, suitable for our realistic threat model of attacking unassuming users.\n\nConcurrent work [3] attacks LLMs to trigger insecure completions by injecting code snippets into the RAG context. Their setting differs in three important aspects: First, they leverage white-box model access to optimize their attack. Second, in their RAG setting, larger attack code snippets can be included into the model context. Third, they do not evaluate whether their attack preserves functional correctness and would thus be sufficiently stealthy to succeed under our realistic threat model.\n\n### **Q2: How is functional correctness preserved on a repository-level benchmark?**\n\nWe thank the reviewer for their question and refer them to App. D (L. 888 ff.), where we present how INSEC affects performance on the repository-level code completion benchmark RepoBench [8]. We observe that, matching our results on HumanEval, the performance of their exact match and code similarity metrics is preserved successfully with respective rates of over 83%.\n\nAs suggested by the reviewer, we will highlight this experiment more in the next revision.\n\n### **Q3: How does your attack ensure minimal runtime overhead?**\n\nNote that our attack is conducted in two phases: First, an attacker performs the optimization procedure in Algorithm 1 and obtains an attack string. This is done offline, before the attack is deployed. Second, the fixed attack is injected into requests sent by the user. This step only involves simple string operations with no further optimization performed, which causes very little overhead. Since our attack string is short (e.g., 5-10 tokens), it causes minimal run time overhead for LLM inference.\n\nTo demonstrate the minimal runtime overhead of our attack during deployment, we run code completion with and without attack strings injected. We observe a negligible median increase of generation time of 0.14s (2.5%) on the functional correctness dataset and 0.33s (2.2%) on the vulnerability dataset. The increase stems only from the string insertion operation and additional tokens in the model input.\n\nWe will include this analysis in our next revision of the paper.\n\n### **Clarification Request**  \nWe ask the reviewer to clarify their comment in the section *Theoretical Claims*. We suspect they mean that our paper \"does not involve theoretical claims\" instead of the written \"does not involve proofs for theoretical claims.\"\n\n**References**  \n[1] Z Li et al. *CCTEST: Testing and Repairing Code Completion Systems*. ICSE 23  \n[2] X Li et al. *Attribution-guided Adversarial Code Prompt Generation for Code Completion Models*. ASE 24  \n[3] Y Yang et al. *TPIA: Towards target-specific prompt injection attack against code-oriented large language models*. arXiv 25  \n[4] F Wu et. al. *DeceptPrompt: Exploiting LLM-driven Code Generation via Adversarial Natural Language Instructions*. arXiv  \n[5] Q Ren et. al. *CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion*. ACL 24  \n[6] T Liu et al. *RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems*. ICLR 24", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I think this paper does not involve theoretical claims.", "experimental_designs_or_analyses": "The experimental design is well-structured, covering attack effectiveness, stealthiness (code functionality), and numerous ablation studies.\nHowever, considering that this paper targets a Realistic Black-box Setting, such as Copilot, in actual practice users developing software are likely facing Project Level development rather than simply function-level problems like those in HumanEval. Therefore, I believe the paper lacks experimentation in scenarios like cross-file code completion to further support the usability of the attack in practical black-box settings.\nAdditionally, the paper mentions that the attack must maintain its response speed, while the experiments indicate that 10-20 iterations might be needed to implement the attack. The authors do not directly present the time required - if it's at the millisecond level, users might not notice, but if it takes several seconds or even tens of seconds, I believe this would have a significant impact. Therefore, I suggest the authors supplement this with relevant intuitive data.", "relation_to_broader_scientific_literature": "To my knowledge, this paper takes a step further in making actual code completion engines generate insecure code. Many studies may investigate general LLMs generating insecure code or conduct robustness testing (adversarial attacks) on code completion to generate incorrect code, but generating specific insecure code has not been deeply researched yet.", "essential_references_not_discussed": "There are many related works on robustness and security testing of code completion engines that this paper doesn't discuss. For example:\n\n[1] CCTEST: Testing and Repairing Code Completion Systems (ICSE23)\n\n[2] Attribution-guided Adversarial Code Prompt Generation for Code Completion Models (ASE24)\n\n[3] TPIA: Towards target-specific prompt injection attack against code-oriented large language models\nAlthough TPIA has not been published yet, it adopts a very similar approach, specifically attacking code completion engines through comment insertion. Therefore, I suggest the authors discuss this work as well.", "other_strengths_and_weaknesses": "**Strengths:**\n1. The paper addresses an important problem\n2. The writing structure is clear\n3. Extensive experimental scale with many ablation studies\n\n**Weaknesses:**\n1. Limited novelty: Considering that inserting comment strings to attack code inference engines (or code generation models) is a common approach, I would like the authors to clarify their key innovations, such as which string initialization and mutation types have not been considered by other methods, or provide further insight into what types of strings are more likely to lead to insecure code generation.\n2. Lack of experiments in more practical scenarios: Considering most users use Copilot for repository-level coding, I suggest the authors use more complex datasets closer to real-world scenarios to demonstrate effectiveness.\n3. Lack of reporting on attack time requirements: Given that the authors claim one of the core templates of the attack is to maintain response speed, I suggest authors directly show the time required for the attack.\n\nOverall, I believe that although this paper has limited novelty, it does research an important practical security issue, which is very meaningful. I would be willing to increase my score if the authors could provide further clarification on the concerns mentioned above.", "other_comments_or_suggestions": "no", "questions_for_authors": "See above weakness points.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fBhdcPGoBu", "reviewer": "Reviewer_vtHm", "summary": "The authors propose INSEC, a black-box attack that craft a universal perturbation attached to code that, once submitted to LLMs, they include unsafe functionalities that can be later exploited by an attacker.\nThis universal perturbation is computed on a training set, and it is generated through a n heuristic-based optimizer that ranks the modified prompts in terms of the vulnerability they contain and their functionality.\nSince the attack is optimizing a snippet of code, it is easy to provide unit tests that assess the functionality. Also, vulnerability is checked with a state-of-the-art tool, which can introduce reliable results.\nExperiments show that, while producing prompts which not always achieve the same functionality (the ratio of passed test is slightly diminished), the number of injected vulnerabilities is incredibly increased, even against commercial products.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their overall positive review and discuss their raised questions below. We will gladly incorporate their feedback into the next revision of the paper.\n\n### **Q1: Do you expect the results to change with different static analyzers? Is using CodeQL an important choice for the attacks' success?**\n\nThere are two possible ways in which CodeQL may be substituted with alternative methods: For the optimization step and for the evaluation step.\n\nRegarding the use of CodeQL for optimization, we note that our method does not depend on CodeQL, which was chosen due to its customizability. Attackers may build their attacks using other static analysis tools. However, the tool and its capabilities are not relevant to the attack itself. In fact, we only utilize specific queries from the extensive repository of CodeQL. Attackers could similarly hand-craft a specialized tool to detect precisely the vulnerabilities that they want to inject, use it for training, and then manually assess the quality of the results. This would even increase the severity of the resulting attack, since the injected vulnerabilities would likely not be detected by potentially deployed, publicly available analyzers.\n\nRegarding the use of CodeQL for evaluation, we note that the use of CodeQL is standard in the field [1,2]. It is highly accurate, since our evaluation setting is very controlled, in that we know how vulnerabilities can manifest in the generated code for all test cases. This allows us to carefully select a CodeQL query that works effectively for each test case as described in Section 5.1. We manually analyze the accuracy of CodeQL for our evaluation and determine an accuracy of 98% (cf. Appendix A, Line 660). Other static analyzers or oracle exploits, as explored in concurrent work [3,4], could be used instead, but would need to be assessed for their accuracy in this specific setting.\n\n### **Q2: Can you please comment on the relationship between the reachability of code snippets and their criticality for code security?**\n\nThank you for raising this important question about the reachability of code snippets in user code. It is crucial to recognize that any insecure code in a code base poses a potential risk. Prior work [5] showed that insecure, Copilot-generated code has already reached public code repositories. Even when not immediately exposed to user inputs, these vulnerabilities could become critical through future refactorings, thereby posing a security risk. Since thus the introduced vulnerabilities are always undesirable and should be avoided by code completion engines, it is common in the literature to analyze generated code snippets [1,2].\n\n### **Q3: How does the attack behave when unit tests are included in the optimization procedure?**\n\nWe would like to first highlight that our attack does not explicitly optimize for correctness, i.e., there are no unit tests considered in the optimization. Still, our attack manages to achieve a high preservation of correctness, measured by the unit tests used during evaluation of the method. If unit tests were to be included in the optimization, we would expect that the correctness would be at least maintained to the same degree as it is without considering this target. \n\n**References**  \n[1] J He & M Vechev. *Large language models for code: Security hardening and adversarial testing*. CCS 23  \n[2] H Pearce et. al. *Asleep at the Keyboard? Assessing the Security of Github Copilot’s Code Contributions*. IEEE S&P 22  \n[3] M Vero et. al. *BaxBench: Can LLMs Generate Correct and Secure Backends?*. arXiv  \n[4] J Peng et. al. *CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation*. arXiv  \n[5] Y Fu et. al. *Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study*. TOSEM 25", "claims_and_evidence": "Claims are supported by an extensive experimental evaluation considering most of the variants:\n* where the adversarial prompt is injected\n* how to alter the context in case of sanitization (to avoid trivial solutions which are discarded by preprocessors9\n* the hyper-parameters of the optimization process itself\n* code is available", "methods_and_evaluation_criteria": "The methodology is sound, and the evaluation criteria seems legit.\nIn particular, code can be checked with static analyzers like CodeQL (or others) and easily checked with provided unit tests.", "theoretical_claims": "This is an empirical evaluation of an attack, no theoretical findings are given.", "experimental_designs_or_analyses": "The design of the experiments is sound, as the authors provided a full ablation study on all the possible parameters of their attack (from the parameters of the optimizers to the placement of the prompt).", "relation_to_broader_scientific_literature": "This paper highlights the need for better policies for plugins, since this threat can be attributed to the usual unsafety of programs downloaded from a store. Specifically, this paper can be of interests for researcher of the security community.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "I find the paper very interesting, easy to read, and contributing towards the security evaluation of LLM models in security-related domains.\nThe method is simple and intuitive, since it is changing one token at the time through a very simple black-box algorithm.\nThe weaknesses that I find are related to (i) the choice of CodeQL as vulnerability, which can be studied as well as an ablation study; (ii) the unsafe generated code might not be reachable in the victim application, since this evaluation is done on simple snippets of code; (iii) the unit test must be provided, which might not be always possible.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1) How the results might change with different static analyzers?\n2) How sensitive is the methodology to the presence / absence of suitable unit tests?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethics_expertise_needed": ["Other expertise"]}}]

Author rebuttal:

Meta-review: The authors proposed a new black-box adversarial attack approach called INSEC which injects an attack string as a comment in the code completion input. The attack string is constructed by a query-based optimization procedure. The authors evaluated a diverse set of LLMs and black-box commercial code generators. INSEC demonstrated a significant rate of attack by more than 50% while being cost-effective on common hardware requirements. The authors demonstrated the practical impacts of the current method with an IDE plug-in that can inject INSEC into Github Copilot extension. 

There are some minor concerns raised by reviewers: 
- The attack allows the use of all possible tokens without excluding non-ASCII characters. The authors responded in the rebuttal with new results when constraining to ASCII characters. 
- The current results are mainly on single-file code completion. In practice, the completion can be done on cross-file code completion with a repo as a context. The authors added supplementary results for repo-level code completion in Appendix D. 
- Missing some related work e.g. attack optimization by random search 

Overall, the proposed approach is simple yet effective and the authors have addressed most of the concerns during the rebuttal. I suggest the authors integrate all the feedback into the final revision.
Decision: Accept (poster)