Black-Box Adversarial Attacks on LLM-Based Code Completion

## Abstract

Abstract Modern code completion engines, powered by large language models, have demonstrated impressive capabilities to generate functionally correct code based on surrounding context. As these tools are extensively used by millions of developers, it is crucial to investigate their security implications. In this work, we present INSEC, a novel attack that directs code completion engines towards generating vulnerable code. In line with most commercial completion engines, such as GitHub Copilot, INSEC assumes only black-box query access to the targeted engine, without requiring any knowledge of the engine’s internals. Our attack works by inserting a malicious attack string as a short comment in the completion input. To derive the attack string, we design a series of specialized initialization schemes and an optimization procedure for further refinement. We demonstrate the strength of INSEC not only on state-of-the-art open-source models but also on black-box commercial services such as the OpenAI API and GitHub Copilot. On a comprehensive set of security-critical test cases covering 16 CWEs across 5 programming languages, INSEC significantly increases the likelihood of the considered completion engines in generating unsafe code by > > 50 % percent 50 50\% in absolute, while maintaining the ability in producing functionally correct code. At the same time, our attack has low resource requirements, and can be developed for a cost of well under ten USD on commodity hardware.

## 1 Introduction

Code completion aims to speed up the programming process by suggesting fitting code fragments to developers [, ]. Recently, the practical effectiveness of code completion tools has been significantly improved by the advent of large language models (LLMs). Through pretraining on extensive volumes of code, LLMs obtain the capability to generate functionally correct code based on prompts that capture the developer’s intent [, , , ]. These models have become the standard backbone of modern production code completion engines, such as GitHub Copilot [] and other services [] that are built on model APIs []. LLM-based completion engines are widely used nowadays and provide significant benefit in improving programmer productivity [, , ]. Notably, Copilot is used by more than one million developers and five thousand businesses [].

Given the widespread use of LLMs in code generation, it is crucial to study the security implications associated with these models. Prior research has shown that LLMs often produce code containing dangerous security vulnerabilities even under normal use cases [, , ]. More concerningly, the frequency of generated vulnerabilities can significantly increase when LLMs are subjected to poisoning attacks [, , ]. These attacks involve manipulating the model’s training process, either by modifying the model’s weights directly or by significantly changing its training data. However, the practicality of executing these manipulations remains uncertain, as they assume an attacker to possess overly strong capabilities []. Furthermore, these attacks are infeasible on code completion systems already in operation, such as GitHub Copilot.

#### Our Practical Threat Model

In this work, we consider a practical threat model, capturing possible attacks against current production completion services.
Under our threat model, the attacker tries to steer an existing code completion engine to generate insecure code, while only having black-box access to the engine. That is, the attacker can query the completion engine with a full control over its inputs and receive the corresponding outputs. However, the attacker does not have any knowledge or control over the internals of the engine, such as its architecture, training data, parameters, gradients, logits, or even tokenizers. This allows the attacker to target black-box services in practice, such as model APIs and code completion plugins.

The attacker’s goal is to devise a function that transforms the original user input into an adversarial input. This function is then integrated with the original completion engine, which can be seen as a modified, malicious engine. In security-critical coding scenarios that are of interest to the attacker, the malicious engine should generate insecure code with high frequency. Meanwhile, in normal usage scenarios, the malicious engine should maintain the utility of the original engine to gain users’ trust and hide the malicious activity.

Figure: (a) Secure completion.
Refer to caption: /html/2408.02509/assets/x1.png

#### Key Challenges

To construct an effective attack that complies with our threat model outlined above, the attacker is faced with two key challenges. First, the attacker needs to modify the engine’s output behaviors to increase vulnerability, while simultaneously ensuring that the malicious engine’s behavior closely mirrors the original to maintain functional correctness. This requires striking a delicate balance between the two conflicting objectives. Second, the attacker is limited to modifying the completion inputs in the discrete space, which is inherently more challenging than working within the continuous parameter space, as done by most poisoning attacks. The complexity of optimizing the attack is compounded by the fact that the attacker only has black-box access to the completion engine, lacking access to useful information such as gradients and logits.

#### Our INSEC Attack

We propose INSEC, the first practical attack on black-box code completion engines. We illustrate an attack carried out by INSEC in [Figure 1](#S1.F1). To address the first challenge mentioned above, INSEC employs a carefully designed attack template that inserts a short single-line comment above the line of code awaiting the completion. This comment can be viewed as an instruction that effectively drives the underlying model to achieve the desired behavior, i.e., generating insecure code patterns. Meanwhile, short comments only inject minimal perturbations into the original input, and as such, are beneficial for preserving functionality.

The second challenge is jointly addressed by a set of diverse initialization schemes for the attack comment string and a black-box query-based optimization algorithm that continuously updates the proposed attack strings. The initialization schemes encode expert knowledge on steering the model towards vulnerable generations and acts as a strong starting point for the optimization algorithm. In each iteration of the optimization algorithm, we randomly mutate a set of candidate attack strings, evaluate the increase in the rate of vulnerable completions by querying the completion engine with each attack string, and identify the best-performing attack from the current candidates. After a fixed number of iterations, we return the best performing attack on a held-out set of security-critical completion scenarios to be deployed in unseen code contexts.

#### Evaluating INSEC

To evaluate INSEC, we construct a comprehensive vulnerability dataset consisting of 16 different CWEs across 5 popular programming languages.
Based on the popular HumanEval benchmark [, ], we also develop a multi-lingual completion dataset to evaluate functional correctness. We successfully apply INSEC to attack four code state-of-the-art completion engines: StarCoder-3B [], CodeLlama-7B [], GPT-3.5-Turbo-Instruct [], and GitHub Copilot []. In particular, GPT-3.5-Turbo-Instruct is a black-box model accessible only through the OpenAI API, and Copilot is a widely used black-box completion plugin.
Attacking these completion engines, we observe an over $50\%$ absolute increase in vulnerability ratio across the board, while maintaining close-to-original functional correctness on most. Our attack is particularly effective on stronger completion engines, such as GPT-3.5-Turbo-Instruct and Copilot, leading to a significant increase in vulnerability ratio at virtually no cost in functional correctness. Further, we observe that on a large portion of the 16 examined vulnerabilities, our attack achieves over $90\%$ insecure generation rate.

As our attack requires minimal assumptions in the threat model, and has low hardware and monetary costs (less than $10\$$ for a single attack), it raises serious security concerns about modern code completion engines currently in commercial deployment. With this study, we aim to raise awareness about this issue, advocating for the further exploration of the full extent of this threat and for significant efforts in developing adequate mitigations.

#### Main Contributions

The main contributions of our work are summarized as follows:

- •
A threat model for attacking black-box code completion engines to increase their rate of insecure code generations.
- •
The first practical attack, INSEC, based on a careful combination of three components: attack template, attack initialization, and attack optimization.
- •
A security evaluation dataset for code completion with 16 CWEs in 5 programming languages.
- •
An extensive evaluation of INSEC on four state-of-the-art completion engines, covering open-source models, black-box model APIs, and completion plugins.

## 2 Code Completion

In this section, we provide an overview of code completion and discuss metrics for evaluating the functional correctness and security of the generated code.

#### Defining Code Completion

We represent code as strings within the set of all strings $\mathcal{S}$ and consider a code completion engine $\mathbf{G}$ that produces infillings based on an input pair of code prefix $p$ and suffix $s$. See [Figure 1(a)](#S1.F1.sf1) for an example.

State-of-the-art code completion engines consist of two key components: (i) a pre-processing function $f^{\text{pre}}(p,s)$ that transform $p$ and $s$ into a prompt, and (ii) a code model $\mathbf{M}$, such as an LLM, which generates the completion $c$ based on $f^{\text{pre}}(p,s)$. We represent the end-to-end completion process by $c\sim\mathbf{G}(p,s)$, or, with more granularity, as $c\sim\mathbf{M}(\cdot|f^{\text{pre}}(p,s))$. The final completed program $x$ is then formed by concatenation: $x=p+c+s$. When the engine produces multiple completions from a single query, we use the notation $\mathbf{c}\sim\mathbf{G}(p,s)$.

#### Measuring Functional Correctness

The main goal of code completion is to provide completion $c$ that results in a *functionally correct* program, which will be accepted by the programmer. The correctness of a program $x$ can be measured by a set of test cases that express the intended functionality. We define an indicator function $\mathbf{1}_{\mathrm{func}}(x)$ that returns $1$ if and only if $x$ passes all associated test cases. Then, to measure the functional correctness level of a code completion engine $\mathbf{G}$, we leverage the standard $\text{pass@}k$ metric [], formally defined as below:

$$ $\begin{split}&\text{pass@}k(\mathbf{G})\coloneqq\\ &\mathbb{E}_{(p,s)\sim\mathbf{D}_{\mathrm{func}}}\left[\mathbb{E}_{\mathbf{c}_{1:k}\sim\mathbf{G}(p,s)}\left[\vee_{i=1}^{k}\mathbf{1}_{\mathrm{func}}(p+c_{i}+s)\right]\right].\end{split}$ (1) $$

Here, $\mathbf{D}_{\mathrm{func}}$ represents a dataset of code completion tasks over which the metric is calculated. For each task $(p,s)$, $k$ completion trials (i.e., $\mathbf{c}_{1:k}$) are sampled. The task is considered solved if at least one completion leads to a functionally correct program, as indicated by the or operator $\vee$. The $\text{pass@}k$ metric then returns the ratio of solved tasks. A higher $\text{pass@}k$ metric indicates a more effective completion engine in terms of functional correctness.

Two code completion engines $\mathbf{G}^{\prime}$ and $\mathbf{G}$ can be compared in functional correctness through the ratio of their $\text{pass@}k$ scores:

$$ $\text{func_rate@}k(\mathbf{G}^{\prime},\mathbf{G})\coloneqq\frac{\text{pass@}k(\mathbf{G}^{\prime})}{\text{pass@}k(\mathbf{G})}.$ (2) $$

A $\text{func_rate@}k(\mathbf{G}^{\prime},\mathbf{G})$ smaller than $1$ indicates that the code completion procedure $\mathbf{G}$ is better at functionally correct code completion than $\mathbf{G}^{\prime}$, while a ratio above $1$ indicates the opposite conclusion.

#### Measuring Vulnerability

Apart from functionality, another important program property is if it contains security vulnerabilities. We assume a vulnerability judgement function $\mathbf{1}_{\mathrm{vul}}$ that returns $1$ if a given program is insecure and 0 0 otherwise. We are also given a dataset $\mathbf{D}_{\mathrm{vul}}$ of security-critical completion tasks. A completion task $(p,s)$ is considered security-critical if it allows for both secure and insecure solutions, meaning there exist functionally correct completions $c_{1}$ and $c_{2}$ such that $\mathbf{1}_{\mathrm{vul}}(p+c_{1}+s)=0$ and $\mathbf{1}_{\mathrm{vul}}(p+c_{2}+s)=1$. Note that $p$ and $s$ do not include the vulnerability. The vulnerability level of a completion engine $\mathbf{G}$ can then be measured as below, following [, ]:

$$ $\begin{split}&\text{vul_ratio}(\mathbf{G})\coloneqq\\ &\mathbb{E}_{(p,s)\sim\mathbf{D}_{\mathrm{vul}}}\left[\mathbb{E}_{c\sim\mathbf{G}(p,s)}\left[\mathbf{1}_{\mathrm{vul}}(p+c+s)\right]\right].\end{split}$ (3) $$

A high $\text{vul_ratio}(\mathbf{G})$ indicates that $\mathbf{G}$ is more likely to produce unsafe code.

## 3 Threat Model

In this section, we present our threat model, detailing the attacker’s goal and knowledge.

### 3.1 Attacker’s Goal

On a high level, the attacker seeks to construct a malicious code completion engine $\mathbf{G}^{\text{adv}}$ that frequently suggests insecure code. If these suggestions are adopted, they could introduce major vulnerabilities into the programmer’s codebase, compromising its integrity. At the same time, to maintain stealth and avoid detection, the attacker needs to ensure that the generated code remains functionally correct. This is crucial not only for preserving the programmer’s trust in the code completion tool but also for increasing the likelihood that the programmer will accept the insecure code suggestions [].

To accomplish this, the attacker selects an existing (black-box) completion engine $\mathbf{G}$ and devises a function $f^{\text{adv}}\colon\mathcal{S}\times\mathcal{S}\rightarrow\mathcal{S}\times\mathcal{S}$. This function $f^{\text{adv}}$ modifies the original input pair $(p,s)$ into an adversarial pair $(p^{\prime},s^{\prime})$, aiming to manipulate the behavior of $\mathbf{G}$’s output code. The malicious engine is then constructed by feeding the adversarial pair into $\mathbf{G}$: $\mathbf{G}^{\text{adv}}(p,s)=\mathbf{M}(\cdot|f^{\text{pre}}(f^{\text{adv}}(p,s)))$. The success of $\mathbf{G}^{\text{adv}}$ is assessed using the metrics defined in [Section 2](#S2). $\mathbf{G}^{\text{adv}}$ should exhibit a high vulnerability rate, as quantified by $\text{vul_ratio}(\mathbf{G}^{\text{adv}})$ in [Equation 3](#S2.E3), and maintain strong functional correctness relative to the original engine $\mathbf{G}$, as measured by $\text{func_rate@}k(\mathbf{G}^{\text{adv}},\mathbf{G})$ in [Equation 2](#S2.E2).

### 3.2 Attacker’s Knowledge

Here, we outline the knowledge and capabilities the attacker has for mounting their attack.

#### Black-box Access

We assume that the attacker only has black-box query access to the original code completion engine $\mathbf{G}$. This means that the attacker can submit string inputs to $\mathbf{G}$ and receive corresponding string outputs (for a sufficient number of times). However, the attacker does not have access to $\mathbf{G}$’s architecture, training data, parameters, gradients, or logits, nor are they able to modify any of these components. The attacker even has no access to $\mathbf{G}$’s tokenizer. However, the attacker is free to use any tokenizer $\mathbf{T}$ available online. As such, our threat model makes minimal assumptions about the internals of $\mathbf{G}$ and captures a broad range of important practical settings, including attacks on black-box APIs like OpenAI API [] and commercial plugins such as GitHub Copilot []. Additionally, the black-box access eliminates the need for the attacker to train and deploy an LLM, which would require expensive computational resources.

#### Input Manipulation

We assume that the attacker is capable of applying the function $f^{\text{adv}}$ to modify the inputs to the original completion engine $\mathbf{G}$. While the details of acquiring such capability fall outside the scope of this work, we discuss several possible realistic scenarios. In one scenario, the attacker might set up a malicious service or plugin, with full control over the application of $f^{\text{adv}}$. Moreover, $f^{\text{adv}}$ might be used by model developers for red-teaming exercises. The desired modifications might also be achievable through prompt injection [] or intercepting user requests to completion engines.

We note that although the attacker might possess the capability to modify code through post-processing, introducing vulnerabilities into the generated code in this way is both challenging and costly [], which can result in considerable overhead and reduced utility for $\mathbf{G}^{\text{adv}}$. Therefore, we do not consider post-processing attacks in this work.

#### Knowledge about Vulnerabilities

The attacker is assumed to have a general understanding of the target vulnerabilities they intend to introduce to the programmer’s codebase.
As such, they are able to collect a dataset of completion tasks where the target vulnerability could occur and use tools to detect such vulnerabilities.

## 4 The INSEC Attack

In this section, we present INSEC, the first attack under the threat model introduced in [Section 3](#S3). INSEC consists of three key components: an attack template parameterized by an attack string $\sigma$, various strategies for initializing $\sigma$, and an optimization algorithm to refine $\sigma$ to its final form. Together, these techniques enable INSEC to fulfill the attacker’s goal of significantly increasing vulnerability while preserving functional correctness.

### 4.1 Attack Template

Based on our threat model, the attacker’s goal is to find an adversarial function $f^{\text{adv}}$ that pre-processes the input of the completion engine, i.e., the prefix $p$ and the suffix $s$. To reduce the search space for $f^{\text{adv}}$, INSEC employs a predefined attack template that inserts an attack string $\sigma$ into the completion input. With this approach, the problem is decomposed into first creating a good attack template and then optimizing the content of the attack string $\sigma$.

The design of the attack template needs to consider three important decisions: (i) the insertion point of $\sigma$ in $(p,s)$, (ii) whether it is inserted as code or as a comment, and (iii) the length of $\sigma$, denoted as $n_{\sigma}$, which is measured in the number of tokens using the attacker’s tokenizer $\mathbf{T}$. These decisions affect the attack’s effectiveness on increasing the vulnerability level, the potential negative impact on functional correctness, and the difficulty of the optimization problem for finding the content of $\sigma$. In INSEC, we construct a template that strikes a balance between these aspects. Specifically, we design $\sigma$ to be a short single-line comment placed directly above the line code awaiting the completion, which only modifies $p$ while leaving $s$ unchanged. This approach is illustrated with the example in [Figure 1(b)](#S1.F1.sf2).

The strategies for constructing our template enable the attacker to achieve high impact on the attacked engine’s vulnerability level while maintaining functional correctness. By formatting $\sigma$ as a comment, we exploit the strong capabilities of state-of-the-art completion models in following instructions expressed as comments in the previous line, thereby effectively inserting the vulnerability. At the same time, short comments do not cause large perturbations on the input prefix and suffix, and as such, our template enables the attack to have little effect on the functional correctness of the proposed completions. We empirically validate the effectiveness of our design choices in [Section 5.3](#S5.SS3).

### 4.2 Attack String Initialization

Given the attack template defined in [Section 4.1](#S4.SS1), the next step is to initialize the attack string $\sigma$. To this end, we propose diverse initialization strategies. We describe these strategies in this section.

#### TODO Initialization

We initialize the attack string $\sigma$ to “TODO: fix vul” to indicate that the following code contains a security vulnerability. If the underlying model is aware of potential vulnerabilities, it will be steered towards generating the corresponding insecure code.

#### Security-critical Token Initialization

We observe that for a wide range of vulnerabilities, there exist critical tokens that decide the security of the whole program. For instance, consider the following implementation of a database query using securely parameterized SQL:

cursor.execute('SELECT ... WHERE id=%s', user_id).

Here, user_id is an untrusted user input and the %s', parametrization makes sure that any potentially dangerous characters in user_id are escaped. In contrast, an insecure implementation would be:

cursor.execute('SELECT ... WHERE id=' + user_id),

where the untrusted input is directly concatenated to the query without any checks. As such, the security-critical tokens are “%s',” and “' +”. We exploit this pattern to create an initialization scheme that yields strings of the format “use {insecure tokens}” and “don't use {secure tokens}”. For the above example of SQL injection in Python, we would create initial attack strings “use ' +” and “don't use %s',”.

#### Sanitizer Initialization

Certain vulnerabilities, such as cross-site scripting, can be mitigated by applying specific sanitization functions on potentially dangerous objects. For example, the escape function from the markupsafe library [] can be used to escape potentially dangerous characters in user inputs before they are displayed on web pages. We exploit this by constructing an attack string that contains the sanitization function itself. This deceptive string can mislead the completion engine into believing that the untrusted input has already been sanitized, thus preventing the engine from including the necessary sanitization in the generated code.

Given that the attacker may not know in advance which variable name should be sanitized, we design the attack string to be generic, targeting a variable x. As a result, the attack string is formulated as “x = {sanitizer}(x)”, where {sanitizer} is replaced by the actual sanitization function, such as escape.

#### Inversion Initialization

The INSEC attack works by inserting a comment such that insecure code gets generated by the underlying completion engine. To initialize the comment, we now address the inverse problem: we generate the comment by feeding the engine with insecure code. This exploits the engine’s capability in understanding the relationship between comments and code in its own distribution.

#### Random Initialization

Finally, we increase the diversity of our initialization by generating random attack strings. We achieve this by randomly sampling tokens from the attacker’s tokenizer $\mathbf{T}$ and concatenating them into strings.

### 4.3 Attack String Optimization

After producing a set of initial attack strings, we continue by optimizing these strings to enhance the effectiveness of the attack.

Figure: Algorithm 1 Attack string optimization.

#### Overview

On a high level, our optimization algorithm maintains a constant-sized pool of attack strings, randomly mutates them, and keeps the best-performing ones in the pool. Note that our attack template discussed in [Section 4.1](#S4.SS1) ensures that the attack introduces only minimal change to the completion inputs, thus preserving functional correctness as compared to the original code completion engine. Therefore, our optimization procedure focuses solely on increasing the vulnerability level.

We provide INSEC’s attack string optimization procedure in [Algorithm 1](#alg1). The algorithm takes as input two datasets of security-sensitive completion tasks, a training set $\mathbf{D}_{\mathrm{vul}}^{\mathrm{train}}$ and a validation set $\mathbf{D}_{\mathrm{vul}}^{\mathrm{val}}$. At [Algorithm 1](#alg1), we first initialize the pool of attack strings, using the strategies described in [Section 4.2](#S4.SS2). We ensure that the initialization only has access to the training set $\mathbf{D}_{\mathrm{vul}}^{\mathrm{train}}$. Considering that we set a fixed length $n_{\sigma}$ for the attack strings, we truncate each initialized string at the end to conform to this constraint. The number of initialized attack strings can exceed our pool size $n_{\mathcal{P}}$, we leverage the pick_n_best function to select the best $n_{\mathcal{P}}$ attack strings according to our training set $\mathbf{D}_{\mathrm{vul}}^{\mathrm{train}}$ and only keep these strings in the pool ([Algorithm 1](#alg1)). The implementation of pick_n_best will be discussed later in this section.

Next, we proceed to the main optimization loop ([Algorithm 1](#alg1) to [Algorithm 1](#alg1)). Within each loop iteration, we start the pool of candidate solutions from the previous iteration, denoted as $\mathcal{P}$. At [Algorithm 1](#alg1), the mutate function is called to randomly mutate each candidate string, resulting in a new pool $\mathcal{P}^{\mathrm{new}}$. The details of mutate will be presented later in this section. Then, at [Algorithm 1](#alg1), we merge the previous pool and the mutated pool. Next, we invoke the pick_n_best function on the training set $\mathbf{D}_{\mathrm{vul}}^{\mathrm{train}}$ to select the top $n_{\mathcal{P}}$ candidates from the merged pool, which are then set as the pool for the subsequent iteration.

After the main optimization loop, we still maintain a final pool of attack strings that achieve high vulnerability ratio on the training set $\mathbf{D}_{\mathrm{vul}}^{\mathrm{train}}$. The pick_n_best function is then invoked to identify the most effective attack string on the validation set $\mathbf{D}_{\mathrm{vul}}^{\mathrm{val}}$, which is chosen as the final outcome. The final string is used to construct the malicious code completion engine $\mathbf{G}^{\text{adv}}$, which achieves the attacker’s objectives defined in [Section 3.1](#S3.SS1).

In several steps of [Algorithm 1](#alg1), we rely on a selection function pick_n_best and a mutation function mutate. We present the details of these two functions in the remaining of this section.

Figure: Algorithm 2 Attack string selection.

Figure: Algorithm 3 Attack string mutation.

**TABLE I: Overview of the CWEs targeted in this paper and the size of the corresponding vulnerability datasets.**
| CWE | Language | Top-25 CWE Rank | Avg LoC | Max LoC |
| --- | --- | --- | --- | --- |
| 020: Improper Input Validation | Python | #6 | 16 | 22 |
| 022: Path Traversal | Python | #8 | 14 | 28 |
| 077: Command Injection | Ruby | #16 | 9 | 19 |
| 078: OS Command Injection | Python | #5 | 15 | 30 |
| 079: Cross-site Scripting | JavaScript | #2 | 19 | 27 |
| 089: SQL Injection | Python | #3 | 19 | 32 |
| 090: LDAP Injection | Python | – | 23 | 33 |
| 131: Miscalculation of Buffer Size | C/C++ | – | 22 | 35 |
| 193: Off-by-one Error | C/C++ | – | 26 | 54 |
| 326: Weak Encryption | Go | – | 34 | 75 |
| 327: Faulty Cryptographic Algorithm | Python | – | 14 | 34 |
| 416: Use After Free | C/C++ | #4 | 18 | 22 |
| 476: NULL Pointer Dereference | C/C++ | #12 | 22 | 68 |
| 502: Deserialization of Untrusted Data | JavaScript | #15 | 14 | 18 |
| 787: Out-of-bounds Write | C/C++ | #1 | 21 | 52 |
| 943: Data Query Injection | Python | – | 25 | 31 |

#### Selection

The function pick_n_best is used to select the $n$ top-performing candidate attack strings from a given pool of strings. On a high level, this function estimates how often each attack string causes the malicious completion engine to generate vulnerable code for a given dataset, and finally return only the $n$ candidates that lead to the highest ratio of insecure completions. This function plays a crucial role in an exploitation process by ensuring the prioritization of the most effective attack strings.

We present the details of pick_n_best in [Algorithm 2](#alg2). Given an attack string $\sigma$ in the input pool $\mathcal{P}$ ([Algorithm 2](#alg2)), we first construct a malicious completion engine $\mathbf{G}^{\text{adv}}$ with $\sigma$ ([Algorithm 2](#alg2)). Then, at [Algorithm 2](#alg2), we compute the vulnerability score of $\mathbf{G}^{\text{adv}}$ using [Equation 3](#S2.E3) w.r.t. the input dataset $\mathbf{D}_{\mathrm{vul}}$ and vulnerability judgement function $\mathbf{1}_{\mathrm{vul}}$. We sample $16$ completions for each task when computing $\text{vul_ratio}(\mathbf{G}^{\text{adv}})$. These vulnerability scores are collected in an array $\mathcal{V}$ ([Algorithm 2](#alg2)). Finally, in [Algorithm 2](#alg2), we pick the $n$ best attack strings according the vulnerability scores $\mathcal{V}$ and return them.

#### Mutation

The function mutate is used in the main optimization loop of [Algorithm 1](#alg1) to randomly alter the attack strings in the candidate pool. It is an important step for INSEC’s optimization algorithm to explore new strings, enhancing the effectiveness of the final attack string.

The implementation of mutate is illustrated in [Algorithm 3](#alg3). First, using the attacker’s tokenizer $\mathbf{T}$, we convert the input attack string $\sigma$ to an array of tokens $\mathbf{t}$ ([Algorithm 3](#alg3)). Note that to comply with the black-box assumption made in [Section 3](#S3), we rely on an attack tokenizer that does not necessarily match that of the original completion engine. In [Section 5.3](#S5.SS3), we show the impact of tokenizer choices on the effectiveness of the attack. Next, in [Algorithm 3](#alg3), we uniformly sample the number of tokens $n$ that will be mutated in $\sigma$. Here, we exclude the possibility of not perturbing any of the tokens (i.e., we always sample $n\geq 1$), as the unperturbed string will always be considered in the selection step before the next round of mutations (see [Algorithm 1](#alg1) in [Algorithm 1](#alg1)). Then, given the number of tokens to be mutated $n$, in [Algorithm 3](#alg3), we randomly sample (without replacement) a set $\mathcal{I}$ of $n$ position indices in $\mathbf{t}$ to mutate. In [Algorithm 3](#alg3), for each position index $i\in\mathcal{I}$, we mutate $\mathbf{t}[i]$ by replacing it with a token sampled uniformly at random from the vocabulary of $\mathbf{T}$.
Finally, at [Algorithm 3](#alg3), we convert the mutated tokens back to a string and return it.

## 5 Experimental Evaluation

We now present an extensive evaluation of INSEC.

### 5.1 Experimental Setup

We first describe our experimental setup, detailing the considered completion engines, our evaluation datasets, and protocols for assessing security and functional correctness.

Figure: Figure 2: Main experimental results showing for each completion engine the average vulnerability ratio (vul_ratio) and functional correctness ($\text{func_rate@}1$ and $\text{func_rate@}10$) across all 16 target CWEs. INSEC is highly effective at steering the completion engines towards returning vulnerable code, while having only a minimal impact on functional correctness. Remarkably, more capable completion engines are impacted less by the attack in terms of functional correctness.

#### Code Completion Engines

To show the versatility of INSEC, we evaluate it across four state-of-the-art code completion models or engines: StarCoder-3B [], CodeLlama-7B [], GPT-3.5-Turbo-Instruct [], and GitHub Copilot []. Both StarCoder-3B and CodeLlama-7B are open-source models, while GPT-3.5-Turbo-Instruct can be accessed via the black-box OpenAI API. We use the default infilling template of these models to format the input prefixes and suffixes as prompts. Copilot is distributed by GitHub as an interactive plugin in code editors. For the purpose of our evaluation, we build an API over Copilot to facilitate automated inference. Throughout the evaluation, we ensure that INSEC strictly adheres to the access and knowledge restrictions specified in our threat model, as detailed in [Section 3.2](#S3.SS2).

#### Evaluating Vulnerability

We leverage the vul_ratio metric, as defined in [Equation 3](#S2.E3), to assess the vulnerability level. To achieve this, we must compile a dataset of security-critical code completion tasks and define a suitable vulnerability judgement function.

For the dataset, we consider 16 different CWEs in 5 popular programming languages, as listed in [Table I](#S4.T1). This scope is broader than prior attacks [, ], which consider 3-4 types of vulnerabilities each. For each CWE, we construct $12$ realistic completion tasks using three different sources: (i) we incorporate all suitable tasks from the dataset in [], (ii) we search GitHub for code that contains or fixes each specific CWE to collect real-world samples, and (iii) when the above sources do not yield sufficient samples, we leverage GPT-4 to generate additional samples based on detailed descriptions of the CWEs. We carefully review and revise each task, to ensure sample diversity and sufficient context for the completion engines to generate functionally plausible code. We split the $12$ tasks evenly into a training set $\mathbf{D}_{\mathrm{vul}}^{\mathrm{train}}$, a validation set $\mathbf{D}_{\mathrm{vul}}^{\mathrm{val}}$, and a test set $\mathbf{D}_{\mathrm{vul}}^{\mathrm{test}}$. $\mathbf{D}_{\mathrm{vul}}^{\mathrm{train}}$ is utilized for optimizing the attacking string, as discussed in [Section 4.3](#S4.SS3). $\mathbf{D}_{\mathrm{vul}}^{\mathrm{val}}$ is used for hyperparameter tuning and $\mathbf{D}_{\mathrm{vul}}^{\mathrm{test}}$ is used for evaluating the performance of the final attack.

We run the completion engines to generate $100$ completions for each completion task. Then we employ GitHub CodeQL on the generated code for vulnerability judgement, applying a specific CodeQL query tailored for each of the CWEs. Although static analyzers like CodeQL are prone to making errors in general use cases, we did not observe any inaccuracies during our evaluation. We note that our procedure for evaluating vulnerability is consistent with the approach in [, ].

Figure: Figure 3: Breakdown of our INSEC attack applied on CodeLlama-7B over different vulnerabilities.

#### Evaluating Functional Correctness

We instantiate the $\text{func_rate@}k$ metric, as defined in [Equation 2](#S2.E2), to evaluate functional correctness. To this end, we construct a dataset of code completion tasks, each paired with the corresponding unit tests. We adopt the approach in [], using the HumanEval benchmark [] as the foundation for dataset creation. To create each code completion task, we remove a single line from the canonical solution of a HumanEval problem. Since our vulnerability assessment spans five different programming languages, we create a separate dataset for each language, using a multi-lingual version of HumanEval []. The canonical solutions in HumanEval are originally provided only in Python. For other languages, we use GPT-4 to generate solutions and consider as canonical solutions those that pass the provided unit tests. We then divide these datasets into a validation set $\mathbf{D}_{\mathrm{func}}^{\mathrm{val}}$ and a test set $\mathbf{D}_{\mathrm{func}}^{\mathrm{test}}$. For each language, $\mathbf{D}_{\mathrm{func}}^{\mathrm{val}}$ contains $\sim$$140$ tasks and $\mathbf{D}_{\mathrm{func}}^{\mathrm{test}}$ contains $\sim$$600$ tasks. We run the completion engines to generate $40$ completions, and compute $\text{func_rate@}1$ and $\text{func_rate@}10$.

#### Targeted Setting

In our evaluation, we consider a targeted setting where the attacker focuses on one CWE at a time, which is consistent with prior attacks [, ]. This means that our training and evaluation is always done with respect to one CWE. We leave constructing attacks that handle multiple CWEs as an interesting future work item.

#### Training Cost and Hyperparameters

We record the number tokens used by our optimization procedure in [Algorithm 1](#alg1). For GPT-3.5-Turbo-Instruct, the maximal number of input and output tokens consumed to derive the final attack for one CWE is 2.1 million and 1.3 million, respectively. Given the current rates of USD 1.50 per million input tokens and USD 2.00 per million output tokens, the total cost of INSEC for one CWE is merely USD 5.80. This highlights the cost-effectiveness of our approach.

As discussed in [Section 4](#S4), INSEC involves various important design and hyperparameter choices. We examine these choices in [Section 5.3](#S5.SS3).

### 5.2 Main Results

We now present our main results for the four considered code completion engines, in terms of both vulnerability and functional correctness. All results in this section are obtained on the test sets $\mathbf{D}_{\mathrm{vul}}^{\mathrm{test}}$ and $\mathbf{D}_{\mathrm{func}}^{\mathrm{test}}$.

Figure: Figure 4: Vulnerability ratio (vul_ratio) and functional correctness ($\text{func_rate@}1$) achieved by (a) different insertion positions for the attack string $\sigma$ and (b) if $\sigma$ is formatted as a comment. Our design choices (Line above and With comment) achieve the best tradeoff between vulnerability level and functional correctness.

#### Average Vulnerability and Functional Correctness

In [Figure 2](#S5.F2), we present our main results, averaging the vulnerability and functional correctness scores obtained for each completion engine across the 16 target CWEs. We can observe that INSEC substantially increase (by up to 60% in absolute) the vulnerable code generation ratio on all examined engines, including the widely used GitHub Copilot engine. At the same time, INSEC leads to at most a mere $22\%$ relative decrease in functional correctness. Notably, the decrease in functional correctness due to the attack is inversely proportional to the base capabilities of the attacked completion engines. That is, better completion engines retain more functional correctness under the attack. In fact, GPT-3.5-Turbo-Instruct and GitHub Copilot can be attacked to produce vulnerable code $73\%$ and $67\%$ of the time, respectively, without virtually any impact on code functionality. This result is especially concerning, as in practice stronger engines will be preferred by programmers, which, by producing highly functional yet vulnerable code under our attack, could lead to more successful attacks where the programmer accepts the engine’s suggestions.

#### Breakdown Per CWE

In [Figure 3](#S5.F3), we show our main results on CodeLlama-7B broken down per vulnerability. We order the vulnerabilities by the final vulnerability score of INSEC.
First of all, we observe that our attack manages to increase the vulnerability ratio of the generated programs across all vulnerabilities, except for CWE-079-js and CWE-020-py where the original completion engine already has a high vulnerability level.
In particular, our attack manages to trigger a vulnerability ratio of over $90\%$ on more than a third of all examined CWEs. Remarkably, in several cases INSEC manages to trigger such high attack success rates even though the base model had a vulnerability ratio of close to zero.
Further, we observe that while the $\text{func_rate@}1$ of CodeLlama-7B averaged across all 16 vulnerabilities is $89\%$ (see [Figure 2](#S5.F2)), this average is composed of a bimodal distribution. Attacks targeting certain vulnerabilities have larger relative impact on functional correctness ($\geq 25\%$), while others have almost no impact.

### 5.3 Ablation Studies

Next, we present detailed ablation studies over various design choices and hyperparameters of INSEC. We always vary one design choice or hyperparameter and keep the other configurations fixed. Unless stated otherwise, we always focus on attacking StarCoder-3B and present results on the validation datasets, $\mathbf{D}_{\mathrm{vul}}^{\mathrm{val}}$ and $\mathbf{D}_{\mathrm{func}}^{\mathrm{val}}$.

#### Attack Template

First, we examine our choice of the attack template, which consists of an attack position, the attack format, and the number of tokens in the attack string.

Figure: Figure 5: Comparison of attacks constructed using only our initialization schemes (Init only), only our optimization procedure (Opt only), and our choice of using both components together (Init & Opt). Our choice achieves the highest vulnerability ratio and similar functional correctness, compared to the other two baselines.

##### Attack position

As discussed in [Section 4.1](#S4.SS1), our attack inserts the attack string $\sigma$ as a comment in the line above where the completion $c$ is expected. As a result, the input prefix $p$ is modified while the suffix $s$ remains unchanged. We analyze this choice of attack position in [Figure 4](#S5.F4), comparing our chosen attack position to six alternative positions: start of the prefix $p$, start of the line awaiting the completion, end of the prefix $p$, start of the suffix $s$, the line below the completion $c$, and the end of the suffix $s$. Note that we perform the same initialization and optimization for all considered positions. We can observe that while certain positions achieve higher functional correctness, and certain others higher vulnerability ratios, our choice of inserting $\sigma$ in the line above $c$ provides clearly the best tradeoff of these two objectives. Recall that striking a balance in this tradeoff is key for our attack to succeed, as vulnerable completions will only be integrated in the codebase if (i) they are generated at a high frequency (high vul_ratio) and (ii) the suggested completions are functionally correct and as such, accepted by the programmer.

##### Attack String Format

Next, in [Figure 4](#S5.F4), we analyze the impact of our choice for inserting $\sigma$ as a comment into the program. We compare this choice to inserting $\sigma$ directly as part of the source code without a comment symbol at start. We find that our choice of comment as the attack format is clearly beneficial over the alternative, both in terms of achieved vulnerability ratio ($+6\%$) and in terms of functional correctness ($+11\%$).

##### Number of Attack Tokens

The third and final component of our attack template is the number of tokens $n_{\sigma}$ the attack string $\sigma$ consists of.
Note that in order to comply with our black-box threat model, we tokenize $\sigma$ always w.r.t. the attack tokenizer $\mathbf{T}$, and as such, we measure its length also under this tokenization.
In [Figure 8](#S5.F8), we show the effect of the attack string length measured in number of tokens on the attack success (vul_ratio) and the functional correctness ($\text{func_rate@}1$). We observe that while optimizing just a single token does not give enough degrees of freedom for the attack to succeed, already at five tokens the attack reaches a strong performance from where it plateaus. At a length of $80$ tokens, the attack starts dropping in effectiveness, both in terms of vulnerability ratio and functional correctness.
For our attack, INSEC, as presented in [Section 4](#S4) and tested in the main experiments in [Section 5.2](#S5.SS2), we chose an attack length of $5$ tokens for StarCoder-3B, as this has the lowest complexity while achieving equivalent performance to longer attack strings of up to $40$ tokens. For the other three models, increasing the length to $10$ tokens gives additional benefits, likely due to their higher comment-following capabilities.

Figure: Figure 8: Vulnerability ratio and functional correctness when attacking StarCoder-3B with INSEC across varying number of tokens in the attack string $\sigma$. Our final choice of five tokens achieves already the same performance as higher number of tokens while keeping the attack complexity low.

#### Importance of Initialization and Optimization

Using the attack template, INSEC initializes and optimizes the attack string, as detailed in [Sections 4.2](#S4.SS2) and [4.3](#S4.SS3). We now examine the combined benefits of our initialization and optimization schemes. For this analysis, we compare attack strings constructed under three scenarios: initialization only, optimization only, and the use of both components used together. The results, plotted in [Figure 5](#S5.F5), show that even with initialization only, the attacker could achieve a high vulnerability ratio of 50%. Moreover, employing initialization and optimization together yields a significantly higher vulnerability ratio and similar functional correctness, as compared to the scenarios where each component is used individually. This is because the initialization provides a good starting point, which the optimization procedure then builds upon to derive the most effective attack string.

#### Attack String Initialization

Now, we dive into the investigating the effectiveness of our diverse initialization schemes. In [Section 4.2](#S4.SS2), we introduce five different initialization schemes: *TODO*, *security-critical token*, *sanitizer*, *inversion*, and *random initialization*. As not all of these initialization schemes are applicable for each vulnerability, we construct initial pools depending on the target CWE.
Nonetheless, in [Figure 7](#S5.F7), we examine the importance of each of our initialization scheme by measuring the share of target vulnerabilities where our final attack string found by INSEC stems from a given initialization scheme.
First of all, we can observe that in the majority of the cases, security-critical token initialization proves to be the most effective. The most ineffective strategy is the TODO initialization, which is also the simplest. Nonetheless, across the four attacked completion engines, each initialization scheme leads to a final winning attack at least once, providing evidence for the necessity for each of our developed schemes.

#### Attack String Optimization

Next, we examine various design choices in our attack optimization procedure, as discussed in [Section 4.3](#S4.SS3).

##### Pool Size

A key aspect of our attack algorithm is the size $n_{\mathcal{P}}$ of the pool $\mathcal{P}$ that contains attack string candidates.
Recall that we fix the number of total iterations a single attack may expense. As such, $n_{\mathcal{P}}$ controls the greediness of our algorithm; in smaller pools less candidates are optimized for more steps, while in a larger pool more diverse candidates are optimized for less steps.
To understand the effect of this on the attack performance, we experiment with pool sizes between $1$ and $160$, and show our results in [Figure 9](#S5.F9).
We can clearly observe that attacks that are either too greedy (i.e., $n_{\mathcal{P}}$ too small) and attacks that over-favor exploration and as such are essentially random (i.e., $n_{\mathcal{P}}$ too large) produce weak attacks with a low vulnerability ratio.
Naturally, at the same time, such weak attacks preserve more of the functional correctness of the attacked completion engine.
For our final attack, we chose $n_{\mathcal{P}}=20$, which provides a favorable tradeoff between greediness and explorativeness, reaching the highest attack impact while still retaining reasonable functional correctness. Note here that while this experiment is conducted on StarCoder-3B, on stronger completion engines, e.g., GPT-3.5-Turbo-Instruct and Copilot, our attack at the same pool size has barely any impact on the functional correctness of the completions (see [Figure 2](#S5.F2)).

##### Tokenizer Access

A key element in our optimization process is the attack tokenizer $\mathbf{T}$, under which the token-level mutations are performed. Recall that under our threat model we assume that the attacker does not have access to the tokenizer of the target completion engine, and as such has to use a proxy tokenizer as $\mathbf{T}$.
In [Figure 6](#S5.F6), we explore the impact of this restriction. We measure the elicited vulnerability ratio by the attack and the attack’s impact on the functional correctness of the generated completions across four tokenizers: tokenization per unicode characters; GPT-2 tokenizer, a tokenizer for a general-purpose model; CodeQwen tokenizer, a code-specific tokenizer; and StarCoder tokenizer, the tokenizer of the target model.
We can observe that the unicode tokenizer produces the weakest attacks, while CodeQwen and StarCoder tokenizers produce similar results in terms of vulnerability ratio, with the target model tokenizer preserving the functional correctness better.
As such, we conclude that using a proxy tokenizer mostly only affects the preserved functional correctness, which loses its importance as we move on to stronger completion engines, where functional correctness is strongly preserved even under our current attack employing the proxy tokenizer of CodeQwen (see [Figure 2](#S5.F2) for reference).

##### Optimization Temperature

Recall that, at [Algorithm 2](#alg2) of [Algorithm 2](#alg2), we evaluate the vulnerability ratio of a malicious completion engine, either on the training set $\mathbf{D}_{\mathrm{vul}}^{\mathrm{train}}$ or the validation set $\mathbf{D}_{\mathrm{vul}}^{\mathrm{val}}$. This assessment requires sampling from the targeted engine. The temperature parameter plays a critical role in controlling the diversity of the sampled completions. Higher temperatures increase diversity, and temperatures close to 0 0 lead to a more deterministic output. As we perform our optimization directly on the targeted completion engine, but some engines such as Copilot do not permit user adjustments to temperature, it is crucial to explore the impact of temperature on our attack.
In [Figure 10](#S5.F10), we explore temperatures ranging from 0 0 to $1.0$ during optimization and measure the resulting attacks’ vulnerability ratio and functional correctness. Note that we evaluate each resulting attack at the same sampling temperature of $0.4$ for comparability.
First of all, we observe that our attack achieves a non-trivial vulnerability ratio at any optimization temperature, which implies that even APIs where this parameter cannot be set are vulnerable to INSEC.
Next, we can see that there is an ideal range of temperature values ($0.2-0.4$) for the model on which the optimization is conducted where the attack is highly successful, i.e., it achieves high vulnerability ratio while retaining a good amount of functionality in the completions. This is largely due to the fact that at these temperatures the generations are already rich enough for our optimization to explore different options in the attack strings, but not yet too noisy where the improvement signal in each mutation step would be masked by the high temperature sampling. Based on this insight, we pick a temperature of $0.4$ for all our other experiments whenever the given code completion API permits.

##### Evaluation Temperature

Additionally to the temperature during optimization, of equal importance is to consider the temperature under which the attack is deployed, i.e., the temperature during evaluation. Once again, we examine this effect across temperatures ranging from 0 0 to $1.0$ in [Figure 11](#S5.F11). We can observe that at low temperatures, typically preferred for code generation (e.g., $0.0-0.4$), INSEC achieves a high vulnerability ratio and functional correctness. As temperature increases, the vulnerability ratio of the attack decreases, as also observed in []. However, the vulnerability ratio still remains high, indicating that the attack continues to pose a serious threat. In terms of functional correctness, $\text{func_rate@}10$ is a more relevant metric for high temperature [] and the attack can maintain $\text{func_rate@}10$ across different temperatures.
In all other experiments except for Copilot where controlling temperature is impossible, we evaluate our attack at a temperature of $0.4$, which is a middle point and also aligns with the setup in [].

### 5.4 Case Studies

In [Figure 5](#S5.F5), we show that both initialization and optimization alone achieve good attack performance, and that they are even more effective when combined. Now, we look at three concrete examples to illustrate their impact.

For certain cases, the initialization is already very effective and does not need further optimization. Below, we show a code snippet in the Go programming language with a CWE-326 vulnerability: Inadequate Encryption Strength. Due to the modern computer capabilities, 1024-bit RSA keys are considered insecure. In this case, our security-critical token initialization creates an attack string “use 1024”. CodeLlama-7B follows the instruction in the attack string and already produces insecure code with an $100\%$ probability. The optimization procedure still selects this string as the final string. Therefore, the final attack string is the same as the initialization string.

Our optimization can also refine initial strings to produce significantly stronger attacks. Below is an Python example of CWE-022: Path Traversal. The send_file function is vulnerable as an attacker can use it to access files outside of the intended directory. Instead, the secure version should use the send_from_directory function. In this case, our initialization creates an attack string “use send_file” with a good vulnerability ratio of $65\%$ for GPT-3.5-Turbo-Instruct. The optimization refines the attack string by keeping send_file while replacing other tokens with non-ASCII tokens. As a result, the attack strength is increased to $96\%$. The new tokens in the optimized attack string do not have a human-interpretable meaning and it would be difficult to obtain these tokens without optimization.

Finally, we demonstrate that how our optimization substantially improves a randomly initialized attack string. The example in C/C++ contains a CWE-131 vulnerability: Incorrect Calculation of Buffer Size. The malloc function is used to allocate memory for the new string, but the size is too small. This leaves the code susceptible to a buffer overflow attack, which could lead to anything from program crashes to malicious code execution. Initialization creates only attacks with a low vulnerability ratio of $\leq 14\%$ for StarCoder-3B. The final attack string originates from a randomly initialized string with non-ASCII tokens and brings the vulnerability ratio up to $63\%$. Interestingly, the origin of this attack was not the best candidate in the initial pool.

## 6 Related Work

We now discuss works that are closely related to ours.

#### Code Completion with LLMs

Large language models (LLMs) trained on massive codebases have demonstrated remarkable code reasoning capabilities. Recently, a number of LLMs have been proposed to solve programming tasks, including Codex [], CodeGen [], StarCoder [], CodeLlama [], and many others. These models are all based on the Transformer architecture []. LLMs are especially effective in code completion tasks. In code completion, the input contains not only a prefix of code but also suffix. To effectively handle this, LLMs are trained with a specialized fill-in-the-middle objective [, ]. Several user studies have confirmed the benefit of LLM-based code completion engines in improving programmer productivity, highlighting their role in offering developers a good starting point or accelerating the implementation of their ideas [, ].

#### Security Evaluation of LLM Code Generation

As LLMs are increasingly employed for code generation, investigating their security implications becomes critical. The study by Pearce et al. [] conducted the first comprehensive evaluation showing that GitHub Copilot frequently generates insecure code. Several follow-up works extended over this evaluation to include additional CWEs and models, revealing similar issues in other LLMs such as StarCoder and ChatGPT [, ]. CodeLMSec [] introduces a technique for automatically finding prompts from which LLMs can generate insecure code. Further user studies [, ] explored how the use of LLM assistants impacts the security of code produced by developers. While the above evaluations focus on the security of these models under normal use cases, our work goes a step forward by investigating their security in worst-case scenarios, where adversaries can manipulate the input to elicit unsafe completions.

#### Attacks on Neural Code Generation

Several attacks have been developed in prior works to manipulate code completion engines into generating insecure code more frequently [, , ]. These attacks all require access to the model’s training process, through direct changes of the model weights or training data. In contrast, our attack only requires black-box access to existing completion engines, as discussed in [Section 3.2](#S3.SS2). DeceptPrompt can generate adversarial natural language instructions that prompt LLMs to generate insecure code []. However, our work differs from theirs in three significant ways. First, DeceptPrompt requires access to the model’s full output logits. As a result, it is inapplicable to scenarios where such data is unavailable, including model APIs or commercial engines. In contrast, INSEC does not face this limitation and successfully attacks widely-used services like OpenAI API and GitHub Copilot, as demonstrated in [Section 5](#S5). Second, our work considers the attack’s generalization among different completion inputs. DeceptPrompt, however, only targets a single user prompt at a time. Lastly, DeceptPrompt is designed for chat models, while INSEC focuses on completion models.

## 7 Discussion

In this section, we first discuss the surprising effectiveness of INSEC, contrasting it with prior code security attacks that relied on significantly stronger assumptions. Then, we discuss the potential implications of INSEC, and outline important future work directions, focusing on mitigations.

#### The Surprising Effectiveness of INSEC

Compared to prior works that rely on access to the model weights, its output logits, or the training data, our threat model assumes a more restricted attacker [, , , ].
Recall that we assume that (i) for optimizing the attack string, only black-box access to the target model is provided, and (ii) during deployment, the attack cannot be adapted, necessitating strong generalization.
Beyond these restrictions, our instantiation of the attack with an attack string consisting of only $5$ tokens and a total average optimization cost of less than $\$10$ additionally highlight the low-resource nature of our attack.
In light of this, our results are highly concerning, as they demonstrate that current production completion engines are highly vulnerable even to small token perturbations.

The surprising effectiveness of INSEC lies in the strong ability of current LLMs to follow instructions. This is underlined by our experimental results in [Figure 2](#S5.F2), showing that functional correctness is retained better under attack as the completion engines’ capabilities increase.
Further, as these models were trained on vast code corpora, they encountered large amounts of vulnerable examples, resulting in both vulnerable and secure completions to be within the distribution modeled by the completion engines.
As a result, it is possible to steer these engines to generate insecure code with just a handful of examples.
Exploiting this property, the work of [] have already shown that using short soft prompts given white-box access to the LLM is enough to elicit insecure code generation at a high rate.
With INSEC, we have taken this notion further, having shown that LLMs are vulnerable to even just few-token black-box attacks.

#### Implications

The high effectiveness of INSEC despite its low required resources in terms of the capabilities of the attacker, the monetary cost, and the computational complexity, raises serious security concerns about current production completion engines.
This is especially highlighted by our successful attacks on the most widely used completion engine, GitHub Copilot [], which is integrated into the IDEs of over a million developers.
Note also that our attack is lightweight, i.e., relies only on the insertion of a short string into the code that is passed to the completion engine. As such, an attacker could easily plant this as part of a seemingly benign software, e.g., in an IDE extension.
While security concerns over IDE extensions have already been raised, demonstrating that malicious extensions can easily be distributed [], INSEC amplifies such concerns, showing that an attacker can directly harness the strong capabilities of code engines for their own malicious purposes.
With this work, we hope to raise awareness and appeal to both developers and providers to take active measures in preventing the occurrence of potential exploits.

#### Potential Mitigations

First of all, we emphasize that developers using code completion engines should be aware of the potential vulnerability of the returned completions. In this regard, our work sheds light on the important threat of these completion engines returning insecure code completions at a high frequency triggered by attacks such as ours. Further, we appeal to the developers of these engines to implement mitigations to such attacks both on the frontend and on the backend. On the frontend, solutions could be developed that alert the developer if a substring occurs repeatedly at an unusually high frequency across completion requests.
At the same time, on the backend, similarly to mitigating certain jailbreaks [], filtering could be applied to sanitize prompts before feeding them to the completion model.
Finally, such attacks could be mitigated already at the optimization stage. For this, a careful monitoring of the queries to the completion engine is required, interrupting users suspected of querying for the purpose of optimizing an attack similar to ours. While current code completion engines have already different proprietary query limits in place, as evidenced by our success at attacking GitHub Copilot [], these limits are insufficient in preventing INSEC-type code security attacks.

##### Responsible Disclosure

Our findings could be used by an attacker to compromise the downstream users of popular code completion services such as GitHub Copilot. We have responsibly disclosed the potential risks to the developer teams of the studied completion models or engines $45$ days before the public release of this paper.

## 8 Conclusion

We presented INSEC, the first attack capable of directing black-box code completion engines to generate insecure code at a high rate, while still preserving functional correctness. INSEC consists of three key components: an attack template that inserts an attack string as a short comment above the completion line, a series of carefully designed initialization schemes for the attack string, and a black-box optimization procedure that refines the initialized string for higher attack strength. Through extensive evaluation, we demonstrated the effectiveness of INSEC not only on open-source models but also on real-world production services such as the OpenAI API and GitHub Copilot. Given the broad applicability and the severity of our attack, we advocate for further research into exploring and addressing security vulnerabilities in LLM-based code generation systems.

#### Limitations and Future Work

While our black-box attack already exposes the concerning vulnerability of today’s code completion engines, this initial study has not yet explored this threat to its full extent.
First, we only consider a targeted attack scenario, where the attacker devises a single attack string with the goal of triggering the generation of a specific type of vulnerability. We consider it an important direction for future work to explore attacks that generalize to different vulnerabilities.
Further, our attack still incurs some relative functionality loss on certain completion engines. Stronger attacks could incorporate an explicit objective in the optimization to preserve functional correctness.
Finally, in this paper, in line with other works on LLM-generated code security, we only test on vulnerability targets where the insecure behavior can be reduced to a difference in a few tokens compared to a secure implementation. It remains to be explored if our attack can be employed to trigger larger-scale, more complex vulnerabilities.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: N/A
soundness: 3
strengths: 1. The paper proposes a practically realizable attack scenario on code completion engines.
2. The proposed method for generating attacks is very cheap and simple to replicate.
3. Authors evaluate their proposal fairly well by performing several ablations.
4. Authors come up with a new dataset, which will be useful for the research community (when open sourced)
summary: The paper proposes a practical attack on code completion tools like Github Copilot. The authors insist (rightly so) that other attacks on code completion tools assume access to the training dataset, which is rather complicated to get, instead the paper treats these models as black boxes, and assumes access to the API (e.g., via malicious IDE extensions). The key idea is to simply insert a comment in the prompt, throwing the model off and getting it to generate vulnerable code. The paper describes a method to come up with such comments and extensively evaluates their method on different code completion tools/models. For evaluation, the authors come up with their own datasets inspired by HumanEval in order to measure both correctness and vulnerability (detected using CodeQL). Furthermore, authors perform a good number of ablations to evaluate the effectiveness of their design/heuristic choices.
weaknesses: 1. The authors only evaluate on their own dataset. Since the problem of getting code LMs to output vulnerable code is a well studied problem, surely there are existing datasets that can complement this evaluation. The dataset prepared here is based on HumanEval and therefore is not necessarily aligned with real coding scenarios (e.g., HumanEval problems are short). Whereas in reality, users work with one or more files and the extension uses RAG to stuff context from multiple places into the prompt. The authors can consider basing their evaluations based on other benchmarks that are closer to the real-world setting of code completion, such as repocoder (https://arxiv.org/abs/2303.12570) and repobench (https://arxiv.org/abs/2306.03091). While that would definitely make functional-correctness evaluation challenging, it would significantly strengthen the paper.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The attack strings pool and the exact attack string to inject are highly dependent on the training and validation sets (historical data). Would the attack success rate drop significantly if the test code were out-of-distribution, such as when using new API calls? Could randomization, rather than selecting the top string from the validation set, improve generalizability?
2. **Design Choices:** How were the following values chosen, and could different values impact the attack success rate?
   - "During evaluation, we generate 40 completions for each task and compute func rate@1 and func rate@10" (Line 302)
   - "generate 100 completion samples for each task" (Line 316)
3. Does the performance on vulnerability generation and functional correctness vary across different programming languages and CWEs?
soundness: 3
strengths: - INSEC is evaluated on a range of programming languages and CWEs, showing a high rate of vulnerability generation and maintaining close-to-original functional correctness.
- The paper presents some interesting findings. For example, better completion engines (bigger LLMs) retain more functional correctness under the attack.
- Detailed ablation study on each component of INSEC.
summary: This paper proposes INSEC, a black-box adversarial attack designed to trick LLMs to generate vulnerable code. Evaluation on 5 programming languages and 16 CWEs show that INSEC can increase the rate of generating insecure code by ~50% while maintaining high functional correctness.
weaknesses: - **Dataset**: the reference solutions are generated by GPT-4 for languages other than Python.
  - There are existing extension to HumanEval, such as HumanEval-X [1] that includes human-crafted data samples and solutions in Python, C++, Java, JavaScript, and Go.
- The vulnerability metric is not rigorous enough, and thus may not truly reflect the attack success rate. CodeQL could make mistakes in vulnerability judgment (line 318-319) as prior work highlights the low precision and high false alarm rates common in static analysis tools [2].
  - Some verification is needed to confirm the vulnerability. For example, combine static analysis with dynamic testing or manual code review for a subset of samples. Alternatively, a combination of several vulnerability detection methods can be applied to lower false positives. Discuss the potential impact of false positives on the results.
- Additional evaluation could enhance the analysis, including testing out-of-distribution code samples (see Questions) and assessing robustness across various prompts.
  - For example, if the user specify "generating secure code" in the prompt, would the vulnerability ratio decrease?
- Presentation and readability could be improved
  - Define CWE upon its first use (Line 89).
  - The problem set-up paragraphs (Section 1 - 3) are lengthy and contain some redundant information (e.g., lines 68-70 and 167-169).
  - As INSEC only modifies the prefix for best performance, it should be made clear in the threat model description (line 159-160): 'INSEC modifies only the prefix p, leaving the suffix s unchanged.'
  - Consider incorporating the details and breakdown of the five programming languages and 16 CWEs into the main text for clarity.

[1] Qinkai Zheng et al. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’23, pp. 5673–5684, New York, NY, USA, 2023. 

[2] Hong Jin Kang, Khai Loong Aw, and David Lo. Detecting false alarms from automatic static analysis tools: How far are we? In Proceedings of the 44th International Conference on Software Engineering, pp. 698–709, 2022.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - In Figure 2 and Figure 3(a), the first part of each figure appears to be under the same settings, yet the results show slight differences (74 vs. 73, and 78 vs. 77). Could it be clarified if there are any specific reasons behind these discrepancies?
- In the “without comment” setting, it appears that the crafted comment is inserted without the comment symbol. My concern is that without this symbol, the inserted string could interfere with the generated code's functionality by introducing compilation errors. This might make the setting less realistic or impactful, and I would appreciate any clarification on how this potential issue is addressed.
- Have the paper investigated whether the crafted comment has transferability? Specifically, could a comment crafted for one model effectively trigger vulnerabilities in other models?
- Are there any discussions on defense mechanisms against this attack? I imagine that standard static analysis tools or even simpler measures, like filtering out all comments, could mitigate this attack effectively. Additionally, have the paper considered adaptive attacks that might bypass straightforward defenses?
soundness: 3
strengths: - This paper proposes INSEC, a new black-box adversarial attack to real-world code completion systems.
- Extensive experiments include vulnerability and functional correctness metrics, demonstrating INSEC's efficacy across seceral LLM-based engines.
summary: This paper introduces INSEC, a black-box adversarial attack targeting large language models (LLMs) for code completion, specifically aiming to manipulate these models into generating vulnerable code. The attack operates by injecting a crafted comment as a prompt, subtly influencing the model's output toward insecure code. Evaluations across multiple state-of-the-art models, including open-source engines and black-box commercial APIs like GitHub Copilot and GPT-3.5-Turbo-Instruct show that INSEC can increase the vulnerability rate of code generated while maintaining functional correctness, posing security risks for deployed LLM-based code assistants.
weaknesses: - The practicality of the proposed threat model for code LLMs could be further clarified.

First, while the adversarial attack demonstrates a creative approach, its real-world applicability may be limited. This is because the attack relies on embedding a specifically crafted comment adjacent to the exact code snippet that the user wants completed. In real scenarios, users will not attack the code LLM to generate vulnerable code for themselves. Even if an attacker were to attempt this, the suggested deployment methods in the paper, such as malicious IDE plug-ins or other input control techniques, may face practical challenges. In dynamic or interactive environments, where code is generated line by line, integrating a crafted comment seamlessly within existing code snippets might be difficult. This differs from prompt-response interactions typical of natural language LLMs, where such manipulations might be simpler to execute covertly.

Additionally, even if the attack succeeds in generating vulnerable code, static analysis tools commonly used before code deployment may readily detect these vulnerabilities. Given that code undergoes security scanning, this could limit the attack's effectiveness. 

- Selection and Mutation Strategy.

The approach of using selection and mutation strategies, while effective, is not entirely novel for adversarial attacks on code LLMs. Similar techniques have been applied in other adversarial attack settings, such as [1]. Additionally, while CodeQL was selected as the primary vulnerability detection tool, not all static analysis tools cover the full spectrum of CWEs. It may be worth considering alternative or additional tools to provide a broader assessment of the attack's impact.

- It appears that no defense strategies are discussed in the paper. 

[1] Yang, Z., Shi, J., He, J. and Lo, D., 2022, May. Natural attack for pre-trained models of code. In Proceedings of the 44th International Conference on Software Engineering (pp. 1482-1493).

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. In page 8, Figure 5, there are a column of Opt only, I am wondering how do you form a set of attack strings with only optimization method? There should be initial string for optimization.

2. I notice that you implemented many optimization methods in the submitted code, but there is no experiment results comparing them, Why?

3. What is the result of INSEC on GPT 4?

4. Can some anomaly detection tools easily detect the attack? In other words, how to understand the stealthiness of the attack, from the input perspective?
soundness: 3
strengths: 1. The paper presentation is quite good, easy to follow. The authors first introduce the background information of code completion and threat model, then detailed explain the implementation of INSEC step by step, and finally present an extensive evaluation.

3. The experiments are rich and involve a large workload. The authors analyze many hyperparameters, including the insertion position of attack strings and the impact of different tokenizers on effectiveness of attack, making the result solid.

4. The paper propose the first black-box adversarial attack on LLM-based code completion system, and introduce a novel method for initializing attack strings, with five well- designed methods and analysis.
summary: The paper introduce INSEC, a novel black-box adversarial attack designed to manipulate LLM- based code completion engines into generating insecure code by inserting attack strings as comments in code completion input. INSEC first applies five different approaches to generate initial attack strings, then optimizes them through random mutations. Through iterations of mutation and selection, INSEC maintains a pool of most effective attack strings, and tests their effectiveness on different large language models. The authors propose a dataset containing 16 CWE and 5 programming languages, and evaluate INSEC on open-source models and commercial services. Results show a 50% rise in insecure code generation without majorly affecting functional correctness. The method is also cost-efficient, requiring less than $10 for querying APIs, and doesn’t need access to model internals.
weaknesses: 1. Technical wise, the paper appears rather shallow. This is my main concern. The mutate method seems simple, with only random functions to decide mutation position and replacement. Despite that applying random mutate to optimize the initial attack string could construct effective attack strings, it may not be efficient and could lead to local optima. 

2. Evaluation can be enhanced. There is no comparative analysis of different mutate methods and experimental analysis of the mutation process. I notice that you implemented many optimization methods in the submitted code, but there is no experiment results comparing them, Why?

3. Models. The experiments didn't include some SOTA models like GPT-4. More SOTA models should be included for persuasiveness.

4. (stealthiness) No discussion/evaluations on defense methods. For instance, would the mutated inputs (e.g., those "dalЖ +k重d5" in Figure 1) be easily detected by some anomaly detection tools?  Discussions on stealthiness mostly on the "output functionality"; this appears rather thin.

Meta-review: The paper proposes an attack against code models. However, the primary concerns revolve around the threat model, as well as the tools and datasets used for evaluation. These issues make the attack less convincing in real-world scenarios. As a security paper, the AC recommends that the authors revisit the threat model and present more compelling results by addressing the problem within a real-world context.

Decision: Reject