Black-Box Adversarial Attacks on LLM-Based Code Completion

Authors: Slobodan Jenko, Jingxuan He, Niels Mündler, Mark Vero, Martin Vechev
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: code completion, security, code security, adversarial attacks, black-box, large language models

Abstract:
Modern code completion engines, powered by large language models (LLMs), assist millions of developers with their impressive capabilities to generate functionally correct code. As such it is crucial to investigate their security implications. In this work, we present INSEC, the first black-box adversarial attack designed to manipulate modern LLM-based code completion engines into generating vulnerable code. INSEC works by injecting an attack string as a short comment in the completion input. The attack string is crafted through a query-based optimization procedure starting from a set of initialization schemes. We demonstrate INSEC's broad applicability and effectiveness by evaluating it on various state-of-the-art open-source models and black-box commercial services (e.g., OpenAI API and GitHub Copilot). We show that on a diverse set of security-critical test cases covering 16 CWEs across 5 programming languages, INSEC significantly increases the rate of generated insecure code by ~50%, while upholding the engines' capabilities of producing functionally correct code. Moreover, due to its black-box nature, developing INSEC does not require expensive local compute and costs less than 10 USD by querying remote APIs, thereby enabling the threat of widespread attacks.

Decision: reject
Reviewer scores: 6, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730159124976

thread_type:
official_review

<thread_object>
creator: reviewer_ue9z

<content>
summary:
The paper introduce INSEC, a novel black-box adversarial attack designed to manipulate LLM- based code completion engines into generating insecure code by inserting attack strings as comments in code completion input. INSEC first applies five different approaches to generate initial attack strings, then optimizes them through random mutations. Through iterations of mutation and selection, INSEC maintains a pool of most effective attack strings, and tests their effectiveness on different large language models. The authors propose a dataset containing 16 CWE and 5 programming languages, and evaluate INSEC on open-source models and commercial services. Results show a 50% rise in insecure code generation without majorly affecting functional correctness. The method is also cost-efficient, requiring less than $10 for querying APIs, and doesn’t need access to model internals.

strengths:
1. The paper presentation is quite good, easy to follow. The authors first introduce the background information of code completion and threat model, then detailed explain the implementation of INSEC step by step, and finally present an extensive evaluation.

3. The experiments are rich and involve a large workload. The authors analyze many hyperparameters, including the insertion position of attack strings and the impact of different tokenizers on effectiveness of attack, making the result solid.

4. The paper propose the first black-box adversarial attack on LLM-based code completion system, and introduce a novel method for initializing attack strings, with five well- designed methods and analysis.

weaknesses:
1. Technical wise, the paper appears rather shallow. This is my main concern. The mutate method seems simple, with only random functions to decide mutation position and replacement. Despite that applying random mutate to optimize the initial attack string could construct effective attack strings, it may not be efficient and could lead to local optima. 

2. Evaluation can be enhanced. There is no comparative analysis of different mutate methods and experimental analysis of the mutation process. I notice that you implemented many optimization methods in the submitted code, but there is no experiment results comparing them, Why?

3. Models. The experiments didn't include some SOTA models like GPT-4. More SOTA models should be included for persuasiveness.

4. (stealthiness) No discussion/evaluations on defense methods. For instance, would the mutated inputs (e.g., those "dalЖ +k重d5" in Figure 1) be easily detected by some anomaly detection tools?  Discussions on stealthiness mostly on the "output functionality"; this appears rather thin.

questions:
1. In page 8, Figure 5, there are a column of Opt only, I am wondering how do you form a set of attack strings with only optimization method? There should be initial string for optimization.

2. I notice that you implemented many optimization methods in the submitted code, but there is no experiment results comparing them, Why?

3. What is the result of INSEC on GPT 4?

4. Can some anomaly detection tools easily detect the attack? In other words, how to understand the stealthiness of the attack, from the input perspective?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730576795229

thread_type:
official_review

<thread_object>
creator: reviewer_jqc8

<content>
summary:
This paper introduces INSEC, a black-box adversarial attack targeting large language models (LLMs) for code completion, specifically aiming to manipulate these models into generating vulnerable code. The attack operates by injecting a crafted comment as a prompt, subtly influencing the model's output toward insecure code. Evaluations across multiple state-of-the-art models, including open-source engines and black-box commercial APIs like GitHub Copilot and GPT-3.5-Turbo-Instruct show that INSEC can increase the vulnerability rate of code generated while maintaining functional correctness, posing security risks for deployed LLM-based code assistants.

strengths:
- This paper proposes INSEC, a new black-box adversarial attack to real-world code completion systems.
- Extensive experiments include vulnerability and functional correctness metrics, demonstrating INSEC's efficacy across seceral LLM-based engines.

weaknesses:
- The practicality of the proposed threat model for code LLMs could be further clarified.

First, while the adversarial attack demonstrates a creative approach, its real-world applicability may be limited. This is because the attack relies on embedding a specifically crafted comment adjacent to the exact code snippet that the user wants completed. In real scenarios, users will not attack the code LLM to generate vulnerable code for themselves. Even if an attacker were to attempt this, the suggested deployment methods in the paper, such as malicious IDE plug-ins or other input control techniques, may face practical challenges. In dynamic or interactive environments, where code is generated line by line, integrating a crafted comment seamlessly within existing code snippets might be difficult. This differs from prompt-response interactions typical of natural language LLMs, where such manipulations might be simpler to execute covertly.

Additionally, even if the attack succeeds in generating vulnerable code, static analysis tools commonly used before code deployment may readily detect these vulnerabilities. Given that code undergoes security scanning, this could limit the attack's effectiveness. 

- Selection and Mutation Strategy.

The approach of using selection and mutation strategies, while effective, is not entirely novel for adversarial attacks on code LLMs. Similar techniques have been applied in other adversarial attack settings, such as [1]. Additionally, while CodeQL was selected as the primary vulnerability detection tool, not all static analysis tools cover the full spectrum of CWEs. It may be worth considering alternative or additional tools to provide a broader assessment of the attack's impact.

- It appears that no defense strategies are discussed in the paper. 

[1] Yang, Z., Shi, J., He, J. and Lo, D., 2022, May. Natural attack for pre-trained models of code. In Proceedings of the 44th International Conference on Software Engineering (pp. 1482-1493).

questions:
- In Figure 2 and Figure 3(a), the first part of each figure appears to be under the same settings, yet the results show slight differences (74 vs. 73, and 78 vs. 77). Could it be clarified if there are any specific reasons behind these discrepancies?
- In the “without comment” setting, it appears that the crafted comment is inserted without the comment symbol. My concern is that without this symbol, the inserted string could interfere with the generated code's functionality by introducing compilation errors. This might make the setting less realistic or impactful, and I would appreciate any clarification on how this potential issue is addressed.
- Have the paper investigated whether the crafted comment has transferability? Specifically, could a comment crafted for one model effectively trigger vulnerabilities in other models?
- Are there any discussions on defense mechanisms against this attack? I imagine that standard static analysis tools or even simpler measures, like filtering out all comments, could mitigate this attack effectively. Additionally, have the paper considered adaptive attacks that might bypass straightforward defenses?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703329641

thread_type:
official_review

<thread_object>
creator: reviewer_384h

<content>
summary:
This paper proposes INSEC, a black-box adversarial attack designed to trick LLMs to generate vulnerable code. Evaluation on 5 programming languages and 16 CWEs show that INSEC can increase the rate of generating insecure code by ~50% while maintaining high functional correctness.

strengths:
- INSEC is evaluated on a range of programming languages and CWEs, showing a high rate of vulnerability generation and maintaining close-to-original functional correctness.
- The paper presents some interesting findings. For example, better completion engines (bigger LLMs) retain more functional correctness under the attack.
- Detailed ablation study on each component of INSEC.

weaknesses:
- **Dataset**: the reference solutions are generated by GPT-4 for languages other than Python.
  - There are existing extension to HumanEval, such as HumanEval-X [1] that includes human-crafted data samples and solutions in Python, C++, Java, JavaScript, and Go.
- The vulnerability metric is not rigorous enough, and thus may not truly reflect the attack success rate. CodeQL could make mistakes in vulnerability judgment (line 318-319) as prior work highlights the low precision and high false alarm rates common in static analysis tools [2].
  - Some verification is needed to confirm the vulnerability. For example, combine static analysis with dynamic testing or manual code review for a subset of samples. Alternatively, a combination of several vulnerability detection methods can be applied to lower false positives. Discuss the potential impact of false positives on the results.
- Additional evaluation could enhance the analysis, including testing out-of-distribution code samples (see Questions) and assessing robustness across various prompts.
  - For example, if the user specify "generating secure code" in the prompt, would the vulnerability ratio decrease?
- Presentation and readability could be improved
  - Define CWE upon its first use (Line 89).
  - The problem set-up paragraphs (Section 1 - 3) are lengthy and contain some redundant information (e.g., lines 68-70 and 167-169).
  - As INSEC only modifies the prefix for best performance, it should be made clear in the threat model description (line 159-160): 'INSEC modifies only the prefix p, leaving the suffix s unchanged.'
  - Consider incorporating the details and breakdown of the five programming languages and 16 CWEs into the main text for clarity.

[1] Qinkai Zheng et al. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’23, pp. 5673–5684, New York, NY, USA, 2023. 

[2] Hong Jin Kang, Khai Loong Aw, and David Lo. Detecting false alarms from automatic static analysis tools: How far are we? In Proceedings of the 44th International Conference on Software Engineering, pp. 698–709, 2022.

questions:
1. The attack strings pool and the exact attack string to inject are highly dependent on the training and validation sets (historical data). Would the attack success rate drop significantly if the test code were out-of-distribution, such as when using new API calls? Could randomization, rather than selecting the top string from the validation set, improve generalizability?
2. **Design Choices:** How were the following values chosen, and could different values impact the attack success rate?
   - "During evaluation, we generate 40 completions for each task and compute func rate@1 and func rate@10" (Line 302)
   - "generate 100 completion samples for each task" (Line 316)
3. Does the performance on vulnerability generation and functional correctness vary across different programming languages and CWEs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712322489

thread_type:
official_review

<thread_object>
creator: reviewer_xra8

<content>
summary:
The paper proposes a practical attack on code completion tools like Github Copilot. The authors insist (rightly so) that other attacks on code completion tools assume access to the training dataset, which is rather complicated to get, instead the paper treats these models as black boxes, and assumes access to the API (e.g., via malicious IDE extensions). The key idea is to simply insert a comment in the prompt, throwing the model off and getting it to generate vulnerable code. The paper describes a method to come up with such comments and extensively evaluates their method on different code completion tools/models. For evaluation, the authors come up with their own datasets inspired by HumanEval in order to measure both correctness and vulnerability (detected using CodeQL). Furthermore, authors perform a good number of ablations to evaluate the effectiveness of their design/heuristic choices.

strengths:
1. The paper proposes a practically realizable attack scenario on code completion engines.
2. The proposed method for generating attacks is very cheap and simple to replicate.
3. Authors evaluate their proposal fairly well by performing several ablations.
4. Authors come up with a new dataset, which will be useful for the research community (when open sourced)

weaknesses:
1. The authors only evaluate on their own dataset. Since the problem of getting code LMs to output vulnerable code is a well studied problem, surely there are existing datasets that can complement this evaluation. The dataset prepared here is based on HumanEval and therefore is not necessarily aligned with real coding scenarios (e.g., HumanEval problems are short). Whereas in reality, users work with one or more files and the extension uses RAG to stuff context from multiple places into the prompt. The authors can consider basing their evaluations based on other benchmarks that are closer to the real-world setting of code completion, such as repocoder (https://arxiv.org/abs/2303.12570) and repobench (https://arxiv.org/abs/2306.03091). While that would definitely make functional-correctness evaluation challenging, it would significantly strengthen the paper.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1732573055843

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Paper Reviews

comment:
We would like to thank the reviewers for their insightful and constructive comments. Overall we are glad that reviewers appreciated the scope of our evaluation and ablations (Ue9Z, Jqc8, 384H, Xra8), the effectiveness of our method (384H, Jqc8), and the novelty of our work (Ue9Z, Xra8).

To clarify the reviewers’ remaining questions, comments, and concerns, we have responded in individual comments.

Further, we uploaded a revised version of our paper, incorporating the requested changes and our clarifications, with changes highlighted in blue color. Overall, we made the following changes to the paper:

- Added Discussion of possible defenses (Line 485-486, Appendix E)  
- Discussion of Yang et al. [1] in Related Work (Line 523-527)  
- Experimental Results regarding generalization of INSEC between models and the effect of the choice of mutation function (Appendix C)
- Details regarding the manual confirmation of CodeQLs accuracy (Line 313, Appendix A)
- Introduction of prior work, which used genetic algorithms on LLM inputs (Line 197-198)
- Details regarding the choice of evaluation parameters (Line 303) and the construction of the vulnerability dataset (Line 311-312)
- Clarification that random initialization is used in the “Opt only” ablation (Line 398-401)
- Explicit introduction of Common Weakness Enumeration (Line 88)
- Several rephrasings to enhance clarity and reduce wordiness (Line 68-70, 84, 168-170, 186)


[1] Z Yang et al., Natural Attack for Pre-trained Models of Code. ICSE 2022.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes an attack against code models. However, the primary concerns revolve around the threat model, as well as the tools and datasets used for evaluation. These issues make the attack less convincing in real-world scenarios. As a security paper, the AC recommends that the authors revisit the threat model and present more compelling results by addressing the problem within a real-world context.