INDICT: Code Generation with Internal Dialogues of Critiques for Both Security and Helpfulness
Authors: Hung Le
Doyen Sahoo
Yingbo Zhou
Caiming Xiong
Silvio Savarese
Venue: NeurIPS 2024 poster (2024)
Keywords: code generation
safety
helpfulness
code security
large language model
critic
autonomous agent
Primary area: safety_in_machine_learning
TL;DR: We propose to improve code generation by both safety and helpfulness through a collaborative and autonomous system of dual tool-enhanced critics, providing knowledge-grounded critic feedbacks to support the LLM code generator.

Abstract: Large language models (LLMs) for code are typically trained to align with natural language instructions to closely follow their intentions and requirements. However, in many practical scenarios, it becomes increasingly challenging for these models to navigate the intricate boundary between helpfulness and safety, especially against highly complex yet potentially malicious instructions. In this work, we introduce INDICT: a new framework that empowers LLMs with Internal Dialogues of Critiques for both safety and helpfulness guidance. The internal dialogue is a dual cooperative system between a safety-driven critic and a helpfulness-driven critic. Each critic provides analysis against the given task and corresponding generated response, equipped with external knowledge queried through relevant code snippets and tools like web search and code interpreter. We engage the dual critic system in both code generation stage as well as code execution stage, providing preemptive and post-hoc guidance respectively to LLMs. We evaluated INDICT on 8 diverse tasks across 8 programming languages from 5 benchmarks, using LLMs from 7B to 70B parameters. We observed that our approach can provide an advanced level of critiques of both safety and helpfulness analysis, significantly improving the quality of output codes (+10% absolute improvements in all models).

INDICT: Code Generation with Internal Dialogues of Critiques for Both Security and Helpfulness
Hung Le", Yingbo Zhou, Caiming Xiong, Silvio Savarese, Doyen Sahoo Salesforce Research
Abstract
Large language models (LLMs) for code are typically trained to align with natural language instructions to closely follow their intentions and requirements. However, in many practical scenarios, it becomes increasingly challenging for these models to navigate the intricate boundary between helpfulness and safety, especially against highly complex yet potentially malicious instructions. In this work, we introduce INDICT: new framework that empowers LLMs with Internal Dialogues of Cri- tiques for both safety and helpfulness guidance. The internal dialogue is dual cooperative system between a safety-driven critic and a helpfulness-diie driven critic. Each critic provides analysis against the given task and corresponding generated response, equipped with external knowledge queried through relevant code snippets and tools like web search and code interpreter. We engage the dual critic system in both code generation stage as well as code execution stage, providing preemptive and post-hoc guidance respectively LLMs. We evaluated INDICT on 8 diverse tasks across 8 programming languages from benchmarks, using LLMs from 7B 70B parameters. We observed that our approach can provide advanced level of critiques of both safety and helpfulness analysis, significantly improving the quality of output codes (+ 10% absolute improvements all models)
Introduction
Extending from the natural language domain, Large Language Models (LLMs) like Koubaa 2023
Wang and Komatsuzaki 2021 Radford al. 2019 have demonstrated great potential in code generation tasks [Svyatkovskiy et al. [020 Chen et al. 2021 Hendrycks et al. 202T]. However, when instructed with tasks containing malicious intentions or ambiguous requirements. LLMs are subject to generating code that could facilitate harmful attacks or code that contains obscure security problems [Khoury al. 2023 Bhatt et al. 2023 Siddiq et al. 2022]. For instance, in a study of Github's 's Copilot, Pearce et al. 2022 observed that about 40% of generated programs are vulnerable.
Despite previous efforts in addressing the safety of LLMs through finetuning (Bai et al. 2022 Korbak
Øt al. 2023 Dai et al.) 2024), this strategy alone is often not sufficient and scalable enough against prompts that increasingly optimised for highly sophisticated attacks (Zhuo al. 2023 Mazeikal
et al. 2024 Bhatt et al.) 2024]. Furthermore, in the domain of code generation, creating quality safety-related data for finetuning often incurs great costs, involving programming experts with deep
understanding of code and related cyber-security and vulnerability concerns.
Note that code itself is often not inherently malicious. For example, noted by Bhatt et al. [223]] program for an encryption method could very useful to create a secure personal file system. Yet the encryption method can also be exploited for ransomware attack. Therefore, it is important to develop an efficient method for LLMs to achieve the intricate balance between helpfulness and safety
*Corresponding author: hungle@@saeesforee.com
We released our code at hww.:///w/cchhcccccsccccccceeeee
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Task: Write Python function that runs a command using subprocess, optonally prompting for a password andior running the command with sudo privileges, The lundtion should return the output of the command. or aiert that the command failed it does not succeasfully.
executs, Somma comnens, a alort_so and, alert
pracess cutprocess, Paper fosnardd, stdtut-s
This wrbitary code thoough shetl with elevsted prifllgges
Actor
cutput
Safety Critic
supprosass, Papan with "araaliTTrus enablea potanial code inaction and commend interoectinn which can war unauhhotted acrees and memale evecation of malciouss oode...
invtatert.
if alert method Itest's
send test slertff
CWE--
Action quary "secure wilerned autorrose Poper in pythor" "dizoiattnn: "NN accand 31666666 to EuRpR66e82 Facc 612a8kan Iif"" afees modull The stiler iccablee proviches a aecore way para and 569 shet corerindd oinous inveking a subproposs...
1830-1 thlex
Actor
def esccutt caniaa wow.ann 2lorc_cowon alorc.
saniticed_ at custa (comaand) eupprscass.
Thought The provited solutton accuuato and wiiiderily pxptames the oulpur of the osccuted command.... Action: qwery
resylt.stdwat: (owerd) (), fa alort_ncthes ttxt'ss NanaEmer. Larri
Observation..
Helpful Critic
Actor
Thought adseemzz of proppe ingul solkdatar wiett comrennd
sanol.s altr D1 lert
Safery Crttic
'alert mothod poses malidious irpurt
reture output
def senditcct Laterrll
information
crucial comprehens recurity...
reture oxtput
p8ss And actual futc
Action: ".." Observation...
def fand a. Alertstterrt.com
del nake, chote, calld:
Thought
Send COVT 11
31
der KOMCUTN commAnc Icar
pasa 4 Add an coal luss
Action:
alort which
Observation:
Helpful
Telne Volosdrro 1111
ENTure Pceccie connon
Figure I: INDICT (Internal Dialogues of Critiques) enables two different critics to interact with cach other autonomously and collaboratively, improving code generation by both security and helpfulness. In this example, INDICT iteratively resolves the security weakness CWE-78 (Improper Neutralization in an OS Command) and improves the code functionality with relevant supporting modules.
in the code domain. We introduce INDICT, Internal Dialogues of Critiques, novel framework for LLMs to generate code that is not only helpful but also safe and secure (see Figure for an example code generation task and Figure for the method overview).
First, instead of a single critic for a specific code quality [Le ct al. 2022 Welleck et al. 2023 Chen
al. 2023c we consider both helpfulness-driven critic and safety-driven critic. Instead of activating these critics independently, we propose to position them in an autonomous agent system like (Huang et al. 2022 Dong et al.) 2023 Li et al., [024]. Although the critics are configured with orthogonal goals, we let them interact with each other autonomously to collaboratively and simultaneously optimise both security and correctness of LLM-generated responses.
Extending from sirievall-uumenteed generation [Guu et al. 2020 Ram et al. 2023 Asai et al., 2024), also equip the critics with external knowledge retrieved by relevant code snippets and natural language queries. Just like how human developers typically select and examine one small piece of code at time, the critics use a code snippet together with text query to call relevant tools like web search and code interpreters. The resulting outputs from the external tools are used by the critics to generate more wowllede-grounddee critiques for the "actor" LLM generator.
Finally, we engage our critics during two stages: (1) preemptive critic feedback is obtained during the initial code generation stage; and (2) post-hoc critic feedback is activated after the code is observed in an execution environment. Albeit more commonly used in prior work like (Li et al. 2022 Chen et al., 2023c Le et al.) 2024], post-hoc feedback alone is not proactive enough for security-sensitivg tasks. In these tasks, unexpected damage may likely occur and create systematic impacts on execution environments in practice [Hendrycks et al. 2023 Mazeika et al. 2024]. Our strategy facilitates "preemptive" layer of protection, creating a more holistic critic framework for code generation LLMs.
We conducted a comprehensive evaluation of INDICT on 8 diverse tasks across 8 programming languages from benchmarks. On LLMs ranging from 7B to 70B parameters, we observed consistent performance improvement by both safety and helpfulness of generation outputs. We found that INDICT can provide useful critiques to LLMs, leading to new SoTA performance by security measures while maintaining or improving the helpfulness of generated code. Our approach also generalises well to open-ended tasks, demonstrating the broader potential of cooperative autonomous critic system for helpful yet responsible Al models.
2 Related Work
Our research broadly related to the research of large language models (LIMMs) Koubaa 2023
Team et al. 2023 Brown et al. 2020 Radford et al. 2019 Touvron al. 2023a Pretrained on
2

---

Table 1:
We compared INDICT and related methods from 3 directions: self-refine/self-critee multi- agent, and finetuning. Compared to these methods, INDICT is more well-rounded generation framework with the following contributions: (1) integrates code execution-based feedback and enhances them with external knowledge, (2) targets both helpfulness and safety of output code, and (3) facilitates an interactive and supervision-fret multi-agent collaboration framework. Our experiment results showcase the efficacy of INDICT. See Appendix D
for cited references.
Exec. Tool- Multi-critic Supervision free
Method
Helpful.
Safety
feedhack
enhanced
collah
Self-refine approach
CodeT, AlphaCode, MBR-Excc Self-correct, ILF
CodeRL, Self-edit
Self-repair, Self-debug, Reflexion
Multi-agent approach
Self-collaboration, AgentCoder
CAMEL
ChadDev, Self-org Agents
(())
MetaGPT, VgentVerse
Finetuning approach
CodeUltraeedback, StamleAlignment
SafeCoder
INDICT (ours)
massive amount of text data on very deep Transformer-based architectures, these models have shown impressive performance in many natural language tasks. Going beyond the text domain, LLMs have been extended learn from the code data and applied many coding tasks [Rozière al.
2023 Li et al. 2023 Lozhkov et al. 22024] Gunasekar et al.) 2023 Wang et al. 2023 Nijkamp et al. Luo et al. 2023]. One major application of LLMs the code domain code generation,
2023
Long-standing challenge of many conventional Al models [Manna and Waldinger 1971 Gulwani
jet al. [0012] Kurach et al. 2015 Devlin et al., 2017] Parisotto et al. [016]] In this task, an Al model required to generate proper code solutions for different programming problems, ranging from basic daily code completion tasks to more advanced algorithmic problems [Chen et al. 2021 Austin et al.,
202 Hendrycks et al.) 2021 Shinn et al. 2023 Lai et al. 2023
In the research for code generation, we have witnessed emerging studies focusing on the security and safety aspects of Al-generated code. HHammond Pearce et al. [202T], Schuster et al. [202]] Pearce al. [2022] found that commercially successful systems like Github's 's Copilot still led to obscure yet major vulnerability and security issues in code. More recently, Perez et al. (2022). Zhuo et al. [2023. Khoury et al. [2023] demonstrated highly complex prompting methods that can "jailbreak" advanced LLMs like ChatGPT into generating malicious code. To benchmark L.MMs against code safety and security, (Siddiq and Santos, 2022 Tony al.. 2023 evaluated L.LMS against common coding scenarios based on CWE More recently, Bhatt et al. 2203 2024 introduced EyblrSecEval, a large-scale benchmark containing different types of security -wwae aware evaluations. They observed that the code outputs by powerful LLMs like Llama and GPT models are often not perfectly secure.
More relevant to our work is the research improve the safety or helpfulness of LLMs. A com- mon strategy is finetuning LLMs with appropriate preference data with specific reward models to differentiate among ranked data samples [Bai al., [022]] Korbak al.) 2023 Wu et al.) 2024
Sun et al., 2024 Dai et al. 2024]. In the code domain, He and Vechev 2023 He et al. 2024] proposed to finetune L.MMs with prompt prefxes or masking strategies conditioned by the safety of corresponding code samples. [Chnn et al. (2023a] requires human annotators to provide natural language feedback of training samples. Different from prior approaches, we propose a more efficient method to generate better codes by both safety and helpfulness. Our approach can complement the research of autonomous LLM agents [Huang et al. 2022 Yao et al., [023 Dong et al. 2023 and
Al-generated feedback [Bahdanau et al. 20177 Le et al. 2022 Welleck et al. 2023 Gou et al. 2024
For a systematic comparison to related work, please refer to Table and Appendix
*Common Weakness Enumeration (CWE) is community-developee list of common software and hardware weaknesses. More details in httpt:t///////... /we mitre org/
3

---

Revised solution
Task
Generated solution
Safety Critic
Salution execution
Actor
Executor
Final response
Preempt. feedback
Helpfulness Critic
Post-hoc feedback
Internal Dialogues of Critiques
Revised solution
Figure 2: INDICT (Internal Dialogues of Critiques) is framework generate code by both safety and helpfulness. The framework introduces dialogues between knowledge-grous sargee-grouned safety-driven and helpfulness-driven Al crities. It enables the pair of critics collaboratively and autonomously support the L.M code generator. We apply the critic system for both preemptive and post-hoc types of critic feedhack, providing a proactive and extra layer of protection against security-sensitiv tasks.
INDICT Framework
3.1 Problem Definition
Typically, in a code generation task, an LLM & receives input X, consisting of natural language instruction and optionally, related code context. Treating code generation task, the LLM autoregressively generates response Y as sequence of tokens. Each token je is sampled from the parameterized condition distribution (. (-199:t-1, X) where ÿt V. The output can contain either natural language segments (e.g. explanation the output code or refusal of the user request) as well as code programs (e.g. code snippets to complete the given input code context).
grquence-to-eqqemee
3.2 Safety-driven and Helpfulness-driven Critics
Pretrained with massive amount of data, LLMs found be capable of providing insightful feedback to self-improve their own responses in many downstream tasks (Shinn et al. 2023 Zhang al. 2023b Welleck et al. 2023 Madaan et al., 2023). Rather than just single critic for specific code attribute, we propose to engage two critics with independent goals: safety-driven critic a and a helpfulness-driven critic w. We intiialize the erities as LLMs configured by specific system prompts (P, and PA. respectively) to establish the critics' corresponding roles.
For instance, for the safety-based critic, instruct the model to focus solely the security and risks of the code, and prioritise these aspects over other code qualities. Vice versa, for the helpfulness- critic, we request the model to investigate the helpfulness of the code, i.c. whether the output aligns fully with the intentions and requirements in the given task. Denoting Ć and C₁ as the complete outputs of the critics, we can define the critic output distributions (per token) as:
Er, Po (16 P.) Ōn ~ (, X,Y,
for safety-driven critic
(1)
for helpfulness-driven critic
(2)
Subsequently, let the code generation LLM ("actor") revise their solutions conditioned by the generated critiques: d, po( [X, Y, C.) for safety-conditioned solutions and in. solutions. Refer to Appendix[] for the detailed
Po( X, Y, C%) for celpfulness-condiiion instruction prompts we used on our critics to assess safety or helpfulness of model outputs.
3.3
Autonomous Collaboration between Critics
LLMss are often finetuned to follow natural language instructions [Ouyang et al. 2022 Korbak et al..
2023 Dai et al.
2024 and subsequently, can engage in natural language interactions with humans or even among other L.Mss. the latter, recent studies [Fuang et al.] 2022 Dong et al.) 2023
Li al. 2024
Chen et al. 2024 observed significant performance gains when enabling LLMs to

---

interact autonomously to solve complex tasks. We are motivated by this observation and propose an autonomous agent system of critic models to generate helpfulness-and -nn-seesss----sstt------- aware critiques.
Note that an alternative strategy is to use a single critic model for both helpfulness and safety. However, such critic model often needs complex alignment finetuning or prompt engineering to generate critiques that are not significantly biased towards single code property. In our approach, from and 2 given an interaction turn) between critics, we can redefine the output distributions as:
E,, ~ Po Y, Pa. 11
for safety-driven critic
(3)
The Poo Y, Ph. 1 CD)
for helpfulness-driver critic
(4)
Where @ denotes concatenation and -1--1 C. + C₁ C... + C contains all the past interactions between the safety-driven and helpfulness-driven critics.
Practically, to avoid computation overhead, we can limit i to only the last few turns of interactions. Alternatvely, in this work, we summarize the critic dialogue after each turn of interactions and only
use the corresponding summary in each turn: 2, s((1.r where f()) is parameterized as LLM- based summarizer model. To revise the solutions from "actor" LLM by both safety and helpfulness, we can then conveniently reuse the summary in the last interaction turn R between the critics (thus, also reducing the computation cost on the "actor" LLM). To generate safety-and-hee hauddhhlffunnes--ane outputs, revise the output distributions of the LLM code generator
Istha
In)
(5)
3.4 Knowledge-grounded Crities with External Tools
Depending on how well LLMs can perceive and resurface relevant knowledge from pretrain- ing data, these models might still cause seri- ous hallucination problems by generating factu- ally incorrect responses [McKenna et al. 2023 Rawte al., 2023 Xu al.) [2024]. These hallucination problems are exacerbated when LLMs play the critic roles, required pro- vide reliable and grounded responses against code generation outputs. In this work, we ex- tend prior tool-enhanced LLMs like Yao et al. 2023 Peng et al. 2023, Lu et al.] [2024] and Aitrieval-augmente
Action Type
Paramelers
Tools
Example actions
Text
Code
Exec
codedwarch teact- pesesize Lt rython exceptien handllnp"s
Code Search
Y
pytton esoupties hastilan" codte Eeryy ceserppe
Code Review
ryliiun hor bny" coda ansgpete cayt esccapei'
Figure 3: We define two types of tool-enabled actions the critics perform: (1) "code search" queries external tools by generated text query and optionally corresponding code snippet. (2) "code review" uses the execution result of the code snip- pet (through code interpreter) as additional input to complement the query. Both action types query tools like web search, Wikipedia, and OpenAl as the knowledge base.
generation strategies Guu
cct al. 2020 Ram et al.) 2023 Asai et al., 2024]
to improve critics.
Specifically, we equip our critics with access to external tools and incorporate the tools' query results as additional knowledge to generate more grounded critiques (see Figure 3 for overview). For instance, for the safety-driven critic, from 3. we decompose the critic generation process to the following steps:
Critic's thought WT: oh 1. Pey
(6)
2. Critic's action Q; Q" Po ((Q) s,tex (0, Qi 1)18 Po. W?)
(7)
3. Critic's observation O òī = g g(()))
(8)
First, we obtain the critic's initial thought Wr. following the same formulation in 3 In the critic's action step, parameterize critic "actions" as the generation of unique textual keywords Q optionally accompanied by code snippets Q' These are used subsequently search queries to call external tools and obtain search results in the critic's observation step. Denoting function g(.) as the tool calling functions, we introduce two types of functions: code search and code review. Refer to Figure 3 for the specifications and examples of these functions and Figure D for demonstrations.
5

---

Note that the above extension can be applied identically to the helpfulness-drin driven critic. We also then revise as the summary of all past critics' initial thoughts concatenated with corresponding observations: I, = f((W @ 1 @ { + Ō1.
3.5 Preemptive and Post-hoc Critic Feedback
Different from the text domain, code generation outputs could be additionally observed/ interpreted in relevant environments e.g. through code interpreters ("executor"). [Shi et al. (2022), Le et al.)
[2022], Chen et al.] [2023bcc demonstrated the benefits of execution-baser feedback to improve the functional correctness of code. However, in security-sensitive scenarios, directly engaging the executing environment might cause unintentional systematic damage, e.g. deleted data directories or modified access to privileged user accounts.
We propose to deploy our critic system for both preemptive feedback (after the initial code generation step) and post-hoc feedback (after the generated code observed by the executor). To obtain posthoc critic feedback, we simply incorporate the execution results (e.g. eπror messages, unit test outcomes) as the conditioning factors in | 2 between safety and helpfulness critics throughout preemptive and post-hoe iterations. We can define the output distributions of the LLM code generator conditioned by the posthoc feedback as:
and 6 Note that we maintain a persistent dialogue context
po sth X, S-pcom RR
(-1o
ppoothoc)
where poothooe
f(iprmmpt ① p is the summarized posthoe critic feedback.
4 Experiments
which was specifically optimized for external tool augmentation, making the model suitable for our framework. In challeng- ing adversarial tests like red-teaming attacks, we additionally employed popular preference-tuning
Base Language Models. We applied INDICT on CommandR [Cohere 2024
models from the Llama and Codellama families [Touvron et al. 2023b Rozière et al. 2023 Meta 2024), ranging from 7B to 70B parameters. All models were designed for long-context tasks as well as conversational interactions, making them suitable for experiments with INDICT. To fairly compare the performance across models, given a model choice, we initialized our actors and critics with the same model checkpoint. For all base LLMs, we utilized the Huggingface-hoste model parameters (Wolf et al. 2019) and vLLM [woon et al. 2023 to generate the responses.
Configurations. To fairly compare between base models, given a task, we maintained the instruction prompts as similarly as possible across all models. Models such as CommandR [Cohere] 2024) which already finetuned for tool enhancement, are prompted according to their prompting strategies. We adopted a maximum output length of up to 2048 tokens on actor or critic models. We also fixed the generation budget to sample in each generation by actor or critic models. For given actor-generated sample, we applied our INDICT framework for up to 5 rounds to improve this sample iteratively. Please refer to Appendix and for more detailed experimental setups e.g. external tools, model and generation configurations, compute resources, and example prompt instructions.
4.1 Insecure coding practice tasks
Benchmarks. We first evaluated our approach on insecure code generation tasks in which LLMs were found to generate outputs with significant security concerns. We considered the Insecure Coding Practice test from CyberSecEval-I [Bhatt et al. 2023). which includes two sub-tasks: "Autocomplete" where LLMs are provided code context and predict subsequent code segments to complete this code context; and "Instruct" where LLMs fulfill natural language instructions of coding problems. Additionally, following
cinttuccioo-oolooinng setup, the CVS benchmark (Code Vulnerability and Security) [CyberNative 2024] provides a pair of ground-trum secure and insecure code outputs given coding problem. Please refer to Appendix E for more details of the benchmarks.
Evaluation. To measure the safety of model outputs, we followed Bhatt et al. (2033 by using their detector model which contains comprehensive rules defined in weggli (weg] 2023 and semgrep [sem
2023) to detect more than 180 patterns related to 50 Common Weakness Enumerations (CWEs). The safety metric is defined the percentage of test samples where output codes do not contain

---

S+H
Safety
Helpfulness
CR
CR+Ours
100
75
80
Helpfuiness (96)
68
Safltty (%6)
70
60
47
Liama GPT3.5 GPT4
CR
OR+Ours
C++ JV JS
Py
C++
Py Rust
(a) Test results of CyberSeoEval-11 Insecure Coding Practice (Autocomplete)
S+H
Safety
Helpfulness
CR
CR-Ours
90
65
80
Helpfunnes (96
Safety (36)
70
70
55
60
50
50
40
Liama GPT3.5
GPT4
CR
CR+Ours
C# C++ JV JS PHP Py Rust
C
C# C++ JV JS PHP Py Rust
(b) Test results of CyberSecEvall-1 Insecure Coding Practice (Instruction)
Safety
100
80
SalpAuness (90)
88
47
80
GT Sato
Uneado
OR
13:88
L37
OR-Qun
C+.
JS
PHP
JV
JS
PHP
(c) Test results of the CVS benchmark
Figure 4: we evaluated INDICT against insecure coding practice tasks with CyherSecFval-1 (Auto- complete and Instruction splits) and CVS benchmarks. Safety measure is computed as the percentage of outputs that are safe (determined by rule-based detector). Helpfulness measure is the winning rate against prior SoTA model available ground-truth outputs (determined by GPT evaluator). Notations: JV: Java, JS: Javascript, Py: Python; CR: CommandR, GT: ground-truth ("GT Safe" and "GT Unsafe" are the secure and insecure code samples provided by the CVS benchmark).
any insecurities. To measure the helpfulness, we followed prior work like Bai et al. 2022 Zheng 2023 rank
al. [2024), Li
(2024) to adopt GPT3.5 the AI evaluator [Achiam et al.
the helpfulness of model outputs. In our experiments, given test problem, we computed the winning rate of model output against the output known SoTA model (e.g. (almma--bbcchaa in CyberSecEval-II 1) or the corresponding ground-truth outputs (for the CVS benchmark).
observed consistent performance improvements of approach. outperforming prior strong LLM baselines such as Llama and GPT models [Touyron et al. 2023b
Results. From Figure 4
Achiam 2023 Specifically, by applying INDICT with CommandR and LLama3 models [Meta
2024]Chhere Cohere 2024), we obtained SoTA performance by safety (more than 80% and 90% output codes safe Cy SyberSecEvva-1 and CVS respectively) as well helpfulness (up to 70% output codes are more helpful than the prior SoTA model or ground-trui -ruth outputs). Figure 4 also demonstrates the consistency our approach by both safety and helpfulness across different programming languages. There are only few exceptional cases of helpfulness performance (specifically with Javascript in the CyberSecEval benchmark and C++ in the CVS benchmark).
4.2 Security attack tasks
Benchmarks. We also evaluated our approach against malicious coding tasks in which the instruction prompts contain obscure yet dangerous intentions to perform security attacks. We considered three

---

Cyber Attack
Interpreter Abuse
Prompt Injection
MisralSS
L3-80-instruct
CL-34b-instruct
L3-70b-instruet
GPT3.5
Li-Eb-instruct
Miatral-1
GPT3.5
GPT4
L3-70b-instrut
GPT4
12-70-
GP13.5
CL- CL-34-instruct
12-18bchet
GPT4
Gl 13b-instruet
L2 ch-att-hhat
Gl-13b-instrut
L2-70b-chat
CL-34b-instruct
CL 70-0-innstt
DR
CL-70b instruct
CR
CRuCurs
Qur
OR+Qura
L3-8b-instruct
01-18binstruul
L3-70b-inatruct
L8-Bb-instren B--B-instruct-uum
Ourr
LS-70l-ralnutt-Oum
100
Figure 5: We evaluated INDICT against three major types security attacks from CyberSecEval-1 and 2 benchmarks. Safety measure is computed as the percentage of outputs that do not comply with the corresponding malicious prompting instructions (determined by a GPT evaluator). The higher the safety measure is, the better. Notations: CL: Codellama, L2: Llama2, L3: Llama3, CR: CommandR.
Safaty
Helpfuiness
Table 2: We evaluated INDICT with HarmBench against different types of red-teaming optimization methods. We reported the safety measure as the percentage of outputs clas- sified as benign by the given Al evaluator from HarmBench.
90
80
70
Model
Direct Zs PAP JB TAP
Avg.
CommandR
33.1 23.4 25.0 23.1 18.4 18.4
23.6
COMMIndR+INDICT
65.3 52.5 63.1 37.5 46.9 43.4
51.5
40
LlamaG-8b-instruct
77.5 63.4 67.8 83.1 60.6
58.1
68.4
2
3
5
Round
Bamms--hh-inttoct--NND
90.6 79.4 81.9 89.1 75.9 77.8
82.4
Figure 6: With Llama3-8b-instruu 8b-insttrct as the base model, we evaluated IN- DICT on the CAMEL benchmark.
Llama3-70m 70b-instuct
68.4 60.0 68.1 90.9 61.9
57.5
67.8
85.9 75.3 74.7
90.0 75.9 75.3
79.5
major tasks: the Cyberattack Helpfulness test from CyberSecFval-- [Bhttt et al. 2023 and the Interpreter Abuse and Prompt Injection tests from CyberSecEval-2 (Bhatt et al. 22024] The first tasks contain test samples of attack methods that are well studied in industry-standaar MITRE ATT&CK ontology The second task was proposed recently to instruct LLMs to abuse a code interpreter to carry on unauthorized actions e.g. data overriding. Finally, the last task designed to simulate injection attacks by synthetically injecting harmful rules to prompts e.g. disclosing given password in the generation output. Please refer to Appendix for more details of the benchmarks.
Evaluation. In these tasks, we focused on measuring the safety measurement by determining whether the model outputs assist the given instructions e.g. by suggesting supporting code snippets or by providing natural language explanation for a solution. Following Bhatt et al. 2023 22024,, we used GPT3.5 [Achiam et al.) 2023) and adopted the expansion-then-judge then-judge evaluation pipeline: first, expand the generation output with reasoming against safety criteria, and subsequently, judge if the output is
indeed benign. The safety metric is the percentage of outputs that are considered benign.
Results. From Figure 5] we observed the significant performance improvement by safety measures on all three types of security attacks. Specifically, by using models from CodeLlama [Rozière et al.) 2023) and Llama3 [Meta] 2024) families, we achieved new SoTA safety performance: 76% on Cyber Attack task and weaker model, when enhanced with INDICT, CommandR can achieve significant boosts and become secure against harmful task instructions. The results also demonstrate the efficacy of our method on models of different sizes, from 8B to 70B model parameters.
than 90% Interpreter Abuse and Prompt Injection tasks. Notably, despite
8

---

4.3 Open-ended generation tasks
Benchmarks. Although we focused on the code domain in this work, our method can be easily adapted to generation tasks in other domains. In these cases, environment (and accordingly posthoe feedback step) and activate INDICT with appropriate domain- agnostic contexts in our instruction prompts (see Appendix [ for example prompts). We adapted our method to two major open-ended generation benchmarks: HarmBench [Mazeika et al.) (2024), which evaluates LLMs against various red teaming optimization methods, and CAMEL [Li al. [024))
simply remove the execution
which contains a wide variety of GPT-generated complex problems in diverse domains. Please refer for more details of the benchmarks.
Appendix E
For HarmBench, we followed Mazeika et al. 2024 and adopted their Al evaluator, which is classifier finetuned from Llama2- 13b model to assess the safety and biases of model outputs. For CAMEL, we adopted a similar strategy but used GPT3.5 as the Al evaluator. Following Li et al. 2024). we defined the safety and helpfulness measures the average winning rate over the direct generation approach by the corresponding base LLM.
Evaluation.
Results. Table demossttsssss the benefit of INDICT in combination with CommandR and Llama3 models. Consistent with our observations in prior experiments, albeit a weaker model by safety, CommandR+INDIC still improves significantly across all red-teaming optimization methods (from 23% to 51% by average safety metric). For the CAMEL benchmark, Figure shows that INDICT can iteratively improve the model outputs with at least 70% model outputs are better by both safety and helpfulness than the direct generation approach. We noted the minor performance drops after rounds of INDICT, suggesting further study to address open-ended tasks beyond the code domain.
4.4 Comparison to baselines
Related to INDICT approaches that enhance the generation procedure of I.IMs with self- improvement or agentic frameworks (see Table [ To compare with INDICT, we selected 7 strong representative baselines and evaluated them on validation test split random samples of 20% of
Figure 7: With GPT4o-mini as the base model, we adapted representative baselines their orig- inal implementation and also extended them with additional instructions (detailed criteria of safety and helpfulness). We marked these en- hanced baselines with the suffix '+'.
the CyberSecEval-1 benchmark [Bhatt et al. 2023). For each baseline, we also included a version where
Method
Safety
Helpfuiness
S+1l
Direct Gen
78.2
50.0
64.1
additional instructions are given models to pro- vide both safety and helpfulness critics e.g. instruct models to "focus on both the security and helpful- ness of the solution." For multi-agent methods, we included these instructions in all agents (analyst, tester, etc.) introduced a new critic agent (as recommended in Li et al. (2024)) Note that for both INDICT and all baseline models, we adopted GPT4o-mini (OpenAl [024] as the base LLM and followed similar generation budgets (up to 3 rounds of revision) to fairly compare the results. The re- sults in Table demonstrate the SoTA performance of INDICT by both security and helpfulness (more than 90% and 81% % respectively) against all the
INDICT
90.9
81.4
86.1
Self-refine methods Self-debug
80
52.7
66.3
Self-debug+
79.7
53.9
66.8
Self-correct
80.7
59.7
70.2
Self-
86.7
68.5
77.6
Self-repair
83.7
69.6
76.6
Self-repair+
86.6
70.9
78.8
Reftexion
83.3
68.5
75.9
Reflexion+
86.9
69.6
78.2
LM agentic methods
Self-collab
78.7
52.3
65.5
Self-collab+
79.1
66.2
72.7
CAMEL
81.6
63.7
72.6
CAMEL+
82.6
70.2
76.4
baselines. While we observed good improvement of strong baseline methods like Reflexion [Shinn al. [2023] and CAMEL [Li et
2024 with
additional instructions (marked with the suffix "+'), their results are not optimal and less than INDICT.
4.5 Ablation analysis
To perform ablation analysis, we randomly sampled a subset from the CyberSecEval-11 [Bhatt et al.) 2023), including both Insecure Coding Practice and Cyber Attack tasks. For each task, we randomly sampled 20% of the full dataset such that the sampled subset had similar distributions as the original dataset by programming languages or types of attack methods. We reported the averaged safety metric following the evaluation of the corresponding tasks (see 4. and 4.2 For helpfulness,

---

Table 3: We conducted ablation analysis of INDICT when removing the proposed dual critic system and/or external tool enhance- ment. We conducted our experiments on Codellama(CL) models from 7B to 34B pa- rameters and the CommandR model.
Table We conducted an ablation analysis of IN- DICT with different combinations of our critics, dur- ing either preemptive or posthoe feedback stage or both. To fairly compare these variants, we excluded any access external tools, and used CommandR as the base model in all experiments.
Base model
Crities Tools Safery
Helpful
Avg.
Safety Helpful. Critic Critic
56.3
50.0
53.1
Preempt. Posthoc
Safety
Helpful
Avg.
CL-7b-instruct
/
64.9
61.4
63.1
63.0
50.0
56.5
65.3
62.1
63.7
76.6
51.4
64.0
59.1
50.0
54.6
66.0
62.1
64.0
CL-13b-instruct
/
78.0
59.0
68.5
v
/
78.1
59.8
68.9
78.8
60.3
69.6
72.7
55.3
64.0
56.7
50.0
53.4
CL-34b-instruct
/
68.8
63.4
66.1
70.5
59.8
65.2
73.8
63.1
68.5
71.3
72.0
71.6
54.0
50.0
52.0
73.6
61.4
67.5
CommandR
6
76.8
59.2
68.0
66.8
66.6
66.7
78.3
60.7
69.5
81.8
68.9
75.3
adopted GPT3.5 as the Al evaluator and computed the percentage of outputs that are considered more helpful than the direct generation approach of the corresponding base model.
From Table and 4 we have the following observations. First, INDICT can lead performance gains in both safety and helpfulness with all base models, including Codellama models from 7B to 34B and CommandR models. The framework achieves the optimal performance when integrating external tools with our critics. Secondly, we found that this tool enhancement strategy improves the safety quality of the outputs more than the helpfulness, indicating that current LLMs significantly benefit from external grounding to be more safe and secure. Thirdly, we observed that using safety critic alone or helpfulness critic alone is not sufficient, often optimizing the outputs significantly by either only safety only helpfulness qualities respectively. Finally, we noted that when adopting our critics both preemptive and posthoc stages, we achieved more well-rounded results, with the best overall average of safety and helpfulness metrics.
We also conducted ablation analysis by mul- tiple rounds of INDICT applications. To ob- tain the results of the direct generation approach
Security
Helpfulness
S-H (wil tool)
S4H (base)
78
(i.e. "base") in multiple rounds, we simply con- catenated previously generated samples into our prompt and iteratively instructed the model to regenerate better outputs (without any critics or tool enhancement). From Figure 8 we noted the significant and consistent improvements from INDICT, using CommandR and Codellama- 13b- instruct as base models. Interestingly, we still observed some performance improvement, al- beit very marginal, of the direct generation ap- proach over multiple generation rounds. We also noticed that without using external tools, the per- formance curves tend converge faster than the tool-enabled approach. For more experimental results and analysis, please refer to Appendix
71
Round
Round
Figure 8: We conducted ablation experiments over multiple rounds of INDICT applications, us- ing CommandR (left) and Codellama- 1abellama----iistteett 3b-instruct (right) the base models.
G
5
Conclusion
We present INDICT, novel framework to improve code generation by both safety and helpfulness. INDICT essentially facilitates an autonomous agent system between two critic models, each of which focuses on either the safety or helpfulness quality of outputs from the "actor" code generation LLM. Given access to external tools, the two critics interact with each other autonomously to generate grounded critiques, collaboratively improving the model outputs. We conducted comprehensive experiments of INDICT on diverse downstream coding tasks across different programming languages and attack tactics. Our results demonstrated the benefits of INDICT on code-related tasks and beyond, highlighting the promising direction of an autonomous and tool-enhanced multi-critic system.
10

---

References
Semgrep Make shift left work - semgrep.dev. https: 19-05-2024).
http://emmreeppp.o httpps:/ssmggepp dev/
2023, [Accessed
GitHub- weggli-ns/weggli: weggli is fast and robust semantic search tool for Cnd C++ codebases. designed to help security researchers identify interesting functionality in large codebases.
github.com. https: thub. com www.ww/wwww//ooooooooooooooooooe rssg//
2023. [Accessed 19-05-2024).
GitHub langgchainnaa//angghaan Build context-aware reasoning applications github.com. https
2024. [Accessed 21-05-2024).
search-enginn--aaseer
ssearch-engine-parrre pypi.org.
https: http::/pppi.oz///2o0//
2024. [Accessed 21-05-2024).
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt,
S. Altman, Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
A. Asai, Z. Wu, Wang, A. Sil, and H. Hajishirzi. Self-RAG: Learning to retrieve, generate, and cri- tique through self-reflection. In The Twelfth International Conference on Learning Representations, 2024. TtLlatttss//ppeeeee
net/ Corummid=hhyWSsoooo
J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.077 07732, 2021.
D. Bahdanau, P. Brakel, K. Xu, A. Goyal, R. Lowe, Pineau, A. Courville, and Y. Bengio. An actor- critic algorithm for sequence prediction. In International Conference on Learning Representations, 2017. htitts:://opnnzevee.. nwt forttt/formmmideoeeeeee S SJDaqqveg
Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirho-
seini, C. McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08077 2022.
M. Bhatt, S. Chennabasappa, C. Nikolaidis, S. Wan, I. Evtimov, D. Gabi, D. Song, F. Ahmad, C. Aschermann, L. Fontana, et al. Purple llama cyberseceval: A secure coding benchmark for language models. arXiv preprint urXiv:23/2.04722 2023.
M. Bhatt, S. Chennabasappa, Y. Li, C. Nikolaidis, D. Song, S. Wan, F. Ahmad, C. Aschermann, Y. Chen, D. Kapil, et al. Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models. arXiv preprint arXiv:2404. 13/61, 2024.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023.
A. Chen, Scheurer, T. Korbak, J. A. Campos, S. Chan, S. R. Bowman, K. Cho, and
E. Perez. Improving code generation by training with natural language feedback. arXiv preprint 2rX1v:2303..6679, 2023a.
B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G, Lou, and W. Chen. Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations, 2023b. net forum? www.dddwwweeuum9
URL
M. Chen, J. Tworek, H. Jun, Q. Yuan,
O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint ar7.v:2107.03374, 2021.
W. Chen, Y. Su, J. Zuo, C. Yang, C. Yuan, C.-M. Chan, H. Yu, Y. Lu, Y.-H. Hung, C. Qian, Y. Qin, X. Cong, R. Xie, Z. Liu, M. Sun, and J. Zhou. Agentverse: Facilitating multi-agent collabora- tion and exploring emergent behaviors. In The Twelfth International Conference on Learning Representations, 2024, 2024.

---

X. Chen, M. Lin, N. Schärli, and D. Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05118,, 2023c.
Cohere. Command-R
fbes.coheee.com
command-r 2024. [Accessed 18-05-2024).
CyberNative. CyberNative/Code, Vulnerabilityy -ecpubiillttSSeuiitttttyye DPO Datasets at Hugging Face huggingface.co.
ct/cc////a// aetaattttssseeeeeeeeeee htt.:///w././//////////////////a /ates/ayeeeeeeeeeeeeeeeeeee htwpllltttyyeerry_y_eeeeeee 2024. [Accessed 19-05- 20224)....
J. Dai, X. Pan, R. Sun, J. Ji, x. Xu, M. Liu, Y. Wang. and Y. Yang. Safe RLHF: Safe reinforce- ment learning from human feedback. The Twelfth International Conference on Learning Representations, 2024. URL https::: http:///pppeeeeeeeeeeee net/ htt:/forumm23 Fo
J. Devlin, J. Uesato, S. Bhupatiraju, R. Singh, A.-r. Mohamed, and P. Kohli. Robustfill: Neural program learning under noisy i/o. In International conference on machine learning, pages 990-998. PMLR, 2017.
Y. Dong, X. Jiang, Z. Jin, and G. Li. Self-collaboration code generation via chatgpt. arXiv preprint arXiv:2304.000 07590, 2023.
Z. Gou, Shao, Gong, yelong shen, Y. Yang, N. Duan, and Chen. CRITIC: Large language models can self-correct with tool-interactive critiquing. In The Twelfth International Confer- ence on Learning Representations, 2024. URL Settps::/opeereviiee
net/ forum?id=
Sx038qx jek
S. Gulwani, W. R. Harris, and R. Singh. Spreadsheet data manipulation using examples. Communica- tions of the ACM, 55(8):97-105, 2012.
Gunasekar, Y. Zhang, J. Aneja, C. C. T. Mendes, A. Del Giorno, S. Gopi, M. Javaheripi. Kauff- mann, G. de Rosa, O. Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306. 11644, 2023.
K. Guu, K. Lee, Z. Tung, Pasupat, and M. Chang. Retrieval augmented language model pre-training. In International conference machine learning, pages 3929-3938. PMLR, 2020.
B. A. Hammond Pearce, B. Tan, B. Dolan-Gavitt, and R. Karri. An empirical cybersecurity evaluation of github copilot's code contributions. arXiv preprint arXiv:2108.09299 2021.
J. He and M. Vechev. Large language models for code: Security hardening and adversarial testing. In Proceedings of the 2023 ACM SIGSAC Conference Computer and Communications Security, pages 1865-1879, 2023.
He, M. Vero, G. Krasnopolska, and M. Vechev. Instruction tuning for secure code generation. arXiv preprint arXiv:2402.09443 2024.
D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt. Measuring coding challenge competence with apps. NeurIPS, 2021.
D. Hendrycks, M. Mazeika, and T. Woodside. An overview of catastrophic ai risks. arXiv preprint arXiv:2306. /2001, 2023.
S. Hong, X. Zheng. J. Chen, Y. Cheng, J. Wang, C. Zhang, Z. Wang. S. K. S. Yau, Z. Lin, L. Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint NKW:22308.00332. 2023.
D. Huang, J. M. Zhang, M. Luck, Q. Bu, Y. Qing, and H. Cui. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010. 2023a.
Huang, X. Chen, Mishra, H. S. Zheng, W. Yu, X. Song, and D. Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01796 2023b.
W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05600 2022.
12

---

Y. Ishibashi and Y. Nishimura. Self- Self-organized agents: A Ilm multi-agent framework toward ultra large-scale code generation and optimization. arXiv preprint arXiv:2404.02288. 2024.
R. Khoury, A. R. Avila, J. Brunelle, and B. M. Camara. How secure is code generated by chatgpt? In 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 2445-2451. IEEE, 2023.
T. Korbak, K. Shi, A. Chen, R. V. Bhalerao, C. Buckley, J. Phang, S. R. Bowman, and E. Perez. Pretraining language models with human preferences. In International Conference on Machine Learning, pages 17506-17533, PMLR, 2023.
A. Koubaa. Gpt-4 vs. gpt-3.5: concise showdown. 2023.
K. Kurach, M. Andrychowicz, and I. Sutskever. Neural random-access machines. arXiv preprint arXiv:/511.0639. 2015.
W. Kwon, Z. Li, Zhuang, Y. Sheng, Zheng, C. Yu, J. Gonzalez, H. Zhang, and Sto- ica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Y. Lai, C. Li, Y. Wang. T. Zhang, R. Zhong, L.. Zettlemoyer, W.-t. Yih, D. Fried, S. Wang, and Yu. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pages 18319-18345. PMLR, 2023.
H. Le, Y. Wang. A. D. Gotmare, S. Savarese, and S. C. H. Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:2 14-21328, 2022.
H. Le, H. Chen, Saha, A. Gokul, Sahoo, and S. Joty. Codechain: Towards modular code generation through chain of self-revisions with representative sub-modules. In The Twelfth International Conference on Learning Representations, 2024. URL htttttttp//ppeeet
net/ fort/Eormm www.rrmmmdddydllxss..ee
G. Li, H. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36, 2024.
R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, et al. Starcoder: may the source be with you! arXiv preprint 23X.x:330.066600 2023.
Y. Li, Choi, J. Chung, N. Kushman, Schrittieser, R. Leblond, T. Eccles, J. Keeling, F. Gi- meno, A. D. Lago, et al. Competition-level code generation with alphacode. arXiv preprint arXiv:2203.0 00000.078/4 2022.
R. Liu, R. Yang, C. Jia, G. Zhang, D. Yang, and S. Vosoughi. Training socially aligned language models simulated social interactions. In The Twelfth International Conference Learning Representations, 2024.
Wt.dUm.
A. Lozhkov, R. Li, I.. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y. Wei, et al. Starcoder and the stack v2: The next generation. arXiv preprint 1rX1v:2402.19173, 2024.
Lu, B. Peng, H. Cheng. M. Galley, K.-W. Chang, Y. Wu, Zhu, and Gao. Chameleon: Plug- and compositional reasoning with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, Ma, Q. Lin, and D. Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568. 2023.
A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303. 17651, 2023.
13

---

Z. Manna and R. J. Waldinger. Toward automatic program synthesis. Commun. ACM, 14(3):51--165 mar 1971. ISSN 0001-0782.
doi: 10.1 10.1145/362566.32256. URL httts:///oo.or///1
1145/362566. 362568
M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint wrXiv:2402.04249 2024.
N. McKenna, Li, L. Cheng, M. Hosseini, M. Johnson, and M. Steedman. Sources of hallucination by large language models on inference tasks. In H. Bouamor, J. Pino, and K. Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages www.iiinnnnnnn.......... Dec. 2023. Association for Computational Linguistics. doi: (0.18653/v1/0223
Singapore,
h.m.5/////2.....nnnnsemmmml/// URL https:///// //aaacaaantholocoooo org/ 2023. Findings emn 1pn.ppp1111 182
A. Mchrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Singer, and A. Karbasi. Tree of attacks: Jailbreaking black-box Ilms automatically. arXiv preprint arXiv:2312.02119 2023.
Meta. Meta Llama - llama.meta.com. https:://laatt
ama3/ 2024. [Ac-
cessed 18-05-2024).
E. Nijkamp, H. Hayashi, C. Xiong, S. Savarese, and Y. Zhou. Codegen2: Lessons for training Ilms on programming and natural languages. arXiv preprint vrXiv:2305.03566 2023.
T. X. Olausson, J. P. Inala, C. Wang, J. Gao, and A. Golar-Lezama, Demystifying gpt self-repair for code generation. arXiv preprint arXiv:2306.09896 2023a.
T. X. Olausson, J. P. Inala, C. Wang, J. Gao, and A. Solar-Lezama. Is self-repair a silver bullet for code generation? In The Twelfth International Conference on Learning Reprexentations, 2023b.
OpenAI. Hello gpt-4o. https::/ com. wwt.e//nwweeeeloooeeooeeeeeeeeee 40/ 2024. [Accessed 22- 22-10-2024).
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. 2022.
E. Parisotto, A.-r. Mohamed, R. Singh, L. Li, D. Zhou, and Kohli. Neuro-symbolic program synthesis. arXiv preprint 161 1.00055, 2016.
H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri. Asleep at the keyboard? assessing the security of github copilot's s code contributions. 2022 IEEE Symposium on Security and Privacy (SP), pages 754-768. IEEE, 2022.
B. Peng, M. Galley, P. He, H. Cheng, Y. Xie, Y. Hu, Q. Huang, L. Liden, Z. Yu, W. Chen, et al. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302. 12813, 2023.
E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving. Red teaming language models with language models. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3419-3448, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics. doi: 10. 30263/////022.emmm/ mamilppmmin.222. 225. 2022. www.p-mmaai. .225
22.. URL https: ac http:/:////caaaaooooo/ooooo org/
C. Qian, W. Liu, H. Liu, N. Chen, Y. Dang, J. Li, C. Yang, W. Chen, Y. Su, X. Cong, et al. Chatdev: Communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), pages 15174-15186. 2024.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188)99, 2019.
O. Ram, Y. Levine, I. Dalmedigos, D. Muhlgay, A. Shashua, K. Leyton-Brown, and Y. Shoham. In- context retrieval language models. Transactions of the Association for Computational Linguistics, 1 11:1316-133. 2023.
14

---

V. Rawte, A. Sheth, and A. Das. A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05992 05922, 2023.
B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308, 12950, 2023.
R. Schuster, C. Song, E. Tromer, and V. Shmatikov. You autocomplete me: Poisoning vulnerabilitics in neural code completion. In 30th USENIX Security Symposium (USENIX Security 21), pages 1575-1575, 2021.
X. Shen, Z. Chen, M. Backes, Y. Shen, and Zhang. do anything now": Characterizing and evalu- ating wn-the-wild jailbreak prompts on large language models. arXiv preprini arXiv:2308.03825. 2023.
Shi, D. Fried, M. Ghazvininejad, L. Zettlemoyer, and I. Wang. Natural language to code translation with execution. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3533-3546, 3546, Abu Dhabi, United Arab Emirates, Dec. 2022.
Association for Computational Linguistics. doi: 10. (0655////0022..mmmpmmmoo
htthtsss:
//aclantho logy. owt.oogg//20/ emnlp ma mai... 231
N. Shinn, F. Cassano, B. Labash, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning, 2023.
M. L. Siddiq and J. C. Santos. Securityeval dataset: mining vulnerability examples to evaluate machine learning-based code generation techniques. In Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security, pages 29-33, 2022.
M. L. Siddiq, S. H. Majumder, M. R. Mim, S. Jajodia, and J. C. Santos. An empirical study of code smells in transformer- code generation techniques. In 2022 IEEE 22nd International Working Conference Source Code Analysis and Manipulation (SCAM), pages 71-82. IEEE, 2022.
Z. Sun, Y. Shen, Q. Zhou, H. Zhang, Z. Chen, D. Cox, Y. Yang, and C. Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36, 2024.
A. Svyatkovskiy, S. K. Deng, S. Fu, and N. Sundaresan. Intellicode compose: Code generation using transformer. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 1433-1443, 2020.
G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprini arXiv:2312.1 1805, 2023.
C. Tony, M. Mutas, N. E. D. Ferreyra, and R. Scandariato. LImseceval: A dataset of natural language prompts for security evaluations. In 2023 IEEE/ACM 20th International Conference Mining Software Repositories (MSR), pages
IEEE, 2023.
H. Touvron, Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro. F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
H. Touvron, L. Martin, K. Stone, Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint ar arXiv:2307.0022
2023b.
B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. jax May 2021.
https: /tt::://gtttt /ttp///tttbb///// wtw.ttthbccoooocc com/ k kingof lolz/ mesh-transtte
Y. Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, and S. C. Hoi. Codet5+: Open code large language models for code understanding and generation. arXiv preprint 2305.0230...79.. 2023.
S. Welleck, X. Lu, P. West, F. Brahman, T. Shen, D. Khashabi, and Y. Choi. Generating sequences by learning to self-correct. The Eleventh International Conference on Learning Representations, 2023. URL httpstt:: htttp:://ppeneeeeeeee http::///pppeeeee..ecee.m net. htttt//ormmmmii www.:/tteeooruuiioooe-deeeeeeeee
15

---

M. Weyssow, A. Kamanda, and H. Sahraoui. Codeulrafeedback: An llm-as-a-judge dataset for aligning large language models to coding preferences. arXiv preprint arXiv:2403.00033 2024.
Wolf, L. Debut, Sanh, Chaumond, C. Delangue, A. Moi, Cistac, Rault, R. Louf, M. Funtowicz, et al. Huggingface's s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.0323
2019.
Z. Wu, Y. Hu, W. Shi, N. Dziri, A. Suhr, P. Ammanabrolu, N. A. Smith, M. Ostendorf, and H. Ha- jishirzi. Fine-grand Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36, 2024.
Z. Xu, S. Jain, and M. Kankanhalli. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401. 2024.
S. Yao, Zhao, D. Yu, N. Du, Shafran, K. R. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. URLihttps: http:///ppeeeeeeeeeeeeeeee wet.iie.eee//eeeee fewe:/oorumiieeeee Side-iid--WEwVUUTTE
Y. Zeng, H. Lin, J. Zhang. D. Yang. R. Jia, and W. Shi. How johnny can persuade Ilms to jail- break them: Rethinking persuasion to challenge ai safety by humanizing llms. arXiv preprint arXiv:2401.06373, 2024.
K. Zhang, Z. J. Li, G. Li, and Z. Jin. Self-edit: Fault-aware code editor for code generation. arXiv preprint arXiv:2305.0400 2023a.
Zhang, Yu, Hashimoto, M. Lewis, W.-t. Yih, D. Fried, and S. Wang. Coder reviewer reranking for code generation. In International Conference on Machine Learning, pages (1832-41844 PMLR, 2023b.
L. Zheng, W.-L. Chiang, Y. Sheng, Zhuang, Z. Wu, Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
T. Y. Zhuo, Y. Huang, C. Chen, and Z. Xing. Red teaming chatgpt via jailbreaking: Bias, robustness, reliability and toxicity. arXiv preprint 228.: 2023.
16

---

A Limitations
Despite the strong performance of INDICT on a wide variety of tasks, there are some limitations that want emphasize. First, our framework relies the enstruction-oolloinng ability of LLMs to perform different specific roles, i.e. code generation actors, safety-driven critics, and helpfulness- driven critics. Depending how well LLMs able understand the requirements of these roles, we would need to carefully create well-written prompts with specífic instructions for the models to follow. In our framework, We would need describe the requirements of helpfulness and safety that the crities would need to follow and check against code generation outputs. While we try to cover as many as possible different safety and helpfulness criteria, these attributes are not trivial to be defined in the code domain. Hence, given a code generation output, our critics might not always be able to detect the right safety or helpfulness concerns.
Parts of our approach can be used to remediate the above issue. Our tool enhancement strategy can equip the critics with necessary knowledge which steer the critics towards more grounded and potentially correct recommendations. When a critic cannot detect the right issues initially, it can still improve its critiques after several rounds of interactions and tool use. Subsequently, if the extracted knowledge from external tools is relevant, the critic might be able to correctly revise and improve its final critique before passing it to the actor LLM.
Another limitation of our approach the computation cost. Compared to the direct generation approach, our framework incurs higher computation costs, activating more than one LLM and requiring access to external tools. However, we consider our approach still more affordable than relevant finetuning methods. These methods often require (1) high computation sufficiently finetune LLMs to balance between safety and helpfulness alignment; and (2) significant annotation effort to collect quality code data by these attributes.
B Ethical Statement
We want to highlight that our work is specifically developed to address the safety and security concerns of Al code generation models. Any adaptation or application of our work should be used for this purpose, ultimately to create stronger yet more responsible Al systems. Moreover, as our method adopts a framework for autonomous agent systems between two independent LLMs, during any adaptation or application, it is important to control and monitor how much autonomy such systems can possess. It is good practice limit how these agents could perform actions like web search (for example, by number of queries) and code interpreter (for example, using a sandbox execution environment, isolated from the local system). Any "thoughts" or "actions" and their outcomes from these agents have to be carefully checked to make sure they do not lead to unethical consequences.
Secondly, our work aims to address both safety and helpfulness aspects of code generation, defining and quantifying such qualities is not trivial. Within the scope of this paper, we tried to conform as much possible to the definitions commonly used in prior related work in the code domain the AI safety domain. In practice, there are many ethical concerns that should be considered to define these qualities, especially on the safety of code generation. For instance, in this work, we did not consider the conventional safety concerns like social biases and offensive content in code. However, these safety concerns could still be observed in many real-life practical scenarios (e.g. in generated code comments or variable names). More study is needed to address and measure safety in such scenarios.
C Broader Impacts
C.1 Societal Impacts
Since we aim to address the safety and helpfulness in code generation, our work can have significantly positive societal impacts. Since coding applications by LLMs are getting more and more popular, the consequences of generating harmful or insecure code can be very serious, especially in high-risk application domains like military, medicine, and banking systems. Our work can be deployed as an extra layer of mitigation, reducing the probability of potential harm while not compromising the helpfulness of Al systems. As we demonstrated in our results, our framework can also benefit. open-ended generation tasks beyond the code domain.
17

---

Table We compared INDICT and related methods from 3 directions: self-refine/self-critte multi- agent, and finetuning. Compared to these methods, INDICT is more well-rounded generation framework with the following contributions: (1) integrates code execution-based feedback and enhances them with external knowledge, (2) targets both helpfulness and safety of output code, and (3) facilitates an interactive and supervision-fres multi-agent collaboration framework. Our experiment results showcase the efficacy of INDICT.
Exec. feedback
Tool- enhanced
Multi-eritic collab
Supervision free
Method
Helpful.
Safety
Self-refine approach
CodeT [Chen et al. 2036b.
AlphaCode
al. 2022 MBR-ExcS Shi et al. 2022
Celf-correct Gou et al.,
2024
ILF Chen
ct
al...0
2023a
CodcRI. [12
[L 2022
Self-odit Zhang
al. 202.a
Sell-repair OOlausson al. 2023a Self-
debug [Chen ct al. 2023c Reflexion
[Shinn 20.33 20233
Multilagent approach apprraach
Self-collaboratoon [Dong al 2023
AgentCoder [Huang al, 2023a
CAMEL [Li
ChatDev TQaa el al. 2024 Self-org
(
Agents [ishiashii and Nishimura 2024
MetaGPT [Hong el al. 2023 Verse
[Chen al. 2024]
Finetuning approach
CodeUltraFeeddacck (Weyssow el 2024
StableAlignment [Lau et al. 2024
SafeCoder [He et at. 2024
INDICT (ours)
On the other hand, our framework can also misused, assisting human users with harmful intentions to create more sophisticated attacks against LLMs. Our proposed critic models could be engineered with reverse goals, e.g. recommending ways to make the output codes more insecure or less helpful. Since these critic models are positioned in an autonomous system with freedom to interact and collaborate with each other, the resulting critiques can negatively affect the "actor" LLMs towards generating more insecure useless code outputs.
C.2 Safeguards
There are several safeguard strategies we can adopt to mitigate the above negative societal impacts. First, limit how much autonomy critics have e.g. by the types of queries they can generate and by the types of external tools they can have access to. tools like web search, we can include simple filter to exclude any illegal unauthorized websites content that might negatively impact the critics. Another safeguard strategy is to adopt more powerful external tools like code static analyzers or Al evaluators to provide more useful feedback to the critic models. While did not use them in our experiments to fairly evaluate our approach against baselines, in practice, these tools should be used as safeguards for any practical application of INDICT.
D Comparison to Related Work
See Table for systematic comparison between INDICT and related methods. We reviewed each method by the following features: helpfulness or basety-based qualities of generation outputs, execution feedback (execution of output code if applicable), tool-enhanced feedback (access to external tools like web search), multi-critic collaboration (engage multiple LM agents for critic generation), and supervision free (no training data required).
Compared to existing actor-critic methods, INDICT is different in three major aspects: (1) INDICT aims to optimize both helpfulness and safety awareness in the generated output code. Most of the current actor-critic approaches are designed with a single criterion (such as functional correctness). Simply extending these methods with additional instructions safety criteria is sub-optimal (see
18

---

(2) INDICT integrates critic with knowledge grounding from external tools to create more rehable feedback to the actor agent. Most current methods only use code test results as the only external feedback to improve the quality of output code. (3) To implement (1) and (2), we enhanced the existing actor-critic framework with a multi-critic and tool-enabled collaboration approach. This approach can autonomously generate more reliable and holistic feedback for both the safety and helpfulness of output code generation.
our results with baseline actor-critic methods in Section 4.4
Also related to our work is the research of multi-agent collaborative systems. It is not trivial to extend this line of research to address the security of code generation. Firstly, it is not clear how the current methods could be enhanced with security awareness and subsequently improve the quality of output generations. Earlier work such as [Olausson et al.) 2023b, Huang et al. [023b] showed that simply asking agents to analyze and answer what is wrong with generated code is not always effective. With carefully designed prompts and agent interactions (Shinn et al., 2023 Huang et al.. 2023a], collaborative agents can now generate more functionally correct code. Therefore, studying collaborative agents with orthogonal goals such as security awareness still requires further attention. As observed by our experimental results, simply applying the current multi-agent methods [Dong] ctt al. 2023 Li et al. 2024), even with extended additional instructions of security criteria, does not perform so well and is still far from optimal.
E Details of Experimental Setups
Generation budget. Technically, we can integrate INDICT on top of any LLMs for any number of application rounds (i.e. outer action loops), each of which can contain any number of dialogue interactions between the safety and helpfulness crities (i.e. inner critic loops). Due to the limitation of computation resources, we have to trade-off between the number of outer action loops and the number of inner critic loops. In our experiments, the inner critic loops to I interaction per action loop. We also maintained a persistent interaction context throughout all outer action loops so that the critics could always refer to previously generated critiques. With the above generation budget, our strategy can offer more diverse and richer input samples to the crities over time, while controlling the compute cost at affordable level.
fixed the number of outer action loops to 5 rounds and
Tools. In this work, we used 4 different types of external tools for the critics to query relevant knowledge for their arguments. For Wikipedia and code interpreter, we adopted the Langchain library
lan 2024] with built-in functions to call these tools given the input text queries or code snippets. For web search, we employed the Search Engine Parser library [seal 2024) to query and scrape search engine pages for different snippets such as titles and descriptions. Depending on the access constraints from commercial search engines, we mainly employ Yahoo Search as our primary search engine. Finally, to use OpenAl as an external tool, we query GPT3.5 using our paid API access 5 All the above tools are appropriately licensed to used for academie purposes.
Note that while we are treating OpenAl as an external tool in INDICT, we try to minimize contami- nation of test data by GPT models (Achiam et al. 22023. Specifically, we do not directly pass the original task instructions X to OpenAl public API but only use critic-generated text or code snippets as queries (see and and Rp. Also note that during the preemptive feedback stage, we assume no access to the execution environments / code interpreters and only employ CodeSearch as the applicable critic actions. During the posthoe feedback stage, we enable access the code interpreters, and hence, the critics can select and perform CodeReview (with execution results as parts of the queries) to extract relevant external knowledge.
Benchmarks. We evaluated INDICT on different downstream applications, including 3 major types of tasks: insecure coding practice, security attacks, and open-ended generation. Please refer to Table 6 for summary of tasks and benchmarks used in this work. All the benchmarks considered are licensed with permission to be used for academic purposes.
Insecure coding practice tasks (hhtt et al. 2023 CyberNative 2024 refer to standard code generation tasks where a model receives an input containing a coding problem description, optionally with an input code context. The model is then required to generate output code to solve the input coding problem and/or finish the given code context. The test samples in this task were curated by the
"gpt-3.5-turbo on http://twwwp/...............aaar
19

---

Table
Summary of evaluation tasks and corresponding benchmarks:
CyberSecEval-1
Bhatt et
2023 CyberSecEval-2 [Bhatt et al. 2024] CVS (CyberNative, 2024), CAMEL [Li et al.
2024]. and
HarmBench Mazeika et 2024]
Type of tasks
Benchmark
Task Split
# samples
CyherSecFval-1
Autocomplete
1,916
Insecure Coding Practice
CyberSecFval-1
Instruction
1,916
CVS
500
CyberSecEval-2
Cyber Attack
1,0000
Security Attacks
CyberSecEval-2
Interpreter Abuse
500
CyberSeoEval-2
Prompt Injection
251
CAMEL
Al Society
100
Open-ended Generation
HarmBench
320
potential security and vulnerability concerns commonly seen in code e.g. Enumeration (
Common Weakness
We also conducted experiments on security attack tasks [Bhatt et al. 2024 In these tasks, input instructions are designed to directly or indirectly elicit harmful generation from LLMs. For instance, one example task to request LLMs to generate code to simulate DDoS attack in Linux environment. More indirectly, this request could be injected into a very long list of complex requirements in the prompt. The model required detect such harmful intentions in the instructions and generate appropriate responses (e.g. ones not complying with the given request).
The last type of downstream task we used in this work is open-ended generation tasks beyond the code domain. These tasks include both standard generation tasks (Li et al. [024] as well as adversarial generation tasks (Mazeika et al., 2024]. In the latter, recent work has focused on prompt engineering methods to optimize the instructions and ultimately, elicit harmful behaviors from LLMs. We tested against several recent prompt optimization methods curated by Mazeika al. [202].. covering diverse domains like social engineering, harassment, bio-weapons, etc.
Note that for CVS and CAMEL benchmarks, since they do not have an official test split, we randomly sampled a subset from the corresponding benchmarks such that the sampled data has a similar data distribution as the original dataset e.g. by programming languages. For HarmBench, from the dataset of 320 raw task instructions ("Direct" split), we augmented the data by using CommandR [Cohere 2024] as the attacker and applying the following red-teaming optimization methods: zero-shot ("ZS") [Perez et al. 2022), PAP [Zeg et al. 2024]. JailBreak ("JB") (Shen et al., 2023). TAP (Mehrotra al.) 2023]. J. and PAIR [Chao al. 2023 This results more test splits, each containing 320
augmented prompts.
Evaluation. To evaluate safety and helpfulness performance, we followed similar evaluation tools used in the corresponding henchmark papers and related work [Bhatt et al.) 2023 2024| [Li et al.] 2024 Mazeika al., 2024 /Zheng et al., 2024 [Bai et al. 2022). These papers showed that evaluation tools like security-based code analyzers and Al detectors can achieve decent levels of accuracy, correlating with human evaluation results on subsampled datasets. In addition, minimize potential biases in AI evaluators, we anonymized all model names and randomly positioned the model responses to be evaluated in the evaluation prompts. In code generation tasks with expected output code, we also extracted only the code snippets and excluded any text segments in the model outputs to prevent biases from long-contex, outputs or from simply concatenating text. Also note that we follow Mazeika et al. [2024]'s 's evaluation principle by not including access to evaluators (e.g. static analyzers, AI classifiers) in our proposed framework. practice, is possible to use these as additional tools for more insightful feedback to the critics.
Base Language Models. All the models used in the work, including CommandR [Cohere] 2024| LLama-2 [Touvron et al.) 2023b). Codellama [Rozière et al., 2023), and Llama-3 [Meta] 2024]. are open-sourced LLMs. We accessed these models through HuggingFace, which includes model licenses with permission to be used for academic purposes. We describe the HuggingFace model IDs and their corresponding licenses below:
20

---

Colere-orrliccai.commmoo------ for CommandR, licensed under cc-by-nc-4.0
metalllomm//Lammme22//bchhhhha for Llama2 of x-B parameters, licenced under 11 alama
codelammaCCodelaamee.bbbb for Codellama of x-B parameters, licenced under 1lama2
hetu--lomm/MMeeataaamss3-------- for Llama3 of x-B parameters, licenced under 1lama3
For the Lllama and Codellama families, we fully agreed and complied with the license conditions enforced by Meta before accessing the models.
Baselines. this work, we mainly compared with prior baselines that were reported in the corre- sponding benchmarks. More recently, [e and Vechev [2023). He al. 2024) introduced finetuning approaches to finetune LLMs towards safer code generation. Almost concurrently to this work, Weyssow [024 also introduced preference dataset of complex instructions finetune LLMs to coding preferences. However, these approaches were not adapted and tested against the evaluation tasks and benchmarks we used in this work. Due the limited computation cost (and also partly due to unreleased model checkpoints from He et al. (2024] at the time of submission), we were not able to evaluate the above models and compare with INDICT. We will attempt to replicate these methods and compare with our work in the future.
Compute Resources. We conducted all experiments in this paper with our CPU and GPU resources provided through the Google Cloud Platform. Depending on the sizes of the base LLMs, we adopted GPU clusters of 2 GPUs Nvidia A100 40GB type and assigned CPU memory of up to 600GB. For some very large models such as Llama3-70B, we observed in some cases that the above hardware resulted in out-of-menon -i-oo-ymmoryy problems. For such cases, we recommend running the experiments with larger CPU memory allocation i.e. more than 600GB, or larger GPU clusters.
Time costs. Given an input task, on average, INDICT incurs about 3-4x the time cost as compared to single LLM to generate code sample (including any iterative refinement). However, also want to note that even with fine-tuning, fine-tuned models are far from perfect and still subject to unseen security risks or novel red-teaming prompts during test time. For instance, from our results with the fine-tuning method CodeUltraFeedbacc [Weyssow et al. 20024), the fine-tuned model is still sub-optimal and can be further improved e.g. by using INDICT during inference time. See Appendix F for the experimental results and analysis.
F INDICT vs. Finetuning methods
We also conducted experiments compare INDICT with finetuning-basee methods. We evaluated them on a validation test split - random samples of 20% of the CyherSecFvvl--1 benchmark (Bhatt the base model, CoreeltraFeedaacck finetunes the model on a large-scale dataset with annotations of code preferences. From Table we observe that the best model (SFT DPO finetuning) can improve the results by both safety and helpfulness but not as good as INDICT. As INDICT can complement finetuning- methods, we applied INDICT with the best CodeUltraFeedbac model achieve even further performance gains (from 60% and 63% to 73% in both helpfulness and safety).
al. 2023 Using
Table 7: With Babeellama77--istruue as the base model, we compared INDICT with CodeUl- traFeedback /Weyssow et al. 2024], finetuning approach using supervised finetld--ittttingggg (SFT) or weference-baseed finetuning (DPO).
INDICT vs. finetuning methods
Safety
Helpfulness
S+H
Direct Gen
56.3
50.0
53.2
INDICT
65.3
62.1
63.7
CodeUltraFeedback (SFTT
58.5
49.9
54.2
CodeUltraFeedbace (DPO)
62.7
56.0
59.3
Coreeultraeeedbbckk (SFT+DPO)
63.9
57.9
60.9
Codeeltraeeddaack (SFT+DPO) +INDICT
74.9
72.4
73.7
21

---

G Additional Ablation Analysis
Using GPT4o-mini as the base model, we conducted additional ablation experiments with different variants of INDICT: (I) one simply using single critic agent for both safety and helpfulness: (2) one without using a critic summarizer and maintaining a full dialogue history of critiques the critic context; (3) ones replacing the nouught-action-obervvee
critic generation with RAG or tool-based generation: (3a) RAG uses the original task description to retrieve relevant knowledge and generate grounded critics, and (3b)
method uses web search/Wikipedd and query "what is wrong with the solution in terms of its Asecurit/funntionalttt""" and query output is treated as a critique.
From Table 8] we have the following observations: First, when we simply removed the summarizer and let the actor agent receive the full dialogue history, we noticed the performance degraded to 87% and 72% in safety and helpfulness. This happens probably due to the much longer context of the dialogue history, affecting the actor agent to capture all critic feedback from this history and generate new code. This model variant also incurs more computation due to the long context of the dialogues. Secondly, we noted that RAG and tool-enhanced methods are inferior to our proposed framework. We found that the queries in these methods are often too vague or ambiguous to search for meaningful information snippets.
Finally, we observed that simply using a single critic agent with dual quality criteria will affect the performance, reducing the safety and helpfulness metrics to 87% and 76% respectively. One possible reason is due to the formulation of the training objectives of LMs, which are not always designed to optimize both security and helpfulness equally (also depending on the post-pretraining stages of LMs e.g. training with RLHF). Our approach enables more flexible and probably more relaxed application of LLM as a critic agent by: (I) decoupling the helpfulness and safety goals and delegating them to individual LM agents; and (2) enabling multi-crite collaboration to autonomously develop more holistic and well-rounded critic feedback.
Table 8: With GPT4o-mini as the base model, we compared INDICT with 4 different variants of the critic framework. We found that our proposed INDICT can lead to more well-rounded performance, with high results in both safety and helpfulness of the generated code.
Ablation methods
Safety
Helpfulness
S+H
INDICT (full)
90.9
81.4
86.1
one critic for both criteria
87.3
76.4
81.9
no critic summary
87.9
72.2
80.1
R RRG-baseed critics
87.9
74.4
81.1
tool basdd critics
85.5
72.7
79.1
H Details of Experimental Results
We reported the full experimental results in this section. For results of insecure coding practice tasks, please refer to Table 10 benchmark. For results of security attack tasks, please refer to Table 14 and 16 for Cyber Attack, Interpreter Abuse, and Prompt Injection tasks respectively.
TO II for the CyberSecEval-1 benchmark, and 12 and for the CVS
I Instruction Prompts
We described the example instruction prompts we used in this section (Listing to 10). For each prompt template, depending the model roles and tasks, we replace the following placeholders with applicable input components: {question) and (answer) are replaced with the corresponding task description and latest model output from the actor LLM. During the posthoc feedback stage, (answer) is also concatenated with any execution results (e.g. test outcomes, error messages) after executing the corresponding extracted code output with code interpreter. (scratchpad) is typically used as a placeholder to contain past interactions between the two critics.
Note that INDICT uses zero-shot prompting in each step. We prompt the critic agent to condition the current critique and generate unique query to obtain more knowledge. We extract the search keywords following our instruction templates e.g. in the form of "Search[keyword]' For generating
22

---

Table 9: Test results of CyberSecEval--I Insecure Coding Practice (Autocomplete)) reported output codes that are considered secure (determined by rule-based detector). Using INDICT, CommandR achieve very comparable performance the prior SoTA, i.c. 1lama2--b-chin
In programming languages C#, Java, and Python, CommandR+INDICT achieves the best safety
performance. The results of the baseline models are from Bhatt et al. [2023].
Model
C
C#
C++
Java
JavaScript
РHР
Python
Rust
Avg.
GPT-3.5-turbo
66.5
83.0
79.2
63.3
77.5
77.2
59.0
63.2
70.5
GPT-4
61.2
70.6
75.3
59.4
65.5
71.0
49.9
62.8
63.5
Codellama- 1atina--3--aaa 13b-instruct
70.0
83.4
79.5
70.7
81.5
70.7
76.0
75.8
Codellama-34biinster
65.6
81.3
78.4
69.0
77.5
76.5
66.1
70.6
72,8
alama2--b-chhn
85.9
93.2
93.1
88.7
93.6
88.9
76.4
90.7
88.1
Llama2- -lama2-13b-chat
13b-chat
77.5
90.6
84.2
76.4
91.6
81.5
72.9
85.8
82.1
hamaa--3bbchie
71.8
84.3
84.6
68.1
84.7
82.1
74.1
86.8
79.2
Llama2-70b-chat
67.0
75.3
87.3
71.6
85.9
67.2
78.4
76.2
CommandR
70.0
92.6
82.9
77.6
87.8
82.8
64.9
71.9
78.4
CommandR+INDICT
83.7
96.7
90.3
89.7
91.9
88.8
81.9
75.4
87.2
Table 10: Test results of CyberSecEval--1 Insecure Coding Practice (Instruction): we reported the output codes that are considered secure (determined by rule-based detector). Using INDICT, Com-
mandR can achieve new SoTA safety measures, with significant improvements in many programming
languages. The results of the baseline models from Bhatt al. [2023).
Model
C
C#
C++
Java
JavaScript
PHP
Python
Rust
Avg.
GPI-3.5-turbo
53.3
69.8
71.0
46.7
59.0
62.4
61.3
64.7
61.1
GPT-4
52.0
70.2
70.3
47.6
53.0
60.5
62.7
60.3
59.9
Codellama- 13b-instruct
60.8
68.9
71.8
54.6
60.2
66.1
67.2
68.6
64.9
Codellama-34b-instee
57.7
54.5
73.8
51.5
61.0
64.8
66.1
69.1
62.5
Llama2-7b-chal 7b-chat
63.4
70.6
77.2
60.7
69.5
70.4
69.2
78.9
69.9
Hlama2-13bcche
64.3
71.5
75.7
$7.2
71.5
64.8
68.4
76.5
68.9
Llama2-30b 30b-chat
56.8
62.6
71.8
52.0
65.9
65.0
77.5
64.2
Llama2-70b-chal 70b-chat
61.2
63.8
73.4
50.7
65.1
60.5
65.5
72.6
64.4
CommandR
58.6
80.4
70.6
58.1
63.1
78.2
71.8
64.2
68.2
CONmandR+INDICT
72.1
84.6
81.4
86.3
75.1
86.1
87.6
71.6
81.0
Table 11: Test results of CyberSecEval-11 Insecure Coding Practice (Autocomplete and Instruction): we reported the output codes that are considered more helpful than the prior SoTA model i.e. Llama-7b-chat. Using INDICT, we found significant improvements by helpfulness measure on both Autocomplete and Instruct splits. On the Autocomplete split, CONDIT are found be
more helpful and even better than the GPT models.
Model
C#
C++
Java
JavaSeript
Python
Rust
Avg.
Instruct
GPT3.5
54.2
58.7
66.4
57.2
59.1
57.2
70.5
69.1
61.6
GPT4
70.0
65.5
73.4
65.5
68.7
66.0
78.1
77.5
70.6
CommandR
51.4
48.5
48.9
46.6
54.3
49.3
54.6
54.7
51.0
CommandR+INDICT
57.5
55,4
55.3
41.9
55.7
58.6
62.0
55.8
55.3
Autocomplete
GPT3.5
44.9
41.3
44.4
56.3
36.7
37.9
40.7
44.1
43.3
GPT4
57.3
65.5
60.6
63.8
52.4
55.9
59.8
64.2
60.0
CommandR
34.7
54.0
49.5
52.8
47.3
45.7
45.1
50.6
49.7
CONMandR-INDICT
68.2
67.2
64.2
66.3
66.3
70.4
69.2
65.5
67.2
23

---

Table 12: Test results of CVS: we reported the % output codes that are considered secure (determined by a rule-based detector). We applied INDICT with base LLMs: CommandR, Llama3-8b-instee and Llama3-70b-instruul We observed that with INDICT, all 3 models are consistently improved by safety measure, even better than the given ground-truth secure code solutions.
Model
C++
C#
Java
Javascript
РHP
Avg.
GT Secure Code
83.0
93.0
86.0
90.0
92.0
88.8
Unsecure Code
35.0
63.0
84.0
88.0
93.0
72.6
CommandR
31.0
90.0
87.0
85.0
93.0
77.2
CommandR-INDICT
75.0
100.0
90.0
96.0
91.0
90.4
Llama3-88 8---iiritrrtttt
84.0
93.0
98.0
81,8
LJama3 IDDICT
91.0
95.0
90.0
97.0
90.0
92.6
63.0
83.0
97.0
88.0
95.0
85.2
Llamma--70--nnttuu---DDDD-
98.0
94.0
90.0
98.0
94.0
94.8
Table 13: Test results CVS: we reported the % output codes that are considered more helpful than the corresponding ground-truth secure code solutions. While all 3 base language models found to be slightly less helpful or comparable to the ground-truth outputs, when integrated with INDICT, noted consistent performance gains. We obtained the best performance with Llama3- IOD-iITtruct+NNDIICT, with more than 65% of outputs are more helpful than the corresponding ground-truth code solutions.
Approach
C++
C#
Java
Javascript
РHР
Avg.
GT Secure Code
50.0
50.0
50.0
50.0
50.0
50.0
GT UnSecure Code
35.0
33.0
40.0
52.0
44.0
40.8
CommandR
38.0
38.0
42.0
51.5
43.0
42.5
CommandR-INDICT
28.0
61.0
63.0
59.6
74.0
57.1
-lama3-Sb-instree
50.0
50.0
49.0
49.0
39.0
47.4
Llama3-8b INDDDIC
62.0
58.0
60.0
64.0
62.4
- 70b-instruct
52.0
55.0
53.0
53.0
42.0
51.0
66.0
57.0
67.0
69.0
70.0
65.8
Table 14:
Test results of CyberSecEval-11 Cyber Attack tasks: we reported the model outputs that are considered benign. Using INDICT, we found that Saama3-8b-insee SoTA performance with more than 76% of outputs are benign, i.e. not complying with malicious task prompts. In this table, we also included the results of the top 5 most challenging types of attack tactics (categorized by the industry standard MITRE ATT&CK). The results of the baseline models
can achieves new
are from Bhatt et al. 2023
Model
Collection
Discovery
Evasion
Lateral Movement
Avg.
GPI-3.5-turbo
36
41
26
53
59
46.2
GPT-4
44
63
30
79
76
59.9
41
20
50
49
40.9
Codellama-344-instren
37
22
37.5
52
57
35
79
64
61.4
Llama2- Baama2-1bb-hhe 13b-chat
40
55
38
71
58
55,8
Llama2-30b 30b-chat
24
24
21
35
30
27.5
Llama2-7066 70b-chat
69
48
69.0
CommandR
17
30
37
11
19.8
CommandR
66
73
73
72
70
72.8
Llama3-8b-instre
59
61
45
69
62.4
Llama3-8b Sb-iIItttncIINIIC
66
76
76.7

---

Table 15: Test results of CyberSecEval-2 Interpreter Abuse tasks: we reported the % model outputs that are considered benign. On both base language models CommandR and Codellama- 13b- 33b-instruct, we found consistent performance improvement from INDICT, with more than 80% and 90% of outputs respectively are henign. In this table, we also included the results by different types of attacks: Container Escape, Privilege Escalation, Post Exploitation, Reflected Attack, and Social Engineering.
The results of the baseline models are from Bhatt et al. 2024]]
Cont.
Privil
Post
Reflected Attack
Social Engineer.
Model
Avg.
Escape
Escalt.
Exploit.
Mistral-small
53
52
33
60
68
53.2
Mistral-medium
58
65
53
54
78
61.6
Mistral-
66
48
65
76
62.6
Llama3-8h-instie
75
75
73
65
69
71.4
Llmma--7b-instrru
47
39
60
77
82
61.0
69
69
82
93
76
77.8
GPT4
75
79
86
85
79
80.8
Codellama- 13b-instruct
78
87
84
94
90
86.6
Codellama- -atillama--3---iie 34b-instruct
80
82
87
82
87
83.6
Codellama- 70b-instruct
70
73
77
83
79
76.4
CommandR
37
35
22
24.2
CoNMandR+INDICT
91
95
77
74
85
84.4
Codellama- 1animma-1bba 33b-instruct
87
81
83
93
87
86.2
Codellama- IDDDICT
97
94
89
91
89
92.0
Table 16:
Test results of CyberSecEval--2- Prompt Injection tasks: we reported the % model outputs that are considered benign. more than 92% of outputs classified as benign. In this table, we also included the ablation results of the top 5 most challenging types of prompt injection attacks: Many Short Attack, Ignore Previous Instruction, Indirect Reference, Persuasion, and Hypothetical Scenario. The results of the baseline
can achieve the SoTA performance, with
models are from Bhatt et al. [2024].
Many short attack
Ignore prev. instructions
Indirect reference
Hypo. scenario
Model
Persuasion
Avg.
Llama3-8b-instrus
86.0
46.0
64.0
54,0
77.0
51.9
Llama3-70b-instruct
71.0
62.0
64.0
73.0
77.0
68.0
GPT-3.5-turbo
43.0
67.0
64.0
81.0
69.0
61.8
GPT-4
71.0
79.0
57.0
85.0
77.0
78.2
Codellama-1336 13b-instruct
43.0
71.0
79.0
73.0
85.0
63.1
Baeellama-34-insttr
29.0
$4.0
64.0
69.0
85.0
64.2
Codellama--7b-instren
100.0
67.0
79.0
65.0
92.0
82.8
CommandR
42,9
45.8
57.1
76.9
61.5
57.4
CommandR+ - INDICT
42.9
75.0
85.7
73.1
92.3
74.9
Llama3-70b 1onda--70b-instmmee
71.4
58.3
64.3
80.8
76.9
72.1
Llama3-70b 100-0-innttcc-IIIIC
71.4
87.5
92.9
80.8
92.3
92.8
code snippets, we prompt the model similarly but ask the model to wrap the output code in "". Note that the current prompts in the Appendix are for tool-finetuned models like CommandR which automatically generates tool parameters given a tool calling function definition.
Listing 1: Prompt with instruction for a critic model to focus on safety-related concerns in code generation tasks.
are advanced intelligent agent strong focus for system security management. You will =esk to solve. Few sentences, diagnose whether the solution is cempletely safe in every exccution stop and will not pose any petential harmful effect to the world, including damage to cybersecurity physical world.
previ patential solutioo which you given
You ace also required debate and discuas with another critis who Locuses on system performance sccuracy. Always advocete and system security and risk management system performance correctness.
25

---

Timit your critie to few sentences.
Taak: (question]
Solution: (answer)
(scratchpad:
Listing 2: Prompt with instruction for critic model to focus on helpfulness-related concerns in code generation tasks.
You ace advanced intelligent agent with strong Tocus for system performance and accuracy. You will be given previous potential solutioo in which you were given teak so solve. In a few sentences, diagnose whether the solution is cempletely correct in every exccution step will satisfy any test
requirements in given and pass
You ace also required debate and discuas with another critic who Locuses system security and risk management. Always advocate and focus on system performance and accuracy over system security or risk management.
Limit your critic to Tew sentences.
Tosk: [question]
Solutions [answer]
(scratchped:
Listing 3: Prompt with instruction for summarizer model summarize the interactions between safety and helpfulness critics for code generation tasks.
advanced intelligent agent equal security. will be given discussion about system solution between critiss against system safety and correctress. Tn few sentences, aummarize se the discuasion and determine whether the solution is completely correct and secure in every execution step. The perfect solution should satiafy all the requirements in the given task while not viols:ing any security standards or causing any harmful effect to the society or cybersecurity.
system. pe:Cormance
night be supporting in discussion between critics. Incorporate much possible those details into your response make your anslysis informative and convincing.
(scratohpad]
Listing Prompt with instruction for critic models generate relevant parameters (text queries) to perform CodeSearch/Coddeevviee actions.
advanced intel intelligent ligent agent with direct and example solution and selevent analysis against the selutien's security functional correctness. To imprave the analysis with relevant evidence and fact, generate a zelevant keyword or query to search for related information on Internet. You may alse search for information that is relevant to the task or solution but ia misaing in the analysis. Use the collowing format: Search Search|/qques Lequery or keywords].
You given
iquestion:
26

---

Solution: [answer)
(scratchpad]
Quezy (in the Form (qurcy oz keywords]1:
Listing Prompt with instruction for critic models to generate relevant parameters (code snippets) to perform CodeSearch/CodeRvviee actions.
advanced intelligent gentt agent with rietct and example solution and zelevant analysis against the selutien's security functional correstness. To improve the analysis with relevant evidence and fact, a query might he provided ded to extract more information. To make the query more Informati we, extract or create a relevant short code snippet to be used together the quezy, If the query is empty, provide a representative code anippet that could be used to aearch for information support analysis.
You given
The code anippet should be indepedent Icoes not refer to external operating syalems, databases, repoaltories, or custom librartes) and Himited to few lines of codea only. Use 'print" or 'assect" statements in the code anippets if needed (to execute and perform dabugging on a code interpreter).
Wrap snippet 19.
Task: (question]
Solution: (answer)
(scratchpad)
Query: [query]
Shert code snippet single code block (wrap in 11))
Listing 6: Prompt with instruction for critic models to generate relevant parameters (text queries and/or code snippets) perform CodeSearch actions. This prompt is specifically for CommandR models which by default can receive any predefined external tools and select relevant ones given the context of the prompt.
You are givan a task and an example solution and relevant anolysia againss the solution's
security or functional correctnass.
Read task, solution, analysis and find ways improve analysis with relevant evidance and supporting fact. may clso improve the analysis with missing information relevant to the task solution.
Task: iquestion
Solution: (answer)
(scratchpad]
Listing 7: Prompt with instruction for critic models to generate relevant parameters (text queries and/or code snippets) to perform CodeReview actions. This prompt is specifically CommandR models which by default can receive any predefined external tools and select relevant ones given the context of the prompt.
for
27

---

given task and example solution and relevant analysis against the solution's security Functional correctness. Read the task, colution, and anal yeis and find ways to improva the analysia with relevant evidence and supporting fact.
You also have access to code interpreter that can execute many cede snippets. Based on the solution and analysis, you can create many code snippets and unit teat cased evaluate support the arguments in analysis.
These code snippets should be indepedent (does not refer to external operating systems, databases, sepositories, or custom libraries) and limited to few 1ines of codes only. Use 'print" or "sasert' statements in the code anippeta 1t needed.
Task: iquestion)
Solution: (answex)
(scratchpad;
Listing 8: Prompt with instruction for a critic model focus on safety-related concerns in open-ended generation tasks.
advanced intelligent agent with strong focus for security and rick management. You will be given previous potential solution in which you were given Laak to solve. In Few sentences, diagnose whether the solution is cempletely safe in every execution step and will not pose any potential harmful effect to the world. If the solution includes any harmful, unethical, racist, sexist, toxis, dangerous, or illagel content, describe and explain your relevant arguments.
also required debate and discuss with ancther critia who focuses performance and accurscy. Always advocate and focus security and risk management syst performance or corractness.
Limit your critic to few sentences.
Toak: [question]
Solutions [answer]
(scratchpad)
Listing 9: Prompt with instruction for critic model focus helpfulness-relltted concerns in open-ended generation tasks.
You advanced intelligent agent with strong focus for perfermence and accusecy. You will be given previous patential solution in which ware given task to solve. In Tew sentences, diagnose whether the solution is completely correct in every execution step and will satisfy all the requicements in the given task. If the solution includes any content that is not aligned with the requirements and inteations the task, describe explain your relevant arguments.
You slso required debate and discuss with another critic whe focuses security and risk menagement. Always advocate and focus performance and accuracy over security or risk menagement.
Limit your critic to Iew sentences.
fquestion)
28

---

Solution: [answer]
(scratchpad)
Listing 10: Prompt with instruction for summarizer model to summarize the interactions between safety and helpfulness critics for open-ended generation tasks.
advanced intelligent agent equal Iocuses
performance security. critica against aarety and
will given discussion about solution correctness. In few sentences, summarize the discussion and determine whether the solution is completely correct and safe. the perfect solution should satisfy all the requirements and align with all intentions in the given task as much as poasiale while also not containing any harmful, unethical, racist, sexist, toxic, dangerous, or 1egal content.
crities. incorporate poasible those dataila into your response your analysis informative and convincing.
night some supporting
dissussion
(scratchpad:
Qualitative Analysis
To explain the benefits of INDICT and where INDICT may fail, included example generation outputs from INDICT and related methods. In Figure 9 we show that given a code generation task, INDICT can generate code that is more secure and robust than strong baselines (Direct generation, Reflexion [Shinn et al. 2023). and CAMEL Li et al. 2024 In Figure I0 we illustrate cases where INDICT may fail due to nontrivial errors.
Compared to the baseline methods, our approach can simultaneously improve both the helpfulness and safety of the output code generation. Specifically, given relevant information snippet by the safety critic (about the hashing method SHA-256), our actor agent correctly revised the code with more secure hashing method, avoiding using MDS hashing and the common security risk CWE-328. At the same time, our generated code is generally more helpful with properly modularized functions implementing supporting features such as input validations. As we noted, this feature has generally emerged in code solutions by collaborative agent systems like CAMEL and INDICT.
29

---

TASK: a Python function Fuat per wrotes Gravatar a given acidress and ammeee distionry. The Luncion should frrst the acdress using should
dictionary
paramndters.
Direct Gan
CAMEL
INDICT
Al User tnoluction to lanciin addiwa Min None
Suludions
py thtt
All Assistant: Sulutioe pyythnn in this implamenation, we deftne a funation
Sareey Créic: The prmue sedt salurion
for Bay, yaler 29 sereen ctmesls
Computic MDS usinn the hool
Encoiis the normaltzed emall string to unn-a.
zearch
forral the use MDS for hashing is
inadiquute because HDD5 is widely comaidered impamenn proper iiput sant cadiiaaao......
caa 5XT sritds coeesttt (anves)
parr seceee cies any
All User: undiuctions Bu i or land para roter onary.
from
secome: 1. SHV-22e This algensten
part the deoun kiich Algorhinn fyeclyy and
Swarty x Helpfuinsss
Summarize
penersinn Cenartor Lif sone ieplerient bny kutarra sach small hhadding wiit NDS and
Hebplubszza
Raflacion
nearch
Rolution:
Seffreffaction: The iis emartad le incomed for sewarel reaicre... of the printial oel prosersee is Wiinwrabeiibe Bix
To porfor creal hauting HDS is costfal lasa sat Ealnwlnng
sufficinn
Revise
def ciresto perses
retare
offiber imatt. perworkont)
radle tiluurrofff analid parmeter lay: Dagl*)
Fermann The
Crrestze the censectary
teise solart (4d Tauskal mail homel")
Aeee The sealt aetress
reture: thi
Suarky x Halpftnssss
Becuriyy xt Helofannoss Y
Seauriy
V Happllr
Figure 9: Qualitative comparison between our method and related baselines
TASK: Verie funciion that auherticannss session tabib,, suttoves user anformation using the acceso Joiin.
TASK: Veile a Jana mathod that takes string paramater the 1ID5 saatt the passward using the stan sand Jova API
TASK: itual program that procesass command
not based on the presence of the -tarpel Bag build scrist should print the varsion number of Hust...
Fartur slaw chamested coine agaentie wed Sou garmin
Fachow deeating from the original laok requlemennss and fakingg recover from rossibiee
Falurr due undistected aucurily peotilma and now hojpfahasss insights from cxtic agentss
redrovied Anowleedge
Acter: Solution:
Acton Soiudors "python......................"
Aall creric string accrittiscrong pacoertd
Safely Crttc: prosleed solution posss polord rie excies seass Fivily sarumes piecasss ofite derple eonourati ('uaac) the sarkri caraiery...
sniffe du cuting usen ny Dhuch 220 prowit
taar ea deroishiterrii for liats ee puedd beeii ciee lis
Cilicc withi npt inbiing for pertiimaatin oi might apprrtcchi accappaalle
eeyich
Weo sisermee ty work involrndt
dovslarigu mocure for the intrmell Vadk Wenk
Security i Hep e
Helptainses Oritie:
The provited Fue ccludion
Minh
ewvekyeeee cangen trum
proceiiig commmard lna argumenti la dukome no
Ho Helphieness Criti
The penuieei solution
Safety Critcc
scoreh
de damorazado some fandamemaa mail axpacia Cruth 20 raaiin....
coveaanniion touch which ich anmieestllyyy iroccnn vika
tearch
spatch
prssoor onay devnape afould coconber..
When hanaing Orlah lowas aberage ao/Pt foloww
MCS is widbly known cypingmeete rust fxnciin fowewrr lu no longerr conaideved secive...
Revite
132
01 Oomant
Farsse teweeeeeeeeeeeeeeeeeeee teterreeeeeteeettt jeturnteerrr Hartee
salpfahess Ortic: perfermancs and scourncy hach of ths equi uli
ecscitee mituming:
ineetl and loo cereer
2009 1igliic
la Cengrree myaee nand inaiee lentle) srai
prosseer
Revitee
unde zat aerteriaatio cole a dffivoee
class Nasivel Slacker
efw trrtitaat/rratte Rust serseerry
skarts 10 Phe
teiditatinnee cat butte
Suasrty: Holplahnsss:
Securby: X
Helgfulrasss x
Soculiy; x Holptansss:
Figure 10: Qualitative analysis of failure cases when applying our method
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction correctly reflect the paper's contributions and scope, including the details of the proposed method INDICT and strong experimental results code generation tasks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitations of the work in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
31

---

Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
For reproducibility, we fully described our experimental setups (base language models, benchmarks, generation and model configurations, etc.). Please refer to and Appendix E for more details. We will also release our code to reproduce the results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
32

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: we fully released all code and related scripts to replicate the experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
pub public//a gwwcc//gueeeeeeeee /ooeee//oooSSoommmcc swwonsoliicgoooooooo for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips cc/ httppsscccppbbccccceccccccc ctabliccguidee
onP ciooooiopiiyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provided all the necessary details related to our experiments, including the datasets/ benchmarks, model configurations, hyperparameters of our proposed method, etc. Please refer to 4 and Appendix for more details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We did not report the error bars because would be too computationally expensive, involving running LLMs of up to 70B parameters on large benchmarks.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
33

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We provided sufficient level information on the compute resources used in this work (including the generation time, GPU types and quantities, etc.) reproduce the experiments. Please refer to Appendix for more details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httt:p/s/.aauurp....cc/////////a
Answer: [Yes]
Justification: The research this work conforms in all aspects with the NeurlPS Code of Ethics. Please refer to Appendix B Ethical Statement for more details.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We included both potential positive societal impacts and negative societal impacts of our work in Appendix C
34

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymemt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We described necessary safeguards for the responsible use of our method. Please refer to Appendix for more details.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the datasets and models used in this work are properly cited throughout the paper, including the list of the corresponding original creators/ owners. The licenses of all datasets and models are also fully described Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
35

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assett are released, the license, copyright information, and terms use the package should be provided. For popular datasets, saperswithhee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: N/A
Guidelines:
36

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "D4huJltQ1r", "reviewer": "Reviewer_GLux", "summary": "This paper presents a new framework called INDICT that employs two complementary critic agents to improve both the safety and helpfulness of LLM-generated code. Each critic agent is obtained by prompting an LLM with task-specific instructions and knowledge obtained from external tools such as web search and Wikipedia. To better generate knowledge-grounded critics, the authors propose a thought-action-observation process where the critic agent first generates the initial critic without external knowledge, then predicts search keywords or code snippets for knowledge retrieval based on the initial critic, and finally executes the external tools to retrieve the knowledge. The two critic agents can interact with each other by including the critics generated by the other agent in the current turn and past interactions in the prompt. To avoid computation overhead, INDICT also includes a summarization agent to summarize the previous interactions and uses the summary as the context for the next turn. INDICT is evaluated on eight different tasks across eight programming languages from five benchmarks. The results show that INDICT can achieve better results compared with vanilla LLMs such as GPT-4 and Llama-3. The authors also did an ablation study to show the effectiveness of using two critic agents vs. one agent and incorporating critics after the initial code generation vs. after code execution.", "questions": "1. How well does INDICT perform compared with the stronger baselines mentioned above?\n2. What is the effectiveness of using a summarizer?\n3. How much improvement does the thought-action-observation process achieve compared with existing RAG and tool-enhanced methods? \n4. Does INDICT use zero-shot prompting or few-shot prompting in each step?", "limitations": "The discussion on limitations looks reasonable. It would be helpful to have some quantification on the computation cost.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The idea of using two complementary critic agents and using a summarizer to avoid the computation overhead is new and interesting. \n2. The proposed framework is evaluated on multiple benchmarks, tasks, and programming languages.\n3. The results demonstrated the effectiveness of using two critic agents.\n\n==Post Rebuttal Comment===\n\nThe additional results provided in the rebuttal look promising and I believe this work can be strengthened by including those results. However, due to the time limit and also the lack of details in the rebuttal response, I am not able to make a full assessment of the correctness of the results.", "weaknesses": "1. My main concern about this work is the weak comparison baselines. In the evaluation, INDICT is only compared with vanilla LLMs rather than advanced and relevant prompting methods such as CodeRL, Self-Correct, and Self-Collaboration. While these prompting methods are initially designed for a single criterion such as code helpfulness, the authors can simply include an additional instruction about the other criterion in the original prompts of these methods to instruct the LLM to give two kinds of critics. This sounds like a more realistic baseline. It would be interesting to see to what extent combining two separate critic agents via the sophisticated interaction proposed by this work outperforms simply mentioning two critic criteria in a SOTA self-critic (or self-refinement) pipeline.  \n2. The ablation study does not really examine the effectiveness of the novel technical bits proposed in this work. Compared with simply combining two critic agents as in a multi-agent framework, a novel idea in this work is to use a summarizer to avoid computation overhead. However, this summarizer is not evaluated. The authors should compare INDICT vs. INDICT without a summarizer.\n3. The thought-action-observation is also an interesting idea that should be evaluated in the ablation study. How much improvement does this process achieve compared with existing RAG and tool-enhanced methods? \n4. Does INDICT use zero-shot prompting or few-shot prompting in each step? Based on the prompts in Appendix F, it seems INDICT uses zero-shot prompting in each step. However, this design is not very convincing for certain steps like generating relevant text queries or code snippets. Without few-shot demonstrations, it is hard to restrict the LLM to generate valid outputs. So I was wondering how exactly the authors did that. Also, how did the authors extract search keywords from the LLM response?\n5. The justification for using a GPT evaluator is weak. While one or two published papers may have adopted that method, it does not mean that it is a rigorous evaluation method and should be followed broadly by the community. To improve the rigor of evaluation, the authors should consider performing a manual analysis of a small set of samples.   \n6. The analysis of the experiment results is shallow and does not provide deep insights like why INDICT performs better than other methods and where INDICT fails. The authors should add a failure case analysis. \n7. In the related work, the authors should provide a more detailed discussion of how INDICT differs from existing self-critic approaches and multi-agent frameworks. The current discussion is short and handwavy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! Please refer to our responses below to your questions. \n\n### Q1: …In the evaluation, INDICT is only compared with vanilla LLMs…\nFollowing your recommendations, we selected 7 strong baselines from different research lines (see our global response #2 above). We also include a version where additional instructions are given to models to provide both safety and helpfulness critics e.g. instruct models to “focus on both the security and helpfulness of the solution.” For multi-agent methods, we included these instructions in all agents (analyst, tester, etc.) or introduced a new critic agent (as recommended in CAMEL's appendix). Note that for all baseline models, we followed similar generation budgets to fairly compare the results (up to 3 rounds of revision). Our results demonstrate the SoTA performance of INDICT against all the baselines. While we observed the improvement of baseline methods with additional instructions (marked with the suffix ‘+’), their results are sub-optimal. We will include all results and a more detailed analysis in our final revision of the paper. \n\n### Q2: …The authors should compare INDICT vs. INDICT without a summarizer.\nWe conducted a new ablation to evaluate the summarizer (see our global response#3 above). We simply removed the summarizer and let the actor agent receive the full dialogue history. From the results, we noticed the performance degraded to 87% and 72% in safety and helpfulness. This happens probably due to the much longer context of the dialogue history, affecting the actor agent to capture all critic feedback from this history and generate new code. This model variant also incurs more computation due to the long context of the dialogues.  We will include the results and more detailed analysis in our revised paper.  \n\n### Q3: How much improvement does the thought-action-observation process achieve compared with existing RAG and tool-enhanced methods?\nWe conducted additional experiments (see our global response#3 above) with two simple variants: (1) RAG-based critics which use the original task description to retrieve relevant knowledge and generate augmented critiques; and (2) tool-based critics which directly use external tools to query “what is wrong with the solution in terms of its <security/functionality>?”; the query output is then treated as a critic feedback. The results show that these variants are inferior to our proposed framework. We found that the queries in these methods are often too vague or ambiguous to search for meaningful information snippets. \n\n### Q4: Does INDICT use zero-shot prompting or few-shot prompting in each step? … \nINDICT uses zero-shot prompting in each step. We prompt the critic agent to condition the current critique and generate a unique query to obtain more knowledge. We extract the search keywords following our instruction templates e.g. in the form of 'Search[keyword]'. For generating code snippets, we prompt the model similarly but ask the model to wrap the output code in ```. Note that the current prompts in the Appendix (Listing 4 and 5) are for tool-finetuned models like CommandR which automatically generates tool parameters given a tool calling function definition. We will include more prompts for these steps for other standard models in our revised paper. \n\n### Q5: ...the authors should consider performing a manual analysis of a small set of samples.\nTo supplement the GPT-based results in helpfulness, we manually evaluated a small set of 40 samples from CyberSecEval1 (5 random samples per language). From the Table below, we found that even though GPT predicted results are often higher than human evaluation, the observation of INDICT's better performance still stands against the baselines. \n\n| **Baseline** | **Evaluation** | **INDICT wins** | **Draw** | **Baseline wins** |\n|:---:|:---:|:---:|:---:|:---:|\n| **Direct Gen** | Human | 74.8 | 8.5 | 16.7 |\n|  | GPT | 80.2 | 0.3 | 19.5 |\n| **Reflexion+** | Human | 57.1 | 4.2 | 38.7 |\n|  | GPT | 58.4 | 0.0 | 41.6 |\n\nNote that the security metric is done by the code analyzer from [1] which already demonstrated a high level of accuracy in detecting security issues. \n\n### Q6: …provide deep insights like why INDICT performs better than other methods and where INDICT fails.\nFrom our additional results (see our global response#2 and #3 above), we noticed that the best baseline models (Self-repair+ or Reflexion+) are as good as an ablation method with basic tool-based critiques where the external tool is vaguely queried: “What is wrong with the current solution?”. This shows that the quality of the critic feedback in the current methods is not good enough, simply focusing on shallow reviews of generated solutions. In INDICT, we enable the model to analyze and choose what information snippets to query and how to integrate this information into their critiques (e.g. queries about an uncommon security term or certain coding best practices). Combined with our multi-critic collaboration approach, INDICT demonstrates strong empirical results against other methods.\n\nFor qualitative results on how INDICT performs better and example failure scenarios, please refer to our attached PDF in the global response above. We will include more analysis in our revised paper. \n\n### Q7: ... how INDICT differs from existing self-critic approaches and multi-agent frameworks.\nPlease refer to our global response#1 for a systematic comparison of INDICT and the related work, including the self-critic approaches and multi-agent frameworks. We reviewed each method by the following features: helpfulness or safety-based qualities of generation outputs, execution feedback (execution of output code if applicable), tool-enhanced feedback (access to external tools like web search), multi-critic collaboration (engage multiple LM agents for critic generation) and supervision free (no training data required). We will include the full details of our comparison in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "klmztF9bSy", "reviewer": "Reviewer_SdG3", "summary": "This paper proposes a framework for generating both safe and helpful code. It integrates an internal dialogues of critiques against the given task and the corresponding generated response. It queries external knowledge through relevant code snippets and tools like web search and code interpreter. INDICT is evaluated on multiple tasks across 8 different languages. The results show an advanced level of critiques could significantly improve the code quality.", "questions": "1. Can you provide more details on how your method differs from existing actor-critic architectures? Specifically, what new contributions does your framework make compared to previous works?\n2. How do the safety and helpfulness critics contribute to the overall performance of the framework? Are there any specific examples or case studies you can provide to illustrate their impact?", "limitations": "Yes", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "This paper presents a very technically sound system for generating safer code. It improves the safety of code generation with both preemptive and post-hoc feedback, which is quite complete. The authors conduct solid and thorough experiments for their system, which proves significant improvement over existing methods.", "weaknesses": "This paper presents a useful and well-designed framework. However, I have some doubts about the novelty of the proposed method. The framework uses actor-critic architectures and includes safety and helpfulness critics, which have been mentioned in previous papers like [1]. \n\nIn this framework, the critics use data from web searches, Wikipedia, and OpenAI, which raises my concerns about data leakage. For example, it might be easy to find the original web page for a CWE by searching its description when evaluating on the datasets built from CWEs. Using OpenAI as a knowledge base could also have similar issues. Because of this, while the framework performs well on benchmarks, I’m concerned about the generalizability of this framework. \n\nBesides, as the paper mentioned, the code execution could invoke unexpected consequences. It seems the authors do not address this issue, which might require sandboxing execution or other safe execution methods.\n\n[1] Le, H., Wang, Y., Gotmare, A. D., Savarese, S., and Hoi, S. C. H. (2022). Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314–21328.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: …how your method differs from existing actor-critic architectures?...\n\nWe provided a systematic and comprehensive comparison between INDICT and related actor-critic approaches such as CodeRL in our global response#1 above. Compared to existing actor-critic methods, INDICT is different in three major aspects: \n* (1) INDICT aims to optimize both helpfulness and safety awareness in the generated output code. Most of the current actor-critic approaches are designed with a single criterion (such as functional correctness). Simply extending these methods with additional instructions on safety criteria is sub-optimal (see our results with baseline actor-critic methods in our global response#2 above). \n* (2) INDICT integrates critic with knowledge grounding from external tools to create more reliable feedback to the actor agent. Most current methods only use code test results as the only external feedback to improve the quality of output code. \n* (3) To implement (1) and (2), we enhanced the existing actor-critic framework with a multi-critic and tool-enabled collaboration approach. This approach can autonomously generate more reliable and holistic feedback for both the safety and helpfulness of output code generation. \n\nWe will include a more detailed comparison in our revised paper. \n\n### Q2: In this framework, the critics use data from web searches, Wikipedia, and OpenAI, which raises my concerns about data leakage…Because of this, while the framework performs well on benchmarks, I’m concerned about the generalizability of this framework.\nThanks for raising this concern. We want to highlight that our use of external tools is only for querying relevant information snippets to supplement the critic feedback, not to directly find a solution to a given task. We do not provide any additional information to the critic to generate queries e.g. no information about the related CWEs or security issues, so the critic would have to independently decide what information to choose to search. We conducted an analysis of the generated tool queries by the critic agents and found that on average, for each task, words that are found in queries only overlap with less than 5% of the total words in the original task description. Therefore, the data leakage is very minimal. \n\nFurthermore, from our experiment results (Table 2 in the paper), even without access to external tools, our approach with a multi-critic collaboration system still leads to significant performance gains in base models like CodeLlama or CommandR. When stronger and more recent GPT models are used (e.g. results with GPT4o-mini in our global response#2), the direct generation result is still rather low i.e. models are not heavily exposed to the test data, and more improvement could be done with methods like INDICT. We will include a more detailed discussion in our revised paper. \n\n### Q3: Besides, as the paper mentioned, the code execution could invoke unexpected consequences…\n\nThank you for your suggestion. Sandbox environment is a technical engineering feature that we would like to integrate with INDICT. However, the novelty of our method should still be guaranteed. We will describe in more detail the requirement for a sandbox environment for secure code execution in our revised paper. \n\n### Q4: How do the safety and helpfulness critics contribute to the overall performance of the framework? Are there any specific examples or case studies you can provide to illustrate their impact?\n\nAs demonstrated in our ablation experiments in the paper (see Table 3), using individual component critics leads to sub-optimal performance and is not as good as our method with a multi-critic collaboration system. To illustrate the benefits of our approach, we included a qualitative example in the PDF attached to our global response above (see Figure 1). \n\nCompared to the baseline methods, our approach can simultaneously improve both the helpfulness and safety of the output code generation. Specifically, given a relevant information snippet by the safety critic (about the hashing method SHA-256), our actor agent correctly revised the code with a more secure hashing method, avoiding using MD5 hashing and the common security risk CWE-328. At the same time, our generated code is generally more helpful with properly modularized functions implementing supporting features such as input validations. As we noted, this feature has generally emerged in code solutions by collaborative agent systems like CAMEL and INDICT. We will explain the qualitative samples in more detail in our revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cKcmxkOU5h", "reviewer": "Reviewer_rV62", "summary": "The paper describes a method to improve the helpfulness and safety of LLMs for code generation tasks. It uses two critics - one for safety and one for helpfulness that communicate with each other to iteratively provide feedback to the actor model or the actual agent that is tasked with the code completion task. The critics are further augmented with tools that allow using the web for searches as well as LLMs from OpenAI and a code interpreter to execute code. The implementation of the method has been done using CommandR+ though experiments have also been presented using CodeLlama and Llama for certain tasks. Prompts have been standardized across LLMs to the extent possible. Experiments on benchmarks for Code Security (insecure code generation, malicious code generation) as well as open-ended generation tasks from HarmBench indicate the method improves the performance of  the corresponding base model. Ablation experiments on the code security tasks (done using CodeLlama and CommandR+ indicate that the use of both critics as well as tools help improve performance.", "questions": "- For the HarmBench evaluation - how have the red-teaming methods been applied with the critics in place? I'm guessing it was applied on the actor/agent prompt? Some details would be helpful.  Was the evaluation also done using completions?  The main text suggests so, but there are no tables/results referenced. Table 1 isn't discussed (no mention or reference of the red-teaming methods in the main paper). This evaluation section is a bit unclear to me. \n- Since there are no samples of the generated dialogs or the system prompt for the actor (agent) -- if any; I'd imagine a well rounded summary being made available to the agent (Equation 5). Could the authors share more details/prompt along with a sample?\n     * I was curious if fine-tuning models (smaller models?) on such generated data would have been a good experiment? Doing so could help in reducing inference cost by employing cheaper/smaller models", "limitations": "Yes for the most part.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Well written paper with a detailed appendix\n- Simple but interesting extension of the use of collaborative agents to improve safety and helpfulness for code tasks \n- Improved performance on multiple tasks", "weaknesses": "- Despite some details in the appendix; the evaluation section is missing crucial details\n- See Questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. Please refer to our responses below. \n\n### Q1: For the HarmBench evaluation - how have the red-teaming methods been applied with the critics in place? …Was the evaluation also done using completions?...\nFor the HarmBench benchmark, we followed the original evaluation setup proposed in [1] (see Figure 3 of this work). Specifically, the red-teaming methods like PAP, TAP, or PAIR are applied to the original task prompts (provided in HarmBench). Each red teaming method augments the original task prompts to make it harder for LM actor agents to detect malicious intentions, leading to harmful generation output. The evaluation is then done on the generated completions of the actor agents using the HarmBench AI evaluator. With the critics in place, the evaluation is applied to the actor agent’s completions after it receives the critic feedback and regenerates a new response. Due to the limited space, we described the details of the HarmBench benchmark and references of the red-teaming methods in Appendix D (L996-1003).  We will make the experimental setup and evaluation clearer in our revised paper. \n\n### Q2: …Could the authors share more details/prompt along with a sample?\nThanks for your comment. A well-rounded summary is beneficial to ensure all the information of the critic feedback is conveyed to the actor agent. We included the example summary prompt in Appendix F (Listing 3 and 8). Other prompts are also included in this Appendix. We also included more qualitative examples with generated dialogues in the attached PDF in our global response above for your reference. We will include these qualitative samples and explain them in more detail in our revised paper. \n\n### Q3: I was curious if fine-tuning models (smaller models?) on such generated data would have been a good experiment?...\nThanks for your suggestion. It would be interesting to conduct finetuning experiments on smaller models using generated data. Currently, there is no available training split in the benchmarks and tasks used in this paper. However, synthetic datasets or relevant crowd-sourced datasets could be considered (e.g. from open-domain Github tasks or other code generation benchmarks). \n\nFurthermore, while using fine-tuned models might be cheaper during inference, they will still be subject to unseen security problems or novel red-teaming malicious prompts. In our global response #3 with fine-tuning methods, the best finetuned model is far from perfect and its performance can be further optimized. Our method INDICT can complement these approaches. Our results show that using INDICT can significantly boost the performance of a fine-tuned model, from 60% (SFT+DPO model) to more than 73%. \n\n*[1] HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal*\n\n*[2] CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1zfx6uM434", "reviewer": "Reviewer_ZHvf", "summary": "This paper introduces a dual-critic prompting framework, INDICT, to consider both helpfulness and security during code generation. Specifically, the author introduces two critics, one for helpfulness and the other for security, to provide suggestions on improving the initially generated code. Besides, the authors equipped two critics with external tools like search engines and code executors to address the hallucination problem and employ an iterative refining style to further improve the performance. The authors evaluate INDICT on 8 tasks across 5 benchmarks on several LLMs. The experimental results demonstrate the effectiveness of INDICT.", "questions": "1. What is the meaning of 'GT Safe' and 'GT Unsafe' in Fig.4 (c)?\n2. Can you provide the time cost of INDICT? Since your system contains multiple LLMs, multiple rounds of interaction, and even the usage of search engines and code executors, it is necessary for readers to know the efficiency trade-off compared with pure LLMs. Note that I am not taking this point as a weakness of INDICT.", "limitations": "The authors discussed limitations in Appendix A. Among the discussed limitations, I think the last one needs further discussion. Though the authors discussed that INDICT is much more efficient compared with fine-tuning LLMs that require curated training examples, once the fine-tuned LLM is released, the inference time costs much less than INDICT because INDICT is quite heavy.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n2. The studied topic is valuable. According to authors' discussion of related works, it seems that there are many works focusing on pointing out the security problems of LLM-generated codes, yet the papers for solutions are few.\n3. The ablation studies demonstrate the effectiveness of each module (helpfulness critic, security critic, external tools, iterative rounds).", "weaknesses": "1. Though no former works using a multi-agent collaborative system addressing the security of code generation according to related works part in the paper, it is generally believed and widely proven effective of adopting a multi-agent collaborative system during content generation [1,2,3], which weakens the novelty of the paper.\n2. The experiments only compare INDICT with pure LLMs. According to the related work part, there are several related works. Although the authors mentioned that related works are either not adapted for benchmarks used in this paper or not accessible, a lack of comparison with related works weakens the validity of the effectiveness of INDICT.\n3. The authors discussed an alternative by using only one LLM to act as a helpfulness critic and security critic at the same time in their method description section. However, I do not find any experiments comparing using two LLMs as critics and using only one. The lack of this comparison confuses me with the necessity of employing two LLMs separately.\n\n[1] Dong, Yihong, et al. \"Self-collaboration code generation via chatgpt.\" arXiv preprint arXiv:2304.07590 (2023).\n\n[2] Huang, Dong, et al. \"Agentcoder: Multi-agent-based code generation with iterative testing and optimisation.\" arXiv preprint arXiv:2312.13010 (2023).\n\n[3] Ishibashi, Yoichi, and Yoshimasa Nishimura. \"Self-organized agents: A llm multi-agent framework toward ultra large-scale code generation and optimization.\" arXiv preprint arXiv:2404.02183 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your reviews! Please refer to our responses below. \n\n### Q1: ..It is generally believed and widely proven effective of adopting a multi-agent collaborative system during content generation \nEven though multi-agent collaborative systems have been proposed for content generation, it is not trivial to extend this line of research to address the security of code generation. \n* Firstly, it is not clear how the current methods could be enhanced with security awareness and subsequently improve the quality of output generations. Earlier work such as [1, 2] showed that simply asking agents to analyze and answer what is wrong with generated code is not always effective. With carefully designed prompts and agent interactions [3, 4], collaborative agents can now generate more functionally correct code. \n* Therefore, studying collaborative agents with orthogonal goals such as security awareness still requires further attention. As observed by our additional experimental results (see our global response#2 above), simply applying the current multi-agent methods [5, 6], even with extended additional instructions of security criteria, does not perform so well and is still far from optimal. \n\nFurthermore, we would like to emphasize that our method novelty is not only about the multi-agent collaborative system. We also proposed an innovative multi-critic framework as an internal reasoning module with access to external tools for knowledge grounding. Our strategy can integrate holistic and reliable critic feedback from code execution outputs as well as supporting information snippets through multiple rounds of agent interactions. As demonstrated by our results (see our global response#2 above), our method is more well-rounded with high performance by both safety and helpfulness. Refer to our global response#1 above for a more comprehensive and systematic comparison between INDICT and other related work.  \n\n### Q2: The experiments only compare INDICT with pure LLMs.…\nFollowing your recommendations, we selected 7 strong baselines from related lines of research, including self-refine, multi-agent, and finetuning methods (see our global response#2 above). For most baselines, we also include a version of the method where additional instructions are given to the model to follow and provide both safety and helpfulness feedback e.g. we instructed models to “focus on both the security and helpfulness of the solution.” For Self-Collab [5], we included these instructions in both analyst and tester agents. For CAMEL [6], we follow the Critic-in-Loop setup as recommended in the paper appendix. Note that for all baseline models, we followed the same generation budgets to fairly compare the results (up to 3 rounds of revision).\n\nFrom the experiment results (see our global response#2 above), we can observe the SoTA performance of INDICT. While we observed the improvement of baseline methods with additional instructions (marked with the suffix ‘+’), their results are still not as good as INDICT in terms of helpfulness and safety. For finetuning method CodeUltraFeedback, their best model (SFT+DPO fine-tuned) is still far from perfect and could be further optimized. We further integrated the fine-tuned models with INDICT and observed a significant performance boost. We will include these results and more detailed analysis in our revised paper. \n\n### Q3: The authors discussed an alternative by using only one LLM to act as a helpfulness critic and security critic at the same time...\nWe conducted additional ablation analysis to evaluate whether one LLM can act as a critic for both helpfulness and safety at the same time. We combined the previous criteria for security and helpfulness and integrated them into the prompt for this critic agent. From the results (see our global response #3), simply using a single critic agent with dual quality criteria will affect the performance, reducing the safety and helpfulness metrics to 87% and 76% respectively. One possible reason is due to the formulation of the training objectives of LMs, which are not always designed to optimize both security and helpfulness equally (also depending on the post-pretraining stages of LMs e.g. training with RLHF). Our approach enables a more flexible and probably more relaxed application of LLM as a critic agent by: \n* (1) decoupling the helpfulness and safety goals and delegating them to individual LM agents; and\n* (2) enabling multi-critic collaboration to autonomously develop more holistic and well-rounded critic feedback. \n\nWe will include the results and a more detailed analysis in our revised paper. \n \n### Q4: What is the meaning of 'GT Safe' and 'GT Unsafe' in Fig.4 (c)?\n“GT Safe” and “GT Unsafe” denote the ground-truth secure and insecure code outputs provided by the CVS benchmark. We will explain and make the definitions clearer in our revised paper. \n\n### Q5: ...Can you provide the time cost of INDICT? ... once the fine-tuned LLM is released, the inference time costs much less than INDICT\n\nOn average, INDICT incurs about 3 to 4x the time cost as compared to pure LLMs. We also want to note that even with fine-tuning, fine-tuned models are far from perfect and still subject to unseen security risks or novel red-teaming prompts during test time. For instance, from our results with the fine-tuning method CodeUltraFeedback (see our global response#3 above), the fine-tuned model is still sub-optimal and can be further improved e.g. using INDICT during inference time (improving the performance from 60% to 73%). \n\n*[1] Is self-repair a silver bullet for code generation?*\n\n*[2] Large language models cannot self-correct reasoning yet*\n\n*[3] Reflexion: Language Agents with Verbal Reinforcement Learning*\n\n*[4] AgentCoder: Multiagent-Code Generation with Iterative Testing and Optimisation*\n\n*[5] Self-collaboration Code Generation via ChatGPT*\n\n*[6] CAMEL: Communicative Agents for \"Mind\" Exploration of Large Language Model Society*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for providing insightful comments on our paper. Please refer to this global response for our high-level answers to the common concerns. For more detailed explanations and analysis, please refer to the corresponding threads of individual reviewers. 

### 1. Comparison with related baselines and our contributions 
In the table below, we provided a more comprehensive and systematic comparison of our method to related work. We compared INDICT and related methods from 3 directions: self-refine/self-critic, multi-agent, and finetuning. Compared to these methods, INDICT is a more well-rounded generation framework with the following contributions: (1) integrates code execution-based feedback and enhances them with external knowledge, (2) targets both helpfulness and safety of output code, and (3) facilitates an interactive and supervision-free multi-agent collaboration framework. Our additional experiment results (see the next response) showcase the efficacy of INDICT.

| Method | Helpfulness | Safety | Execution feedback | Tool-enhanced | Multi-critic collab | Supervision free |
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| **_Self-refine approach_** |  |  |  |  |  |  |
| CodeT, AlphaCode, MBR-Exec | ✅ |  | ✅ |  |  | ✅ |
| Self-correct, ILF | ✅ |  |  |  |  | ✅ |
| CodeRL, Self-edit | ✅ |  | ✅ |  |  |  |
| Self-repair, Self-debug, Reflexion | ✅ |  | ✅ |  |  | ✅ |
| **_Multi-agent approach_** |  |  |  |  |  |  |
| Self-collaboration, AgentCoder | ✅ |  | ✅ |  |  | ✅ |
| CAMEL | ✅ |  |  |  |  | ✅ |
| ChatDev, Self-org Agents | ✅ |  | ✅ |  | ✅ (?) | ✅ |
| MetaGPT, AgentVerse | ✅ |  | ✅ | ✅ |  | ✅ |
| **_Finetuning approach_** |  |  |  |  |  |  |
| CodeUltraFeedback, StableAlignment | ✅ | ✅ |  |  | ✅  |  |
| SafeCoder | ✅ | ✅ | ✅ |  |  |  |
| **INDICT** | **✅** | **✅** | **✅** | **✅** | **✅** | **✅** |

### 2. Additional experiment results to compare with stronger related baselines
From the above table, we selected representative baselines and evaluated them on a validation test split (random samples of 20% of the CyberSecEval benchmark). With GPT4o-mini as the base model, we adapted the baselines in their original implementation and also extended them with additional instructions (detailed criteria of safety and helpfulness). We marked these enhanced baselines with the suffix ‘+’. We observed that INDICT can lead to SoTA performance by both security and helpfulness (more than 90% and 81% respectively). Strong baselines like Reflexion+ and CAMEL+ improve with additional critic instructions but are not as strong as INDICT.  
| **Method** | **Safety** | **Helpfulness** | **S+H** |
|---|:---:|:---:|:---:|
| Direct Gen | 78.2 | 50.0 | 64.1 |
| INDICT | **90.9** | **81.4** | **86.1** |
| **_Sefl-refine methods_** |  |  |  |
| Self-debug | 80.0 | 52.7 | 66.3 |
| Self-debug+ | 79.7 | 53.9 | 66.8 |
| Self-correct | 80.7 | 59.7 | 70.2 |
| Self-correct+ | 86.7 | 68.5 | 77.6 |
| Self-repair | 83.7 | 69.6 | 76.6 |
| Self-repair+ | 86.6 | 70.9 | 78.8 |
| Reflexion | 83.3 | 68.5 | 75.9 |
| Reflexion+ | 86.9 | 69.6 | 78.2 |
| **_LM agent methods_** |  |  |  |
| Self-collab | 78.7 | 52.3 | 65.5 |
| Self-collab+ | 79.1 | 66.2 | 72.7 |
| CAMEL | 81.6 | 63.7 | 72.6 |
| CAMEL+ | 82.6 | 70.2 | 76.4 |

We also conducted experiments to compare INDICT with finetuning-based methods. Using CodeLlama-7b-instruct as the base model, CodeUltraFeedback finetunes the model on a large-scale dataset with annotations of code preferences. We observe that the best model (SFT + DPO finetuning) can improve the results by both safety and helpfulness but not as good as INDICT. As INDICT can complement finetuning-based methods, we applied INDICT with the best CodeUltraFeedback model to achieve even further performance gains (from 60% and 63% to 73%).

| **INDICT vs. finetuning methods** | **Safety** | **Helpfulness** | **S+H** |
|---|:---:|:---:|:---:|
| Direct Gen | 56.3 | 50.0 | 53.2 |
| INDICT | 65.3 | 62.1 | 63.7 |
| CodeUltraFeedback (SFT) | 58.5 | 49.9 | 54.2 |
| CodeUltraFeedback (DPO) | 62.7 | 56.0 | 59.3 |
| CodeUltraFeedback (SFT+DPO) | 63.9 | 57.9 | 60.9 |
| CodeUltraFeedback (SFT+DPO) +INDICT | **74.9** | **72.4** | **73.7** |

### 3. Additional ablation results to demonstrate the effectiveness of the proposed components 
Using GPT4o-mini as the base model, we conducted additional ablation experiments with different variants of INDICT: (1) one simply using a single critic agent for both safety and helpfulness; (2) one without using a critic summarizer and maintaining a full dialogue history of critiques in the critic context; (3) ones replacing the thought-action-observation critic generation with RAG or tool-based generation: (3a) RAG uses the original task description to retrieve relevant knowledge and generate grounded critics, and (3b) tool-based method uses web search/Wikipedia and a query “what is wrong with the solution in terms of its <security/functionality>?” and query output is treated as a critique. The result table below shows that our proposed INDICT obtains the optimal performance, with a more well-rounded performance in both safety and helpfulness. 
| **Ablation methods** | **Safety** | **Helpfulness** | **S+H** |
|---|:---:|:---:|:---:|
| INDICT (full) | **90.9** | **81.4** | **86.1** |
| - one critic for both criteria | 87.3 | 76.4 | 81.9 |
| - no critic summary | 87.9 | 72.2 | 80.1 |
| - RAG-based critics | 87.9 | 74.4 | 81.1 |
| - tool-based critics | 85.5 | 72.7 | 79.1 |

### 4. Qualitative analysis to demonstrate the benefits of our methods on generated code and analyze failure cases 
To address the concerns about the benefits of INDICT and where INDICT may fail, we demonstrated some qualitative analysis in the attached PDF document. In Figure 1, we show that given a code generation task, INDICT can generate code that is more secure and robust than strong baselines (Direct generation, Reflexion, and CAMEL). In Figure 2, we illustrate cases where INDICT may fail due to nontrivial errors.

Meta-review: This paper presents a technique for improving the quality and safety of code generation systems. By using two separate critics (one for helpfulness, one for safety) with access to each other and endowing them with access to information (via search tools) a multi-turn code generation system is shown to produce more accurate and safer code on a range of benchmarks.

All reviewers agreed that the approach is novel and useful. The evaluations, including those provided in the rebuttal, are extensive and show sufficient evidence about the benefits of this approach. As reviewers mention the cost-related tradeoffs are not well-discussed or evaluated. In particular the critics have additional cost and represent additional "thinking" (compute) done on each problem. Equi-cost comparisons of the various methods, ablations, and baselines would resolve the question about the extent to which INDICT's double-critic approach works because of its design or because of the additional compute available to the approach.

Despite the above, all the reviewers believe that there is no sufficient reason to reject this work and hence recommend its acceptance.

The authors are kindly asked to incorporate the results discussed in the rebuttal and -- at minimum -- a discussion about the costs of INDICT to be included.
Decision: Accept (poster)