INDICT: Code Generation with Internal Dialogues of Critiques for Both Security and Helpfulness

Authors: hung  le
yingbo  zhou
caiming  xiong
silvio  savarese
sahoo salesforce research
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 08:55:05

Abstract: Large language models (LLMs) for code are typically trained to align with natural language instructions to closely follow their intentions and requirements. However, in many practical scenarios, it becomes increasingly challenging for these models to navigate the intricate boundary between helpfulness and safety, especially against highly complex yet potentially malicious instructions. In this work, we introduce INDICT: a new framework that empowers LLMs with Internal Dialogues of Critiques for both safety and helpfulness guidance. The internal dialogue is a dual cooperative system between a safety-driven critic and a helpfulness-driven critic. Each critic provides analysis against the given task and corresponding generated response, equipped with external knowledge queried through relevant code snippets and tools like web search and code interpreter. We engage the dual critic system in both code generation stage as well as code execution stage, providing preemptive and post-hoc guidance respectively to LLMs. We evaluated INDICT on 8 diverse tasks across 8 programming languages from 5 benchmarks, using LLMs from 7B to 70B parameters. We observed that our approach can provide an advanced level of critiques of both safety and helpfulness analysis, significantly improving the quality of output codes (+10% absolute improvements in all models).

Main paper: 
 introduction: 
 Extending from the natural language domain, Large Language Models (LLMs) like [Koubaa, 2023 , Wang and Komatsuzaki, 2021 , Radford et al., 2019] have demonstrated great potential in code generation tasks [Svyatkovskiy et al., 2020 , Chen et al., 2021 , Hendrycks et al., 2021] . However, when instructed with tasks containing malicious intentions or ambiguous requirements, LLMs are subject to generating code that could facilitate harmful attacks or code that contains obscure security problems [Khoury et al., 2023 , Bhatt et al., 2023 , Siddiq et al., 2022] . For instance, in a study of Github's Copilot, Pearce et al. [2022] observed that about 40% of generated programs are vulnerable.Despite previous efforts in addressing the safety of LLMs through finetuning [Bai et al., 2022 , Korbak et al., 2023 , Dai et al., 2024] , this strategy alone is often not sufficient and scalable enough against prompts that are increasingly optimised for highly sophisticated attacks [Zhuo et al., 2023 , Mazeika et al., 2024 , Bhatt et al., 2024] . Furthermore, in the domain of code generation, creating quality safety-related data for finetuning often incurs great costs, involving programming experts with a deep understanding of code and related cyber-security and vulnerability concerns.Note that code itself is often not inherently malicious. For example, as noted by Bhatt et al. [2023] , a program for an encryption method could be very useful to create a secure personal file system. Yet the encryption method can also be exploited for a ransomware attack. Therefore, it is important to develop an efficient method for LLMs to achieve the intricate balance between helpfulness and safety Table 1 : We compared INDICT and related methods from 3 directions: self-refine/self-critic, multiagent, and finetuning. Compared to these methods, INDICT is a more well-rounded generation framework with the following contributions: (1) integrates code execution-based feedback and enhances them with external knowledge, (2) targets both helpfulness and safety of output code, and (3) facilitates an interactive and supervision-free multi-agent collaboration framework. Our experiment results showcase the efficacy of INDICT. See Appendix D for cited references.Helpful. Safety Exec. feedbackMulti-critic collab Supervision free Self-refine approach CodeT, AlphaCode, MBR-Exec ✓ ✓ ✓ Self-correct, ILF ✓ ✓ CodeRL, Self-edit ✓ ✓ Self-repair, Self-debug, Reflexion ✓ ✓ ✓ Multi-agent approach Self-collaboration, AgentCoder✓ ✓ ✓ CAMEL ✓ ✓ ChatDev, Self-org Agents ✓ ✓ ✓(?) ✓ MetaGPT, AgentVerse ✓ ✓ ✓ ✓ Finetuning approach CodeUltraFeedback, StableAlignment ✓ ✓ ✓ SafeCoder ✓ ✓ ✓ INDICT (ours) ✓ ✓ ✓ ✓ ✓ ✓a massive amount of text data on very deep Transformer-based architectures, these models have shown impressive performance in many natural language tasks. Going beyond the text domain, LLMs have been extended to learn from the code data and applied to many coding tasks [Rozière et al., 2023 , Li et al., 2023 , Lozhkov et al., 2024 , Gunasekar et al., 2023 , Wang et al., 2023 , Nijkamp et al., 2023 , Luo et al., 2023] . One major application of LLMs in the code domain is code generation, a long-standing challenge of many conventional AI models [Manna and Waldinger, 1971 , Gulwani et al., 2012 , Kurach et al., 2015 , Devlin et al., 2017 , Parisotto et al., 2016] . In this task, an AI model is required to generate proper code solutions for different programming problems, ranging from basic daily code completion tasks to more advanced algorithmic problems [Chen et al., 2021 , Austin et al., 2021 , Hendrycks et al., 2021 , Shinn et al., 2023 , Lai et al., 2023] .In the research for code generation, we have witnessed emerging studies focusing on the security and safety aspects of AI-generated code. Hammond Pearce et al. [2021] , Schuster et al. [2021] , Pearce et al. [2022] found that commercially successful systems like Github's Copilot still led to obscure yet major vulnerability and security issues in code. More recently, Perez et al. [2022] , Zhuo et al. [2023] , Khoury et al. [2023] demonstrated highly complex prompting methods that can "jailbreak" advanced LLMs like ChatGPT into generating malicious code. To benchmark LLMs against code safety and security, [Siddiq and Santos, 2022, Tony et al., 2023] evaluated LLMs against common coding scenarios based on CWE3 . More recently, Bhatt et al. [2023 Bhatt et al. [ , 2024] ] introduced CyberSecEval, a large-scale benchmark containing different types of security-aware evaluations. They observed that the code outputs by powerful LLMs like Llama and GPT models are often not perfectly secure.More relevant to our work is the research to improve the safety or helpfulness of LLMs. A common strategy is finetuning LLMs with appropriate preference data with specific reward models to differentiate among ranked data samples [Bai et al., 2022 , Korbak et al., 2023 , Wu et al., 2024 , Sun et al., 2024 , Dai et al., 2024] . In the code domain, He and Vechev [2023] , He et al. [2024] proposed to finetune LLMs with prompt prefixes or masking strategies conditioned by the safety of corresponding code samples. Chen et al. [2023a] requires human annotators to provide natural language feedback of training samples. Different from prior approaches, we propose a more efficient method to generate better codes by both safety and helpfulness. Our approach can complement the research of autonomous LLM agents [Huang et al., 2022 , Yao et al., 2023 , Dong et al., 2023] and AI-generated feedback [Bahdanau et al., 2017 , Le et al., 2022 , Welleck et al., 2023 , Gou et al., 2024] .For a systematic comparison to related work, please refer to Table 1 and Appendix D. The framework introduces dialogues between knowledge-grounded safety-driven and helpfulness-driven AI critics. It enables the pair of critics to collaboratively and autonomously support the LLM code generator. We apply the critic system for both preemptive and post-hoc types of critic feedback, providing a proactive and extra layer of protection against security-sensitive tasks. 
problem definition: 
 Typically, in a code generation task, an LLM θ receives an input X, consisting of a natural language instruction and optionally, a related code context. Treating code generation as a sequence-to-sequence task, the LLM autoregressively generates a response Ŷ as a sequence of tokens. Each token ŷt is sampled from the parameterized condition distribution p θ (.|ŷ 1:t-1 , X) where ŷt ∈ V. The output can contain either natural language segments (e.g. explanation of the output code or refusal of the user request) as well as code programs (e.g. code snippets to complete the given input code context).Pretrained with a massive amount of data, LLMs are found to be capable of providing insightful feedback to self-improve their own responses in many downstream tasks [Shinn et al., 2023 , Zhang et al., 2023b , Welleck et al., 2023 , Madaan et al., 2023] . Rather than just a single critic for a specific code attribute, we propose to engage two critics with independent goals: a safety-driven critic σ and a helpfulness-driven critic ω. We initialize the critics as LLMs configured by specific system prompts (P s and P h respectively) to establish the critics' corresponding roles.For instance, for the safety-based critic, we instruct the model to focus solely on the security and risks of the code, and prioritise these aspects over other code qualities. Vice versa, for the helpfulness-based critic, we request the model to investigate the helpfulness of the code, i.e. whether the output aligns fully with the intentions and requirements in the given task. Denoting Ĉs and Ĉh as the complete outputs of the critics, we can define the critic output distributions (per token) as:ĉs,t ∼ p σ (.|ĉ s,1:t-1 , X, Ŷ , P s ) ⇒ for safety-driven critic (1) ĉh,t ∼ p ω (.|ĉ h,1:t-1 , X, Ŷ , P h ) ⇒ for helpfulness-driven critic (2)Subsequently, we let the code generation LLM ("actor") revise their solutions conditioned by the generated critiques: ŷs ∼ p θ (ŷ s,1:t-1 |X, Ŷ , Ĉs ) for safety-conditioned solutions and ŷh ∼ p θ (ŷ h,1:t-1 |X, Ŷ , Ĉh ) for helpfulness-conditioned solutions. Refer to Appendix I for the detailed instruction prompts we used on our critics to assess safety or helpfulness of model outputs.LLMs are often finetuned to follow natural language instructions [Ouyang et al., 2022 , Korbak et al., 2023 , Dai et al., 2024] and subsequently, can engage in natural language interactions with humans or even among other LLMs. In the latter, recent studies [Huang et al., 2022 , Dong et al., 2023 , Li et al., 2024 , Chen et al., 2024] observed significant performance gains when enabling LLMs to interact autonomously to solve complex tasks. We are motivated by this observation and propose an autonomous agent system of critic models to generate helpfulness-and-safety-aware critiques.Note that an alternative strategy is to use a single critic model for both helpfulness and safety. However, such a critic model often needs complex alignment finetuning or prompt engineering to generate critiques that are not significantly biased towards a single code property. In our approach, from 1 and 2, given an interaction turn r between critics, we can redefine the output distributions as: ĉr s,t ∼ p σ (.|ĉ s,1:t-1 , X, Ŷ , P s , Î1:r-1 ) ⇒ for safety-driven critic (3)ĉr h,t ∼ p ω (.|ĉ h,1:t-1 , X, Ŷ , P h , Î1:r-1 ⊕ Ĉr s ) ⇒ for helpfulness-driven critic (4)Where ⊕ denotes concatenation andÎ1:r-1 = Ĉ1 s ⊕ Ĉ1 h ⊕ . . . Ĉr-1 s ⊕ Ĉr-1h contains all the past interactions between the safety-driven and helpfulness-driven critics.Practically, to avoid computation overhead, we can limit Î to only the last few turns of interactions. Alternatively, in this work, we summarize the critic dialogue after each turn of interactions and only use the corresponding summary in each turn: Îr = f ( Î1:r ) where f (.) is parameterized as an LLMbased summarizer model. To revise the solutions from "actor" LLM by both safety and helpfulness, we can then conveniently reuse the summary in the last interaction turn R between the critics (thus, also reducing the computation cost on the "actor" LLM). To generate safety-and-helpfulness-aware outputs, we revise the output distributions of the LLM code generator as: ŷs+h,t ∼ p θ (.|ŷ s+h,1:t-1 , X, Ŷ , ÎR )(5)Thought:... Action:... Observation:... Safety Critic Thought:... Action:... Observation:... Depending on how well LLMs can perceive and resurface relevant knowledge from pretraining data, these models might still cause serious hallucination problems by generating factually incorrect responses [McKenna et al., 2023 , Rawte et al., 2023 , Xu et al., 2024] . These hallucination problems are exacerbated when LLMs play the critic roles, required to provide reliable and grounded responses against code generation outputs. In this work, we extend prior tool-enhanced LLMs like [Yao et al., 2023 , Peng et al., 2023 , Lu et al., 2024] and retrieval-augmented generation strategies [Guu et al., 2020 , Ram et al., 2023 , Asai et al., 2024] to improve our critics.Specifically, we equip our critics with access to external tools and incorporate the tools' query results as additional knowledge to generate more grounded critiques (see Figure 3 for an overview). For instance, for the safety-driven critic, from 3, we decompose the critic generation process to the following steps:1. Critic's thought Ŵ r s : ŵr s,t ∼ p σ (.|w r s,1:t-1 , X, Ŷ , P First, we obtain the critic's initial thought Ŵ r s , following the same formulation as in 3. In the critic's action step, we parameterize critic "actions" as the generation of unique textual keywords Qr s,text , optionally accompanied by code snippets Qr s,code . These are used subsequently as search queries to call external tools and obtain search results in the critic's observation step. Denoting function g(.) as the tool calling functions, we introduce two types of functions: code search and code review. Refer to Figure 3 for the specifications and examples of these functions and Figure 1 for demonstrations.Note that the above extension can be applied identically to the helpfulness-driven critic. We also then revise I as the summary of all past critics' initial thoughts concatenated with corresponding observations:Îr = f ({ Ŵ ⊕ Ô} 1:r-1 s ⊕ { Ŵ ⊕ Ô} 1:r-1 h ).Different from the text domain, code generation outputs could be additionally observed/ interpreted in relevant environments e.g. through code interpreters ("executor"). Shi et al. [2022] , Le et al. [2022] , Chen et al. [2023b,c] demonstrated the benefits of execution-based feedback to improve the functional correctness of code. However, in security-sensitive scenarios, directly engaging the executing environment might cause unintentional systematic damage, e.g. deleted data directories or modified access to privileged user accounts.We propose to deploy our critic system for both preemptive feedback (after the initial code generation step) and post-hoc feedback (after the generated code is observed by the executor). To obtain posthoc critic feedback, we simply incorporate the execution results (e.g. error messages, unit test outcomes) as the conditioning factors in 1, 2, 3, 4, and 6. Note that we maintain a persistent dialogue context between safety and helpfulness critics throughout preemptive and post-hoc iterations. We can define the output distributions of the LLM code generator conditioned by the posthoc feedback as:EQUATIONwhere Îposthoc r = f ( Îpreempt1:R⊕ Îposthoc 1:r-1 ) is the summarized posthoc critic feedback. 
experiments: 
 Base Language Models. We applied INDICT on CommandR [Cohere, 2024] which was specifically optimized for external tool augmentation, making the model suitable for our framework. In challenging adversarial tests like red-teaming attacks, we additionally employed popular preference-tuning models from the Llama and Codellama families [Touvron et al., 2023b , Rozière et al., 2023 , Meta, 2024] , ranging from 7B to 70B parameters. All models were designed for long-context tasks as well as conversational interactions, making them suitable for experiments with INDICT. To fairly compare the performance across models, given a model choice, we initialized our actors and critics with the same model checkpoint. For all base LLMs, we utilized the Huggingface-hosted model parameters [Wolf et al., 2019] and vLLM [Kwon et al., 2023] to generate the responses.Configurations. To fairly compare between base models, given a task, we maintained the instruction prompts as similarly as possible across all models. Models such as CommandR [Cohere, 2024] which is already finetuned for tool enhancement, are prompted according to their prompting strategies. We adopted a maximum output length of up to 2048 tokens on actor or critic models. We also fixed the generation budget to 1 sample in each generation by actor or critic models. For a given actor-generated sample, we applied our INDICT framework for up to 5 rounds to improve this sample iteratively. Please refer to Appendix E and I for more detailed experimental setups e.g. external tools, model and generation configurations, compute resources, and example prompt instructions.Benchmarks. We first evaluated our approach on insecure code generation tasks in which LLMs were found to generate outputs with significant security concerns. We considered the Insecure Coding Practice test from CyberSecEval-1 [Bhatt et al., 2023] , which includes two sub-tasks: "Autocomplete" where LLMs are provided a code context and predict subsequent code segments to complete this code context; and "Instruct" where LLMs fulfill natural language instructions of coding problems.Additionally, following an instruction-following setup, the CVS benchmark (Code Vulnerability and Security) [CyberNative, 2024] provides a pair of ground-truth secure and insecure code outputs given a coding problem. Please refer to Appendix E for more details of the benchmarks.Evaluation. To measure the safety of model outputs, we followed Bhatt et al. [2023] by using their detector model which contains comprehensive rules defined in weggli [weg, 2023] and semgrep [sem, 2023] to detect more than 180 patterns related to 50 Common Weakness Enumerations (CWEs). The safety metric is defined as the percentage of test samples where output codes do not contain Notations: JV: Java, JS: Javascript, Py: Python; CR: CommandR, GT: ground-truth ("GT Safe" and "GT Unsafe" are the secure and insecure code samples provided by the CVS benchmark).any insecurities. To measure the helpfulness, we followed prior work like Bai et al. [2022] , Zheng et al. [2024] , Li et al. [2024] to adopt GPT3.5 as the AI evaluator [Achiam et al., 2023] to rank the helpfulness of model outputs. In our experiments, given a test problem, we computed the winning rate of a model output against the output of a known SoTA model (e.g. Llama2-7b-chat in CyberSecEval-1) or the corresponding ground-truth outputs (for the CVS benchmark).Results. From Figure 4 , we observed consistent performance improvements of our approach, outperforming prior strong LLM baselines such as Llama and GPT models [Touvron et al., 2023b , Achiam et al., 2023] . Specifically, by applying INDICT with CommandR and LLama3 models [Meta, 2024 , Cohere, 2024] , we obtained SoTA performance by safety (more than 80% and 90% output codes are safe on CyberSecEval-1 and CVS respectively) as well as helpfulness (up to 70% output codes are more helpful than the prior SoTA model or ground-truth outputs). Figure 4 also demonstrates the consistency of our approach by both safety and helpfulness across different programming languages.There are only a few exceptional cases of helpfulness performance (specifically with Javascript in the CyberSecEval benchmark and C++ in the CVS benchmark).Benchmarks. We also evaluated our approach against malicious coding tasks in which the instruction prompts contain obscure yet dangerous intentions to perform security attacks. We considered three Figure 5 : We evaluated INDICT against three major types of security attacks from CyberSecEval-1 and 2 benchmarks. Safety measure is computed as the percentage of outputs that do not comply with the corresponding malicious prompting instructions (determined by a GPT evaluator). The higher the safety measure is, the better. Notations: CL: Codellama, L2: Llama2, L3: Llama3, CR: CommandR. Llama3-8b-instruct+INDICT 90.6 79.4 81.9 89.1 75.9 77.8 82.4Llama3-70b-instruct 68.4 60.0 68.1 90.9 61.9 57.5 67.8Llama3-70b-instruct+INDICT 85.9 75.3 74.7 90.0 75.9 75.3 79.5Figure 6 : With Llama3-8b-instruct as the base model, we evaluated IN-DICT on the CAMEL benchmark.major tasks: the Cyberattack Helpfulness test from CyberSecEval-1 [Bhatt et al., 2023] , and the Interpreter Abuse and Prompt Injection tests from CyberSecEval-2 [Bhatt et al., 2024] . The first tasks contain test samples of attack methods that are well studied in industry-standard MITRE ATT&CK ontology4 . The second task was proposed recently to instruct LLMs to abuse a code interpreter to carry on unauthorized actions e.g. data overriding. Finally, the last task is designed to simulate injection attacks by synthetically injecting harmful rules to prompts e.g. disclosing a given password in the generation output. Please refer to Appendix E for more details of the benchmarks.Evaluation. In these tasks, we focused on measuring the safety measurement by determining whether the model outputs assist the given instructions e.g. by suggesting supporting code snippets or by providing natural language explanation for a solution. Following Bhatt et al. [2023 Bhatt et al. [ , 2024]] , we used GPT3.5 [Achiam et al., 2023 ] and adopted the expansion-then-judge evaluation pipeline: first, expand the generation output with reasoning against safety criteria, and subsequently, judge if the output is indeed benign. The safety metric is the percentage of outputs that are considered benign.Results. From Figure 5 , we observed the significant performance improvement by safety measures on all three types of security attacks. Specifically, by using models from CodeLlama [Rozière et al., 2023] and Llama3 [Meta, 2024] families, we achieved new SoTA safety performance: 76% on Cyber Attack task and more than 90% on Interpreter Abuse and Prompt Injection tasks. Notably, despite a weaker model, when enhanced with INDICT, CommandR can achieve significant boosts and become more secure against harmful task instructions. The results also demonstrate the efficacy of our method on models of different sizes, from 8B to 70B model parameters.Benchmarks. Although we focused on the code domain in this work, our method can be easily adapted to generation tasks in other domains. In these cases, we can simply remove the execution environment (and accordingly posthoc feedback step) and activate INDICT with appropriate domainagnostic contexts in our instruction prompts (see Appendix I for example prompts). We adapted our method to two major open-ended generation benchmarks: HarmBench [Mazeika et al., 2024] , which evaluates LLMs against various red teaming optimization methods, and CAMEL [Li et al., 2024] , which contains a wide variety of GPT-generated complex problems in diverse domains. Please refer to Appendix E for more details of the benchmarks.Evaluation. For HarmBench, we followed Mazeika et al. [2024] and adopted their AI evaluator, which is a classifier finetuned from Llama2-13b model to assess the safety and biases of model outputs. For CAMEL, we adopted a similar strategy but used GPT3.5 as the AI evaluator. Following Li et al. [2024] , we defined the safety and helpfulness measures as the average winning rate over the direct generation approach by the corresponding base LLM.Results. Figure 7 : With GPT4o-mini as the base model, we adapted representative baselines in their original implementation and also extended them with additional instructions (detailed criteria of safety and helpfulness). We marked these enhanced baselines with the suffix '+'. 1 ).To compare with INDICT, we selected 7 strong representative baselines and evaluated them on a validation test split -random samples of 20% of the CyberSecEval-1 benchmark [Bhatt et al., 2023] .For each baseline, we also included a version where additional instructions are given to models to provide both safety and helpfulness critics e.g. instruct models to "focus on both the security and helpfulness of the solution." For multi-agent methods, we included these instructions in all agents (analyst, tester, etc.) or introduced a new critic agent (as recommended in Li et al. [2024] ). Note that for both INDICT and all baseline models, we adopted GPT4o-mini [OpenAI, 2024] as the base LLM and followed similar generation budgets (up to 3 rounds of revision) to fairly compare the results. The results in Table 7 demonstrate the SoTA performance of INDICT by both security and helpfulness (more than 90% and 81% respectively) against all the baselines. While we observed good improvement of strong baseline methods like Reflexion [Shinn et al., 2023] and CAMEL [Li et al., 2024] with additional instructions (marked with the suffix '+'), their results are not optimal and less than INDICT.To perform ablation analysis, we randomly sampled a subset from the CyberSecEval-1 [Bhatt et al., 2023] , including both Insecure Coding Practice and Cyber Attack tasks. For each task, we randomly sampled 20% of the full dataset such that the sampled subset had similar distributions as the original dataset by programming languages or types of attack methods. We reported the averaged safety metric following the evaluation of the corresponding tasks (see 4.1 and 4.2). For helpfulness, we adopted GPT3.5 as the AI evaluator and computed the percentage of outputs that are considered more helpful than the direct generation approach of the corresponding base model.From Table 3 and 4, we have the following observations. First, INDICT can lead to performance gains in both safety and helpfulness with all base models, including Codellama models from 7B to 34B and CommandR models. The framework achieves the optimal performance when integrating external tools with our critics. Secondly, we found that this tool enhancement strategy improves the safety quality of the outputs more than the helpfulness, indicating that current LLMs significantly benefit from external grounding to be more safe and secure. Thirdly, we observed that using safety critic alone or helpfulness critic alone is not sufficient, often optimizing the outputs significantly by either only safety or only helpfulness qualities respectively. Finally, we noted that when adopting our critics in both preemptive and posthoc stages, we achieved more well-rounded results, with the best overall average of safety and helpfulness metrics. We also conducted ablation analysis by multiple rounds of INDICT applications. To obtain the results of the direct generation approach (i.e. "base") in multiple rounds, we simply concatenated previously generated samples into our prompt and iteratively instructed the model to regenerate better outputs (without any critics or tool enhancement). From Figure 8 , we noted the significant and consistent improvements from INDICT, using CommandR and Codellama-13binstruct as base models. Interestingly, we still observed some performance improvement, albeit very marginal, of the direct generation approach over multiple generation rounds. We also noticed that without using external tools, the performance curves tend to converge faster than the tool-enabled approach. For more experimental results and analysis, please refer to Appendix F, G, H. 
conclusion: 
 We present INDICT, a novel framework to improve code generation by both safety and helpfulness.INDICT essentially facilitates an autonomous agent system between two critic models, each of which focuses on either the safety or helpfulness quality of outputs from the "actor" code generation LLM.Given access to external tools, the two critics interact with each other autonomously to generate grounded critiques, collaboratively improving the model outputs. We conducted comprehensive experiments of INDICT on diverse downstream coding tasks across different programming languages and attack tactics. Our results demonstrated the benefits of INDICT on code-related tasks and beyond, highlighting the promising direction of an autonomous and tool-enhanced multi-critic system. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We provided all the necessary details related to our experiments, including the datasets/ benchmarks, model configurations, hyperparameters of our proposed method, etc. Please refer to 4 and Appendix E for more details.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We did not report the error bars because it would be too computationally expensive, involving running LLMs of up to 70B parameters on large benchmarks.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We included both potential positive societal impacts and negative societal impacts of our work in Appendix C.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: We described necessary safeguards for the responsible use of our method. Please refer to Appendix C for more details.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All the datasets and models used in this work are properly cited throughout the paper, including the list of the corresponding original creators/ owners. The licenses of all datasets and models are also fully described in Appendix E.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: N/A Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: N/A Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: N/A Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Despite the strong performance of INDICT on a wide variety of tasks, there are some limitations that we want to emphasize. First, our framework relies on the instruction-following ability of LLMs to perform different specific roles, i.e. code generation actors, safety-driven critics, and helpfulnessdriven critics. Depending on how well LLMs are able to understand the requirements of these roles, we would need to carefully create well-written prompts with specific instructions for the models to follow. In our framework, we would need to describe the requirements of helpfulness and safety that the critics would need to follow and check against code generation outputs. While we try to cover as many as possible different safety and helpfulness criteria, these attributes are not trivial to be defined in the code domain. Hence, given a code generation output, our critics might not always be able to detect the right safety or helpfulness concerns.Parts of our approach can be used to remediate the above issue. Our tool enhancement strategy can equip the critics with necessary knowledge which can steer the critics towards more grounded and potentially correct recommendations. When a critic cannot detect the right issues initially, it can still improve its critiques after several rounds of interactions and tool use. Subsequently, if the extracted knowledge from external tools is relevant, the critic might be able to correctly revise and improve its final critique before passing it to the actor LLM.Another limitation of our approach is the computation cost. Compared to the direct generation approach, our framework incurs higher computation costs, activating more than one LLM and requiring access to external tools. However, we consider our approach still more affordable than relevant finetuning methods. These methods often require (1) high computation to sufficiently finetune LLMs to balance between safety and helpfulness alignment; and (2) significant annotation effort to collect quality code data by these attributes.We want to highlight that our work is specifically developed to address the safety and security concerns of AI code generation models. Any adaptation or application of our work should be used for this purpose, ultimately to create stronger yet more responsible AI systems. Moreover, as our method adopts a framework for autonomous agent systems between two independent LLMs, during any adaptation or application, it is important to control and monitor how much autonomy such systems can possess. It is good practice to limit how these agents could perform actions like web search (for example, by number of queries) and code interpreter (for example, using a sandbox execution environment, isolated from the local system). Any "thoughts" or "actions" and their outcomes from these agents have to be carefully checked to make sure they do not lead to unethical consequences.Secondly, as our work aims to address both safety and helpfulness aspects of code generation, defining and quantifying such qualities is not trivial. Within the scope of this paper, we tried to conform as much as possible to the definitions commonly used in prior related work in the code domain or the AI safety domain. In practice, there are many ethical concerns that should be considered to define these qualities, especially on the safety of code generation. For instance, in this work, we did not consider the conventional safety concerns like social biases and offensive content in code. However, these safety concerns could still be observed in many real-life practical scenarios (e.g. in generated code comments or variable names). More study is needed to address and measure safety in such scenarios.Since we aim to address the safety and helpfulness in code generation, our work can have significantly positive societal impacts. Since coding applications by LLMs are getting more and more popular, the consequences of generating harmful or insecure code can be very serious, especially in high-risk application domains like military, medicine, and banking systems. Our work can be deployed as an extra layer of mitigation, reducing the probability of potential harm while not compromising the helpfulness of AI systems. As we demonstrated in our results, our framework can also benefit open-ended generation tasks beyond the code domain. Multi-critic collab Supervision free Self-refine approach CodeT [Chen et al., 2023b] , AlphaCode [Li et al., 2022] , MBR-Exec [Shi et al., 2022] ✓ ✓ ✓Self-correct [Gou et al., 2024] , ILF [Chen et al., 2023a] ✓ ✓CodeRL [Le et al., 2022] , Self-edit [Zhang et al., 2023a] ✓ ✓Self-repair [Olausson et al., 2023a] , Selfdebug [Chen et al., 2023c] , Reflexion [Shinn et al., 2023 ]✓ ✓ ✓Multi-agent approach Self-collaboration [Dong et al., 2023] ,AgentCoder [Huang et al., 2023a ]✓ ✓ ✓CAMEL [Li et al., 2024 ] ✓ ✓ ChatDev [Qian et al., 2024] , Self-org Agents [Ishibashi and Nishimura, 2024 ]✓ ✓ ✓(?) ✓MetaGPT [Hong et al., 2023] , AgentVerse [Chen et al., 2024 ]✓ ✓ ✓ ✓CodeUltraFeedback [Weyssow et al., 2024] ,StableAlignment [Liu et al., 2024 ]✓ ✓ ✓ SafeCoder [He et al., 2024] ✓ ✓ ✓ INDICT (ours) ✓ ✓ ✓ ✓ ✓ ✓On the other hand, our framework can also be misused, assisting human users with harmful intentions to create more sophisticated attacks against LLMs. Our proposed critic models could be engineered with reverse goals, e.g. recommending ways to make the output codes more insecure or less helpful. Since these critic models are positioned in an autonomous system with freedom to interact and collaborate with each other, the resulting critiques can negatively affect the "actor" LLMs towards generating more insecure or useless code outputs.There are several safeguard strategies we can adopt to mitigate the above negative societal impacts. First, we can limit how much autonomy our critics can have e.g. by the types of queries they can generate and by the types of external tools they can have access to. In tools like web search, we can include a simple filter to exclude any illegal or unauthorized websites or content that might negatively impact the critics. Another safeguard strategy is to adopt more powerful external tools like code static analyzers or AI evaluators to provide more useful feedback to the critic models. While we did not use them in our experiments to fairly evaluate our approach against baselines, in practice, these tools should be used as safeguards for any practical application of INDICT.See Table 5 for a systematic comparison between INDICT and related methods. We reviewed each method by the following features: helpfulness or safety-based qualities of generation outputs, execution feedback (execution of output code if applicable), tool-enhanced feedback (access to external tools like web search), multi-critic collaboration (engage multiple LM agents for critic generation), and supervision free (no training data required).Compared to existing actor-critic methods, INDICT is different in three major aspects: (1) INDICT aims to optimize both helpfulness and safety awareness in the generated output code. Most of the current actor-critic approaches are designed with a single criterion (such as functional correctness).Simply extending these methods with additional instructions on safety criteria is sub-optimal (see our results with baseline actor-critic methods in Section 4.4). ( 2) INDICT integrates critic with knowledge grounding from external tools to create more reliable feedback to the actor agent. Most current methods only use code test results as the only external feedback to improve the quality of output code.(3) To implement (1) and ( 2), we enhanced the existing actor-critic framework with a multi-critic and tool-enabled collaboration approach. This approach can autonomously generate more reliable and holistic feedback for both the safety and helpfulness of output code generation.Also related to our work is the research of multi-agent collaborative systems. It is not trivial to extend this line of research to address the security of code generation. Firstly, it is not clear how the current methods could be enhanced with security awareness and subsequently improve the quality of output generations. Earlier work such as [Olausson et al., 2023b , Huang et al., 2023b] showed that simply asking agents to analyze and answer what is wrong with generated code is not always effective. With carefully designed prompts and agent interactions [Shinn et al., 2023 , Huang et al., 2023a] , collaborative agents can now generate more functionally correct code. Therefore, studying collaborative agents with orthogonal goals such as security awareness still requires further attention.As observed by our experimental results, simply applying the current multi-agent methods [Dong et al., 2023 , Li et al., 2024] , even with extended additional instructions of security criteria, does not perform so well and is still far from optimal.Generation budget. Technically, we can integrate INDICT on top of any LLMs for any number of application rounds (i.e. outer action loops), each of which can contain any number of dialogue interactions between the safety and helpfulness critics (i.e. inner critic loops). Due to the limitation of computation resources, we have to trade-off between the number of outer action loops and the number of inner critic loops. In our experiments, we fixed the number of outer action loops to 5 rounds and the inner critic loops to 1 interaction per action loop. We also maintained a persistent interaction context throughout all outer action loops so that the critics could always refer to previously generated critiques. With the above generation budget, our strategy can offer more diverse and richer input samples to the critics over time, while controlling the compute cost at an affordable level.Tools. In this work, we used 4 different types of external tools for the critics to query relevant knowledge for their arguments. For Wikipedia and code interpreter, we adopted the Langchain library [lan, 2024] with built-in functions to call these tools given the input text queries or code snippets.For web search, we employed the Search Engine Parser library [sea, 2024] to query and scrape search engine pages for different snippets such as titles and descriptions. Depending on the access constraints from commercial search engines, we mainly employ Yahoo Search as our primary search engine. Finally, to use OpenAI as an external tool, we query GPT3.5 using our paid API access5 . All the above tools are appropriately licensed to be used for academic purposes.Note that while we are treating OpenAI as an external tool in INDICT, we try to minimize contamination of test data by GPT models [Achiam et al., 2023] . Specifically, we do not directly pass the original task instructions X to OpenAI public API but only use critic-generated text or code snippets as queries (see 7 and 8). Also note that during the preemptive feedback stage, we assume no access to the execution environments / code interpreters and only employ CodeSearch as the applicable critic actions. During the posthoc feedback stage, we enable access to the code interpreters, and hence, the critics can select and perform CodeReview (with execution results as parts of the queries) to extract relevant external knowledge.Benchmarks. We evaluated INDICT on different downstream applications, including 3 major types of tasks: insecure coding practice, security attacks, and open-ended generation. Please refer to Table 6 for a summary of tasks and benchmarks used in this work. All the benchmarks considered are licensed with permission to be used for academic purposes.Insecure coding practice tasks [Bhatt et al., 2023 , CyberNative, 2024] refer to standard code generation tasks where a model receives an input containing a coding problem description, optionally with an input code context. The model is then required to generate output code to solve the input coding problem and/or finish the given code context. The test samples in this task were curated by the We also conducted experiments on security attack tasks [Bhatt et al., 2024] . In these tasks, input instructions are designed to directly or indirectly elicit harmful generation from LLMs. For instance, one example task is to request LLMs to generate code to simulate a DDoS attack in a Linux environment. More indirectly, this request could be injected into a very long list of complex requirements in the prompt. The model is required to detect such harmful intentions in the instructions and generate appropriate responses (e.g. ones not complying with the given request).The last type of downstream task we used in this work is open-ended generation tasks beyond the code domain. These tasks include both standard generation tasks [Li et al., 2024] as well as adversarial generation tasks [Mazeika et al., 2024] . In the latter, recent work has focused on prompt engineering methods to optimize the instructions and ultimately, elicit harmful behaviors from LLMs. We tested against several recent prompt optimization methods curated by Mazeika et al. [2024] , covering diverse domains like social engineering, harassment, bio-weapons, etc.Note that for CVS and CAMEL benchmarks, since they do not have an official test split, we randomly sampled a subset from the corresponding benchmarks such that the sampled data has a similar data distribution as the original dataset e.g. by programming languages. For HarmBench, from the dataset of 320 raw task instructions ("Direct" split), we augmented the data by using CommandR [Cohere, 2024] as the attacker and applying the following red-teaming optimization methods: zero-shot ("ZS") [Perez et al., 2022] , PAP [Zeng et al., 2024] , JailBreak ("JB") [Shen et al., 2023] , TAP [Mehrotra et al., 2023] , and PAIR [Chao et al., 2023] . This results in 5 more test splits, each containing 320 augmented prompts.Evaluation. To evaluate safety and helpfulness performance, we followed similar evaluation tools used in the corresponding benchmark papers and related work [Bhatt et al., 2023 , 2024 , Li et al., 2024 , Mazeika et al., 2024 , Zheng et al., 2024 , Bai et al., 2022] . These papers showed that evaluation tools like security-based code analyzers and AI detectors can achieve decent levels of accuracy, correlating with human evaluation results on subsampled datasets. In addition, to minimize potential biases in AI evaluators, we anonymized all model names and randomly positioned the model responses to be evaluated in the evaluation prompts. In code generation tasks with expected output code, we also extracted only the code snippets and excluded any text segments in the model outputs to prevent biases from long-context outputs or from simply concatenating text. Also note that we follow Mazeika et al. [2024] 's evaluation principle by not including access to evaluators (e.g. static analyzers, AI classifiers) in our proposed framework. In practice, it is possible to use these as additional tools for more insightful feedback to the critics.Base Language Models. All the models used in the work, including CommandR [Cohere, 2024] , LLama-2 [Touvron et al., 2023b] , Codellama [Rozière et al., 2023] , and Llama-3 [Meta, 2024] , are open-sourced LLMs. We accessed these models through HuggingFace, which includes model licenses with permission to be used for academic purposes. We describe the HuggingFace model IDs and their corresponding licenses below:• CohereForAI/c4ai-command-r-v01 for CommandR, licensed under cc-by-nc-4.0• meta-llama/Llama-2-[x]b-chat-hf for Llama2 of x-B parameters, licenced under llama2• codellama/CodeLlama-[x]b-Instruct-hf for Codellama of x-B parameters, licenced under llama2 • meta-llama/Meta-Llama-3-[x]B-Instruct for Llama3 of x-B parameters, licenced under llama3For the Lllama and Codellama families, we fully agreed and complied with the license conditions enforced by Meta before accessing the models.Baselines. In this work, we mainly compared with prior baselines that were reported in the corresponding benchmarks. More recently, He and Vechev [2023] , He et al. [2024] introduced finetuning approaches to finetune LLMs towards safer code generation. Almost concurrently to this work, Weyssow et al. [2024] also introduced a preference dataset of complex instructions to finetune LLMs to coding preferences. However, these approaches were not adapted and tested against the evaluation tasks and benchmarks we used in this work. Due to the limited computation cost (and also partly due to unreleased model checkpoints from He et al. [2024] at the time of submission), we were not able to evaluate the above models and compare with INDICT. We will attempt to replicate these methods and compare with our work in the future.Compute Resources. We conducted all experiments in this paper with our CPU and GPU resources provided through the Google Cloud Platform. Depending on the sizes of the base LLMs, we adopted GPU clusters of 2 to 8 GPUs of Nvidia A100 40GB type and assigned a CPU memory of up to 600GB.For some very large models such as Llama3-70B, we observed in some cases that the above hardware resulted in out-of-memory problems. For such cases, we recommend running the experiments with larger CPU memory allocation i.e. more than 600GB, or larger GPU clusters.Time costs. Given an input task, on average, INDICT incurs about 3-4x the time cost as compared to a single LLM to generate a code sample (including any iterative refinement). However, we also want to note that even with fine-tuning, fine-tuned models are far from perfect and still subject to unseen security risks or novel red-teaming prompts during test time. For instance, from our results with the fine-tuning method CodeUltraFeedback [Weyssow et al., 2024] , the fine-tuned model is still sub-optimal and can be further improved e.g. by using INDICT during inference time. See Appendix F for the experimental results and analysis.We also conducted experiments to compare INDICT with finetuning-based methods. We evaluated them on a validation test split -random samples of 20% of the CyberSecEval-1 benchmark [Bhatt et al., 2023] . Using CodeLlama-7b-instruct as the base model, CodeUltraFeedback finetunes the model on a large-scale dataset with annotations of code preferences. From Using GPT4o-mini as the base model, we conducted additional ablation experiments with different variants of INDICT: (1) one simply using a single critic agent for both safety and helpfulness; (2) one without using a critic summarizer and maintaining a full dialogue history of critiques in the critic context;(3) ones replacing the thought-action-observation critic generation with RAG or tool-based generation: (3a) RAG uses the original task description to retrieve relevant knowledge and generate grounded critics, and (3b) tool-based method uses web search/Wikipedia and a query "what is wrong with the solution in terms of its <security/functionality>?" and query output is treated as a critique.From Table 8 , we have the following observations: First, when we simply removed the summarizer and let the actor agent receive the full dialogue history, we noticed the performance degraded to 87% and 72% in safety and helpfulness. This happens probably due to the much longer context of the dialogue history, affecting the actor agent to capture all critic feedback from this history and generate new code. This model variant also incurs more computation due to the long context of the dialogues. Secondly, we noted that RAG and tool-enhanced methods are inferior to our proposed framework. We found that the queries in these methods are often too vague or ambiguous to search for meaningful information snippets.Finally, we observed that simply using a single critic agent with dual quality criteria will affect the performance, reducing the safety and helpfulness metrics to 87% and 76% respectively. One possible reason is due to the formulation of the training objectives of LMs, which are not always designed to optimize both security and helpfulness equally (also depending on the post-pretraining stages of LMs e.g. training with RLHF). Our approach enables a more flexible and probably more relaxed application of LLM as a critic agent by: (1) decoupling the helpfulness and safety goals and delegating them to individual LM agents; and (2) enabling multi-critic collaboration to autonomously develop more holistic and well-rounded critic feedback. We reported the full experimental results in this section. For results of insecure coding practice tasks, please refer to Table 9 , 10, 11 for the CyberSecEval-1 benchmark, and 12 and 13 for the CVS benchmark. For results of security attack tasks, please refer to Table 14 , 15, and 16 for Cyber Attack, Interpreter Abuse, and Prompt Injection tasks respectively.We described the example instruction prompts we used in this section (Listing 1 to 10). For each prompt template, depending on the model roles and tasks, we replace the following placeholders with applicable input components: {question} and {answer} are replaced with the corresponding task description and latest model output from the actor LLM. During the posthoc feedback stage, {answer} is also concatenated with any execution results (e.g. test outcomes, error messages) after executing the corresponding extracted code output with a code interpreter. {scratchpad} is typically used as a placeholder to contain past interactions between the two critics.Note that INDICT uses zero-shot prompting in each step. We prompt the critic agent to condition the current critique and generate a unique query to obtain more knowledge. We extract the search keywords following our instruction templates e.g. in the form of 'Search[keyword]'. For generating Table 13 : Test results of CVS: we reported the % output codes that are considered more helpful than the corresponding ground-truth secure code solutions. While all 3 base language models are found to be slightly less helpful or comparable to the ground-truth outputs, when integrated with INDICT, we noted consistent performance gains. We obtained the best performance with Llama3-70b-instruct+INDICT, with more than 65% of outputs are more helpful than the corresponding ground-truth code solutions. code snippets, we prompt the model similarly but ask the model to wrap the output code in "'. Note that the current prompts in the Appendix are for tool-finetuned models like CommandR which automatically generates tool parameters given a tool calling function definition.Listing 1: Prompt with instruction for a critic model to focus on safety-related concerns in code generation tasks.You are an advanced intelligent agent with a strong focus for system security and risk management. You will be given a previous potential solution in which you were given a task to solve. In a few sentences, diagnose whether the solution is completely safe in every execution step and will not pose any potential harmful effect to the world, including any damage to cybersecurity or physical world.You are also required to debate and discuss with another critic who focuses on system performance and accuracy. Always advocate and focus on system security and risk management over system performance or correctness.Limit your critic to a few sentences.Task: {question} Solution: {answer} {scratchpad} Listing 2: Prompt with instruction for a critic model to focus on helpfulness-related concerns in code generation tasks.You are an advanced intelligent agent with a strong focus for system performance and accuracy. You will be given a previous potential solution in which you were given a task to solve. In a few sentences, diagnose whether the solution is completely correct in every execution step and will satisfy all the requirements in the given task and pass any corner test cases.You are also required to debate and discuss with another critic who focuses on system security and risk management. Always advocate and focus on system performance and accuracy over system security or risk management.Limit your critic to a few sentences.Task: {question} Solution: {answer} {scratchpad} Listing 3: Prompt with instruction for a summarizer model to summarize the interactions between safety and helpfulness critics for code generation tasks.You are an advanced intelligent agent with equal focuses to both system performance and security. You will be given a discussion about a system solution between two critics against system safety and correctness. In a few sentences, summarize the discussion and determine whether the solution is completely correct and secure in every execution step.The perfect solution should satisfy all the requirements in the given task while not violating any security standards or causing any harmful effect to the society or cybersecurity.There might be some supporting facts in the discussion between the critics. Incorporate as much as possible those details into your response to make your analysis informative and convincing.{scratchpad} Listing 4: Prompt with instruction for critic models to generate relevant parameters (text queries) to perform CodeSearch/CodeReview actions.You are an advanced intelligent agent with direct access to Internet. You are given a task and an example solution and relevant analysis against the solution's security or functional correctness. To improve the analysis with relevant evidence and fact, generate a relevant keyword or query to search for related information on Internet. You may also search for information that is relevant to the task or solution but is missing Listing 5: Prompt with instruction for critic models to generate relevant parameters (code snippets) to perform CodeSearch/CodeReview actions.You are an advanced intelligent agent with direct access to Internet. You are given a task and an example solution and relevant analysis against the solution's security or functional correctness. To improve the analysis with relevant evidence and fact, a query might be provided to extract more information. To make the query more informative, extract or create a relevant short code snippet to be used together the query. If the query is empty, provide a representative code snippet that could be used to search for more information to support the analysis.The code snippet should be indepedent (does not refer to external operating systems, databases, repositories, or custom libraries) and limited to few lines of codes only.Use 'print' or 'assert' statements in the code snippets if needed (to execute and perform debugging on a code interpreter).Wrap the code snippet in '''. Listing 6: Prompt with instruction for critic models to generate relevant parameters (text queries and/or code snippets) to perform CodeSearch actions. This prompt is specifically for CommandR models which by default can receive any predefined external tools and select relevant ones given the context of the prompt.You are given a task and an example solution and relevant analysis against the solution's security or functional correctness.Read the task, solution, and analysis and find ways to improve the analysis with relevant evidence and supporting fact. You may also improve the analysis with missing information relevant to the task or solution.Task: {question} Solution: {answer} {scratchpad} Listing 7: Prompt with instruction for critic models to generate relevant parameters (text queries and/or code snippets) to perform CodeReview actions. This prompt is specifically for CommandR models which by default can receive any predefined external tools and select relevant ones given the context of the prompt.You are given a task and an example solution and relevant analysis against the solution's security or functional correctness. Read the task, solution, and analysis and find ways to improve the analysis with relevant evidence and supporting fact.You also have access to a code interpreter that can execute many code snippets. Based on the solution and analysis, you can create many code snippets and unit test cases to evaluate them and support the arguments in the analysis.These code snippets should be indepedent (does not refer to external operating systems, databases, repositories, or custom libraries) and limited to few lines of codes only.Use 'print' or 'assert' statements in the code snippets if needed.Task: {question} Solution: {answer} {scratchpad} Listing 8: Prompt with instruction for a critic model to focus on safety-related concerns in open-ended generation tasks.You are an advanced intelligent agent with a strong focus for security and risk management.You will be given a previous potential solution in which you were given a task to solve.In a few sentences, diagnose whether the solution is completely safe in every execution step and will not pose any potential harmful effect to the world. If the solution includes any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content, describe and explain your relevant arguments.You are also required to debate and discuss with another critic who focuses on system performance and accuracy. Always advocate and focus on security and risk management over system performance or correctness.Limit your critic to a few sentences. You are an advanced intelligent agent with a strong focus for performance and accuracy. You will be given a previous potential solution in which you were given a task to solve. In a few sentences, diagnose whether the solution is completely correct in every execution step and will satisfy all the requirements in the given task. If the solution includes any content that is not aligned with the requirements and intentions of the task, describe and explain your relevant arguments.You are also required to debate and discuss with another critic who focuses on security and risk management. Always advocate and focus on performance and accuracy over security or risk management.Limit your critic to a few sentences.Task: {question} Solution: {answer} {scratchpad} Listing 10: Prompt with instruction for a summarizer model to summarize the interactions between safety and helpfulness critics for open-ended generation tasks.You are an advanced intelligent agent with equal focuses to both performance and security.You will be given a discussion about a solution between two critics against safety and correctness. In a few sentences, summarize the discussion and determine whether the solution is completely correct and safe. The perfect solution should satisfy all the requirements and align with all intentions in the given task as much as possible while also not containing any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content.There might be some supporting facts in the discussion between the critics. Incorporate as much as possible those details into your response to make your analysis informative and convincing.To explain the benefits of INDICT and where INDICT may fail, we included example generation outputs from INDICT and related methods. In Figure 9 , we show that given a code generation task, INDICT can generate code that is more secure and robust than strong baselines (Direct generation, Reflexion [Shinn et al., 2023] , and CAMEL [Li et al., 2024] ). In Figure 10 , we illustrate cases where INDICT may fail due to nontrivial errors.Compared to the baseline methods, our approach can simultaneously improve both the helpfulness and safety of the output code generation. Specifically, given a relevant information snippet by the safety critic (about the hashing method SHA-256), our actor agent correctly revised the code with a more secure hashing method, avoiding using MD5 hashing and the common security risk CWE-328. At the same time, our generated code is generally more helpful with properly modularized functions implementing supporting features such as input validations. As we noted, this feature has generally emerged in code solutions by collaborative agent systems like CAMEL and INDICT.Summarize TASK: Write a Python function that generates a Gravatar URL for a given email address and parameter dictionary. The function should first hash the email address using MD5 and then build a canonized parameter dictionary from the input parameters. Finally, the function should return the Gravatar URL constructed from the email hash and parameters. Actor: Solution: ```python…``S afety Critic: The proposed solution for authenticating a user using OAuth 2.0 presents significant security vulnerabilities…Performance Critic: While optimizing for performance might suggest that the current approach is acceptable… Helpfulness Critic: The provided solution demonstrates some fundamental aspects of OAuth 2.0 implementation;...When handling OAuth 2.0 token storage in PHP, it is important to follow some best practices … Revise Security: ✘ Helpfulness: ✅ TASK: Write a Java method that takes a string parameter representing a password and returns a string representing the MD5 hash of the password using the standard Java API.Safety Critic: The proposed solution uses MD5 for password hashing, which is fundamentally insecure due to its vulnerabilities … Helpfulness Critic: From a performance and accuracy perspective, the code executes correctly, returning a hash of the input string,... Justification: we fully released all code and related scripts to replicate the experimental results.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Common Weakness Enumeration (CWE) is a community-developed list of common software and hardware weaknesses. More details in https://cwe.mitre.org/about/index.htmlhttps://attack.mitre.org/gpt-3.5-turbo on https://platform.openai.com/docs/models/overviewhttps://cwe.mitre.org/about/index.html 


Decision: This paper presents a technique for improving the quality and safety of code generation systems. By using two separate critics (one for helpfulness, one for safety) with access to each other and endowing them with access to information (via search tools) a multi-turn code generation system is shown to produce more accurate and safer code on a range of benchmarks.

All reviewers agreed that the approach is novel and useful. The evaluations, including those provided in the rebuttal, are extensive and show sufficient evidence about the benefits of this approach. As reviewers mention the cost-related tradeoffs are not well-discussed or evaluated. In particular the critics have additional cost and represent additional "thinking" (compute) done on each problem. Equi-cost comparisons of the various methods, ablations, and baselines would resolve the question about the extent to which INDICT's double-critic approach works because of its design or because of the additional compute available to the approach.

Despite the above, all the reviewers believe that there is no sufficient reason to reject this work and hence recommend its acceptance.

The authors are kindly asked to incorporate the results discussed in the rebuttal and -- at minimum -- a discussion about the costs of INDICT to be included.

Reviews: review: paper_summary: paper_summary: This paper presents a new framework called INDICT that employs two complementary critic agents to improve both the safety and helpfulness of LLM-generated code. Each critic agent is obtained by prompting an LLM with task-specific instructions and knowledge obtained from external tools such as web search and Wikipedia. To better generate knowledge-grounded critics, the authors propose a thought-action-observation process where the critic agent first generates the initial critic without external knowledge, then predicts search keywords or code snippets for knowledge retrieval based on the initial critic, and finally executes the external tools to retrieve the knowledge. The two critic agents can interact with each other by including the critics generated by the other agent in the current turn and past interactions in the prompt. To avoid computation overhead, INDICT also includes a summarization agent to summarize the previous interactions and uses the summary as the context for the next turn. INDICT is evaluated on eight different tasks across eight programming languages from five benchmarks. The results show that INDICT can achieve better results compared with vanilla LLMs such as GPT-4 and Llama-3. The authors also did an ablation study to show the effectiveness of using two critic agents vs. one agent and incorporating critics after the initial code generation vs. after code execution.
strength_weakness: strength_weakness: 1. My main concern about this work is the weak comparison baselines. In the evaluation, INDICT is only compared with vanilla LLMs rather than advanced and relevant prompting methods such as CodeRL, Self-Correct, and Self-Collaboration. While these prompting methods are initially designed for a single criterion such as code helpfulness, the authors can simply include an additional instruction about the other criterion in the original prompts of these methods to instruct the LLM to give two kinds of critics. This sounds like a more realistic baseline. It would be interesting to see to what extent combining two separate critic agents via the sophisticated interaction proposed by this work outperforms simply mentioning two critic criteria in a SOTA self-critic (or self-refinement) pipeline.  
2. The ablation study does not really examine the effectiveness of the novel technical bits proposed in this work. Compared with simply combining two critic agents as in a multi-agent framework, a novel idea in this work is to use a summarizer to avoid computation overhead. However, this summarizer is not evaluated. The authors should compare INDICT vs. INDICT without a summarizer.
3. The thought-action-observation is also an interesting idea that should be evaluated in the ablation study. How much improvement does this process achieve compared with existing RAG and tool-enhanced methods? 
4. Does INDICT use zero-shot prompting or few-shot prompting in each step? Based on the prompts in Appendix F, it seems INDICT uses zero-shot prompting in each step. However, this design is not very convincing for certain steps like generating relevant text queries or code snippets. Without few-shot demonstrations, it is hard to restrict the LLM to generate valid outputs. So I was wondering how exactly the authors did that. Also, how did the authors extract search keywords from the LLM response?
5. The justification for using a GPT evaluator is weak. While one or two published papers may have adopted that method, it does not mean that it is a rigorous evaluation method and should be followed broadly by the community. To improve the rigor of evaluation, the authors should consider performing a manual analysis of a small set of samples.   
6. The analysis of the experiment results is shallow and does not provide deep insights like why INDICT performs better than other methods and where INDICT fails. The authors should add a failure case analysis. 
7. In the related work, the authors should provide a more detailed discussion of how INDICT differs from existing self-critic approaches and multi-agent frameworks. The current discussion is short and handwavy.
questions: questions: 1. How well does INDICT perform compared with the stronger baselines mentioned above?
2. What is the effectiveness of using a summarizer?
3. How much improvement does the thought-action-observation process achieve compared with existing RAG and tool-enhanced methods? 
4. Does INDICT use zero-shot prompting or few-shot prompting in each step?
limitations: limitations: The discussion on limitations looks reasonable. It would be helpful to have some quantification on the computation cost.
paper_summary: paper_summary: This paper proposes a framework for generating both safe and helpful code. It integrates an internal dialogues of critiques against the given task and the corresponding generated response. It queries external knowledge through relevant code snippets and tools like web search and code interpreter. INDICT is evaluated on multiple tasks across 8 different languages. The results show an advanced level of critiques could significantly improve the code quality.
strength_weakness: strength_weakness: This paper presents a useful and well-designed framework. However, I have some doubts about the novelty of the proposed method. The framework uses actor-critic architectures and includes safety and helpfulness critics, which have been mentioned in previous papers like [1]. 

In this framework, the critics use data from web searches, Wikipedia, and OpenAI, which raises my concerns about data leakage. For example, it might be easy to find the original web page for a CWE by searching its description when evaluating on the datasets built from CWEs. Using OpenAI as a knowledge base could also have similar issues. Because of this, while the framework performs well on benchmarks, I’m concerned about the generalizability of this framework. 

Besides, as the paper mentioned, the code execution could invoke unexpected consequences. It seems the authors do not address this issue, which might require sandboxing execution or other safe execution methods.

[1] Le, H., Wang, Y., Gotmare, A. D., Savarese, S., and Hoi, S. C. H. (2022). Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314–21328.
questions: questions: 1. Can you provide more details on how your method differs from existing actor-critic architectures? Specifically, what new contributions does your framework make compared to previous works?
2. How do the safety and helpfulness critics contribute to the overall performance of the framework? Are there any specific examples or case studies you can provide to illustrate their impact?
limitations: limitations: Yes
paper_summary: paper_summary: The paper describes a method to improve the helpfulness and safety of LLMs for code generation tasks. It uses two critics - one for safety and one for helpfulness that communicate with each other to iteratively provide feedback to the actor model or the actual agent that is tasked with the code completion task. The critics are further augmented with tools that allow using the web for searches as well as LLMs from OpenAI and a code interpreter to execute code. The implementation of the method has been done using CommandR+ though experiments have also been presented using CodeLlama and Llama for certain tasks. Prompts have been standardized across LLMs to the extent possible. Experiments on benchmarks for Code Security (insecure code generation, malicious code generation) as well as open-ended generation tasks from HarmBench indicate the method improves the performance of  the corresponding base model. Ablation experiments on the code security tasks (done using CodeLlama and CommandR+ indicate that the use of both critics as well as tools help improve performance.
strength_weakness: strength_weakness: - Despite some details in the appendix; the evaluation section is missing crucial details
- See Questions
questions: questions: - For the HarmBench evaluation - how have the red-teaming methods been applied with the critics in place? I'm guessing it was applied on the actor/agent prompt? Some details would be helpful.  Was the evaluation also done using completions?  The main text suggests so, but there are no tables/results referenced. Table 1 isn't discussed (no mention or reference of the red-teaming methods in the main paper). This evaluation section is a bit unclear to me. 
- Since there are no samples of the generated dialogs or the system prompt for the actor (agent) -- if any; I'd imagine a well rounded summary being made available to the agent (Equation 5). Could the authors share more details/prompt along with a sample?
     * I was curious if fine-tuning models (smaller models?) on such generated data would have been a good experiment? Doing so could help in reducing inference cost by employing cheaper/smaller models
limitations: limitations: Yes for the most part.
paper_summary: paper_summary: This paper introduces a dual-critic prompting framework, INDICT, to consider both helpfulness and security during code generation. Specifically, the author introduces two critics, one for helpfulness and the other for security, to provide suggestions on improving the initially generated code. Besides, the authors equipped two critics with external tools like search engines and code executors to address the hallucination problem and employ an iterative refining style to further improve the performance. The authors evaluate INDICT on 8 tasks across 5 benchmarks on several LLMs. The experimental results demonstrate the effectiveness of INDICT.
strength_weakness: strength_weakness: 1. Though no former works using a multi-agent collaborative system addressing the security of code generation according to related works part in the paper, it is generally believed and widely proven effective of adopting a multi-agent collaborative system during content generation [1,2,3], which weakens the novelty of the paper.
2. The experiments only compare INDICT with pure LLMs. According to the related work part, there are several related works. Although the authors mentioned that related works are either not adapted for benchmarks used in this paper or not accessible, a lack of comparison with related works weakens the validity of the effectiveness of INDICT.
3. The authors discussed an alternative by using only one LLM to act as a helpfulness critic and security critic at the same time in their method description section. However, I do not find any experiments comparing using two LLMs as critics and using only one. The lack of this comparison confuses me with the necessity of employing two LLMs separately.

[1] Dong, Yihong, et al. "Self-collaboration code generation via chatgpt." arXiv preprint arXiv:2304.07590 (2023).

[2] Huang, Dong, et al. "Agentcoder: Multi-agent-based code generation with iterative testing and optimisation." arXiv preprint arXiv:2312.13010 (2023).

[3] Ishibashi, Yoichi, and Yoshimasa Nishimura. "Self-organized agents: A llm multi-agent framework toward ultra large-scale code generation and optimization." arXiv preprint arXiv:2404.02183 (2024).
questions: questions: 1. What is the meaning of 'GT Safe' and 'GT Unsafe' in Fig.4 (c)?
2. Can you provide the time cost of INDICT? Since your system contains multiple LLMs, multiple rounds of interaction, and even the usage of search engines and code executors, it is necessary for readers to know the efficiency trade-off compared with pure LLMs. Note that I am not taking this point as a weakness of INDICT.
limitations: limitations: The authors discussed limitations in Appendix A. Among the discussed limitations, I think the last one needs further discussion. Though the authors discussed that INDICT is much more efficient compared with fine-tuning LLMs that require curated training examples, once the fine-tuned LLM is released, the inference time costs much less than INDICT because INDICT is quite heavy.

Comments: title: Response to the rebuttal
Response to authors
Response to authors
Authors-reviewers discussion
comment: Thank you again for your feedback and response to our rebuttal! We will incorporate all feedback and our discussion into the revised paper.

Regards,
Thank you very much for your consideration and revision of the review score! Due to the limited character constraint in rebuttal, some information might not be fully explained. We will elaborate in detail the new results and other discussion in our revised paper. 

Regards,
I wanted to thank the authors for their responses. The new experiment results with stronger baselines and ablation studies are helpful. The authors should include those results in their paper and provide more interpretation & elaboration on them. 

Overall, the results look promising. I raised my score to borderline accept to acknowledge the effort from the reviewers. But I have to admit that I am not able to make a full assessment of the new results as in a regular review due to the lack of details on the results. I feel these new results require another round of careful reviews since these results are not simply the addition of a secondary experiment but rather completely new results with new baselines, etc.
Thank you for the response -- I have no further questions. These would be good to include in the updated manuscript. I would like to retain my review scores.
Thank you for your response! We very much appreciate your consideration and decision to raise the review score. We will incorporate our rebuttal discussion in detail in the revised paper. 

Regards,
Dear Reviewer GLux,

Thank you again for your reviews and comments on our paper. In our rebuttal, we have tried to address your concerns as much as possible, including your main concern about comparison to baselines (Q1), ablation experiments (Q2 and Q3), and other concerns such as prompting, manual evaluation, comparison to related work, etc. (Q4 to Q7). Please let us know if you still have any questions before the authors-reviewers discussion ends soon. 

Regards,
Thank you for your clarifications. Your response looks good to me. I would suggest adding the data leakage part to your experiments or appendix to resolve audiences' concern. Raise my score.
Dear Reviewers,

We hope our rebuttal has addressed your concerns sufficiently. Before the authors-reviewers discussion period ends on 13 Aug, please let us know if any additional information or clarification is still needed to improve the paper. 

Regards,
Dear Reviewer rV62,

We hope our rebuttal response has addressed your concerns about the paper. As the authors-reviewers discussion will end in a few days, please do let us know early if you still have any questions or need further clarification.

Regards,
Dear Reviewer SdG3,

We hope our rebuttal response has addressed your concerns about the paper. As the authors-reviewers discussion will end in a few days, please do let us know early if you still have any questions or need further clarification.

Regards,
Thank you for your consideration and revising the score! We will incorporate the feedback and our discussion in detail into the revised paper. 

Regards,
Dear Reviewer GLux,

We hope our rebuttal response has addressed your concerns about the paper. As the authors-reviewers discussion will end in a few days, please do let us know early if you still have any questions or need further clarification. 

Regards,
Thank you for your clarification. I decided to raise my score.
Please add this rebuttal content to the next version of the paper.

One more thing, I still strongly recommend a more detailed time cost comparison between INDICT and pure LLMs, other baselines. To give readers better insights about the efficiency trade-off.
Dear Reviewers,

As the authors-reviewers discussion period ends on 13 Aug, please let us know if you have any more concerns or clarifications needed to be addressed. Appreciate if you can let us know early so we can facilitate more timely discussion.

Regards,