Training Language Models to Generate Quality Code with Program Analysis Feedback
Authors: Feng Yao
Zilong Wang
Liyuan Liu
Junxia Cui
Li Zhong
Xiaohan Fu
Haohui Mai
Vish Krishnan
Jianfeng Gao
Jingbo Shang
Venue: NeurIPS 2025 poster ()
Keywords: Code Generation
Reinforcement Learning
Program Analysis
Primary area: applications
TL;DR: We adopt reinforcement learning to train LLMs to generate quality code with rewards derived from program analysis.

Abstract: Code generation with large language models (LLMs), often termed vibe coding, is increasingly adopted in production but fails to ensure code quality, particularly in security (e.g., SQL injection vulnerabilities) and maintainability (e.g., missing type annotations). Existing methods, such as supervised fine-tuning and rule-based post-processing, rely on labor-intensive annotations or brittle heuristics, limiting their scalability and effectiveness. We propose REAL (Reinforcement rEwards from Automated anaLysis), a reinforcement learning framework that trains LLMs to generate production-quality code using program analysis–guided feedback. Specifically, REAL integrates two automated signals: (1) static analyzers detecting security and maintainability defects and (2) unit tests ensuring functional correctness. Unlike prior work, our framework is prompt-agnostic and reference-free, enabling scalable supervision without manual intervention. Experiments across multiple datasets and model scales demonstrate that REAL outperforms state-of-the-art methods in simultaneous assessments of functionality and code quality. Our work bridges the gap between rapid prototyping and production-ready code, enabling LLMs to deliver both speed and quality.

Training Language Models to Generate Quality Code with Program Analysis Feedback
Feng Yao Zilong Wang!*
Liyuan Liu
Li Zhong' Jianfeng Gao² Jingbo Shang
Junxia Cui
Xiaohan Fu'
Haohui Mai Vish Krishnan
UC San Diego,
Miirosoof Research, *CausalFlow Inc.
(fengyao, zlwang,
jshang) edu, (lucliu, jfgao)@nicrosoft.cco
Abstract
Code generation with large language models (LLMs), often termed vibe coding, is increasingly adopted in production but fails to ensure code quality, particularly in security (c.g.. SQL injection vulnerabilities) and maintainability (c.g., missing type annotations). Existing methods, such as supervised fine-tuning and rule-based post-processing, rely labor-intensive annotations brittle heuristics, limiting their scalability and effectiveness. We propose REAL, a reinforcement learning framework that incentivizes LLMs to generate production-qualigy code using pro- gram analysis- guided feedback. Specifically, REAL integrates two automated signals: (1) program analysis detecting security maintainability defects and (2) unit tests ensuring functional correctness. Unlike prior work, our framework is prompt-agnostic and reference-free enabling scalable supervision without manual intervention. Experiments across multiple datasets and model scales demonstrate that REAL outperforms state-of-the-as methods in simultaneous assessments of functionality and code quality. Our work bridges the gap between rapid prototyping and production-ready code, enabling LLMs to deliver both speed and quality.
sallysii-guieed
Baxe-0.58
Baxe-3
BB
Base model: wwwnn2-CCorrinssuuuee
80
Pass Rata (N) Funetionanty Quality
60
85.37
$9.02
04
16-03
27.08
20.00
3.833
5.01
SecCodePLT4
SafeSQL
APPS+
"Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Large language models (LLMs) have revolutionized code generation, enabling rapid workflows collo- quially termed vibe coding. Coding assistants like Copilot [GitHub] 202T], Cursor, and Windsurf (Xu et al. 2022]) exemplify this shift, with developers increasingly relying on LLMs to automate tasks from prototyping to production, highlighting the critical need to ensure code quality.
In production settings, code quality extends beyond functional correctness to encompass: security (e.g., resistance injection attacks or misuse of unsafe functions) and maintainability (e.g.. proper type annotations, consistent style, and modular structure). These properties are crucial for long-term reliability and team collaboration. However, LLMs are known to generate code that is syntactically plausible but flawed in subtle or dangerous ways (Yang et al. 2024 Wan et al. 2024].
Existing methods improve code quality either through supervised fine-tuning on large corpora of manually curated, vulnerability-fiee free code [He et al. [024] or by applying rule-based post-processing
at inference to enforce security constraints [Fu et al. 22024 Nazzal et al. 2024). The former incurs
high annotation costs, while the latter depends on hand-crafted constraints specific to each coding task. Both strategies exhibit limited scalability and effectiveness in real-world production scenarios.
We introduce REAL (Reinforcement rEwards from Automated program analyysi)), a reinforcement learning framework that trains LLMs generate quality code through program analysis-guided guided feed- back. Unlike prior methods that either teach LLMs to mimic human- -verffed code examples or correct their outputs post hoc with brittle hcuristics, REAL employs verifiable and reference-fiee free reward signals to incentivize quality code generation with minimal human efforts. As illustrated in Figure| [ REAL'S compound reward combines: (1) program analysis- based detection of vulnerabilities in security or maintainability, and (2) unit-test- based verification of functional correctness.
To demonstrate REAL'S 's efficacy, we evaluate it across multiple benchmarks spanning diverse pro- duction scenarios, assessing code quality along two key dimensions. (1) For security evaluation, we augment SecCodePLT dataset (Yang et al., 2024] with program maalysis-based detector built by us that effectively identifics 17 Common Weakness Enumerations (CWEs) $ A..I resulting in an enhanced benchmark term SecCodePLT+. To enable fine- tiee-grained evaluation of high-impace vulnerabilitics, we additionally introduce SafeSQL, a targeted dataset featuring realistic database query tasks susceptible SQL injection attacks.
(2) For maintainability assessment, APPS dataset [Hendrycks et al.] [2021] to APPS+ with comprehensive static analysis, including type checking, unreachable code detection, and function signature verification for Python code.
augment
Policy Gradient
Task Description
Generated Program
Count the number of Items in with price
"Nen?"1y
within the range specifiee by the user's inputs.
O
"SLLECT CoUNt (*) FROM itens A
WHEEE pe ice do (max)
Advantage
Policy Model
AND price >n [min)"
execute
Hybrid Reward
Vulnerability Detector (via Program Anelysk)
Functionality Verifier Isis Unit Teats
Grample SQL Injection
Tainted Analysis
Unit Tests aal--sse---ppp Test 52.5-510-3
(maas, min) Use inputs as toint
Cinas, min, Q)2. Propogate cointed
Test $3-51
Puss
pererae SGL as sink.
Test $7-$1000
Generated Program
Convertt the into a Control Flow Graph (CFG) and conduct Injection vulnerabitties.
Reward: poamed texts
Taint Analysis
4 total teets
Overview of the REAL framework. Given coding task, the LLM policy generates candidate program, which is then evaluated along two automated axes: (1) Vulnerability Detector applies program analysis to flag security and maintainability defects, (2) Functionality Verifier runs unit tests to assess correctness. The two reward signals are averaged and fed into a policy-gradient update, steering the LLM toward high-quality, functionally correct code with minimal human effort.
Figure 1:

---

In addition to code quality evaluation, we assess functional correctness using unit tests. To provide holistic evaluation, we introduce a composite metric that jointly measures both functionality and quality. This resolves a critical gap: structurally sound code that fails functionally should not be prioritized. By integrating both objectives, our evaluation reflects the real-world needs, where code quality and functionality are inseparable. Extensive experiments across diverse benchmarks and model sizes demonstrate that REAL consistently outperforms state-of-the-ar baselines, confirming its scalability and effectiveness in delivering reliable and production-quality code generation. To summarize, our contribution are three-fold:
We propose REAL. novel reinforcement learning framework that integrates program analysis as automated feedback, enabling LLMs generate quality code with minimal manual intervention.
We contribute three datasets for quality code generation: (1) SecCodePLT+, enhancing (Yng et al.)
2024) with detectors for 17 CWEs,, (2) APPS+, augmenting (Hendrycks al. 2021 with static analysis for maintainability, and (3) SafeSQL, a targeted dataset for SQL. injection vulnerabilities. We design a holistic evaluation protocol that jointly prioritizes functionality and code quality, resolving the oversight of prior work that treats these objectives independently.
Problem Formulation
In this section, we first formalize the concept of quality code we investigate in this paper ($ 2.2 l.
then outline the existing paradigms and their limitations that motivate our work (.2.2) and finally introduce our task formulation and holistic evaluation protocol for quality code generation ($ 2.3)
2.1 Quality Code: Beyond Functional Correctness
In real-world production, quality code should both functionally correct and vulnerability resistant. In this work, concentrate two classes of vulnerabilities that critically impact production code:
Security Vulnerabilities. These include exploits such SQL injection and Cross-Site Request Forgery (CSRF), corresponding to entries in the Common Weakness Enumeration (CWE) [MITRE] 2024]. These vulnerabilities pose significant risks and critical threats in production environments. Maintainability Vulnerabilities. In dynamically typed languages (e.g.. Python), the absence of explicit type information, unreachable code paths, or inconsistent function signatures often leads to latent bugs, runtime errors, and degrade long-term code reliability and maintainability.
2.2 Limitations of Existing Approaches
Existing methods for quality code generation have primarily targeted reducing security vulnerabilities, with limited attention to maintainability aspects. We further identify some other critical limitations in both their evaluation protocols and methodological frameworks as follows.
First, prior work suffers from incomplete evaluation protocols, manifesting in three key shortcomings:
Chulity-Functiionlly Isolation. Code quality and functionality are evaluated separately using disjoint datasets (He et al. 2024) omitting functionality entirely (Bhatt al.) [2023), failing to capture real-world production requirements of satisfying both criteria at the same time.
Single-Vull Vulneraiiunr Assumption. Evaluations by default assume each coding problem contains only one vulnerability type and rely on this false premise for methodology design [Fu et al. 20044)
ignoring production scenarios where the code can involve multiple defects simultaneously.
Limited Detection Paradigms. Existing evaluations rely on two constrained strategies: (I)
Unreliable static analyzers like CodeQL [GitHub] 2019). which we found unexpectedly ineffective for SecCodeLTT+, and (2) Handcrafted unit tests Yang et al. 2024). which inherently have limited coverage and do not support maintainabilityy issues (e.g., type checking).
Second, existing methods for quality code generation follow two paradigms with inherent trade-offs:
Over-Reliance on Human Annotations. Data-driven approaches like supervised finetuning
heavily depend on extensive human annotations vulnerability-fre code examples, which are labor-intensive, costly, and impractical for scaling (He and Vechev 2023 He et al. 2024
Our code and datasets released httpp://ggthuu..oo/yyoof////aa
3

---

Presumption of Vulnerability Knowledge. Training- free methods enforce predefined rules for each coding problem (e.g.. filtering code with insecure patterns) that inherently presume prior knowledge of potential vulnerabilities (Fu et al. 2024 Nazzal et al. 2024). This creates paradox: avoiding known vulnerabilities renders generation redundant, while unknown ones evade detection.
2.3
Investigation Setup
We investigate quality code generation in two scenarios: (1) security-sensitive tasks requiring robust mitigation of vulnerabilities (c.g., generating database queries resistant to SQL injection shown in Figure DT and (2) maintainabiliy-avarrr tasks demanding adherence structural best practices (e.g., Python code with type annotations). These scenarios reflect real-world demands where code must simultaneously achieve functional correctness and defect resistance.
Task Formulation.
Holistic Evaluation. Our protocol addresses prior evaluation limitations through three principles: (I) jointly assess qualityfunctiionaliy: we evaluate code on quality and functionality simultancously, (2) detect multiple vulnerabilities: we detect multiple security vulnerabilities and maintainability issues via program analysis ($ ($1..I) and (3) propose unified metrics: we introduce holistic metrics that prioritize functionality and quality jointly (detailed in 4.1
3 Methodology
In this section, we propose REAL to address the limitations discussed in $ 9222by integrating program analysis feedback in model training. We first present the development of our vulnerability detector- the program analysis tool tailored for code quality ($ (. and then describe how its outputs are combined with functionality unit tests form hybrid reward for reinforcement learning ($ 3.2)
3.1 Vulnerability Detector
As noted in 2.1 quality code extends beyond functional correctness to vulnerability resistance. While correctness can be validated with unit tests, vulnerability detection is significantly more challenging: unit tests offer limited coverage and are expensive craft for each problem. In contrast, program analysis provides a scalable and general solution, capable of identifying a broad range of issues without task-specific design. To leverage this, we develop dedicated vulnerability detectors based on program analysis techniques, targeting both security and maintainability.
Security Vulnerability. encompasses a wide spectrum of weaknesses in software development. In REAL, we target a total of 18 CWEs ($ A.I) covered by SecCodePLT+ and SafeSQL, such as path traversal, command injection, and cross-site scripting. We develop static analysis to identify vulnerabilities in the code. On high level, most of the analysis follows the tactics of information flow analysis [Myers] 1999| : systematically checks whether there exists a code path where sensitive information unsanitized inputs (i.e., sources) go undesired destinations (i.e., sinks).
Take SQL injection as an example. It arises when unsanitized user inputs are directly embedded into database queries, allowing attackers to execute arbitrary SQL statements. To detect such issues, the detector transforms the program into Static Single Assignment (SSA) form [Aho et al. 2006) to analyze control flow and data dependencies. It treats user inputs and database APIs as sources and sinks, then traverses the control flow graph identify data flows connecting unsanitized inputs to APIs without proper sanitation. In Figure [ path connects the user inputs (max, min) to the database API (cur.. sour.ex execute (-)) without safeguards such as parameterized queries, indicating a vulnerability. Compared to other tools HiiHHb 2019 PyCQA 2014 Bearer [021] that emphasize precisions, REAL's analysis focuses more on soundness (i.c., identifying vulnerabilities more comprehensively) to guide the RL process to generate codes that are easier to reason about. We find that REAL's 's context-insee seneexttinsenstivee, flow-sensitive analysis is sufficient for detecting vulnerabilities, which is typically short and self-contained. REAL currently uses heuristics to conservatively identify sanitation and applies the same analysis principles to other types of vulnerability.
Maintainability Vulnerability. Beyond security vulnerabilities, we also address main- tainability requirements essential to modern software development, where generated code

---

must follow strict standards to ensure long-term reliability. Such requirements in- clude, but are not limited to, enforcing proper type annotations throughout the codebase, eliminating unused or redundant code segments, and following consistent naming conventions across modules and functions. To assess main- tainability, we use MyPy [Lehtosalo] 2025).
Task: Convert the temperature between scales def footC: 2rom, ocale: pealer
and to scalu tet: retura (2 9/5) 32
fron scale
static analysis tool for Python that inspects the abstract syntax tree and performs type infer- to detect missing annotations, type mis- matches, implicit conversions, and other quality wsshes- executing the code. We apply MyPy to model-generated code to extract rich signals that guide reinforcement learning toward producing maintainable code.
elif fron scale *x" and to scale tatt: 273.15) 9/5
reture
Furiction missing return type annotation [no-untyped-def] # Unsupported operand types for ("str" and "antt) [operator] Unsupported oporand types for ("str' and "foaat") [operator]
Figure 2: Maintainability issues detected by MyPy
3.2 Reinforcement Learning with Hybrid Rewards
REAL incorporates both code quality and functionality rewards into the reinforcement learning framework, guiding the model to generate code that is both correct and vulnerability resistant.
Motivation. While our vulnerability detectors can automatically identify defects and provide feed- back during training, optimizing for code quality alone leads to reward hacking. In early experiments on the PurpleLLaMA dataset (Bhatt et al. 2023). the model learned to produce trivial outputs sutputs-suchh such as empty code or comments -thtt maximize quality scores while lacking functionality. This failure mode underscores the need to jointly optimize for both code quality and functionality during training. To this end, we introduce REAL, a reinforcement learning framework with hybrid rewards that balances these two objectives, promoting code that is both safe and functionally meaningful.
our re- inforcement learning algorithm, following a standard framework guided by our enhanced reward design. In REAL, candidate programs are generated by the policy model, i.e., 0 = (2)), where me is the policy parameterized by 6, and represents the problem description. The generated programs y are then evaluated from two perspectives: code quality and functional correctness.
Framework. We adopt Proximal Policy Optimization (PPO) [Schulman al. 2017
Quality Reward. We pass the generated candidate program ý through our curated vulnerability detector to check whether is safe in terms of security maintainability. We denote the reward provided by the detector as Iquality
Equality = Detector (j) 1, if no vulnerabilities are detected
0, otherwise
where Detector(f) is binary reward function that assigns a positive reward only when the generated program passes the vulnerability checks without any detected security or maintainability issues.
Functionality Reward. we follow prior work and use unit tests as a verifiable reward signal for functionality [Guo et al.) 20025. Correctness evaluates the functionality of each specific task, making it hard to develop universal detectors similar to those for security and maintainability. Finally, measure the pass rate of the unit tests our functionality reward:
N
Σ 1 (in())) = out;} 2m
Prunction
N
where N is the total number of unit tests, 1 i generates ground truth result out, against the i-th unit test inp, as expected.
If (inp,, )ut,)} denotes whether the candidate program
Hybrid Reward. Taking both vulnerability concerns and functional correctness into account, final hybrid reward is formulated as,
Thybrid Tquality + (1 a) Ffunction
where E [0, 1] is weighting coefficient that controls the trade-off between code quality and functionality. If the policy model my fails to generate runnable code (e.g., due to syntax errors, runtime exceptions, etc.), assign penalty reward of -1.
5

---

After obtaining the hybrid reward, we estimate the advantage using Generalized Advantage Estimation (GAE) and update the policy model with the PPO clipped loss to ensure stable learning. This process iteratively improves the model's s ability to generate quality yet functional code.
4
Experiment
In this section, we verify the effectiveness of REAL in both security-sensitive and maintainability- aware tasks. We first introduce the experimental setup ($ 4.1.. then present comparative results ($ 4.2)
and ablation studies ($ 4.3) validate our design choices, and conclude with a case study ($ 4.4).
4.1 Experiment Settings
We evaluate REAL across three curated code generation benchmarks that cover a broad range of security-sensirigg and maintainaaility-aaaa coding problems. Each task requires the model to generate functionally correct code while meeting specific quality constraints.
Table 1: Overview of the benchmarks. Task/solution lengths averaged value measured in tokens.
Dataset
Train Size
Test Size
Task Length
Solution Length
Scenario
Source
SecCodePLT+
164
224
Security Sensitive
Enriched
SafeSQL
339
337
203
Security-Sensitiee
Constructed
APPS+
2,038
519
373
152
Maintainability-Auee Enriched
Benchmarks. Given the scarcity of benchmarks for evaluating the overall quality of generated code, we curate the benchmarks used in our experiments by extending existing datasets evolving data with large language models [Luo et al.) [024]. Specifically, we employ SecCodePI.T+ and SafeSQL study security vulnerabilities, and APPS+ to study maintainability concerns. The details
of these datasets are described below, and summary statistics are provided in Table
SecCodePLT+:
We enhance the original SecCodePLT (Yang et al.) 2024 dataset by integrating
dedicated vulnerability detectors ($ (.II for each associated CWE category, resulting in a unified and comprehensive evaluation platform for assessing security risks in code generation.
SafeSQL: We construct SafeSQL dataset by evolving seed programs using GPT-4.1 [OpenAI 2025), focusing on SQL injection vulnerabilities. Each task involves generating code that constructs SQL. queries resistant to injection attacks while retrieving correct results from a given database. APPS+: We construct APPS+ by filtering and verifying subset of APPS (Hendrycks et al. [2021), then augmenting it with a maintainability checker ($ [.1].. The benchmark evaluates whether models can solve algorithmic problems while producing clear, maintainable, and robust code.
Evaluation. We evaluate REAL along two key dimensions: functionality and quality. For each dimension. report the Pass Rate as the metrie, representing the percentage of tasks that pass all the unit tests or pass the vulnerability detector, respectively. To jointly assess both dimensions, we compute the Pass Rate by requiring the generated code to meet both criteria simultaneously.
Baselines. For the security-sensitive scenario, we consider two categories state-of-the-ar secure code generation methods: (1) Data-driven methods, including SVEN (He and Vechev 2023 and
SafeCoder [He et al. 2024), which finetune LLMs on curated vulnerability- free code, and supervised finetuning (SFT) baseline trained directly
fround-truth safe solutions from dataset. (2) Training- free methods, such as CodeGuard+ Fu et al.] 2024], which constrains decoding to favor secure outputs, and PromSec Nazzal et al. 2024), which refines prompts using GAN-based feedback. For the maintainability-wver -watt scenario, where no existing methods target maintainable code generation, we introduce: (1) prompt-based baseline that explicitly instructs the model generate maintainable code, and (2) an SFT baseline trained on ground-truth maintainable solutions. All methods use Sven2.5-Coder-Inse (Hui al. 2024 as the backbone, evaluated at 0.5B, 3B, and 7B scales.
4.2 Quantitative Results
We evaluate the performance of REAL and baseline methods across both security-sensitive and maintainability-awaarr scenarios. The results summarized in Table and Table respectively.

---

Table 2: Performance comparison of REAL and baseline models on security-sensitive tasks across different model scales. (Bold indicates the best performance; underline indicates the second-best.)
SecCodePLT+
SafeSQL
Params
Method
Function
Quality
Fune.-uul.
Function
Quality
Func--Qual.
Vanilla
0.1280
0.3598
0.0366
0.4118
0,5412
0.2000
SafeCoder
0.1524
0.3963
0.0488
0,3412
0.8824
0.3294
SVEN
0.1707
0,3780
0,0549
0,3176
0.8824
0,3059
CodeGuard+
0.0732
0.5061
0.0061
0.3176
0.4471
0.0824
0.5B
PromSec
0.1341
0.3537
0.0366
0.3529
0.7412
0.2235
SFT
0.8720
0.5061
0.4573
0.5765
0.8706
0.5527
REAL
0,5854
0.7988
0,3963
0.7647
0.8941
0.6941
Vanilla
0.2805
0.3354
0. 0.1098
0,6000
0.4588
0.2706
SafeCoder
0.3476
0.4146
0,1585
0.4824
0.8706
0.4471
SVEN
0.3476
0.4024
0.1666
0.4471
0,9294
0.4353
CodeGuard+
0.2927
0.3963
0.1280
0.5882
0.4824
0.2471
38
PromSec
0.2134
0.4146
0.0854
0,2000
0.9647
0.1882
SFT
0.8902
0.5061
0.4573
0.7529
0.9176
0.6824
REAL
0.7378
0,8476
0.6037
0.8471
1.0000
0.8471
Vanilla
0.2988
0,3902
0.1098
0.6471
0.8824
0,5882
SafeCoder
0.3293
0,3902
0,1402
0.4706
0.8941
0.4353
SVEN
0.3171
0.4024
0,1280
0,5059
0,9176
0,4706
CodeGuard+
0.2988
0.3476
0,1098
0.6353
0.8824
0.5647
7B
PromSec
0.2195
0.4878
0.0976
0,0941
0.8588
0.0824
SFT
0.8659
0.5122
0.4634
0.8118
0.8941
0.7294
REAL
0.7561
0,8293
0.6159
0.8588
1.0000
0.8588
Security-Sensitive Scenario. The results in Table 2 highlight three key findings. First, REAL consistently achieves the best overall performance on the SafeSQL. benchmark across all model sizes, outperforming all baselines in functionality, security quality, and joint metries. Second, on SecCodePLTT+, REAL leads in security quality and joint metrics at the 3B and 7B scales. However, at the 0.5B scale, supervised finetuning (SFT) slightly outperforms REAL. This gap is relatively small and can be attributed to the limited capacity of the 0.5B requires a reasonably strong base model. Third, training-free methods like CodeGuard+ and PromSec perform poorly across most metrics, highlighting the limitations of decoding time interventions for secure code generation. Overall, REAL demonstrates strong scalability and a robust balance between functionality and security, validating the its effectiveness in sccurity-sensittee tasks.
learning often
Maintainabillty-Aaaal Scenario. According to Table 3 REAL achieves the best overall perfor- mance across all metrics and model sizes. It surpasses both prompt- and supervised finetuning (SFT) baselines in functionality, maintainability quality, and their joint measurement. The improve- ments especially clear in the joint metrics, where REAL significantly outperforms all alternatives, demonstrating its effectiveness in generating not only correct but also clean and maintainable code. Furthermore, REAL scales well with model size- delivering consistent gains in both functionality and joint performance as the model capacity in- creases from 0.5B to 7B.
Table 3: Performance comparison of REAL and baselines on maintainability-sennnn sensitive tasks.
APPS+
Method
Function
Quality
Func--Qual.
Vanilla
0.1965
0.2177
0.0501
PromptEng
0.0597
These results collectively demonstrate the ef- fectiveness of REAL across wide range of real-world production scenarios, including both security-sensitive and maintainability-awee By jointly optimizing for correctness and code quality through reinforcement learning with pro- gram analysis feedback, REAL consistently im- proves performance across all metrics and model sizes-outperformin strong baselines without sacrificing either dimension significantly.
0.5B
SFT
0.2274
0.7476
0,1888
REAL
0,3064
0.9587
0.2909
Vanilla
0.4990
0.1407
0.0983
tasks.
Promptlnn
0.4913
0.1946
0.1272
SFT
0.4586
0.8189
0,4046
REAL
0.5241
Vanilla
0,5896
0.1580
0.1079
Promptting
0,5645
0,2312
0.1599
7B
SET
0.4663
REAL
0.6667
0.9229
0.6204

---

Table 5: Comparison of training strategies using functionality-onke quality-only, and hybrid rewards in both security-sene seniitivve and maintainabilttyawaare scenarios. (Bold indicates the best performance; underline indicates the second-best.)
SecCodePI.I+
SafeSQL.
APPS+
Method
Function
Quality
Func--uall
Function
Quality
Punc--uul.
Function
Quality
Func--uual.
Vanilla
0.1280
0,3598
0.0366
0,4118
0,5412
0.2000
0,1965
0.2177
0,0501
Prancees
0,1402
0.0854
0.7765
0.0471
0.0235
0,2505
0.1734
0.0751
0.58
w/ Fipuity
0.0732
0,9268
0.0732
0.3529
1,0000
0.3529
0.044.
0.9981
0.0445
0.544
0.7988
0.3963
0.7647
0.8911
0.3064
0.9557
0.2909
Vanilla
0808
0,3354
0.1098
0.6000
0.4588
0.2706
0.4990
0.1407
0.0983
0.7683
0,2805
0.2073
0,8588
0,0824
0.0588
0.5607
0.1464
0,1137
3B
w/ Fiaaty
0.1585
0,9024
0.1220
0.5529
1.0000
0.5529
0.4663
0.9383
0.4432
REAL
0.7378
0.8476
0.6037
0.8471
1,0000
0.8471
0.5549
0.9268
0.5241
Vanilla
0.2988
0.3902
0.1098
0.6471
0.8824
0.5882
0.5896
0.1580
0.1079
0.7805
0,3476
0.2866
0.8706
0.0588
0.0353
0.6667
0.1503
0.1214
7B
Fipulty
0.3415
0.2866
0.6353
1,0000
0.6353
0,5645
0.9441
0.5414
REAL
0.7561
0.8293
0,6159
0.8588
1,0000
0,8588
0.6667
0.9229
0.6204
To further validate generalization, we trained REAL on APPS+ and evaluated it on an unseen benchmark, HumanEval [Chen et al. [002]. The result is shown in Table 4 REAL 7B achieves substantial improvements compared to the base- line across all metries despite not being trained on HumanFyal, demonstrating strong generalization.
Table 4: Performance comparison of REAL on the unseen benchmark, HumanEval.
HumanEval
Method
Functionality
Quality
Func--uull.
Vanilla
0.7927
0.2561
0,2317
7B
REAL
0.8476
0,9573
0.8171
4.3 Ablation Study
In this section, we analyze the impact of key design choices in our proposed REAL framework, focusing on (1) the hybrid reward balancing code quality and functionality, (2) the weighting hyperparameter in the hybrid reward, and (3) program analysis versus unit tests for quality supervision during reinforcement learning.
Hybrid Reward vs. Single Reward Our proposed REAL framework incorporates a hybrid reward that combines correctness unit tests (for functionality) with program analysis (for quality). To better understand the contribution of each component, we conduct an ablation study where we isolate the reward signal to either functionality--on (unit tests) or quality-only (program analysis). keeping the overall training pipeline unchanged. The results are presented in T
We observe that when using only the functionality reward, the model achieves strong functional correctness but suffers from notable decline in security quality. Conversely, when using only the quality reward, the model generates safer code but at the expense of reduced functional correctness. In both cases, the combined metric drops significantly, indicating lack of balance.
In contrast, adopting the hybrid reward leads to a substantial improvement in the joint functionality- quality metric. While there is a slight trade-off in each individual dimension compared to their respective single-reward counterparts, the hybrid approach enables the model to achieve a balanced optimization. This balance results in significantly better overall performance, demonstrating that our method effectively harmonizes the competing objectives of functionality and quality.
Sensitivity to Hybrid Reward Weight a In the hybrid reward of REAL, we balance the quality reward and functionality reward with hyperparameter a, where Thybrid = & qqqualtty + (1 a.) Fffncctoon To probe its sensitivity, we swept a over (0, 0.3, 0.7, 1.0} on SecCodePLT+ and Sa As shown in Figure 3 we observe clear trade-off: optimizing solely for functionality or for quality tends harm the other. A low a (favoring functionality) leads to poor quality scores, whereas a high a (favoring quality) degrades functional correctness. Both extremes result in reduced joint pass rates. In contrast, moderate values (e.g., a = 0.3 ~ 0.5) achieve a more desirable balance, yielding the strongest overall performance across model scales and datasets. These findings validate our hybrid reward design, demonstrating that our RL. framework can flexibly optimize multiple objectives through careful tuning the value of a.
8

---

Function
Quality
Func- Qual
04
0.3
03
3.5
04
0.4
83
02
60
0.
07
0.3
5.5
1.
Alpta Vilue
Arpha Valas
Alpte Valle
SecCodePLT- SecCodePLTe-3
SecCodeeLT---78
SafeSO
SafeeCa
Figure 3: Ablation study the value of a on SecCodePLT+ and SafeSQL datasets.
Table 6: Comparison of training with program analysis rewards (REAL.) versus safety unit tests across different model sizes on the SecCodePLT+ and SafeSQL. datasets. REAL consistently outperforms unit lest-based training across functionality, security quality, and their conjunction. (Bold indicates the best performance.)
SecCodePLT+
SafeSQL
# Params
Equality
Function
Quality
Func-Qual.
Function
Quality
Func--uul.
w/ Safety Unit Tests
0.4573
0.3415
0.1341
0.4588
0.8235
0.3882
0.5B
w/ Detector (REAL)
0,5854
0.7988
0.3963
0.7647
0.8941
0.6941
w/ Safety Unit Tests
0.6524
0.4634
0.2439
0.8235
0.8824
0.7412
Detector (REAL)
0.7378
0.8476
0.60337
0.8471
1.0000
0.8471
Safety Unit Tests
0.6341
0.4695
0.3171
0.8471
0.8941
0.7647
7B
w/ Detector (REAL)
0.7561
0.8293
0.6159
0.8588
1.0000
0.8588
Program Analysis vs. Unit Test While our REAL framework uses program analysis as feedback to train models for generating quality code, prior work has explored using unit tests to evaluate security properties (Yang et al. 2024 Dai et al. [025]. Although not originally intended for training, these unit tests can also be repurposed as reward signals in reinforcement learning. To assess the effectiveness of our design choice, compare these two reward strategies: (1) using safety unit tests, and (2) using our program analysis basee detector. We conduct experiments on the SecCodePLT+ and SafeSQL datasets, both of which provide safety unit tests for each coding problem problem-allooing for a fair comparison. As illustrated in Table models trained with our program analysis- based feedback consistently outperform those trained with unit tests across functionality, quality, and their conjunction, demonstrating the robustness and scalability of our approach across all model sizes.
4.4 Case Study
In Figure 4 we illustrate the code generated by REAL at different training stages of reinforcement learning with hybrid rewards on the SafeSQL benchmark. We observe that the individual reward com- ponents within the hybrid reward framework converge at different rates, with the code quality reward saturating earlier than the unit test- best-based correctness reward. In this example, the task requires the model to correctly interpret the constraints described in the task prompt and construct an appropriate SQL query retrieve the desired results from the database. At the initial stage, the generated code is vulnerable and incorrect. It suffers from a SQL injection vulnerability by directly incorporating user inputs into the SQL query through string formatting without proper sanitization. Additionally, the semantics of the query are problematic, as the task specifies the use of an OR condition for the price constraint rather than an AND condition. As training progresses, the model gradually learns to satisty the security requirement but the constructed query remains semantically incorrect. The generated code adopts parameterized queries, leveraging the implicit sanitization provided by the sqlite3 library (e.g., cursor execute (query, (room_ ttpee ..)). In parameterized execu- tion, placeholders (i.e., ?) are used in the query, and the corresponding user inputs are safely injected through the provided parameters, automatically handling escaping and preventing injection attacks (e.g., escaping special characters and enforcing correct data types). By the final phase of training, the model successfully learns to both generate a correct SQL query while securely us- ing parameterized execution. Notably, the hybrid reward further guides the model to apply explicit input sanitization by enforcing proper type conversion (c.g., max_price = float (input(...))

---

White Pychen senge that rater check-ln oree check and lipuc then generetes query reservatons mettohing type, checi-ii ofter given check--ut bxfore given
Generated Code at Global Step lllitt
Generated Code at Global Step.180
Sonerated Code at Global Step 500 (tnaal
cona
aqlisas cassact....
Easa aq-458). .onsact
osea aglisad cocoact
Lapont Tiiner wosl pelne Pinense T1
pello Lapen (Tniter the wax see pe lise
maN priie Flonk Laran ("enner maxtess pays 112
Dischmen mork n01, saarty
not evory
XMD cter 1n Jate ((therk_ in _esoe)
rowe type i smn stem inuditee on san shonk port jott aen pelinn
300 cbeek teaeate AND
cmm chort mark Jane (ohorkk www.t__taaaaa _
sneel 24 AAte de prine do.
Rescuse
with the case lnpata
Eoocate the Linan, Sapee, nansor conreborreeryy
cureae peteereoseeeteeooeeeee (overr crose Jypey,
pzices)
Incorrect quary w/ SQL Injection
Incorrect query w/o SQL injection
Correct query w/o SQL Injection
Figure 4: Examples of code generated by REAL 0.5B at different training stages on SafeSQL with hybrid rewards. Initially, the model produces incorrect and insecure code, misinterpreting "or" as AND and directly incorporating unsanitized user inputs. Later on, it adopts parameterized execution (using ? placeholders in the query with separate parameter binding) to implicitly address vulnerabilities. Finally, it corrects the query logic and explicitly sanitizes user inputs with proper type conversion (using float (-) to enforce correct type conversion of the input).
further demonstrating the effectiveness of our proposed REAL in producing high-quality, secure code and enabling a seamless, confident vibe coding experience.
5 Related Work
Existing secure code generation methods generally follow two paradigms: (1) Data-driven approaches apply supervised fine-tuning to train LLMs on large corpora of secure code [He et al. 2024 Yang et al., 2024], under the assumption that mimicking "clean" code is sufficient for generalization. While effective against known vulnerabilitiess these methods struggle with novel defects and require extensive human annotation. (2) Training ffre approaches rely on rule-based post-processie either enforcing security constraints during decoding [Fu et al. 2024 or using feedback or in-context examples iteratively refine prompts [Nazzal al. 2024 Zhang et al.,
Secure Code Generation
2024]. However, such heuristics are task-specific, brittle to unseen vulnerabilities, and vulnerable to vulnerability-wware leakage, where models exploit patterns in the rules evade detection. Some recent work explores reinforcement learning to fix vulnerabilities, but these methods still depend on ground-truth code for each problem and define rewards based on semantic equivalence those references- thhs still requiring costly annotations [Islam et al. 2024). In contrast, our method does not rely on ground-truth annotations or prior knowledge of specific vuinerabilities.
Reinforcement Learning for Code Generation Reinforcement learning has recently shown to be highly effective in improving model capabilities in tasks that come with "verifiable rewards" such as math and coding [Guo al. 2025 OpenAl 2025). For code generation, reward signals can be
determined by executing the candidate code against unit tests or testcases (Le et al. 2022 Yang al.] 2024, Yu et al., [024] Gehring et al. [2025] Wei et al.] [025]. Since the availability of unit tests may be limited in practice, [Li et al. [024] explores generating unit tests for specific coding tasks automatically in scale. addition to unit tests derived rewards, [Dou et al., (2024] incorporates
compiler feedback to address the challenge of long code sequence, while [Xie et al. 2025 explores
using LLMM themselves as a critic to improve the code generation. However, these work all focus on enhancing the functionalitycooreettess of the generation, but not code quality.
Conclusion
We presented REAL, a reinforcement learning framework that leverages program analysis to guide LLMs toward generating quality code that is both functionally correct and vulnerability resistant. By integrating feedback from program analysis and functionality verification, REAL enables scalable training without relying on human- wwitten references or handcrafted rules. Extensive experimental results demonstrate the effectiveness of REAL. While our current vulnerability detectors are designed to prioritize soundness and generality, they rely on heuristic approximations and do not yet cover the full breadth of CWE types. In the future, we will explore more robust and comprehensive detectors to expand coverage, enabling even more reliable feedback for large-scale training.
10

---

References
Alfred V. Aho, Monica S. Lam, Ravi Sethi, and Jeffrey D. Ullman. Compilers: Principles, Techniques, and Tools (2nd Edition). Addison-Wesley Longman Publishing Co., Inc., USA, 2006. ISBN
0321486811.
Bearer. Bearer: Static application security testing (sast) tool. https://github.com/BBeea comeearer/ bearer 2021. Accessed: 2025-05-15.
Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evtimov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, et al. Purple llama cyberseceval: A secure coding benchmark for language models. arXiv preprint urXiv:2312.04724, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Shih-Chieh Dai, Jun Xu, and Guanhong Tao. comprehensive study of IIm secure code generation.
arXiv preprint arXiv:2503. 25.3::2500335555555 /5554, 2025.
Shihan Dou, Yan Liu, Haoxiang Jia, Limao Xiong, Enyu Zhou, Wei Shen, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, et al. Stepcoder: Improve code generation with reinforcement learning from compiler feedback. arXiv preprint wrXiv:2402.0139, 2024.
Yanjun Fu, Ethan Baker, Yu Ding, and Yizheng Chen. Constrained decoding for secure code generation, 2024. /tttps::://axxi http://xxxvvorro//////// abs/ 2405.0028
Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Quentin Carbonneaux. Taco Cohen, and Gabriel Synnaeve. Rlef: Grounding code 1lms in execution feedback with reinforcement learning, 2025. URL hRtps::////axxit org/ abs /2///410.022000
GitHub. Codeql: Semantic code analysis engine. https: https://codeql. wit.gbt.hh.ooooo 2019. Accessed:
2025-05-15.
GitHub. Github copilot: Your ai pair programmer.
https://github. blog/
2021-06-29-gi10 20--99--itt t---githubccopillteeeee paai-ppar-pppe moair-programmer/ 2021.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprins arXiv:2501. 2025.
Jingxuan He and Martin Vechev. Large language models for code: Security hardening and adversarial testing. In ACM CCS, 2023. httpst httpaaawwwwwr...aaaaaaaaa abs /200./2322222 05319
Jingxuan He, Mark Vero, Gabriela Krasnopolska, and Martin Vechev. Instruction tuning for secure code generation. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL http://://ppppoorrer..tettetttte MgTzMaYHvG
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. NeurIPS, 2021.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12106, 2024.
Nafis Tanveer Islam, Mohammad Bahrami Karkevandi, and Peyman Najafirad. Code security vulnerability repair using reinforcement learning with large language models. arXiv preprint arXiv:2401.07031, 2024.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning, 2022. URL
httts://arxii... orggiiiioorr/aeee 2b5/ 01780

---

Jukka Lehtosalo. Mypy: Optional static typing for python. https://mypy-lant org/ 2025.
Version accessed: May 2025.
Jia Li, Yunfei Zhao, Yongmin Li, Ge Li, and Zhi Jin. Acecoder: An effective prompting technique specialized in code generation. ACM Transactions Software Engineering and Methodology, 33
(8):1-26, 2024.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, 2024.
MITRE. Common weakness enumeration (cwe). MITRE Corporation, 2024. URL https://cwe
mi mitre.ooe wwwpeorggnii..iieeeeee html Accessed: 2025-05-14. Version 2025a.
Andrew C. Myers. Jflow: practical mostly-static information flow control. Proceedings of the 26th
ACM SIGPLAN-SIGATT Symposium on Principles of Programming Languages, POPL '99, pages New York, NY, USA, 1999. Association for Computing Machinery. ISBN 1581130953.
doi: 00.1445/92540.292556 https:/// org/ 10. 11.111..1555 1144/29224200 292561
Mahmoud Nazzal, Issa Khalil, Abdallah Khreishah, and NhatHai Phan. Promsec: Prompt optimization for secure generation of functional source code with large language models (Ilms.. In Proceedings of the 2024 on ACM SIGSAC Conference Computer and Com- munications Security, CCS '24, pages 2266-2280, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400706363.
doi: 10.11455368644436009999 URL
bittp://doo..rg// 10. 1145/36586444 3690298
OpenAl. Gpt-4.1, 2025. http:://opeaa..coo/mdndx/pppp--
PyCQA. Bandit: Security linter for python source code. httpps///githuub.cm///ooo/ 2014. Accessed: 2025-05-15.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X7v:1707.06347, 2017.
Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman, Stephanie Ding, Vlad Ionescu, Yue Li, and Joshua Saxe. Cyberseceval 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models, 2024. URL
wwwpaawwwwwv..rrraaa/aae//e////e 01605
Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. Swe-rl: Advancing IIm reasoning via reinforcement learning on open software evolution, 2025. URL https:ts:://aaa http://axxvv....o org/ www.orgg//bbbbroeeeeeee abs2 18449
Zhihui Xie, Liyu Chen, Weichao Mao, Jingjing Xu, Lingpeng Kong, et al. Teaching language models to critique via reinforcement learning, arXiv preprint 2xiv:2502.03492, 2025.
Frank Xu et al. Windsurf: Empowering developers with ai-assisted code completion. arXiv preprint arXiv:2212. 10943, 2022.
Yu Yang, Yuzhou Nie, Zhun Wang, Yuheng Tang, Wenbo Guo, Bo Li, and Dawn Song. Seccodeplt: A unified platform for evaluating the security of code genai, 2024. URL Ittpps//s/// org/
Zishun Yu, Yunzhe Tao. Liyu Chen, Tao Sun, and Hongxia Yang. B-coder: Value-based deep reinforcement learning for program synthesis, 2024. URL https://arxiv. wwwpr/iiv.orgabb//21//0
03173
Boyu Zhang, Tianyu Du, Junkai Tong, Xuhong Zhang, Kingsum Chow, Sheng Cheng, Xun Wang, and Jianwei Yin. Seccoder: Towards generalizable and robust secure code generation. arXiv preprint arXhv:2410.01488 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and Section clearly state the paper's contributions and scope, in- troducing REAL as reinforcement learning framework that uses program analysis feedback to generate quality code.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section discusses limitations of existing approaches and addresses the limitations of the REAL framework.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
13

---

Answer:
[NA]
Justification:
This paper proposes a practical system without theoretical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper provides comprehensive details about the implementation of REAL, the datasets used, and the evaluation protocol in Sections 3 and 4, making the results reproducible.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: According to Section A.2, the authors implement REAL based on the VeRL framework, which is publicly available. The datasets and implementation details are de- scribed thoroughly.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nips.cc/public/////cccc heallii/ggiiss//Ce Cotpss//dodeSbbiiice SoroodeeumisssioPPPe.. wwriiisssiiiiiyiyioo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4 and Appendix A.2 specify training details including model architec- ture, hyperparameters, training procedures, and evaluation criteria needed to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow the same protocol as in our baseline methods, e.g., SafeCoder, SVEN, CodeGuard+.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section mentions the experiments were conducted on a server with NVIDIA H100 GPUs and provides details about batch size, learning rate, and other training parameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httph:ttt/////t httppp/:////u/pss././/// httffhighttrrr//rrrrp....pp.pppp
Answer: [Yes]
Justification: The research focuses on improving code quality and security, which aligns with the NeurIPS Code of Ethics by promoting safer and more reliable software systems.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: maintainability, while acknowledging challenges RFAL-world deployment in Sections
The paper discusses positive impacts in terms of improving code security and
and 2.
16

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymemt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
The paper describes program analysis-base based safeguards to detect security vulnerabilities in Section 3.1, and how these are integrated into the reinforcement learning framework.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The paper properly cites the original sources for datasets, models, and frame- works used, including SecCodePLT, APPS, and the Owen2.5-Coder--ni model.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
17

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The paper introduces three new datasets: SecCodePLT+, SafeSQL, and APPS+, with documentation of their construction and statistics provided in Section 4.1.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects, as it focuses on computational methods for code generation.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research does not involve human subjects require IRB approval.
18

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Section 4.1 mentions that GPT-4.1 used to evolve seed programs for constructing the SafeSQL dataset, which is an integral part of their research methodology.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy hect:::/hearrip..cc/ooffeee http://s....ccoooeeneeeeeeeeeeee for what should or should not be described.
19

---

A Appendix
A.1
CWE List
We provide list of Common Weakness Enumerations (CWEs) we cover in the paper in $ A.1
CWE ID
CWE NAME
CWE RISKY SCENARIOS
74
Improper Neutralization of Special Elements in Output Used by a Downstream Component ("Injection')
The product constructs all part of command, data structure, or record using extemally-infin inffuenced input from an upstream component, but it does not neutralize or incorrectly neutralizes special elements that could modify how it is parsed interpreted when it is sent to downstream component.
Improper Neutralization of Special Elements used in Command ("C I'Command Injection')
product constructs all or part of command using externaly--nfllenneee influenced input from an upstream component, but it does not ncutralize or incorrectly neutralizes special elements that could modify the intended command when it component.
downstream
79
Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
The product does not neutralize or incorrectly neutralizes user-controllable input before it is placed in output that is used web page that is served to other users.
89
Improper Neutralization of Special Elements used in an SQL Command ('SQL Injection')
The product constructs all or part of an SQL command using externally-influencee input from an upstream component, but it does not neutralize or incorrectly neutralizes special elements that could modify the intended SQL command when it sent to downstream component. Without sufficient removal or quoting of SQL syntax in becr-contoollabe inputs, the generated SQL query cause those inputs be interpreted SQL instead of ordinary user data.
94
Improper Control Generation of Code ('Code Injection")
The product constructs all or part of code segment using externally-in citrmalll-iffuueeeedd input from an upstream component, but it does not neutralize or incorrectly neutralizes special elements that could modify the syntax or behavior of intended code segment. The product receives input from an upstream component, but it does not neutralize or incorrectly neutralizes code syntax before using the input in dynamic evaluation call (c.g. eval).
95
Improper Neutralization of Directives in Dynamically Evaluated Code ("Eval Injection")
200 Exposure of Sensitive Information to an Unauthorized
The product exposes sensitive information to an actor that is not explicitly authorized to have access to that information.
Actor
327
Use of a Broken Risky Cryptographic Algorithm
The product
broken risky cryptographic algorithm protocol.
347
Improper Verification of Cryptographic Signature
The product does not verify, or incorrectly verifics, the cryptographic signature for data.
352
Cross-Site Request Forgery (CSRF)
The web application does not, can not, sufficiently verify whether well-formed, valid, consistent request was intentionally provided by the user who submitted the request.
502
Deserialization of Untrusted Data
The product deserializes untrusted data without sufticiently verifying that the resulting data will be valid.
601
URL. Redirection to Untrusted Site ('Open Redirect")
weh application accepts a user-controlled input that specifics link to an external site, and uses that link in Redirect. This simplifies phishing attacks. The product allocates reusable resource or group of resources on behalf of an actor without imposing any restrictions on the size or number of resources that can be allocated, in violation of the intended security policy for that actor.
770
Allocation of Resources Without Limits Throtting
20

---

CWE ID
CWE NAME
CWE RISKY SCENARIOS
862
Missing Authorization
The product does not perform authorization check when attempts access resource perform an action.
863
Incorrect Authorization
The product performs an authorization check when an actor attempts to access resource perform an action, but does not correctly perform the check. This allows attackers bypass intended access restrictions. The product receives input from an upstream component that specifies multiple attributes, properties, fields that are be initialized or updated in an object, but does not properly control which attributes can be modified.
915
Improperly Controlled Modification of Dynamically-Determie Determined Object Attributes
918
Server-Side Request Forgery (SSRF)
The web server receives a URL or similar request from an upstream component and retrieves the contents of this URL, but does not sufficiently ensure that the request being
the expected destination. The product uses regular expression with an inefficient, possibly exponential worst-case computational complexity that consumes excessive
1333
Inefficient Regular Expression
Complexity
CPU cycles.
A.2 Implementation Details
We implement REAL based the VeRL framework and conduct all experiments node equipped with 8 NVIDIA H100 GPUs. The backbone model used in our experiments is Qwenn255-Coder-Insttii with pretrained weights obtained from the public HuggingFace platform
For reinforcement learning, we adopt the Proximal Policy Optimization (PPO) algorithm with hybrid reward design that balances functional correctness and code quality, focusing both security and maintainability. The policy model is initialized from the -eeene.--oddr--nnseee
checkpoint and fine-tuned using PPO with a learning rate of Ic-6, batch size of 256, and a KL divergence penalty coefficient of le-3 to ensure stable policy updates. Advantage estimates are computed using Generalized Advantage Estimation (GAE) with discount factor of 1.0 and GAE lambda of 1.0. To promote exploration, entropy regularization is applied, and hybrid rewards are normalized to further stabilize training.
curate the SafeSQL benchmark by constructing diverse set manually designed seed programs covering common database query patterns and known security pitfalls. These seed programs are further evolved with GPT 4. usinng code mutation and transformation strategies inspired by (Luo cet al.) 2024 producing comprehensive benchmark that captures wide range of realistic and challenging SQL generation scenarios.
htrp:://ugggingfacce.o//
21


Reviews:
[{"review_id": "mLWEiELKmR", "reviewer": "Reviewer_sExN", "summary": "The authors propose a reinforcement learning-based framework for quality code generation, named as REAL, which integrates an automated feedback policy through program analysis with minimal human intervention. They also contribute three curated datasets to support their primary goal of generating high-quality code, with a particular focus on vulnerability detection and maintainability. Finally, they evaluate their proposed methodology in terms of balancing the trade-off between these two objectives, demonstrating improvements over prior approaches that address them separately.", "questions": "Q1. Instead of curated datasets, what if we evaluate on previous SecCodePLT [Yang e al., 2024] and APPS [Hendrycks et al., 2021] without any preprocessing. Can you please demonstrate similar benchmarking as Table 3-5? Can REAL, as a novel stand alone RL framework, applicable for any unknown real world raw datasets (generalizability) towards satisfying maintainability and vulnerability check (can we expect self-automated)?\nQ2. Authors mentioned as \"with minimal manual intervention\", however used 3 curated datasets for evaluation. How minimal was this \"curation\" as \"by extending existing datasets or evolving data with LLM\"? Is there any bias introduced due to these curated datasets in training? Can it anyhow misguide the RL agent in specific scenario, for complex logic codes especially in dynamic contexts?\nQ3. Program analysis-based feedback might not always provide explainable rationales for the improvements or rejections. Any concerns there?", "limitations": "Limitations are mentioned in weakness part.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The authors well discussed and presented metrics and limitations of prior research works that targeting similar scope of quality code generation framework, which is highly beneficial for understanding the fundamental background, research roadmap trajectory and the expected improvements.\n2. Experimental set up and benchmark strategy are well-structured and supported with evidence. The proposed methodology consistently  outperforms baseline and previous approaches across various evaluation metrics.\n\nWeakness:\n\n1. Methodology section appears to be more theoretical, putting more emphasis on \"what\" rather going in depth of \"how\" only mentioning as a generalized PPO based RL-based framework with reward/penalty mechanisms. It lacks deeper technical elaboration. More implementations details are needed for better clarity. \n2. Experimental validation: Please make it more clear that as benchmarked on 3 curated datasets as SecCodePLT+, SafeSQL and APPS+, proposed method outperforming because of these specifically curated dataset or for employing a RL-based framework or combining both. Generalization over real-world raw dataset should be explored.\n3. Authors should also points towards current limitations and potential breakpoints of the REAL framework (if any), particularly in the context of real-world quality code generation applications\n4. Several typos and grammatical errors have been noticed. A through proofreading is encouraged.", "was_revised": true, "final_justification": "The authors have provided satisfactory and substantial responses, including relevant demonstrations and clarifications, to all my concerns. Accordingly, I am raising my score to 4. Borderline Accept.", "rebuttal": "We thank Reviewer sExN for their thorough review and insightful comments. We are pleased that the reviewer recognized ReaL’s well-structured experiment settings and superior performance over baselines. We address the reviewer’s concerns as follows:\n\n---\n\n**Response to Weakness 1**\n\n> *“More implementations details are needed for better clarity.”*\n\nWe provide additional technical details to help the reviewer better understand our work. In the **REAL** framework, we fine‑tune **Qwen2.5‑Coder‑Instruct** using **Proximal Policy Optimization (PPO)** with a **hybrid reward** that balances **functional correctness** and **code quality** (security and maintainability). At each training step, the model generates candidate solutions for coding prompts, evaluates them using unit‑test pass rates and static analyzers, and combines these scores into a normalized hybrid reward. The static analyzer leverages control‑flow graphs and taint analysis to detect unsafe data flows from user input to execution. Based on the reward signals, we compute advantages using Generalized Advantage Estimation (GAE) and update the policy with PPO’s clipped objective, augmented by KL‑penalty and entropy regularization to ensure stable learning and maintain exploration. This framework allows **REAL** to improve steadily using scalable and verifiable feedback.\n\n---\n\n**Response to Weakness 2**\n\n> *“Please make it more clear that as benchmarked on 3 curated datasets as SecCodePLT+, SafeSQL and APPS+, proposed method outperforming because of these specifically curated dataset or for employing a RL-based framework or combining both. Generalization over real-world raw dataset should be explored.”*\n\nWe clarify that the datasets used in this paper are **not specifically curated** for our method, but are used to **establish a new experimental setting** and to **train and evaluate all baseline methods**, ensuring a fair comparison. The original SecCodePLT and APPS datasets remain unchanged; we only add new verifiers (security and maintainability) as extra evaluation dimensions. The SafeSQL dataset was curated solely to study SQL development due to the lack of existing datasets. And it is also used to train the SFT baseline. Thus, REAL’s superior performance stems from our RL‑based framework with hybrid reward mechanisms, not from data differences.\n\nTo further validate generalization, we trained REAL on APPS+ and evaluated it on the unseen HumanEval [1] benchmark. The result is as follows. REAL‑7B achieves substantial improvements across all metrics despite not being trained on HumanEval, demonstrating strong generalization. \n\n| HumanEval | Functionality | Quality | Functionality-Quality |\n| --- | --- | --- | --- |\n| Qwen-Coder-Instruct-7B | 0.7927 | 0.2561 | 0.2317 |\n| REAL-7B | 0.8476 | 0.9573 | 0.8171 |\n\n[1] Chen, Mark, et al. \"Evaluating large language models trained on code.\" *arXiv preprint arXiv:2107.03374* (2021).\n\n---\n\n**Response to Weakness 3**\n\n> *“Authors should also points towards current limitations and potential breakpoints of the REAL framework (if any), particularly in the context of real-world quality code generation applications”*\n\nWe appreciate the reviewer’s request. We would like to highlight our novelty in our focus on the quality code and the use of program analysis in reward mechanism, and discuss current limitations of the **REAL** framework.\n\n- **Focus on quality code:** REAL is the **first** to apply reinforcement learning to optimize for **code quality**—specifically **security and maintainability**—rather than just functional correctness. This is increasingly critical as coding assistants are integrated into real-world workflows. Prior RL efforts mainly focus on solving tasks via unit tests, overlooking real risks like unsafe or unmaintainable code. Our work addresses this important and under-explored problem.\n- **Use of program analysis in reward:** Unlike prior works that rely on unit tests as rewards—which are task-specific and insufficient for quality aspects—we leverage **program analysis** (e.g., taint analysis and control-flow graphs) as a **general-purpose reward signal**. This enables scalable and automated evaluation of vulnerabilities and maintainability across diverse prompts, without the need for handcrafted test cases.\n- **Limitations and future directions:** Our current focus is on **single-turn** quality code generation. We acknowledge that many real-world applications involve **multi-turn interaction** and iterative refinement. Extending REAL to such settings—e.g., through agent-based frameworks that incorporate feedback loops—is a key direction for future work.\n\n---\n\n**Response to Question 1**\n\n> *“Instead of curated datasets, what if we evaluate on previous SecCodePLT [Yang e al., 2024] and APPS [Hendrycks et al., 2021] without any preprocessing. Can you please demonstrate similar benchmarking as Table 3-5?”*\n\nWe’d like to clarify that we did **not curate or preprocess** new data for the SecCodePLT and APPS benchmarks. Our updated versions (SecCodePLT+ and APPS+) use **exactly the same data**—we simply add static analysis tools as an **additional evaluation metric** to assess code quality (security and maintainability), which was not covered in the original benchmarks that only used unit tests.\n\nDataset statistics are shown below:\n\n|  | Training set | Testing set |\n| --- | --- | --- |\n| SecCodePLT | 655 samples | 164 samples |\n| SecCodePLT+ | 655 (exactly the same data as SecCodePLT) | 164 (exactly the same data as SecCodePLT) |\n| APPS | 2038 samples | 519 samples |\n| APPS+ | 2038 (exactly the same data as APPS) | 519 (exactly the same data as APPS) |\n\n> *“Can REAL, as a novel stand alone RL framework, applicable for any unknown real world raw datasets (generalizability) towards satisfying maintainability and vulnerability check (can we expect self-automated)?”*\n\nYes, this is a core strength of the REAL framework. By integrating **program analysis tools as reward functions**, REAL can generalize to **unseen, real-world datasets** without manual curation. To demonstrate this, we trained on APPS+ and evaluated on **HumanEval**—a benchmark with different styles and no overlap in training data. Despite no exposure to HumanEval, REAL significantly improves code quality:\n\n| HumanEval | Functionality | Quality | Functionality-Quality |\n| --- | --- | --- | --- |\n| Qwen-Coder-Instruct-7B | 0.7927 | 0.2561 | 0.2317 |\n| REAL-7B | 0.8476 | 0.9573 | 0.8171 |\n\n[1] Chen, Mark, et al. \"Evaluating large language models trained on code.\" *arXiv preprint arXiv:2107.03374* (2021).\n\n---\n\n**Response to Question 2**\n\n> *“Authors mentioned as \"with minimal manual intervention\", however used 3 curated datasets for evaluation. How minimal was this \"curation\" as \"by extending existing datasets or evolving data with LLM\"? Is there any bias introduced due to these curated datasets in training? Can it anyhow misguide the RL agent in specific scenario, for complex logic codes especially in dynamic contexts?”*\n\nWe’d like to clarify that “minimal manual intervention” refers to the use of **program analysis tools** as automated reward functions in REAL. These tools significantly reduce human effort in detecting vulnerabilities in generated code, especially when compared to manually crafting case-specific unit tests.\n\nFor example, given 1,000 SQL programs, a single static analyzer can check for SQL injection vulnerabilities—eliminating the need to manually write and annotate 1,000 individual test cases.\n\nAs for the datasets mentioned in the question, we’d like to clarify that the three mentioned datasets were **not curated specifically for training REAL**, but rather to establish a sound experimental setting for evaluating security and maintainability—both for our method and future work. We will include these clarification in our future version to make this point clearer.\n\n---\n\n**Response to Question 3**\n\n> *“Program analysis-based feedback might not always provide explainable rationales for the improvements or rejections. Any concerns there?”*\n\nIn REAL, we use program analysis–based feedback as part of the reward function, represented as a binary signal (1 for secure, 0 for vulnerable). While this feedback is not explicitly explainable, our experiments show that it is **sufficient to guide learning** and significantly improve code quality.\n\nThis practice aligns with **standard approaches in the RL with verifiable rewards (RLVR)** literature [1,2,3,4], where binary or scalar reward signals are commonly used. The advantage estimation mechanism in PPO effectively captures reward differences between successful and unsuccessful outputs, enabling the policy to learn the desired behaviors—even in the absence of detailed rationale.\n\n[1] AI2: Lambert, Nathan, et al. \"Tulu 3: Pushing frontiers in open language model post-training.\" *arXiv preprint arXiv:2411.15124* (2024).\n\n[2] DeepSeek: Guo, Daya, et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" *arXiv preprint arXiv:2501.12948* (2025).\n\n[3] Microsoft Research: Wen, Xumeng, et al. \"Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs.\" *arXiv preprint arXiv:2506.14245* (2025).\n\n[4] Meta AI: Gehring, Jonas, et al. \"Rlef: Grounding code llms in execution feedback with reinforcement learning.\" *arXiv preprint arXiv:2410.02089* (2024)."}, {"review_id": "88TQ3qCI32", "reviewer": "Reviewer_YAqt", "summary": "This work proposes a method called REAL, whose core idea is to introduce a hybrid reward mechanism during the training process of large code models. The hybrid reward consists of two components: one is a program vulnerability detector based on static program analysis techniques, and the other is a functional validator that verifies whether the function's intended behavior is achieved. By combining these two rewards, the large code model achieves performance improvements on three datasets: SecCodePLT+, APP+, and SafeSQL. The authors also present concrete examples to demonstrate how the code model \"evolves\" into a more secure model.", "questions": "1.\tWhy does the model perform so well on the SafeSQL dataset? Does this suggest overfitting or that the task itself lacks sufficient challenge?\n2.\tWhy are the scores in the “Quality” column significantly higher than those in the other two columns?", "limitations": "N/A", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.\tUsing static program analysis techniques as a signal for code quality is meaningful, as it helps alleviate the issue of sparse rewards.\n2.\tThe paper is clearly written.\nWeaknesses:\n1.\tThe idea of using reinforcement learning to train code models is not particularly novel. In particular, it remains unclear whether reinforcement learning — especially preference optimization algorithms like PPO — is truly beneficial for code-related tasks. According to the authors’ experimental results, REAL does not appear to deliver significant performance improvements.\n2.\tThe paper lacks important experimental details. For example, how was reinforcement learning applied? What data was used as seed data? How were the experiments conducted? How does the hyperparameter $\\alpha$ affect the trade-off between code quality and functional correctness?", "was_revised": true, "final_justification": "The core contribution represents only an evolutionary refinement of existing approaches. It falls significantly short of the innovative breakthrough anticipated in the introduction. The claimed novelty exceeds the actual incremental advancement delivered. Not sufficient for acceptance.", "rebuttal": "We sincerely thank the reviewer for their time and thoughtful feedback. We are pleased to hear that the reviewer finds our method meaningful and appreciates the clarity of our writing. Below, we address the reviewer’s concerns in detail:\n\n---\n\n**Response to Question 1**\n\n> *“Why does the model perform so well on the SafeSQL dataset? Does this suggest overfitting or that the task itself lacks sufficient challenge?”*\n\n- This is not overfitting, as we use separate data for training and evaluation. Moreover, during RL training, the model receives no ground-truth answers—only feedback from program analysis and unit tests.\n- The strong performance likely comes from the dataset focusing on a single CWE (SQL injection), enabling the model to specialize in avoiding this vulnerability.\n- Despite targeting one CWE, SafeSQL remains challenging. As shown in Table 2, untrained base models perform poorly, with joint pass rates of only **20%**, **27%**, and **58%** for 0.5B, 3B, and 7B models, respectively.\n\n---\n\n**Response to Question 2**\n\n> “Why are the scores in the Quality column significantly higher than those in the other two columns?”\n\nFunctionality and quality capture different aspects of code generation and are **not directly comparable**. Their difference arises from **how each metric is defined**:\n\n- **Functionality**: Measures the percentage of prompts where the generated code passes *all* unit tests—an all-or-nothing criterion.\n- **Quality**: Measures the percentage of prompts passing *all* program analysis checks. However, this metric can be gamed—for example, by generating empty code—so it should not be used in isolation.\n- **Joint (Functionality & Quality)**: Requires passing *both* unit tests and analysis checks, making it the strictest metric. As a result, its score is naturally lower than either individual metric.\n\n---\n\n**Response to Weakness 1**\n\n> *“The idea of using RL to train code models is not particularly novel.“*\n\n- We do **not** claim novelty in applying RL to code generation broadly. Instead, our contribution lies in **what** we optimize and **how**.\n- We are the **first to use RL with Program Analysis feedback** to train LLMs that generate not just functional, but also **secure and maintainable** code—an urgent and under-explored problem, as also recognized by **Reviewers 5a54** and **qzWn**. Prior RL-for-code work focuses primarily on **functionality** via unit tests. In contrast, we address **quality**—i.e., security and maintainability—which prior work overlooks.\n- We are also the **first** to propose using **Program Analysis** as the reward signal. Unlike unit tests, which are impractical and problem-specific, program analysis offers a **general, scalable, and task-agnostic** mechanism to detect vulnerabilities and enforce maintainability standards (e.g., type annotations), making it uniquely suited for RL optimization in this setting.\n\n> *“In particular, it remains unclear whether reinforcement learning — especially preference optimization algorithms like PPO — is truly beneficial for code-related tasks.”*\n\n- **RL has demonstrated clear benefits for coding tasks**: Recent models like DeepSeek-R1 [1], Kimi-K2 [2], and Qwen3-Coder [3] are all post-trained with RL (typically using unit test feedback) and show strong performance on code benchmarks. While details are not public, Anthropic’s blog suggests Claude also uses RL [4]. While the broader impact of RL on reasoning remains open, its value for math and coding tasks is well supported.\n- **Clarifying PPO**: PPO is not inherently a \"preference optimization\" algorithm—it is a general-purpose RL method. In RLHF, it’s often used to optimize learned preference models, but in our case, PPO directly optimizes **hybrid rewards** based on program analysis and unit test feedback. This encourages the model to generate code that is not just correct, but also secure and maintainable.\n\n[1] DeepSeek: Guo, Daya, et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" *arXiv preprint arXiv:2501.12948* (2025).\n\n[2] Kimi: Team, Kimi, et al. \"Kimi K2: Open Agentic Intelligence.\" *arXiv preprint arXiv:2507.20534* (2025).\n\n[3] Qwen Team: \"Qwen3-Coder: Agentic Coding in the World.\" Blog post. Qwen Blog, 2025.\n\n[4] Anthropic Team. “Claude’s Constitution.” Anthropic News blog post. May 9, 2023. Anthropic.\n\n> *“REAL does not appear to deliver significant performance improvements.”*\n\nWe respectfully disagree. REAL consistently yields **substantial gains** over both the vanilla base model and the strongest baseline (SFT), especially when evaluated on the Joint Pass-Rate, a strict metric requiring the code to pass both unit tests and static analysis checks.\n\nAcross all datasets and model sizes, REAL improves over the vanilla model by up to **+57.7%**, and over SFT by up to **+16.5%**. Notably, for the 7B model, REAL outperforms SFT by **+15.3%** on SecCodePLT+, **+12.9%** on SafeSQL, and **+15.4%** on APPS+. These consistent gains across benchmarks and scales clearly demonstrate the effectiveness of our program-analysis-guided RL framework.\n\n---\n\n**Response to Weakness 2**\n\nThe implementation details can be found in Appendix A.2 and we add more details below to help improve the quality of our paper and we will definitely include more in the final revision.\n\n> *“how was reinforcement learning applied?”*\n\nWe employ **Proximal Policy Optimization (PPO)** on top of a pretrained code model (Qwen2.5‑Coder) using our hybrid reward. We use the VeRL framework to implement this and we mainly focus on the reward design. All the code will be released and we will also add more details in the paper.\n\n> *“What data was used as seed data? ”*\n\nTo curate the SafeSQL dataset, we asked human experts to write 10 database programming tasks that are potentially vulnerable to SQL injection. These tasks span diverse, realistic scenarios—including filtering, sorting, and aggregation—to ensure broad coverage of query types.\n\nEach prompt was carefully reviewed by both human experts and a strong LLM (GPT‑4.1) to ensure clarity and solvability. For example, one task asks the model to query a `books` table for entries within a user-specified price range and genre, simulating a typical real-world use case prone to SQL injection risks in the code solutions.\n\n> *“How were the experiments conducted?”*\n\nOur experiment setup is introduced in Section 4.1 and more implementation details are provided in Appendix A.2.  Generally, we implement ReaL based on the widely used VeRL framework and conduct all experiments on an 8 x H100 GPUs server. The backbone model used in our experiments is Qwen2.5-Coder-Instruct. We will elaborate more on the experiment details in the final revision.\n\n> *“How does the hyperparameter α affect the trade-off between code quality and functional correctness?”*\n\nOur experiments reveal a clear trade-off: optimizing solely for correctness or quality often hurts the other. To study this, we vary the weighting factor **α** in the hybrid reward and evaluate on SecCodePLT+ and SafeSQL:\n\n### SecCodePLT+\n\n| Scale | Metric | α=0 | α=0.3 | α=0.5 | α=0.7 | α=1 |\n| --- | --- | --- | --- | --- | --- | --- |\n| 0.5 B | Function | 0.7317 | 0.7012 | 0.5854 | 0.4207 | 0.0732 |\n|  | Quality | 0.1402 | 0.6524 | 0.7988 | 0.9512 | 0.9268 |\n|  | **Func‑Qual** | **0.0854** | **0.3659** | **0.3963** | **0.4024** | **0.0732** |\n| 3 B | Function | 0.7683 | 0.7622 | 0.7378 | 0.7317 | 0.1585 |\n|  | Quality | 0.2805 | 0.7378 | 0.8476 | 0.8598 | 0.9024 |\n|  | **Func‑Qual** | **0.2073** | **0.5183** | **0.6037** | **0.5915** | **0.1220** |\n| 7 B | Function | 0.7805 | 0.7622 | 0.7561 | 0.7012 | 0.3415 |\n|  | Quality | 0.3476 | 0.872 | 0.8293 | 0.872 | 0.8354 |\n|  | **Func‑Qual** | **0.2866** | **0.6402** | **0.6159** | **0.6037** | **0.2866** |\n\n### SafeSQL\n\n| Scale | Metric | α=0 | α=0.3 | α=0.5 | α=0.7 | α=1 |\n| --- | --- | --- | --- | --- | --- | --- |\n| 0.5 B | Function | 0.7765 | 0.7647 | 0.7177 | 0.6588 | 0.3529 |\n|  | Quality | 0.0471 | 0.8941 | 1.0000 | 1.0000 | 1.0000 |\n|  | **Func‑Qual** | **0.0235** | **0.6941** | **0.7177** | **0.6588** | **0.3529** |\n| 3 B | Function | 0.8588 | 0.8471 | 0.8353 | 0.8235 | 0.5529 |\n|  | Quality | 0.0824 | 1.0000 | 1.0000 | 1.0000 | 1.0000 |\n|  | **Func‑Qual** | **0.0588** | **0.8471** | **0.8353** | **0.8235** | **0.5529** |\n| 7 B | Function | 0.8706 | 0.8706 | 0.8235 | 0.8118 | 0.6353 |\n|  | Quality | 0.0588 | 1.0000 | 1.0000 | 1.0000 | 1.0000 |\n|  | **Func‑Qual** | **0.0353** | **0.8706** | **0.8235** | **0.8118** | **0.6353** |\n\nAs shown, **low α** (favoring correctness) leads to poor quality scores, while **high α** (favoring quality) degrades functional correctness. Both extremes result in lower **joint pass rates**. In contrast, moderate values (e.g., **α = 0.3–0.5**) strike a better balance, yielding the strongest overall performance across scales and datasets.\n\nThese findings validate our **hybrid reward design**, demonstrating that RL can flexibly optimize different objectives through careful tuning of α. In our main results, we use **α = 0.5** as a balanced default, but the framework remains adaptable to varied priorities.\n\nWe hope this analysis addresses the reviewer’s question. We’re happy to clarify further if needed."}, {"review_id": "Bqoau3elkc", "reviewer": "Reviewer_5a54", "summary": "This manuscript proposes ‘REAL’, a framework that uses reinforcement learning (RL) model to integrate dual automated signals; i.e., a program analyzer to detect maintainability defects and unit tests to ensure functional correctness. REAL was designed to enhance the quality of code generated by LLMs using a prompt-agnostic and reference-free approach. The automated, hybrid program analysis-based rewards mechanism gives REALs an upper hand compared to traditional methods that depend on manual annotations or brittle heuristics. REAL is leveraged to generate production-ready code by combining static detection of security and maintainability flaws along with unit testing to provide comprehensive feedback, which guides the LLM's code generation workflow. To accomplish this, the authors develop three extended benchmarks: (1) SecCodePLT+ (covering 18 Common Weakness Enumerations), (2) SafeSQL (targeting SQL injection vulnerabilities), and (3) APPS+ (assessing Python maintainability), to evaluate the REAL across diverse quality metrics. Results suggest that REAL-trained models outperform both the baseline fine-tuning and rule-based approaches, thus achieving an 85.9% success rate in functionality on SafeSQL and 92.3% maintainability on APPS+. REAL advances scalability and demonstrates elegant generalization across model sizes within 0.5B to 7B parameters. The novelty of REAL’s is its ability to leverage static analysis within a hybrid reward function to guide code generation toward safer and more maintainable outputs.", "questions": "1. How is the hybrid reward function (e.g., $r_{\\text{total}} = \\alpha r_{\\text{quality}} + (1 - \\alpha) r_{\\text{func}}$) calibrated? I would recommend that the authors experiment with multiple values of α to assess its sensitivity.\n\n\n2. How did the authors validate the correctness and robustness of the custom detectors (e.g., for sanitization or code style)? Do they generalize outside the training distribution?\n\n\n3. Can the authors confirm if they measured or observed the RL-trained model exploiting brittle or sparse reward patterns, i.e., engaging in reward hacking?\n\n\n4. Why were strong recent baselines (e.g., DeepSeekCoder, Claude-Code, StarCoder2) excluded from comparison? Would adding them affect the paper's conclusions?\n\n\n5. Can the author report statistical confidence in the evaluation metrics (e.g., via multiple seeds, bootstrapping, or CI bars) to rule out that observed gains are not noise?", "limitations": "No. While the paper acknowledges the challenges of program analysis integration, it does not adequately address the potential brittleness or bias in static detectors, nor does it fully explore generalization risks or limitations of the hybrid reward design. Ethical considerations related to overtrust in detector-based training are also missing.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## **Strengths**\n\n1. **Novel**\n\n Referencing Section: 3.1 Vulnerability Detector: \n\n - This work uses Program Analysis and integrates symbolic vulnerability detectors in the RL framework reward signal. This integration provides a significant contribution to the safe LLM code generation process.\n\n2. **Scalability** \n\n - Section: 4.1 Experiment Settings (Pg. 5, Lines: 194–202), Section: 4.2 Quantitative Results (Pg. 7, Lines: 240–260), and Table 3 with surrounding discussion (Lines: 240–260) confirming scalability:\n - This RL framework scales across model sizes (i.e., from 0.5B to 7B) and is customizable with open-source tools, thus aiding reproducibility and increasing practical real-world adoption.\n\n3. **Empirically grounded**\n\n- This manuscript illustrates consistency in performance improvements across different benchmarks, with clear gains in both functionality and security-related metrics.\n\n- Table 5, SecCodePLT+ and SafeSQL (security) and APPS+ (maintainability), with REAL outperforming all baselines in both quality and functionality.\n\n\n4. **Hybrid Reward Design**: \n\n- Section: 3.2 Reinforcement Learning with Hybrid Rewards (Pg. 4, Lines: 156–186)\n\n- The authors implement a thoughtful balance of complementary objectives in SWE activities by blending functional correctness with program quality indicators.\n\n\n\n5. **Reproducibility Pipeline**: \n- Section: Introduction, Pg. 1, Lines: 10–11, Pg. 5, Lines: 194–202,\n\n- The methodology used in this paper relies on openly available models and datasets, thus increasing its accessibility to the entire research community for replication and verification of results.\n\n- Both data sources, APPS and SecCodePLT, are available and use open source tools like MyPy [Lehtosalo, 2025].\n\n- REAL is prompt-agnostic and reference-free, hence scalable without manual intervention.\n\n\n\n\n\n### **Weaknesses**\n\n#### **Quality**\n\n1. **Lack of proper validation** Pg. 3, Lines 36–50; Pg. 4, Lines 74–85\n\n   - The authors did not formally evaluate the static analysis tools used to compute `r_quality` in a SWE context. Leaving the reader to wonder about their accuracy and susceptibility, especially to false positives/negatives. The absence of this evaluation undermines confidence in the reward signal's reliability and Static Analysis. Between false positives and false negatives, which is mostly severe and unacceptable in this use case?\n   \n\n2. **Binary Reward Granularity** Pg. 4, Eq. 2; Figure 5\n\n   - The vulnerability reward is binary (0/1), and the hybrid reward lacks ablation or sensitivity testing with respect to the weighting parameter α. Without understanding how the reward channels interact, training robustness remains unclear.\n   \n\n3. **Lack of Qualitative Analysis** Section 5–6\n    \n    - Strange enough, no examples of incorrect, hallucinated, or low-quality code generations are reported in the paper. Failure case breakdowns are essential for understanding where the model goes wrong and for establishing trustworthy usage in production.\n   \n\n4. **No Human Evaluation**\n   \n   - Even though the paper automates the evaluations, yet, quality code at production still depends on subjective human judgment. I.e., metrics like readability, structure, and idiomatic expression—dimensions are best judged by humans. Even in this case, a solid measure of IRR should be implemented in the study design.\n\n\n\n#### **Significance**\n\n5. **Benchmarks Generalization Concerns**  Pg. 2, Lines 41–55\n   \n   - In this study, the authors extended the APPS+, SecCodePLT+, and SafeSQL Benchmarks on their use case. However, little is known about their representative assessment of actual codebases, thus leaving generalization to production code unverified.\n  \n\n6. **Benchmarks Overfitting Risk**\n   \n   - The authors performed an extensive fine-tuning and validation on the curated datasets. However, they did not test against distribution shift, using industrial-grade data that is usually more complex with interdependencies and sometimes suffers from poorly documented code patterns. These limitation (gaps) increases the risk of generalizability and robustness under distribution shift. Thereby reducing the quality and safety of the code generated by REAL.\n\n7. **Unvalidated Metric** Section: 3.2, Pg. 5, Lines 182-186; Section 4.2, Table 3, Pg. 7 \n   - The paper treats code quality and functionality as a single metric, but does not provide theoretical or empirical support for its weighting scheme or assess whether the metric masks failures in one dimension. Pg. 5 line 184 simply provides the interval [0,1]\n   \n   - The authors provide no empirical/conceptual justification for how 𝛼 was chosen. Even though the paper claims trade-off between quality and functionality, tit is not clear how 𝛼 was calibrated.  \n   - While conducting a post-hybrid analysis using GAE and PPo is essential, I still do not understand why there is no sensitivity analysis on varying 𝛼 to observe the relationship between r_quality and r_function against performance.\n\n\n\n\n\n#### **Originality**\n\n8. **Algorithmic Contributions**\n\n  - Overall, it seems that REAL suffers a bit of novelty concerns because the framework uses LLM as its policy model, the environment is the code generation task, and nothing special about that. The reward is derived from symbolic program analysis and unit tests combined (not human feedback). The authors also used PPO as-is; no modification was done architecturally or optimally.\n\n  - While I appreciate and see some novelty contextually in the use of reward design, the underlying RL setup (PPO with reward shaping) is basic. Therefore, the primary originality lies only in the integration of symbolic signals into an RL framework for LLMs.\n\n\n#### **Clarity** Pg. 8, Section 5.2\n\n9. **Lack of Transparency in the Dataset Annotation Process**\n   \n   - It is not clear to me how the paper addresses potential human biases in the annotation process. Little is known about the labelling of vulnerability indicators and how the dataset was curated, validated, or verified. At least, I would appreciate a discussion of the IRR and how agreements/disputes were resolved among human annotators.\n\n\n10. **Scalability Claims**\n\n   - The paper claims that REAL scales across model sizes (0.5B, 3B, and 7B) with consistent improvements in SFT and RL-based baselines. Despite these claims, evidence for training stability and reproducibility is not discussed or presented. \n\n   - Despite the superficial demonstrations of scale through performance metrics, the paper fails to show evidence through Training reward/loss curves, Standard deviation/error bars on results, Convergence discussion (e.g., speed, stability, and Hardware resource comparison across sizes.\n\n\n   - Although the method is tested across LLM sizes, training stability, convergence behavior, and variance across runs are not sufficiently discussed.", "was_revised": true, "final_justification": "The rebuttal has improved the manuscript's clarity substantially, and provides additional empirical evidence addressed some of my earlier concerns considerably. Notably:\nReward Sensitivity – The authors added a sweep over α values across two benchmarks (SecCodePLT+ and SafeSQL), confirming that α ≈ 0.5 balances functionality and quality while showing predictable trade-offs when shifting emphasis.\nDetector Validation – They now present in-domain (SecCodePLT+) and out-of-distribution (HumanEval) evaluations, including comparisons to existing tools (Bandit, Bearer) with strong gains in precision, recall, and F1, reinforcing the reliability of their static analysis signals.\nReward Hacking Mitigation – Clear evidence is provided of reward hacking with static-analysis-only rewards, and the corrective effect of combining unit tests with quality checks is demonstrated.\nHuman Evaluation – A small-scale expert study (50 APPS+ problems, 80% win rate for REAL outputs) was added to address readability and maintainability aspects not captured by automated metrics.\nNovelty Clarification – The contribution is now explicitly framed as the first integration of program-analysis–driven reward shaping in RL for code generation, targeting maintainability and security alongside functionality.\nAnnotation Process Transparency – More detail is given for SafeSQL construction, including multi-pass expert verification and cross-checking, though IRR remains unreported.\nStatistical Reliability – After discussion, the authors provided preliminary multi-seed results (SafeSQL, five seeds) with low standard deviations, indicating stable performance; they committed to updating the manuscript with these results.\nThese additions address the majority of the technical and methodological issues I raised. The paper still has some remaining limitations:\nStatistical rigor has improved, but the full set of multi-seed results and statistical analyses (e.g., CIs, variance across all benchmarks) should be incorporated in the final manuscript for complete transparency.\nQualitative failure analysis is still somewhat limited; more diverse real-world failure case studies would further strengthen the diagnostic value.\nGeneralization to production settings remains only partially addressed; evaluation on more complex, interdependent industrial codebases is still an open step.\nOverall, the rebuttal demonstrates responsiveness and meaningful improvements. The core idea, leveraging program analysis as a scalable, domain-relevant reward signal in RL for LLM code generation, is well-motivated, and the new results reinforce its technical merit. While some gaps remain for future work, the contribution is solid and timely. I maintain a borderline accept rating, leaning toward acceptance given the strengthened empirical grounding and practical relevance.", "rebuttal": "We sincerely thank Reviewer 5a54 for the insightful review and detailed suggestions. We are grateful that the reviewer carefully checked our paper and found our method novel, scalable, and empirically grounded. \n\nThe concerns are mainly about **detail clarifications**. \n\nWe first answer the questions and then address the remaining concerns.\n\n---\n\n**Response to Question 1**\n\n> *“How is the hybrid reward function (e.g., ) calibrated? I would recommend that the authors experiment with multiple values of α to assess its sensitivity.”*\n> \n\nIn our experiments we fixed α = 0.5, which delivered a strong balance of correctness and quality. To probe its sensitivity, we swept α over {0, 0.3, 0.5, 0.7, 1.0} on two benchmarks (SecCodePLT+ and SafeSQL). \n\nAs α increases, the functionality score predictably declines, but by tuning α just above or below 0.5 we can actually surpass the default’s joint “Func-Qual” performance.\n\n(1) SecCodePLT+\n\n|Scale|Metric|α=0|α=0.3|α=0.5|α=0.7|α=1|\n|---|---|---|---|---|---|---|\n|0.5 B|Function|0.7317|0.7012|0.5854|0.4207|0.0732|\n| |Quality|0.1402|0.6524|0.7988|0.9512|0.9268|\n| |**Func-Qual**|**0.0854**|**0.3659**|**0.3963**|**0.4024**|**0.0732**|\n|3 B|Function|0.7683|0.7622|0.7378|0.7317|0.1585|\n| |Quality|0.2805|0.7378|0.8476|0.8598|0.9024|\n| |**Func-Qual**|**0.2073**|**0.5183**|**0.6037**|**0.5915**|**0.1220**|\n|7 B|Function|0.7805|0.7622|0.7561|0.7012|0.3415|\n| |Quality|0.3476|0.8720|0.8293|0.8720|0.8354|\n| |**Func-Qual**|**0.2866**|**0.6402**|**0.6159**|**0.6037**|**0.2866**|\n\n(2) SafeSQL\n\n|Scale|Metric|α=0|α=0.3|α=0.5|α=0.7|α=1|\n|---|---|---|---|---|---|---|\n|0.5 B|Function|0.7765|0.7647|0.7177|0.6588|0.3529|\n| |Quality|0.0471|0.8941|1.0000|1.0000|1.0000|\n| |**Func-Qual**|**0.0235**|**0.6941**|**0.7177**|**0.6588**|**0.3529**|\n|3 B|Function|0.8588|0.8471|0.8353|0.8235|0.5529|\n| |Quality|0.0824|1.0000|1.0000|1.0000|1.0000|\n| |**Func-Qual**|**0.0588**|**0.8471**|**0.8353**|**0.8235**|**0.5529**|\n|7 B|Function|0.8706|0.8706|0.8235|0.8118|0.6353|\n| |Quality|0.0588|1.0000|1.0000|1.0000|1.0000|\n| |**Func-Qual**|**0.0353**|**0.8706**|**0.8235**|**0.8118**|**0.6353**|\n\n---\n\n**Response to Question 2**\n\n> *“How did the authors validate the correctness and robustness of the custom detectors (e.g., for sanitization or code style)? Do they generalize outside the training distribution?”*\n> \n\nWe validate our detectors both **in-domain** and **out-of-distribution**:  \n\n- In-Domain Validation (Security Detector)\n    \n    Generally, we evaluate the detector using positive and negative pairs. For example, we leverage the SecCodePLT+ test set, which comprises 164 coding problems—each annotated with one “safe” and one “unsafe” solution for a specific CWE. For each detector, we compute true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN), then derive precision, recall, and F1-score. \n    \n    Below is a representative subset of results, comparing our detector against two popular open-source tools (Bandit and Bearer):\n    \n    |CWE ID|Metric|Bandit|Bearer|Ours|\n    |---|---|---|---|---|\n    |77|Precision|0.6721|0.1961|1|\n    |77|Recall|0.8039|0.3279|1|\n    |77|F1-score|0.7321|0.3279|1|\n    |770|Precision|0.5|0|0.9808|\n    |770|Recall|0.0196|0|1|\n    |770|F1-score|0.0377|0|0.9903|\n- Out‐of‐Distribution Generalization (Maintainability Detector)\n    \n    To assess generalization, we evaluate on the **HumanEval** benchmark, whose problem styles differ markedly from our training data (APPS+). Crucially, none of the HumanEval samples were seen during training. We measure both functional correctness and code quality to compute a hybrid “Functionality-Quality” score.\n    \n    |Model|Function|Quality|Function-Quality|\n    |---|---|---|---|\n    |Qwen-Coder-Instruct-7B|0.7927|0.2561|0.2317|\n    |**REAL-7B**|**0.8476**|**0.9573**|**0.8171**|\n\n---\n\n**Response to Question 3**\n\n> *“Can the authors confirm if they measured or observed the RL-trained model exploiting brittle or sparse reward patterns, i.e., engaging in reward hacking?”*\n> \n\nWe explicitly monitored for reward‐hacking behaviors and confirmed that our performance gains were not due to exploiting brittle or sparse reward signals. \n\nWe did observe reward‐hacking when using **only** the static‐analysis signal, and here is an example code that the model generates null code to bypass the static-analysis tool:\n\n```python\ndef generate_post_html(author: str, title: str, body: str, post_tags: list[str]) -> str:\n    # Implement the function to generate HTML for the post\n    pass\n```\n\nWe fixed this by adding correctness unit test and the model cannot simply generate null code to achieve a high reward. We manually checked and found the code was functional and high-quality.\n\n---\n\n**Response to Question 4**\n\n> *“Why were strong recent baselines (e.g., DeepSeekCoder, Claude-Code, StarCoder2) excluded from comparison? Would adding them affect the paper's conclusions?”*\n> \n- Our method focuses on improving LLMs' capability to generate both correct and high-quality code (security and maintainability), not chasing SOTA performance. Performance comparisons against base model and SFT show significant gains, demonstrating our method's effectiveness.\n- These stronger models will not affect our conclusion — we have shown that REAL works across different model scales (0.5B, 3B, 7B), so stronger backbone models will also benefit from our training paradigm.\n\n--- \n\n**Response to Question 5**\n\n> *“Can the author report statistical confidence in the evaluation metrics (e.g., via multiple seeds, bootstrapping, or CI bars) to rule out that observed gains are not noise?”*\n> \n\nWe can confirm that the observed gains are not from randomness or data noise. Extensive experiments on three different datasets (SecCodePLT+, SafeSQL, and APPS+) and three different model scales (0.5B, 3B, and 7B) show consistently significant gains.\n\n---\n\n**We find that most of the Weaknesses are already addressed in the answers above. We respond to the remaining uncovered ones as follows:**\n\n---\n\n**Response to Weaknesses 3 & 4**\n\n> *“…Failure case breakdowns are essential…” & “… best judged by humans…”*\n> \n- For the failure cases, we will add more in the revision. The failure cases mostly covers (1) null code with empty implementation, (2) correct but unsafe code, and (3) incorrect and unsafe code.\n- For human evaluation, we recruited two experts to do a **winning-rate–based evaluation** via pairwise comparisons on 50 coding problems from APPS+. For each problem they picked between the Qwen2.5-7B base output and the Qwen2.5-7B+REAL output, judging maintenance, readability, and fault tolerance (e.g., comments, type annotations, brevity, and error resilience). The REAL-trained model won 80% of the time. We will include more details in the revision.\n\n---\n\n**Response to Weakness 7**\n\n> *“The paper treats code quality and functionality as a single metric ... Pg. 5 line 184 simply provides the interval [0,1]”*\n> \n\nThe reviewer’s comment conflates the hybrid *training* reward with our *evaluation* metrics. In fact, we evaluate **functionality**, **quality**, and their **combination** separately; each is reported as a pass rate and defined in the paper (Line 213, Page 6).\n\n---\n\n**Response to Weakness 8**\n\n> *Algorithmic Contributions: Overall, it seems that REAL suffers a bit of novelty concerns …*\n> \n\nWe do **not** claim novelty in the general application of RL to code generation. Our contribution lies in **what** we optimize and **how** we do it.\n\n- We are the **first to leverage RL with Program Analysis feedback** to train LLMs that generate code that is not only functional but also **secure and maintainable**—a critical yet under-addressed problem, as highlighted by **Reviewers 5a54** and **qzWn**. While previous RL-for-code efforts focus almost exclusively on **functional correctness** using unit tests, we tackle **code quality**, including security and maintainability, which has been largely overlooked.\n- Additionally, we are the **first to introduce Program Analysis as a reward signal**. In contrast to unit tests—which are narrow, task-specific, and hard to scale—program analysis provides a **general, scalable, and task-agnostic** way to detect security flaws and enforce maintainability (e.g., checking for type hints). This makes it an ideal signal for guiding RL in this domain.\n\n---\n\n**Response to Weakness 9**\n\n> *“It is not clear to me how the paper addresses potential human biases in the annotation process …”*\n> \n\nThank you for pointing this out. The SafeSQL dataset was constructed through a multi-step process to reduce bias:\n\n- **Seed Tasks:** Human experts wrote 10 seed tasks covering SQL injection–prone scenarios across diverse query types.\n- **LLM Expansion:** We used **GPT-4.1** to mutate these tasks (e.g., changing constraints or merging cases), followed by **deduplication and expert verification**.\n- **Cross-Verification:** Multiple annotators reviewed each example, and disagreements were resolved via discussion. We also applied **program analysis tools** to enforce consistency.\n- **Functionality Tests:** For each task, we generated I/O pairs from reference solutions to create unit tests.\n\nWhile we did not compute IRR scores, we reduced annotation variance through multi-pass review and plan to include formal IRR in future work.\n\n---\n\nWe sincerely thank the reviewer for the detailed feedback and constructive suggestions. We believe our comprehensive responses and additional experimental results address all the raised concerns. We welcome any further questions and are committed to addressing any remaining issues.\n\nWe appreciate the reviewer's consideration of our responses and hope they find our clarifications satisfactory."}, {"review_id": "MaExSfQ5n3", "reviewer": "Reviewer_qzWn", "summary": "The authors propose training an LLM model to generate safe, functional, and maintainable code. To measure safety, they use program analysis-based techniques. For functionality, they use unit test cases. For maintainability, they use some simple static analysis tools.\n\nThey augment SecCodePLT by identifying 17 CWEs to create the SecCodePLT+ dataset.\nThey create a new dataset called SafeSQL, a targeted dataset with realistic database query tasks susceptible to SQL injection attacks.\n\nThe authors train the model in an RL setting, with a score that is a tradeoff between security and utility. They can also detect multiple vulnerabilities simultaneously.   \n\nThe quality reward measures the presence of a vulnerability in the code.\nThe functionality reward measures the pass rate of the model.\nThe hybrid reward combines the above metrics using a linear combination of the scores.", "questions": "Will the authors open-source the tool they created for the detection of 17+ vulnerabilities?\n\n\n(Q) Can the authors provide insight into the following observation: Based on the results, the SFT model seems to be a very strong competitor model, especially for the functionality metric on SecCodePLT+.\n\nCan the authors share insight into why this is the case?", "limitations": "Although there seems to be an in-depth discussion on the limitations of related work, I did not find any explicit paragraph on the limitations of REAL, and also the possible negative societal impacts of their training strategy. \n\nLLMs that have the ability to generate both safe and functional code can have important implications on the software engineering industry through the means of automation.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n\nThe paper is clear and well written. It is easy to follow and understand. The key contributions are evident in the writing, in both the datasets and in the methodology. The paper proposes a novel technique to integrate both the practical aspects of code safety and functionality.  It addresses an important gap in existing work and bridges this gap through a joint optimization framework that optimizes for both functionality and security. \n\nThe approach is novel and is unique. It is an elegant way to combine both the functional and security scores in a unified format. It also seems like an original idea and a good contribution to this downstream task. The problem being tackled is significant and important to this research area.\n\n\nWeakness\n\nBased on the results, the SFT model seems to be a very strong competitor model, especially for the functionality metric on SecCodePLT+.", "was_revised": true, "final_justification": "I appreciate the authors’ thoughtful and timely rebuttal. It clarified several points from my review and addressed some questions at a high level. Balancing the helpful clarifications against the remaining limitations, my overall evaluation is unchanged, and I maintain my original score.", "rebuttal": "We sincerely thank Reviewer qzWn for their positive and thoughtful assessment. We appreciate the recognition of our paper’s clarity, the novelty of our joint optimization framework, and the significance of addressing both functionality and security in code generation.\n\nWe address the reviewer’s concerns as follows:\n\n**Response to Weakness**\n\n> *“Can the authors provide insight into the following observation: Based on the results, the SFT model seems to be a very strong competitor model, especially for the functionality metric on SecCodePLT+.”*\n> \n\nThis is a critical point! Indeed, the SFT model performs strongly on **functionality** in SecCodePLT+, even surpassing REAL. We attribute this to the relatively straightforward nature of the benchmark: SFT benefits from direct supervision using ground-truth solutions, allowing it to learn precise patterns that match test cases.\n\nIn contrast, **REAL**, trained via reinforcement learning, **does not rely on ground-truth answers**. Instead, it explores candidate solutions and learns from reward feedback. While this can lead to slightly lower functionality in simple benchmarks, it offers greater flexibility in satisfying **complex and diverse constraints**—such as the 17 CWEs and broader code quality goals. As shown in our results, REAL excels in **security and maintainability**, demonstrating its ability to go beyond pattern-matching and adapt to real-world quality requirements.\n\nIn short, while SFT is strong at mimicking known solutions, REAL is designed to generalize and optimize for broader, verifiable objectives without relying on heavy supervision.\n\n---\n\n**Response to Question**\n\n> *“Will the authors open-source the tool they created for the detection of 17+ vulnerabilities?”*\n> \n\nYes, we recognize the importance of open-sourcing to support the research community. We plan to release the vulnerability detection tool after completing internal validation and adding documentation to ensure its usability and reliability. This will help promote reproducibility and enable future work on quality code generation.\n\n---\n\n**Response to Limitation**\n\n> *“Although there seems to be an in-depth discussion on the limitations of related work, I did not find any explicit paragraph on the limitations of REAL, and also the possible negative societal impacts of their training strategy.”*\n> \n\nThank you for raising this important point. A clear discussion of REAL’s limitations and potential societal impacts will indeed help strengthen the paper.\n\n- **Limitations:** REAL currently focuses on **single-turn code generation**, where the model outputs a complete solution in one step. However, many real-world scenarios involve **multi-turn or interactive coding**, where bugs and vulnerabilities may emerge through complex interactions across multiple code fragments. Detecting and resolving such issues is more challenging and remains an open area for extension of our framework.\n- **Potential Societal Impact:** While REAL is designed to enhance code quality and security, it could be **misused by malicious actors** to generate more sophisticated or harmful code. Moreover, improved automation may reduce the need for human oversight in certain development tasks, potentially impacting **entry-level programming roles**. These risks underscore the importance of responsible deployment—through access control, robust evaluation safeguards, and continued human oversight in the code generation process."}]

Author rebuttal:

Meta-review: The paper proposes REAL, a reinforcement learning (RL) framework to incentivize LLMs to generate production-quality code that goes beyond functional correctness by incorporating security and maintainability qualities. REAL leverages program analysis to detect defects and unit tests to ensure correctness, offering prompt-agnostic and reference-free supervision for scalability.

Strengths:
- Clear experimental setup and benchmarks, showing strong performance over baselines (sExN, 5a54)
- While RL for code models has been studied before (5a54, YAqt), the paper innovatively extends it to target security and maintainability, integrating program analysis into the reward signal (5a54, qzWn)
- Scalable across model sizes and customizable with open-source tools (5a54)

Weaknesses:
- Method description remains somewhat theoretical, lacking deeper technical/implementation details (sExN, YAqt, qzWn), though many concerns were clarified in rebuttal
- Limited discussion of current limitations and possible breakpoints of the framework (sExN), along with missing qualitative analysis (5a54)
- Absence of human evaluation on subjective qualities such as readability (5a54). The authors addressed this issue during rebuttal. 

In summary, the paper addresses an important challenge in LLM-based code generation by moving beyond correctness toward production-level qualities. The approach is well-motivated, simple yet promising with comprehensive experimental results. I advise the authors review all the feedback and improve the paper with more concrete technical details, analysis of limitations, and human validation to strengthen their contributions.
Decision: Accept (poster)