Training Language Models to Generate Quality Code with Program Analysis
  Feedback
Authors: name: Feng Yao; name: Zilong Wang; name: Liyuan Liu; name: Junxia Cui; name: Li Zhong; name: Xiaohan Fu; name: Haohui Mai; name: Vish Krishnan; name: Jianfeng Gao; name: Jingbo Shang
Published: 2025-05-28 17:57:47

Abstract: Code generation with large language models (LLMs), often termed vibe coding,
is increasingly adopted in production but fails to ensure code quality,
particularly in security (e.g., SQL injection vulnerabilities) and
maintainability (e.g., missing type annotations). Existing methods, such as
supervised fine-tuning and rule-based post-processing, rely on labor-intensive
annotations or brittle heuristics, limiting their scalability and
effectiveness. We propose REAL, a reinforcement learning framework that
incentivizes LLMs to generate production-quality code using program
analysis-guided feedback. Specifically, REAL integrates two automated signals:
(1) program analysis detecting security or maintainability defects and (2) unit
tests ensuring functional correctness. Unlike prior work, our framework is
prompt-agnostic and reference-free, enabling scalable supervision without
manual intervention. Experiments across multiple datasets and model scales
demonstrate that REAL outperforms state-of-the-art methods in simultaneous
assessments of functionality and code quality. Our work bridges the gap between
rapid prototyping and production-ready code, enabling LLMs to deliver both
speed and quality.