Code Aesthetics with Agentic Reward Feedback
Authors: name: Bang Xiao, name: Lingjie Jiang, name: Shaohan Huang, name: Tengchao Lv, name: Yupan Huang, name: Xun Wu, name: Lei Cui, name: Furu Wei
Organization: Microsoft Research

Large Language Models (LLMs) have become valuable assistants for developers
in code-related tasks. While LLMs excel at traditional programming tasks such
as code generation and bug fixing, they struggle with visually-oriented coding
tasks, often producing suboptimal aesthetics. In this paper, we introduce a new
pipeline to enhance the aesthetic quality of LLM-generated code. We first
construct AesCode-358K, a large-scale instruction-tuning dataset focused on
code aesthetics. Next, we propose agentic reward feedback, a multi-agent system
that evaluates executability, static aesthetics, and interactive aesthetics.
Building on this, we develop GRPO-AR, which integrates these signals into the
GRPO algorithm for joint optimization of functionality and code aesthetics.
Finally, we develop OpenDesign, a benchmark for assessing code aesthetics.
Experimental results show that combining supervised fine-tuning on AesCode-358K
with reinforcement learning using agentic reward feedback significantly
improves performance on OpenDesign and also enhances results on existing
benchmarks such as PandasPlotBench. Notably, our AesCoder-4B surpasses GPT-4o
and GPT-4.1, and achieves performance comparable to large open-source models
with 480B-685B parameters, underscoring the effectiveness of our approach.