A.S.E: A Repository-Level Benchmark for Evaluating Security in
  AI-Generated Code
Authors: name: Keke Lian, name: Bin Wang, name: Lei Zhang, name: Libo Chen, name: Junjie Wang, name: Ziming Zhao, name: Yujiu Yang, name: Haotong Duan, name: Haoran Zhao, name: Shuang Liao, name: Mingda Guo, name: Jiazheng Quan, name: Yilu Zhong, name: Chenhao He, name: Zichuan Chen, name: Jie Wu, name: Haoling Li, name: Zhaoxuan Li, name: Jiongchi Yu, name: Hui Li, name: Dong Zhang

The increasing adoption of large language models (LLMs) in software
engineering necessitates rigorous security evaluation of their generated code.
However, existing benchmarks often lack relevance to real-world AI programming
scenarios, making them inadequate for assessing the practical security risks
associated with AI-generated code in production environments. To address this
gap, we introduce A.S.E (AI Code Generation Security Evaluation), a
repository-level evaluation benchmark designed to closely mirror real-world AI
programming tasks, offering a comprehensive and reliable framework for
assessing the security of AI-generated code. Our evaluation of leading LLMs on
A.S.E reveals several key findings. In particular, current LLMs still struggle
with secure coding. The complexity in repository-level scenarios presents
challenges for LLMs that typically perform well on snippet-level tasks.
Morever, a larger reasoning budget does not necessarily lead to better code
generation. These observations offer valuable insights into the current state
of AI code generation, assisting developers in selecting the most appropriate
models for practical tasks, while laying the foundation for refining LLMs to
generate secure and efficient code in real-world applications.