CODESIM: Multi-Agent Code Generation and Problem Solving through
  Simulation-Driven Planning and Debugging
Authors: name: Md. Ashraful Islam; name: Mohammed Eunus Ali; name: Md Rizwan Parvez
Published: 2025-02-08 18:43:59

Abstract: Large Language Models (LLMs) have made significant strides in code generation
and problem solving. Current approaches employ external tool-based iterative
debuggers that use compiler or other tool-based runtime feedback to refine
coarse programs generated by various methods. However, the effectiveness of
these approaches heavily relies on the quality of the initial code generation,
which remains an open challenge. In this paper, we introduce CodeSim, a novel
multi-agent code generation framework that comprehensively addresses the stages
of program synthesis-planning, coding, and debugging-through a human-like
perception approach. As human verifies their understanding of any algorithms
through visual simulation, CodeSim uniquely features a method of plan
verification and internal debugging through the step-by-step simulation of
input/output. Extensive experiments across seven challenging competitive
problem-solving and program synthesis benchmarks demonstrate CodeSim's
remarkable code generation capabilities. Our framework achieves new
state-of-the-art (pass@1) results-(HumanEval 95.1%, MBPP 90.7%, APPS 22%, and
CodeContests 29.1%). Furthermore, our method shows potential for even greater
enhancement when cascaded with external debuggers. To facilitate further
research and development in this area, we have open-sourced our framework in
this link (https://kagnlp.github.io/codesim.github.io/).