Learning to Generate Unit Tests for Automated Debugging
Authors: name: Archiki Prasad; name: Elias Stengel-Eskin; name: Justin Chih-Yao Chen; name: Zaid Khan; name: Mohit Bansal
Published: 2025-02-03 18:51:43

Abstract: Unit tests (UTs) play an instrumental role in assessing code correctness as
well as providing feedback to a large language model (LLM) as it iteratively
debugs faulty code, motivating automated test generation. However, we uncover a
trade-off between generating unit test inputs that reveal errors when given a
faulty code and correctly predicting the unit test output without access to the
gold solution. To address this trade-off, we propose UTGen, which teaches LLMs
to generate unit test inputs that reveal errors along with their correct
expected outputs based on task descriptions and candidate code. We integrate
UTGen into UTDebug, a robust debugging pipeline that uses generated tests to
help LLMs debug effectively. Since model-generated tests can provide noisy
signals (e.g., from incorrectly predicted outputs), UTDebug (i) scales UTGen
via test-time compute to improve UT output prediction, and (ii) validates and
back-tracks edits based on multiple generated UTs to avoid overfitting. We show
that UTGen outperforms UT generation baselines by 7.59% based on a metric
measuring the presence of both error-revealing UT inputs and correct UT
outputs. When used with UTDebug, we find that feedback from UTGen's unit tests
improves pass@1 accuracy of Qwen-2.5 7B on HumanEvalFix and our own harder
debugging split of MBPP+ by over 3% and 12.35% (respectively) over other
LLM-based UT generation baselines.