Otter: Generating Tests from Issues to Validate SWE Patches
Authors: Toufique Ahmed
Jatin Ganhotra
Rangeet Pan
Avraham Shinnar
Saurabh Sinha
Martin Hirzel
Venue: ICML 2025 poster (2025)
Keywords: LLMs
SWE Patches
Reproduction Tests
Primary area: applications->language_speech_and_dialog
TL;DR: This paper introduces Otter, a system that generates tests from software issues, supporting test-driven development and validation of automated code patch generation, and outperforms state-of-the-art systems in experiments.

Abstract: While there has been plenty of work on generating tests from existing code, there has been limited work on generating tests from issues. A correct test must validate the code patch that resolves the issue. This paper focuses on the scenario where that code patch does not yet exist. Doing so supports two major use-cases. First, it supports TDD (test-driven development), the discipline of "test first, write code later" that has well-documented benefits for human software engineers. Second, it also validates SWE (software engineering) agents, which generate code patches for resolving issues. This paper introduces TDD-Bench-Verified, a benchmark for generating tests from issues, and Otter, an LLM-based solution for this task. Otter augments LLMs with rule-based analysis to check and repair their outputs, and introduces a novel self-reflective action planner. Experiments show Otter outperforming state-of-the-art systems for generating tests from issues, in addition to enhancing systems that generate patches from issues. We hope that Otter helps make developers more productive at resolving issues and leads to more robust, well-tested code.

Otter: Generating Tests from Issues to Validate SWE Patches
Toufique Ahmed!
Jatin Ganhotra' Rangeet Pan
Avraham Shinnar' Saurabh Sinha Martin Hirzel
Abstract
2002). The recent introduction of SWE-Bench (Jimenez et al., 2024) has spurred work resolving SWE issues automatically with LLMs, typically with agents (Yang et al., 2024; Ruan et al., 2024; Wang et al., 2024b). Such issue- resolution systems are often called SWE agents. We posit that, just like up-front tests help human SWE engineers, they also help automated SWE agents.
While there has been plenty of work on generating tests from existing code, there has been limited work on generating tests from issues. A correct test must validate the code patch that resolves the issue. This paper focuses on the scenario where that code patch does not yet exist. Do- ing so supports two major use-cases. First, it supports TDD (test-driven development), the dis- cipline of "test first, write code later" that has well-documented benefits for human software en- gineers. Second, it also validates SWE (software engincering) agents, which generate code patches for resolving issues. This paper introduces TDD- Verifiee benchmark for generating tests from issues, and Otter, LLM-based solution for this task. Otter augments LLMs with rule-based analysis to check and repair their outputs, and introduces a novel self-reflective action planner. Experiments show Otter outperforming state-of- the-art systems for generating tests from issues, in addition to enhancing systems that generate patches from issues. We hope that Otter helps make developers more productive at resolving is- sues and leads to more robust, well-tested code.
While test generation is active area of research, prior work focuses on creating tests for existing code, not on creating tests from issue descriptions alone. A recent solu- tion SWE-Agent+ (Mündler et al., 2024) attempts to create tests from issue descriptions albeit with limited success (the state-of -frr-ar SWE-Agent+ only generates a fail-to-pass test in 19.2% cases and is quite expensive, requiring several LLM calls and tokens). See Section for more in-depth discussion of related work.
This paper introduces Otter (acronym for "Otter: TDD-Test gEnerator for Reproducing issues"). It takes as input an issue description and the original code in the repository, and generates as output a set of tests y. Otter contains a novel self-reflective action planner for deciding which code to read and which tests write. It also uses rule-based code analyses and transformations throughout the workflow to curate LLM inputs, validate LLM outputs, and repair generated tests, Otter generates fail-to-pass tests in 31.4% of cases, and in an ensemble of size 5 dubbed Otter++, that increases to 37.0%. At the same time, the cost (even with ensembling) less than $0.10 per issue with GPT-4o. Tests generated by Otter adhere to the same testing framework as repository's existing test suite, to which they can be added.
1. Introduction
A software engineering (SWE) issue is bug report fea- ture request for improving the code in repository. Before software engineer attempts to write new code that resolves a bug, they typically write a reproduction test to confirm the presence of the bug in the old code. In fact, even when work- ing on a feature request, before writing new code it rec- ommended to create an acceptance test first, to confirm the presence of the feature in the new code once written. This practice is known as test-driven development (TDD), and it improves the quality of both tests and the code itself (Beck,
Today the most popular benchmark for automatically resolv- ing SWE issues is SWE-bench Verified (Chowdhury et al., 2024). To evaluate solutions such as Otter that automatically generate tests from issues, we introduce a new benchmark, TDD-Benchh Be-DBenh--eerifed. MDD-Bench-- Verified evaluates tests by checking whether the tests a) fail the old code before issue resolution, b) pass the new code, and c) cover the code changes well. The fact that TDD-Bench-Veriee derived from SWE-bench Verified enables us to empirically study the effects of generated tests on SWE agents. We observe that the tests from Otter++ can be used to trade precision for recall on the SWE-bench Verified leaderboard. For instance, for the system ranked 3rd on the leaderboard,
is
'IBM Research, Yorktown Heights, New York, USA. Corre- spondence to: Toufique Ahmed stfahmed@ibm.comss com, Martin Hirzel <hirzel @us. wom.ibm.com...
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Otter: Generating Tests from Issues to Validate SWE Patches
filtering by the generated tests boosts precision from 60.8% to 91.9% while reducing recall to 33%. This paper makes the following contributions:
Libro to Python and measured a fail-to-pass rate of 15.2%.) Plein et al. (2024) proposed another test-generation system for Defects4J, reporting a fail-to-pass rate of 6%. Both systems have relatively low success rates, and unlike our work, neither evaluates the impact of generated tests on issue-resolving systems.
Otter, a system that generates tests from issues, using LLMs with novel self-reflective action planning tech- nique along with rule-based pre- and post-processing. Verified', a benchmark for evaluating tests generated from issues, including a high-quality dataset and a metric based on test results and coverage.
When resolving issues, some SWE agents also generate tests along the way. The original Swe-Agent (Yang et al.. 2024), a single-agent system, attempts to reproduce the is- sue, as explicitly instructed in its prompt. Some multi-agent systems- Destems-CodeR (Chen et al., 2024) and SpecRover (Ruan et al., 2024) start with a Reproducer agent for generating tests. However, none of the three (SWE-Agent, CodeR, or SpecRover) are evaluated for the effectiveness of their gen- erated tests. Agentless (Xia et al., 2024) relies on inference scaling, generating several candidate patches and several tests. It then uses the tests to help rank the patches, ulti- mately choosing single patch to submit. The effectiveness of the tests is evaluated indirectly by their impact on issue resolution rate (from 27% to 32%), not directly for their own fail fail-to-pass rate or coverage like in our work.
An empirical study on using tests generated from issues to filter rssuu-resolution candidates for SWE-bench Verified.
Generated tests can assist software engineers both before and after resolving an issue, and can increase trust in auto- mated SWE agents.
2. Problem Statement
This work focuses on the problem of generating tests from issues. Specifically, the input x is a pair (dissue, Cold) of an issue description dissue alongside the old version Cold of the code before the issue is resolved. The issue description is typically in natural language, but it may sometimes contain embedded code snippets or stack traces. The code is snapshot of all the files and folders in Python source code repository. The expected output y is a set of tests that should go from failing on Cold to passing on Cnew, which is the new version of the code after the issue is resolved. By failing on Cold, the tests reproduce the issue, and by passing on Cnews they validate its resolution. Besides going from failing to passing, the tests should also maximize coverage of the code change (formalized in Section 5). A solution to this problem is thus function gentests that takes an input x and returns tests = gen (e)) The new code Cnew is not available to genTesss, which must generate tests y based on x alone. This is representative of the real world where source code repositories may have regression tests for existing code but lack tests for open issues. Otter provides an implementation of the Tests function and, thus, solution to this problem.
Three very recent systems are dedicated generating tests from Python issues. Aegis (Wang et al., 2024a) is multi- agent system that uses inference scaling, but the exact dataset for their evaluation is unclear (the paper says SWE- bench Lite, but then compares against numbers from another system on SWT-bench Lite, which is different). Aegis is more costly than Otter++, and unlike our work, the Aegis paper (a) does not report coverage numbers and (b) does not evaluate how tests can help trade off precision vs. recall w.r.t. performance of SWE agents. EvoCoder (Lin et al., 2024) uses experiences from prior issues to help with the latest issue at hand, which is complementary to Otter. Un- like our work, the generated tests are not integrated with the existing CI pipeline. Furthermore, the experiments do not use exccution-based metries, making it hard to compare empirically. SWE-Agent+ adapts patch-generating agent to generate tests instead (Mündler et al., 2024) and achieves 9..2% tail-to-pass rate on the SWT-bench I Lite dataset intro- duced by the same paper. SWT-bench applies less rigorous quality filters than TDD-Benchh Verified, and measures cov- erage in a round-about way by first running additional tests than just generated ones and then subtracting them back out, Using tests from SWE-Agent+ as filter improves preci- sion of SWE agents to 47. 47.8% while reducing recall to 20%. Otter++ outperforms S SWE-Agent+ on all of these metrics.
3. Related Work
Prior to SWE-bench (Python) (Jimenez et al., 2024), De- fects4J (Java) (Just et al., 2014) has been popular bench- mark in the community. The creators of Defects4J care- fully curated and cleaned up each issue by hand. Unlike SWE-bench, Defects4J only contains bug reports, no fea- ture requests. The earliest system we are aware of that generates tests from issues, Libro (Kang et al., 2023). fo- cuses on Defects4J. Libro achieves fail-to-pass rate of 19.9% with one generation. (Mündler et al. (2024) ported
CodeT (Chen et al., 2023) is one of the first approaches that leverage the same LLM to automatically generate both code samples and corresponding test cases. CodeT then executes generated code samples using the generated test cases and performs dual execution agreement to choose the best solution and test. CodeT is not directly applicable
hhtps://gihuu..co///////II BenhD-BBehhh---- Verified
2

---

Otter: Generating Tests from Issues to Validate SWE Patches
the project. They do this by localizing relevant existing tests and focal functions (i.e., functions exercised by those tests and likely places for fix). Depending on the developer's familiarity with the project and the nature of the issue, the difficulty of localization may vary. Inspired by human devel- opers' actions, our approach also starts with test and focal function localization. The localizer phase collects files from Cold that contain at least one test function for test local- ization. presents the list of files and the issue description dissue to the LLM and asks it to generate 10 relevant test file names. Our initial findings on the dev set of SWE-bench indicated that the Top- Top-5, and Top-1 accuracy for test file localization with GPT-4o are 83.6%, 76.0%, and 59.1 1,, respectively. We restricted file retrieval to 10 files so as not to overwhelm the contexts for subsequent LLM calls. Next, the localizer validates the file names by comparing them with the previously collected file list and drops the ones that do not match. After localizing test files, it makes a second LLM call with file names and test function names from those files. The model chooses the test files and functions relevant to dissue- The localizer again validates the retrieved file names, but instead of dropping any hallucinated file names, replaces them with the file names with minimal edit distance. This ensures validity of identified files, which is essential for the subsequent rest-generation phase. Note that even if the localizer chooses the wrong test file, if the file is at least valid, the test generator may still succeed. Focal file and function localization follows a similar two-LLM-call approach. Figures 14 and 15 in the Appendix illustrate the LLM prompts for localization.
description
imput
code
dican
Otter
I Localize test files
Localize focal files
Localizer
Localize best functions
Localize local functions
local files and functions
files and functions
Execule read actions
Validate planned actions
Self-refleetive action planner
Make initial plan
Reflect and improve plan
planned actions
Execute read actions
Flesh out write or modify action
Test generator
Repair generated tests
Legend:
generated tests
oulpul. y
Jata
LLM
Rule-based
Figure 1. Overview of Otter.
Self-Reflective Action Planner. The second phase of Ot- creates a plan, which is list of actions for generating the fail-to-pass tests y. There write, modify. A read f action reads a function f from Cold to use as context in a prompt. A write f or modify f action declares the intent to write a new test function or modify an existing test function. Here, f a file and func- tion name (in the planner, write and modify actions do not yet include the exact code for the test function, which left to the test generator phase of Otter). The planner starts by executing read actions for the files and functions provided by the localizer. Next, it prompts an LLM with the function definitions, the issue description dissue. and instructions to make an initial plan, restricted to only read actions. The next step of the planner validates the planned actions: checks whether the actions generated by the LLM refer to valid file and function names. The final step of the planner is "reflect and improve plan", an LLM call with prompt including feedback from validation. At this point, the plan is no longer restricted to only read actions, and can also contain a write or modify action. Section B in the appendix provides statistics on read/write/modify actions generated by the planner. The model also instructed self-reflect on
three kinds of actions: read,
current setup because we have multiple tests but code patch. CodeMonkeys (Ehrlich et al., 2025) is another relevant work that iteratively and jointly improves patch with test, which interesting and complementary to Otter. Unlike our work, their experiments do not evaluate the fail- to-pass rate nor the coverage of generated tests.
4. Methodology
This section describes three solutions to the problem stated in Section 2: Otter, Otter++, and a baseline approach.
4.1. Otter: Test Generation Guided by Self-reflective Action Planner
Figure presents an overview of Otter, which has three main components: a localizer, planner, and a test generator.
Localizer. Before developers tackle an issue, they usually start by gaining an understanding of the current state Cold of
3

---

Otter: Generating Tests from Issues to Validate SWE Patches
the proposed plan with one of three possible outcomes: "Sat- isfied". "Unsatisfied", and "Unsure". If the model chooses "Sattsfied",, Otter moves forward to the test-generation phase. For other options, it returns to the validation step and then repeats the "reflect and improve plan" step. This process is repeated at most five times. In most cases, the model is satisfied with the plan in the first two turns. The two planner prompts are presented in Figures 18 and 19 (Appendix).
of localization may produce different fail-to-pan fail-to-pass tests that are not generated by Otter. Otter++ uses the test generated by Otter (TI) and adds four new tests (T2-T5), obtained by skipping the planner stage, and including neither, one, or both of focal and test localization. In other words, Otter++ runs the test generator stage five times with different, her- erogeneous prompts. We favor heterogeneous prompts with greedy decoding over homogeneous prompts with higher model temperature because our initial experiments showed that the latter yields poorer tests and lower diversity.
Test Generator. The cest-generation phase of Otter exe- cutes the actions computed by the planner, which can in- volve generating
To pick a single test among the five candidate tests, we run the five tests on Cold (recall that Grew is not available yet) and analyze the execution logs. If a test passes on Cald, we just discard that test because violates the fail-to- pass criterion. We classify the remaining tests into three groups: assertion failure, other failure (the test runs but produces the wrong output), and error (the test does not run properly, e.g., because of wrong syntax or an exception in a fixture). We pick a test from the first non-empty group to maximize the chance that failed for the right reason (i.e., it reproduces the bug described in the issue). If the selected group has multiple tests, we break the tie with pre-determined ordering of the five prompts that favors tests from prompts with more or better information: Otter, followed by the prompts with both localizers, test localizer only, focal localizer only, and neither localizer, in that order.
test (write action) or updating existing test (modify action). To guide the LLM in this task, we extract the test structure and imports from the localized test file and make them available in the prompt. This reduces the burden on the LLM to generate imports. The file struc- ture is relevant for new tests, to determine their insertion point in the test file. Otter uses different prompt for a write vs. modify action, illustrated in Figures 20 and 21 in the Appendix. For new tests, the model needs to generate the preceding function name in addition to the test. Unlike most SWE agents, Otter does not try to generate diffs; instead, it asks the model to generate complete test functions, even for the modification case. Since model training data tends to contain more complete functions than diffs, we expect the model to perform better at generating functions.
To handle missing or hallucinated imports, Otter includes an soport-fixing step in this phase, where it looks at model- generated imports and linting errors detected using Flake82 (a static analysis tool) to identify missing imports. Note that Flake8 reports different styling errors; we manually curated the error codes catch name-related errors. case of missing imports, we add dummy import to the function. Then, we take the model-generated and dummy imports and try to find the imported module among the files in the codebase. If we find the module, we replace the model- generated/dummy import with the one from the codebase; otherwise, we continue with the model pondel-generated/ddmmmy import as a fallback. Finally, we add the function in the codebase and generate a git diff create the test patch.
4.3. Baseline: Zero-shot Test File Generation
Recent cioructiiontunede LLMs excel at following instruc- tions (Peng et al., 2023; Zhang et al., 2023). We propose a simple zero-shot approach to generate a fail-to-past test given the repository name and the issue description. Given the prompt, the model generates a complete test file with all necessary imports to make it compilable. In real scenarios, test files usually have multiple test cases, but this bascline usually generates only a single test per file.
5. TDDD-Bnnch-eeriie Benchmark
This section presents TDD-Bench- Verified, a new bench- mark that supports evaluation of techniques for generating tests from issue description and an old code version, with- out access to new code that resolves the issue (see problem statement in Section 2).
4.2. Otter++: Ensemble using Multi-Sampling with Heterogeneous Prompting
Otter has several components (e.g., localizer, planner), but they are not perfect. The file localizer accuracy for focal and test localization is 82.4% and 70.6% with GPT-4o, re- spectively. As the output from localization serves as input to later LLM calls, those later calls may be affected by inac- curacies in localization. Conversely, LLMs can sometimes generate fail-to-pass pass tests in zero-shot setup, even without any context. So, selectively including and excluding parts
5.1. FDDD-ench--o
Evaluation Harness
Figure 3 shows the harness for evaluating tests y, which typically come from genTests solution, but the harness can also be applied on golden tests ý mined from a pull request (PR). The evaluation harness runs in a containerized environment. Starting at the top left, tests come the form of patch, which is applied (via git apply) on the old
4

---

Otter: Generating Tests from Issues to Validate SWE Patches
Verifled by human annotators
Not sble recreaie X
Not lal-lo-pass or zero coverage
Random sample of SWE-Bench 1,699
SWE-Bench Verified
44
500
Ran successfully 493
Instances 449
TDD-Bench- BeD-Bench-Verilied
Recreate
449
Not verified 1,199
docker
golden test patch (before and golden code patch)
Add line coverage (before and after patch)
Figure 2. Overall flow of TDD-bench dataset filtering starting from SWE-bench verified.
ende with
reselve and Inration
ified or the golden tests á were overly specific, i.c., would reject some valid new code Cnew- They also removed some instances where tests failed due environment problems.
icandidate or goldent
xit apply
old code Cosd
mant Texts and
tests
In the same spirit, TDD-Bench- Verified applies more filters to obtain an even higher-quality dataset. In a nutshell, the filtering process applies the -e--Benn--eeriieddd evaluation harness (Figure 3) to the golden tests ý from the original PR. Specifically, substituting á wherever y occurs in Figure checks whether the PR indeed contributed tests that went from failing to passing. We filter out any instance where the contributed tests do not satisfy that criterion. Although the human annotators of SWE-bench Verified were diligent, a few instances slipped past their filters, and we drop those for TDD-Bench Verified.
golden code peach
Eail jpass (orppct Gull H)
git apply
gulden new code
PAUH Touds and
Ballpass
(espoct
Figure 3. Evaluation hamesss for TDD-Bench- Verified.
Fi Figure 2 illustrates the filtering process. Starting from the 500 instances of SWE-bench Verified, we first drop in- stances whose environment could not recreate. Next, run the test harness on the golden tests This filters out 44 additional instances because the tests do not have the ex- pected fail-to-pass behavior (25 instances) or have zero line coverage on the golden code patch (19 instances). In the end, 449 high-quality instances remain across 12 repositories. Ta- ble summarizes key statisties of TDD-Bench Verified.
code Cold. Next, the harness analyzes the resulting code Cold y to resolve the contributed test functions y. The harness then executes y while avoiding running other tests that occur in the same file but were not contributed in the patch. This yields test results, including coverage achieved on the old code. At least one of these tests should fail, indicating that the tests reproduce the issue at hand.
Moving to the bottom part of Figure 3, the code changes come from the golden code patch mined from the same PR. which is applied Cold to obtain the golden new code &new- The harness executes the tests y again, this time on the new code, to obtain a second set of test results. This time, all tests should pass, to validate that the issue was indeed resolved. An example test patch is presented in Figure 9.
5.3. Evaluation Metrie
Passing a test does not necessarily mean a patch adequate to address the issue. Aleithan et al. (2024) reported that 31.1% % of the passed code patches in SWE-Bench are suspi- cious due to weak test cases. To evaluate test adequacy, compute the coverage of the submitted test patch. One key difference between SWE-Bench Verified and TDD-Bench- Verified is that the former runs an entire test file to evaluate the submitted patch, whereas we only run the contributed tests y. Not running other test cases enables us to precisely compute coverage of y. If the tests are relevant, they should cover the deleted lines in Cold and the added lines in Gnew- We integrated the Python Coverage package into the 12 repositories and updated the test scripts to allow us to run specific test cases and compute coverage for them.
5.2. Dataset Filters
TDD-Bench- BoD--Bnch--Vrrffied builds upon prior work from SWE- bench (Jimenez et al., 2024) and SWE-Bench Veri- fied (Chowdhury et al., 2024). SWE-bench uses filters to keep only those mined instances for which the set of golden tests ÿ contains at least some tests that fail Cold and pass on the golden new code Ćnew from the same PR. SWE-Bench Verified is subset of SWE-Bench, con- sisting of 500 instances further vetted by human annota- tors (Chowdhury et al., 2024). The annotators filtered out instances where the issue description disse was underspec-
We define the tddScore metric that evaluates the qual- ity of tests generated by solution gen Tests
Tests over set
5

---

Otter: Generating Tests from Issues to Validate SWE Patches
x (zig .1y.... of instances. It returns a number be- tween 0 and 100, the higher the better. is defined as 100 times the arithmetic mean of the per-instance scores:
Table I. Performance of Otter, Otter++, and haseline technique on IDD-Beench Verified.
fail-to-pass
of Bail-to-pass lest in (%)
100 X FEX
Model
Approach
tadScore
Coverage
tddScore(X, X, gen Tests) =
tddScore(z, gen (ests())))
test
Zero-shot
57
12.7
11.8
60.6
Mistral-Large
Otter
121
26,9
25.1
70.5
Given set of tests y gen gen (2) submitted for an instance, the per-instance score is a product of two factors:
Otie++
32.1
28.6
70.4
Zero-shot
84
18.7
17.2
60.0
GPT-O
Otter
141
31.4
29.4
70.6
tddScore(x, v) = faiTToPass (z. y) adequacy (z. y)
Octer++
166
37.0
32.4
71.5
The first factor is a binary correctness metric, using the indicator function for the tests y failing the old code times the indicator function for the tests y passing on the
Table 2. Contribution of each component of Otter.
Mudel
Compuneni
Approads
A frll-
Change MddSooee
new code. While the solution gen Tesls only access to the old code Cold. the evaluation metric also uses the hidden golden new code Enew right after the issue was fixed.
to pass
Oter (TT)
121
25.7
widhout Action Pluming (comppeete) (T2)
20.2
-214
Action Plazner
without Plun Refinemten Oust I uttempo
115
23.8
-7.4
widuou Action Validtioon
22.1
144.0
Misiral-
(asTToPass( y) I [faill E run Tuntests (y, cona)) [fail & run runTeslsyy Cnew)
large
Lecolooess
without Focal Localization" (13)
widiout Test Localicaion"
96
20.2
35.5
21.4
without Focal & Text Localtzation (TS)
51
17.2
33.1
widhout Fixing Import
4.3
Test Generater
The second factor is fraction between O and 1 based on test coverage on the old and new code:
without Impors at Gereration
114
23.7
7.88
Olaer (TI)
141
20.4
without Action Plannigg (completoj (17)
130
23.6
-19.7
Action Plazner
without Poes Resinement (lus latceppt)
130
27.5
-6.5
adequacy(2; y) = cooey, Cold) ) n (cold ' [neww]
without Action Valiationn
25.7
-12.6
+ cov(y, Cnew) ) n (Cnow \
Gota)
GPT-40
without Focal La reafioeiiss (TS)
115
25.2
-14.3
Localtuera
without Test Localicatoon (T4)
24.2
-17.7
Cold \ Cnew!
+ Knew
Cold
without Focal & Tes Lacalication" (TS)
110
24.2
-17.7
Test Generator
without Fixing Import
26.7
.9.2
Adequacy focuses on just the coverage of lines added and deleted when going from the old code to the new code, because those are the most relevant lines to be tested. In the above, cov (y, c) is the set of lines covered by running tests y on code (Cold \ (.eww) is the set of lines deleted by the PR patch; and (Enew \ Cola) is the set of lines added by the PR patch. We evaluate adequacy jointly for added and deleted lines, as some code patches may contain only added or deleted lines.
without Imports Genzaaiin
130
267
-92
"i au followed by action plansing
experiment with 22 systems from the SWE-Bench leader-
board. We ran 22 x 449 x 5 = 49,390 Docker containers or tests (one Docker container per test) to report the results.
6.2. Effectiveness of Otter, Otter++, and the Baseline
Table 1 shows that GPT-4o-based Otter and Otter++ per- form well on test generation, creating fail-to-pass tests for 31.4% and 37% of the instances, respectively, whereas the baseline produced such tests for 18.7% of the instances. The improvements are also reflected in tddScore. We observe similar performance improvements with Mistral-large. The pass 5 rate (where one of the five tests is fail-to-pass) for Otter++ is 44% for GPT--4o and 37% for Mistral-large.
6. Evaluation
We conducted an extensive set of empirical studies, evaluat- ing the effectiveness of our approach ($6.2) and the compo- nents of Otter and Otter++ ($6.3), comparing our approach against existing techniques ($6.4), and investigating the cost effectiveness of Otter ($6.5), characteristics of the generated tests ($6.6), usefulness of the generated tests in supporting automated program repair ($6.7), and possible effects of data contamination ($6.8).
6.3. Ablation Study
Table 2 shows the ablation study for Otter (T1). We also present the individual performance for the other four tests (T2-T5) produced by Otter++. We can see that all the components contribute to Otter's performance. Without action planning, we lose more than 14%-20% of fail-to-pass tests for GPT-4o and 21 11%- 7-%6% of the tests for Mastral
6.1. Experiment Setup
The evaluation used the closed-source GPT-4o (gpt-4o-2024- 08-06) and the open-source Mistral- model (123 billion parameters). All experiments used greedy decoding. For each instance, Otter makes 7-11 LLM calls for T1. Otter++ makes one additional call for each of the other four tests (T2-T5) after the localization stage. To evaluate using the generated tests for SWE agents, we conducted a large-scale
6.4. Comparison with the Approaches of Mündler et al.
Mündler et al. (2024) proposed a set of approaches for gen- erating fail-to-pass tests. We ran Otter and Otter++ on their
6

---

Otter: Generating Tests from Issues to Validate SWE Patches
Table Comparing with approaches proposed by Mündler et al. (2024) on the 276 instances of their SWT-Lite.
Table 4. Cost for running Otter and Otter++ with GPT-4o.
Component
Cost
Cost/Sumple
A of Fail-to pass Tosts
Approach
in (5%
Focal Localization
$8.61
$0.02
Test Localization
$9.63
50.02
ZeroShot (GPT-4)
16
5.8
Action + Generate
$10.94
$0.02
ZeroShotPlus* (GPTT-4
28
10.1
Total for Otter
$29.18
$0.06
LIBRO* (GPT-4)
42
15.2
AutoCodsRover (GPT-4)
25
9.1
Additional Tests (T2-T5)
$11.20
$0.02
SWE-Agent (GPT-4)
46
16.7
Total for Otter++
$40.38
$0.09
SWE-Agent+ (GPT-4)
53
19.2
Table 5. Comparing the coverage of model-generated and developer-wrine written golden tests.
Otter (GPT-4o)
70
25.4
Otter++ (GPT-44)
29.0
6 uses "proposed patch" while generating
Model
Fail to pass?
Otter
Golden test
Yes
93.1
95.5
dataset to study how the approaches compare. They also evaluated zero-shot approaches, which differ from our zero- shot baseline. All of their approaches (including the zero- shot ones) instruct the model to generate a novel code diff format introduced by their paper. Two of their approaches use a proposed patch in the prompt. Mündler et al. 's SWE- agent and SWE-agent+ approaches are derived from SWE- Agent, which was originally designed for generating code patches (Yang al., 2024). Table 3 shows the results. Otter and Otter++ perform better than their best- best-performing ap- proach, generating 70 (25.4%) and 80 (29.0%) fail-to-pass tests compared to 53 (19.2%) fail-to-pass tests generated by SWE-agent+.
Mistral-targe
No
63.8
93.4
93.7
95.6
GPT-4o
No
60.0
93.3
model was hosted locally. Otter++ makes four additional calls and reuses the output from localizers. The total cost for Otter++ (which includes Otter) is $0.09 per instance.
Sample
200-
Sample Sample
Number of Samples
Otter++ uses execution logs on the current code base cold which give it advantage in the final selection stage. The feedback works as a contributing factor to the superior per- formance of Otter++. Note that Otter does not use any execution logs, yet it still performs significantly better than LIBRO (25.4% vs. 15.2% in Table 3). The novel code diff format proposed by Mündler el al. has similarities to our approach. However, their format requires the model per- form additional tasks such writing the file name, change type, and line number in response to one LLM call. Addi- tionally, to instruct the model to generate a specific format, the authors had to include an example that is not relevant to the issue itself. In our generation step, the model only needs to generate the test (and prior function name for positioning new tests).
211%
12.0%.
Number Tums
Figure 4. Number of turns taken in "reflect and improve plan" step. 6.6. Characteristics of the Generated Tests
Coverage of the generated tests. We compared the cov- erage achieved by the Otter-generates tests and the golden tests written by developers. We observe that, for the fail- to-pass tests, Otter-generated and golden tests have very similar and high coverage-more than 90% with both mod- els (Table 5). However, for the other tests, the coverage is quite low for Otter. This indicates that tests with higher coverage are more likely to be fail-to-pase tests.
6.5. Cost Effectiveness of Otter
Prompts complementarity Figure 5 presents the overlap among instances for which fail-to-pass tests could be gener- ated by different prompts using the GPT-4 model. Overall, each of the prompts is successful on some instances on which the other prompts fail, with TI (Otter with GPT-4o) achieving the most success in this respect prrecccingoiiigggggo fail- to-pass tests for 24 instances on which none of the other prompts succeeded in generating such tests. Thus, com- bining the results from the different prompts increases the fail-to-pass rate to 44% (38% for Mistral) at p This
Table 4 presents the cost for invoking the GPT-4o model to process 449 instances with Otter. Each sample requires an average of $0.06. The "reflect and improve plan" step can make 1-5 LLM calls. Figure 4 shows that, for more than 80% of the instances, both GPT-4o and Mistral are satisfied within two calls. Therefore, the total calls vary from 7-8 for most instances. The cost is very low because we do not accumulate context from prior calls in subsequent calls. We do not discuss the cost for Mastral-large because the

---

Otter: Generating Tests from Issues to Validate SWE Patches
instead of higher temperature generate multiple samples. We make multiple LLM calls in different stages and multi- sampling in each stage would exponentially increase the test counts. Therefore, to compare heterogenous prompting with temperature, we generated 5 samples at high temperature (1.0) in the last LLM call in the Test Generator phase of Otter (our best solution). Though the average number of fail- to- to-pass test goes up (117.8 vs. 116.6), the Top-5 and Top-1 1 (using Otter++'s ranker) results remain lower (Top-5: 173 vs. 197 and Top-1: 146 vs. 166). That means heterogenous prompting boosts up the overall ensemble performance if we compare at the same number of samples. Appendix D discusses more details.
T2
74
Figure 5. Number of instances with fail-to-pass tests generated by different prompts.
6.7. Test Generation and SWE Agents
Table 6. Analysis of the Otter-generated tests using GPT-4o model from different perspectives.
The tests generated by our approach can be used for filtering bad code patches and increasing the precision of solutions proposed by different systems from the SWE-bench Verified leaderboard. We take the top 22 systems from the leader- board and run the five tests generated by Otter++. We filter out code patch if all the tests fail on it. Figure 6 shows that this achieves a precision of 65% to 92% while maintaining decent recall of 30%-41% %. except for one system where the precision increased by 22% to 167%. Note that Mündler et al. (2024) achieved 47.8% precision at 20% recall on SWE--geett Using tests generated by Otter++ achieved much higher precision while maintaining greater recall.
Perspective
Calegory
of Sample
Aof fail-to-puss
fail-to-pass ralc
PatchEsisting
122
50
41.0
Type Test
AdditionOnly
327
91
27.8
Pass
69
0
AssertionFai
170
97
$7.1
Test old Codebase
Fail
101
40
39.6
Eroor
109
3.7
Correct
370
118
31.9
Rocal Localization
Wrong
79
23
29.1
Crerect
317
117
36.9
Test Localization
Wrong
132
24
18.2
indicates the potential of incorporating different prompts for test generation.
Apart from filtering SWE-patches, we could use tests to choose the best SWE-patch, which would be a good ap- plication of the Ottergenerated tests. We have tried CodeT ranking on candidates from the top 3 leaderboard systems on SWE- BeeeB-echhWe Verified and observed 2% improvement. Note that some of these leaderboard solutions have already been through good rankers and used superior models. Improving upon these samples using ranker may be difficult.
Detailed analysis of the tests. Table 6 presents the analy- sis of the Otter-generated tests using GPT-4o model from different perspectives. We see that the fail-to-pass rate is higher when patching an existing test than when adding a new test, which is expected because the model can have better context from the existing test. Writing a new test is in- herently more difficult than modifying an existing test. It is expected that the test will fail on the old codebase. However, we found 69 samples in Otter where the test passed on the old codebase. Also, our analysis shows that tests with asser- tion failures have a higher success rate (57.1 % fail-to-pass rate) compared to other groups. We did not see much impact of focal localization on the performance, with 31.9% and 29.1% for correct and incorrect localization, respectively. Test localization has significant impact on the performance (36.9% vs. 18.2% for correct and incorrect localization). In our ablation study, we also found that test localization is more important than focal localization. In Appendix E, we discuss the impact of hallucination replacer in our pipeline.
6.8. Effect of Data Contamination
As HOD--ench--Vriffeen is based on historic GitHub issues, they may be included in the pre-training data of the LLMs we use. To see whether the model simply generates memo- rizes tests, we performed two different experiments.
Model data cut-off date. The cutoff date for the GPT-4o model is October 2023. Unfortunately, we have only one sample dated post-cutoff and could not compare the two groups. Popular GitHub repositories evolve quickly. It is likely that only the snapshot taken during the data collection process was seen by the model and that it performed well on a specific year of data. Figure 7 shows the total and fail-to-pass test distribution by year. We did not observe any pattern in performance among the distributions by year. For example, for 2020, 2021, and 2023, we have very similar
Heterogenous prompts vs. temperature. Otter++ scaled well with samples up to 5, giving 0.5%-5.6% improvement (see Appendix C). Otter++ uses heterogeneous prompting
8

---

Otter: Generating Tests from Issues to Validate SWE Patches
buckkono
33000
codestory, 3.
fram.ty, www.rrtrrrtaa... cousees
Pyse:
doeo ve
energenc
gt
apam
K33
dove
Operliance- Codeten 2.10-
Systems
google Jure gimint 2.0
ensterbbe
Masiina bimbberty
Figure 8. Cumulative percentage of Onter-gnnerated new tests, us- ing GPT-4o, with maximum similarity less than the similarity value shown on the x-axis.
open S.
boo d.
7. Limitations
comocebo sance
--3726
nave goste sonnet
One limitation of TDD-Bench- Be--c--neer---ie is that it is mined from 12 popular Python repositories, so our findings may not apply to other programming languages and repositories. We note that SWE-bench, despite having the same limitation, has been impactful, and one of the findings in the SWE- bench paper is that "difficulty does not correlate with issue resolution date", indicating that contamination problems (if any) are minor (Jimenez et al., 2024). Our results on data contamination ($6.8) indicate the same thing. A limitation of Otter is that it considers only one test file and generates only one block of code. In real-world projects, test code can be spread across multiple files or blocks of code. Despite that, Otter exceeded the state-of-the-an performance, so we leave further improvements to future work. We use the Python coverage package for computing test coverage, but this package can fail for various reasons, such permission issues, version incompatibilityy configuration problems. In Otter, we computed coverage for all projects, including SymPy. However, upon manual validation, we found the coverage information for SymPy to be unreliable. Therefore, we removed coverage from the final tddScore metric for SymPy instances (<15% of the total instances). Note that coverage does not affect the reported fail-to- soll-to-ass scores.
mergent
ogen gol4c
Prectaion
300
Figure 6. Precision of all 22 systems from the leaderboard. Recall is also mentioned the top of each bar.
Mail Sampre Fal te Pars Sample
40-
2010
2023
2022
2002
2013
Year
Figure 7. Sample distribution of total and fail-to-pass tests by year.
performance by the model.
Test similarity. We conducted another experiment follow- ing the approach of Schäfer et al. (2024). We compute the similarity score between the generated test from Otter and the most similar test from the repository, as follows: dis(t" (f))
8. Conclusion
mar (1- meat(ent(") (e)) ), where TP is the set of test functions and t* is the generated test. Figure 8 shows the similarity scores for new tests. For 90% the instances, the similarity score is less than 0.6. Table 12 in the appendix shows some samples with more than 0.5 similarity give the reader some idea. From our observation, even at simi- larity score of 0.7, the tests are significantly different. As expected, for modified tests, the similarity is higher. How- ever, we did not observe any difference between fail-to-pass tests and other tests (see appendix for figures). Therefore, the model is not simply generating memorized tests.
The primary contribution of this paper is Otter, a system for generating tests from issue descriptions before issue resolution. Otter outperforms the prior state of the art in fail- to- co-pass tests generated while also costing less. This paper also contributes the same problem statement, mined from real- weal-world GitHub issues with strict filters and evaluation metrics. Finally, this paper demonstrates that generated tests can improve patches generated by SWE agents, helping them reach a precision of between 65% and 92%. TDD- BODD-Bncch- Bench-Verified and Otter gen- erated tests are at https://githh..oom/// cotph////.coo////DDDD.... Bench- Verified.
Cee--Bennh--eieeee new benchmark for

---

Otter: Generating Tests from Issues to Validate SWE Patches
Impact Statement
Kang, S., Yoon, J., and Yoo, S. Large language models are few-shot testers: Exploring LLM-based general bug reproduction. In International Conference Software Engineering (ICSE), pp. 2312-2323, July 2023.
This paper presents work whose goal to advance the field of software testing and program repair. Developers spend significant amount of time resolving bugs and testing them. We believe this work will significantly improve the devel- opers' experience in their day-to-day life. We envision two ways of integrating Otter into existing developer workflows. First, Otter can run on a new issue to propose test, which the stakeholders can use to clarify requirements for the de- sired behavior after issue resolution and, following that, a developer can use the test for test-driven development to resolve the issue. Second, Otter can be paired with a patch- generation solution (a "SWE agent") to create PR that includes both patch and test.
Lin, Y., Ma, Y., Cao, R., Li, B., Huang. F., Gu, X., and Li, Y. Lims as continuous learners: Improving the reproduc- tion of defective code in software issues. arXiv preprint arXiv:2411.33941 2024.
Mündler, N., Mueller, M. N., He, J., and Vechev, M. SWT- bench: Testing and validating real-world bug-fixes with code agents. In Conference on Neural Information Pro- cessing Systems (NeurlPS), December 2024.
Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruction tuning with GPT-4, arXiv preprint arXiv:2304.03277, 2023.
References
Aleithan, R., Xue, H., Mohajer, M. M., Nnorom, E., Ud- din, G., and Wang, S. SWE-Bench+: Enhanced coding benchmark for LLMs. arXiv preprint arXiv:2410.06992, October 2024.
Plein, L.., Ouédraogo, W. C., Klein, J., and Bissyandé, T. F. Automatic generation of test cases based on bug reports: a feasibility study with large language models. In Inter- national Conference on Software Engineering: Compan- ion Proceedings (OSE--Commanion, pp. 360-361, May 2024.
Beck, K. Test driven development: By example. Addison- Wesley Professional, 2002.
Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J.-G., and Chen, W. Codet: Code generation with gen- erated tests. In International Conference on Learning Representations (ICLR), May 2023.
Ruan, H., Zhang, Y., and Roychoudhury, A. SpecRover: Code intent extraction via LLMs. arXiv preprint 1rXiv:2488.02232. August 2024.
Schäfer, M., Nadi, S., Eghbali, A., and Tip, An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions Software En- gineering, 50(1):85-105, 2024.
Chen, D., Lin, S., Zeng. M., Zan, D., Wang. J.-G., Cheshkov, A., Sun, J., Yu, H., Dong, G., Aliev, A., al. CodeR: Issue resolving with multi-agent and task graphs. arXiv June 2024.
preprint arXiv:2406.0..3
Wang. X., Gao, P., Meng, X., Peng. C., Hu, R., Lin, Y., and Gao, C. AEGIS: An agent-based framework for general bug reproduction from issue descriptions. arXiv preprint arXiv:2411.18015, November 2024a.
Chowdhury, N., Aung, J., Shern, C. J., Jaffe, O., Sherburn, D., Starace, G., Mays, E., Dias, R., Aljubeh, M., Glaese, M., Jimenez, C. E., Yang. J., Liu, K., and Madry, A. In- troducing SWE-bench Verified, August 2024. URL https: //openai. pnpeenai.oom/ineee/nttddduuiiiii http:///ee/iindciiiineeweeeeeeee
Wang. x., Li, B., Song, Y., Xu, F.F. Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., Tran, H. H., Li, F,, Ma, R., Zheng, M., Qian, B., Shao, Y., Muennighoff, N., Zhang, Y.. Hui, B., Lin, J., Brennan, R., Peng, H., H., and Neubig, G. OpenDevin: An open platform for Al software developers as generalist agents. arXiv preprint arXiv:2407.1674 July 2024b.
Ehrlich, R., Brown, B., Juravsky, J., Clark, R., Ré, C., and Mirhoseini, A. CodeMonkeys: Scaling test-time compute for software engineering. arXiv preprint 1.X1:25001.14723, January 2025.
Jimenez, C. E., Yang. J., Wettig, A.. Yao, S., Pei, K., Press, O., and Narasimhan, K. R. SWE-bench: Can language models resolve real-world GitHub issues? Interna- tional Conference on Learning Representations (ICLR), May 2024.
Xia, C. S., Deng, Y., Dunn, S., and Zhang, L. Agentless: Demystifying LLM-based software engineering agents. arXiv preprint 2rxiiv2407.01482 July 2024.
Just, R., Jalaii, D., and Ernst, M. D. Defects4J: A database of existing faults to enable controlled testing studies for java programs. In International Symposium on Software Testing and Analysis (ISSTA), pp. 437-440, July 2014.
Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O. SWE-Agent: Agent- computer interfaces enable automated software engineer- ing. arXiv preprint orXiv:2405. 15793, May 2024.
10

---

Otter: Generating Tests from Issues to Validate SWE Patches
Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wu, F,, et al. Instruction tuning for large language models: A survey, arXiv preprint arXiv:2308.1079,, 2023.

---

Otter: Generating Tests from Issues to Validate SWE Patches
A. Different attributes of the TDD- HoD-Bench-Veriffee Instances & Sample Test Patch
Table 7 presents different attributes of the TDD instances to give some idea about the nature of the issues we are dealing with in -DD-Bench-Verfiee. We also present a sample test patch in Figure 9.
Table 7. Different attributes of the TDDD-enchh Verified instances.
Average of Lines
Fraction of
Average Word Count in Issue Description
Project
of Instances
# of Files
# Tfst Test Files
Deleted and Added
Dataset (in %)
On Code
On Tests
Astropy
18
4.0
1,990
351
11.9
28.7
304.5
Django
212
47.2
6,863
810
12.0
24.7
145.6
Flask
0.2
275
27
3.0
5.0
35.0
Matplotlib
32
7.1
4,656
102
9.3
20.0
260.5
Pylint
10
2.2
3,833
51
24.7
33.8
347.1
Pytest
16
3.6
639
114
24.6
53.5
250.1
Requests
1.1
155
9
3.6
6.6
85.2
Scikit-lcarn
25
5.6
1,772
242
11.8
17.1
297.6
Seaborn
2
0.5
353
34
13.5
18.5
182.5
Sphinx
41
9.1
1,917
137
17.5
26.1
186.2
Sympy
67
14.9
2,050
617
12.1
11.9
114.2
Xarray
20
4.5
394
67
17.1
24.3
301.0
Overall
449
100.0
24,897
2.561
13.2
23.3
182.0
FFlle counts are based on the main branches of the project (cloned on October 29, 2024).
-4,6 + 88
fron testing Smport assert, wreay, equal, aosert, assert_array, www.sattstttooo equit from cesting import frame,
fron seabom__oore groupby import BecupBy reaborn. iesoemm..tttee state, regression
PolyFit
-68, 3 +51, 11 def test cou_peoupeeeaal1 of):
61
grid a np. ww.1iinpaceepartt I.nint), parti"x"1. I.nxxt,, gridsizel equnt (partt( (xt), grid)
assert part("y").diff()..c (i.f..e)......)........ atr.cirf.o...oonnn.aaaa..l. aaaeemmn....at)ttt.........
missinng datacself, dr1:
Contributing
36
groupay Groupty(["
c1oefoof6:281 na.nan
PolyFsti)tdfL "y*ll, groupoy,
PolyFint)(er(( ")*))_drmpaa(), groupby, *x*, ())
swwect, eepleed, cwx2)
Figure 9, Example test patch with one contributed test. Although the test file name test.regression.c and test name test mi wrissing.de a are available in this diff, the class TestPolyFit enclosing test pest_missing.der data missing. By applying the test patch to the base commit and parsing the file, retrieve TestPolyFic. which is required to test.missing data.
B. Action Counts in Self-Reflectiv Action Planner Phase
Table 8 presents the stats on the different types of actions proposed by the model in the self-reflective action planner phase.
C. Scaling of Otter++
Otter++ uses heterogeneous prompting and execution logs to select the best solution. Therefore, we cannot significantly increase the number of samples. Otter++ scaled well with samples up to 5, giving 0.5%-5.6% improvement (see Table 9).
12

---

Otter: Generating Tests from Issues to Validate SWE Patches
Table
Stats action count.
Model
Action
Average
Max
Min
Read
5.4
GPT-4o
Write
1.2
Modify
0.5
5
0
Read
6.5
20
Mistral-large
Write
Modify
0,5
Table 9. Scaling of Otter++
#t of Candidate I of Fail-to-pass
In %
141
31.4
2
154
34.3
35.0
36.5
166
37.0
D. Heterogenous Prompts Temperature
In Table 10, the Top-1 column shows "fail-to-pass @ 1" based on our actual ranker, whereas the Oracle column shows "fail-to-pass N", i.e. the result we had a perfect ranker. Heterogeneous prompts yield better results than high-temperature samples in both the Top-1 column and the Oracle column. This evidence that the difference indeed comes from the diversity of heterogeneous prompts and is not ranker specific. The difference between the Top-1 and the Oracle column is almost the same in the first three rows (i.e., all settings with 5 samples). That said, there is still room for improvement in the ranker, illustrated by the last row. In the last row, oracle that could reliably choose among all 10 samples would perform amazingly well, motivating further work on better rankers.
E. Impact of hallucination replacer.
The localizer makes two LLM calls during which hallucinations can occur. In the first call, we drop hallucinated file names. In the second call, replace hallucinated file names with existing ones ensure our pipeline functions correctly. Table 11 show that proposed technique helps Otter achieve better performance. For example, if we didn't 't replace the hallucinated file name in the 2nd call of test localization (the last row), we would have lost 5 samples because our pipeline would have exited with an error. Replacing the hallucinated name generated 4 fail-to-pass tests for those samples.
F. Data Contamination
Figure 10 to Figure 13 present cumulative percent of Otter generated test with different criteria, using GPT-4o, with maximum similarity less than the similarity value shown the x-axis, Table 12 shows examples with higher (> 0.5) similarity score.
G. Prompts for Otter
Figure to Figure present all the prompts used for Otter.
13

---

Otter: Generating Tests from Issues to Validate SWE Patches
Table 10. Performance of Our Test Generation Approach in Different Settings.
Setting
Individual Performance
Max
Average
Top-1
Oracle
5 heterogeneous prompts (Otter++)
141, 11,, 107, 110
116.6
166
197
high-temperature samples (temp=1.0)
111, 118, 126, 121, 113
126
117.8
146
173
5 I greedy 4 hiigh-eemerrature samples
141, 111, 118, 126, 121
141
123,4
152
183
10 = 5 heterogeneous + 5 high-temperature
141, 10, 115, 107, 110 111, 118, 126, 121, 113
141
117.2
168
218
Table 11. Impact of hallucination replacer on GPT-4o generated tests.
Halluciantion
Roo passs rate
Focal/Test
LLM Call
#of fail-to-pass
Happen
LLM Call
19
8
42.1
Focal
LLM Call
LLM Call 1
39
22
56.4
Text
LLM Call
80.0
Table 12. Similarity between Generated Tests & Existing Tests with Similarity Score
Guerenl
Voa lives Rago
Seclail Soer
conturel wntoaoneeennnn wnw.enttttttttttn
der Lost wawgaagnggggeeennn presereeeteee
with aaaen0 (oomundaceos) As
KIES searalat icn, ovare didad
parraad of pooeesadd" chopuid-s, atdc
sedout-out)
Assostin in Phone of the acauennce anileppedd exthir is copuledt are tot.oxooption) iont?
And aaseeattttiiiaa cnn on Carouaaee dyy <dett
paza
Selatediooaa
ded tast cedacing poiseing to Screigshay Eseld p43s
TeatNode1 (aodeis Modec3s
ordering [
24ce0t podals www.la..oralpee.... woo.iggnee (Tazent, modeis. CASCADDC
055
txys
cyder d'gareen
(0)*)
cast updata caturn value i
for note in colf Foteos
with welt aeseeeteeuueeeeee updatesuccunt 201e bolkkupaate lasit cnotas, ("apce"))
inotass
aoce.id
viss seif, sssectiien 21
AoCc-AancttCoonncige leat, rote for not in self meltinetes)
("nota"1)
welt. waieerase.. exp.ated..come ientrals.
dud aoproooeooddd grort wtt.etthaaaeeeeec (oellty
Caaee"a at, order=2)
gee pkassvediaase oopeesa.dic di:
rasoceedya coseecedd 2sis snnsetiuue antr.sanatiiuua (cal)
(nel) valusc ("oae"" "codec', "Lar)-
sncf assertiuua lssaziti, ("a', 21) quget)
self ensertii tremiiti. (.1)
(aaaee "oodazt, (1at)-
ded tast anstance attrz pwwples1on wew.nnoooeessss...
wettttppttttssstttiil wtw.m//sttttmmmeeeeeeeee resereseer (preject)
Vlane
klass inteanaattesyyy [ppr
socoet lala,, *inetanongat syorty
sssedt csype dicti
Kays socted coppo depeddies..e..... ()ye
sasnet bays (topssiett*aa potecid bases, Instancel, typs
asuaet isiraratcr coppedies en ("osletion" [2], asseoid bawss, Inataroe), sppeatiet
saseet typecdist inses
thee an (00 *homorhing" ansert Eypraliot ("ale") (0) 16 cntieel www.eeo.tteeeeeeeee Ueleferalle
"prapetyy proporti olaoswetholgeep))
'property',
propert pro.grop"
ASSNEt 11s8 lectusl]
abSect (acouall
pripoopertyi pww.claannttoogrroo.
20 poproppatyys Poo.prop",
(eeddee: peopectles"
tacget. 1type: int',
012
soyper atat.
class peopersy.!,
docateing',
14

---

Otter: Generating Tests from Issues to Validate SWE Patches
Cumuistivs Percnbage of Tast Cooos with Hakmum Smis//t
Soart Merirto
Figure 10. Cumulative percent of Otter generated all tests, using GPT-4o, with maximum similarity less than the similarity value shown the x-axis.
189
Curruativ Parcottage Rust Carcs wtt Horimum Siitaity
New
Semty Tail
32
Smilority
Figure 11. Cumulative percent of Otter generated modified tests, using GPT-4o, with maximum similarity less than the similarity value shown x x-aiss.
15

---

Otter: Generating Tests from Issues to Validate SWE Patches
Cumuisttc Porcnbag of Tust Cooos with Hoxmum Smio//t
Sour Mererto
Figure 12. Cumulative percent of Otter generated fail-to-pass tests, using GPT-4o, with maximum similarity less than the similarity value shown the x-axis.
189
Curruative Parcottage Bust Carcs wtt Horimum Siilaity
Semty Tail
32
66
9.5
Figure 13, Cumulative percent of Otter generated non fail-to-pass to-pass tests, using GPT-4o, with maximum similar- ity less than similarity value shown the x a-axis
16

---

Otter: Generating Tests from Issues to Validate SWE Patches
Suppose you are very experienced developer. An has been created, and you need choose the best possible file to make the changes. You will be given the following pieces of information. Please write down the file name after the "Answer"
Repository name: SREPO_NAAA NAME
Issue Description: $ISSUE DESCRIPTION
List of file(s), that will be updated or added for addressing the pull request:
SLIST FILES
Please write down the most suitable 10 file names from the list above based on the issue description. Write one file name in each line. Do not add any number index. Also do not add any explanation.
Answer:
Figure 14. Prompt (1 of 2) for focal function localizer.
Suppose you a very experienced developer. An been created, and you need choose possible files functions make changes. will be given the following pieces of information. Please write the file and function name after the "Answer" token.
Repository name: $REPO_NAM NAME
2. Issue Description: $ISSUE DESCRIPTION
3. List of file(s) and function(s), that will be updated added for addressing the pull request:
SLIST_ _LIE FUNCTION
Please write down the most suitable program files and relevant function names on issue description. You can choose multiple functions from different files but keep the list as short as possible.
Filename should be written between <illename> </Filename> tags. Function should be written between <Function> & </Function> tags.
Each line should start with file name and the function names written in that file. Please write down the function names from the same file in one line.
Answer:
Figure 15. Prompt (2 of 2) for focal function localizer.
17

---

Otter: Generating Tests from Issues to Validate SWE Patches
Suppose you very experienced developer. An issue has been created, and need to choose the best possible test file to write fail-to-pass test. You will be given three pieces of information. Please write down the file after "Answer" token.
Repository name: $REPO_ NAME
Issue Description: $ISSUE_ DESCRIPTION
3. List of file(s), that will be updated added for pritingfail-be aai--tt-pass test:
$LIST_ BLIST_FILES
Please write the suitable test from above based description. Write file name in line. add any number index. Also do not any explanation.
Answer:
Figure Prompt of 2) test function localizer.
Suppose are a experienced developer. An issue has been created, and you need to choose the best possible test file to write fail-to to-pass test. You will be given three pieces of information. Please write down the file name and function name after the "Answer" token.
Repository name: $REPO_ NAME
NAME
Issue Description: SISSUE DESCRIPTION
3. List of file(s), that will be updated or added for writing fail-to-pass test:
SLIST_ FILE FUNCTION
Please write down the most suitable test files and relevant function names based on the issue description. You choose multiple from different files but keep the
possible.
Filename should be written between <Filename> </Filename> tags. Function should be written between <Function> </Function> tags.
Each line should start with a file name and the function names written in that file. Please write down the function names from the same file in one line.
Answer:
Figure 17. Prompt (2 of 2) for test function localizer.
18

---

Otter: Generating Tests from Issues to Validate SWE Patches
Suppose you are a very experienced developer. An issue has been created, and you need to write a test to ensure issue has been resoived. will be given the following pieces of information.
Repository $REPO_ NAME
Description: DESCRIPTION
Teat fil((s) function(s), that updated added for testing the pull request:
WIST_EET__TT FILE, FUNCTION
4. Program filess) and functions(s), that will be updated or added for addressing the issue:
SLITT FOCAL FILE FUNCTION
Now make action list write fail-to- test. You can read necessary test and focal function to finish the task. You can only perform
Read: You can read a focal function/test given a file, You cen read only the version of the focal funnction/s/teettttt.s
Each line will have one action, file name, and focal function/test.
Action should be written between <Actions & <jActionn tags
<Filename>> Focal function/tesa sh should be written between of FFunction> </Function> tags
tegg
Please do not repeat any specific action the seme file and focal function/test peirs. You should byy identify both focal function and test, based on the issue description. In most cases, choosing one focal function and one test is enough.
Answer:
Figure 18. Prompts for "make initial plan" step.
Secoasos axperianc
andivna
thw mecoved iilswiing interrme
sotions We Welfian
setlens sebesween ware volid, while othene hod lava Muscrioninne rimes We preseee a aerlons belnw.
awaad Actimbbo:
Now: Aeseet conmrnlly ontuoctoo previded intommation
funotico Veramoo with prrcooonedd rntormattooo
hasas Deaceptive Biscue DESCOEPTION
rlage fel chat care
Program kunctionati,
addied addrnas
Acdion Rutas tullsstoffell follow::
read ffanctiongi ai and functicn nome. You can mad cument fanction.
sati
Modily- canmually function
taxt
mydig
maniral charges.
geassthe "Wiis MMudty" from descriction unc ediaiing Banci I you
espa balito Lunction tygng two than wibc lest for Youree also proporee with rntta editing
Fack liwwe howe non sodinn, sana,,, and sorrion nanns
Aciian shouldd zetween vietiaer stActioni
Farerme tor caranns zaaan ronas
After reuding the funciaoni mentored the vouid action iol, moy mnsite somss of furetberze touta ure instevent L. "heas dios the unnoconaars reddectiona, You at lo adendlly boch locul fanciin and wat based onithe taae deacepoos moal cuara shoccaiin lanciion and taxt croughs touare alooes modily oniyram enta dlown actiion lat emzlam anythig
Alter writing now auaow
the hawe' "Suthrhed" "Umaaital "Ursan""
cThrughhen wot.uggges rags
suggen.
Anaver
Figure 19. Prompts for "reflect and improve the plan" step.
19

---

Otter: Generating Tests from Issues to Validate SWE Patches
you reperienced develope:
owetee, and
At nerding related and wald texts, derided teet. New wrthe teats); We the tollowing information.
Reponatory namer SREPO. NAME
2. Issus Descript one BIBLEE DESCRIFTION
3. Rolevond Function:
SF FUNCTION
4. Test import and str actuus::
SFILE L.L.EFUNCCON FUNCTION
b. function
SNAME
ting Bule:
complete beoweon withing the funetton usiing following
S/COMFLETE FUNC- noed any explanction any olsss tago,
keep
CCOMPLET ETE FUNCH
Teemmberr baned drecctiption seooldd vension, which Relwaant Flnnc and include mossing funstiun botty Maka name
repuoat most the
present lext imports antrrcuuurre, Do
5 NNome of the function no modifted), Plcase write down the simplest fall to poss and discord aethorindeewom factss) from the oxiscing
Anower:
Figure 20. Prompt for modifying existing test.
Supparsas
the octting you decidldd vitt westetacciccc you
Sepoeicoy survic SREFO, MAKE
boan Descripa sisuE DESORITIIN
INCh
ttaie onl sinetraee
fanchannaa wittar
Weuse witte sams phep aatec, atheraa reatsaccion
conaler ches nore andrd praarpp rppanar cam www.liaeeaaoooorr.......
whed ie chrorfandie Noreaa
trte zie conglees betovere CCOMPLETE LTS.F JUNCH MOPLET WPPEEERERRRRRR RV rags Mouner with anction and with Withe doun thw nou owep envertaiin eonantie
o/OOe #TS PMlCts
Wester withe Sowo the oongnee nn
Figure 21. Prompt for writing new test.
20


Reviews:
[{"review_id": "bRE6OsR3hR", "reviewer": "Reviewer_G2zQ", "summary": "The paper introduces Otter, an LLM-based solution that generates unit tests directly from issue descriptions before a code patch is written. This approach supports test-driven development scenario and reduces LLMs overfitting of unit tests to the specific focal code. Otter employs a self-reflective action planner to iteratively decide which code to read and how to construct meaningful tests. Additionally, Otter applies rule-based post-processing to fix missing imports and validate test correctness. To evaluate its effectiveness, the paper introduces TDD-Bench-Verified, a benchmark derived from SWE-Bench Verified, which measures test quality based on fail-to-pass rates.\n\nEmpirical results show that Otter++ (an ensemble version of Otter) achieves a 37.0% fail-to-pass rate, outperforming previous methods like SWE-Agent+ (19.2%). Additionally, incorporating Otter-generated tests into automated patch validation improves precision from 60.8% to 91.9%, helping filter out incorrect fixes. The approach remains cost-effective, with test generation costing less than $0.10 per issue when using GPT-4o.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "a) Alternative Localization Strategies: Thank you for your suggestion. To further assess the impact of localization, we re-ran Otter using oracle golden localizations for both focal and test functions. We did not observe any significant performance gain (32.5% vs 31.4% with GPT-4o) in the fail-to-pass rate. This indicates that Otter's localization is already good enough. Perfection is not required in this phase; for instance, it is fine to add a test to a different file than the golden test.\n\nb) Analyze Otter-generated tests: As per your suggestion, we have analyzed the Otter-generated tests using GPT-4o model from different perspectives, as outlined in the following table. We will include additional analysis in the final version of the paper.\n|      Perspective     |    Category   | #of Sample | #of fail-to-pass  | in %|\n|:-:|:-:|:-:|:-:|:-:|\n| Type of Test         | PatchExisting |         122 |                50 |                   41.0 |\n|                      | AdditionOnly  |         327 |                91 |                   27.8 |\n| Test on old Codebase | Pass          |          69 |                 0 |                      0 |\n|                      | AssertionFail |         170 |                97 |                   57.1 |\n|                      | Fail          |         101 |                40 |                   39.6 |\n|                      | Error         |         109 |                 4 |                    3.7 |\n| Focal Localization   | Correct       |         370 |               118 |                   31.9 |\n|                      | Wrong         |          79 |                23 |                   29.1 |\n| Test Localization    | Correct       |         317 |               117 |                   36.9 |\n|                      | Wrong         |         132 |                24 |                   18.2 |\n\nModified/New: We see that the fail-to-pass rate is higher when patching an existing test than when adding a new test, which is expected because the model can have better context from the existing test. Writing a new test is inherently more difficult than modifying an existing test.\n\nTest on old codebase: It is expected that the test will fail on the old codebase. However, we found 69 samples in Otter where the test passed on the old codebase. Also, our analysis shows that tests with assertion failures have a higher success rate (57.1% fail-to-pass rate) compared to other groups.\n\nFocal Localization: We did not see much impact of focal localization on the performance, with 31.9% and 29.1% for correct and incorrect localization, respectively. \n\nTest Localization: Test localization has a significant impact on the performance (36.9% vs 18.2% for correct and incorrect localization). In our ablation study, we also found that test localization is more important than focal localization. \n\nAlternative diff: We did not explore alternative diff formats. Mundler et al. compared two diff formats in their paper and concluded that custom diff outperforms unified diff. Our approach shares some similarities with the custom diff format, as explained in response to reviewer ZeyC (Diff format). In brief, custom diff provides the file name, type of change (new/modified), position of the edit, and the complete class or function in response to a single call. We extract similar information from the LLM using multiple calls, while keeping the total number of calls low, with specialized prompts. This approach resulted in significantly better performance than those using different diff formats.\n\nc) CodeT Ranking: CodeT is not directly applicable to our current setup because we have multiple tests but not the solution. However, apart from filtering SWE-patches, we could use our tests to choose the best SWE-patch, which would be a good application of the Otter-generated tests. Following your suggestion, we have tried CodeT ranking on candidates from the top 3 leaderboard systems on SWE-Bench-Verified and observed 2% improvement. Note that some of these leaderboard solutions have already been through good rankers and used superior models. Improving upon these samples using a ranker may be difficult. We will add a discussion on CodeT ranking in the paper.\n\nd) Scaling of Otter++: Otter++ uses heterogeneous prompting and execution logs to select the best solution. Therefore, we cannot significantly increase the number of samples. Otter++ scaled well with samples up to 5, giving 0.5%-2.9% improvement.\n\n| #of  Candidate | #of Fail-to-pass  | In % |\n|-|-|-|\n|              1 |                    141 | 31.4 |\n|              2 |                    154 | 34.3 |\n|              3 |                    157 | 35.0 |\n|              4 |                    164 | 36.5 |\n|              5 |                    166 | 37.0 |\n\nWe conducted some experiments with temperature (see Reviewer ZeyC -Heterogeneous Prompts vs Temperature) to generate more samples, but we did not find it beneficial.", "claims_and_evidence": "The claims are generally well supported by the evaluation, although the cost-effectiveness claim only takes into account the GPU inference costs but not the infrastructure for executing tests.", "methods_and_evaluation_criteria": "The paper evaluates Otter’s test generation quality using fail-to-pass rates and TDDScore, both measured on the TDD-Bench-Verified dataset. The fail-to-pass rate directly assesses whether generated tests meaningfully capture the reported issue by failing on the buggy code and passing after the fix, with Otter achieving 31.4% (Otter) and 37.0% (Otter++) -- outperforming prior methods like SWE-Agent+ (19.2%). The TDDScore extends this evaluation by incorporating test adequacy, measuring whether the generated tests effectively cover relevant code changes. This metric provides a more granular evaluation of test quality, ensuring that the generated tests are both correct and comprehensive.\n\nFor patch filtering, the paper measures how well Otter-generated tests improve precision and recall in validating patches using the SWE-Bench Verified dataset. The results show that filtering patches with Otter-generated tests increases precision from 60.8% to 91.9%, though recall drops to 33%, indicating a trade-off between accuracy and completeness. These results suggest that Otter-generated tests effectively eliminate incorrect patches while preserving some valid ones.", "theoretical_claims": "The Otter paper relies on empirical validation of its novel test generation approach, there are no theoretical proofs to verify.", "experimental_designs_or_analyses": "The Otter paper uses a structured ablation study to isolate the impact of key components, such as self-reflective action planning, focal/test localization, and import fixing. Removing action planning reduces fail-to-pass rates by 14–20%, while eliminating test localization leads to a steeper drop (up to 36%), confirming that each step significantly improves test quality. The study also shows that test localization contributes more than focal localization, suggesting that retrieving relevant tests is more critical for fail-to-pass success. While these results validate Otter’s design, a deeper breakdown of failure cases (e.g., incorrect test logic vs. mislocalization) would further clarify its contributions. In addition, it would be good to include alternative localization approaches -- for instance, using analysis of import statements, or based line test coverage (dynamic contexts).\n\nThe paper also compares Otter against prior test-generation models on SWT-Bench Lite, showing that Otter++ (29.0%) outperforms SWE-Agent+ (19.2%) and Libro (15.2%), highlighting the benefits of self-reflection and execution-guided filtering over static LLM outputs. By evaluating Otter across TDD-Bench-Verified and SWT-Bench Lite, the study ensures its findings generalize beyond a single dataset. However, statistical significance testing is could also be conducted, making it more clear whether performance gains consistently hold across different types of issues. In addition, a comparison to other patch filtering techniques like those introduced in CodeT paper could be helpful.", "relation_to_broader_scientific_literature": "The Otter paper builds upon and extends prior research in LLM-driven test generation, automated software debugging, and self-correction. The techniques discussed in the paper are instrumental to building TDD-style SWE agents, where tests are generated upfront to guide issue resolution. This aligns closely with recent work in SWE-Agent, which explores GH issue resolution, SWT-Bench, which benchmarks test generation from issues, and CodeT, which to the best of my knowledge is the first to introduce the concept of test generation conditioned on the natural language problem description rather than focal code.", "essential_references_not_discussed": "\"CodeT: Code Generation with Generated Tests\", B. Chen et. al", "other_strengths_and_weaknesses": "Strengths:\nOtter’s focus on test generation from issues aligns with real-world SWE agentic workflows, making it highly applicable for automated debugging and test-driven development in software engineering. \n\nThe introduction of TDD-Bench-Verified also provides a structured benchmark for evaluating fail-to-pass test generation, ensuring practical impact.\n\nWeaknesses\nThe evaluation effectively quantifies success rates, but it does not deeply analyze why some generated tests fail (e.g., mislocalization, incorrect assertions, gaps in execution filtering, or patch applicability errors). A more detailed error categorization would provide stronger insights into where Otter struggles and how it can be further improved.", "other_comments_or_suggestions": "None", "questions_for_authors": "The paper includes an ablation study evaluating the impact of test and focal localization on fail-to-pass rates, but could you provide further analysis on alternative localization strategies (e.g. based on imports or dynamic coverage context driven localization)? \n\nWhile the evaluation measures fail-to-pass success rates, it does not deeply analyze cases where Otter-generated tests fail to apply to patches. Have you experimented with alternative diff formats, can you break down these metrics for addition-only, patches, etc? \nA breakdown of these cases would help assess the robustness of Otter’s generated tests in practical debugging workflows.\n\nThe study demonstrates that Otter-generated tests improve patch filtering, but have you compared Otter’s filtering mechanism to other patch ranking algorithms, such as those used in CodeT?\n\nHow does Otter++ scale when increasing the number of generated tests per issue or expanding the number of candidate patches evaluated?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XSZkJjBc4Z", "reviewer": "Reviewer_BFCj", "summary": "The study describes a system to produce tests from code issues by leveraging LLMs with rule-based analysis and multi-step process with feedback. In addition, the study provides an associated benchmark derived from the SWE-bench Verified benchmark for evaluating tests generated from issues.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "a) Issue with Sympy Coverage: We use the Python coverage package in the TDD-Bench-Verified benchmark, which works well with Pytest. However, the Sympy project does not use Pytest for testing. Additionally, the coverage package may fail due to dependency issues. In Sympy, we were able to generate the coverage report but found that many fail-to-pass tests had zero coverage. We then manually checked several of the fail-to-pass tests and found inconsistencies in the coverage report. There were lines that were executed but not covered by the package. Therefore, we decided to exclude coverage from the TDD-Score metric for Sympy.\n\nb) Handling Inheritance: The Otter pipeline does not explicitly handle inheritance, but the localizer can include functions from the parent class if it deems them relevant to the issue description. In several instances, the localizer selects multiple functions from different files. Also, the localizer does not consider library functions because it only examines user-defined functions from the current repository.\n\nc) Clarification on Ablation: We apologize for the confusion with respect to Table 2, which presents the ablation study of Otter, not Otter++. Figure 2 offers an overview of Otter, which comprises three major components: localizers, a self-reflective action planner, and a test generator. Please see the modified ablation study for the GPT-4o model (the Mistral-large version will be added in the final release) in the table below.\n\nIn the first group, we present the ablation on the action planner. The first row shows the results when the action planner is removed. The following two rows demonstrate the outcomes when action planning is retained, but action refinement and action validation input to the LLM are disabled. Note that without self-reflective action planning, we cannot refine our localizations. Therefore, we must present all the functions collected during focal and test localizations.\n\nIn the second group, we present the ablation on the localizer. The table shows the results of omitting either or both localizations. We did not run self-reflection on partial or no localization because, without both localizations, self-reflective action planning does not add any value.\nThe last two entries, “fixing import” and “imports at generation”, belong to our third component, the test generator. These are the only two rule-based entries.\n\n|  Model |    Component   |                 Approach                 | # of fail-to-pass | tddScore | Change in tddScore% |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n| GPT-4o | NA             | Otter                                    |               141 |     29.4 |                  NA |\n|        | Action Planner | Otter - Action Planning (complete)       |               110 |     23.6 |               -19.7 |\n|        |                | Otter - Plan Refinement (Just 1 attempt) |               130 |     27.5 |                -6.5 |\n|        |                | Otter - Action Validation               |               120 |     25.7 |               -12.6 |\n|        | Localizers     | Otter - Focal Localization              |               115 |     25.2 |               -14.3 |\n|        |                | Otter - Test Localization               |               107 |     24.2 |               -17.7 |\n|        |                | Other - Focal & Test Localization      |               110 |     24.2 |               -17.7 |\n|        | Test Generator | Otter - Fixing Import                    |               128 |     26.7 |                -9.2 |\n|        |                | Otter - Imports at Generation            |               130 |     26.7 |                -9.2 |\n\nTo clarify, Otter++ uses heterogeneous prompting and a rule-based ranker on execution logs. For heterogeneous prompting, we used the same prompts used for the ablation on Otter. Therefore, the execution log is not used by Otter.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I read the descriptions of the experiments.", "relation_to_broader_scientific_literature": "Test generation for issues is generally relevant. The workflow described combining LLMs with rule-based steps is interesting but could be presented better.", "essential_references_not_discussed": "I do not identify and major omissions in the references.", "other_strengths_and_weaknesses": "The study is fairly comprehensive and clear– describing the methodology, associated benchmark, and assessments of both (including cost).\nHowever, I think that one area of potential improvement is clarifying the ablation study to capture the effect of the novel components of the workflow (the rule based and reflective components) clearly.", "other_comments_or_suggestions": "Text and numbers in many of the figures are too small to read (Fig 4-8)\n\nIn the future, please enable the submission status on the template. It makes it easier to review.\n\nThe textual reference to Mundler et al. should be standardized: Mundler et al. 2024\n\nThe use of “cov” in the equations is confusing since usual “cov” refers to covariance.\n\nTable 2 is confusing but one that I am interested in. It looks like this is an ablation test focusing on the “fixing import” and “import at generation”  steps for Otter. It’s hard to match where those are in Figure 1. It would be interesting (and increase the impact of the work) to explicitly split out between the LLM, LLM w/ feedback, and rule-based components. The other entries look like they are ablation tests for Otter++ and not Otter.", "questions_for_authors": "Can you provide further detail into what the issue is with coverage on sympy and what you did for manual validation?\n\nWhen identifying “local files” how do you handle inheritance? E.g. if a class inherits a method from a parent class do you include both files?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b6Ca3jOwDc", "reviewer": "Reviewer_b7T8", "summary": "This paper introduces Otter, a system for generating tests from issue descriptions to validate software engineering (SWE) patches. Unlike prior work that focuses on generating tests for existing code, Otter addresses the scenario where a code patch does not yet exist, supporting test-driven development (TDD) and validation of SWE agents. The system takes an issue description and original code as input, and generates tests that should fail on the old code but pass on the new code after the issue is resolved.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "a) Limited novelty: Our approach introduces self-reflective action planning for generating bug-reproduction tests, which is a novel application in this problem setting. Our approach also incorporates heterogeneous prompting for multi-sampling, which is shown to be better than multi-sampling at higher temperatures. Although applying static analysis for repairing name-related errors, prompt design, and rule-based checking are not new, we have adapted them to work well in our problem setting. Together these features enable our approach to achieve significantly better results than the techniques presented by Mundler et al. \n\nb) Incomplete ablation studies: We have modified and restructured the ablation study to place more focus on self-reflective action planning (See reviewer BFCj “Clarification on Ablation”). We have added the last two rows in the following table (Table 2 in the paper): Row 2 shows the number of fail-to-pass tests obtained if plan refinement is disabled; Row 3 shows the results obtained if we do not provide the model with valid and invalid action lists.\n\n|  Model |    Component   |                 Approach                 | # of fail-to-pass | tddScore | Change in tddScore% |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|   GPT-4o  | Action Planner | Otter - Action Planning (complete)       |               110 |     23.6 |               -19.7 |\n|        |                | Otter - Plan Refinement (Just 1 attempt) |               130 |     27.5 |                -6.5 |\n|        |                | Otter - Action Validation               |               120 |     25.7 |               -12.6 |\n\nIn Figure 4 of the paper, we present the number of turns taken in the “reflect and improve the plan” step. In the following table, we show the total number of actions proposed by the model in the final stage and how the performance changes with it. We find that the number of actions tends to be around 1-3, with no clear correlation between the number of actions and whether the test ultimately satisfies the fail-to-pass criterion.\n\n| #of Action | #of Sample | #of fail-to-pass  | in % |\n|-|-|-|-|\n| 1          | 18         | 7                 | 38.9 |\n| 2          | 157        | 49                | 31.2 |\n| 3          | 258        | 78                | 30.2 |\n| 4          | 15         | 7                 | 46.7 |\n| 5          | 1          | 0                 | 0    |\n\nc) Limited qualitative analysis: We have added a detailed analysis of the Otter-generated test in response to Reviewer G2zQ (Analyze Otter-generated tests). We discuss how performance changes with mislocalization, the test status on the current codebase, and the type of change (new/modified).  \n\nd) Missing comparison with concurrent work: Thank you for pointing out the references to Aegis, EvoCoder, and CodeMonkeys. All three of them are highly relevant to our work, as they involve generating reproduction tests from issues. We will make sure to cite and discuss them.\n \n[1] Aegis: Our related-work section already cites and discusses this paper.\n \n[2] EvoCoder: This paper proposes the interesting idea of using experiences from prior issues to help with the latest issue at hand, which is complementary to Otter. Unlike our work, the generated tests are not integrated with the existing CI pipeline. Furthermore, the experiments do not use execution-based metrics, making it hard to compare empirically.\n \n[3] CodeMonkeys: This paper leads with inference scaling and patch generation. It appeared on arXiv on Jan 24, 2025, and we did not realize its test-generation contributions until after we had submitted our work to ICML on Jan 30. CodeMonkeys iteratively and jointly improves a patch with a test, which is interesting and complementary to Otter. Unlike our work, their experiments do not evaluate the fail-to-pass rate nor coverage of generated tests.  \n\n\ne) Example & Integrating Otter: Thank you for your suggestions. We will add more examples in the main text. We envision two ways of integrating Otter into existing developer workflows. First, Otter can run on a new issue to propose a test, which the stakeholders can use to clarify requirements for the desired behavior after issue resolution and, following that, a developer can use the test for test-driven development to resolve the issue. Second, Otter can be paired with a patch-generation solution (a \"SWE agent\") to create a PR that includes both a patch and a test.", "claims_and_evidence": "The paper's main claims are supported by evidence.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are generally appropriate for the problem:\n- The TDD-Bench-Verified benchmark is a reasonable extension of SWE-bench Verified with additional quality filters.\n- The tddScore metric combines fail-to-pass behavior with code coverage, which is appropriate for evaluating test quality.\n- The comparison with existing approaches (SWE-Agent+, LIBRO, etc.) provides context for Otter's performance.\n\nHowever, there are some limitations:\n\n- The paper would benefit from more qualitative analysis of the generated tests to understand their characteristics beyond just pass/fail rates.\n- The evaluation focuses primarily on Python repositories, which may limit generalizability.", "theoretical_claims": "The paper doesn't make significant theoretical claims requiring formal proofs.", "experimental_designs_or_analyses": "The experimental design is generally sound:\n\n- The comparison with baselines uses the same dataset and evaluation metrics.\n- The ablation studies help understand component contributions.\n- The data contamination analysis addresses potential concerns about model memorization.", "relation_to_broader_scientific_literature": "The paper positions itself well within the literature on test generation and SWE agents. It builds upon:\n- SWE-bench and SWE-bench Verified for issue resolution benchmarking\n- Recent work on LLM-based SWE agents (SWE-Agent, CodeR, SpecRover)\n- Test generation systems like LIBRO", "essential_references_not_discussed": "As mentioned above, the following references are essential to understanding the context but are not cited:\n\n```\n[1] Wang X, Gao P, Meng X, et al. (2024) \"AEGIS: An Agent-based Framework for General Bug Reproduction from Issue Descriptions\" - This work directly addresses bug reproduction from issue descriptions using a multi-agent framework, which is highly relevant to Otter's goals.\n\n[2] Lin Y, Ma Y, Cao R, et al. (2024) \"LLMs as Continuous Learners: Improving the Reproduction of Defective Code in Software Issues\" - This paper explores similar problem space of reproducing issues from descriptions.\n\n[3] Ehrlich R, Brown B, Juravsky J, et al. (2025) \"CodeMonkeys: Scaling Test-Time Compute for Software Engineering\" - This work presents related approaches for test generation that would provide important context.\n```", "other_strengths_and_weaknesses": "Strengths:\n- The paper addresses an important practical problem in software engineering - generating tests from issue descriptions before patches exist.\n- The comprehensive evaluation across multiple models and comparison with existing approaches provides a clear picture of Otter's capabilities.\n- The introduction of TDD-Bench-Verified is a valuable contribution to the research community.\n\nWeaknesses:\n\n- **Limited novelty**: The primary novelty is in the combination of existing techniques rather than fundamentally new approaches. \n- **Incomplete ablation studies**: While Table 2 shows component contributions, more detailed analysis of the novel components (especially the self-reflective planning) would strengthen the paper.\n- **Limited qualitative analysis**: Beyond pass/fail rates, deeper analysis of test characteristics would be valuable.\n- **Missing comparisons with concurrent work**: The paper doesn't compare with or discuss several highly relevant recent papers in the same problem space.", "other_comments_or_suggestions": "- The paper would benefit from more examples of generated tests in the main text.\n- The paper should address how Otter might be integrated into existing developer workflows.", "questions_for_authors": "See about \"Other Strengths And Weaknesses\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1L6XjTU2Dp", "reviewer": "Reviewer_ZeyC", "summary": "This paper introduces a novel code agent specifically designed to generate test cases for real-world repositories, comprising a localizer, action planner and generator. A variant of the agent executes all but one of these components and picks the best results using some heuristics and execution feedback. The resulting agent achieves state of the art performance on the related SWT-bench and high performance on the similar benchmark introduced by the authors for the same purpose, TDD-Bench.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "a) Reporting Coverage: We will add a coverage column to Table 1.\n\n|     Model     |  Approach | Coverage |\n|-|-|-|\n| Mistral | Zero-shot |     60.6 |\n|               | Otter     |     70.5 |\n|               | Otter++   |     70.4 |\n| GPT-4o         | Zero-shot |     60.0 |\n|               | Otter     |     70.6 |\n|               | Otter++   |     71.5 |\n\n\n\nb) Ablation: We have conducted a detailed analysis of Otter tests in response to Reviewer G2zQ's comments (see- Analyze Otter-generated tests) and have modified our ablation study to further break down self-reflective action planning (Reviewer BFCj- Clarification on Ablation).\n\nFile Retrieval: We evaluated the effect of the number of localized files on the results and found it to be minor:\n\nFocal (#of files, accuracy) = {(5, 82.4%), (10, 82.4%), (15, 83.3%)}\n\nTest (#of files, accuracy) = {(5, 69.9%), (10, 70.6%), (15, 71.9%)}\n\nHallucination: The localizer makes two LLM calls during which hallucinations can occur. In the first call, we drop hallucinated file names. In the second call, we replace hallucinated file names with existing ones to ensure our pipeline functions correctly.\n\n| Type |  LLM Call  | #of Hallucinations Happen | #of fail-to-pass  | in %|\n|-|-|-|-|-|\n| Focal      | 1 |                       19 |                 8 |              42.1 |\n|            |  2 |                        5 |                 3 |              60.0 |\n| Test       |  1 |                       39 |                22 |              56.4 |\n|            |  2 |                        5 |                 4 |              80.0 |\n\nThe results show that our proposed technique helps Otter achieve better performance. For example, if we didn’t replace the hallucinated file name in the 2nd call of test localization (the last row), we would have lost 5 samples because our pipeline would have exited with an error. Replacing the hallucinated name generated 4 fail-to-pass tests for those samples.\n\nPlan revision up to 5 times: The model does not always revise the plan 5 times. If the model is satisfied before making 5 calls, it does not make further revisions. Figure 4 shows that the LLM is usually satisfied within 2 calls. Our approach could go beyond 5 calls, but considering the cost and time, we limit the number of plan revisions to 5.\n\nTest File Structure: It contains lines where classes and functions are defined. We build the test file structure by omitting lines that do not define a new class or function. Note that 72.8% tests are new, and test file structure is necessary for those tests. LLMs are unable to indicate the position of new tests without it.\n\nc) Analysis for various fixing steps: We have already added the results in Table 2 for the import fixer. We will lose 11 and 4 instances with GPT-4o and Mistral-large, respectively. We address the hallucination above.\n\nd) Heterogenous Prompts vs Temperature: We make multiple LLM calls in different stages and multi-sampling in each stage will exponentially increase the test counts. To compare heterogenous prompting with temperature, we generated 5 samples at high temperature (1.0) in our last LLM call in Test Generator phase of Otter (our best solution). Though the average number of fail-to-pass test goes up (117.8 vs 116.6), the Top-5 and Top-1 (using Otter++ ranker) results remain lower (Top-5: 173 vs 197 and Top-1: 146 vs 166). That means heterogenous prompting boosts up the overall ensemble performance if we compare at the same number of samples.  \n\ne) Diff format: The custom-diff has similarities to our approach. However, the model needs to perform additional tasks such as writing the file name, change type, and line number in response to one LLM call. Additionally, to instruct the model to generate a specific format, the authors had to include an example that is not relevant to the issue itself. In our generation step, the model only needs to generate the test (and prior function name for positioning new tests).\n\nf) Usage of patch in Mundler et al.'s: Thank you for pointing this out. Figure 12 of the SWT-Bench paper presents the prompt that was used for ZeroShotPlus and LIBRO. Line 7 indicates that the “proposed patch” has been used to generate the test. We mistakenly thought this refers to the golden patch, which we now realize is not the case. We will fix this statement in the paper.  \n\ng) Execution feedback: Otter++ uses execution logs on the current code base and gains an advantage in the final selection stage. We will mention the use of feedback as a contributing factor to the superior performance of Otter++. Note that Otter does not use any execution logs, yet it still performs significantly better than LIBRO (25.4% vs 15.2% in Table 3). \n\nh) Supplementary Material: We will improve the Otter repository as suggested. We removed the license following the ICML author instructions. We will address this in our open-source version. V5 is our internal name used for Otter; we apologize for the confusion it caused.", "claims_and_evidence": "Several claims regarding related work are mentioned further below.\nClaims regarding state-of-the-art performance appear well-founded.", "methods_and_evaluation_criteria": "The authors introduce fail to pass (FTP) scores and a tddScore, which is the multiplication of the FTP score with the % of changed lines covered. While the latter is an interesting score, many readers are more familiar with line coverage and may be interested in having line coverage displayed along FTP and tddScore.", "theoretical_claims": "No theoretical claims made/checked.", "experimental_designs_or_analyses": "While the proposed method proposes interesting components to combat likely issues with test generation, from a scientific perspective I feel there is a lack of rigor in assessing the individual component's impact (listed below in more details).\n\n**No ablation on the localization, planner, test generator component**\nIn Section 4.1 a Localizer is introduced, and it is introduced that only the top 10 retrieved files are passed to the LLM. It would be interesting to see an ablation here on how the choice of less or more files influences the model performance (either qualitatively or quantitatively).\nAlso hallucinated file names are not dropped but replaced with minimal edit distance files. Why was this introduced and can the authors provide examples or statistics on how frequent such hallucinations are (and how useful their replacement is)?\nThe same holds for the planner component - the model revises its plan 5 times. Why this many times?\nFinally the test generator extracts file structure and imports for the model. How exactly is the \"test structure\" extracted? What happens when this is not provided to the LLM?\n\n**No analysis of relevance for various fixing steps**\nThe authors introduce the above mentioned hallucination replacer and an import fixer (Section 4.1 Test Generator). It would be interesting to see their impact, i.e. how many instances would fail if not for these components?\n\n**No analysis of heterogenous prompts vs temperature**\nThe authors mention they use 5 heterogenous prompts at temperature 0 for their models because this proved useful in early experiments. It would be interesting to see concrete examples of how this change improves the method or some quantitative comparison of performances.", "relation_to_broader_scientific_literature": "The approach is similar to the mentioned AEGIS. However, the field of repository-level test case generation is evolving and it is interesting to see different proposals for code agents for test generation (similar to code agents for code repair).\n\n\nIn section 4.1 \"Test generator\", the authors claim that Mundler et. al. attempt to generate diffs directly, however telling from the corresponding paper, Mundler et.al. actually introduce a custom diff format in their zero-shot setups that mainly asks the model to generate complete functions and where to insert them, similar to the method that the authors propose. The agents do not generate diffs either but are equipped with different editing tools. I suggest the authors adapt these claims accordingly or clarify how they consider their method to differentiate.\n\nIn Section 6.4, the authors moreover claim that \"two of [Mundler et al.'s] approaches use a golden patch in the prompt\". Could the authors clarify which methods these are?\n\nFor Otter++, an execution feedback oracle is used, concretely, the best model generation is picked based on execution logs in the final setting. While this is similar to the LIBRO setting in Mundler et. al., it is a clear advantage over their analysed code agents which do not have access to a correctly set-up environment and the test results. The authors may consider mentioning this, since it may be a contributing factor to their superior performance.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "The paper introduces a novel and interesting follow-up for the work of Mundler et. al., demonstrating a variety of components designed for addressing common issues during test generations.", "other_comments_or_suggestions": "Table 8 is barely readable when printed on paper, I suggest to make it slightly larger.", "questions_for_authors": "Please see my open questions in the above fields.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Otter, a system leveraging large language models combined with rule-based analysis and self-reflective planning to generate software tests directly from issue descriptions, aiming to support test-driven development and validate automated code patches. Reviewers broadly acknowledged the practical relevance of this problem setting, the system's strong empirical performance compared to state-of-the-art methods like SWE-Agent+ (G2zQ, ZeyC), and the value of the introduced TDD-Bench-Verified benchmark (G2zQ, BFCj). However, concerns were raised regarding the degree of methodological novelty (b7T8), the completeness of the initial ablation studies (G2zQ, BFCj, b7T8, ZeyC), the lack of deep qualitative analysis of failure modes (G2zQ, b7T8), and missing comparisons or discussions of some related work (G2zQ, b7T8).

The authors provided a detailed rebuttal with significant new experimental results and analyses. They presented expanded ablation studies clarifying the impact of individual components like localization and the novel self-reflective action planner, addressing points raised by multiple reviewers (G2zQ, BFCj, b7T8, ZeyC). They also included a qualitative breakdown of generated test outcomes and failure reasons (addressing G2zQ, b7T8) and committed to incorporating discussions of the related work highlighted by reviewers (addressing G2zQ, b7T8). These additions were well-received, notably leading reviewer ZeyC to increase their score, indicating that the clarifications sufficiently addressed their concerns about component contributions and analysis rigor.

While reviewer b7T8 maintained their assessment regarding limited fundamental novelty, acknowledging the work primarily as effective engineering, the demonstrated state-of-the-art results on a challenging and practically relevant software engineering task carry significant weight. Considering the strong empirical contribution, the introduction of a useful benchmark, the system's relevance to the growing field of AI for software engineering, and the positive reception of the rebuttal by several reviewers, the paper represents a solid contribution that meets the acceptance criteria for ICML.
Decision: Accept (poster)