REDO: Execution-Free Runtime Error Detection for Coding Agents

## Abstract

Abstract As LLM-based agents exhibit exceptional capabilities in addressing complex problems, there is a growing focus on developing coding agents to tackle increasingly sophisticated tasks. Despite their promising performance, these coding agents often produce programs or modifications that contain runtime errors, which can cause code failures and are difficult for static analysis tools to detect. Enhancing the ability of coding agents to statically identify such errors could significantly improve their overall performance. In this work, we introduce Execution-free Runtime Error Detection for COding Agents ( REDO ), a method that integrates LLMs with static analysis tools to detect runtime errors for coding agents, without code execution. Additionally, we propose a benchmark task, SWE-Bench-Error-Detection ( SWEDE ), based on SWE-Bench (lite), to evaluate error detection in repository-level problems with complex external dependencies. Finally, through both quantitative and qualitative analyses across various error detection tasks, we demonstrate that REDO outperforms current state-of-the-art methods by achieving a 11.0% higher accuracy and 9.1% higher weighted F1 score; and provide insights into the advantages of incorporating LLMs for error detection.

## 1 Introduction

Figure: Figure 1: REDO outperforms current SOTA methods (Bieber et al., ) with respect to accuracy and weighted F1 (W.F1) on different tasks.
Refer to caption: x1.png

Large language models (LLMs) and LLM-based agents have exhibited significant potential in code generation, code editing, and code evaluation.
This progress has culminated in the development of advanced LLM-based agents (hereafter referred to as coding agents) designed to address increasingly complex tasks. For example, SWE-Bench (Jimenez et al., ) presents a demanding benchmark comprising repository-level coding challenges. This benchmark requires coding agents to generate a modification patch that solves a given problem within a GitHub repository, based on a problem statement expressed in natural language. To effectively navigate complex tasks such as those posed by SWE-Bench, coding agents must demonstrate proficiency in the following core competencies: 1) comprehension of the problem statement and retrieving relevant code, 2) reasoning towards a functionally correct solution, and 3) generation of programs free from runtime errors such as SyntaxError, AttributeError, or TypeError.

While the majority of coding agents across different tasks focus on enhancing comprehension, retrieval and reasoning capabilities, the systematic detection of runtime errors has received comparatively limited attention. However, ensuring that generated code is free from runtime errors is as critical as the aforementioned capabilities. For example, an AttributeError can cause the modified code to fail, irrespective of the agent’s comprehension and reasoning processes. Indeed, coding agents are not immune to runtime errors. On SWE-Bench-lite (Jimenez et al., ), the top six coding agents as of August 2024
(CodeStory, Mentatbot, Marscode, Lingma, Droid, AutoCodeRover) produce, on average, 1.8 SyntaxErrors, 25.8 TypeErrors, 5.2 NameErrors, and 11.3 AttributeErrors. Moreover, SWE-Bench (lite) is deliberately curated to include only straightforward problems, suggesting that the incidence of runtime errors could be substantially higher on the full SWE-Bench dataset. Additionally, detecting these runtime errors enables coding agents to rapidly iterate solutions, thereby reducing both time and cost.

Recent coding agents do often incorporate modules for detecting runtime errors. For example, SWE-agent (Yang et al., ) and AutoCoderRover (Zhang et al., ) employ static analysis tools such as Pylint (Foundation, ), while CodeR (Chen et al., ) utilizes dynamic analysis by generating unit tests and executing the modified code. However, these methods have three critical limitations. First, because runtime errors are often triggered by specific inputs that are typically unpredictable from the code alone, static analysis approaches struggle to detect errors such as many TypeErrors. Second, although dynamic analysis techniques may identify these runtime errors, they require execution of the underlying code, which is problematic for three reasons. First, execution ceases upon encountering the first runtime error, allowing only one error to be detected at a time, thereby increasing the cost of detection. Secondly, dynamically configuring execution environments for diverse project setups and testing frameworks poses significant challenges and might trigger technical, legal or privacy concerns (Puddu et al., ; Lacoste & Lefebvre, ). Furthermore, although these mechanisms are commonly employed as separate modules, their performance has never been comprehensively evaluated. This lack of modular evaluation makes it difficult to assess the extent to which these error detection modules enhance the performance of coding agents.

In this work, we introduce an execution-free runtime error detection method, termed REDO, which integrates static analysis tools, such as Pyflakes (Foundation, ) or PyRight (pyright, ), with a large language model (LLM). This approach extends the capabilities of static analysis tools to detect a broader range of errors without the need for code execution. The combination of these tools is specifically designed to maximize the advantages of both tools, achieving a balanced trade-off between reliability and the breadth of detectable errors. Similar to other works (Jimenez et al., ; Bieber et al., ), we specifically focus on the runtime error detection in Python repositories; but our method could be straightforwardly extended to other languages.
Moreover, we present a challenging and practical benchmark, SWE-Bench-Error-Detection (SWEDE), which is the first repository-level error detection in the presence of complex external dependencies. Beyond SWEDE, we perform a suite of experiments encompassing various tasks to further evaluate error detection algorithms. As shown in Figure 1, REDO significantly outperforms previous methods, obtaining SOTA performance across diverse scenarios; and through qualitative analysis, we provide insights into the underlying mechanisms of REDO.

## 2 Related Work

### 2.1 LLM-Based Code Generation, Coding Agents, and SWE-Bench

LLMs (Ouyang et al., ; et al., ; ; ; Anthropic, ) have been increasingly leveraged for automatic code generation (Nijkamp et al., ; et al., ; Chai et al., ; et al., ; Nijkamp et al., ; et al., ; Gunasekar et al., ) and code repair (Xia et al., ; Shypula et al., ; Gunasekar et al., ; Prenner & Robbes, ; Huang et al., ). The advent of LLM-based agents has further expanded the scope of problem-solving in complex coding tasks, leading to the development of specialized coding agents. For example, SWE-Agent (Yang et al., ) is built on the ReAct framework (Yao et al., ) and incorporates a custom Agent-Computer Interface (ACI), enhancing the agent’s ability to interact with the environment effectively. Similarly, AutoCodeRover (Zhang et al., ) provides an integrated set of search tools and includes Pylint within its toolkit, which serves to statically detect errors. Another significant contribution is CodeR (Chen et al., ), a multi-agent framework that facilitates code editing by coordinating multiple agents through structured graphs. Within this framework, specific agents like the Reproducer and Verifier are designed to generate unit tests and validate modified implementations, respectively. These coding agents are frequently evaluated using the SWE-Bench benchmark Jimenez et al. (), which is composed of coding issues extracted from public GitHub repositories. This benchmark poses significant challenges by requiring coding agents to comprehend issues, localize relevant code segments, and produce correct modifications.

### 2.2 Static Analysis Tools and Runtime Error Prediction

Static analysis, a technique for examining computer programs without execution, is particularly valuable in contexts where executing the program might lead to legal, privacy, or computational concerns. Due to its non-executive nature, static analysis has found widespread application in error detection (Zheng et al., ; Dillig et al., ; Chow et al., ), bug identification (Ayewah et al., ; Mashhadi et al., ), and vulnerability discovery Charoenwet et al. (); Sonnekalb et al. (); Esposito et al. (); Chess & McGraw (); Livshits & Lam (); Evans & Larochelle (). In the context of Python programming, various professional static analysis tools have been developed to enhance code quality. For instance, Pylint (Foundation, ) and Pyflakes (Foundation, ) are designed to identify errors, while Bandit PyCQA () focuses on detecting common security vulnerabilities. Additionally, tools such as PyRight (pyright, ) and MyPy (Foundation, ) perform type checking, contributing to more robust software development. Although the integration of LLMs with static analysis is still in its infancy, some recent studies have proposed combining these technologies to enhance bug detection in complex systems, such as the Linux kernel (Li et al., ), and to identify security vulnerabilities (Li et al., ). However, the exploration of LLMs in conjunction with static analysis for runtime error detection remains limited. Additionally,  Bieber et al. () presents a notable effort in this domain by leveraging Graph Neural Networks (GNNs) for predicting runtime errors, along with proposing a dataset, referred to as STA, to evaluate their approach’s efficacy.

Figure: Figure 2: REDO employes a two-step process. When a repository-level modification is given, REDO first applies differential analysis to compare the original and modified implementations. If runtime errors are identified, REDO triggers an alert and rejects the patch. Conversely, if no errors are detected, the patch is forwarded to the LLM-based detection. The final determination regarding the patch’s safety is made based on the detection provided by the LLM.
Refer to caption: x2.png

## 3 Execution-free Runtime Error Detection for Coding Agents

In this study, we introduce REDO, which serves to check the safety of modification patches. Here, “Unsafe” instances are those that might crash due to runtime errors; and “Safe” instances are those that can be successfully run. REDO operates through a two-phase process: differential analysis and LLM-based detection.

The differential analysis component employs a static analysis tool, which provides a dependable method for detecting runtime errors. However, its detection capabilities are generally constrained to SyntaxError, AttributeError, and NameError. To address this limitation, the LLM-based detection is incorporated to reason about the input contexts. It extends REDO’s detection capabilities to errors such as TypeError and ValueError, which are typically beyond the scope of static analysis.

By integrating these mechanisms, REDO achieves a balanced trade-off between reliability and the breadth of error detection. An overview of REDO’s architecture is illustrated in Figure 2.

### 3.1 Differential analysis

Static analysis tools like Pyflakes and PyRight typically ensure detection through reliable methods, such as verifying syntax correctness and maintaining data type consistency, making them essential for identifying runtime errors in coding agents. In this study, we employ PyRight as our static analysis tool since it is fast and lightweight; however, our framework is designed to be flexible, allowing the integration of any static analysis tool.

Despite their utility, static analysis tools are affected by two challenges. First, they are prone to generating false positives, where potential vulnerabilities are incorrectly flagged (Kang et al., ; Kharkar et al., ; Murali et al., ). For example, when PyRight is applied to original python scripts containing the modified functions,
which do not contain runtime errors, it falsely classifies an average of 267 instances, considering 89% of all testing instances as “Unsafe” across various coding agents. To mitigate this issue, particularly in code edit tasks, we introduce the concept of differential analysis. This method involves applying static analysis tools to both the original and modified implementations separately. By comparing the errors detected in the original implementation ($S_{\text{Orig}}$) with those in the modified implementation ($S_{\text{Mod}}$), we can identify any new errors introduced by the modifications. If new errors are detected in the modified implementation, the patch is flagged as “Unsafe”. Differential analysis effectively refines static analysis tools to focus specifically on runtime errors introduced by modifications, thereby filtering out false positives from the original implementation. Notably, this removes almost all positives induced by the original implementation.

The second challenge with static analysis tools is their inability to detect errors that are triggered under specific inputs. In dynamically typed languages like Python, variable data types are unknown before execution and also can change in response to different inputs. Since static analysis tools cannot reason about input contexts, they often fail to detect these errors, resulting in low recall in error detection performance. We show a concrete example in Figure 4. To address this challenge, we propose the LLM-based detection described in the next section.

### 3.2 LLM-based Detection

In comparison to static analysis tools, LLMs possess the ability to comprehend both the problem statement and the modification patch. This capability allows them to reason about potential input contexts and anticipate runtime errors that might be overlooked by static analysis tools. However, the reasoning process of LLMs is not always as reliable as the error detection mechanisms inherent in static analysis tools. To harness the complementary strengths of both approaches, we restrict the application of LLMs to instances deemed ’Safe’ by static analysis tools. Accordingly, we have designed the LLM prompt template to identify potential runtime errors that static analysis tools may have missed.

Specifically, for each modification patch, we provide two additional pieces of information: the problem statement and the Python script containing the original version of the modified functions. The problem statement outlines the input contexts and describes the potential verification process for the modified implementation. The original implementation provides the running context of the modified functions, including the safe utilization of variables and functions. Subsequently, we prompt the LLM to enumerate potential runtime errors that may arise due to the modification and ultimately assess the safety of the patch. A detailed prompt template is provided in Appendix E.2.

Given the cost and latency associated with LLM calls, the LLM API is restricted to a single invocation per instance in our study. For the LLM, we utilize Claude-SONNET-3-5, with the temperature setting fixed at zero. A pseudo-code of whole REDO framework is given in Algorithm 1.

## 4 SWE-Bench-Error-Detection (SWEDE)

The difficulty of runtime error detection can vary drastically among different coding problems. For instance, on the task proposed by Bieber et al. (), only one python script is considered on each data point. This task is practical as it resonates the competitive programming scenario where one python script should contain all functionality; and external dependencies are simple. However, as coding agents become more powerful, additional challenging and practical scenarios should be considered. In this work, we propose an repository-level error detection task, which is based on SWE-Bench (lite) (Jimenez et al., ) and its evaluation results using different coding agents. We name this task as SWE-Bench-Error-Detection or SWEDE.

SWE-Bench (lite) is a popular benchmark dataset containing instances of repository-level coding problems. Taking a GitHub repository and a problem statement as inputs, SWE-Bench (lite) asks coding agents to generate a modification patch to resolve the problem. SWEDE extends SWE-Bench (lite) to include generated patches and evaluation logs from SWE-Bench leaderboard (Jimenez et al., ); but with a focus on detecting runtime errors induced by generated patches, without executing the code.

The task is challenging for two reasons. First, the modified scripts usually call other python scripts, referred to as external dependencies, within the same repository. For instance, as shown in Table 1, when only one directory level above the location where the modified file resides is considered, there already are many dependencies on average. When all files in the repository are considered, the number of dependencies could become even more intimidating. Furthermore, the unit tests might not directly interact with the modified files. These two factors make SWEDE challenging for error detection algorithms as running contexts of variables and functions are harder to infer. The task is practical because, when coding agents autonomously modify repositories, detecting runtime errors early offers instrumental information; and can potentially reduce cost and time.

**Table 1: Average External Dependencies in Parent Folder on SWEDE Across Coding Agents.**
| Method | CodeStory | Demo | Aider | Lingma | Droid | ACR |
| --- | --- | --- | --- | --- | --- | --- |
| Average dependencies | 5.56 | 4.56 | 6.07 | 5.75 | 5.61 | 5.98 |

As the problem is challenging, in this work, we focus on detecting if a modification patch will induce any runtime errors, e.g., SyntaxError, AttributeError, TypeError, etc. As a result, given a patch, we label it as positive if it contains an runtime error; and negative if the patch passes all unit tests or fails the unit tests only because of wrong functionality. We propose to measure the performance of error detection algorithms on SWEDE using precision, recall and F-1 score.

## 5 Experiments

### 5.1 Quantitative results

#### SWE-Bench-Error-Detection (SWEDE).

We first evaluate REDO on SWEDE task. We consider six State-of-the-art (SOTA) coding agents on the SWE-Bench-lite leaderboard, including CodeStory, Mentatbot, Marscode, Lingma, Droid, and AutoCodeRover (ACR) Zhang et al. (). We compare REDO to several baselines. First, we include two widely used static analysis tools, namely Pyflakes and PyRight. To eliminate false positive detection, we apply differential analysis (introduced in Section 3.1. Second, we include an LLM-only method, denoted as LLM. The LLM is prompted with the same template introduced in Section 3.2. Lastly, to study how the choice of static analysis tool affects REDO, we include a REDO framework with Pyflakes, named as REDO-Pyflakes. Due to the inherent stochasticity in generation, even with a temperature setting of zero, we generate the LLM responses three times and report the mean and standard deviation of the outcomes.

As shown in Table 2, static analysis tools (Pyflakes and PyRight) usually have higher precision scores, while LLM obtains better recall scores. This justifies our analysis on the difference between static analysis tools and LLM (as in Section 3). Furthermore, confusion matrices in Figure 3 show that the static analysis tool is more prudent in claiming unsafe patches, and therefore misses more failed patches than LLM does. By appropriately combining these two kinds of tools, REDO makes a good trade-off between reliability and detectable score. This better trade-off contributes to superior F1 scores. Second, the performance of LLM is closer to REDO because REDO utilizes LLM on more than 70% instances. However, we note that despite the similarity, REDO makes less API calls because of the differential analysis step. We show confusion matrices using REDO and different coding agents in Figure 6. Also, as can be seen on the other dataset (STA) below, when static analysis tools performs better, REDO can significantly outperform the LLM-only baseline. Third, the performance of REDO-Pyflakes and REDO are close and outperform others, demonstrating the robustness of REDO against static analysis tool choices. We further report results using Anthropic Claude OPUS in Table 4 and SONNET-3.5 with temperature being 0.5 in Table 5. We can observe that REDO and REDO-Pyflakes consistently outperform static analysis tools and LLM, demonstrating the robustness of our proposed method against LLM configurations.

**Table 2: Performance metrics by method and benchmark**
| Coding agent | Metric | Method |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Pyflakes | PyRight | LLM | REDO-Pyflakes | REDO |  |  |
| CodeStory | Precision | 32.1 | 43.7 | 34.0 0.2 | 33.3 0.2 | 33.9 0.2 |
| Recall | 22.8 | 48.1 | 69.2 0.7 | 70.5 0.7 | 75.5 0.7 |  |
| F1 | 26.7 | 45.8 | 45.6 0.4 | 45.3 0.3 | 46.8 0.3 |  |
| Demo | Precision | 34.1 | 33.9 | 30.7 0.6 | 31.2 0.6 | 31.1 0.3 |
| Recall | 34.9 | 45.3 | 61.2 1.8 | 70.2 1.3 | 74.4 1.2 |  |
| F1 | 34.5 | 38.8 | 40.9 1.0 | 43.2 0.8 | 43.8 0.5 |  |
| Marscode | Precision | 22.9 | 33.8 | 28.7 1.0 | 27.9 0.7 | 29.1 0.8 |
| Recall | 16.4 | 37.3 | 68.7 1.5 | 74.6 1.5 | 77.6 1.5 |  |
| F1 | 19.1 | 35.5 | 40.5 1.2 | 40.6 1.0 | 42.3 1.0 |  |
| Lingma | Precision | 45.8 | 48.3 | 40.5 0.1 | 40.3 0.2 | 41.3 0.2 |
| Recall | 27.8 | 29.9 | 66.3 0.6 | 71.5 0.6 | 71.5 0.6 |  |
| F1 | 34.6 | 36.9 | 50.3 0.2 | 51.5 0.3 | 52.3 0.3 |  |
| Droid | Precision | 53.3 | 42.1 | 41.8 0.3 | 40.1 0.3 | 42.0 0.3 |
| Recall | 22.0 | 14.7 | 63.6 0.5 | 65.4 0.5 | 68.2 0.5 |  |
| F1 | 31.2 | 21.8 | 50.4 0.4 | 49.7 0.4 | 52.0 0.4 |  |
| ACR | Precision | 45.3 | 54.7 | 39.2 0.2 | 38.3 0.2 | 40.6 0.2 |
| Recall | 23.5 | 34.3 | 65.7 1.0 | 69.3 0.6 | 73.9 0.6 |  |
| F1 | 31.0 | 42.2 | 49.1 0.3 | 49.3 0.0 | 52.4 0.3 |  |

Figure: (a) PyRight
Refer to caption: x3.png

#### Ensemble patch

To demonstrate how the detection errors serve to enhance coding agents, we propose a preliminary algorithm to ensemble a base coding agent with an auxiliary agent, basing on the detection results from REDO. Specifically, given an instance in SWEDE, REDO first checks the safety of the patch $p_{\text{base}}$ from the base agent. If $p_{\text{base}}$ is safe, it will be accepted; otherwise, REDO will check the safety of the patch $p_{\text{aux}}$ from the auxiliary agent. If $p_{\text{aux}}$ is safe, it will be alternatively accepted; otherwise, the algorithm falls back to the base agent and accept $p_{\text{base}}$. Note that, since the top agents on the SWE-Bench leaderboard are proprietary, we are unable to run their methods to generate new patches. Instead, we utilize the patches reported in the SWE-Bench repository. A pseudo-code of the ensemble algorithm is given in Algorithm 2. The quantitative results and analysis can be found in Section D.

#### STA dataset.

To assess the general applicability of REDO, we conduct evaluations on another dataset: the balanced test set introduced by Bieber et al. (), hereafter referred to as STA. This dataset is derived from a code generation dataset CodeNet (et al., ) and comprises approximately 27,000 Python submissions for competitive programming tasks. The balanced test set was designed to ensure that the number of submissions without errors is approximately equal to those containing runtime errors. The dataset categorizes 26 distinct error types, including the category ”no error.” The detailed enumeration of errors and their corresponding indices are consistent with those listed in the prompt template provided in Appendix E.3. Unlike the SWEDE dataset, each submission in STA consists of a single Python script that processes input via standard input (stdin) and performs its functionality without relying on external dependencies. In the context of STA, error detection algorithms are tasked with predicting runtime error types based on the Python submission, with or without the presence of input contexts. These input contexts describe the potential values that stdin may take, providing clues regarding possible runtime errors. We refer to the scenario involving input context as With Context, and the scenario without input context as Without Context.

Given that the task in STA differs from that in SWEDE, we adapt REDO using modified prompt templates. The templates for both the scenarios with and without input contexts are provided in Appendix E.3 and Appendix E.4, respectively. The performance of the algorithms is assessed using three metrics: accuracy (Acc), weighted F1 score (W.F1), and weighted error F1 score (E.F1). Weighted metrics are computed by calculating the F1 score for each class independently and averaged using a weight that depends on the number of true instances for each class; the E.F1 is calculated exclusively for data points containing a runtime error. We compare our methods with those reported by Bieber et al. (). To further investigate the contributions of individual components within REDO, we also evaluate PyRight and LLM separately. Since the original dataset samples do not include error-free submissions, we employed three different random seeds to sample an equivalent number of submissions for a fair comparison. The means are reported in Table 3, and the standard deviations are presented in Table 6.

First, as demonstrated in Table 3, REDO-PyRight achieves state-of-the-art (SOTA) performance across all six evaluated metrics, underscoring the superior performance of REDO in the context of STA. Additionally, we observed discrepancies between the annotated error types and the results obtained from our running results. Consequently, we also present evaluation results, enclosed in brackets, based on error types identified in our runs. Under these conditions, REDO exhibits even more pronounced improvements over the baselines. Second, our ablation PyRight obtains decent performance in both Without and With Context tasks. Considering that the input to STA submissions are usually most common cases, as opposed to corner cases in SWEDE, our result demonstrates that static analysis tools could perform well if the inputs and dependencies are simple. Furthermore, when comparing REDO to PyRight, we can also see that REDO obtains extra benefits by including the LLM based tool. This shows the effectiveness of our proposed framework.

Moreover, the differing conclusions regarding static analysis tools and LLMs between SWEDE and STA highlight that each excels in distinct tasks. By integrating these two tools, we can leverage the strengths of both, rendering our framework more versatile and effective across a broader range of tasks.

**Table 3: Quantitative results on detecting runtime errors in CodeNet dataset. Each method is evaluated with and without the input context. Three metrics are included, namely accuracy (Acc), weighted F1 score (W. F1), and weighted error F1 score (E. F1). Bold values indicate the best performance for each metric, while underlined values represent the second-best performance.**
|  | Without Context | With Context |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | Acc | W. F1 | E. F1 | Acc | W. F1 | E. F1 |
| GGNN | 62.8 | 58.9 | 45.8 | 68.3 | 66.5 | 56.8 |
| TRANSFORMER | 63.6 | 60.4 | 48.1 | 67.3 | 65.1 | 54.7 |
| LSTM | 66.1 | 61.4 | 48.4 | 68.1 | 66.8 | 58.3 |
| IPA-GNN | 68.3 | 64.8 | 53.8 | 71.4 | 70.1 | 62.2 |
| E. IPS-GNN | 68.7 | 64.9 | 53.3 | 71.6 | 70.9 | 63.5 |
| PyRight | 74.5 (76.4) | 67.2 (70.1) | 52.2 (54.2) | 74.5 (76.4) | 67.2 (70.1) | 52.2 (54.2) |
| LLM | 61.8 (64.1) | 54.0 (57.2) | 32.7 (35.8) | 64.3 (66.8) | 58.6 (62.1) | 40.4 (43.7) |
| REDO | 75.1 (76.8) | 69.9 (72.6) | 57.0 (59.1) | 77.4 (79.3) | 74.5 (77.5) | 64.3 (66.6) |

### 5.2 Qualitative analysis

In this section, we qualitatively analyze how key design factors in REDO affects the performance. Consequently, we target answering the following two questions:

- 1.
When does LLM succeed and fail to help REDO?
- 2.
How could the detected errors help?

#### When Does the LLM Succeed?

Figure 4 illustrates a case in which the LLM successfully identifies a runtime error that PyRight overlooked. This example is on the django-13551 instance using coding agent Lingma. The complete LLM response and modification patch are presented in Figure 8. Specifically, the evaluation log in Figure 4 reveals that an AttributeError is triggered when the invalid attribute email is accessed on the variable user, which is an instance of a CustomEmailField object. PyRight failed to detect this runtime error due to the inability to infer the data type of user through static analysis. In contrast, the LLM, capable of reasoning about potential runtime contexts, successfully identifies the runtime error, thus marking the patch as Unsafe. This example demonstrates the advantage of leveraging LLMs to anticipate runtime errors. Another successful instance is documented in Appendix F.1.

Figure: Figure 4: Successful example
Refer to caption: x5.png

#### When Does the LLM Fail?

Figure 5 presents an instance where both PyRight and the LLM failed to detect a runtime error. This example pertains to the instance django-11797 using coding agent CodeStory. The complete modification patch and the LLM’s response are detailed in Figure 11. The runtime error occurs when an invalid column name is referenced in a query. Given the current input context, it is nearly impossible to infer the content of the query, leading to the LLM’s failure to predict this runtime error. This example highlights the limitations of REDO in inferring potential inputs with limited contextual information, a challenge we plan to address in future work. Another instance of failure is documented in Appendix F.2.

Figure: Figure 5: Failed example
Refer to caption: x6.png

#### How could the detected errors help?

This section presents a qualitative example illustrating how detected errors can facilitate the correction of flawed patches. We introduce a preliminary patch-fixing algorithm that leverages error messages generated by REDO. Specifically, when presented with a modified patch (referred to as the original generated patch) and its associated detected errors, our approach first applies the patch and extracts the modified code chunks, with ten lines of code added before and after each chunk. We then prompt the LLM to refine the code chunks according to the identified errors. Finally, a fixed patch is generated based on the modifications to the code chunks. The detailed LLM prompt is provided in Appendix E.6. For instance, in the case of Django-12308 utilizing ACR, REDO initially identifies a risky attribute call that could potentially lead to an AttributeError. Based on this detection, the patch-fixing algorithm generates an fixed patch that avoids invoking the risky attribute, thereby preventing the AttributeError present in the original patch. A comprehensive description of the algorithm and a detailed analysis of this example are provided in Section G.

## 6 Conclusion

In this study, we first present REDO, an innovative error detection framework that operates through a two-step process: differential analysis followed by LLM-based detection. This approach achieves a balanced trade-off between reliability and the scope of detectable errors. We also propose SWE-Bench-Error-Detection (SWEDE), a novel and challenging runtime error detection task that aligns with the increasing deployment of autonomous coding agents responsible for repository-level modifications. Furthermore, we conduct a comprehensive set of quantitative experiments to empirically demonstrate the efficacy of REDO across various tasks. In addition, our qualitative analysis offers insights into the conditions under which LLM integration proves beneficial or falls short, and how detected runtime errors using REDO could help fix the previous flawed patches.

## 7 Limitation and future work

The current implementation of the LLM-based detection step constrains the number of LLM API call on each data point to just one. Expanding the number of API calls, potentially by leveraging agentic AI techniques, could significantly improve error detection capabilities. Additionally, the SWEDE task is presently confined to a binary classification of ’Safe’ and ’Unsafe.’ As error detection algorithms become more sophisticated, it is crucial to consider expanding the classification schema to encompass a wider spectrum of error types, similar to those addressed in STA. Finally, the advancement of more refined ensemble and patch-fixing algorithms holds the potential to enhance the efficacy of REDO in supporting coding agents.

## Ethics statement.

Our work leverages results from previous methods, including publicly available sources and the SWE-Bench dataset and leaderboard, as cited in the Experiment section. To the best of our knowledge, this study does not pose any risks related to harmful insights, discrimination, bias, fairness, privacy, or security concerns.

## Reproducibility statement.

We provide details of our experiments and implementation in Sections 3 and 5, including the models used and a description of the data processing steps. Given the limited time, we are unable to wrap up all the code files before the submission deadline. However, we will gladly provide them during the rebuttal stage if required. Additionally, for each experiment, we used three random seeds and report both the means and standard deviations.

## Appendix A REDO pseudocode

Figure: Algorithm 1 Execution-Free Runtime Error Detection for Coding Agents

## Appendix B Confusion matrices

Figure: (a)
Refer to caption: x7.png

## Appendix C Additional quantitative results

**Table 4: Performance metrics by method and benchmark using Claude-3 OPUS**
| Coding agent | Metric | Method |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Pyflakes | PyRight | LLM | REDO-Pyflakes | REDO |  |  |
| CodeStory | Precision | 32.1 | 43.7 | 43.1 | 35.6 | 39.2 |
| Recall | 22.8 | 48.1 | 39.2 | 46.8 | 62.0 |  |
| F1 | 26.7 | 45.8 | 41.1 | 40.4 | 48.0 |  |
| Demo | Precision | 34.1 | 33.9 | 34.0 | 33.9 | 33.6 |
| Recall | 34.9 | 45.3 | 20.9 | 47.7 | 55.8 |  |
| F1 | 34.5 | 38.8 | 25.9 | 39.6 | 41.9 |  |
| Marscode | Precision | 22.9 | 33.8 | 30.8 | 28.7 | 32.8 |
| Recall | 16.4 | 37.3 | 35.8 | 46.3 | 59.7 |  |
| F1 | 19.1 | 35.5 | 33.1 | 35.4 | 42.3 |  |
| Lingma | Precision | 45.8 | 48.3 | 48.6 | 45.9 | 49.1 |
| Recall | 27.8 | 29.9 | 35.1 | 51.5 | 54.6 |  |
| F1 | 34.6 | 36.9 | 40.7 | 48.5 | 51.7 |  |
| Droid | Precision | 53.3 | 42.1 | 56.0 | 50.6 | 48.2 |
| Recall | 22.0 | 14.7 | 25.7 | 40.4 | 36.7 |  |
| F1 | 31.2 | 21.8 | 35.2 | 44.9 | 41.7 |  |
| ACR | Precision | 45.3 | 54.7 | 43.1 | 41.2 | 45.9 |
| Recall | 23.5 | 34.3 | 27.5 | 41.2 | 49.0 |  |
| F1 | 31.0 | 42.2 | 33.5 | 41.2 | 47.4 |  |

**Table 5: Performance metrics by method and benchmark using Claude-3 SONNET and temperature=0.5**
| Coding agent | Metric | Method |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Pyflakes | PyRight | LLM | REDO-Pyflakes | REDO |  |  |
| CodeStory | Precision | 32.1 | 43.7 | 33.3 | 32.9 | 34.1 |
| Recall | 22.8 | 48.1 | 67.1 | 69.6 | 75.9 |  |
| F1 | 26.7 | 45.8 | 44.5 | 44.7 | 47.1 |  |
| Demo | Precision | 34.1 | 33.9 | 31.8 | 31.4 | 32.1 |
| Recall | 34.9 | 45.3 | 65.1 | 70.9 | 77.9 |  |
| F1 | 34.5 | 38.8 | 42.7 | 43.6 | 45.4 |  |
| Marscode | Precision | 22.9 | 33.8 | 29.5 | 28.2 | 29.3 |
| Recall | 16.4 | 37.3 | 68.7 | 74.6 | 76.1 |  |
| F1 | 19.1 | 35.5 | 41.3 | 41.0 | 42.3 |  |
| Lingma | Precision | 45.8 | 48.3 | 40.9 | 40.7 | 41.4 |
| Recall | 27.8 | 29.9 | 69.1 | 74.2 | 74.2 |  |
| F1 | 34.6 | 36.9 | 51.3 | 52.6 | 53.1 |  |
| Droid | Precision | 53.3 | 42.1 | 43.7 | 42.3 | 43.0 |
| Recall | 22.0 | 14.7 | 67.0 | 70.6 | 70.6 |  |
| F1 | 31.2 | 21.8 | 52.9 | 52.9 | 53.5 |  |
| ACR | Precision | 45.3 | 54.7 | 39.6 | 39.1 | 40.9 |
| Recall | 23.5 | 34.3 | 63.7 | 68.6 | 72.5 |  |
| F1 | 31.0 | 42.2 | 48.9 | 49.8 | 52.3 |  |

**Table 6: Means and standard deviations using PyRight, LLM, and REDO on STA.**
| Metric | Method |  |  |
| --- | --- | --- | --- |
| PyRight | LLM | REDO-PyRight |  |
| Without context |  |  |  |
| Accuracy | 74.5 0.1 | 61.8 0.0 | 75.1 0.3 |
| Running Accuracy ^† | 76.4 0.0 | 64.1 0.1 | 76.8 0.3 |
| W.F1 | 67.2 0.1 | 54.0 0.0 | 69.9 0.2 |
| Running W.F1 | 70.1 0.0 | 57.2 0.0 | 72.6 0.2 |
| With context |  |  |  |
| Accuracy | 74.5 0.1 | 64.3 0.1 | 77.4 0.4 |
| Running Accuracy | 76.4 0.0 | 66.8 0.2 | 79.3 0.3 |
| W.F1 | 67.2 0.1 | 58.6 0.1 | 74.5 0.3 |
| Running W.F1 | 70.1 0.0 | 62.1 0.1 | 77.5 0.2 |

## Appendix D Ensemble patches

Figure: Algorithm 2 Patch ensemble

For a base agent, we denote the number of instance on SWEDE switching from failed to passed as $M$; and from passed to failed as $N$. The performance enhancement $E$ of the base agent is therefore measured by the difference in number between newly passed instances to newly failed instances, or $E=M-N$. We further consider two scenarios. First, only passed instances or instances with runtime errors using the base agent are considered; second, all instances are considered. These two scenarios evaluates how well REDO detects specifically runtime errors and any errors, respectively. We report the results in Figure 7(a) and 7(b), where the rows represent different base agents and columns represent auxiliary agents.

First, the enhancements $E$’s are positive for the majority of the entries, and are especially significant when ensemble a base agent with a more powerful auxiliary agent (as shown by numbers in lower triangular entries). The average enhancement is 3.1 in Figure 7(a) and 4.6 in Figure 7(b). These enhancements are meaningful as more powerful agents usually involve more API calls. Switching to more powerful agents only when necessary can reduce over cost and time. Second, when the base agent is CodeStory, the enhancements are negative. This could results from the fact that the ensemble algorithm currently does not consider the difference in the coding editing performance between base and auxiliary agents, therefore being over-confident with less powerful auxiliary agents. Since the ensemble algorithm is orthogonal to our contribution, we regard improving the algorithm as an important future work.

Figure: (a) Scenario 1: passed and runtime error instances
Refer to caption: x13.png

## Appendix E Prompt templates

### E.1 Prompt template: system prompt

### E.2 Prompt template: error detection on Code Editing

### E.3 Prompt template: error detection on CodeNet with input context

### E.4 Prompt template: error detection on CodeNet without input context

### E.5 Prompt template: error detection on SWE-Bench-lite

### E.6 Prompt template: patch fixing

## Appendix F Additional qualitative results

Figure: Figure 8: Generated patch and LLM response on Django-13551 using Lingma.
Refer to caption: x15.png

Figure: Figure 9: Generated patch and LLM response on Django-11797 using Codestory.
Refer to caption: x16.png

### F.1 Another successful example

This example is on instance Django-10924 using Lingma. As shown in Figure 10, the modification fails due to calling a non-callable object field.path. According to the modification patch in Figure 10, this happens if a non-callable path variable is input into the init function. As this error is only triggered during runtime, static analysis tools fail to capture it. On the other hand, as shown in Figure 10, the LLM succeeds to identify this situation and foresee the runtime error.

Figure: Figure 10: Generated patch and LLM response on Django-10924 using Mentatbot.
Refer to caption: x17.png

### F.2 Another failed example

This example is on instance Sympy-13471 using Mentabot, which actually passed the unit test. As shown in Figure 11, the LLM identifies a corner case where the variable num is not a string, which will fail the program. However, since this corner case is not triggered, the instance passed the unit test, making the LLM prediction a false positive alarm.

Figure: Figure 11: Generated patch and LLM response on Sympy-13471 using Mentatbot.
Refer to caption: x18.png

## Appendix G Patch fixing

As depicted in Figure 12, REDO identifies two potential runtime errors, including an AttributeError triggered by invoking the prepare_value attribute from JSONField. The evaluation log after executing the modified implementation confirms that this AttributeError indeed caused a crash. Subsequently, only basing on the results from REDO and without knowing the true error, Figure 13 shows that the corrected patch avoids invoking JSONField, which successfully mitigates the AttributeError.

We remark that although the AttributeError is resolved, the fixed patch may introduce new AssertionError instances or other runtime errors. The AssertionError lies beyond the detection capabilities of REDO, and the emergence of additional runtime errors suggests that an iterative process may be required to fully eliminate all such errors.

Figure: Figure 12: REDO analysis and evaluation log on the fixed patch.
Refer to caption: x19.png

Figure: Figure 13: Original generated patch and fixed patch
Refer to caption: x20.png

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: Que: Could the authors clarify the motivations of the paper beyond integration with agents, if any? That would maybe help view the current set of results with more context.
soundness: 3
strengths: I think the paper and the overarching idea of using existing program analysis tools in conjunction with LLMs within agents/systems is quite timely. Several recent breakthroughs in tasks such as SWEBench have been made by designing and integrating several classical tools such as linters, type checkers, repository dependency graphs, etc. I also found the data recipe to create the SWEDE benchmark from existing agent failures to be quite interesting.
summary: This paper introduces REDO which is a system that attempts to detect runtime errors in code modifications without execution by combining static analysis tools with LLMs.
weaknesses: **Motivations unclear**. The paper starts of with SWEBench and software agents as the motivation. However the task suddenly switches to simply predicting execution errors. The paper never circles back on how this prediction could actually help agent/systems that generate code. This makes the overall motivation of the paper quite weak in my opinion. While it is clear that setting up execution-environments for real-world code is challenging and an execution-free analyzer helps, the paper does not show any results that motivate that hypothesis.

**Execution feedback will almost always be better than REDO**. Adding to the point above, assume you have an execution environment for a repository. For most cases, the error messages from execution would contain a lot more information (traceback, type of error, etc.) than the assessment that REDO can provide. Furthermore you can even use an LLM to analyze the failed execution trace and code to produce even better feedback for agents. In that case, I'm wondering if one even needs a system like REDO if the motivation is to integrate with software agents.

**SWEDE an afterthought?**. I am not sure of the contributions of SWEDE as a benchmark as currently presented. While letting models figure out execution-based errors before execution takes place seems quite important and powerful, the current analysis or results do not explore this at all. Further more the benchmark reduces complex error detection and root cause analysis to binary classification. With that it seems like SWEDE is a means to evaluate REDO but it's unclear the value beyond that. There's potential for future versions of SWEDE to be a more comprehensive benchmark around predicting execution results that tap into whether or not LLMs truly understand code.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. What are the benefits of being execution-free? Static analysis is not necessarily execution-free. For instance, symbolic execution is a very powerful static analysis technique. It can be used to detect input-dependent errors. 

2. Can LLMs work more tightly with static analysis tools for this task, for instance, LLM-based static analysis or static analysis for LLMs? 

3. What are the key challenges that are solved by the proposed approach? Dubbing LLM-generated codes is generally a very challenging task. It could be valuable to identify what these challenges are and how they could be solved separately.
soundness: 3
strengths: 1. important problem. Coding agents face a very big problem of generating buggy and unreadable codes.  Automatically generated codes are difficult to debug manually. It is important to find automatic ways of analyzing LLM-generated codes. The authors provide an alternative solution in this direction. The proposed hybrid approach is reasonable and intuitive. Undoubtedly, LLMs can help to some extent find bugs in programs and could be leveraged with other algorithm-based approaches such as static analysis. This direction deserves further investigation. 


2. a clear and simple solution. The proposed approach is clear and easy to understand. The mechanism is simple and easy to implement. That might can be used to solve practical problems, as shown in the experiments. Errors can be found efficiently. As an auxiliary functionality, the approach could be integrated like grammar checkers or syntactic checkers for coding agents, providing useful suggestions and alerts to programmers to be careful of potential bugs that are detected.
summary: This paper studies how to apply LLM to runtime error detection of source codes that are generated by coding agents. The authors employ a hybrid approach in which a static analysis approach collaboratively works with LLM to detect runtime errors. Specifically, PyRight and Pyflakes are used to detect syntactical errors such as SyntaxError, NameError, while LLMs are used to detect runtime errors in codes. The authors claim that their hybrid approach is execution-free in that no execution is needed to detect such errors. A prototype is implemented and evaluated on a self-designed benchmark SWEDE. The experimental results show the outperformance of the proposed approach REDO over other related techniques.
weaknesses: 1. The capability of detecting bugs purely relies on backend LLMs and static analysis tools. I didn't see how the two approaches can be deeply integrated and collaborated. Apparently, the simple loose coupling of static analysis tools and LLMs cannot find all bugs in codes. Actually, no method exists to find all bugs. It would be very important to investigate collaborative ways of leveraging the advantages of tools and LLMs to find bugs or errors as many as possible. The current integration approach is very preliminary and shall be further explored. 

2. The use of static analysis tools is quite limited. The authors emphasize that their approach is execution-free. I didn't see why we have to consider execution-free. Are there some specific reasons or scenarios we have performance execution-free analysis? In the paper, the authors mentioned that static analysis cannot find input-dependent errors (or struggle to find them in their words). That is not the truth. There are various static analysis approaches to detect input-dependent errors such as symbolic execution, conclic testing. Why not apply these sophisticated approaches to LLM-generated codes? 

3. Technical contributions are limited. This point might be related to the above two issues. Throughout the paper, I didn't see any technical contributions, which I think is important to a research paper. To me, the improvement by REDO is not surprising, when comparing it with the purely static analysis tool PyRight. LLMs are extraordinarily powerful.  However, imposing LLMs on PyRight only gives very limited improvement (See Table 3). I believe LLMs can do more for this task if the collaboration between the tool and LLM can be tighter.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Could the authors clarify whether REDO is uniquely designed for coding agents or applicable to general program code error detection?

2. What specific information, if any, is transferred between PyRight and the LLM as indicated by the arrow in Figure 2?

3. Given that LLMs may introduce false positives, what measures are taken to control for this, especially when PyRight has deemed code safe?
soundness: 2
strengths: - Originality

REDO combines static analysis with LLMs, and although similar approaches are becoming more common, this implementation remains useful.

- Quality 

The empirical evaluation of REDO  provides meaningful insights into the effectiveness of the proposed approach.


- Clarity 

Overall, the paper is not difficult to follow.

- Significance 

REDO addresses an important issue by ensuring program code security with LLMs.
summary: The paper introduces REDO, a framework for error detection with large language modes (LLMs) without executing code. REDO combines static analysis with LLM-based detection to capture errors.

- The static analysis tools (e.g., PyRight) detects runtime errors by comparing the error profile of the original code with that of the modified code. By focusing on new errors introduced by modifications, this approach filters out false positives related to the original code and flags modifications that introduce potential runtime issues, such as SyntaxErrors or AttributeErrors.

- For instances where no errors are found through static analysis, an LLM (like Claude) is prompted with the problem context, original code, and modified patch to reason about runtime errors that might depend on specific inputs or complex conditions. This step helps detect context-sensitive errors, such as TypeErrors, that static analysis might miss, improving error detection without executing the code.


Additionally, the paper presents the SWEDE benchmark, an extension of SWE-Bench, to evaluate error detection capabilities in repository-level tasks with complex dependencies. 

In the experiments, REDO demonstrates improved accuracy and F1 scores compared to baseline methods, but it shows a reduction in precision.
weaknesses: - Overclaim

This paper claims to focus on error detection for coding agents; however, the REDO approach appears to be a more broadly applicable method for general program code error detection, which creates a misalignment in the work's focus.

- Limited Novelty

REDO combines PyRight with LLMs in a relatively straightforward way, with the LLM engaged only when PyRight deems the code safe. There seems to be no interaction or integration between the static analysis tool and the LLM beyond this sequential setup. In Figure 2, while an arrow suggests information flow from PyRight to the LLM, it is unclear what, if any, information is being transferred. Furthermore, other works also use LLMs alongside static analysis and various other tools, further undermining the novelty of REDO's design.

- Inconclusive Performance Improvement

The LLM in REDO is intended to catch potential errors that PyRight might miss; however, LLMs can sometimes mistakenly flag safe code as erroneous, thereby reducing precision. This limitation is reflected in REDO’s lower precision scores in the experiments, indicating that the addition of LLMs does introduce new issues, instead of consistently improve error detection performance.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - (Referring to W1) Can you provide a more detailed score on how precise LLM predictions are when differentiating between error types? Or alternatively provide some insight on why such differentiation is not needed?
- (Referring to W2) Could you provide some insight on the possibility and impact of combining static analysis with the method proposed by Bieber et. al.?
- What are the error margins in Figure 1?
- What does the bolded text in "STA dataset" refer to (see Nitpicks)?
- What is the expected impact of applying the proposed method in languages with stronger type safety guarantees? Or put differently, which kind of errors is most reliably predicted by static analysis tools and LLMs respectively and what is the expected impact when errors caught by type systems are covered by the compiler?
- Is there a reason for the different metrics of CodeNet and SWEDE and can the assessments be unified in their metrics?
soundness: 3
strengths: - The story is coherent and reads in an engaging way and mostly well-written and properly formatted.
- The figures and tables are well sized and easy to read and interpret.
- The proposed method is interesting in that it combines traditional conservative and novel more sensitive tooling, to obtain the best of both worlds.
summary: The paper presents REDO, a novel approach at predicting runtime errors in Python code patches by first querying traditional static analysis (SA) tools and extending them with LLM predictions if no error is detected. They compare the method with pure SA and pure LLM methods and one recent work on two datasets, one based on real world GitHub projects and one based on standalone scripts, and conclude that such methods are superior based on obtained F1 scores.
weaknesses: I think the paper presents an interesting idea, however I am not convinced that the claimed SOTA results have merit in the sense that it is claimed in the paper, therefore, without further changes I lean towards reject. Based on my current understanding addressing the following points would be sufficient to change my recommendation towards accept.

1) Lack of precision in assessing method prediction in SWEDE

The benchmark SWEDE considers all error types as one for its evaluation, considering coinciding "method predicts some error" and "some error occurred" as a successful prediction (Sec. 4). I think this score is significantly less interesting than the score used by Bieber et. al (2022) in the previous work, i.e. to consider each error _type_ as a separate class, for several reasons. 

- The purpose of error prediction is the potential use of the predicted information for subsequent repair of suggested fixes. The more precise such a prediction the more useful it is to whichever tool (i.e. LLM) is to implement such a fix. Predicting an AttributeError for a non-existing method when the actual error is completely unrelated may be unhelpful at best or confusing at worst for subsequent processing. Ideally one would even want to correctly predict the cause of the error, though this is admittedly difficult to assess in an automatic fashion. 
- The previous work, which this work compares to, uses exactly such a metric and a comparison might be unfair or misleading.
- Traditional, rule-based static analysis tools usually predict errors (and their sources) precisely. Related guarantees are often important for applications (i.e. see above) and are already violated by extending results with LLM predictions. It would be interesting to see how precise LLMs are, in order to obtain insight into whether further research in this direction will be required.


I suggest for these reason that either 
- the authors provide such a more detailed score for SWEDE or
- the authors present a compelling experiment to investigate whether the predicted errors are already helpful for repair (i.e. they could also still be helpful because they rarely point to entirely unrelated errors, even when not exactly matching)


2) No attempt at generalization of the proposed method

The authors compare the combined performance of static analysis tools and LLMs with preceding work that trains a tailored model on error prediction. Since the proposed method of combining i) precise but conservative static analysis with  ii) an imprecise but more sensitive learned model, is very general, I am wondering a) if it is not possible to combine the preceding work by Bieber et. al. with static analysis in such a way and b) what the performance of such an approach would be.

Consequently it is also not clear whether the improvement in the results stems from the involvement of LLMs or from the clever combination with static analysis tools. A more rigorous analysis of the involved factors would be interesting.

3) Presentation

Overall, the paper could benefit from a number of improvements in the general presentation. Below I present a list of smaller concerns regarding the form:

- In Figure 1 the chart is zoomed in a lot, to highlight results varying around 12% (min-max plotted value). Since the percentages are rather small I would like to see error bars that help assessing the significance of the results.
- Section 4 analyses external dependencies of python files in SWE-Bench "when only one directory level above the location where the modified files resides" is considered. It is not very clear to me what this means and even less so why the dependencies are not calculated for the entire repository (instead of vaguely mentioning that it "could become even more intimidating").
- Algorithm 1 is referenced in Sec. 3.2. Since it was not mentioned that the algorithm is in the appendix, it was difficult to find. I recommend either including the algorithm in the main body (which might also help general understanding or mentioning this) or explicitly referring to "Algorithm 1 in the Appendix"
- I don’t understand why the bolded text in the "STA dataset" section of Sec. 5.1 is bolded and which "different conclusions" it is referring to.
- Sec 5.1. is generally a large swath of text that might benefit from refactoring (i.e. moving details to the appendix and highlighting the main points more easily digestible)
- Table 3 would better fit on page 7 together with the corresponding dataset paragraph
- (A shortened version of) the analyzed patch and code for figures 4 and 5 would aid its understanding. The respective evaluation log could correspondingly be shortened.
- Appendix Section B is apparently completely empty.
- Row spacing in several tables is quite tight, resulting in intersections between presented numbers, underlines and table lines.
- Somehow no recall is provided for the CodeNet dataset and overall very different types of numbers are presented while the datasets do not fundamentally seem different. Similar metrics would IMO aid understanding here.

Meta-review: This paper introduces REDO, which attempts to detect runtime errors in code modifications without execution by combining static analysis tools with LLMs.
The reviewers agree that the motivation and technical contributions of the paper are limited and the authors are encouraged to further improve the technical depth of the paper for the next submission.

Decision: Reject