MLDebugging: Towards Benchmarking Code Debugging Across Multi-Library Scenarios
Authors: JinYang Huang, Xiachong Feng, Qiguang Chen, Hanjie Zhao, Zihui Cheng, Jiesong Bai, Jingxuan Zhou, Min Li, Libo Qin
Accepted at: acl25

MLDebugging: Towards Benchmarking Code Debugging Across Multi-Library Scenarios

Jinyang Huang♣*, Xiachong Feng♣*, Qiguang Chen◊ Hanjie Zhao♣ Zihui Cheng♣ Jiesong Bai♡ Jingxuan Zhou♣ Min Li♣ Libo Qin♣†
♣ School of Computer Science and Engineering, Central South University, China
♦ The University of Hong Kong
◊ Research Center for SCIR, Harbin Institute of Technology, Harbin, China
♡ School of Communication and Information Engineering, Shanghai University, China
hjy.tsuki@gmail.com, lbqin@csu.edu.cn

Abstract

Code debugging is a crucial task in software engineering, which attracts increasing attention. While remarkable success has been made in the era of large language models (LLMs), current research still focuses on the simple no-library or single-library setting, ignoring the complex multi-library scenario in real-world applications. To address this limitation, we make the first attempt to introduce MLDebugging (Multi-Library Debugging), a comprehensive benchmark designed to assess debugging challenges within multi-library Python code. Specifically, MLDebugging encompasses 126 distinct Python libraries, covering a wide range of multi-library code issues, categorized into seven distinct types. Furthermore, we conduct a thorough evaluation of MLDebugging using both mainstream open-source and closed-source LLMs and highlight that current LLMs still struggle to correctly perform code debugging across multi-library scenarios. We hope this work can uncover the potential of LLMs in multi-library debugging scenario and offer insights for future research.

1 Introduction

Code debugging emerges a significant urge for code review, requiring bug location first and then fixing the bug for correct functionality, which has garnered increasing attention for software engineering (Just et al., 2014; Lin et al., 2017). In light of the need to enhance the efficiency of code debugging and repair, a series of work consider adapting Automatic Code Debugging (ACD) techniques to serve as a fast and promising solution to the persistent issue of software defects (Austin et al., 2021; Chen et al., 2021; Li et al., 2024; Shi et al., 2024).

With the advancement of large language models (LLMs) (Zhao et al., 2025; Qin et al., 2024), a considerable body of research has been dedicated to effective code debugging. Specifically, Berabi et al. (2021) was the first to reframe the code debugging task as a Text-to-Text problem. Tian et al. (2024) introduced the first debugging dataset specifically designed for LLMs, which leveraged code snippets from the LeetCode (2025) platform. Khan et al. (2024) proposed multiple debugging sub-tasks, thereby expanding the code debugging task to encompass multi-language and multi-task scenarios. Furthermore, Liu et al. (2024b) further extended this benchmark to include multilingual debugging contexts, covering 18 programming languages. This extension facilitates a more thorough assessment of LLM debugging capabilities across various programming languages.

Despite its success, as shown in Figure 1 (a), the current research are still limited to the simple

![A representative static bug example (a) alongside our newly introduced multi-library bug instance (b) is shown. In (a), the error arises from using the assignment operator '=' instead of the equality comparison operator '==', while (b) involves an issue of variable adaptation between two library functions.](page_1012_320_393_312.png)

Figure 1: A representative static bug example (a) alongside our newly introduced multi-library bug instance (b) is shown. In (a), the error arises from using the assignment operator '=' instead of the equality comparison operator '==', while (b) involves an issue of variable adaptation between two library functions.
<table>
  <tr>
    <th>Benchmark</th>
    <th>Language Type</th>
    <th>Libraries</th>
    <th>Reference Code</th>
    <th>Task Scenario</th>
  </tr>
  <tr>
    <td>xCodeEval</td>
    <td>11 types</td>
    <td>[0, 1]</td>
    <td>✓</td>
    <td>Algorithm Competition</td>
  </tr>
  <tr>
    <td>HumanEval</td>
    <td>Python</td>
    <td>[0, 1]</td>
    <td>✓</td>
    <td>Simple Requirements</td>
  </tr>
  <tr>
    <td>MdEval</td>
    <td>18 types</td>
    <td>[0, 1]</td>
    <td>✓</td>
    <td>Simple Requirements</td>
  </tr>
  <tr>
    <td>QuickBugs</td>
    <td>Python, Java</td>
    <td>[0, 1]</td>
    <td>✗</td>
    <td>Algorithm Competition</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>Python</td>
    <td>[2, 6]</td>
    <td>✓</td>
    <td>7 Real-World Scenarios</td>
  </tr>
</table>

Table 1: Comparison of representative code debugging and generation benchmarks.

no-library or single-library setting, which fails to satisfy the requirements of some complex multi-library scenario in real-world applications. Actually, in real-world software development, the use of multiple libraries is a common practice, as evidenced by research (Feng et al., 2024), which emphasizes the importance of multi-library scenarios for code debugging. Unlike the previous no-library or single-library scenarios, as illustrated in Figure 1 (b), multi-library debugging naturally introduce two distinct challenges: (1) Understanding multiple libraries for bug location and (2) Utilizing multiple libraries for bug fixing, which cannot be addressed by previous approaches.

Driven by this motivation, we propose MLDebugging (Multi-library Debugging) in this study, a benchmark designed to evaluate debugging across 126 libraries, comprising 1,175 samples. As shown in Table 1, the task involves providing a code snippet that integrates multiple libraries, along with descriptions of the required functionality, test cases, and reference code. We employ GPT-4o (Ope-nai, 2024a) to produce buggy code samples derived from the multi-library code generation benchmark (Zhuo et al., 2024), which are then debugged through the application of multiple LLMs. Next, we design a bug category balancing process, enabling the generation of more stable and balanced bugs. Finally, we implement rigorous quality control to measure and validate the quality and authenticity of our dataset by comparing it with the distribution of real-world multi-library bugs.

To assess the limitations of current LLMs, we carry out a comprehensive evaluation of both open- and closed-source LLMs using MLDebugging. Our experiments reveal the following insights: (1) Current LLMs excel at debugging method class errors but struggle with conceptual mistakes and missing imports. (2) The structured nature of the MLDebugging, widespread use of libraries, and access to complete runtime information, such as test cases and feedback, enhance LLM performance. (3) In MLDebugging, models like DeepSeek-r1 (Guo et al., 2025), which are built on distillation techniques, fall short of enhancing task performance.

Our contributions are summarized as follows:

(1) We introduce a complex scenario of multi-library code debugging, addressing challenges encountered in real-world development tasks.

(2) We construct a multi-library code debugging benchmark with a total of 1,175 samples, covering 126 commonly used software libraries and categorized into 7 distinct bug types relevant to multi-library environments.

(3) We conduct a comprehensive analysis of the dataset’s performance across multiple models and provide detailed insights, with further exploration following the experiment.

To promote further investigation, the complete dataset can be accessed via https://github.com/hjyTsuki/MLDebugging.

2 Task Formulation

Consider a complete library set \( \mathcal{L} \), an error code \( C_{\mathcal{R},l} \) that implements a particular requirement \( \mathcal{R} \), and utilizes the subset of libraries \( l \subseteq \mathcal{L} \). Given an ideal test case set \( \mathcal{T} \), the error code \( C_{\mathcal{R},l} \) is defined to satisfy the following condition:

\[
\exists t \in \mathcal{T}, \text{exec}(C_{\mathcal{R},l}|t) = \text{error},
\]

where \( \text{exec}(x|y) \) denotes the execution of code \( x \) with input-output assertion \( y \), returning \( \text{error} \) if execution encounters a fault, or \( \text{pass} \) if the execution is successful without errors.

In the context of multi-library code debugging, the task involves generating the correct code \( \hat{C}_{\mathcal{R},l} \) from the erroneous code \( C_{\mathcal{R},l} \). This process can be formally expressed as:

\[
\hat{C}_{\mathcal{R},l} = \mathcal{D}(C|C_{\mathcal{R},l}, \mathcal{R}, \mathcal{L}),
\]

where \( \mathcal{D} \) represents the debugger, which utilizes the library set \( \mathcal{L} \) to correct the original code \( C_{\mathcal{R},l} \).
<table>
  <tr>
    <th>Type</th>
    <th>Count</th>
  </tr>
  <tr>
    <td>Type Mismatch (TM)</td>
    <td>97</td>
  </tr>
  <tr>
    <td>Data Transfer Issues (DTI)</td>
    <td>127</td>
  </tr>
  <tr>
    <td>Function Parameter Errors (FPE)</td>
    <td>88</td>
  </tr>
  <tr>
    <td>Parameter Configuration Errors (PCE)</td>
    <td>60</td>
  </tr>
  <tr>
    <td>Function Misuse (FM)</td>
    <td>101</td>
  </tr>
  <tr>
    <td>Requirement Misunderstanding (RM)</td>
    <td>143</td>
  </tr>
  <tr>
    <td>Import Errors (IE)</td>
    <td>23</td>
  </tr>
</table>

Table 2: The count of samples per category in the unbalanced dataset. In the subsequent sections of the article, we use the first letter shortened forms to replace the full names of the categories (TM, DTI, PFE, PCE, FM, RM, IE).

The resulting corrected code \( \hat{C}_{\mathcal{R},t} \) will pass all test cases, satisfying the following condition:

\[
\forall t \in \mathcal{T}, \text{exec}(\hat{C}_{\mathcal{R},t}|t) = \text{pass}.
\]

(3)

3 Data Collection

3.1 Source Code Collection

To obtain realistic erroneous code, as shown in Figure 2, we collect practical source code errors as follows: (1) First, to collect queries that involve multiple libraries, we adapt BigCodeBench (Zhuo et al., 2024), a dataset consisting of code snippets that address real-world programming tasks using two or more Python libraries selected from a pool of 179 widely-used libraries. (2) Next, we leverage GPT-4o (Openai, 2024a) following the methodology in Zhuo et al. (2024) to generate 1,038 code snippets involving multiple libraries, ensuring a broad range of real and diverse code errors. (3) Finally, we test all these generated code snippets using the provided test cases, which successfully identifies 609 buggy code snippets.

To further enable a thorough analysis of the dataset, we examine a set of common multi-library bugs preliminary. As shown in Table 2, we classify these bugs into 7 categories, each based on one of three perspectives: variable transfer between libraries, library function parameters, and Functionality comprehension. Based on this analysis, we introduce a clearer and more analyzable classification framework for evaluating debugging in practice, enabling more precise assessments of debugging performance across different bug types.

3.2 Annotating & Debugging With LLM

As shown in Figure 2, based on the previous classification, we manually provide detailed descriptions and examples for each bug category. Using this bug category information, we instruct the LLMs to classify each bug and generate a detailed bug description for each code snippet. This process is designed to assist and accelerate human annotation and next model debugging.

To enhance diversity and improve debugging performance, we employ three LLMs: GPT-4o (Openai, 2024a), DeepSeek-V3 (Liu et al., 2024a) and Claude-3-5-sonnet (Anthropic, 2024), so that we systematically collected the corrected code results for subsequent comparative analysis. However, it is worth noting that LLMs cannot always generate and repair code correctly on the first attempt. Inspired by the idea of test-time scaling (Wu et al., 2024; Chen et al., 2025b), for any unsuccessful debugging attempts, we conduct up to 5 additional trials to obtain a correct repaired code. After that, we obtain an unbalanced dataset, each comprising the bug category, corresponding correct and erroneous code pairs, and relevant test cases.

3.3 Bug Category Balance

As shown in Table 2, an imbalance in bug category distribution leads to evaluation bias, particularly for less frequent errors. To address this, as illustrated in Figure 2, we employ a balancing strategy.

Multi-Library Information Preparation  Analyzing source code in isolation often fails to capture the abstract semantics of code that incorporates multiple libraries, limiting a deeper understanding of its functionality and hindering the generation of debugged code. To address this, We utilize the Abstract Syntax Tree (AST) to capture the structure of code. Specifically, we prompt the LLM with the AST, ensuring it reflects variable transfers between libraries, the role of each library at each step, and how they collaborate to accomplish the task.

Category Balance  Based on the Information Preparation for Multi-Library code, we select a specific bug type from Table 2 and randomly extract corresponding bug instances from the unbalanced dataset. Specifically, we equally sample each unbalanced category to generate more code with bugs and automatically generate debugged code based on strategies in Sec. 3.1 and Sec. 3.2. Next, we manually filter the generated samples, and finally keep sample size of all categories is left are close. This methodology allowed for the successful injection of 566 bugs, standardizing the number of instances per category to approximately 200.
Figure 2: A pipeline diagram illustrating the dataset construction process.

3.4 Quality Control

To ensure the integrity of the MLDebugging dataset, we conduct thorough manual quality checks across all data instances.

Manual Bug Recheck & Fixing Due to the model’s inability to resolve all bugs, a manual review and correction process is employed for the unresolved code. Specifically, 4 experienced programmers, each with over 4 years of coding experience, are assigned to the task of bug fixing. Prior to beginning their work, these programmers undergo training on 50 sample cases to ensure consistency in labeling and to standardize the review process. To ensure the reliability of the bug-fixing process, overlapping cross-checks are organized, allowing for multiple reviews of the same cases. Any discrepancies identified during these reviews are resolved through collaborative discussions, ensuring accuracy and consistency in the final corrections.

Category & Description Recheck Finally, we manually assess the correctness of bug categories and bug descriptions to ensure that MLDebugging provides a reliable assessment of the model’s effectiveness in debugging different categories of bugs. Specifically, this process involves annotators comparing the fixed code with the buggy version to validate the correctness of the generated bug category and bug descriptions, as discussed in Sec. 3.2.

As a result of these efforts, as detailed in Table 3, we corrected 119 bug descriptions, 340 classifications of multi-library bugs, and manually fixed 185 buggy samples. Additionally, we removed 356 unreasonable samples from the generated dataset.

<table>
  <tr>
    <th>Criteria</th>
    <th>Correction Count</th>
  </tr>
  <tr>
    <td>BUG Description</td>
    <td>119</td>
  </tr>
  <tr>
    <td>BUG Type</td>
    <td>340</td>
  </tr>
  <tr>
    <td>Manual Debugging</td>
    <td>185</td>
  </tr>
</table>

Table 3: The number of erroneous samples identified through manual inspection at the quality control threshold.

4 Dataset Analysis

In this study, we design distribution-based experiments to evaluate the alignment between error distributions in our dataset and real-world debugging scenarios. First, we extract question-answer pairs focused on issues related to multiple libraries from Stack Overflow (2025). We then apply textual embeddings to the error descriptions in both the MLDebugging and DebugBench datasets.

To quantify the distributional similarity, we use two key measures: (1) Centroid-Based Comparison We calculate the centroids of bug description embedding vectors for MLDebugging, DebugBench, and StackOverflow separately, then evaluate the cosine similarity and Euclidean distance between them. (2) Real-World Proximity For each sample, we measure its distance to the nearest real-world sample from StackOverflow. Consider the text embedding vector \( \mathbf{a} \) from the benchmark (MLDebugging, DebugBench) and the real bug description embedding vector \( \mathbf{b} \) from StackOverflow. We retain the \( b_j \) points closest to \( a_i \) and compute the sum of their Euclidean distances to obtain the
<table>
  <tr>
    <th>Comparison</th>
    <th>C Sim(↑)</th>
    <th>E Dist(↓)</th>
    <th>DD(↓)</th>
  </tr>
  <tr>
    <td>ML and ST</td>
    <td>0.731</td>
    <td>0.376</td>
    <td>46.68</td>
  </tr>
  <tr>
    <td>DB and ST</td>
    <td>0.660</td>
    <td>0.432</td>
    <td>56.46</td>
  </tr>
</table>

Table 4: Comparison of the distances among MLDebugging (ML), DebugBench (DB), and StackOverFlow (ST), where C Sim represents the cosine similarity, and E Dist represents the Euclidean distance.

![t-SNE visualization of document embeddings for three datasets in a two-dimensional space.](page_186_370_627_393.png)

Figure 3: The t-SNE visualization of document embeddings for three datasets in a two-dimensional space.

Distribution Distance (DD) as:

\[
\text{DD} = \sum_{i=1}^{m} \min_j \|a_i - b_j\|_2.
\]

The results in Table 4 show that the cosine similarity between our dataset and StackOverflow exceeds that between DebugBench and StackOverflow. Additionally, the overall distances for our dataset are smaller than those for DebugBench, indicating that, at the script level, MLDebugging more accurately reflects the real-world bug distribution on StackOverflow. Moreover, as shown in Figure 3, the dimensionality reduction visualizations further reveal that our dataset forms more compact clusters, more closely aligning with real-world samples and emphasizing its practical relevance for multi-library code debugging.

5 Experiment

5.1 Experimental Settings

Model Settings  We conducted experiments on a wide range of models, including both open-source and closed-source systems, to provide a comprehensive understanding of multi-library code debugging. Specifically, we selected open-source models such as Qwen2.5 (Yang et al., 2024a), Qwen2.5-Coder (Hui et al., 2024), LLama3.1 (AI, 2024), Mistral (Jiang et al., 2024), and DeepSeek (DeepSeek, 2024), as well as closed-source models including the GPT series (Openai, 2024b) and Claude (Anthropic, 2024). Additionally, we evaluate emerging reasoning models, including the DeepSeek R1 Distill series (Guo et al., 2025) and QwQ-Preview (Team, 2024), to investigate how advanced reasoning capabilities contribute to improved debugging performance.

Metric Settings  The pass rate represents the proportion of code that pass all test cases. Given test cases \( \mathcal{T}_i \) and the code \( \hat{C}_{R,i}^j \) for i-th sample, the pass rate can be calculated as:

\[
\mathcal{P} = \frac{1}{n} \sum_{i=1}^{n} (\bigwedge_{t \in \mathcal{T}_i} [\text{exec}(\hat{C}_{R,i}^j|t) = \text{pass}]),
\]

where n denotes the number of codes in benchmark. Here, \( \bigwedge_{t \in \mathcal{T}[k]} \) denotes the logical “and” operation across all \( t \in \mathcal{T} \). This expression returns 1 if all test cases pass, and 0 otherwise.

5.2 Main Results

We evaluate MLDebugging with different LLMs varying in size, including closed- and open-source LLMs ranging from 7B to 72B.

All LLMs face challenges in MLDebugging. To assess the debugging capabilities of LLM in multi-library scenarios, we conduct an in-depth evaluation of various models’ performances on MLDebugging. As shown in Table 5, none of the LLMs demonstrate a significantly high pass rate in ML debugging. Specifically, the highest performance observed on the DeepSeek V3 dataset is only 58.7%, which indicates that all LLMs face substantial challenges in MLDebugging. This result also highlights significant room for improvement in the models’ multi-library debugging capabilities.

LLMs with increasing parameter sizes exhibit diminishing returns in MLDebugging. To understand the impact of varying model sizes on our benchmark, we evaluate a series of LLMs with different parameter counts. As shown in Table 5, model performance improves significantly from 7B to 32B parameters. However, from 32B to 72B, the performance gain levels off and may even decline. This suggests that our benchmark cannot be fully addressed by scaling alone and presents unique challenges inherent to multi-library tasks. One key reason is the scarcity of multi-library debugging
<table>
  <tr>
    <th rowspan="2">Category</th>
    <th colspan="6">7B+</th>
    <th colspan="6">14B+</th>
  </tr>
  <tr>
    <th>Qwen2.5</th>
    <th>Qwen2.5-coder</th>
    <th>Llama3.1</th>
    <th>Mistral</th>
    <th>DS Qwen</th>
    <th>DS Llama</th>
    <th>Qwen2.5</th>
    <th>Qwen2.5-coder</th>
    <th>DS Qwen</th>
    <th>DS-coder-Lite</th>
  </tr>
  <tr>
    <td>TM</td>
    <td>47.6</td>
    <td>40.0</td>
    <td>39.7</td>
    <td>28.8</td>
    <td>18.8</td>
    <td>33.5</td>
    <td>50.0</td>
    <td>54.1</td>
    <td>42.4</td>
    <td>30.0</td>
  </tr>
  <tr>
    <td>DTI</td>
    <td>36.1</td>
    <td>33.8</td>
    <td>30.5</td>
    <td>22.7</td>
    <td>15.7</td>
    <td>20.5</td>
    <td>40.7</td>
    <td>43.1</td>
    <td>30.6</td>
    <td>25.5</td>
  </tr>
  <tr>
    <td>PFE</td>
    <td>48.4</td>
    <td>48.8</td>
    <td>43.2</td>
    <td>29.1</td>
    <td>23.5</td>
    <td>26.2</td>
    <td>56.8</td>
    <td>62.0</td>
    <td>49.8</td>
    <td>34.7</td>
  </tr>
  <tr>
    <td>PCE</td>
    <td>57.6</td>
    <td>58.0</td>
    <td>49.8</td>
    <td>42.4</td>
    <td>33.1</td>
    <td>43.0</td>
    <td>66.1</td>
    <td>63.8</td>
    <td>57.6</td>
    <td>40.1</td>
  </tr>
  <tr>
    <td>FM</td>
    <td>38.2</td>
    <td>40.4</td>
    <td>38.8</td>
    <td>26.8</td>
    <td>20.2</td>
    <td>23.0</td>
    <td>44.1</td>
    <td>53.6</td>
    <td>35.5</td>
    <td>31.7</td>
  </tr>
  <tr>
    <td>RM</td>
    <td>12.6</td>
    <td>7.0</td>
    <td>5.6</td>
    <td>8.4</td>
    <td>2.8</td>
    <td>7.5</td>
    <td>15.4</td>
    <td>16.1</td>
    <td>11.2</td>
    <td>4.9</td>
  </tr>
  <tr>
    <td>IE</td>
    <td>26.1</td>
    <td>8.7</td>
    <td>13.0</td>
    <td>4.3</td>
    <td>3.3</td>
    <td>19.0</td>
    <td>30.4</td>
    <td>30.0</td>
    <td>17.4</td>
    <td>17.4</td>
  </tr>
  <tr>
    <th>Avg.</th>
    <th>42.7</th>
    <th>40.6</th>
    <th>36.7</th>
    <th>28.0</th>
    <th>20.6</th>
    <th>27.6</th>
    <th>48.6</th>
    <th>51.4</th>
    <th>40.2</th>
    <th>29.9</th>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Category</th>
    <th colspan="4">32B+</th>
    <th colspan="4">72B</th>
    <th colspan="4">MOE</th>
  </tr>
  <tr>
    <th>Qwen2.5</th>
    <th>Qwen2.5-Coder</th>
    <th>DS Qwen</th>
    <th>QwQ</th>
    <th>Qwen2.5</th>
    <th>Llama3.1</th>
    <th>DS-V3</th>
    <th>Claude</th>
    <th>GPT3.5</th>
    <th>GPT-4</th>
  </tr>
  <tr>
    <td>TM</td>
    <td>58.8</td>
    <td>56.5</td>
    <td>56.5</td>
    <td>46.5</td>
    <td>52.9</td>
    <td>53.5</td>
    <td>60.0</td>
    <td>45.9</td>
    <td>50.0</td>
    <td>55.3</td>
  </tr>
  <tr>
    <td>DTI</td>
    <td>50.0</td>
    <td>50.5</td>
    <td>45.8</td>
    <td>42.6</td>
    <td>47.2</td>
    <td>54.1</td>
    <td>52.8</td>
    <td>39.8</td>
    <td>37.5</td>
    <td>49.1</td>
  </tr>
  <tr>
    <td>PFE</td>
    <td>62.0</td>
    <td>59.2</td>
    <td>59.6</td>
    <td>54.5</td>
    <td>62.9</td>
    <td>45.4</td>
    <td>67.0</td>
    <td>43.7</td>
    <td>54.9</td>
    <td>67.1</td>
  </tr>
  <tr>
    <td>PCE</td>
    <td>70.4</td>
    <td>71.5</td>
    <td>67.3</td>
    <td>58.8</td>
    <td>70.4</td>
    <td>62.4</td>
    <td>76.3</td>
    <td>52.1</td>
    <td>60.3</td>
    <td>70.4</td>
  </tr>
  <tr>
    <td>FM</td>
    <td>55.9</td>
    <td>54.6</td>
    <td>50.5</td>
    <td>42.6</td>
    <td>53.8</td>
    <td>68.9</td>
    <td>56.2</td>
    <td>41.5</td>
    <td>44.8</td>
    <td>53.0</td>
  </tr>
  <tr>
    <td>RM</td>
    <td>20.3</td>
    <td>18.2</td>
    <td>20.3</td>
    <td>19.6</td>
    <td>16.1</td>
    <td>21.7</td>
    <td>23.8</td>
    <td>25.2</td>
    <td>9.1</td>
    <td>21.0</td>
  </tr>
  <tr>
    <td>IE</td>
    <td>26.1</td>
    <td>30.4</td>
    <td>21.7</td>
    <td>30.4</td>
    <td>26.1</td>
    <td>30.4</td>
    <td>34.8</td>
    <td>21.7</td>
    <td>21.7</td>
    <td>30.4</td>
  </tr>
  <tr>
    <th>Avg.</th>
    <th>55.7</th>
    <th>54.8</th>
    <th>52.6</th>
    <th>46.5</th>
    <th>53.7</th>
    <th>53.5</th>
    <th>58.7</th>
    <th>43.0</th>
    <th>45.7</th>
    <th>55.6</th>
  </tr>
</table>

Table 5: The table presents the results of various models in the MLDebugging benchmark, including Qwen2.5, Qwen2.5-Coder, Llama3.1, Mistral, closed-source models, and the DeepSeek(DS) R1 distillation series(The models we use are all based on the Instruct version). Bolded numbers indicate the highest pass rate achieved within models of the same parameter size. The Category column represents the initials of the category names listed in Table2

data in the training corpus (Du et al., 2025a); therefore, scaling the model does not substantially enhance reasoning capabilities, with accuracy remaining around 50–60% (Du et al., 2025b). Moreover, direct prompting strategies fall short in overcoming the intricate knowledge dependencies and interactions among libraries during debugging. These issues highlight inherent difficulties in multi-library tasks that require further research.

*LLMs exhibit varying capabilities across different multi-library debugging categories in MLdebugging.* To evaluate the debugging abilities of LLMs on different types of bugs, we conduct a detailed experiment based on the classification in Table 2. Table 5 presents the pass rate of various models on MLDebugging. While LLMs are somewhat effective for addressing certain bug types, multi-library debugging continues to pose significant challenges. Specifically, the categories TM & DTI, which involve *parameter-level* debugging with variable types and specific forms, and the following categories (PFE, PCE, & FM), which focus on *function-level* debugging, demonstrate relatively better performance. In contrast, the last two categories (RM & IE), which require reasoning and debugging at the *library-level*, show notably lower performance, with a gap of nearly 20% in pass rates. This disparity underscores the varying capabilities of LLMs across different multi-library debugging tasks and highlights the need for targeted improvements, particularly in handling debugging challenges of different complexity.

![The usage scenarios of Python libraries are categorized into eight distinct domains, several of the most representative libraries are selected.](page_1012_1342_420_246.png)

Figure 4: The usage scenarios of Python libraries are categorized into eight distinct domains, several of the most representative libraries are selected.

5.3 Analysis

In this section, we thoroughly analyze the behaviors exhibited by LLMs, with a particular focus on examining their performance from the perspective of library-specific understanding.

5.3.1 Impact of Library Usage Scenario

To gain a comprehensive understanding of the LLMs’ debugging performance across various libraries, we adopt the scenario classification from BigCodeBench (Zhuo et al., 2024). Figure 4 illustrates the debugging pass rates for representative libraries within each scenario. The experimental results indicate that the LLMs’ debugging perfor-
<table>
  <tr>
    <th>Variable</th>
    <th>Correlation</th>
    <th>P-Value</th>
  </tr>
  <tr>
    <td>lines of code</td>
    <td>-0.0071</td>
    <td>0.9654</td>
  </tr>
  <tr>
    <td>library count</td>
    <td>-0.2113</td>
    <td>0.1906</td>
  </tr>
  <tr>
    <td>prevalence</td>
    <td>0.4094</td>
    <td>0.0087</td>
  </tr>
</table>

Table 6: The table presents the point-biserial correlation coefficients and p-values between these three factors (lines of code, library count, and library prevalence) and the pass/fail outcomes.

mance varies across different types of libraries, revealing the following insights:

*LLMs perform well in commonly well-regularized and structured libraries.* As illustrated in Figure 4, models perform well in structured and well-regularized libraries covering general algorithms (General), data processing (Computation), and tasks related to encryption and visualization (Visualization). Specifically, they achieve high pass rates in libraries like itertools (0.641), collections (0.570), sklearn (0.654), base64 (0.724), and PIL (0.714) on GPT-4. This strong performance is likely due to the well-structured and clear definition of tasks in large-scale corpora, which allow LLMs to effectively learn and apply general debugging.

*LLMs struggle with dynamic and unstructured multi-library debugging.* As shown in Figure 4, LLMs underperform in tasks involving network communication (NetWork) and time processing (Time), particularly when using libraries like bs4 (0.286), urllib (0.375), and nltk (0.167) on GPT-4. These deficiencies in time-related libraries highlight the model’s limitations in dynamic debugging, specifically in its understanding of time logic and cross-timezone processing. This results in challenges when trying to accurately detect anomalous behaviors in dynamic environments.

5.3.2 Impact of Library Prevalence

*High prevalence of libraries elicits the models’ capacity on MLDebugging.* In the experiments presented in Table 5, LLMs demonstrate poor performance in MLDebugging. To explore the underlying causes of these difficulties, we hypothesized that the challenges in debugging are associated with factors such as code length, the quantity of libraries used, and the prevalence of LLMs encountered on the internet corpus. To test this hypothesis, we computed the correlation between each of these factors and the pass rate. As shown in Table 6, we find that the prevalence of libraries exhibits the strongest correlation with the debugging difficulty, suggesting that models tend to perform better when handling libraries that are more commonly encountered.

![Bar chart showing debugging effect by runtime information for different bug categories](page_1012_120_388_246.png)

Figure 5: The debugging effect achieved by incorporating runtime information for different bug categories in Table 2.

![Bar chart showing impact of inference ability on debugging](page_1012_420_388_246.png)

Figure 6: The results of various models under baseline conditions, with CoT prompts, and following R1 distillation training.

5.3.3 Exploration

*Both test cases and runtime error messages are essential for MLDebugging.* Execution feedback consistently acts as a vital information source and plays an essential role in effective debugging (Zhong et al., 2024). To explore its impact, we introduce detailed test case and runtime error message information separately into prompts to validate the effectiveness for multi-library debugging. As illustrated in Figure 5, the results illustrate the debugging scenarios based on test cases, runtime feedback, and their combined use. The results demonstrate that providing either test cases or runtime feedback individually offers additional information, thereby improving the model’s performance in most cases. However, in certain experiments involving only test case feedback or only runtime feedback, we incorporate both information to the LLMs’ inputs, leading to better and more robust performance. This suggests that supplying both comprehensive test cases and runtime error messages is essential for ensuring stable performance improvements in debugging tasks.
CoT achieves superior performance in MLDebugging. The Chain-of-Thought (CoT) prompt-based approach has been shown to exhibit enhanced reasoning capabilities in various tasks (Wei et al., 2022; Qin et al., 2023; Zhang et al., 2024; Chen et al., 2024). To investigate this effectiveness for MLDebugging, we adopt the approach introduced by Kojima et al. (2022), wherein the prompt “Let’s think step by step” is used to trigger LLMs to generate a CoT reasoning process. Specifically, we assess the performance of contemporary LLMs with parameter sizes ranging from 7B to 32B, including Qwen2.5 and LLaMA 3.1. As shown in Figure 6, the CoT prompt-based approach consistently demonstrates a significant improvement in reasoning capabilities, particularly in debugging tasks. These enhanced logical reasoning skills prove to be crucial for improving performance in multi-library debugging scenarios.

Reasoning models based solely on distillation fail to improve task performance. Emerging research shows that test-time scaling techniques in reasoning models greatly boost the effectiveness of LLMs (Snell et al., 2024; Jaech et al., 2024; Guo et al., 2025; Chen et al., 2025a,b). Motivated by this insight, we evaluated the Deepseek-R1-Distill-Qwen models. However, as shown in Figure 6, although CoT demonstrates improved performance, reasoning models trained with distilled long CoT data exhibit worse performance. We attribute it to the fact that the use of supervised fine-tuning (SFT) with distilled data alone may not sufficiently enhance the model’s capabilities. To achieve a more substantial improvement, further exploration of alternative strategies, such as reinforcement learning (RL), is warranted.

5.4 Error Analysis
To understand the key challenges in MLDebugging task, we conduct a comprehensive review of the debugged code output by the LLMs. We attribute the primary causes to the following two factors:

Absence of library knowledge A key challenge of (LLMs) is their limited knowledge (Huang et al., 2024). To examine this issue in the context of multi-library debugging, we conduct a detailed manual analysis of LLM-generated outputs, with particular emphasis on their use of specialized knowledge. As demonstrated in Case 1 of Error Analysis, LLM fails to understand the “virtual memory” method in the psutil library, leading to a misidentification that hindered its ability to extract the relevant attributes. This highlights a major limitation of LLMs: a superficial or incomplete understanding of specialized software libraries. As a result, effectively localizing errors and reconstructing faulty code with a sufficient and accurate grasp of the relevant programming libraries present significant and persistent challenges in the field of MLDebugging.

Requirements for efficient cross-library debugging When dealing with interactions between multiple libraries, the inherent complexities introduce significant challenges. As illustrated in Case 2 in Error Analysis, the model’s inability to comprehend cross-library variables impedes its capacity to detect redundant operations on the dataframe. The variation in variable types across libraries, such as differing classes and structures, prevents the model from performing fine-grained debugging, resulting in inaccuracies when handling task-specific details. As a result, effectively correcting code involving cross-library interactions also presents significant challenges in MLDebugging.

6 Related Work
The rise of large language models (LLMs) has also had a considerable impact on Automated Code Debugging (ACD) tasks, as previous datasets have been vulnerable to data leakage risks (Just et al., 2014; Lin et al., 2017). To facilitate a smoother transition from traditional datasets to those suited for LLM evaluations, numerous remarkable contributions have surfaced. DebugBench (Tian et al., 2024) stands as the first dataset designed specifically to assess the debugging capabilities of large models. This work collects data from LeetCode, subsequently introducing bugs via model injection. In the realm of APR, xCodeEval (Khan et al., 2024) has proposed three distinct sub-tasks, spanning multiple programming languages, to comprehensively evaluate a model’s ability to repair code. Meanwhile, MdEval (Liu et al., 2024b) adopts a multi-language approach, encompassing a benchmark across 18 programming languages. In addition, the challenges presented by specific real-world scenarios have prompted research into niche areas, with several efforts concentrating on executable code, data processing, and other specialized contexts (Yang et al., 2024b; Prenner and Robbes, 2023; Galimzyanov et al., 2024).

These advancements have undeniably propelled the field forward. However, the majority of these
datasets are sourced from algorithmic competition platforms, often overlooking the need for Python multi-library scenarios. Therefore, we have constructed a benchmark specifically aimed at evaluating the debugging abilities of models in the context of multiple libraries, providing an in-depth assessment of their proficiency in both static knowledge comprehension and multi-library code interaction.

7 Conclusion

This work introduces MLDebugging, a benchmark designed to assess debugging challenges in multi-library code. We conduct a comprehensive analysis on MLDebugging and the experimental results reveal that the current LLMs still struggle in multi-library scenario. This work emphasizes the need for further research to improve LLM performance in multi-library settings and provides insights to guide future developments in this field.

8 Acknowledgement

This work was supported by the National Natural Science Foundation of China (NSFC) via grant 62306342. This work was sponsored by the Excellent Young Scientists Fund in Hunan Province (2024JJ4070), the Science and Technology Innovation Program of Hunan Province under Grant 2024RC3024 and CCF-Zhipu Large Model Innovation Fund (NO.CCFZhipu202406). We are grateful for resources from the High Performance Computing Center of Central South University.

Limitations

We propose MLDebugging, the first multi-library code debugging benchmark, yet there are still two main limitations: (1) Although the analysis results indicate that the data distribution of MLDebugging closely resembles real-world data, most of the data in MLDebugging are automatically generated by models, which means there will still be some differences compared to real data. In the future, we consider incorporating more real-world data to further enhance the realism and usability of MLDebugging. (2) Despite that our experiments comprehensively evaluate various models and error categories, the entire evaluation process can be cumbersome due to the need to configure several external dependencies and complex environments, which consumes a significant amount of time.

References

Meta AI. 2024. Llama3.1.

Anthropic. 2024. Claude 3.5 sonnet. Available at: https://www.anthropic.com/news/clause-3-5-sonnet.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.

Berkay Berabi, Jingxuan He, Veselin Raychev, and Martin Vechev. 2021. Tfix: Learning to fix coding errors with a text-to-text transformer. In International Conference on Machine Learning, pages 780–791. PMLR.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. 2025a. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. arXiv preprint arXiv:2503.09567.

Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiaqi Wang, Mengkang Hu, Zhi Chen, Wanxiang Che, and Ting Liu. 2025b. Ecm: A unified electronic circuit model for explaining the emergence of in-context learning and chain-of-thought in large language model. arXiv preprint arXiv:2502.03325.

Qiguang Chen, Libo Qin, Jiaqi WANG, Jingxuan Zhou, and Wanxiang Che. 2024. Unlocking the capabilities of thought: A reasoning boundary framework to quantify and optimize chain-of-thought. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

DeepSeek. 2024. deepseek.

Enjun Du, Xunkai Li, Tian Jin, Zhihan Zhang, Rong-Hua Li, and Guoren Wang. 2025a. Graphmaster: Automated graph synthesis via llm agents in data-limited environments. A preprint.

Enjun Du, Liu Siyi, and Zhang Yongqi. 2025b. Graphoracle: A foundation model for knowledge graph reasoning. A preprint.

Jia Feng, Jiachen Liu, Cuiyun Gao, Chun Yong Chong, Chaozheng Wang, Shan Gao, and Xin Xia. 2024. Complexcodeeval: A benchmark for evaluating large code models on more complex code. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pages 1895–1906.
Timur Galimzyanov, Sergey Titov, Yaroslav Golubev, and Egor Bogomolov. 2024. Drawing pandas: A benchmark for llms in generating plotting code. arXiv preprint arXiv:2412.02764.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2024. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems.

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kening Lu, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186.

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai ol system card. arXiv preprint arXiv:2412.16720.

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lampe, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024. Mixtral of experts. Preprint, arXiv:2401.04088.

René Just, Dariosh Jalali, and Michael D Ernst. 2014. Defects4j: A database of existing faults to enable controlled testing studies for java programs. In Proceedings of the 2014 international symposium on software testing and analysis, pages 437–440.

Mohammad Abdullah Matin Khan, M Saiful Bari, Do Long, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. 2024. Xcodeeval: An execution-based large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6766–6805.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213.

LeetCode. 2025. Leetcode: The world’s leading online programming learning platform. https://leetcode.com.

Jierui Li, Hung Le, Yinbo Zhou, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. 2024. Codetree: Agent-guided tree search for code generation with large language models. arXiv preprint arXiv:2411.04329.

Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. 2017. Quixbugs: A multi-lingual program repair benchmark set based on the quixey challenge. In Proceedings Companion of the 2017 ACM SIGPLAN international conference on systems, programming, languages, and applications: software for humanity, pages 55–56.

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437.

Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Ke Jin, Wei Zhang, Hualei Zhu, Shuyue Guo, et al. 2024b. Mdeval: Massively multilingual code debugging. arXiv preprint arXiv:2411.02310.

Openai. 2024a. Hello gpt-4o.

Openai. 2024b. Introducing chatgpt.

Julian Aron Prenner and Romain Robbes. 2023. Runbugrun—an executable dataset for automated program repair. arXiv preprint arXiv:2304.01102.

Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip S. Yu. 2024. Large language models meet nlp: A survey. Preprint, arXiv:2405.12819.

Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. 2023. Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages. Preprint, arXiv:2310.14799.

Yuling Shi, Songsong Wang, Chengcheng Wan, and Xiaodong Gu. 2024. From code to correctness: Closing the last mile of code generation with hierarchical debugging. arXiv preprint arXiv:2410.01215.

Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314.

Stack Overflow. 2025. Stack overflow: Newest questions. https://stackoverflow.com/questions.

Qwen Team. 2024. Qwq.

Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Haotian Hui, Weichuan Liu, Zhiyuan Liu, et al. 2024. Debugbench: Evaluating debugging capability of large language models. arXiv preprint arXiv:2401.04621.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. 2024. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00724.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024a. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115.

Jian Yang, Jiajun Zhang, Jiaxyi Yang, Ke Jin, Lei Zhang, Qiyao Peng, Ken Deng, Yibo Miao, Tianyu Liu, Zeyu Cui, et al. 2024b. Excrepobench: Multi-level executable code completion evaluation. arXiv preprint arXiv:2412.11990.

Yongheng Zhang, Qiguang Chen, Min Li, Wanxiang Che, and Libo Qin. 2024. Autocap: Towards automatic cross-lingual alignment planning for zero-shot chain-of-thought. arXiv preprint arXiv:2406.13940.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2025. A survey of large language models. Preprint, arXiv:2303.18223.

Li Zhong, Zilong Wang, and Jingbo Shang. 2024. Debug like a human: A large language model debugger via verifying runtime execution step by step. In Findings of the Association for Computational Linguistics: ACL 2024, pages 851–870, Bangkok, Thailand. Association for Computational Linguistics.

Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadiira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. 2024. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877.
Appendix

A  Dataset Construction Details

A.1  Prompts for Dataset Construction

This appendix includes the prompts used during both the dataset construction phase and the evaluation phase.

In the second stage of dataset construction, the prompts provided to the model for debugging tasks include several key components: the instruction, the buggy code, the test case, the correct code from the original dataset, and the error message. The model is then required to format and output the solution accordingly. We present the prompts used during the second stage of dataset construction, which are provided to the model for debugging tasks.

We will provide a Python code snippet, task_func, which involves multiple libraries and contains bugs. Your task is to debug the code, identify the issues across the libraries, and categorize the bugs. Please follow these steps: Review the requirements of the code and thoroughly analyze the task_func.
Using the provided test cases and error messages, identify and list the bugs in the code, along with a description of each issue. Correct each identified bug, provide the updated code, and categorize the bugs according to the relevant multi-library issue types. The following information is provided for debugging:
<instruct>: The task of this code segment.
<bug_code>: The code that needs debugging.
<canonical_solution>: The corrected version of the code.
<test_case>: Sample test cases for validation.
<error>: Relevant error messages.

The identified bugs are categorized as follows:
Please output each identified bug in the following format. Write each part’s content with only one tag. Each part must be included:
<bug_des>
Detailed description of the bug </bug_des>
<code> # Import necessary package(s) and provide the refined function code without comments
import ……
def task_func(
</code>
<category> Only output the category names from the seven categories mentioned above </category>
Few-Shot:

we include the prompts employed during the data generation process, covering the analysis and bug injection phases.

You will receive a piece of code that is a function designed with multiple packages, and its corresponding multi-library AST structure is also provided. Your tasks are as follows:
(1) Analyze the relationships between multiple libraries from the perspective of variable passing, based on the provided code and its corresponding multi-library AST (Abstract Syntax Tree) structure.
(2) Inject a specific type of bug, provide a description of the bug, and the code where the bug is injected (do not include any comments). Type Description:
The input format is as follows:
<Instruction> Code Requirements </Instruction>
<Original_Code> Correct Implementation </Original_Code>
<AST> Abstract Syntax Tree </AST>
The output format is as follows:
<AST_analysis> Analyze the relationships between multiple libraries, focusing on variable passing. Analyze the relationships between multiple libraries from the perspective of variable passing, based on the provided code and its corresponding multi-library AST (Abstract Syntax Tree) structure. the provided code and its corresponding multi-library AST (Abstract Syntax Tree) structure.
</AST_analysis>
<bug_des> This is a description of multi-database bugs for the specific category required.
</bug_des>
<bug_code> # Import the necessary packages and provide the bug-implanted code without comments
import ……
def task_func(
</bug_code>

Finally, we outline the various prompts utilized during the testing phase. Standard evaluation prompt:

There is an important info hidden inside a lot of irrelevant text. Find it and memorize them. I will quiz you about the important information there.
Please review the task_func function for errors. Begin by reading the provided instructions to understand the intended behavior of the function. Ensure the code aligns with the requirements and identify any issues. Correct any errors found and provide the revised code.
Input format:
<instruct>: Code requirements and expected functionality
<bug_code>: The original (bugged) version of the code.

Please output only the debugged code under the label <corrected_code>, without any additional text or comments:
Output Format Example
<corrected_code>
import ……
def task_func(……
<corrected_code>

Reasoning model prompt
Please review the task_func function for errors. Begin by reading the provided instructions to understand the intended behavior of the function. Ensure the code aligns with the requirements and identify any issues. Correct any errors found and provide the revised code.

Input format <instruct>: Code requirements and expected functionality <bug_code>: The original (bugged) version of the code.
Please output the final answer at the end, enclosed in markdown format, without any additional text or comments. Final answer output Format Example:
“python import …… def task_func(…… “

CoT prompt

Please review the task_func function for errors. Begin by reading the provided instructions to understand the intended behavior of the function. Ensure the code aligns with the requirements and identify any issues. Correct any errors found and provide the revised code.

Input format
<instruct>: Code requirements and expected functionality
<bug_code>: The original (bugged) version of the code.

Let’s think Step by Step to Solve this problem. Please output the final answer at the end, enclosed in markdown format, without any additional text or comments.
Final answer Output Format Example
“python
import ……
def task_func(……
”

A.2 The Details of Manual Annotation

To ensure the quality of the dataset, we provided training for the data annotate team using a sample of 50 entries and established a structured workflow for dataset annotation:

(1) The dataset is divided, with each annotator assigned an equal portion of the data for labeling.

(2) Each annotator first reviews 50 labeled samples, subsequently following the established guidelines for further annotation.

(3) It is imperative that the data reviewed each day meets the criteria, with the bug code matching both its category and description, and the "Golden" code passing corresponding test cases.

(4) Any unresolved issues are addressed through consensus within the annotator group.

The guidelines for the annotate process are as follows:

Instruction
To construct a Code Debug dataset, we first need to debug the code containing bugs. The debugging process involves a thorough examination of both the model and the corrected code. The dataset includes the following information: the code’s instructions, the bug categories, the buggy code, the golden code, and testcase. These elements provide essential support for subsequent debugging and model evaluation.

Workflow
The provided files contain a jsonl file, which can be processed using the check.py script to generate a consolidated output file for easier inspection. The Check folder holds the necessary information for performing code debugging, while the RUN folder offers convenient debugging support for code testing. During the debugging process:

(1) It is essential to verify whether the buggy code aligns with the bug description and to assess it according to the classification standards provided in the accompanying materials. If discrepancies are found, corrections must be made accordingly.

(2) The golden code must be executed, and its correctness verified through the corresponding test cases. If it fails to pass the test cases, manual corrections are required.

(3) The correction process for each data entry should be thoroughly documented to ensure traceability of the modifications.

B Error Analysis

Case 1
In the first example, the model fails to locate and refactor the code due to a lack of knowledge regarding the virtual_memory method in the psutil library. As a result, the outputted repair version remains unchanged.

Buggy Code:
import psutil
import platform
def task_func():
    system_info = {}
    system_info['OS'] = platform.
        → system()
    system_info['Architecture'] =
        → platform.architecture()[0]
    memory = psutil.virtual_memory()
    system_info['Memory Usage'] = f"(→ memory.percent)%"
    return system_info
Repaired Code:
import psutil
import platform
def task_func():
    system_info = {}
    system_info['OS'] = platform.
        system()
    system_info['Architecture'] =
        platform.architecture()[0]
    memory = psutil.virtual_memory()
    system_info['Memory Usage'] = f"{memory.percent}%"
    return system_info

Repaired Code:
import pandas as pd
import matplotlib.pyplot as plt
def task_func(data_list):
    df = pd.DataFrame(data_list)
    fig, ax = plt.subplots()
    for column in df:
        ax.plot(df[column].dropna().
            reset_index(drop=True),
            label=column)
    ax.set_title("Student Scores over Tests")
    ax.set_xlabel("Test Number")
    ax.set_ylabel("Score")
    ax.legend()
    return ax

Golden Code:
import psutil
import platform
def task_func():
    system_info = {}

    system_info['OS'] = platform.
        system()
    system_info['Architecture'] =
        platform.architecture()[0]

    total_memory = psutil.
        virtual_memory().total
    used_memory = psutil.
        virtual_memory().used
    system_info['Memory Usage'] = f'{used_memory/total_memory *100:.2f}%'

    return system_info

Golden Code:
import pandas as pd
import matplotlib.pyplot as plt
def task_func(data_list):
    df = pd.DataFrame(data_list)
    fig, ax = plt.subplots()
    for column in df:
        ax.plot(df[column], label=column)
    ax.set_title("Student Scores over Tests")
    ax.set_xlabel("Test Number")
    ax.set_ylabel("Score")

    return ax

Case 2
In the following error sample, the model fails to understand the input information during execution, leading it to overlook the redundant operation of removing missing values and reconstructing the index.

Buggy Code:
import pandas as pd
import matplotlib.pyplot as plt
def task_func(data_list):
    df = pd.DataFrame(data_list)
    fig, ax = plt.subplots()
    for column in df:

        ax.plot(df[column].dropna().
            reset_index(drop=True),
            label=column)
        ax.set_title("Student Scores over Tests")
        ax.set_xlabel("Test Number")
        ax.set_ylabel("Score")

        return ax

Reviewer paper summaries: This paper introduces a novel benchmark MLDebugging designed to evaluate code debugging capabilities in scenarios involving multiple Python libraries. Specifically, the authors construct the benchmark, comprising 126 libraries and a total of 1,175 samples. The benchmark categorizes bugs into seven distinct error types and provides extensive experiments to evaluate the performance of existing large language models (LLMs), offering insightful conclusions.

The paper introduces MLDebugging, the first benchmark designed for multi-library code debugging in Python. It includes 1,175 samples across 126 libraries and seven bug categories, reflecting real-world debugging complexity. The benchmark evaluates the performance of both open and closed-source large language models (LLMs), revealing that current LLMs struggle with multi-library bugs, especially in reasoning and library interaction.

This paper introduces MLDebugging, a new benchmark for evaluating the capability of LLMs to debug Python code snippets that involve multiple external libraries. Unlike previous datasets focused primarily on simpler scenarios, MLDebugging specifically targets multi-library scenarios to simulate realistic software development environments. The benchmark consists of 1,175 buggy Python examples covering 126 libraries, categorized into seven types of common multi-library bugs, such as type mismatches, data transfer issues, incorrect function parameters, and missing imports. The paper creates the dataset through automatic bug injection using GPT-4, followed by manual review and balancing. The paper then evaluates multiple widely-used LLMs to measure their debugging effectiveness, highlighting significant challenges and varying performance across bug categories.

This paper proposes a benchmark, MLDebugging, for debugging multi-library code in Python. It collects buggy codes across 126 libraries and classifies them into 7 types. The dataset contains both real-world codes and model-generated codes. A variety of LLMs (such as GPT-4, Claude, etc.) are evaluated on this benchmark, and the result shows that current models have difficulties in multi-library debugging, especially cross-library reasoning and dynamic error handling.
Summary of strengths: 1. The benchmark addresses the previously unexplored issue of debugging code involving multiple libraries, filling an important research gap in current literature.

1. A dataset for debugging related code used by Python's libary is proposed. This data is valuable for exploring the code debugging capabilities of LLM.
2. This paper evaluates some commonly used LLMs and shows the performance of the current model on this benchmark.

- The paper addresses a relatively underexplored and practical setting: debugging Python code that involves interactions between multiple external libraries, which differs notably from simpler debugging scenarios studied previously.

- It introduces a carefully curated dataset with over a thousand code snippets, systematically categorized into common bug types. This structured approach makes the dataset a potentially useful resource for developing and evaluating code debugging methods.

- Extensive experiments with multiple large language models offer insights into model strengths and weaknesses in handling various bug categories, highlighting particularly challenging aspects of multi-library debugging tasks.

- The paper performs detailed quality control, including manual inspection and balancing of bug types, which strengthens the reliability of the dataset.

• Focuses on the often-overlooked real-world multi-library debugging problem. 

• Comprehensive dataset from 126 libraries and for 7 different error types, and the dataset contains both real-world samples and generated samples to improve its robustness. 

• Comprehensive experiments are conducted for both open-source and closed-source LLMs. And detailed error analysis is provided.
Summary of weaknesses: 1. The paper lacks sufficient detail regarding the selection criteria of the 126 libraries. Information about the domains these libraries represent and whether they comprehensively cover various fields needs to be explicitly clarified to justify the benchmark’s representativeness and generality.
2. Detailed statistics of the MLDebugging dataset are missing, such as the number of bugs per library, types of bugs within each library, and the average token length of the code snippets. Providing these statistics would significantly enhance the dataset’s transparency and usability.
3. The rationale behind the choice of the seven bug categories remains unclear. Specifically, it is uncertain whether these categories directly relate to scenarios involving multiple libraries—for instance, "function parameter errors" may not exclusively or comprehensively represent multi-library interactions. Additional discussion on whether these categories fully cover typical and critical multi-library debugging scenarios is needed.

1. The work has limited novelty and contribution, authors only constructed the evaluation dataset based on bigcodebench.  
2. The analysis conducted in this work does not seem to be sufficient and clear. This work claims to evaluate bugs in multi-library but only evaluates the performance of error type grouping and library types, lacking analysis of multi-library errors (e.g., the impact of the number and combination of libraries on debugging)


Question：
1. I understand that multi-library in this work refers to the usage of multiple libraries involved in a code. Does this work count the number of libraries in each problem and explore the performance between the number of libraries and combinations?

2. Does this work take into account errors caused by the library versions, which is a very common type of bug in Python? 

3. The comparison in Table 1 does not seem reasonable. I think the comparison should be made based on the number of tasks, language types, amount of problems, number of bugs, etc.

- Although the paper validates synthetic bugs against real-world patterns and includes thorough manual verification, it remains unclear if the generation process (seeded from BigCodeBench with AST-based bug injection) fully captures the nuances of genuine multi-library integration issues versus more straightforward API misuse. Further analysis of the benchmark’s representativeness could help clarify its difficulty level.

- While the performance analysis connects results to factors such as bug type and library prevalence, the paper could benefit from a deeper exploration into why these factors affect model performance. For instance, is library prevalence linked more to memorization or to generalization challenges? Understanding these interactions could offer more insight into the core difficulties faced by LLMs on this task.

- The evaluation relies primarily on the pass@1 metric, which may not fully capture the models’ debugging capabilities, especially given the relatively low scores reported. Incorporating additional metrics or exploring different evaluation settings could provide a more nuanced understanding of model performance beyond a simple pass/fail binary.

• The majority of the dataset is automatically generated using GPT-4o. This may be different comparing to real-world data. (has been discussed in the "Limitations")

• It is recommended to discuss why scaling LLMs does not directly solve multi-library challenges.

• Some typos, e.g. "natrually" and "which which", need to be fixed.
Comments, suggestions and typos: See the weaknesses part.

Here are some minor errors I found: 
1. The parameter classification in Table 5 is somewhat inaccurate， llama3.1 don't have 72B version，maybe ~70B is a better description, In addition, ds-v3 is moe, but it is not clear whether the closed-source models gpt and claude are moe models
2. In 768 prompt box:  </<AST>
3. In 772 prompt box: <bug _ code>

- Consider including a brief discussion on why certain bug categories consistently appear easier or more challenging for the models, potentially shedding light on underlying model behaviors.

- A library-specific analysis might be valuable, especially if certain library combinations prove particularly difficult for debugging. Highlighting such cases could offer practical insights and guide future research directions.

- Please proofread the manuscript carefully to address minor writing issues and typographical errors, as improving clarity and presentation will benefit overall readability.

As shown above
Overall assessment scores: 3

2.5

3.5

3.5
Metareview score: 3
Metareview: This paper introduces MLDebugging, a novel benchmark for evaluating LLMs' ability to debug multi-library Python code. The benchmark includes 1,175 examples from 126 libraries across seven bug types. It addresses the critical gap of multi-library debugging, reflecting real-world complexities. The paper is well-written and provides valuable insights for future research, with comprehensive experiments highlighting LLM limitations. However, key concerns include insufficient analysis of library representativeness, reliance on synthetic data, lack of clarity regarding bug category rationale, and limited evaluation metrics. Addressing these issues would enhance the clarity and impact of the work.
Summary of reasons to publish: + Addresses the critical and underexplored challenge of multi-library debugging.
+ Proposes a carefully designed benchmark with diverse examples and manual validation.
+ Provides comprehensive experiments across open- and closed-source LLMs, offering actionable insights.