VersiCode: Towards Version-controllable Code Generation

## Abstract

Abstract Large Language Models (LLMs) have made tremendous strides in code generation, but existing research fails to account for the dynamic nature of software development, marked by frequent library updates.
This gap significantly limits LLMs’ deployment in realistic settings.
In this paper, we propose two novel tasks aimed at bridging this gap: version-specific code completion (VSCC) and version-aware code migration (VACM).
In conjunction, we introduce VersiCode, a comprehensive Python dataset specifically designed to evaluate LLMs on these two tasks, together with a novel evaluation metric, Critical Diff Check (CDC@1), which assesses code generation against evolving API requirements.
We conduct an extensive evaluation on VersiCode, which reveals that version-controllable code generation is indeed a significant challenge, even for GPT-4o and other strong frontier models.
We believe the novel tasks, dataset and metric open up a new, important research direction that will further enhance LLMs’ real-world applicability.
The code and resources can be found at https://wutong8023.site/VersiCode/ .

## 1 Introduction

Large Language Models (LLMs), including OpenAI’s GPT series (OpenAI, ; ; ) and specialized variants such as CodeLLaMA (Rozière et al., ), have demonstrated significant advancements in code generation tasks. Typically evaluated using benchmarks such as HumanEval (Chen et al., ) and MBPP (Austin et al., ), these models are measured on tasks that assume code generation is a *static* activity. However, the reality of software development is inherently dynamic, characterized by frequent updates to software libraries, which necessitate adjustments to API interfaces. This evolving landscape raises crucial challenges for LLMs, particularly their ability to generate code that is functional for different, specific library versions. This dynamic nature of software development leads us to ask the following questions:

- •
How reliably can LLMs generate code compatible with specific library versions?
- •
How effectively can LLMs adapt code for API changes across library versions?

Existing benchmarks (Jiang et al., ; Sun et al., ; Luo et al., ), which are oblivious to version-specific dynamics, do not address these challenges. They fall short of simulating the continuous version management activities undertaken by developers who ensure the software remains functional across updates. The static nature of existing benchmarks represents a significant barrier to the practical deployment of LLMs in professional environments, where handling version-specific dependencies is critical (Zhang et al., ; ; Dilhara et al., ; Liu et al., ; Wang et al., ; Vadlamani et al., ; Haryono et al., ).

To bridge this gap, we propose two novel tasks aimed at evaluating LLMs’ version-controllable code generation capabilities, namely version-specific code completion (VSCC) and version-aware code migration (VACM).
These tasks are crafted to mimic real-world software development scenarios, motivated in Figure 1, requiring models to generate code that not only is syntactically correct but also adheres to version-specific API contracts (Zhang et al., ; ; Dilhara et al., ; Liu et al., ; Wang et al., ; Vadlamani et al., ; Haryono et al., ).
Moreover, we introduce VersiCode, the first dataset specifically designed for these two tasks. VersiCode includes data spanning over 300 Python libraries and more than 2,000 versions across 9 years. It has undergone a careful curation process to ensure high quality. Thus, VersiCode provides a comprehensive and robust testbed for assessing LLMs under realistic conditions.
Furthermore, we propose a new evaluation metric, CDC (Critical Diff Check), which enhances traditional code similarity metrics by incorporating considerations for API usage, parameter handling, and deprecated features management. This metric offers a more granular assessment of a model’s ability to navigate the complexities of evolving software libraries.

Our extensive testing of strong frontier models like GPT-4o and LLaMA3 (Meta LlaMa team, ) reveals significant challenges in version-aware code generation tasks. We uncover that (1) LLMs often retain outdated programming knowledge, particularly concerning version-specific information. (2) Conventional metrics used for evaluating code generation do not effectively capture the nuances of version sensitivity. (3) While leveraging context from various library versions can be beneficial, its utility can be limited.
Guided by these insights, we suggest strategies, such as targeted pretraining, continual learning, and refined evaluation methods, for improving LLMs’ version-controlled code generation capabilities.

Figure: Figure 1: Two motivating scenarios for version-controllable code generation: (left) Interacting with LLMs in a browser, where slight query changes lead to incorrect answers, and (right) Programming in an IDE, explicitly specifying the version of dependency libraries.
Refer to caption: x2.png

Our contributions are summarized as follows:

- •
We propose two novel and important yet under-explored tasks in code generation, namely version-specific code completion and version-aware code migration.
- •
We introduce VersiCode, a comprehensive, well-documented and *versioned* dataset, accompanied by a subset annotated with executable test cases.
- •
We introduce Critical Diff Check, a new metric that extends traditional code similarity metrics by checking syntactic validity, API usage, parameter matching, the use of ‘with‘ statements, and correct keyword arguments in the generated code, providing a more detailed evaluation of version-specific code generation.
- •
Our thorough experiments provide valuable insights and directions for future research in this critical area of software development.

## 2 Version-controllable Code Generation

VersiCode is a large-scale code generation benchmark dataset focusing on evolving library dependen-
cies.
We curated our dataset by initially selecting popular Python repositories from GitHub, confirmed by their star ratings, and ensured they were permissively licensed. For each library, we compiled data from three main sources: (1) Library Source Code, extracting all pip-installable versions and official API usage examples from docstrings; (2) Downstream Application Code, sourcing from top-tier research papers spanning ten years to capture evolving libraries; (3) Stack Overflow, retrieving FAQs that mention specific library versions. We present the dataset statistics, construction process and examples in detail in Appendix 2.

As shown in Figure 2, we define a *meta-instance* as $m=[l;v;d;c]\in\mathcal{M}$, where $l$, $v$, $d$, and $c$ represent the library name, version, functionality description, and code snippet, respectively. Consider an API $a$ added to library $l$ in version $v_{s}$ and deprecated in version $v_{e}$, and is active in the intermediate version $v_{m}$ where $s\leq m\leq e$. We refer to the interval $[s,e)$ as the *lifecycle* of $a$. To analyze model performance in detail, we assess how up-to-date each LLM is concerning newly added or deprecated APIs per version. We compare the source code between any two consecutive versions of each library to detect changes in API or method names. Based on the detection results, we label the source code as follows: “addition” indicates an API newly added in the current version and still applicable in subsequent versions; “deprecation” indicates the current version is the last usable version for the API; and “general” indicates the API usage method is inherited from the previous version.

We introduce the two novel version-controllable code generation tasks below.

Version-Specific Code Completion (VSCC): Given a meta-instance $m_{i}$, the input is $x=[l_{i};v_{i};d_{i};c^{\prime}_{i}]$, where $c^{\prime}_{i}$ is the code snippet $c_{i}$ with selective masking, replacing the library- and version-sensitive contents with a special token. Depending on the length of the masked contents, the special token is defined as “[token-mask]”, “[line-mask]”, or “[block-mask]”, reflecting code completion on different granularity levels. The output $y$ is the masked content, typically containing function names or variables.

Version-Aware Code Migration (VACM): Given a pair of meta-instances $(m_{i},m_{j}|l_{i}==l_{j},d_{i}==d_{j},v_{i}\neq v_{j})$, the input $x=[l_{i};v_{i};d_{i};c_{i};v_{j}]$, and the output $y=c_{j}$. Note that version editing may require refactoring of the code structure, making it difficult to format as detailed as in token-level or line-level completion. Additionally, depending on the numerical relationship between $v_{i}$ and $v_{j}$, various scenarios arise, such as editing from an old version to a new version, or vice versa. Data statistics are detailed in Appendix B

Figure: Figure 2: The post-processing pipeline transforms metadata into specific tasks and the running example per task: (left) Leveraging pairs of metadata that share the same functionality but different library versions to construct block-level code migration instances; (right) Utilizing each metadata sample, masking version-sensitive content to create multi-granularity code completion instances.
Refer to caption: x3.png

## 3 Token-level Version-specific Code Completion

In code generation that targets a specific version of a third-party library, the version-related changes usually involve updates to identifiers, such as the addition, removal, or renaming of classes, functions, and parameters.
The token-level code completion task for a specified version, predicting the evolving identifiers identified in real code, is a fundamental and direct way to evaluate LLMs to generate code for specific versions.
We begin our research by addressing the following three research questions:
(1) How well do LLMs perform on code completion tasks that involve version-specific library usage compared to other benchmarks like HumanEval and MBPP?
(2) How do LLMs handle new, deprecated, and intermediate versions of libraries in code completion tasks?
(3) How does the performance of LLMs in code completion change over time with different library versions?

### 3.1 Experiment Setup

Models:
We benchmarked VersiCode against popular open-domain LLMs and dedicated code-LLMs, including variant families such as GPT (OpenAI, ; ; ), LLaMa (Touvron et al., ), Mistral (Jiang et al., ), CodeLLaMa (Rozière et al., ), CodeQwen (Bai et al., ), CodeGemma (CodeGemma Team et al., ), StarCoder (Lozhkov et al., ), Deepseek-Coder (Guo et al., ), and WizardCoder (Luo et al., ). For smaller open-source models (e.g., <20B parameters), we downloaded them from HuggingFace (^1^11[https://huggingface.co/models](https://huggingface.co/models)) and deployed them locally for inference. For larger models, such as LLaMa3 70B (Meta LlaMa team, ) and GPT-4o (OpenAI, ), we used their online APIs (^2^22[https://together.ai](https://together.ai)) (^3^33[https://openai.com/](https://openai.com/)) for inference.

Data Preparation:
Each instance in VersiCode is tagged with its data source (library source code, downstream applications, or Stack Overflow), feature type (addition, deprecation, or general), and release time, allowing for more detailed performance analysis. We randomly selected 2,000 instances for token-level code completion. (see Appendix A.3).

Baseline Dataset: To assess the difficulty of VersiCode, we compared it with two well-known code generation datasets, HumanEval (Liu et al., ) and MBPP (Jiang et al., ), and observed the overall performance of models. HumanEval (Liu et al., ) measures functional correctness in synthesizing programs from docstrings with 164 original problems, resembling simple software interview questions. MBPP (Austin et al., ), with about 1,000 crowd-sourced Python problems for entry-level programmers, covers programming fundamentals and standard library functionality, including task descriptions, code solutions, and three automated test cases for each problem. We also collected the evaluation results for their upgraded versions HumanEval+ (Liu et al., ) and MBPP+ (Liu et al., ). Please refer to Appendix D.1 for details.

Evaluation Metrics:
We use EM@$k$ for token-level generation: For this metric, we generate $n\geq k$ samples per instance (with $n=100$ and $k\in\{1,3,10\}$ for our experiments). We count the number of correct samples $c\leq n$ judged by exact matching. @$k$ is defined as the average performance over the task, calculated as $\mathbb{E}\left[1-\frac{{\binom{n-c}{k}}}{\binom{n}{k}}\right]$, which is the same with Pass@$k$ (Chen et al., ).

### 3.2 Results and Analysis

Figure: Figure 3: The *EM@1* results for token-level code completion from VersiCode: (a1) Comparison with existing benchmark datasets, (a2) Performance grouped by data sources, and (b) Performance grouped by API lifecycle.
Refer to caption: x4.png

Even token-level code completion is challenging.
We present the EM@1 results for token-level code completion on VersiCode, sorted by release time (highlighted in green, see Figure 3-a1).
Compared to the Pass@1 results on HumanEval (blue) and MBPP (orange), all models perform significantly worse on VersiCode (green). This result indicates the difficulty in disambiguating and recalling version-specific library usage. It is important to note that larger and more recent models, such as GPT-4o (M13) and LLaMA3-70B (M12), demonstrate significantly superior performance compared to other models.
(See Appendix H for the error analysis of GPT-4o.)
However, a substantial performance gap of at least 15 points remains when compared to HumanEval and MBPP (detailed in Appendix D.1). This indicates that state-of-the-art LLMs still struggle to deliver satisfactory results, even for the simplest token-level completion tasks.

Differences in LLM performance across different data sources.
Figure 3-a2 presents the EM@1 results for token-level code completion on VersiCode, categorized by data sources.
Among the three data sources, most models perform significantly better on Stack Overflow, especially compared to handling source code from downstream applications. This discrepancy may be attributed to the greater diversity found in downstream applications, which demands a more robust capability to address varied challenges. This may also indicate that Stack Overflow is heavily represented in the pre-training data of LLMs, increasing the likelihood of data leakage. Similar to Figure 3-a1, GPT-4o (M13) and LLaMA3-70B (M12) stand out as outliers, excelling in handling downstream applications, which may increase the likelihood of models memorizing specific content. Full numeric results are provided in Appendix D.1.

Challenges in casual intermediate library versions.
We present the token-level EM@1 results for the token-level code completion task, categorized by lifespan features: addition (in blue), deprecation (in orange), and general (referring to intermediate versions; in green), as shown in Figure 3-b. Most models perform well in cases of addition and deprecation, likely because newly added or deprecated APIs are often emphasized in documentation and by the community.
However, most models struggle with reasoning and adapting to intermediate versions.
As shown in Figure 3-a2, models like LLaMA3-70B excel in downstream applications and handle intermediate versions more effectively, likely due to the diversity of use cases they encounter.

Figure: Figure 4: The *EM@1* performance for token-level code completion, grouped by year (2015-2023), with a histogram of data distribution for each year.
Refer to caption: x5.png

The programming knowledge of LLMs, particularly regarding version-specific information, is surprisingly outdated.
Figure 4 presents the EM@1 performance for token-level code completion, grouped by year from 2015 to 2023, along with a histogram showing the data distribution for each year.
To ensure precise timestamps and minimize noise, we only used instances collected from library source code.
As shown in Figure 4-a, there is a clear trend: model performance declines as the release time becomes more recent.
This is counter-intuitive compared to temporal knowledge question answering (Zhao et al., ), where performance initially increases before declining. We further filtered for “deprecation” (Figure 4-b) and “addition” (Figure 4-c) to identify version-sensitive cases.
Although data sparsity reduces confidence in the results, both cases show a clear downward trend over time
This suggests that LLMs have outdated programming knowledge, highlighting the need for rapid adaptation to newer libraries and APIs.

## 4 From Token-level to Line- and Block-level Completion

When utilizing third-party code library APIs, LLMs should handle not only API name generation but also parameter preparation and contextual code integration. In this section, we extend the task to line-level (completing a single line) and block-level (completing multiple lines) code generation. This expanded scope presents new challenges for both the model’s capabilities and the evaluation methodologies.
(1) How does increasing complexity in line- or block-level code completion affect the LLMs to handle API usage and parameters?
(2) How does having more context (like import statements and specified library version) improve the accuracy of line- and block-level code generation?
(3) Which evaluation metrics best capture the accuracy of line- and block-level code generation, and which is most reliable?

### 4.1 Experiment Setup

Models:
We selected GPT-4o, GPT-3.5, and LLaMA3 70B, the three models that perform best on token-level code completion, to conduct experiments on line-level or block-level code completion.

Data Preparation:
We sample a subset from VersiCode for dynamic code analysis with executable test cases from library source code, focusing on code snippets with complete context (e.g., import statements). GPT-4 was used to refactor the snippets into task functions, followed by test case generation and validation in a version-specific environment. All of the test cases have been manually verified to ensure their correctness. The code completion tasks are categorized into token, line, and block levels. The test cases include return type, normal input, boundary values, and functionality checks (see Appendix A.3 for details).

Figure: Figure 5: The process of executable code assessment, which includes data refactoring, test case generation, and validation. Starting from code snippets collected from real code involving specific API calls for a given library version, GPT-4 is employed to refactor the code into a task function. The large language model is then prompted to generate test cases from various perspectives (See Appendix F for a running example of instances and test cases.) Each generated test case is verified by experts, and the correctness is ensured by running the code in a specified environment. If issues arise, they are corrected through multiple iterations with GPT-4.
Refer to caption: x6.png

Metrics:
We use the following evaluation metrics for each task granularity:
(1) Pass@$k$ for token-level generation (Chen et al., ):
For this metric, we generate $n\geq k$ samples per instance (with $n=6$ and $k=1$ to compare different metrics). We count the number of correct samples $c\leq n$ judged by executable testing.
(2) Identifier Sequence Match (ISM@$k$) and Prefix Match (PM@$k$) for line-level generation (Agrawal et al., ): These metrics measure how closely the generated sequences match the ground truth. For block-level generation, we adopt the average performance over lines. Following the setup in Agrawal et al. (Agrawal et al., ), we generate $n=6$ independent samples per instance.
(3) Exact Match (EM@$k$): We use regular expression matching to determine whether the specified API is used in the code generated by the model and the formula for calculating the EM@k score is the same as the formula for calculating the Pass@k score ($n=6$ and $k=1$).
(4) Critical Diff Check (CDC@$k$): Unlike traditional code similarity calculations, CDC focuses on the differences between the code generated by the model and the reference answer. CDC extends the EM metric by adding four additional rules: checking whether the generated code is syntactically valid; identifying the line in the generated code where the specified API is used and determining if the number of parameters in the function call is the same; if the answer uses a with statement, checking whether the generated code also uses a with statement; and if the answer uses keyword arguments, verifying whether the generated code uses the same keyword arguments. Please refer to Appendix E for detailed examples, effectiveness analysis, and ablation study, conducted to validate CDC.

### 4.2 Results and Analysis

Figure: Figure 6: The *Pass@1* performance of different models across various granularities and test case types.
Refer to caption: x7.png

Table: Table 1: The performance of different models across various granularities *(Token, Line, Block)*. *Pass@1* refers to dynamic analysis metrics, while green-colored metrics *(EM, ISM, PM)* correspond to static analysis based on string matching. The blue-colored metric *(CDC)* represents a newly proposed metric. The configurations labeled as *“w/o version”* indicate that the prompt does not specify the version of the third-party code libraries, while *“w/o import”* refers to prompts where the provided code context lacks import statements, meaning the model must generate code based entirely on user intent. The Pearson correlation coefficient is computed for each metric’s results against Pass@1 within each granularity.

Less context leads to more errors in code generation.
When models have more context, like import statements, their performance improves significantly. For example, as shown in Table 1, GPT-4o at the token level achieves a Pass@1 score of 65.97 with imports, but this drops to 44.54 without imports. This pattern is consistent across all models and granularity levels (i.e., token, line, and block), as shown in Figure 6. When models lack important context, such as external libraries or other dependencies, they struggle to generate accurate code, which leads to more errors. So, giving models more information upfront is crucial for better results.

Models show limited sensitivity to version-specified instructions. As shown in Table 1, at the token level, models like GPT-4o perform slightly better when provided with version information (52.80 with version v.s. 49.72 without version). However, this advantage diminishes at the line and block levels, where the results become inconsistent. This suggests that while version details can be helpful for short code snippets, they don’t significantly impact the model’s performance for more extended or complex code. This likely indicates that models are not trained to prioritize or heavily rely on version-specific instructions.

The CDC@1 metric closely aligns with Pass@1 scores, making it a strong proxy for dynamic code analysis. As shown in Table 1, at the block level, the Pearson Correlation Coefficient (PCC) between CDC@1 and Pass@1 is 0.9995, indicating a strong correlation. Even though EM@1 has a high correlation with Pass@1 at the token level (PCC = 0.9995), EM@1 becomes less aligned at the block level (PCC = 0.8974). Additionally, the absolute differences between CDC@1 and Pass@1 values are generally smaller compared to other static metrics like EM@1, making CDC a potentially more reliable alternative for assessing code generation accuracy.

## 5 From Code Completion to Code Migration

In addition to generating code for specific third-party library versions, another common challenge is maintaining user projects when these libraries are upgraded or rolled back.
We address version-aware code migration by exploring three key questions:
(1) How well can LLMs handle migrating code across different versions, compared to generating code for a specific version?
(2) What impact do major and minor version changes in third-party libraries have on code migration?
(3) How do forward migrations (from older to newer versions) compare to reverse migrations (from newer to older versions) in terms of trends and challenges?

### 5.1 Experiment Setup

Models: Based on the token-level code completion experimental results in Section 3, we selected the most outstanding performers from each model series for the experiments in this section.

Data Preparation: For code migration, we utilize a subset of VersiCode, in which instances are constructed based on differences between source and target code versions, covering both updates to newer versions and downgrades. Versions were categorized by patterns (e.g., major vs. minor) to capture different migration scenarios. (Detailed in Appendix A.3)

Metrics: Code migration is similar to block-level tasks in code completion. We use the same evaluation metric as for block-level: CDC@$k$ ($n=6$, $k\in\{1,3\}$).

### 5.2 Results and Analysis

Table: Table 2: The performance of various models in different code migration scenarios. The arrow ”$\mapsto$” indicates the direction of migration, where ”Major” corresponds to major version changes (e.g., Torch 2.0.0), and ”Minor” corresponds to minor version changes (e.g., Torch 2.1.3). The ”Old $\mapsto$ New” scenario simulates upgrading from an old version to a new version, while ”New $\mapsto$ Old” represents the maintenance of historical code. The performance of different models in these scenarios is measured using the CDC metrics (CDC@1 and CDC@3), reflecting their adaptability to various code migration tasks.

Model performance across version migrations.
Different models display varying degrees of adaptability when transitioning between major and minor software versions, with some showing exceptional robustness in Table 2. The table categorizes version migrations into four types: Major-to-Major, Major-to-Minor, Minor-to-Major, and Minor-to-Minor. Notably, models like GPT-4o excel in major-to-major migrations, suggesting superior handling of significant changes, whereas transitions involving minor versions tend to show moderate performance. This variability underscores the models’ design focus, whether on broad adaptability or specialized functionality.

Adaptability in code migration based on release timing.
Backward and forward compatibility testing reveals a spectrum of model resilience under different temporal migration scenarios. The evaluation is split into two releasing time directions: Old-to-New and New-to-Old, shown in Table 2. Generally, models perform better when adapting to newer versions from older ones, with GPT-4o standing out for its high scores in both directions. However, the drop in performance when handling older versions after training on newer releases highlights challenges in maintaining backward compatibility, a critical aspect for long-term usability and integration stability in evolving tech environments.

The context code in another version is still helpful, but its benefits are limited.
The comparison between block-level code completion and block-level code migration is shown in Table 2 and Table 1. There is a significant improvement across most models, except for LLaMA3-70B and GPT-4o, detailed in Appendix D.3. When provided with code in another version as context (i.e. in the code migration task), these models can generate correct code with a much higher success rate. However, a bottleneck is evident in LLaMA3-70B and GPT-4o, where the code context hinders their performance than code completion.

## 6 Discussion

How can we enhance pre-training for new code-LLMs?
Figure 4 demonstrates a notable decline in the performance of all models over time. This deterioration is likely attributable to two primary factors: (1) the use of outdated pre-training data, which causes older versions of code to predominate the training set, and (2) the backward compatibility of APIs, which results in a higher prevalence of use cases and examples about older versions of these APIs (Lamothe et al., ). To mitigate this issue and improve the models’ capabilities with newer libraries, we suggest increasing the representation of new-version codebases within the training data. This adjustment aims to enhance the proficiency in utilizing contemporary libraries effectively (Zhao et al., ; Shao et al., ).
Besides, based on the results in Section 4.2, current LLMs show limited use of version information in code generation. To address this, we propose enhancing pre-training by incorporating version-tagged code samples and metadata to help models better differentiate between API versions.

How can we address the challenge of evolving libraries in LLMs?
Generating block-level or repository-level code (Luo et al., ) requires LLMs to understand user demands and library dependencies. Addressing this challenge involves continually training the model with new libraries using continual learning techniques (Jiang et al., ). These techniques enable the model to adapt to changing libraries without forgetting previously learned information. Examples include memory-based methods and various continual learning strategies (Wu et al., ; Yadav et al., ; Wu et al., ). Additionally, developing benchmark datasets that are continuously and automatically curated and maintained is crucial for evaluating the performance of models with new libraries (Jang et al., ). Enriching the taxonomy (Jiao et al., ) and maintaining datasets for evolving libraries (Lamothe et al., ) is also vital (Jiao et al., ). Multi-agent systems can be employed for this purpose.
Aligning development and evaluation efforts will enhance the ability of LLMs in code understanding and generation capabilities, to remain effective as libraries evolve.

Can we address version-controllable code generation with retrieval-augmented generation?
Retrieval-augmented generation (RAG) approaches typically involve two crucial components: retrieval and in-context generation (Gao et al., ). The following challenges need to be addressed in order for RAG to be effectively applied to this problem.
From the retrieval perspective:
(1) It may be difficult to disambiguate version-related queries, as embeddings for version strings like “torch 2.1.3” and “torch 1.3.2” can be very similar (Singh & Strouse, ). This similarity makes it hard for retrievers to differentiate between specific features and capabilities associated with each version.
( 2) Version information of code snippets is rarely explicitly mentioned within the code itself and may instead appear in separate configuration files like “requirements.txt”. This separation necessitates a more sophisticated retrieval approach, where the model must integrate information from multiple sources to accurately understand version dependencies.
From the perspective of in-context generation:
Table 9 shows that even non-matching version contexts (i.e., code migration) can help smaller models generate grammatically correct code.
This observation suggests potential for dedicated RAG approaches (Jiang et al., ), though the benefits are limited and retrieval noise may reduce effectiveness.

What are the effective methods for evaluating the capabilities of LLMs in generating version-controllable code?
Both static analysis (Agrawal et al., ), which reviews code without executing it, and dynamic analysis (Zhuo et al., ), which tests the code by running it, are vital for software development.
However, evaluating LLMs for version-controllable code generation presents unique challenges.
(1) Dynamic analysis is complicated by API calls that rely on specific code contexts, making it difficult and costly to create standalone tests (Zhuo et al., ).
Additionally, using LLM-generated code as test cases introduces further complexity in managing test quality.
Especially, VersiCode, which includes 300 packages and over 2,000 versions in the raw dataset, requires detailed setups for each testing environment and managing various dependencies, complicating the practical deployment of solutions.
(2) Meanwhile, static analysis uses metrics like ISM (Agrawal et al., ) and PM (Agrawal et al., ) for broad coverage but may miss critical details such as indentation and parameter positioning in API-related code, refer to Table 1 and Appendix E.
These omissions suggest that traditional static metrics are not entirely suitable for assessing version-controllable code generation.
Evaluating the effectiveness of these metrics is crucial.
Our study initiates the exploration of more reliable methods; however, extensive research, including approaches like code slicing (Du et al., ), is essential to advance our evaluation techniques.

## 7 Related Work

Code Generation Models:
Recent advancements in code language models (Guo et al., ; CodeGemma Team et al., ; Bai et al., ; Rozière et al., ; Sun et al., ), driven by sophisticated NLP techniques (Jiang et al., ) and extensive code repositories (Hu et al., ), have resulted in substantial breakthroughs. Transformer-based large language models (Luo et al., ; Rozière et al., ; Guo et al., ; Lozhkov et al., ; Bai et al., ; Gunasekar et al., ; Li et al., ) have demonstrated exceptional capabilities in generating syntactically correct and semantically meaningful code from natural language descriptions. Additionally, research efforts that integrate multi-modal data (OpenAI, ; ; Meta LlaMa team, ), including both code and accompanying documentation (Hu et al., ), have significantly improved model accuracy. While in real-world software engineering,

Code Generation Datasets:
The code generation (Jiang et al., ; Sun et al., ; Luo et al., ) includes tasks for both code completion and code editing, ensuring comprehensive coverage of programming scenarios.
Code completion (Yao et al., ; Yin et al., ; Feng et al., ; Chen et al., ; Austin et al., ; Hendrycks et al., ; Lu et al., ; Li et al., ; Fried et al., ; Liu et al., ; Lai et al., ; Yu et al., ; Fu et al., ; Zheng et al., ) is the task of predicting subsequent code tokens based on the given context, benefits from datasets, which provide extensive code repositories from various programming languages. These datasets enable models to learn syntactic and semantic patterns (Jiao et al., ).
Code editing (Just et al., ; Lin et al., ; Zhu et al., ; ; Hu et al., ; Yan et al., ; Ahmad et al., ; Jiao et al., ; Zhang et al., ; Tian et al., ) involves automatically generating changes to existing code, such as bug fixes or refactoring. Datasets like EvalGPTFix (Zhang et al., ) and DebugBench (Tian et al., ), which focus on bug fixing and code refinement tasks, are instrumental in this area.
To our knowledge, given the necessity and challenges in library evolution (Jiang et al., ), refer to the detailed comparison in Table 6 and Appendix C, the proposed dataset VersiCode is the first large-scale code generation dataset, covering both code completion and code editing. Refer to Appendix C for a comprehensive comparison among datasets.

Third-party Library Evolution: Third-party library code is continually updated due to bug fixes, code refactoring, and the addition of new features, making it a significant research topic in software engineering (Zhang et al., ; ; Dilhara et al., ; Liu et al., ; Wang et al., ; Vadlamani et al., ; Haryono et al., ).
Studies by Zhang et al. () show that Python APIs often evolve by adding, deleting, or modifying parameters. Further research by Zhang et al. () notes frequent API changes, including parameter updates.
Dilhara et al. () reveal that developers adjust their use of machine learning libraries in response to updates, while Liu et al. () and Dig & Johnson () find that undocumented changes in Android and Java can cause errors.
Research on API deprecation highlights issues with documentation and the quality of suggested alternatives (Wang et al., ; Vadlamani et al., ; Haryono et al., ; Brito et al., ), showing that improvement in library evolution does not necessarily translate to better suggestions for deprecated APIs.
VersiCode, unlike traditional software engineering research, studies API version evolution from an LLM perspective, exploring its impact on model training, code generation, and evaluation.

## 8 Conclusion

In conclusion, our research underscores the need for updated benchmarks that capture the dynamic nature of software development, better assessing the capabilities of LLMs in code generation. By introducing the VersiCode dataset, we provide a realistic testing ground that reveals significant limitations in current models, like GPT-4o and LLaMA3, when handling version-specific code. Our findings advocate for continuous model improvements and the adoption of our new metric, i.e., critical diff check, which more accurately evaluates model performance against real-world challenges. This work not only introduces valuable tools but also sets a direction for future enhancements in AI-driven code generation, ensuring LLMs remain effective and relevant in professional settings.
For future research, we will investigate a solution for version-controllable code generation based on the insights from this paper, including approaches like continual learning, memory-enhanced methods, or retrieval-based methods. Additionally, we plan to develop a live version of VersiCode, which will continuously incorporate new libraries and downstream use cases.

## Appendix A Dataset Construction

VersiCode is a large-scale code generation benchmark dataset focusing on evolving library dependencies. We propose two tasks to simulate real-world applications: version-specific code completion and version-aware code migration, incorporating version information into code generation constraints.
First, we discuss data curation, and preprocessing of noisy code snippets and FAQs into organized metadata. Based on the metadata, we describe the task design and quality control process. We then address tagging API lifespan features per library version. Finally, we provide data statistics for VersiCode and discuss future dataset extensions.

### A.1 Dataset Curation and Collection

As shown in Figure 7, we first collected permissively licensed Python repositories from GitHub that serve as the source code for Python libraries. These repositories are ranked by their popularity (as indicated by their collected stars). Using the list of popular libraries, we gathered data from three sources for each library:
(1) Library Source Code: We collected all available versions of the library source code from GitHub, verifying with PyPI to ensure that the collected versions are formally released and can be installed via pip.
From the library source code, we extracted official usage examples for each API from the docstrings.
(2) Downstream Application Code: Given Python’s popularity in scientific programming, we collected the source code from top-tier research papers over 10 years as downstream applications. These applications are valuable due to being lightweight yet self-consistent, diverse in their topics, and tagged release timelines associated with publishing venues. Given the time span, this data source implicitly includes evolving libraries.
(3) Stack Overflow: Using the library names as queries, we collected FAQ data from Stack Overflow, which provides real user queries and diverse user answers. We filtered the data to include only those queries that explicitly mention the versions of the libraries used, using heuristic rules, as shown in Table 3. Additionally,
we have made our best efforts to filter all of the source code based on the open-source licenses of the repositories to ensure there is no infringement.

**Table 3: Detailed explanation of annotation stages and the corresponding filtering rules.**
| Procedure | Rules |
| --- | --- |
| Ranked Libraries $\mapsto$ StackOverflow Q&A | Filter out answers that involve the use of libraries from the ranked libraries,<br>and ensure these answers include content in the library version format (e.g., pandas==1.3.5) as well as code snippets. |
| Ranked Libraries $\mapsto$ Library Source Code | Based on the ranked libraries, parse the source code of these libraries to find functions related to version changes. |
| Ranked Libraries $\mapsto$ Downstream Application | (1)Exclude files that do not utilize libraries and version information explicitly listed in requirements.txt.<br>(2)Exclude files with an average line length exceeding 100 characters.<br>(3)Exclude files with a maximum line length exceeding 1000 characters.<br>(4)Exclude files with less than 25% of alphabetic characters.<br>(5)Exclude files with syntax errors. |
| Annotation $\mapsto$ Metadata | StackOverflow: Filter out data that has been annotated by experts with correct library version and code snippet,<br>and utilize GPT to generate functionality descriptions for the code snippets. |
| Library Source Code: Utilize GPT to extract examples from version change function docstrings,<br>filter out successfully extracted data, and employ GPT to generate functionality descriptions for the examples. |  |
| Downstream Application: Utilize GPT to generate functionality descriptions for code snippets. |  |

Given the high diversity and varied quality of the collected raw data, we adopted a hybrid annotation approach involving both human experts and LLMs, such as ChatGPT. (1) Library Source Code: The library version is concrete and explicitly available, but example usage varies across libraries and versions. We used an LLM with in-context learning to help extract example code from docstrings, preparing the library version and code snippets. (2) Downstream Applications: The version can easily be extracted from configuration files, typically named “requirements.txt”. We carefully filtered out Python files that are too long, do not mention the library version, or fail to compile. (3) Stack Overflow: Given the diversity of the questions, we designed strict heuristic rules to preliminarily annotate the library name, version, and corresponding Python code snippets mentioned in answers. We then distributed the pre-annotated data to six qualified human experts for verification and correction, ensuring the library version and code snippets are ready as well.
With all pairs of library versions and code snippets, we employed ChatGPT with in-context learning to generate descriptions of the functionality for each code snippet. Each pair is wrapped in well-organized metadata.

Figure: Figure 7: The preprocessing pipeline to obtain metadata, structured as n-gram tuple of $\langle$library name, version, functionality description, code snippet$\rangle$.
Refer to caption: x10.png

### A.2 Lifecycle Tagging of APIs

Consider an API $a$ added to the library $L$ in version $V_{s}$ and deprecated in version $V_{e}$, and is active in the intermediate version $V_{m}$ where $s\leq m\leq e$. We refer to the interval $[s,e)$ as the *lifespan* of $a$. To analyze model performance in detail, we assessed how up-to-date each LLM was concerning newly added or deprecated APIs per version. We compared the source code between any two consecutive versions of each library to detect changes in API or method names. Based on the detection results, we labeled the datasets obtained from the library source code as follows: “addition” indicates an API newly added in the current version and still applicable in subsequent versions; “deprecation” indicates the current version is the last usable version for the API; and “general” indicates the API usage method is inherited from the previous version.

### A.3 Data Preparation for Evaluation

Data Preparation for Token-level Code Completion. As introduced in Section 2, we designed two types of version-controllable code generation tasks: version-specific code completion and version-aware code migration. The task granularities are categorized into token-level, line-level, and block-level to control difficulty and simulate different application scenarios. To better understand model performance, each instance in VersiCode is also tagged with the following:
(1) Data source, which includes library source code, downstream applications, and Stack Overflow;
(2) Feature type, including addition, deprecation, and general;
(3) Release time, i.e. the timestamp from GitHub and Stack Overflow);
These tags allow us to filter the evaluation dataset and gain sharper insights into model performance.

Data Preparation for Execution-based Multi-granularity Code Completion.
As shown in Figure 5, we have constructed a subset for dynamic code analysis that includes executable test cases. From the data originating from library source code in VersiCode, we filter for data that includes complete context (e.g., import statements) code snippets. Experts interact with the web version of GPT-4 to refactor the code snippets into task functions. After a manual check of the task functions, experts interact with GPT-4 to write test cases for them. During the interaction, experts provide appropriate feedback to GPT-4. The test cases are run in a testing environment containing specific library versions (e.g., pandas==1.3.5); if successful, the annotation is completed after further manual verification, and if failed, more detailed feedback is provided to GPT-4 to assist with corrections. The annotated task function is processed into code completion forms with three levels of $\langle$mask$\rangle$ granularity: token, line, and block.
The executable test cases include four types:
(1) Test return type: tests whether the return type is correct.
(2) Test normal input: tests whether the expected output is produced with normal inputs.
(3) Test boundary values: tests whether special values (such as null values, incorrect types, etc.) are handled properly.
(4) Test functionality: tests whether the function fulfills its primary functionality. The first three types of test cases have one instance per task function, while the fourth type has 1-3 instances.

Data Preparation for Code Migration. As shown in Figure 2, considering code migration instances constructed from pairs of metadata, the differences between source and target code versions result in various situations, such as updates from an older version to a newer version or vice versa. Additionally, we categorized versions according to version patterns, for example, treating torch v1.0.0 as a major version and torch v1.3.1 as a minor version, to identify combinations of major and minor version migration cases.

## Appendix B Data Statistics and Scope

Dataset Statistics: We present the statistics of VersiCode in Table 4, using the StarCoder2’s (Lozhkov et al., ) tokenizer to compute the number of tokens. We also outline the complete version of VersiCode in the table, which furnishes human-labeled data for three additional languages: C#, Java, and JavaScript. Our executable data, applied in Section 4, is a high-quality human-annotated subset from VersiCode, covering 12 libraries, 40 versions, and 119 functionality descriptions. For each functionality description, we matched 4 to 5 test cases.

**Table 4: Data statistics of VersiCode, including multiple languages.**
| # Language | Python | Java | C# | JavaScript |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| # Data Source | StackOverflow; Library Source Code; Downstream Application | StackOverflow | StackOverflow | StackOverflow |  |  |  |  |
| # Num. of Libraries | 300 | 19 | 16 | 33 |  |  |  |  |
| # Num. of Versions | 2,207 | 25 | 16 | 60 |  |  |  |  |
| # Size of Meta Data | 11,268 | 29 | 16 | 62 |  |  |  |  |
| # Task Type | Completion | Editing (old to new) | Editing (new to old) | Completion | Completion | Completion |  |  |
| # Granularity | Token | Line | Block | Block | Block | Block | Block | Block |
| # Avg. Input Token | 2,087 | 2,075 | 55 | 191 | 195 | 57 | 63 | 67 |
| # Avg. Output Token | 2 | 16 | 128 | 131 | 128 | 349 | 255 | 167 |
| # Num. of Instances | 13,488 | 13,490 | 1,617 | 38,037 | 38,037 | 32 | 21 | 82 |

Scope: VersiCode supports version-specific code completion at the token, line, and block levels, enabling developers to navigate through version variations effortlessly. It also facilitates block-level version-aware code editing, empowering users to make precise modifications tailored to requirements of each version.
The collected metadata also serves as a valuable resource for potential customized task modifications, supported domains are illustrated in Figure 8, aiding in fine-tuning workflows and enhancing model training for optimal performance.

Figure: Figure 8: A proportional chart based on the classification system of targeted audience and topics in third-party Python libraries on PyPI.
Refer to caption: x11.png

## Appendix C Related Dataset

Code Completion Datasets.
As shown in Table 5, we compare the VersiCode-completion dataset with existing benchmarks. VersiCode stands out in annotated data size, marking it as the inaugural dataset tailored for version-specific generation.

**Table 5: Comparison of VersiCode and other code completion datasets. VersiCode is the largest annotated dataset, covering multiple languages and granularities, and involving both human and LLM joint annotations.**
| Benchmark | Source | Language | Samples | Completion Task | Granularity | Collection Time | Annotation |
| --- | --- | --- | --- | --- | --- | --- | --- |
| StaQC (Yao et al., ) | StackOverflow | Python, SQL | 267,056 | Function Programming | Line-Level, Block-Level | 2018 | None |
| CoNaLa (Yin et al., ) | StackOverflow | Python, Java | 2,879 | Function Programming | Line-Level, Block-Level | 2018 | Human |
| CT-maxmin (Feng et al., ) | Existing Benchmark | Multi(=6) | 2,615 | Cloze Test | Token-Level | 2020 | None |
| HumanEval (Chen et al., ) | Hand-Written | Python | 164 | Function Programming | Line-Level, Block-Level | 2021 | Human |
| MBPP (Austin et al., ) | Hand-Written | Python | 974 | Function Programming | Block-Level | 2021 | Human |
| APPS (Hendrycks et al., ) | Programming Sites | Python | 10,000 | Function Programming | Line-Level, Block-Level | 2021 | None |
| CT-all (Lu et al., ) | Existing Benchmark | Multi(=6) | 176,115 | Cloze Test | Token-Level | 2021 | None |
| CodeContests (Li et al., ) | Existing Benchmark, Codeforces | Multi(=3) | 13,610 | Function Programming | Block-Level | 2022 | None |
| HumanEval-FIM (Fried et al., ) | Existing Benchmark | Python | 164 | Function Programming | Line-Level, Block-Level | 2022 | None |
| HumanEval+ (Liu et al., ) | Existing Benchmark | Python | 164 | Function Programming | Line-Level, Block-Level | 2023 | LLM |
| MBPP+ (Liu et al., ) | Existing Benchmark | Python | 378 | Function Programming | Block-Level | 2023 | LLM |
| DS-1000 (Lai et al., ) | StackOverflow | Python | 1,000 | Function Programming | Line-Level, Block-Level | 2023 | Human |
| CoderEval (Yu et al., ) | Github | Python, Java | 460 | Function Programming | Block-Level | 2023 | Human |
| CodeApex (Fu et al., ) | Programming Sites | C++ | 476 | Function Programming | Block-Level | 2023 | None |
| HumanEval-X (Zheng et al., ) | Existing Benchmark | Multi(=5) | 820 | Function Programming | Line-Level, Block-Level | 2023 | Human |
| BigCodeBench (Zhuo et al., ) | Existing Benchmark | Python | 1,140 | Function Programming | Block-Level | 2024 | Human, LLM |
| VersiCode | StackOverflow, Github | Python, Java, C#, JavaScript | 28,595 | Cloze Test, Function Programming | Token-Level, Line-Level, Block-Level | 2024 | Human, LLM |

Code Migration Datasets.
As shown in Table 6, we compare the VersiCode-migration dataset with existing benchmarks. VersiCode stands out in annotated data size, marking it the inaugural dataset tailored for version-specific migration.

**Table 6: Comparison between VersiCode and other code editing datasets, with VersiCode standing out as the largest annotated dataset specifically tailored for version adaptation.**
| Benchmark | Source | Language | Samples | Editing Task | Granularity | Collection Time | Annotation |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Defects4J (Just et al., ) | Open Source Programs | Java | 357 | Debug | Block-Level | 2014 | None |
| QuixBugs (Lin et al., ) | Quixey Challenges | Python, Java | 40 | Debug | Line-Level | 2017 | Human |
| CoST (Zhu et al., ) | GeeksForGeeks | Multi(=7) | 132,046 | Code Translation | Line-Level, Block-Level | 2022 | None |
| XLCoST (Zhu et al., ) | GeeksForGeeks | Multi(=8) | 1,083,000 | Code Translation | Line-Level, Block-Level | 2022 | None |
| InstructCoder (Hu et al., ) | Github | Python | 114,000 | Code Refinement | Block-Level | 2023 | LLM |
| MultilingualTrans (Yan et al., ) | Programming Sites | Multi(=8) | 30,419 | Code Translation | Block-Level | 2023 | None |
| NicheTrans (Yan et al., ) | Programming Sites | Multi(¿8) | 236,468 | Code Translation | Block-Level | 2023 | None |
| LLMTrans (Yan et al., ) | Hand-Written | Multi(=8) | 350 | Code Translation | Block-Level | 2023 | Human |
| Avatar (Ahmad et al., ) | Programming Sites | Python, Java | 62,520 | Code Translation | Block-Level | 2023 | None |
| G-TransEval (Jiao et al., ) | Existing benchmark, GeeksForGeeks | Multi(=5) | 400 | Code Translation | Token-Level, Block-Level | 2023 | Human |
| EvalGPTFix (Zhang et al., ) | AtCoder | Java | 151 | Debug | Block-Level | 2023 | Human |
| DebugBench (Tian et al., ) | LeetCode | Multi(=3) | 4,253 | Debug | Block-Level | 2024 | LLM |
| VersiCode | Github | Python | 76,074 | Version Adaptation | Block-Level | 2024 | LLM |

## Appendix D Additional Experiments and Details

### D.1 Extensive Comparative Study on Large Language Models

In addition to the model depicted in Figure 3, comprehensive and detailed evaluation results are presented in Table 7, encompassing 23 models and sorted by the release time of each model.

**Table 7: Full evaluation results of EM@1 on token-level code completion compared to related datasets and different data sources. The results for related datasets are collected from the online leaderboard of Evalplus (Liu et al., ).**
| Release Time | Model | HumanEval | HumanEval+ | MBPP | MBPP+ | VersiCode |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| EM@1 | EM@1 | EM@1 | EM@1 | Library Source Code | Downstream Application | StackOverflow | Total |  |  |
| 2023.06.14 | WizardCoder-15B-V1.0 (Luo et al., ) | 56.7 | 50.6 | 64.3 | 54.2 | 0.17 | 0 | 0.1 | 0.06 |
| 2023.06.14 | WizardCoder-Python-7B-V1.0 (Luo et al., ) | 50.6 | 45.1 | 58.5 | 49.5 | 6.62 | 0.17 | 5.45 | 2.66 |
| 2023.07.18 | Llama-2-7B (Touvron et al., ) | 12.8 | - | 20.8 | - | 6.57 | 0.46 | 4.76 | 2.74 |
| 2023.07.18 | Llama-2-13B-Chat (Touvron et al., ) | 18.3 | - | 30.6 | - | 3.71 | 0.06 | 3.41 | 1.51 |
| 2023.08.25 | CodeLlama-7B-Instruct (Rozière et al., ) | 34.8 | - | 44.4 | - | 17.77 | 0.62 | 17.8 | 7.62 |
| 2023.08.25 | CodeLlama-13B-Instruct (Rozière et al., ) | 42.7 | - | 49.4 | - | 28.45 | 2.47 | 32.05 | 13.5 |
| 2023.08.28 | CodeLlama-7B-Python (Rozière et al., ) | 38.4 | - | 47.6 | - | 3.4 | 0.03 | 2.35 | 1.28 |
| 2023.10.29 | DeepSeek-Coder-6.7B-Instruct (Guo et al., ) | 74.4 | 71.3 | 74.9 | 65.6 | 3.83 | 0.15 | 4.34 | 1.71 |
| 2023.11.11 | Mistral-7B-Instruct-V0.2 (Jiang et al., ) | 42.1 | 36 | 44.7 | 37 | 13.96 | 1.85 | 20.33 | 7.54 |
| 2024.01.25 | DeepSeek-Coder-7B-Instruct-V1.5 (Guo et al., ) | 75.6 | 71.3 | 75.2 | 62.2 | 26.7 | 4.51 | 44.77 | 15.71 |
| 2024.01.25 | GPT-3.5-Turbo (OpenAI, ) | 76.8 | 70.7 | 82.5 | 69.7 | 40.55 | 30.48 | 65.95 | 37.59 |
| 2024.02.27 | StarCoder2-7B (Lozhkov et al., ) | 35.4 | 29.9 | 55.4 | 45.6 | 12.21 | 0.32 | 13.02 | 5.27 |
| 2024.02.27 | StarCoder2-15B (Lozhkov et al., ) | 46.3 | 37.8 | 66.2 | 53.1 | 29.7 | 2.9 | 35.79 | 14.55 |
| 2024.04.09 | CodeGemma-7B-Instruct (CodeGemma Team et al., ) | 60.4 | 51.8 | 70.4 | 56.9 | 31.8 | 0.76 | 31.29 | 13.36 |
| 2024.04.09 | CodeGemma-7B (CodeGemma Team et al., ) | 44.5 | 41.5 | 65.1 | 52.4 | 29.61 | 1.12 | 34.01 | 13.28 |
| 2024.04.10 | aiXCoder-7B (aiXcoder team, ) | 54.9 | - | 66 | - | 17.51 | 1.09 | 26.3 | 8.83 |
| 2024.04.15 | aiXCoder-7B-Base (aiXcoder team, ) | 43.2 | - | 62.2 | - | 20.41 | 0.94 | 26.37 | 9.59 |
| 2024.04.15 | CodeQwen1.5-7B (Bai et al., ) | 51.8 | 45.7 | 73.5 | 60.8 | 11.61 | 0.12 | 7.58 | 4.33 |
| 2024.04.15 | CodeQwen1.5-7B-Chat (Bai et al., ) | 83.5 | 78.7 | 79.4 | 69 | 12.16 | 0.33 | 9.2 | 4.81 |
| 2024.04.18 | Llama-3-8B (Meta LlaMa team, ) | 35.5 | 29.3 | 61.4 | 51.6 | 17.18 | 0.24 | 20.69 | 7.57 |
| 2024.04.18 | Llama-3-8B-Instruct (Meta LlaMa team, ) | 61.6 | 56.7 | 70.1 | 59.3 | 20.79 | 3.67 | 34.08 | 12.23 |
| 2024.04.18 | Llama-3-70B-Chat (Meta LlaMa team, ) | 77.4 | 72 | 82.3 | 69 | 33.76 | 50.93 | 64.35 | 47.55 |
| 2024.05.13 | GPT-4o (OpenAI, ) | 85.4 | 81.7 | 85.7 | 73.3 | 58.37 | 72.98 | 87.21 | 70.44 |

### D.2 Multi-language Analysis

**Table 8: Multi-language performance on VersiCode**
| Model | Python | Java | C# | JavaScript |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ISM@1 | PM@1 | ISM@1 | PM@1 | ISM@1 | PM@1 | ISM@1 | PM@1 |  |
| DeepSeek-Coder-7B-Instruct-V1.5 (Guo et al., ) | 40.03 | 27.35 | 61.55 | 46.62 | 71.43 | 49.68 | 75.22 | 54.24 |
| CodeLlama-13B-Instruct (Rozière et al., ) | 48.83 | 34.63 | 70.92 | 58.87 | 47.62 | 35.54 | 52.87 | 34.11 |
| StarCoder2-15B (Lozhkov et al., ) | 39.71 | 27.36 | 38.63 | 27.43 | 33.33 | 28.63 | 60.67 | 39.33 |
| CodeGemma-7B (CodeGemma Team et al., ) | 8.67 | 5.00 | 34.38 | 23.53 | 0 | 0 | 16.82 | 10.53 |
| GPT-3.5-Turbo (OpenAI, ) | 40.77 | 28.06 | 50.00 | 39.34 | 28.57 | 26.87 | 24.39 | 15.85 |
| GPT-4o (OpenAI, ) | 64.72 | 50.48 | 70.83 | 64.04 | 71.43 | 63.26 | 77.74 | 70.24 |
| Llama-3-70B-Chat (Meta LlaMa team, ) | 57.68 | 41.47 | 61.55 | 58.57 | 66.67 | 56.35 | 75.61 | 67.61 |

As depicted in Table 8, we perform the primary multi-language experiments. Counter-intuitively, the performance of LLMs in Java, JavaScript, and C# surpasses that in Python. This anomaly might be attributed to potential data leakage from the Stack Overflow dataset.

### D.3 Block-level Generation without Grammar Verification

We use Python’s built-in function *“compile()”* to compile the generated code snippets in order to check whether they are syntactically correct. Upon comparing “w/o grammar verification” and “w grammar verification” in Table 9, it becomes evident that the model tasked with editing, alongside reference code snippets from other versions, finds it easier to produce grammar-verified code.

Table: Table 9: Results of block-level code completion and migration with or without grammar verification.

## Appendix E Metric Design of Critical Diff Check

### E.1 Introduction of Critical Diff Check

Critical Diff Check (CDC) focuses on the changes in the code rather than the overall similarity of the entire code segment. CDC has five rules as follows:

- •
*Rule 1: Check whether the generated code contains the core token.*
- •
*Rule 2: Check whether the generated code is valid.*
- •
*Rule 3: Check if the number of arguments in the function using the core token is consistent.*
- •
*Rule 4: If the reference code
uses a with statement, checks whether the generated code also uses a with statement.*
- •
*Rule 5: If the reference
code uses keyword argument assignment, checks whether the generated code uses the same keyword argument
assignment.*

The failure frequency and examples for each rule are shown in Table 10.

Table: Table 10: Each rule of the CDC, along with the frequency, occurrence rate, and examples of mismatches for each rule. ‘a’ represents the core token, ‘c’ represents the code generated by the model, ‘c^′’ represents the reference code, ‘f’ represents the function of the specified token, and ‘params’ refers to the function’s parameter list. ‘Kc^′(f)’ and ‘Kc(f)’ represent the keyword parameter lists of the reference code and the model-generated code, respectively, and ‘p’ represents the parameter assigned using keyword arguments. In detail, *Rule 1* checks whether the generated code contains the core token; *Rule 2* checks whether the generated code is valid; *Rule 3* checks if the number of arguments in the function using the core token is consistent; *Rule 4*, if the reference code uses a with statement, checks whether the generated code also uses a with statement; *Rule 5*, if the reference code uses keyword argument assignment, checks whether the generated code uses the same keyword argument assignment.

### E.2 Ablation Study of Critical Diff Check

We conducted ablation experiments on the five CDC rules and calculated the Pearson correlation coefficient with the Pass@1 metric for each, to demonstrate the reliability of CDC. The specific experimental data is shown in Table 11.

Table: Table 11: Ablation study of Critical Diff Check per rule. The configuration labeled as “CDC w/o Rule i”, where $i\in\{1,2,3,4,5\}$ means that when calculating the CDC score, Rule i is excluded, and only the other four rules are considered. The Pearson correlation coefficient calculates the correlation the metric’s results obtained in each configuration against Pass@1.

## Appendix F Running Example of Executable Test

As shown in Figure 9, this is an example of a task function used for code generation, where the task function is processed in various granular forms of code completion. The “core token” is only provided for visualization, which is unseen for models. “library version” is optional, identified as “w/ or w/o version”, and “import” statements are also optional, identified as “w/ or w/o import” in Table 1. As shown in Figure 10, these are the test cases for the task function illustrated in Figure 9. The test cases were developed by experts through interactions with GPT-4 and include four types of tests.

Figure: Figure 9: The ground truth for block-level code generation, used for Section 4. Note that, “core token” is only provided for visualization, which is unseen for models. “library version” is optional, identified as “w/ or w/o version”, and “import” statements are also optional, identified as “w/ or w/o import” in Table 1.
Refer to caption: x15.png

Figure: Figure 10: The test cases associated with generated code for dynamic code analysis, used for Section 4.
Refer to caption: x16.png

## Appendix G Evaluation Details

### G.1 Hyper-parameter

As illustrated in Table 12, we have itemized the hyper-parameters pertinent to version-controllable code generation.

**Table 12: Hyper-parameters for completion and migration.**
| hyper-parameter | code completion | code migration |  |  |
| --- | --- | --- | --- | --- |
| token-level | line-level | block-level | block-level |  |
| temperature | 0.8 | 0.8 | 0.8 | 0.8 |
| top_p | 0.95 | 0.95 | 0.95 | 0.95 |
| max_tokens | 64 | 128 | 512 | 512 |
| n | 100 | 6 | 6 | 6 |

### G.2 Prompt Template

We introduce the prompt template for token-level, line-level, and block-level evaluations in Figure 11, Figure 12, and Figure 13, respectively.

Figure: Figure 11: Prompt template for token-level version-specific code completion.
Refer to caption: x17.png

Figure: Figure 12: Prompt template for line-level version-specific code completion.
Refer to caption: x18.png

Figure: Figure 13: Prompt template for block-level version-specific code completion.
Refer to caption: x19.png

Figure: Figure 14: Prompt template for version-aware code migration.
Refer to caption: x20.png

### G.3 Data Sampling

For token-level completion tasks(Figure 3), we randomly sampled 2,000 instances for evaluation. We used the entire executable dataset for line- and block-level completion tasks due to its smaller size (Figure 6, Table 1). In the time trend experiment (Figure 4), we sampled 200 data points per quarter or used all available data if fewer. And in the code migration task (Table 2), we randomly sampled 2,000 instances for evaluation.

## Appendix H Error Analysis

### H.1 Error Analysis of GPT4-o

Despite GPT4-o achieving superior performance in general evaluation, it still encounters errors in 30% of instances. We provide several negative examples in Figure 15, Figure 16, and Figure 17.

Figure: Figure 15: The first negative example of GPT-4o on token-level code completion.
Refer to caption: x21.png

Figure: Figure 16: The second negative example of GPT-4o on token-level code completion.
Refer to caption: x22.png

Figure: Figure 17: The third negative example of GPT-4o on token-level code completion.
Refer to caption: x23.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Your work provides valuable advancements for version-aware code generation. However, handling complex API changes—such as alterations in method behaviors, parameter types, or the introduction of new functionalities—can significantly impact code compatibility and functionality. How do you plan to address these more intricate API modifications in future iterations of VersiCode and CDC? 

Additionally, what strategies are you considering to ensure that VersiCode remains up-to-date and scalable as libraries continue to evolve rapidly across different programming languages and ecosystems?
soundness: 3
strengths: - The paper addresses a real-world software development challenge, which is largely ignored by LLM4Code literature, where frequent updates to libraries and APIs require code to be compatible with specific versions.

- It introduces two new tasks focused on code evolution: code completion and version-aware code migration. These tasks are common in real-world development, and the paper provides detailed, separate approaches for each.

- The paper presents VersiCode, a large, high-quality dataset that includes data from over 300 Python libraries across over 2,000 versions over nine years.

- The new CDC metric improves on standard code similarity checks by including essential details like API usage, parameter handling, and managing deprecated features.
summary: Despite current LLMs' remarkable advances in code generation tasks, they are typically trained and evaluated using static benchmarks such as HumanEval and MBPP. However, real-world software development is dynamic, with frequent library updates and API changes that require version-specific code compatibility. Current benchmarks do not address these challenges, highlighting a gap between LLM evaluations and practical development needs. To bridge this gap, the paper proposes two new tasks: (i) version-specific code completion (VSCC) and version-aware code migration (VACM)—and introduce VersiCode, a comprehensive dataset covering over 300 Python libraries and 2,000 versions. Additionally, the authors present a new metric, Critical Diff Check (CDC),  to better assess LLMs' ability to handle version-specific code generation.
weaknesses: - While dealing with API evolution, the paper did not deal with more complex changes involving updates of API parameters or behaviors over time.  

- The proposed VersiCode dataset requires regular updates to remain relevant, and handling thousands of library versions could become difficult as libraries continue to evolve, which may limit long-term usefulness.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: See above.
soundness: 3
strengths: - Strong Insight. Capturing version dynamics is essential for effectively handling version-specific dependencies in practical software development.
- Comprehensive dataset. VersiCode covers 300 libraries over nine years, offering extensive coverage for version-specific testing and paving the way for future research in version-specific code generation and automated code migration.
- Novel metric. The CDC@k metric employs a set of hand-crafted rules to assess the similarity between API usages across different versions, enabling a more detailed, execution-free evaluation of version-specific code generation.
summary: The paper introduces VersiCode, a benchmark aimed at addressing the challenges faced by LLMs in generating version-specific code generation. It proposes two tasks: version-specific code completion (VSCC) and version-aware code migration (VACM). These tasks simulate real-world scenarios where deprecated API invocations must be updated to a newer version. The study includes a novel metric, Critical Diff Check (CDC@1), to access the code generation's adherence to version-specific requirements.
weaknesses: - Lack of in-depth analysis. As noted in subsection 4.2, the performance of GPT-4o drops significantly without import statements, suggesting that the evaluation is heavily influenced by prompt design and provided context. A more thorough analysis on how context information, such as documentation related to added or deprecated APIs, affects performance would be beneficial.
- Misleading analysis. In subsection 5.2, the claim that "The context code in another version is still helpful, but its benefits are limited" is difficult to corroborate based on Table 1 and Table 2, as block-level code migration details are not sufficiently presented. Additionally, the statement, "There is a significant improvement across most models, except for LLaMA3-70B and GPT-4o, detailed in Appendix D.3," is puzzling, as Appendix D.3 focuses on grammar verification impacts. Clarification is needed to confirm if there was an error or oversight.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: The paper makes a significant contribution to understanding the performance of LLMs on version-specific code tasks, yet certain design choices would benefit from further clarification. The following questions aim to probe areas where additional context could strengthen the work:

1. Could the authors elaborate on the rationale behind the specific criteria used for sampling instances in model evaluations?

2. Did the team consider incorporating any RAG-based methods in future work to enhance the model’s access to relevant version documentation?

3. Are there plans to extend VersiCode to support other programming languages, given the diverse needs in software development?
soundness: 4
strengths: Major Strengths

1. Comprehensive Dataset and Benchmark for Version-Specific Tasks: VersiCode is an important contribution as it directly addresses the under-explored area of version-specific code generation. By including metadata such as function descriptions, code snippets, and version numbers, it enables realistic evaluations of LLM capabilities in scenarios that require adherence to specific library versions. As shown in Figure 2, VersiCode’s metadata is utilized to create multi-granularity completion tasks, demonstrating the paper’s methodological depth in representing version-sensitive contexts.

2. Introduction of the CDC Metric: The Critical Diff Check (CDC) metric provides a more detailed assessment of LLM performance by focusing on crucial factors like API usage and parameter handling. This is particularly useful for evaluating tasks where syntax alone does not suffice. Table 1 presents a comparison of CDC and other metrics across different test cases, demonstrating its close alignment with the Pass@1 score, which highlights CDC’s effectiveness as a reliable proxy for assessing version-specific correctness.

3. Insightful Evaluation on Diverse Models and Detailed Error Analysis: The authors evaluate a wide range of models, including GPT-4o and StarCoder, revealing notable performance differences across version migration scenarios. For instance, in Table 2, models are evaluated across major and minor version changes, highlighting the difficulties that LLMs face in backward compatibility. This comprehensive analysis provides valuable insights into model limitations and suggests paths for improvement.

Minor Strengths

1. Detailed Task Decomposition with Multi-Level Completion: The dataset’s structure supports token, line, and block-level completion, making VersiCode versatile and relevant for different development use cases. Figure 3 compares model performance on token-level code completion across various data sources, showcasing the dataset’s capability to test LLM adaptability to different levels of code.

2. Rigorous Evaluation Methodology: The experimental setup is clearly defined, with the authors selecting 2,000 test instances for token-level completion (Section 3.1). This careful setup enhances the credibility of the results, providing a consistent baseline against well-known datasets like HumanEval[1] and MBPP[2].

[1] Chen M, Tworek J, Jun H, et al. Evaluating large language models trained on code[J]. arXiv preprint arXiv:2107.03374, 2021.

[2] Austin J, Odena A, Nye M, et al. Program synthesis with large language models[J]. arXiv preprint arXiv:2108.07732, 2021.
summary: The VersiCode paper addresses the crucial gap in current LLM capabilities by focusing on version-controllable code generation, a requirement in real-world software development due to frequent updates in third-party libraries. The authors introduce two novel tasks, version-specific code completion (VSCC) and version-aware code migration (VACM), to evaluate models on handling evolving software dependencies. A new dataset, VersiCode, consisting of over 300 Python libraries and 2,000 versions, enables comprehensive testing. They also propose the Critical Diff Check (CDC) metric, which enhances traditional metrics by evaluating version-specific aspects such as API usage and parameter handling. Through testing on models like GPT-4 and LLaMA, the study reveals significant challenges in version-sensitive code generation, underscoring the need for targeted pretraining, continual learning, and refined evaluation methods to improve LLM adaptability to evolving software versions.
weaknesses: Major Weaknesses

1. Lack of Implementation of Retrieval-Augmented Generation (RAG): Although the authors acknowledge the potential of RAG techniques, there is no exploration of its integration, which could have significantly enhanced LLM performance in this context. RAG could assist in real-time access to documentation or version-specific information, potentially improving accuracy on challenging cases (see Section 5.2 for model limitations in handling context). Incorporating recent works in RAG-based methodologies for code tasks, such as those explored in [1], could inspire potential enhancements.

2. Absence of Evaluation Metrics for High-Level Reasoning: While CDC is effective for syntactic and structural validation, it doesn’t capture the semantic reasoning required to fully assess code functionality in context. Including metrics that evaluate semantic coherence or logical consistency could provide a more rounded view of LLM limitations, similar to the approaches suggested in studies like [2] which focus on code understanding and error detection.

Minor Weaknesses

1. Potential Bias in Model Evaluation Sampling: Although VersiCode’s dataset is extensive, the paper does not clarify how test samples were selected, which could impact the generalizability of findings. Describing the selection process, such as prioritizing recent or frequently updated APIs, would improve the robustness of conclusions (Section 2).

2. Limited Generalization to Languages Beyond Python: VersiCode primarily targets Python libraries, despite the widespread use of other languages in development. Expanding the dataset to include JavaScript or Java, for example, could enhance the benchmark’s applicability across diverse development environments, as is common in industry setting.

[1] Li R, Allal L B, Zi Y, et al. Starcoder: may the source be with you![J]. arXiv preprint arXiv:2305.06161, 2023.

[2] Le H, Wang Y, Gotmare A D, et al. Coderl: Mastering code generation through pretrained models and deep reinforcement learning[J]. Advances in Neural Information Processing Systems, 2022, 35: 21314-21328.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. Is it possible to use more familiar and possibly more reliable metrics, like pass@1, to evaluate code migration results?

2. The paper uses “Exact Match” to mean the specified API is used in the generated code. This practice is inconsistent with other papers, where “exact match” means the generated code is the same as the oracle. I hope the authors can change the name of this metric to a more appropriate one.
soundness: 3
strengths: Originality: The problem of version-controllable code generation is pretty novel. Previous research in code generation ignores that the libraries and APIs used by the generated code can evolve.

Quality: The paper constructs the first versioned code generation dataset, including version-specific code completion and version-aware code migration tasks. The authors evaluate many popular LLMs on the dataset and give valuable insights about their ability to generate version-controlled code.

Clarity: The paper clearly states the problem of version-controllable code generation, including the motivation and how existing benchmarks fail to address the challenge.

Significance: Version-controllable code generation is very important in practice. The generated code is used with specific library versions, so the code generation process must consider the library versions.
summary: The authors find that code generated by LLMs needs to target specific library versions, and this problem was ignored in previous work. They propose two new tasks, version-specific code completion, and version-aware code migration, to evaluate LLMs’ ability in version-controllable code generation. They construct a Python dataset, VersiCode, to evaluate LLMs on these two tasks. They also propose a new evaluation metric, Critical Diff Check (CDC), to assess code migration results. Evaluation results show that version-controllable code generation is a significant challenge, even for extremely powerful LLMs.

Contributions:

- They propose two novel important tasks, i.e. version-specific code completion and version-aware code migration.

- They construct a versioned dataset, VersiCode.

- They introduce a new metric, Critical Diff Check.

- Their experiments provide valuable insights and directions for future research.
weaknesses: 1. The only metric used in the version-aware code migration is CDC, a metric just proposed in this paper. Although in section 4, the CDC@1 is stated to align closely with pass@1, it is hard to say this new metric can serve as the only metric in code migration.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Regarding the concerns around the (interpretation of the) results, I strongly suggest removing the comparison with HumanEval and MBPP in favor of an analysis that separates out API completion performance on methods that are (almost) never changed vs. ones that are, to better isolate the existence and magnitude of the effect of versioning on API completion specifically. On this note, please try to answer:

Is the difference in performance statistically significant when controlling for time? E.g. in Fig. 4, is there a consistent difference when comparing datapoints for the same year between APIs that don't change (ideally ones that never change, but "general" is fine too) and ones that are added/deprecated for any of these models? Reverse trends for addition vs. deprecation are interesting too.

Please address the questions in the discussion about the definition and evaluation of deprecated methods and on the other methodological concerns above.

What degree of manual validation was used to ensure that the underlying snippets are valid? How were code migration pairs constructed exactly?

Is there a need for the CDC metric compared to pass@1 if the latter can be applied to a sufficiently large set of samples already? How do you address the concerns around the metric's potential for false negatives and the relatively slim difference compared to "Exact Match" (and the concerns with that nomenclature)?

Answers to all other questions in the "Weaknesses" section would be appreciated.
soundness: 2
strengths: This work provides a dataset that fills an important gap in code evaluations specifically, measuring the effect of API changes. The dataset it contributes is quite large and contains useful data from a variety of sources. It provides some of the first large-scale empirical evidence that LLMs perform differently on certain types of APIs, including more recent releases and deprecated ones. It presents a wide range of results.
summary: This work introduces a benchmark for measuring the ability of LLMs trained on code to condition their predictions on API versions. It measures both code completion and code migration performance of a wide range of models across deprecated, newly added, and consistently available API methods across a large set of projects in a new benchmark called VersiCode. It also introduces a new accuracy metric based on static analysis. Results show that more capable LLMs generally fair worse on newly added and deprecated methods than on ones that have consistently been present, while smaller models show the reverse pattern. All models show a decrease in performance for newer APIs. Most models struggle especially with code migration across major version changes.
weaknesses: My main concerns are around the interpretation (and significance) of the results, the clarity and soundness of the methodology, and the relevance of the contributions to this venue. I'll address these in order, followed by a list of minor issues.

**Results:** the interpretation of the results involves questionable claims in a number of places. Some of these are perhaps mostly clarity issues, while a few others seem more foundational.

An example of the latter is the comparison with HumanEval and MBPP in Fig. 3, which shows, in Fig. 3.a1, that completion accuracy is far lower on VersiCode than on the aforementioned benchmarks. The text argues that this shows "the difficulty in disambiguating and recalling version-specific library usage" (L209)  and that LLMs "struggle to deliver satisfactory results" (L232). However, self-contained functions are not a good basis of comparison for examples of real API usage. It is well-established that completing real-world code, especially function calls, is harder than code in self-contained function with detailed specifications. That is not to mention the extensive leakage concerns around the baseline, benchmark programs.

This work doesn't offer a baseline of version-agnostic API completion to compare the results on version-sensitive API completion with. The closest is the comparison in Fig. 3.b, which compares the completion quality on methods that are deprecated/added vs. ones that stay the same ("general") in the particular version used in a sample's prompt. Since unchanging APIs make up more than half the total volume of API calls (Fig. 4), it stands to reason that many of these are consistent across most/all versions of their libraries, making the "general" performance a rough approximation of the performance of models on API call completion disregarding versions. The gap between these and version-sensitive API calls is considerably smaller for the large models in this subfigure, and even inverted for the smaller ones (the explanation for which, on L248, is a bit unsatisfying as it would seem to apply to large models equally). That is, version-sensitive API calls are only slightly harder to complete (for large models) than ones that stay the same.

This combined with Fig. 4's trends suggests that the "real" effect is mostly one of frequency: older APIs and ones that do not get deprecated are used more often in typical training sets, which tend to go back to GitHub's inception, and are therefore more accurately completed. Whether there is an actual difference in performance between these three types of methods is questionable. Figure 4 shows almost indistinguishable curves for each model over time between the subfigures, although separating out "general" might show a more clear contrast. This raises strong concerns around whether there is a difference at all, or whether the differences observed in Fig. 3.b are actually caused by confounds related to statistical frequency over time.

It is also not quite clear why performance on deprecated methods is measured by prompting with the version right before the method is being deprecated. At least, that is implied by L134 ("the last usable version") -- L129 contradicts itself by referring to the lifespan of a method as both $s \leq m \leq e$ and $[s, e)$, making it unclear whether version $e$ is included. If the prompt asks to complete a method that is about to be deprecated in the next version, I assume the goal is to find cases where the models might incorrectly assume that the method is already deprecated? If so, are the samples constructed so that there is a new, valid completion in the exact same context in the subsequent version? If not, the models almost "have no choice" but to complete the soon-to-be-deprecated version for snippets that use an idiomatic style that is made obsolete by the deprecation. Similarly, was there a check that the models aren't simpy using an alternative, valid API? Replacement APIs are often introduced before deprecating older versions.

There are a number of other, mostly minor, questionable claims about the performance as well, including:

L242: why does "excelling in handling downstream applications [...] increase the likelihood of models memorizing specific content"? Is the causal statement meant to be reversed, viz. models memorizing the content increases the likelihood of excelling in downstream applications? That would make more sense, but it would need some empirical evidence to back up that: the smaller models are not memorizing content, the patterns included here show up in the training data, and larger models aren't simply better at pattern matching.

L249: "However, most models struggle with reasoning and adapting to intermediate versions." -- does "intermediate" refer to added/deprecated ones here? If so, Fig. 3b shows that "most models" do relatively better on those than on unchanged ones.

L416: the description of GPT-4o's performance here seems off: its major->major performance is worse than on any other task. Relatedly, it was not clear to me what the sentence on L417/418 meant. Also, the use of "major->minor" and similar syntax is a bit confusing. Typically, we only distinguish between a major or minor version change. This paper seems to use the word "major" to refer just to the first version of a major release in order to distinguish between cases such as the first version of one major change to another vs. jumping from a given minor change within a major release to the start of the next major release. It is unclear why performance on the latter is so much higher than on major->major changes, or even on minor->minor changes, which ought to be much easier.


**Methodology:** as evidenced by some of the questions/concerns above (e.g. the lack of a solid baseline, the definition of deprecation), there are multiple concerns with the details of the approach. These often stem from a lack of clarity in the text, including in the appendix. An example of this is the definition of code migration: the example in Fig. 2 shows a case where the same description applies to two subsequent API versions and the code differs by just the API version. However, the way descriptions are generated (L863) doesn't guarantee that equivalent code snippets in successive versions would get the exact same description. Were the migration pairs collected from cases that happened to have an identical description? Was any manual validation applied to this process to capture the rate of false positives (and ideally FNs, but those would be harder to measure)?

The concern around the lack of manual validation applies to a number of other methodological decisions, such as scraping code from scientific papers. While a convenient dataset for this task, scientific code has a less than stellar reputation for quality. Was any filtering applied to confirm that this code was not using deprecated methods? Many of the results rely on the evaluation samples being unambiguous, in the sense that there is exactly one valid completion, but there is little evidence in the methodology or results that confirms that this has been guaranteed by this benchmark.

The definition of the "CDC" metric also raises some concerns. It appears that "exact match" as used in this paper does not refer to a true (semantic) match between programs, but rather to an identifier ("core token", L1025) match. The CDC metric expands on this with a few additional heuristic checks, such as comparing the number of arguments (but not their exact values) and the presence of a `with` statement. Per Tab. 11, these added heuristics have an almost imperceptible (and perhaps not statistically significant) impact on the correlation between CDC and pass@1 ranking. The main evidence for its usefulness comes from it yielding values closer to those from pass@1 than EM in Tab. 1, at the line and block level and the more noticeably higher Pearson correlation compared to EM at the block level. While that is useful, the lower values may well be in part due to the introduction of false negatives (e.g. matching keyword arguments is not necessarily a prerequisite for correctness) and there is still a considerable gap at the block level. Overall, I find it questionable to define "exact match" as something quite far from a true, semantic match, and the additional heuristics in CDC clearly leave a sizable gap with the actual target metric. I would strongly encourage the benchmark to focus on execution correctness over heuristic match, even if that shrinks the set of usable examples substantially.

**Relevance:** on a slightly less pressing note, this work seems only partially relevant to this venue. While the dataset it contributes shows useful limitations of LLMs, it does not offer proven insights into where these limitations originate and how they can be addressed. It makes a number of claims about why these limitations exist, some of which I identify as dubious above, but it does not support these claims with explorations of the models' training data or learned representations. Without a clear relation between how the models are actually built and the observed effects, any explanations feel rather speculative. That leaves the main contribution in the domain of code evaluation data, rather than representation learning. It may be more suitable for a venue dedicated to code-specific learning.

**Minor Issues:**

- L93: nit: inverted final quote
- L143: no need to use double equals
- L412: please add a mention that Tab. 2 has the results.
- L431: missing "more"?

Meta-review: This paper addresses the issue that code generated by LLMs needs to target specific library versions, a problem overlooked in previous work. While the reviewers found the topic interesting and recognized its potential, several concerns were raised. Several expert reviewers have identify that the interpretation and significance of the results need further clarification, and the methodology requires more transparency and soundness to be fully convincing. Additionally, several technical issues need to be addressed. The authors are encouraged to refine the presentation, clarify the methodology, and strengthen the technical details. With these improvements, the work could make a valuable contribution in future submissions.

Decision: Reject