Unmasking the Version-Switching Capabilities of Code Generation Models

## Abstract

Abstract The rapid evolution of software libraries presents a significant challenge for code generation models, which must adapt to frequent version updates while maintaining compatibility with previous versions. Existing code completion benchmarks often overlook this dynamic aspect, and the one that does consider it relies on static code prediction tasks without execution-based evaluation, offering a limited perspective on a model’s practical usability. To address this gap, we introduce GitChameleon , a novel, manually curated dataset comprising 116 Python code completion problems, each conditioned on specific library versions and accompanied by executable unit tests. GitChameleon is designed to rigorously assess the ability of modern large language models (LLMs) to generate version-specific code that is not only syntactically correct but also functionally accurate upon execution. Our comprehensive evaluations reveal that state-of-the-art LLMs struggle with this task; for instance, GPT-4o achieves a pass@10 of only 39.9% (43.7% when provided with error feedback), highlighting the complexity of the problem and the limitations of current models. By providing an execution-based benchmark that emphasizes the dynamic nature of code libraries, GitChameleon serves as a critical tool to advance the development of more adaptable and reliable code generation models. For facilitation for further exploration of version-conditioned code generation, we make our code repository publicly accessible at https://github.com/NizarIslah/GitChameleon .

## 1 Introduction

Figure: Figure 1: Left: Modern LLMs often struggle with generating version-accurate code, highlighting the need for benchmarks that specifically assess their ability to handle versioning. Right: Cumulative year-over-year (YoY) version releases of popular Python-based machine learning libraries show a consistent upward trend, reflecting the rapid pace of development and version updates of code libraries and packages.
Refer to caption: x1.png

Large Language Models (LLMs) have become highly popular in code completion, to the extent that they are now deployed as virtual coding assistants within popular code editors(^1^11[https://github.com/features/copilot](https://github.com/features/copilot)), enhancing the overall coding workflow.
Code, being a dynamic and constantly evolving environment, necessitates a continuous process of adaptation to stay in sync with the rapidly shifting paradigms, frameworks, and methodologies within the software development domain. The inherent variability in coding styles, the emergence of new programming languages, and the continuous evolution of libraries and packages underscore the imperative for an active approach in updating code generation models.

In response to the needs of practical coding environments, several large language models (LLMs) have been introduced, including StarCoder (Li et al., ), DeepSeek-Coder (Guo et al., ), CodeLlama (Rozière et al., ), among others. Despite these advancements, existing LLMs often struggle to keep pace with the rapid changes in codebases, particularly when tasked with generating version-specific code that is both syntactically and functionally accurate. This issue is especially critical, as developers increasingly depend on AI-assisted coding tools to boost productivity and maintain code quality. A recent Stack Overflow survey revealed that 70% of the participants are using or planning to integrate AI coding tools, 33% citing increased productivity as the primary motivation to incorporate these tools into their workflows(^2^22[https://stackoverflow.co/labs/developer-sentiment-ai-ml/](https://stackoverflow.co/labs/developer-sentiment-ai-ml/)).

Given the rapid development and release cycles of popular libraries, as shown in Figure 1 (right), the need for code generation models to continually adapt to changing API’s is more pressing than ever. For example, prominent machine learning and deep learning libraries like PyTorch (Paszke et al., ), NumPy (Harris et al., ), and Scikit-Learn (Buitinck et al., ) undergo frequent updates, which is reflected in a consistent upward trend in user downloads and version releases. This dynamic nature of code requires models that can adapt and generate code that adheres to the latest versions and practices, a need that current models often fail to meet comprehensively.
In addition, certain hardware is restricted to compatibility with specific versions of commonly used packages, which adds an additional layer of complexity beyond merely updating the knowledge base of a code LLM to the latest library versions.

In response to these challenges, our work introduces a novel benchmark designed to assess the ability of LLMs to generate version-specific code. We propose GitChameleon, a benchmark that evaluates state-of-the-art code models by requiring them to produce executable code based on version-specific prompts. Then, this code is executed to verify its correctness against expected outputs. By highlighting the limitations of current models in generating accurate version-specific code, GitChameleon provides a structured approach to enhance these models and ensure their practical utility in real-world coding environments.

In summary, our contributions are highlighted as follows: 1) we introduce a novel code completion benchmark GitChameleon consisting of 116 Python-based version conditioning problems, including human written unit tests; 2) we systematically analyze the version-specific performance of state-of-the-art code generation LLMs on API change type, version release year, and specific libraries. 3) we demonstrate the effectiveness of utilizing error log feedback as a way to improve version conditioning performance of code generation LLMs.

## 2 GitChameleon Benchmark

We introduce GitChameleon, a benchmark that comprises 116 Python-based version conditioning problems focused on popular code libraries. To evaluate LLM performance on GitChameleon, each problem is accompanied by handwritten assertion-based unit tests, enabling a thorough execution-based assessment of the outputs generated by the code LLMs. This structured approach enables a thorough understanding and categorization of LLM failures in common scenarios involving version-specific code generation problems.
In the following sections, we detail the benchmark statistics, data collection methodology, and sample verification process.

### 2.1 Statistics

GitChameleon consists of 116 python-based version conditioned problems based on 11 libraries: PyTorch (Paszke et al., ), Geopandas (Jordahl et al., ), NLTK (Bird & Loper, ), NetworkX (Hagberg et al., ), GeoPy(^3^33[https://pypi.org/project/geopy/](https://pypi.org/project/geopy/)), Gradio (Abid et al., ), Scikit-Learn (Buitinck et al., ), Matplotlib (Hunter, ), PyCaret(^4^44[https://pycaret.org/](https://pycaret.org/)), Pandas (pandas development team, ; Wes McKinney, ) and NumPy (Harris et al., ). The samples were collected from version releases over a period from the year 2014 to 2023 and exclude legacy and yanked version releases.

**Table 1: Compared to other popular code generation benchmarks, including those evaluating version conditioning, GitChameleon features library- and version-specific problems with unit tests based on real version changes, closely aligning with practical settings.**
| Dataset | Problems | Data Source | Library Specific | Version Specific | Execution based | Real |
| --- | --- | --- | --- | --- | --- | --- |
| HumanEval (Chen et al., ) | 164 | Hand-Written | ✗ | ✗ | ✓ | - |
| MBPP (Austin et al., ) | 974 | Hand-Written | ✗ | ✗ | ✓ | - |
| MTPB (Nijkamp et al., ) | 115 | Hand-Written | ✗ | ✗ | ✓ | - |
| APPS (Hendrycks et al., ) | 10000 | Competitions | ✗ | ✗ | ✓ | - |
| CodeContests (Li et al., ) | 117 | Competitions | ✗ | ✗ | ✓ | - |
| JulCe (Agashe et al., ) | 1518049 | Notebooks | ✗ | ✗ | ✓ | - |
| DSP (Chandel et al., ) | 1119 | Notebooks | ✓ | ✗ | ✓ | - |
| CoNaLa (Yin et al., ) | 2879 | StackOverflow | ✓ | ✗ | ✗ | - |
| DS-1000 (Lai et al., ) | 1000 | StackOverflow | ✓ | ✗ | ✓ | - |
| BigCodeBench (Zhuo et al., ) | 1140 | Expert-Curated | ✓ | ✗ | ✓ | - |
| Versicode (Wu et al., ) | 98692 | GitHub, StackOverflow | ✓ | ✓ | ✗ | ✓ |
| CodeUpdateArena (Liu et al., ) | 670 | LLM-Generated | ✓ | ✓ | ✓ | ✗ |
| (Wang et al., ) | 28125 | API change logs | ✓ | ✓ | ✗ | ✓ |
| GitChameleon (Ours) | 116 | Handwritten and LLM-assisted | ✓ | ✓ | ✓ | ✓ |

Using the cl100k_base tokenizer, we analyzed the token counts of the GitChameleon samples. The problem statements average 20.4 tokens, and the starter code averages 47.4 tokens, leading to a combined average of 67.8 tokens per sample. Including the prompt template utilized for evaluating instruction-tuned LLMs, the total token count across all samples is 19,409 tokens.

Figure: (a) Number of unique versions per library in GitChameleon.
Refer to caption: x3.png

As demonstrated in Fig. 2(b), most of the samples in GitChameleon are from versions of libraries released in the years 2021, 2022 and 2023, with 2021 released version samples accounting for 35% of the total sample count in the benchmark. Since some of the models evaluated on GitChameleon have disclosed their training data cutoff dates, we have ensured that most, if not all, samples fall within the training window of these models.
This approach helps to ensure that the models during their training period are likely exposed to the versions on which the samples are based. Fig. 2(a) shows that NetworkX (Hagberg et al., ) and Gradio (Abid et al., ) have the most versions in our benchmark (8 and 7, respectively). Meanwhile, PyTorch (Paszke et al., ) and NumPy (Harris et al., ) have the highest number of samples (18 and 15, respectively), together accounting for 34% of the total sample count. In addition, we annotate each sample with the type of change that is classified into the following categories:

- •
Argument or Attribute change: The API call to a function, method, or class has a change in arguments (e.g. name, order, new, deprecated argument) between versions.
- •
Function Name change: The name of the API call has changed between versions (e.g. pandas.append to pandas.concat).
- •
Semantics or Function Behavior change: The semantic / runtime behavior of the API call changed between versions (e.g. returning a different type).
- •
New feature or additional dependency-based change: A feature was introduced in a specific version; therefore, to execute the same functionality, a model using an older version should make use of an additional dependency (e.g. torch.special was introduced in torch 1.10, previously one could use numpy for the same).

Most samples in the GitChameleon benchmark fall under the Argument or Attribute and Function Name change category, as these are the most frequent and expected types of changes in mature and stable libraries.

#### Differentiating factor

Several datasets examine LLM interactions with version-specific code, including Versicode (Wu et al., ), CodeUpdateArena (Liu et al., ), and the dataset by (Wang et al., ). While these datasets are valuable, our dataset offers a unique and complementary perspective by focusing on the real-world scenario where developers are often constrained to specific library versions due to technical debt. CodeUpdateArena investigates model adaptation to synthetic API changes, we focus our evaluation on real API changes to assess how effectively an LLM can generate code for version-specific changes of library versions that they have been trained with. In contrast, Versicode and Wang et al. ()’s datasets, while addressing library evolution, primarily rely on string matching for evaluation. Our approach diverges by incorporating executable tests, providing a more practical and rigorous assessment of code generation capabilities.

### 2.2 Collection framework

**Table 2: Example of a problem statement derived from a changelog entry from Numpy 1.25.0**
| Task Description | Expected Result |
| --- | --- |
| [⬇](data:text/plain;base64,ICAgICMgV3JpdGUgYSBmdW5jdGlvbiB0aGF0IGNoZWNrcyBpZiBhbGwgZWxlbWVudHMgaW4gYW4gYXJyYXkgYXJlIHRydWUu)<br># Write a function that checks if all elements in an array are true. | [⬇](data:text/plain;base64,ICAgIG5wLmFsbChhcnIp)<br>np.all(arr) |
| Starter Code | Assertion Test |
| [⬇](data:text/plain;base64,ICAgIGltcG9ydCBudW1weSBhcyBucAogICAgZGVmIGFsbHRydWVfZm4oYXJyKToKICAgICAgICByZXR1cm4=)<br>import numpy as np<br>def alltrue_fn(arr):<br>return | [⬇](data:text/plain;base64,ICAgIGFyciA9IG5wLmFycmF5KFsxLCAxLCAxLCAxXSkKICAgIHJlc3VsdCA9IGFsbHRydWVfZm4oYXJyKQogICAgYXNzZXJ0IHJlc3VsdCA9PSBucC5hbGwoYXJyKQ==)<br>arr = np.array([1, 1, 1, 1])<br>result = alltrue_fn(arr)<br>assert result == np.all(arr) |

The examples were manually crafted by the authors, who divided the task among themselves. We compiled a list of popular Python libraries, focusing on those with which at least one author was familiar and that had detailed changelogs documenting changes between versions. For each library, we reviewed the changelogs to identify deprecated functions, argument changes, alterations in behavior, and newly introduced functions.

For each identified change, we create a concise problem statement, write the starter code, define the expected solution, and develop an assertion test. For instance, in Table 2, we illustrate an example based on the changelog for version 1.25.0 of NumPy (Harris et al., ), a library for scientific computing in Python. This changelog notes that “np.alltrue is deprecated. Use np.all instead.” We used this change to craft a problem statement that tests the LLMs’ ability to recognize and adapt to version-specific updates.

#### Unit test and evaluation framework verification

To assess the correctness of the evaluation framework of GitChameleon, we needed to verify three key aspects:

- •
Compilation: Ensure that the starter code compiles successfully.
- •
Assertion unit tests: Confirm that the assertion tests function correctly.
- •
Dependencies: Verify that all necessary external dependencies are installed, excluding the ones being tested.

We used venv to create and manage virtual environments for testing. This process involved installing the appropriate library version and any additional dependencies. We then combined the starter code, expected result, and the assertion test into a single script, which was executed to verify all three criteria. We provide pseudocode for our verification process in section A.3.

## 3 Empirical Study

We evaluate state-of-the-art large language models (LLMs) using the GitChameleon benchmark to assess their ability to generate version-specific, executable code. This study highlights how well current models adapt to dynamic library versions and produce functionally correct code that passes the provided unit tests.

### 3.1 Experimental setup

For each open-source LLM, we downloaded the corresponding Hugging Face (HF) weights and served the models using Text Generation Inference (TGI) for non-instruct models and VLLM (Kwon et al., ) for instruction tuned models.
We used one NVIDIA 95GB H100 GPU for models with fewer than 70 billion parameters, two GPUs for models more than 70 billion parameters.

We configured the generation parameters with a top_p value of 0.95, top_k of 50, and a temperature of 0.3 for Pass@1 and 0.8 for Pass@10.
The maximum number of new tokens generated was set to 256.
Additionally, we enabled flash attention (Dao et al., ) for all models to enhance inference efficiency.

### 3.2 Evaluation Metrics

To comprehensively evaluate the performance of code generation models using the GitChameleon dataset, we employ a range of execution-based metrics. These metrics assess not only the correctness of the generated code, but also its efficiency and adaptability to different versions.

Pass@k measures the proportion of problems for which at least one of the top k generated solutions passes all assertion tests. This metric provides insight into the model’s ability to generate functionally correct code. For each problem, we generate n code samples, and compute the pass at k metric by the corrected formula:

For instruct models, we run the model’s parsed output as standalone code, and for base models, the concatenation of the starting code and model’s parsed output (completion).

Greedy refers to the standard greedy decoding method, where the most probable token from the next-token distribution is chosen deterministically. This is analogous to setting the temperature to 0.

Error Feedback adds the error log to the prompt (after executing the generated code from the model with the initial prompt). Then, the pass@k metric is recalculated based on the model’s generated code using the prompt with error feedback. See section A.2 for an example.

doc-prompting is a method in which the documentation of the library function (specific to the version that the model is asked to use in the prompt) is given in the prompt. See Appendix A.1 for an example.

### 3.3 Main Results

We report the performance of both base and instruct-tuned models on the GitChameleon benchmark in Tables 3 and 4, respectively. Our analysis reveals a positive correlation between model size and performance in version-specific code generation tasks. For base models, Spearman’s rank correlation coefficients are 0.82 for Pass@1 and 0.69 for Pass@10 (both p-values $<$0.01), indicating that larger models generally perform better. Specifically, DeepSeek-Coder 33B achieved the highest Pass@1 score of 35.7%, highlighting its proficiency in generating correct solutions on the first attempt, while CodeLlama 34B outperformed others at Pass@10 with a score of 42.8%, demonstrating its ability to produce correct solutions given multiple attempts.

**Table 3: Base Model Performance Metrics. Deepseek-coder-33B is the strongest model for Pass@1 (temperature 0.3), while CodeLlama-34B is the strongest model when we compute Pass@10 with an increased number of generations (20) sampled at temperature 0.8. We observe that there is a strong positive correlation between model size and performance, with Spearman’s rank correlation coefficients of 0.82 for Pass@1 and 0.69 for Pass@10.**
| Model | Size | Pass@1 | Pass@10 |
| --- | --- | --- | --- |
|  |  | T=0.3 | T=0.8 |
| CodeLlama-Python (Rozière et al., ) | 7B | 20.4$\pm$1.6 | 36.1$\pm$5.5 |
| 13B | 25.8$\pm$1.0 | 36.4$\pm$2.0 |  |
| 34B | 30.6$\pm$1.4 | 42.8$\pm$1.4 |  |
| Starcoder2 (Lozhkov et al., ) | 3B | 11.9$\pm$1.9 | 27.1$\pm$1.9 |
| 7B | 15.5$\pm$1.1 | 23.1$\pm$2.6 |  |
| 15B | 13.7$\pm$1.7 | 27.0$\pm$3.4 |  |
| Llama-3 (Dubey et al., ) | 8B | 22.3$\pm$1.0 | 32.0$\pm$2.1 |
| 70B | 27.2$\pm$3.0 | 41.3$\pm$2.5 |  |
| Qwen2 (Yang et al., ) | 7B | 27.4$\pm$1.2 | 37.7$\pm$1.8 |
| 72B | 33.2$\pm$2.1 | 39.7$\pm$5.5 |  |
| Starcoderbase (Li et al., ) | 1B | 13.3$\pm$1.0 | 20.3$\pm$1.2 |
| 3B | 15.5$\pm$1.2 | 26.5$\pm$1.5 |  |
| 7B | 20.0$\pm$0.9 | 31.3$\pm$4.1 |  |
| 15B | 16.9$\pm$1.8 | 30.8$\pm$2.6 |  |
| Starcoder (Li et al., ) | 15B | 16.0$\pm$1.2 | 35.9$\pm$1.9 |
| Deepseek-coder (Guo et al., ) | 1.3B | 22.0$\pm$2.5 | 28.0$\pm$1.9 |
| 6.7B | 31.0$\pm$1.8 | 36.1$\pm$0.7 |  |
| 33B | 35.7$\pm$3.0 | 37.9$\pm$4.9 |  |

Similarly, for instruct-tuned models, we observe Spearman’s rank correlation coefficients of 0.45 for Pass@1 and 0.42 for Pass@10 (both with p-values under 1%), confirming the positive correlation between model size and performance. Phi-3.5-MoE (16×3.8B) achieved the highest baseline Pass@1 score of 30.9% (33.6% greedy) and highest baseline Pass@10 (40.5%) for OSS models. Overall, Gemini achieves the highest baseline pass@10 (41.4%). Yi-1.5-Chat (34B) slightly outperformed GPT-4o and Gemini at Pass@10 with error feedback with a score of 44.1% compared to 43.7% and 43.8%, respectively. Incorporating error feedback led to average improvements of 5.4% in Pass@1 and 4.7% in Pass@10 across instruct-tuned models. Additionally, Pass@10 with n=20 samples showed an average performance improvements of 10.6% for base models and 14.8% for instruct-tuned models over Pass@1 with n=5. These findings highlight that scaling model size, utilizing error feedback, and allowing multiple solution attempts are effective strategies for enhancing the ability of LLMs in handling version-specific code generation tasks.

**Table 4: Instruct Model performance metrics. (Top) OSS models, (bottom) closed-sourced models. We observe a 5.4% and 4.7% improvement with error feedback in Pass@1 and Pass@10, respectively. Additionally, there is a positive correlation between model size and performance, with Spearman’s rank correlation coefficients of 0.45 for Pass@1 and 0.42 for Pass@10.**
| Model | Size (Context) / | Pass@1 ($T=0.3$) | Pass@10 ($T=0.8$) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | Version | Baseline | + Error Feedback | Greedy^$\dagger$ | Zero-shot CoT | Baseline | + Error Feedback |
| Deepseek-Coder-V2-Lite (DeepSeek-AI et al., ) | 16B | 27.4$\pm$1.0 | 33.4$\pm$0.5 | 26.7 | 26.7 | 40.3$\pm$0.7 | 43.5$\pm$0.9 |
| Starcoder2-v0.1 (Lozhkov et al., ) | 15B | 22.0$\pm$0.9 | 27.9$\pm$1.6 | 21.6 | 28.4 | 37.8$\pm$0.3 | 39.9$\pm$0.3 |
| CodeLlama (Rozière et al., ) | 7B | 16.4$\pm$1.1 | 18.8$\pm$1.7 | 19.0 | 0.9 | 29.0$\pm$1.5 | 33.1$\pm$1.5 |
| 13B | 20.3$\pm$0.6 | 25.6$\pm$1.3 | 22.4 | 0.0 | 35.7$\pm$0.8 | 41.2$\pm$1.0 |  |
| 34B | 17.4$\pm$0.8 | 23.3$\pm$1.1 | 18.1 | 3.4 | 34.7$\pm$1.0 | 39.9$\pm$1.9 |  |
| Llama-3.1 (Dubey et al., ) | 8B | 16.0$\pm$0.4 | 20.5$\pm$0.7 | 17.2 | 19.8 | 29.2$\pm$0.5 | 34.9$\pm$0.6 |
| Llama-3.2 (Dubey et al., ) | 1B | 9.7$\pm$0.4 | 13.0$\pm$0.8 | 9.5 | 3.4 | 16.2$\pm$0.8 | 21.1$\pm$0.6 |
| 3B | 10.6$\pm$0.8 | 14.5$\pm$1.0 | 12.1 | 11.2 | 20.2$\pm$1.5 | 23.9$\pm$2.7 |  |
| CodeQwen1.5-Chat (Bai et al., ) | 7B | 21.4$\pm$0.8 | 25.9$\pm$0.6 | 19.0 | 27.6 | 38.6$\pm$1.5 | 42.4$\pm$0.8 |
| Qwen2 (Yang et al., ) | 7B | 18.3$\pm$0.7 | 26.7$\pm$1.5 | 19.0 | 19.0 | 38.0$\pm$0.6 | 43.3$\pm$0.5 |
| 72B | 25.0$\pm$0.7 | 30.1$\pm$0.3 | 23.3 | 24.1 | 38.2$\pm$0.9 | 40.7$\pm$0.8 |  |
| Qwen2.5-Coder (Hui et al., ) | 1.5B | 19.7$\pm$0.8 | 22.3$\pm$1.2 | 19.8 | 17.2 | 32.7$\pm$1.0 | 38.8$\pm$0.7 |
| 7B | 21.1$\pm$0.5 | 24.4$\pm$0.4 | 22.4 | 14.7 | 35.6$\pm$1.3 | 41.5$\pm$0.7 |  |
| Codestral-v0.1 (https://mistral.ai/news/codestral/) | 22B | 25.0$\pm$0.4 | 31.3$\pm$0.8 | 24.1 | 34.5 | 37.1$\pm$0.8 | 40.6$\pm$0.9 |
| Yi-Coder-Chat (AI et al., ) | 1.5B | 18.7$\pm$0.5 | 22.0$\pm$0.9 | 18.1 | 21.6 | 32.6$\pm$1.7 | 37.0$\pm$1.7 |
| 9B | 25.7$\pm$0.5 | 30.8$\pm$0.6 | 25.0 | 22.4 | 38.6$\pm$0.9 | 41.5$\pm$0.8 |  |
| Yi-1.5-Chat (AI et al., ) | 6B | 17.3$\pm$1.0 | 23.9$\pm$1.7 | 15.5 | 19.0 | 32.8$\pm$1.0 | 37.4$\pm$1.5 |
| 9B | 20.1$\pm$0.3 | 24.7$\pm$1.0 | 19.8 | 21.6 | 30.5$\pm$0.6 | 39.5$\pm$1.1 |  |
| 34B | 20.8$\pm$0.5 | 26.3$\pm$1.0 | 21.6 | 20.7 | 35.3$\pm$0.8 | 44.1$\pm$0.7 |  |
| codegemma (Team et al., ) | 7B | 17.8$\pm$0.8 | 23.6$\pm$0.2 | 17.2 | 19.8 | 33.9$\pm$0.6 | 38.0$\pm$0.5 |
| stable-code (Pinnaparaju et al., ) | 3B | 14.6$\pm$0.7 | 16.3$\pm$0.9 | 14.7 | 17.2 | 23.9$\pm$1.4 | 25.9$\pm$0.9 |
| granite-code (Mishra et al., ) | 3B (128k) | 23.9$\pm$1.1 | 26.1$\pm$2.0 | 22.4 | 29.3 | 33.6$\pm$1.3 | 36.5$\pm$0.6 |
| 8B (4k) | 24.7$\pm$0.9 | 29.2$\pm$1.5 | 24.1 | 28.4 | 38.5$\pm$0.4 | 42.1$\pm$1.0 |  |
| 8B (128k) | 23.9$\pm$0.7 | 28.7$\pm$1.1 | 25.0 | 29.3 | 35.0$\pm$0.5 | 38.9$\pm$0.6 |  |
| 20B (8k) | 28.2$\pm$0.6 | 30.5$\pm$0.8 | 29.3 | 31.0 | 37.0$\pm$0.7 | 38.3$\pm$0.8 |  |
| 34B (8k) | 29.8$\pm$0.9 | 31.7$\pm$1.2 | 31.0 | 31.0 | 38.3$\pm$0.1 | 39.9$\pm$1.3 |  |
| Phi-3.5-mini (Abdin et al., ) | 3.8B | 24.0$\pm$0.7 | 29.0$\pm$0.9 | 25.9 | 19.8 | 35.1$\pm$1.1 | 39.2$\pm$1.0 |
| Phi-3.5-MoE (Abdin et al., ) | 16x3.8B | 30.9$\pm$0.8 | 34.9$\pm$0.7 | 33.6 | 23.3 | 40.5$\pm$0.5 | 43.2$\pm$0.1 |
| Nxcode-CQ-orpo (Hong et al., ) | 7B | 20.8$\pm$0.5 | 26.3$\pm$1.3 | 19.8 | 27.6 | 38.7$\pm$1.8 | 43.2$\pm$1.3 |
| GPT (OpenAI et al., ) | 3.5 | 19.6$\pm$1.0 | 27.2$\pm$0.8 | 19.8 | 20.7 | 33.3$\pm$1.0 | 37.8$\pm$1.7 |
| 4o (2024-05-13) | 23.6$\pm$2.7 | 34.1$\pm$1.2 | 25.0 | 27.6 | 39.9$\pm$2.4 | 43.7$\pm$2.4 |  |
| Gemini (Team, ) | 1.5-pro-001 | 25.1$\pm$0.2 | 35.9$\pm$0.2 | 32.8 | 37.1 | 41.4$\pm$0.5 | 43.8$\pm$0.6 |

Figure: Figure 3: Correlation of GitChameleon with the representative code benchmarks (HumanEval, EvalPlus, and BigCodeBench-Hard split). For each benchmark, the spearman correlation coefficient was 0.37, 0.50, and 0.36, respectively.
Refer to caption: x6.png

To highlight the uniqueness of our proposed evaluation dataset and difficulty of the samples in GitChameleon, we examine the performance correlation of models on GitChameleon in Figure 3 with three standard code generation evaluation benchmarks: HumanEval (Chen et al., ), EvalPlus (Liu et al., ), and the Hard split of BigCodeBench (Zhuo et al., ). Although performance on these benchmarks shows a positive correlation with that on GitChameleon, the relationship is not strictly predictive, as models that perform well on one or more of these benchmarks do not necessarily achieve equivalent success on GitChameleon.

### 3.4 In-Depth Analysis of Findings

In this section, we delve deeper into the results obtained from our experiments, analyzing model performance across various dimensions, including model size, year of library release, the type of API changes encountered, and sample difficulty.

Figure: Figure 4: Instruct-tuned model performance breakdown by version release year (top) and type of change (bottom): We analyze model performance in terms of pass @ 10 for baseline and with error feedback generation across two dimensions: version release year and type of changes. Darker shaded bars represent values obtained via error feedback generation. Standard deviation is drawn as a black line, obtained from 5 random seeds. (Top) Many models perform poorly on 2022, and generally perform worse on more recent versions. (Bottom) All models perform very poorly at semantic changes, indicating an potential area for massive improvement. Most models perform well on function name changes and new feature (with the exception of Llama-3.2-3B, which is also the smallest model in this comparison).
Refer to caption: x7.png

#### Analysis of Performance by Release Date

At the top of Figure 4, we present the year-over-year performance of a subset of the instruction-finetuned models. The average performance of all models dropped significantly from 2019 (not shown) to 2023. This decline is likely due to the fact that the training datasets contain more data from earlier years, underscoring the need for code LLMs to better adapt to the evolving nature of code libraries and their versions. Interestingly, many models appear to improve with error feedback disproportionately across versions released in the years 2021-2023. For example, Qwen2-72B and CodeLlama-34B improve more with feedback in 2021-2022 compared to 2023, while GPT-4o and Yi-1.5 Chat 34B improve more with feedback in 2023. This raises a question about the extent to which models’ training data influence the effectiveness of error feedback.

#### Analysis of Performance by Type of API Change

At the bottom of Figure 4, our analysis of model performance across different types of API changes in GitChameleon revealed significant variations. The models struggled the most with Semantics or Function Behavior changes.
Argument and Attribute changes were the second most challenging.
In contrast, models performed better on Function Name changes and New Feature or additional dependency based changes.

In general, larger models are more robust to function name, argument/attribute changes, and new feature changes. However, all models perform very poorly on semantic changes, regardless of the availability of error feedback. This indicates a weakness of SotA code generation models, and an area for further investigation. Furthermore, error feedback appears to have a more significant impact in argument/attribute changes compared to the other types of changes. This indicates that 1) the models may be using the error feedback to directly address failures in version-conditioned code generation, rather than non-version-specific errors, and 2) error feedback is most effective for version-specific argument/attribute changes.

#### Sample difficulty analysis

Figure: Figure 5: Comparison of sample and model differences. The left panel shows the distribution of sample difficulty, measured by the frequency of pass@10 scores across all models and seeds. The right panel presents the same distribution, but averaged for specific models across their seeds. Interestingly, individual models tend to exhibit bimodal distributions, indicating they are either consistently good or bad at specific problems. However, the aggregate distribution is less bimodal, suggesting that different models excel at different problems.
Refer to caption: x9.png

Figure 5 shows the distribution of sample difficulty. Notably, individual models (right panel) often display bimodal distributions, meaning they tend to perform consistently well or poorly on specific problems. In contrast, the aggregate distribution (left panel) is not bimodal, indicating that different models perform well on different sets of problems. The availability of error feedback shifts the distribution of the sample-wise difficulty to the right, as expected. Interestingly, some samples are not solved at all across models, even with feedback, and no samples are solved consistently by all models. As a further investigation, we plan to qualitatively examine samples that shift from unsolved to solved given error feedback. Finally, the right panel shows that many samples are either ”easy” or ”hard”, however larger models tend to have more ”medium” difficulty samples, indicating that scale can, at least partially, improve version-conditioned generation from unsolved to solved. We qualitatively demonstrate some of these examples in A.2.

Figure: Figure 6: Doc-prompting vs no doc (greedy decoding) instruct-tuned model performance on a subset of libraries. Doc-prompting refers to the prompting technique in which the prompt contains the documentation of the library function specific to the problem at hand. Darker shaded bars represent doc-prompting values, lighter shaded bars represent no doc values.
Refer to caption: x11.png

#### Doc-prompting vs no doc

Figure 6 shows the instruct-tuned model performance breakdown between doc-prompting and no doc (standard prompt) with greedy decoding for a subset of libraries in the dataset.
There is no consistent trend for any library, and in some cases, doc-prompting performs worse than the standard prompt (no doc). Moreover, performance across models is generally low for torch compared to the other libraries, and does not improve with doc-prompting. This suggests that the poor performance may be due to vague or difficult problem statements making it challenging for the model to interpret the problem.

Figure: Figure 7: Most frequent categories of errors obtained from executing unit tests on model generated code, averaged across models. The most common error by far is Name errors, which represents a variable that was used but not defined. This suggests that models do not grasp the context in many cases. Furthermore, Name, Indentation, Attribute, Import, AssertionError, and ModuleNotFound errors were all reduced with error feedback, while TimeOut and Syntax errors increased.
Refer to caption: x12.png

#### Error Categorization

The objective of error feedback prompting is to allow models to self-correct by interpreting the error trace from the initial execution of the generated solution. Figure 7 reveals that across all instruction-based models, the most frequent error type is Name, indicating a gap in contextual awareness within these models. Although error feedback prompting reduces the occurrence of most types of errors, it unexpectedly increases the frequency of Syntax and Timedout errors.

## 4 Related work

#### Code LLM training and evaluation protocols

Code LLM evaluations mainly revolve around code completion (Zhang et al. (); van Dam et al. (); Lu et al. ()). Existing benchmarks emphasize generic code completion, yet a recognized limitation is the inability of code LLM to generate and complete code that requires library and project-level knowledge (Xu & Zhu, ), let alone version-level knowledge, which is vital for real-world software applications.

Recent initiatives address repository-level code understanding by LLMs (Bairi et al. (); Shrivastava et al. (; ); Liu et al. (); Guo et al. ()). Attempts at library-level code generation (Zan et al. ()) and consideration of dependencies between files (Guo et al. ()) have been made. However, these efforts do not directly address the challenge of accommodating version-sensitive changes, adding complexity.

The core issue arises from models being trained on library code without explicit knowledge of library versions or their functional changes. Consequently, when tasked with generating code specifically compatible with a particular library version, there is a significant risk models often encounter failures.

#### Datasets

Existing datasets like HumanEval (Chen et al., ), MBPP (Austin et al., ), and MTPB (Nijkamp et al., ) provide sets of handwritten prompts and test cases to evaluate code generated by code LLM. However, these datasets are relatively small and lack context regarding a model’s comprehension of repositories. APPS (Hendrycks et al., ) and CodeContest (Li et al., ) offer challenging datasets with coding competition questions, providing insights into a model’s performance on difficult problems but without a focus on library-specific challenges. DSP (Chandel et al., ) and DS-1000 (Lai et al., ) concentrate on the top data science libraries in Python, while JulCe (Agashe et al., ) uses Jupyter Notebooks for training and evaluation, but these notebooks do not necessarily need to be repository-specific. CoNaLa (Yin et al., ) contains problems collected from StackOverflow across multiple programming languages, including both library-specific questions and non-library-specific code. More recently, BigCodeBench (Zhuo et al., ) is constructed to evaluate the comprehensive capabilities of code generation with tool use and instruction following, which poses a great challenge for existing models. Several datasets include version-specific code, such as Versicode (Wu et al., ), CodeUpdateArena (Liu et al., ), and the dataset by Wang et al. Versicode’s dataset, compiled from academic papers, Stack Overflow, and library source code, supports tasks like token, line, and block-level code completion and code editing. Unlike our dataset, Versicode evaluates using exact matches. Wang et al.’s dataset collects API mappings, such as “torch.lstsq() is deprecated in favor of torch.linalg.lstsq(),” and evaluates LLMs using exact match, edit similarity, and fixed rate metrics. Although Versicode and Wang et al.’s datasets address the evolving nature of libraries, their evaluations are limited to string matching.

In contrast, CodeUpdateArena evaluates LLMs’ ability to adapt to API changes, such as adding a boolean flag, by running tests. However, the dataset is synthetic and are not extracted from real-life version changes. For CodeUpdateArena, they also take the approach of training LLMs using the updated API function –using docstrings or examples–. It then tests if without access to the update during inference, the LLM’s reflects the synthetic changes. While these datasets provide valuable resources for training and evaluating models, our GitChameleon dataset advances research into version-conditioned code generation by LLMs. Runnable tests offer insights into LLM adaptability, and GitChameleon further assesses a model’s ability to differentiate between library versions, and successfully use a specific version.

#### Implications for Lifelong Learning

Continual/lifelong learning in code generation models is in its early stages (Yadav et al., ; Weyssow et al., ; Wu et al., ; Gao et al., ). However, current efforts often focus on artificial sequential tasks rather than utilizing the natural distribution shift in the chronological evolution of code. Notably, continual learning mainly targets mitigating catastrophic forgetting and balancing forward- and backward-transfer on a data stream, which may not align optimally with coding environment demands. In coding environments, obsolete or legacy libraries may prompt selective forgetting of irrelevant knowledge, particularly at the library/package level. Previous work, like Caccia et al. (), which explores continual adaptation to shifting data distributions, and Bornschein et al. (), which focuses on rapid adaptation to new distributions, could serve as foundations for enhancing continual learning in LLMs for code. We anticipate that GitChameleon will inspire advancements in LLMs’ continual version adaptability, guiding development in this area. Such adaptability would be highly valuable in practical applications like enterprise-level code migration.

## 5 Limitations

We consider the lack of prompt optimization done for the instruct models as a considerable limitation of our analysis.
Furthermore, the dataset consists of 116 problems, which is relatively small compared to other code benchmarks.
Finally, we do not explore approaches such as RAG, chain of thought, or finetuning on a split of our benchmark to observe an upper bound of performance on this task. Future work could explore such approaches using our dataset.

## 6 Conclusion

Recognizing the crucial need for code LLM adaptation to evolving code environments, particularly in widely used libraries, we introduce a novel and extensive Python-based version-specific benchmark named GitChameleon.
By effectively leveraging GitChameleon, we expose the shortcomings of existing state-of-the-art (SoTA) models in producing version-specific code, representing an inaugural effort to draw attention to this challenge. While our work exposes this shortcoming, we acknowledge the dataset’s limitations. In future endeavors, we aim to enhance the dataset’s comprehensiveness across various programming languages and frameworks. Additionally, we plan to introduce new tasks that can benefit research on code LLM models using GitChameleon.

## Acknowledgments

The authors would like to express their sincere appreciation to Alex Gu for his insightful feedback, which significantly contributed to the development of the concept of the project. Furthermore, we extend our gratitude to Victor May for his technical assistance and expertise in supporting the benchmarking process. The authors acknowledge Bharat Runwal for his assistance during the data curation process and; Deepanway Ghosal and Antonio Orvieto for their valuable feedback on the manuscript. This research was enabled in part by compute resources provided by Mila ([mila.quebec](mila.quebec)) and the Max Planck Institut für Intelligente Systeme ([is.mpg.de](is.mpg.de)). The authors acknowledge support from the Canada CIFAR AI Chair Program and from the Canada Excellence Research Chairs (CERC) Program.

## Appendix A Appendix

### A.1 Example of Doc-prompting

Figure: Listing 1: Sample pytorch problem with documentation

### A.2 Qualitative Samples of Error Feedback

Figure: Listing 2: Original output from Codestral-22B-v0.1

Figure: Listing 3: Regenerated solution with Codestral-22B-v0.1 given error feedback

Figure: Listing 4: Original output from Phi-3.5-MoE-instruct

Figure: Listing 5: Regenerated solution with Phi-3.5-MoE-instruct given error feedback

Figure: Listing 6: Original output with Phi-3.5-MoE-instruct

Figure: Listing 7: Regenerated solution with Phi-3.5-MoE-instruct given error feedback

Figure: Listing 8: Original output with Phi-3.5-MoE-instruct

Figure: Listing 9: Regenerated solution with Phi-3.5-MoE-instruct given error feedback

Figure: Listing 10: Original output with Phi-3.5-MoE-instruct

Figure: Listing 11: Regenerated solution with Phi-3.5-MoE-instruct given error feedback

Figure: Listing 12: Original output with Phi-3.5-MoE-instruct

Figure: Listing 13: Regenerated solution with Phi-3.5-MoE-instruct given error feedback

Figure: Listing 14: Original output with Phi-3.5-MoE-instruct

Figure: Listing 15: Regenerated solution with Phi-3.5-MoE-instruct given error feedback

Figure: Listing 16: Original output with gpt4o

Figure: Listing 17: Regenerated solution with gpt4o given error feedback

Figure: Listing 18: Original output with gpt4o

Figure: Listing 19: Regenerated solution with gpt4o given error feedback

Figure: Listing 20: Original output with gpt4o

Figure: Listing 21: Regenerated solution with gpt4o given error feedback

Figure: Listing 22: Original output with gpt4o

Figure: Listing 23: Regenerated solution with gpt4o given error feedback

Figure: Listing 24: Original output with gpt4o

Figure: Listing 25: Regenerated solution with gpt4o given error feedback

### A.3 Pseudocode for our verificaiton process

Each sample was validated using this method to ensure that it functioned as intended.

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 3
presentation: 3
questions: How are you planning to keep the benchmark up-to-date as the APIs and the models will evolve?
soundness: 2
strengths: - The paper underscores the need for LLMs to keep up with rapidly changing libraries, enhancing the practicality of code generation tools. Thus, GitChameleon focuses on version-specific code to evaluate model limitations and suggest improvements in LLM development.

- A new benchmark specifically targeting version-specific code generation.  The benchmark assesses models based on real execution of version-conditioned prompts, providing a systematic and practical measure of model performance.
summary: The paper argues that LLMs are extensively used as virtual coding assistants but frequently fail to generate accurate version-specific code amidst the fast-evolving software landscape. To address this, the paper presents GitChameleon, a new benchmark that assesses and enhances LLMs' ability to produce executable, version-tailored code. This benchmark highlights existing models' limitations and offers a pathway to improve their effectiveness in dynamic coding environments.
weaknesses: - The benchmark assesses model performance on only a few hundred examples, which may not fully capture the diversity and complexity of real-world codebases.

- Expanding GitChameleon as the modes evolve will be non-trivial. The benchmark will soon be saturated. Also, to cover more libraries and languages may be difficult.

- While GitChameleon addresses an important gap, the benchmark and related evaluation seems to be an incremental improvement.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: In Section 2.1, the authors describe four types of changes. How do you ensure that this classification comprehensively captures the version-switching problem?
soundness: 3
strengths: - The paper tackles the novel topic of assessing LLMs on version-specific code generation. Unlike existing benchmarks that generally focus on static code generation, GitChameleon is designed to evaluate models’ adaptability to changing library versions, a problem particularly relevant for real-world applications.

- The dataset is meticulously curated with executable unit tests, enhancing the robustness of its evaluations. GitChameleon addresses a critical gap in code generation benchmarking. By introducing an execution-based benchmark focused on version-specific compatibility, the work not only highlights current model limitations but also provides a concrete dataset for future improvements in LLM adaptability.

- The paper is well-organized and clear in its presentation.  The motivation behind GitChameleon is well presented as a crucial challenge in the field of code generation.
summary: The paper introduces a new dataset and benchmark, GitChameleon, designed to evaluate the ability of large language models (LLMs) to handle version-specific code generation challenges in Python. 
GitChameleon consists of 116 Python code completion problems, curated to test models' responses to specific library versions and accompanied by executable unit tests for functional validation. 
This benchmark highlights the limitations of LLMs in managing code library version updates, which are critical for practical applications in dynamic software environments.
weaknesses: - Although the focus is clear and the dataset creation process is rigorous, the current GitChameleon dataset remains limited in size. Many libraries in the dataset include only a few API changes, which increases randomness and reduces the stability of the results.

- The benchmark’s limitation to Python makes the version-switching problem less complex in terms of **type information**. In other languages, such as Java, version updates often involve type changes, which are a significant aspect of the version-switching challenge. Thus, Python’s focus may oversimplify some key challenges in version-switching, limiting the paper’s broader contribution to this problem.

- Although the authors acknowledge methodological limitations, some baseline methods, such as RAG, should have been included as a critical evaluation aspect rather than overlooked. For practical benchmark usage, the authors should provide settings for RAG or few-shot learning, as these are more common approaches for addressing knowledge gaps in LLMs and should have been considered.

- Some related work in API learning, which is closely tied to the version-switching problem, is omitted. Including recent popular work on API learning could better contextualize the research. Relevant citations might include:

    - Zan, Daoguang, et al. "When language model meets private library." arXiv preprint arXiv:2210.17236 (2022).
    - Zhang, Kechi, et al. "Toolcoder: Teach code generation models to use API search tools." arXiv preprint arXiv:2305.04032 (2023).

In summary, the paper’s topic is interesting, but it appears to miss several critical dimensions of the version-switching problem, such as choosing a weakly-typed language and omitting a key aspect of version-switching. The current dataset may also be too limited to fully support the range of challenges outlined in the introduction section.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: I also have a few smaller questions:
- Line 177: How many samples fall under the Argument or Attribute and Function Name change categories exactly?
- Line 322: Why is performance better in 2023 than 2022?
- Line 430: Figure 3.4 should be Figure 5.
- Line 495: Wang et al. is not properly linked to the reference.
soundness: 3
strengths: The paper studies an important problem in LLMs for code, i.e., dynamic updates of LLMs when it comes to library changes. I appreciate the authors’ effort in constructing GitChameleon and performing evaluation and analysis over it. This could be useful for future research.
summary: This paper proposes GitChameleon, an execution-based benchmark for evaluating LLMs’ coding capabilities in scenarios involving library updates. The benchmark is manually constructed with a joint effort from multiple authors. The paper provides a detailed analysis of the benchmark and an evaluation of multiple LLMs.
weaknesses: - One major selling point of the paper is providing unit tests for assessing LLM-generated code. However, I don’t really get why unit tests are important for the domain of library changes. For the types of problems considered in this paper (i.e., Lines 167-176), the compiler or interpreter should already tell if a deprecated library version is used. Moreover, when the generated code passes the compiler or interpreter, do you ensure that the LLM actually uses the target API? The LLM could use some other features of the library to implement the same functionality. The LLM could also make errors on parts not concerning the target API.

- The dataset construction is a manual process. While I appreciate such an effort, this unfortunately results in manual bias and a relatively small size of data samples. For example, GitChameleon only covers 116 problems in Python libraries related to machine learning. This could threaten the validity of the results.

- The paper only provides an evaluation, without studying or discussing how to address the issue of LLMs in library updates. This is a relatively thin contribution, especially given that there are already a few benchmarks in the same domain as cited by the paper.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Have you considered using additional techniques, such as prompt engineering, chain of thought (CoT), and prompt fine-tuning, to provide a more comprehensive evaluation of LLMs’ capabilities? Including these methods might better showcase the model's adaptability in version-specific code generation. We would like to see corresponding experimental results for these methods.

2. Could you elaborate on how this dataset could aid in improving the version-specific code generation abilities of existing LLMs? For instance, do you see potential in approaches like fine-tuning on version-specific tasks or using reinforcement learning from unit test feedback? We would be interested in seeing relevant experimental results for these approaches.

3. Would it be feasible to include more mainstream LLMs, such as Claude, in your evaluations? Additionally, do you plan to expand the dataset size? The current dataset is relatively small, which might limit the depth of the benchmark and its ability to generalize across various version-specific challenges.
soundness: 2
strengths: This paper presents GitChameleon, a benchmark specifically designed to test the ability of LLMs in generating version-specific code. It includes 116 Python-based tasks tied to specific versions of popular libraries and equipped with executable unit tests. The paper presents an empirical study evaluating the performance of various state-of-the-art LLMs (e.g., GPT-4, CodeLlama, DeepSeek-Coder) on GitChameleon, highlighting their strengths and limitations in handling version-specific tasks. The paper is well-structured and clearly explains GitChameleon’s dataset creation, evaluation metrics, and model performance analysis, making it accessible to readers who may not be familiar with version-specific challenges in code generation.
summary: The paper presents GitChameleon, a benchmark for evaluating large language models on version-specific code generation tasks. GitChameleon comprises 116 Python tasks tied to specific library versions, with unit tests for execution-based validation. The benchmark highlights current LLMs' limitations in handling dynamic library changes, offering a valuable tool for advancing version-aware code generation models.
weaknesses: While GitChameleon introduces a unique focus on version-specific code generation, the dataset is limited to 116 tasks and 11 Python libraries. This relatively small scale might restrict the generalizability of findings and the benchmark’s robustness. The evaluation lacks experimentation with widely used techniques in prompt engineering (such as chain of thought) and does not consider fine-tuning approaches like prompt-tuning or parameter-efficient tuning. GitChameleon’s current setup seems primarily focused on software engineering tasks and libraries. This specificity may make it more suitable for specialized conferences in software engineering (e.g., ICSE)