CodeUpdateArena: Benchmarking Knowledge Editing on API Updates

Authors: Zeyu Leo Liu, Shrey Pandit, Xi Ye, Eunsol Choi, Greg Durrett
Primary area: datasets and benchmarks
Keywords: knowledge editing, code large language models, program synthesis

Abstract:
Large language models (LLMs) are increasingly being used to synthesize and reason about source code. The libraries and API functions they invoke are continuously evolving, with functionality being added or changing. Yet, no prior work has studied how an LLM's knowledge about code API functions can be updated. To fill this gap, we present `CodeUpdateArena`, a benchmark for knowledge editing in the code domain. An instance in our benchmark consists of a synthetic API function update paired with a program synthesis example that uses the updated functionality; our goal is to update an LLM to be able to solve this program synthesis example without providing documentation of the update at inference time. Compared to knowledge editing for facts, success here is more challenging: a code LLM must reason about the semantics of the modified function rather than just reproduce its syntax. Our dataset is constructed by first prompting GPT-$4$ to generate atomic and executable function updates. Then, for each update, we generate program synthesis examples whose code solutions are prone to use the update. Our benchmark covers updates of various types to 54 functions from seven diverse Python packages, with a total of 670 program synthesis examples. Our experiments show that fine-tuning open-source code LLMs (i.e., DeepSeek, CodeLlama) on documentation of a new update does not allow them to incorporate changes for problem-solving. However, prepending the same information does help, establishing that the information is present, and careful fine-tuning on examples demonstrating the update shows improvement, paving the way for better knowledge editing techniques for code.

Decision: reject
Reviewer scores: 5, 5, 3, 6, 8

Reviews and discussion:
<thread>
creation date:
1730674593283

thread_type:
official_review

<thread_object>
creator: reviewer_jdut

<content>
summary:
This paper presents a new program synthesis dataset focused on evaluating how well an LLM can be adapted to handle changes to an existing API. In each task, an LLM gets an API change description and a few example usages, and after after fine-tuning (or otherwise integrating this information) must solve synthesis examples using the new API, without being prompted with the new API. The evaluation metric ensures that the new API is used and that the old API wouldn't have worked in place of it. The benchmark itself is LLM generated, but with considerable testing and manual inspection for quality control. The evaluation finds that while fine tuning on updated API examples improves performance a bit, none of the evaluated fine tuned models achieves high accuracy. They also ensure the solvability of the dataset by showing that in a few-shot setting with the API functions provided, most of the benchmark is solved.

strengths:
The dataset created in this work deals with an interesting and real-world applicable problem, one that is particularly relevant to large language models which are trained on large amounts of historical code. Broadly I think that such a dataset is a good thing for the community to have, and that it will encourage future research in this area – and this work does a good job of building and evaluating the dataset.

The metrics presented in the work, UPass@K (performance on problems using new API) and SPass@K measuring specificity (i.e. performance degradation on unrelated problems), are solid metrics for evaluating the task. In particular it's important that UPass@K ensures that the new solution not only passes all the unit tests but does so using the new functionality, and that swapping in the old functionality would *not* solve all the tests. Overall this work does a good job of setting up a sensible and precise problem statement / set of metrics for future work to be evaluated on.

While an LLM is used for dataset generation, considerable effort was put into automatically and manually screening the generated tasks – both for the API specifications (4.2) and for ensuring that solving failures in the few-shot setting are not because of incorrect specifications (Table 3 and Appendix C.2).

The evaluation is reasonable for a paper presenting this dataset – a novel approach to solving the problem isn't presented, but doesn't need to be for a dataset paper in a new, compelling domain.

weaknesses:
- I'm confused about how the Base Models could perform so well, which I've elaborated on in the Question section – this is my main concern since it suggests there's something I'm misunderstanding either about those Base Models or the dataset itself.
- (minor) Including the c=1 (so r=2) condition in ablation Table 5 would be nice to see, just to be sure that c=2 really provides the best performance benefit before it starts to drop with c=3.
- Quick fix – Table 8's description should say 77 failures not 66, at least that's what the column adds up to.

> We observed relatively few cases of incorrect test cases or bad specifications, indicating that our dataset is of sufficient quality to test knowledge editing methods.

- From the appendix it seems to be that 19.5% (15/77) of failures are for these reasons, which is not ideal but okay – manual inspection to fix these would be nice for verifying the quality of the dataset. But not a big problem given what a small fraction this is of the total dataset (since 19.5% is only within failures, which themselves are 16.9% of all the problems so it's only about 3% of the whole dataset, if I understand correctly).

questions:
- I would have expected the Base Model row of Table 4 to get essentially 0% for UPass for all models, since UPass requires not only solving the problem but actually solving it using new API in a way that would not work with the old API – but instead models get as high as 22.7% in Pass@1 and 33.3% in Pass@5. My understanding is that the Base Model doesn't get to see the new API at all (neither in fine tuning nor prompting), so in order to solve any problems it would have to somehow hallucinate the correct API change, misunderstanding the old API to behave like the new API despite having never heard about the new API. It seems really surprising that that would happen so often.
  - Am I understanding the Base Model correctly? If so, can you explain this, perhaps providing examples of where this happens to help understand how it could be so common? How could models solve anything without having heard of the new API function?

- In section 6 it's mentioned that 2 program synthesis problems are used for fine tuning, because many update functions only have 3 synthesis problems so the remaining one is held out for testing (and cross validation is used). In cases where where there are more than 3 synthesis problems, what does it mean to be "correct" on this update specification. Does the model need to solve *all* of the held out synthesis problems in one go (with k=1 or k=5 attempts)? Or does it only need to solve one at a time, and these are treated as separate entries in the dataset (where each entry has 1 test synthesis problem and 2 training synthesis problems)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730680577226

thread_type:
official_review

<thread_object>
creator: reviewer_rf97

<content>
summary:
The paper introduces a benchmark to assess LLMs' ability to update their knowledge of API functions. It includes synthetic API function updates and program synthesis tasks using these updates. The paper evaluates various LLM update methods, such as fine-tuning open-source models on update documentation, prepending update information at inference time, and fine-tuning on examples. The dataset comprises 670 program synthesis examples involving updates to 54 functions from seven Python packages, all generated using GPT-4.

strengths:
- **Relevant and timely research**: Updating APIs presents a significant challenge for LLMs. Therefore, assessing how effectively LLMs evolve with API changes addresses a valuable and necessary research area.
- **Novel techniques**:
    - The paper explores code-specific update methods like fine-tuning with examples, recognizing that LLMs must understand the semantics of an update rather than just its syntax. 
   - The paper also establishes a new foundation for measuring LLM update capability in coding using UPass@k and specificity metrics.
- **Interesting insights for future research**: This paper also many interesting insights to help future research in this area. For example, 
    - The paper shows that fine-tuning with examples outperforms fine-tuning on documentation.
    - The paper highlights the issue of specificity loss when fine-tuning with knowledge updates, encouraging future research to address this challenge.

weaknesses:
**Evaluation limitations**: The model is penalized if it generates a correct program without using the updated function, even though it may still achieve the correct outcome. And from the examples given in the paper, it seems like this can occur very commonly. It would be helpful to explore a metric that captures these cases more accurately. **Suggestion**: Comparing pass@k with upass@k in Table 4 could offer some insight here.

**Possible compounding errors in LLM-generated datasets**: Since the dataset is fully generated by LLMs, LLMs can introduce errors affecting the quality of the dataset. It is very hard to establish that the update implementation is correct since the generated test cases themselves could be wrong. It is possible that LLM generated both wrong implementation and wrong test cases, but the wrong implementation might pass the wrong test cases. Similarly, LLM generated test cases for the program synthesis problems are used to the efficacy metric UPass@k and again there is no guarantee that these test cases are correct beyond just random manual inspection done by the authors.

questions:
1.	Can you add pass@k metric to Table 4 in addition to UPass@k metric? 
2.	How are LLMs updated on the docstring? For FT(U), how many times is the update instruction finetuned on? Does this hyper-parameter impact the performance? 
3.	What are the random updates in the training examples in FT(PS) setting? Can you provide some examples?
4.	In Table 4, what is the baseline approach (is it the model without any fine-tuning or prompting for updates)? Additionally, what is the Upass@k for the baseline model? It’s surprising that this value is significantly greater than 0, given that models without update knowledge would likely fail to use the new API correctly.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701356425

thread_type:
official_review

<thread_object>
creator: reviewer_fsyk

<content>
summary:
This work proposes CodeUpdateArena, a benchmark for knowledge editing in the code domain. CodeUpdateArena evaluates the extent to which a pretrained LLM’s knowledge of code API functions can be updated. This benchmark is a set of API updates and program synthesis examples.

The authors evaluate different approaches for updating information, such as pretending the function update’s docstring in the prompt, fine-tuning on code update information, and fine-tuning on program synthesis examples. They find that fine-tuning on documentation of a new
update does not allow LLMs to incorporate changes for problem-solving, while prepending the same information or fine-tuning on examples does help the LLM.

strengths:
- The paper is well-written, with an easy-to-follow running example in the description of the synthetic data generation process.
- The benchmark is open-sourced with 54 functions from seven diverse Python packages with 670 program synthesis examples.

weaknesses:
- While useful, the impact of the paper is limited as the size and diversity of the data set is small (54 functions) in one programming language (python).
- The core contribution is that LoRA fine-tuning on natural language descriptions of code updates is worse than adding the updates in prepended prompts or LoRA fine-tuning on code examples.

questions:
How might these results vary according to the size of the training dataset? E.g. if you employ full-parameter fine-tuning with significantly more samples, how might the efficacy of the FT methods change?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704775604

thread_type:
official_review

<thread_object>
creator: reviewer_plk3

<content>
summary:
This paper proposes a benchmark to assess a model's ability to adapt to changes in API function calling. They show that fine-tuning the model on updated program synthesis problems can improve the model's ability to embrace the new API function call.

strengths:
This paper poses an interesting question of how to evaluate model's robustness to API changes and introduces a dataset to evaluate this. The further provide a testing environment to compare different approaches to augmenting model capabilities with new APIs, testing both in-context and fine-tuning approaches.

weaknesses:
1) It's not clear that the types of updates proposed by GPT-4 are representative of the types of updates found in the wild. With synthetic data, GPT-4 may be idiosyncratic in the updates proposed, step-4 of deduplication removing 53% of the problems seem to indicate this is a significant downside of using GPT-4 for problem generation. A comparison to historic API changes would help justify this.

2) Related work would be better in section 2 to provide better context on what other work has been done in this topic. 

3) The naming of "Arena" seems to imply side-by-side comparison and human input the way ChatbotArena does. This benchmark does not do this. The creation of test cases and sample tasks that involve the new updated API function is more akin to R2E https://r2e.dev/pdfs/paper.pdf

4) It's not clear that updates to APIs happen at a high frequency compared to updates to post-training in proprietary models. In those cases, it may be simper or beneficial to amend the data in the post-training corpus. It would be worth studying if models can be taught to distinguish the difference between package/api versions. 

5) The results are limited to statistical and data processing tools popular in python. It would be interesting to show on other APIs like web applications and visualization packages like plotly and matplotlib.

questions:
1. Do these results replicate when not using LoRA but using full fine-tuning? It's possible that full fine-tuning with data augmentation will allow the model to pay better attention to the in context docstrings and API description. Perhaps something like RAFT (https://arxiv.org/abs/2403.10131). 

2. Does GPT-4 fine-tuning lead to improved performance? Otherwise, this result might be only limited to smaller models with weak reasoning and coding abilities. 

3. Does the fine-tuning lead to improved capabilities for other coding tasks like test-generation for code, self-repair, and code execution?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730918973482

thread_type:
official_review

<thread_object>
creator: reviewer_feee

<content>
summary:
This paper proposes CodeUpdateArena, a dataset of synthetic updates to library APIs in Python. Each instance in the dataset consists of the update (e.g., add/modify a parameter), a code synthesis problem using the changed API, and a set of tests to validate the synthesis results. The authors use GPT-4 to generate most of the dataset, followed by manual inspection to eliminate low-quality or incorrect instances. 

The authors explore three evaluation strategies with closed-source and open-source LLMs (GPT-4, Claude, DeepSeekCoder, and CodeLLama): prepending prompts with api update information, fine-tuning with docstrings, and fine-tuning with synthesis examples. 

The results show that the prepend and fine-tuning options improve the efficiency of the LLMs but also hurt their specificity, i.e., their performance on unrelated tasks.

strengths:
- The paper targets an interesting problem about the integration of API updates in LLM code generation. I believe this is an important task given that LLMs do not have knowledge about the version of the libraries that they use to generate code. Hence, without the updated information, the generated code might be incorrect or not compilable.

- The proposed dataset, CodeUpdateArena, is potentially useful to evaluate if LLMs can generate updated information. 

- The paper covers a reasonable set of LLMs in their evaluation, including both closed-source and open-source LLMs.

weaknesses:
- The fact that dataset is generated using GPT-4, including both the updates, synthesis problems, and the tests to evaluate the generated synthesis, is questionable. This approach limits the quality and realistic nature of the dataset. There is historical data and commits on API updates for many libraries on Github -- which would be more realistic and perhaps still challenging for the LLM to work with. 

- Unlike historical facts, multiple versions of a library API can exist at the same time and are concurrently used by different users due to compatibility issues. E.g., the latest version of numpy may require python 3.12 or greater; such users may want the older API. Hence, getting the latest version of the API is not always important, rather the LLM may have to understand the context or dependencies better for correct prediction. Hence, fine-tuning the LLM with fixed updates does not make sense. 

- Further, numerous libraries are updated every day/week, hence fine-tuning LLMs every time for these updates is not scalable.

- The evaluation approach and metrics do not provide a way to gain high confidence if the LLM learned the API update. There can be many other synthesis scenarios that requires using the API in question. How can we confidently say if the LLM learned the update if the evaluation is only done on a handful of synthesis examples? Hence a more robust evaluation is needed for this aspect.

- Given the fact that the larger LLMs like GPT-4 and Claude already obtain ~60-70% efficacy, this benchmark seems to be challenging only for smaller and slightly outdated LLMs. With the latest LLMs, the benchmark might soon saturate. Hence, it is unclear if the benchmark will be useful to evaluate the upcoming generations of LLMs.

questions:
1. Why do the authors generate synthetic data instead of using historical API updates?
2. How can we rely on tests generated by the LLM?
3. Given the very few examples, the SPass metric is a weak measure of LLM's updating ability? How can we gain more confidence in this metric?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732652511349

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
# General Response Part 1: High performance of base model

Reviewer **rF97** and **jDUt** note that the Base model achieves high UPass@k performance without knowing updated API information. We thank the reviewer for raising this good point.

Our unit tests relied on calling APIs themselves to produce the correct answer. The unit tests were themselves correct, and correctly assessed model performance under new API updates, but this had the effect of causing failures in the unit tests when running with the old API. To obtain a high UPass@k, the predicted solution needs to fail unit tests with the old API in the environment (as an indicator of using updated API), so this instant failure leads to a high false positive of UPass@k.

**To address this, we converted the answers of the unit tests into literal values and reran all impacted numbers.** The changes are as follows:
1. UPass numbers are substantially lower than before, particularly for the baselines. However, the deltas between approaches (e.g., Baseline and Prepend) are roughly the same as before, and the method that works the best (FT (PS)) is unchanged.
2. Pass@k numbers (added in Table 4) are slightly different due to changes in the unit tests (e.g., Pass@1 for GPT-4 baseline was 54.0% and is now 54.1%; Pass@1 with Claude was 69.5% and is now 68.4%). 

We have included Appendix B.4 describing this change. Table 7 includes the results from the previous version of the PDF before the change was made.


# General Response Part 2: Dataset Realism
Reviewer **FeEE** and **PLK3** comment on our choice of using synthetic data for the study from two aspects: 1) synthetic data may not reflect real world API updates; and 2) using historical API updates is a plausible alternative.

We chose synthetic data construction primarily to prevent data contamination, since code LLMs are pretrained on large code corpora and recent pre-training corpora would contain historical API usage info. We see this as a fundamental weakness for a dataset studying knowledge editing methods. Furthermore, real API updates often entangle a large number of changes; for a benchmark dataset on knowledge editing, real updates do not provide the level of modularity needed for focused evaluation.

To make sure our generated updates are as realistic as possible, we built a taxonomy (Line 813-830), and human spot checks for updates during generation (Line 263-268). Our data was intentionally designed to mirror the types of updates seen in the wild. We agree that comparing actual historical data would be an interesting study, though we are not aware of off-the-shelf datasets of past API updates.

# General Response Part 3: Quality Control of an LLM-Generated dataset

Reviewer **FeEE** and **rF97** expressed concern about the quality of an LLM-generated dataset. We address the concern here.

We ensure the correctness through systematic design of our generation pipeline. We generate different parts of material (e.g. update description, update implementation) in a cascading fashion. Taking unit tests as an example,  we have generated each unit test and different sections of each unit test — input, answer, assertion — in separate calls to GPT-4 (middle part of Figure 2). This allows self-consistency [1] to come into play. If the model agrees with itself (measured by executing solution) beyond a certain threshold (we set it to be 60%-70% Line 258, Line 300), it’s likely the case that the entire bundle — problem specification, reference solution, unit tests, etc — is correct.

We further ensured this by conducting multiple quality control practices during development: spot checking generated update (Line 263-Line 268), deduplication of program synthesis examples (Line 304-309), inspection of failed GPT-4 generation (Line 341-348, Line 1757-1765, Table 9), and verifying test coverage (Line 1816-1820). 

We further note that there are examples in the community where imperfect, automatically generated benchmarks are still useful. For example, post-training methods are tested on LLM-generated datasets — UltraFeedback, AlpacaFarm [4,5]; CodeBenchGen [2], an LLM-generated benchmark for code generation; MuSR [3], an LLM-generated benchmark for testing reasoning, etc. 

[1] Self-Consistency Improves Chain of Thought Reasoning in Language Models

[2] CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarks

[3] MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning (ICLR 2024)

[4] UltraFeedback: Boosting Language Models with Scaled AI Feedback

[5] AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper investigates a practical problem: updating a code language model to be aware of a change to API functions. They find prompting is surprisingly more effective than fine tuning on a new synthetic dataset that they introduced generated via prompting GPT4. It is best seen as a benchmarking and evaluation paper.

The primary strength is that the problem is important. The primary weakness is that the dataset itself is not rich enough to serve as a foundation for organizing the community around this problem: It is already reasonably well handled by frontier LLMs (unsurprising, given that it was generated by them), and so likely to saturate soon, and although there is some spot checking of solutions it is not as compelling as a human created benchmark (cf sygus), and lack robust automatic validation of code correctness for anyone who wanted to attempt the benchmark.

Given that this is primarily a benchmark paper it is hard to argue for acceptance when there are important problems with the benchmark itself, and little methodological novelty. Therefore this paper should be rejected.