CodeSync: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
Authors: Chenlong Wang
Zhaoyang Chu
Zhengxiang Cheng
Xuyi Yang
Kaiyue Qiu
Yao Wan
Zhou Zhao
Xuanhua Shi
Hai Jin
Dongping Chen
Venue: ICML 2025 poster (2025)
Keywords: Code Generation
Large Language Models
Knowledge Updating
API Evolution
Primary area: applications->everything_else
TL;DR: This paper introduces CODESYNC, a data engine for identifying outdated code patterns and collecting real-time API knowledge updates from Python third-party libraries.

Abstract: Large Language Models (LLMs) have exhibited exceptional performance in software engineering yet face challenges in adapting to continually evolving code knowledge, particularly the frequent updates of third-party library APIs. This limitation, rooted in the static pre-training datasets, often results in non-executable code or implementations with suboptimal safety and efficiency. To this end, we introduce CodeSync, a data engine to identify outdated code patterns and collect real-time code knowledge updates from Python third-party libraries. Building upon CodeSync, we develop CodeSyncBench, a comprehensive benchmark for assessing LLMs' ability to stay synchronized with code evolution, which covers real-world updates for 220 APIs from six Python libraries. Our benchmark offers 3,300 test cases spanning three evaluation tasks and an update-aware instruction tuning dataset of 2,200 training samples. Extensive experiments on 14 LLMs reveal that they struggle with dynamic code evolution, even with the support of advanced knowledge updating methods (e.g., DPO, ORPO, and SimPO). Our CodeSync lays a strong foundation for developing more effective and robust methods for real-time code knowledge updating in the future. The experimental code is available at:  https://github.com/CGCL-codes/naturalcc/tree/main/examples/codesync.

CODESYNC:
Synchronizing Large Language Models with Dynamic Code Evolution at Scale
Chenlong Wang" 1
Zhaoyang Chu Zhengxiang Cheng"
Xuyi Yang² Kaiyue Qiu Dongping Chen If
Yao Wan
Zhou Zhao Xuanhua Shi Hai Jin
Abstract
V2.0.8
apoote
V2.5.0
aurigned Runeries
nordh avsigne Fanctios für firssrdiets "arga)
sync
firannferes
Large Language Models (LLMs) have exhibited exceptional performance in software engineering yet face challenges in adapting to continually evolving code knowledge, particularly the frequent updates of third-party library APIs. This limitation, rooted in the static pre-training datasets, often results in non-executable code implementations with suboptimal safety and efficiency. To this end, we introduce CODESYNC, a data engine to identify outdated code patterns and collect real-time code knowledge updates from Python third-party libraries scale. Building upon CODESYNC, we develop CONESYNCBENCI, comprehensive benchmark for assessing LLMs' ability to stay synchronized with code evolution, which covers real-world updates for 220 APIs from six Python libraries. Our benchmark offers 3,300 test cases spanning three evaluation tasks and an update-aware instruction tuning dataset of 2,200 training samples. Extensive experiments LLMs reveal that they struggle with dynamic code evolution, even with the support of advanced knowledge updating methods (e.g., DPO, ORPO, and SimPO). Our CODESYNC lays a strong foundation for developing more effective and robust methods for real-time and large-scale code knowledge updating in the future. The experimental code is available at: https:// gi github. wtt.tul.ccccCCCCcccoo CGC.-cooeL-cccoc www.e//ooeeessseeeeeeeeeee natulaccc// tree Cree/maing httpa///exammeee/cceeeeeee
Please help complete this custon class:
class SmoothRe aut.gattaa Funciiai Function): def forward(
def forwardlett, inpor): cbusave for backwarddinput)
Deployed LLN
Typetrror: forward() takes positional argunent
updote
Interpreter
def fooward agal:
angloj input applls cleasee for backwardjjnpu))
Updated LLM
Execution Successfullyl
Interpreter
Figure 1: LLMs struggle to adapt to API updates, leading to potential compatibility issues in generated code. For example, the device parameter was removed from the full function in numpy version 2.1.0, making LLM failed to provide correct invocation. It highlights the need for API knowledge updating to synchronize LLM with the latest API changes and correctly generate updated API invocations.
14
1. Introduction
Large Language Models (LLMs), exemplified by DeepSeek- RI (Guo al., 2025), CodeLlama (Roziere et al., 2023), and GPT-4o (OpenAI, 2024), have demonstrated remarkable performance in automating software development through generating executable code (Jiang et al., 2024). However, due to static pre- pre- training datasets, they often struggle to adapt to the rapidly evolving knowledge in programming, especially the frequent updates of external library APIs (Tao et al., 2012; Zhang et al., 2020).
"Equal contribution 'National Engincering Research Center for Big Data Technology and Systems, Services Computing Technology and System Lab, Cluster and Grid Computing Lab, School of Computer Science and Technology. Huazhong University of Science and Technology, Wuhan, China FWuhuan University 'Zhejiang University. Correspondence Yao Wan <wanyao@ hust.edu.cn..
As illustrated in Figure 1, when prompted to create an array CUDA device, the LLM is unaware of the removal of the device ce parameter in the updated numpy. full function. This oversight results in an error, i.e., "TypeError: full() got an unexpected keyword argument 'device"". The pitfalls of generating code containing outdated APIs can lead parameter compatibility issues, which cause
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
Cade Completion Task (CCT)
Error Correction Task (BCT)
Multiple Choice Question (MCQ)
Ceepseok-Cader
Deepseek Coder
Deepseak Coder
22
3.40
0.40
0.45
3.30
3/2
Owen2.5
3/1
Qwen2.5
Liama- 3/1
Owen2.5
6:19
0:25
3.10
CodeLlama
Qwen2.5-Codd
CodeLlama
Qwen2.5-Code
CodeUlama
Qwen2.5-Codd
Originl
SFT[LORA
DPD
DAPO
SimPG
Figure 2: Performance comparison of knowledge updating methods across three evaluation tasks on five LLMs. All LLMs shown in the figure are instruction-tuned versions. The results reveal that LLMs face challenges in adapting to dynamic API updates, even with the support of knowledge updating approaches, emphasizing the need for improvements in real-time code knowledge updating.
programs to crash or malfunction, undermining the stability and reliability of software (Bai et al., 2024; Zhang et al., 2024c). This challenge highlights the need for LLMs to synchronize with the dynamic evolution of practical code knowledge, particularly the fast-paced API updates that have immediate and visible impacts on software development.
includes real-world updates for 220 APIs (130 functions, initializers, and methods) from 6 Python libraries, along with 3,300 legacy-updated pairs of API invocation instances. The benchmark provides 3,300 test cases across three evaluation tasks, iee, Code Complese Task (CCT), Error Correction Task (ECT), and Multiple Choice Question (MCQ), accompanied by an update- instruction tuning dataset comprising 2,200 training samples. Unlike retrieval- augmented frameworks that enhance LLMs at the expense of increased inference overhead and without reflecting true model updates, CODESYNC and improving LLMs' ability to internalize API update knowledge and accurately recall it during code generation.
Recently, Liu et al. (2024c) made an initial attempt to address this gap by benchmarking LLMS' ability to access API updates through fine-tuning. However, their benchmark relies on unauthentic API updates synthesized by GPT-4 (OpenAI, 2024) rather than real-world library updates, resulting in potentially biased assessments of LLMs' adaptability to practical code evolution. We argue that an authentic evaluation system should be established to answer the key question: Can LLMs be effectively and efficiently updated to handle real-time API modifications?
focuses evaluating
Take-Aways. We benchmark 14 state-of-the-art L.MSs (e.g.. ChatGPT (OpenAI, 2024), DeepSeck (Liu et al., 2024a) and Claude (Anthropic, 2024)), including both proprietary and open-source models, as well as five knowledge updating methods (e.g., DPO (Rafailov et al., 2023), ORPO (Hong et al., 2024), and SimPO (Meng et al., 2024)). Our findings reveal several key insights. First, as shown in Figure 2, assessment results indicate that LLMS struggle to adapt to dynamic API updates, even with the support of advanced knowledge updating approaches, highlighting the need for further advancements in real-time code knowledge updating. Moreover, the number of API invocations available for training and the types of updated APIs significantly impact the effectiveness of knowledge updating, increasing the complexity of handling real-worid API modifications.
To address this gap, this paper introduces CODESYNC, scalable data engine for collecting authentic code knowledge updates from Python third-party libraries across various domains, including data science (e.g., pandas), artificial intelligence (e.g., torch), and web development (e.g., flask). Specifically, CODESYNC systematically identifies real-time API updates at scale by tracking changes to API signatures across library versions. For each identified API with updates, it retrieves relevant code instances invoking the API from GitHub repositories using GitHub Code Search (GitHub). Based on these real-world API invocations, CODESYNC employs DeepSeek-V V3 (Liu et al., 2024a) to synthesize contrastive invocations for the legacy and updated API versions.
Contributions. Our primary contributions are summarized as follows.
Based CODESYNC, develop CODESYNCBENCH, an extensive benchmark for assessing LLMs' ability to stay synchronized with dynamic code evolution, which
A Data Engine. We introduce CODESYNC, a data engine that systematically collects real-time code knowledge updates from various Python third-party libraries.
2

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
Keal-Time API Update Tracking
Real-World API Invocation Retrieval
API Cundinal Sigusuue
Code Search Engine
Import Satementt
API
VEW
Updates
Tamput Seng
Inspect
API Updited
Outdated Signature
qulalii (l.
Updiated VENY
Powes
Updated API Signature
Outdated Signature:
Sullia: coud weweaaatttrreeeeeeeeeee
Updated Signatum::
Instanoo
Extracting API Signatures
Identifying Updates
Retrieving Relevant Code Instances
Locating Valid API Invocations
Rime API
Real- World API Invocation Retrieval
Legacy- Eeeacc-Updatedd API Invocation Synthesis
>>>
>>>
CODESYNCBENCH
Update Tracking
Each Valid API Invocation:
After Modification:
Benchmark Construction
CODESYNCBENCH
Code Contest..
Updated,
2 Target Siq
Sipliilly valuewe, desice=iouds)
Multiple Choice Question
Statistics
Outdated API Signature Updleted Signature
Ouidated. Code:
no np.ullli 21 BI values))
Error Correction Task
Function x 138
Add Updated Code & Outdoted Code to Invocation
Code Completion Task
APIx220
Method x 31
Instruction:
Initializer x 59
Please modify 'torget sexf arcording Updoted Signature' L'Oudated Signature' respectively.
Evaluation
Trainingset Construction
LLM
MetaData:
Code Completion Toak
regacy-Updated API Invocation Synthesis
CODESYNCBENCH
Figure 3:
An overview of our proposed CODES framework. CODESYNC consists of four key steps: (1) Real-Time API Update Tracking tracks and collects API updates by comparing legacy and latest versions of libraries. (2)
API Invocation Retrieval is designed to crawl API invocations and locate valid API calls. (3) Legacy-Updated API Invocation Synthesis leverages LLMs to synthesize new API invocation statements based on legacy and updated signatures, respectively, and then recognizes them into metadata. (4) CODESYNCBENCH is used to evaluate the performance LLMS on API updating tasks, with a period spanning from January 1, 2023 (post-Grt GPT-3.5 release) to current versions.
A Novel Benchmark. We develop CODES YNCHENCH, novel benchmark covering updates for 220 APIs across six Python libraries. It offers 3,300 test cases across three evaluation tasks and an update-aware instruction tuning dataset with 2,200 training samples. This benchmark can serve as a rigorous testbed to facilitate the development of real-time code knowledge updating methods.
systematically tracking changes to API signatures between library versions (see Section 2.1). (2) Real-World API Invocation Retrieval. For each identified API with updates, CODESYNC retrieves relevant code instances invoking the API from GitHub repositories through GitHub Code Search (GitHub) (see Section 2.2). (3) Legacy-Updated API Invocation Synthesis. Building on the retrieved real- world API invocations, CODESYNC employs DeepSeek- V3 (Liu et al., 2024a) to synthesize contrastive code instances that invoke legacy and updated APIs, respectively (see Section 2.3). Based on CODESYNC, we establish CODESYNCBENCH, a benchmark for assessing real-time code knowledge of LLMs, which collects updates for 220 APIs (including 130 functions, 59 initializers, and 31 methods) from Python libraries, totaling 3,300 legacy- updated pairs of API invocation instances (see Section 2.4).
Comprehensive Evaluation. Our extensive experiments on 14 state-of-the-art LLMs, including both proprietary and open-source models, indicate that they still struggle handle dynamic code evolution. Additionally, our results reveal that knowledge updating methods can improve LLM synchronization with API updates, though challenges remain to be addressed.
2. CODESYNC: A Data Engine for Real- Time Code Knowledge Collection
2.1. Step 1: Real-Time API Update Tracking
The functionality of APIs is exposed through their signatures, which provide interface for developers to utilize this functionality within code. This feature enables systematic tracking of library API updates by monitoring changes in their signatures.
As illustrated in Figure 3, we propose CODESYNC, a data engine for real-time collection of code knowledge evolution, which operates through three key steps: (1) Real-Time
API Update Tracking, CODESYNC identifies and extracts API updates across diverse Python third third-party libraries by
3

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
Table I: Statistics of tracked API updates. We systematically identify API updates across diverse Python third-party libraries by monitoring changes in API signatures between the latest version and an outdated version around January 1, 2023. This period coincides with the introduction of the milestone GPT-3.5.
templates. For example, to retrieve code invoking the function torch. Li we match the API name (e.g. .Linear) along with relevant import statements (e.g., import torch. nn as nn and from torch import nn). Further details on the matching templates are provided in Appendix B.1.3.
Locating Valid API Invocations. Code instances retrieved via matching templates may only potentially invoke the target APIs, requiring precise localization to confirm valid invocations. To achieve this, we parse each code instance into an Abstract Syntax Tree (AST) using Python's 's built- in ast module (Python, a) and traverse all statements to identify those that genuinely contain targeted invocations. Moreover, we perform alias resolution on import statements to establish mappings between full module names (e.g.. numpy) and their aliases (e.g., np), ensuring more accurate identification of valid API invocations. For example, we locate statements that contain np. full for the full function and nn Linear for the Linear class initializer. Furthermore, regarding method invocation locating, the ast module enables us to track objects whose types match the target class by examining class instantiations and assignments. For example, in the case of x. reshape (), we identify that x is of type torch. Tensor, confirming valid invocation of the reshape () method from the torch. Tensor class. The strategy guarantees that the filtered instances are absolutely correct. Detailed implementation is provided in Appendix 1.
Library
Legacy Version
Updated Version
Num.
pandas
2.0.3
2.1.4
1,043
numpy
1.24
2.1
55
scipy
1.10,0
1.13.1
494
tensorflow
2.11.0
2.18.0
161
torch
2.0.0
2.5.0
4,260
flask
2.2.2
3.0.0 22
Extracting API Signatures. We target 6 widely used Python third-party libraries: pandas, numpy, sc sci,y. tensorflow torch and flask. To collect complete API signatures from these libraries, we leverage Python's 's built-in inspect module, a dynamic reflection tool provided by the Python standard library (Python, b). This tool enables runtime analysis and collection of information about Python objects, including modules. classes, functions, and methods. For each library, we extract API signatures using inspections within virtual environments configured with specific library versions. Further details are provided in Appendix B.1.1.
Identifying API Updates. To evaluate LLMs' ability to synchronize with real-time API evolution, we consider the most recent library version before ChatGPT's release (OpenAI, 2023) as the legacy version and the current library version as the updated version. Then, we identify API updates by systematically comparing API signatures between versions. To determine whether an update exists for a given API, we perform a static analysis to establish parameter mappings for same-name APIs across versions. These mappings allow us to analyze API changes at the parameter level by examining differences in attributes such as parameter name, position, and type. Using this approach, we identify 6,063 API updates from the six targeted Python libraries, as summarized in Table 1. More implementation details are provided in Appendix 1.2.
Through retrieval and localization, we filter out APIs with fewer than 15 valid invocation instances. Out of 6,036 APIs, 220 meet the criteria, each with 15 valid invocation instances, resulting in total of 3,300 instances.
2.3. Step 3: Legacy-Updated API Invocation Synthesis
While real-world code instances with valid API invocations can be retrieved from GitHub repositories, it is challenging to determine the exact library version of the invoked API. To address this, we synthesize the contrastive API invocation pairs-legacy and updated-using state-of-the- art LLMs, which have demonstrated strong capabilities in revising code while preserving both semantic and syntactic correctness (Guo et al., 2024b).
2.2. Step 2: Real- World API Invocation Retrieval
While API updates are reflected in signature changes, collecting this information alone is insufficient to fully capture the evolution of code knowledge. To address this, we consider real-world API invocation scenarios, focusing on modifications in API usage within actual code contexts. For each API update identified in Section 2.1, we collect relevant code instances that invoke the API from GitHub.
Specifically, for each API invocation instance retrieved in Section 2.2, we prompt DeepSeek-Väs V3 (Liu et al., 2024a) to adapt the target API invocation statement according to the legacy and updated API signatures, respectively, while preserving the integrity of the surrounding context. To ensure data quality, the authors manually verify the divergence between legacy and updated versions, instructing the LLM to re-synthesize cases with insufficient divergence.
Retrieving Relevant Code Instances. We use GitHub Code Search (GitHub) to retrieve Python files that potentially contain API invocations by designing multiple matching
This approach ensures divergence in API usage while maintaining functional equivalence between legacy and
4

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
Code Completion Task
Error Correct Task
Multiple Choice Question
Provided with code context ending with API name, numpy, vectorize init please
[Instruction]
[Instruction] You provided with code contest ending with calls flosk loud. There exists invaking errors. please check and correct appropriate version.
[Instruction]
You provided with code context
ending with API name, numus. masked array.xar
complete the parameter list of current API call statement.
There possible calls. Please up the best
[Question]
[Question]
[Question]
!
co.c context...? (orig ScIyar
def stepinput_patte ousput_path.
interval, base);
</>
def index(var, finame, app):
<...cod context....
[Choices]
""Snaps layer values to boundaries
flaskurl fort'static, filenae-fnamme)
A. (keepdins-a, left parum-1o)
scaler- - lambda round(x/ interval) * interval
var flask joon.load load (open( config..ml") app-app)
Geftliinez, meatsa, under-a) (mean-i, straightt param))
arr- numpy.yectorii.
D. (axis-None, keepdims-z, mean-1)
[Answer]
[Answer]
(scaler, otypes="T T, signature:None)
fla.sl.lsosonnnaa load confn''onnii.jsonn))
[Answer]
Figure 4: An illustrative example of three evaluation tasks of CODESYNCBENCH, (1) CCT only provides the API call name at the end of the question, without explicitly listing the parameters, expecting the completion. (2) ECT includes an incorrect parameter list at the end of the question, expecting the correction. (3) MCQ does not explicitly listing the parameters, but presents one correct option and three incorrect options, expecting the most accurate answer.
Table 2: Statistics of data in CODESYNC. We construct CODESYNC and the associated training set step by step, from identifying real-time API updates, retrieving real-world invocations, and synthesizing legacy-updated invocations to building training and test samples.
in total), with 5 pairs for evaluation (1,100 in total) and 10 for training (2,200 in total). Based on this, our benchmark builds 1,100 tests per evaluation task, accompanied by a training set comprising 2,200 update-aware instructions, providing a rigorous foundation for assessing LLMs' ability to stay synchronized with API evolution.
Step
Setting
Input
Num.
Output
Num.
Code Completion Task (CCT) (Lu et al., 2021). This task evaluates whether LLMs have internalized the updated APIs and can recall them during code generation. Given code snippet ending with an API name, the LLM is prompted to complete the parameter list, with the updated API invocation statement serving as the ground truth. To measure the API invocation completion, we employ three widely used metrics: BLEU (Papineni et al., 2002) for evaluating lexical precision, ROUGE-L (Lin, 2004) for measuring semantic coverage, Relative Edit Distance (Ristad & Yianilos, 1998) for quantifying structural deviation, and CodeBLEU (Ren et al.) for assessing AST matching.
Python Libraries
Updates
6,036
2
API Updates
220
API Invocations
3,300
Logaey-Updated
3
API Invocations
3,300
3,300
Invocation Pairs
Logacy-Updann
Update-Aware
Train
2,200
2,200
Invocation Pains
Instructions
CODESYNC
BENCH
CCT Tests
1,100
Segacy-Updated
Test
1,100
BCT Tests
1,100
Invocation Pairs
MCQ Tests
1,100
Error Correction Task (ECT) (Zheng et al., 2024a). This task simulates real-world debugging scenarios, where an interpreter throws an exception related to a specific API invocation. It evaluates the LLM's ability to actively correct potential errors. Given code snippet ending with a legacy API invocation, the LLM is prompted to rectify it to the updated version. We assess the accuracy of API invocation correction using BLEU (Papineni et al., 2002), ROUGE- L. (Lin, 2004), Relative Edit Distance (Ristad & Yianilos, 1998), and CodeBLEU (Ren al.).
updated implementations, enabling explicit modeling of API evolution. Through this process, we synthesize 3,300 legacy- updated API invocation pairs from 3,300 real-world code instances. The detailed prompt is provided in Appendix C.
2.4. CODESYNCBENCH: A Benchmark for Real-Time Code Knowledge Assessment
Based on CODESYNC, we develop CODESYNc a real-time benchmark for assessing how effectively LLMs adapt to evolving code knowledge, which comprises three evaluation tasks, including Code Completion Task (CCT), Error Correction Task (ECT), and Multiple Choice Question (MCQ), as shown in Figure 4. CODESYNCBENCH covers updates for 220 APIs across 6 Python libraries, including 130 functions, 59 initializers, and 31 methods. Each API is associated with 15 legacy-updated invocation pairs (3,300
Multiple Choice Question (MCQ) (Nguyen et al., 2025). This task evaluates the LLM's ability to discriminate between correct and incorrect API invocations, requiring a deep internalization of the updated APIs. Given four candidate API invocations, including one correct answer and three plausible distractors, the LLM is prompted to select the optimal choice. The distractors, synthesized
5

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
Table 3: The performance of different LLMs in accessing API updates. We evaluate nine popular LLMs on CoDeSYNCBENCH, revealing their poor performance in API invocation tasks. The results highlight significant limitations in LLMs' ability handle updated APIs, with even sfae-oof-the-at models struggling to achieve high scores due to outdated knowledge. (BU for BLEU, RL for ROUGE-L, and RED for Relative Edit Distance)
Knowledge Cutoff Date
CCT
ECT RL1
MCQ P@31
LLM
BUT
RL1
RED
BUT
RED
P@ 1
P@51
Clased Source Models
GPT-4o
Oct. 2023
14.93
47.07
58.87
37.07
67.13
43.06
38,98
42.09
46.07
GPT-44-mini
Oct. 2023
7.45
32.39
67.14
33.69
51.06
49.54
29.58
34.63
35,58
Claude-3.5-Sonme
Apr. 2024
19.29
49.24
57.07
37.91
65.85
43.21
36.08
40.13
41.80
Semini-1.5-Prr
Nov. 2023
17.62
49.65
57.85
32.75
61.93
48.03
34.40
40.55
43.16
Open Source Models
DeepSeek-V3
Jul. 2024
19.24
44.13
57.67
51.57
62.64
34.12
31.54
34.41
35.78
DeepSeek-R1
Jul. 2024
19.32
44.09
57.54
51.81
62.76
34.05
31.61
34.41
35.78
Qwen2.5 5- 1-BB-----ttst
Mar. 2024
10.46
36.94
63.89
30.82
49.60
54.45
37.28
38.88
39.45
Qwen2.5 3-332BIInstrun
Mar. 2024
13.97
39.43
62.24
40.31
55.58
42.81
35.35
37.50
38.16
Mar. 2024
16.06
41.53
59.76
45.03
57.92
38.23
33.49
36.41
37.41
by DeepSeek- V3 (Liu et al., 2024a), include perturbations such as adding an invalid parameter, removing required parameter, and rearranging parameter order. We employ the Pass @k metric (Chen et al., 2021a) to measure the probability that the LLM passes test case within k attempts, which is calculated by drawing n 2 k answers from the LLM for each test case and counting the number of correct answers e S N. We use n = 10 and k E {1,3,5) (abbreviated as P@ 1, P@3, and P@5).
API update settings, e.g, the numbers of API invocations available for training and the rypes of updated APIs, impact the performance of knowledge updating?
3.1. RQ1: Benchmarking Large Language Models
We benchmark nine state-of-the-art LLMs in accessing real-time API updates without retrieval-augmeened settings, including four proprietary models (i.e., GPT-4o, GPT- 4o-mini (OpenAI, 2024), Claude-3.5-Somee (Anthropic, 2024) and Gemini- 1.5-Pro (Team al., 2024)) and five open-source models (i.e., DeepSeek-V3 (Liu al., 2024a), DeepSeek-RI (Guo et al., 2025), and Qwen2.5-11432/77BB Instruct (Qwen Team, 2024)).
Training Set. To evaluate knowledge updating methods, we build instruction tuning dataset D {(i, Oald. Onsw) )- As illustrated in Section F,1, denotes an update-aware instruction containing a code snippet with an incomplete API invocation (e.g.. "array=numpy.. full ("). Oold and Onew are output statements that accomplish the code. Onew represents the correct invocation with the updated API, while Oold reflects the legacy version. Oold and Onew share the same basic functionality, differing only in the parameters affected by the API update. The paired invocations allow the LLMs to identify update-related changes by computing teken-level differences between Oold and Onew-
As shown in Table 3, the results indicate that state-of- the-art LLMs face significant challenges coding tasks involving API updates. For example, leading commercial models like GPT-4o and Claude-3.5-Sonnet exhibit poor performance, with BLEU scores below 20% the code completion task. Similarly, recently released models with up-to-date knowledge cutoffs, such as DeepSeek-VV V3 and DeepSeek-R11 which are expected to incorporate fresher code knowledge, also fail to accurately reflect API updates, yielding similarly low BLEU scores. These findings reveal systemic shortcomings LLMs' ability to adapt evolving APIs, highlighting the fundamental limitations of static pretraining paradigms. Thus, even the latest models suffer from knowledge decay as API versions evolve over time.
3. Can LLMs Sync with Code Evolution?
To assess LLMS' ability synchronize with code evolution, we investigate the following Research Questions (RQs):
RQ1: Benchmarking Large Language Models. Can I.LMs access real-time API updates without relying retrieval-augmenter frameworks?
3.2. RQ2:
Benchmarking Knowledge Updating Methods
RQ2: Benchmarking Knowledge Updating Methods. Can LLMs be effectively and efficiently updated to synchronize with API changes using knowledge updating methods without compromising model utility?
We benchmark five knowledge updating methods including SFT-LoRA (Peng et al., 2023), DPO (Rafailov et al., 2023), SimPO (Meng et al., 2024), and ORPO (Hong et al., 2024), across five open- LLMs including three code- specific L.MMs (iee. Codel II-Isrtruct (Roziere et al.,
RQ3: Impact of API Update Settings. How do different
6

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
Table 4: The overall performance of different knowledge updating methods across five open-source LLMs. We train five models using different methods and evaluate their performance on BEBESYN and HumanEval. All methods demonstrate limited effectiveness on CODESYNCBENCH, (BU for BLEU, RL for ROUGE-I, RED for Relative Edit Distance, and CBU for CodeBLEU.)
CCT
ECT
MCQ
HumanEval
Method
BUT
RL1
RED1
CBUT
BUT
RL1
RED1
CBUT
P@If
P@31
P@st
P@1t It Ratio
Original
7.95
25.70
73.61
30.21
32.24
56.79
50.77
40.71
28.48
41.61
46.91
65.24
SFT-LoRA
12.17
34.59
68.76
32.32
26.63
44.81
57.15
42.85
32.83
47.55
53.21
62.80
96.26
DPO
24.45
52.94
57.12
39.24
46.24
64.87
42.99
49.75
33.39
45.61
50.05
61.59
94.41
ORPO
24.90
52.33
56.37
38.77
40.98
58.92
47.63
46.38
32.85
47.74
53.35
63.41
97.19
SimPO
24.81
52.90
56.88
39.67
45.15
65.51
42.90
51.02
33.14
44.35
48.69
63.41
97.19
Owen2.3-Coder
(eler--BBnn Instrrcctt
Original
5.89
21.56
76.58
29.41
11.64
26.78
71.81
32.68
32.56
41.28
44.57
82.32
SFT-LoRA
15.44
37.40
66.55
31.17
19.20
40.68
60,93
36.03
35.16
48.63
55.02
82.32
100.00
DPO
23.36
51.82
46.12
38.67
55.57
59.07
46.12
44.95
37.00
46,39
50.40
82.93
100.85
ORPO
21.47
48,17
53.43
37.06
56.92
50.20
53.43
40.62
35.42
48.64
54.70
81.71
99.26
SimPO
23.86
53.17
45.22
39.39
54.57
60.31
45.22
45.53
37.87
44.92
47.80
82.93
100.85
Llama-3.7
Original
5.99
22.45
75.70
28.94
17.68
40.98
63.41
35.18
29.08
54.39
66.28
62.20
SFT-LoRA
13.21
36.70
72.01
34.54
43.78
65.76
41.84
49.90
22.28
38.74
47.24
60.98
98.04
DPO
24.13
51.36
55.38
38.85
27.18
51.57
54.83
38.88
36.42
49.88
55.34
58.54
94.12
ORPO
21.55
44.19
60.62
37.12
24.27
42.21
62.09
36,81
31.47
50.30
58.74
60.37
97.06
SimPO
26.83
53.95
56.07
36.79
23.04
44.91
58.74
39.69
36.56
43.96
46.66
62.20
100,00
Original
8.44
28.25
73,20
30.22
18.11
37.71
64,45
35,86
10.89
24.79
33.24
38.41
SFT-LoRA
17.24
44.97
59.57
34.36
30.60
50.42
53.99
43.51
10.34
18.91
24.85
36.59
95.26
DPO
26.54
53.27
26.51
41.66
39.67
60.55
44.79
48.15
20.48
41.09
51.71
36.59
95.26
ORPO
24.37
50.70
54.61
40.01
36.06
55.69
49.00
46.35
18.07
39.17
51.26
35.37
92.09
SimPO
27.78
56.48
50.62
42.04
40.56
65.27
41.65
49.08
25.40
45.50
54.66
35.98
93.67
DeepSeek-Coder-6. Coder-6.
7..8.BBBnat Instruct
Original
5.97
22.55
75.51
29.03
30.07
53.11
52.20
41.69
31.25
24.29
43.60
72.56
SFT-LoRA
14.96
41.42
62.45
33.34
47.79
71.25
34.32
53.76
7.88
8.89
9.32
71.34
98.32
DPO
26.77
55.72
50.86
42.54
43.29
64.95
41.91
50.35
6.37
8.61
9.00
70.12
96.64
ORPO
28.39
56.99
49.23
42.47
43.77
64.86
41.32
48.70
7.02
7.79
8.04
68.29
94.12
SimPO
25.10
53.69
52.97
41.50
41.47
64.06
42.50
50.08
6.75
9.21
10.55
68.29
94.12
2023), Owen2.55Cooder-7BBhmee (Hui al., 2024), and DeepSeek 6 (.eerr....-Inmne In-ttuutt (Guo et al., 2024a)) and two (Dubey
Analysis of the model outputs reveals degraded enstucttoonfoolloimn capabilities, resulting in non-compliant responses. In contrast, other models maintain compliant outputs, indicating lack of robustness in this model.
to significantly lower scores.
LLMs (i.e., 1--aaa-..-----Innneet et al., 2024) and Srren2.5-7B-Insteee (Qwen Team, 2024)). Detailed experiment settings are listed in Appendix D.2.
Evaluation of Updating Effectiveness. As illustrated in Figure and Table 4, the results indicate that knowledge updating methods can improve LLMs' performance handling API evolution across the three evaluation tasks. Notably, fine-tuned LLMs with size 6. 6.7B-8B can achieve scores comparable to those of leading proprietary and open- source LLMs. such as Claude-3.5-Sonned with BLEU scores of 23.86%-31.59 on the CCT task. Despite these improvements, the absolute scores remain low, indicating that current methods are insufficient for effectively updating the code knowledge of LLMs.
Overall, while fine-tuning narrows the gap with larger models some cases, the persistently low scores reveal the limitations of existing approaches. Further advances (i.e. integrating structural code understanding or continual learning) required to reliably update LLMS' code knowledge without compromising their general capabilities.
Fine-grained Analysis on Qwen. Table 4 demonstrates that the evaluated models suffer from severe knowledge obsolescence issues. To assess models' intrinsic capabilities on code, we construct a variant of the CCT benchmark where the reference answer corresponds to outdated code knowledge. Performance this variant serves upper-hound estimate of models' code knowledge. As
Notably, the SeopSeek-Coder. (.deer-6.700 6.7B Instructt model exhibits an anomaly on the MCQ task, where fine-tuning leads

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
Table 5: Estimation of the Upper-Bound Performance on code knowledge.
Evaluation of Updating Efficiency. In addition to effectiveness, updating efficiency is crucial factor that may influence developers' adoption in practice. For each model, we recorded the training time required for four knowledge updating methods, as shown in Figure 5. The results indicate that SFT-LoRA is the most efficient method overall. Moreover, we can observe that, across all models, the training durations follow the pattern: SimPO DPO ORPO, indicating that ORPO is the least efficient and SimPO is the most efficient. Additionally, it can be seen that the training duration for ORPO exhibits relatively larger fluctuation, indicating instability in efficiency.
Model
Original
Best Method
Upper Bound
Qwen2.5 Qwen2.5-Coder
30.21 29.41
39.67 39.39
42.05
45.12
Table 6: Performance of RAG Baseline.
Method
CCT CBU
ECT CBU1
MCQ CBU1
Original
29.41
32,68
32.56
SFT
31.17
36,00
35.16
DPO
38.67
44.95
37.00
Evaluation of Model Utility Post-Updating. We evaluate the general utility of the I.LMS before and after updating using the widely used HumanEval benchmark (Chen et al., 2021b). For each problem, we sample 10 answers (i.e., = 10) and calculate Pass@ 1, Pass @3, and Pass@ 5 scores. To assess the impact of updating, we computed the ratio of the Pass@5 scores for models trained with various methods to those of the original model. The results show that most updating methods incurred a score loss of no more than 10%, indicating minor impact on the models' overall utility.
RAG
35.17
42,26
34.26
SFT+RAG
40.70
51.35
36,89
60
Owen2.5
Oeant Cudder
Time (min)
3.3. RQ3: Impact of API Updating Settings
We further investigate the impact of different API update settings such as the numbers of API invocations available for training and the types of updated APIs, on the performance of knowledge updating in API evolution tasks.
DPO
ORPO
SimPO
Figure 5: Efficiency of different knowledge updating techniques. We measure and compare the time consumption of four knowledge updating techniques across five models. We can observe that the training durations follow the pattern: SimPO DPO ORPO.
Impact of Update-Aware Instruction Number. To evaluate this, we filter 32 APIs from the original training set, each with more than 50 invocation samples, and construct four new training sets with 5, 10, 20, and 50 samples per API, respectively. We then train Qwen-225--BBInntrrucc using four knowledge updating techniques (i.e, SFT-LORA, DPO, ORPO, SimPO) on these sets and evaluate performance on the code completion task. As shown in Figure 6, using only 5 samples per API results in relatively poor performance. When the training sample number increases to 10 per API, the model demonstrates improved recall capabilities the updated APIs. Further increases in sample number lead to performance stabilization with minor additional gains. These findings suggest that a moderate number of samples is sufficient for LLMs to internalize new code knowledge, with 10 samples per API striking an optimal balance between effectiveness and efficiency.
shown in Table 5, select Ow InIstructt and Bwen2.5-Coder--7B In-trrrt for evaluation on CCT variant using CodeBLEU metric. The results indicate that current techniques fall short of the upper bound, underscoring the limitations in their effectiveness.
Furthermore, we introduce retrieval- auggeente generation (RAG) (Lewis al., 2020) as the additional baseline. We construct vector database to store all API signatures from the target library with eext-embedding---ae large (OpenAl) as the embedding model. Table 6 reports the performance of on CODES YNEEECH using CodeBLEU as the metric. Across three different tasks, RAG performs better than SFT but still falls short of DPO. The relatively limited performance can be attributed to its hit rate of only 60%. This reduced hit rate is largely caused by the presence of many similarly named APIs and the complexity introduced by the large number of APIs present in the code context. Notably, combining SFT and RAG achieves improved performance, demonstrating the potential benefits of integrating external retrieval with fine-tuning.
Impact of Updated API Type. We evaluate Qwen-2.5-7B. Instruct on the CCT task across different API types. As illustrated in Figure 7, clear trend can be observed among the three API types. The knowledge updating methods perform similarly on function APIs and initializer APIs yet exhibit significantly lower performance on method APIs. This discrepancy can be attributed to the intrinsic complexity
8

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
STM
30
Favation Intieleer
CPO
oero
5M00
Bleu Score (9)
20-
27.3
25.0
35-
22.5
26.0
0
17.5
15.0
15
20
40
Original
SPTYLoKA)
DPO
OKPO
SmpPO
Figure 6: Model performance with varying numbers of invocation instances per API. We divide the original training set into subsets containing different numbers of samples per API (5, 10, 20, 50). The (wwen..5-7BIInstruu is trained on these subsets and evaluated on the Code Completion Task. The result indicates that 10 samples per API is sufficient for injecting knowledge, keeping a balance between performance and efficiency.
Figure 7: Model performance on different types of APIs. We evaluate the performance of Swen-2.5-7B-Insttree trained using various techniques, as well as reference model, on different categories of APIs (functions, methods, and initializers). The results reveal significant differences in the models' capabilities across different categories. Notably, all models perform relatively worse on methods compared to functions and initializers.
knowledge model editing (KME) (Meng et al., 2022a; Hartvigsen et al., 2023; Meng al., 2022b)) aim to efficiently integrate new information. KME methods optimize specific neurons related to new knowledge with no performance degradation.
of method invocations, which typically involve class instantiations, object references, and dynamic method calls. Unlike function and initializer APIs that follow relatively straightforward invocation patterns, method APIS require LLMs to correctly infer object types, track dependencies, and manage class hierarchies. These additional layers of complexity increase the difficulty of accurately invoking API updates, making more challenging for LLMs to learn and apply correctly. Addressing these challenges may require more sophisticated knowledge updating strategies to improve LLMs' adaptability to complex code knowledge.
Data Synthesized by LLMs. LLMs are widely used to generate synthetic data for pretraining and fine-tuning (Liu et al., 2024b), covering diverse applications like multilingual QA (Riabi et al., 2021), chatbot conversations (Zhao et al., 2023; Zhang et al., 2024b), and data augmentation (Dai et al., 2025; Chung et al., 2023; Chen et al., 2024a; Pu et al., 2025; Huang et al., 2025). Synthetic benchmarks further require generated data to be diverse, accurate, and challenging (Chen et al., 2025; Wu et al., 2024), and are now used to evaluate emergent capabilities, such as trustworthiness (Huang et al., 2024; Ye et al., 2024; Gao et al., 2024) and multimodal reasoning (Zhang et al., 2024a; Bao et al., 2024; Chen al., 2024b; Fu al., 2025). We advance this area by proposing a synthetic benchmark integrating three challenging code generation tasks.
4. Related Work
LLMs for Code Generation. Both proprietary (OpenAI,
2024; Team et al., 2024) and open-source L. (Hui et al., 2024; Roziere et al., 2023; Guo et al., 2024a) have recently demonstrated strong code generation abilities, leading to Al- driven tools such as Copilot (GitHub, 2024) and Cursor (AI, 2024). However, these models often overlook the risks associated with outdated APIs. Existing benchmarks and studies either rely on synthetic API updates (Liu et al., 2024c) vaguely defined knowledge-cditing tasks (Li et al., 2024b), limiting their applicability. Our work addresses these gaps by benchmarking knowledge updating methods on real-world API changes.
5. Conclusion
In this paper, we introduce CODESYNC, an innovative data engine for constructing the structured benchmark CODESYNCBENCH
evaluate LLMs' ability in handling evolving code knowledge. Benchmarking the state-of-the art LLMs and popular knowledge update techniques, we find that LLMss struggle with rapid API evolutions. Furthermore, existing techniques are insufficient for effective code knowledge integration. This highlights the necessity for improved approaches to help models adapt to evolving code knowledge in dynamic environments.
Knowledge Updating for LLMs. LLMs prone to knowledge obsolescence, as retraining is computationally expensive. Knowledge updating methods (i.e. supervised fine-tuning (Liu et al., 2024c; Peng et al., 2023), reinforcement learning (Schulman et al., 2017; Meng et al., 2024; Rafailov et al., 2023; Hong et al., 2024), and

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
Acknowledgements
generate secure code by sidestepping outdated api pitfalls. arXiv preprint arXiv:2409. /6526, 2024.
This work is partially supported by the Major Program (JD) of Hubei Province (Grant No. 2023BAA024). Dongping Chen and Yao Wan are supported by the Fundamental Research Funds for the Central Universities (HUST:
Bao, H., Huang. Y., Wang, Y., Ye, J.. Wang. X., Chen, X., Elhoseiny, M., and Zhang. X. Autobench-v: Can large
vision-language models benchmark themselves? arXiv preprint arXiv:2410.21259, 2024.
62400001). We would like to thank all the anonymous reviewers for their insightful comments.
Ben Allal, L., Muennighoff, N., Kumar Umapathi, L., Lipkin, B., and von Werra, L. A framework for the evaluation of code generation models. https://githie www.://w...oobb.godeeeorrooooooe bigcode-
Impact Statement
In this paper, we present CODESYNC, an innovative data engine designed to systematically monitor real-world API specialized
harness, 2022.
changes and generate CODESYNC benchmark for assessing and improving LLMs' adaptability to API updates. This benchmark establishes a standardized evaluation framework for assessing the challenges posed by outdated API knowledge in LLMs. However, one limitation of our work the efficiency of collecting invocation instances. By enabling LLMs with real-time API adaptation capabilities, our work has the potential to significantly enhance developer productivity and drive advancements in software development, Al-driven coding assistants, and programming education.
Bi, Z., Wan, Y., Wang, Z., Zhang, H., Guan, B., Lu, F. Zhang, Z., Sui, Y., Jin, H., and Shi, X. Iterative refinement of project-level code context for precise code generation with compiler feedback. In Proceedings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 2336-2353, Association for Computational Linguistics, 2024.
Chen, D., Chen, R., Pu, S., Liu, Z., Wu, Y., Chen, C., Liu, B., Huang, Y., Wan, Y., Zhou, P., et al. Interleaved scene graph for interleaved text-and -mage generation assessment. arXiv preprint arXiv:241 17188, 2024a.
References
Chen, D., Chen, R., Zhang, S., Liu, Y., Wang. Y., Zhou, H., Zhang, Q.. Wan, Y., Zhou, P., and Sun, L. MIlm-as-a- judge: Assessing multimodal Ilm an-ass--judge with vision- language benchmark. arXiv preprint arXiv:2402.04788, 2024b.
AI, C. Cursor: An ai-powered coding assistant, 2024. URL https::/ //wts::/www.... cursor. so. Accessed: 2024-12-14.
Allamanis, M., Peng, H., and Sutton, C. A convolutional attention network for extreme summarization of source code. In Proceedings of the International Conference on PMLR, 2016.
Chen, D., Huang, Y., Wu, S., Tang, J., Zhou, H., Zhang, Q., He, Z., Bai, Y. Gao, C., Chen, L., Li. Y., Wang, C., Yu, Y., Zhou, T., Li, Z., Gui, Y., Wan, Y., Zhou, P., Gao, J., and Sun, L. GUI-world: A GUI-oriented dataset for multimodal L.LM-based agents. In Proceedings of the Thirteenth International Conference on Learning Representations, 2025. URL https: ptttts:://opereviieee net forum? Saaerummid-Qaaaee 11-1iddrraraarb22
Machine Learning, pp.
Allamanis, M., Brockschmidt, M., and Khademi, M. Learning to represent programs with graphs. In Proceedings of the 6th International Conference on Learning Representations, 2018
Alon, U., Brody, S., Levy, O., and Yahav, E. code2seq: Generating sequences from structured representations of code. In Proceedings of the International Conference on Learning Representations, 2018
Chen, M., Tworek, J., Jun, H., Yuan, Q. Pinto, H. D. O., Kaplan, J., Edwards, H., Burda, Y.. Joseph. N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint urXiv:2107.03377 2021a.
Alon, U., Zilberstein, M., Levy, O., and Yahav, E. code2vec: Learning distributed representations of code. In Proceedings of the ACM on Programming Languages, volume 3, pp. 1-29. ACM, 2019.
Chen, M., Tworek, J., Jun, H., Yuan, Q.. Pinto, H. D.D.D.O. O., Kaplan, J.. Edwards, H., Burda, Joseph, N., Brockman. G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0337 2021b.
Anthropic, A. Claude 3.5 sonnet model card addendum, 2024. URL https: /www. anthropic com/ news/
Chu, Z., Wan, Y., Li, Q., Wu, Y., Zhang, H., Sui, Y., Xu, G., and Jin, H. Graph neural networks for vulnerability detection: A counterfactual explanation. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 389-401, 2024.
Bai, W. Xuan, K., Huang, P., Wu, Q. Wen, J., Wu, J., and Lu, K. Apilot: Navigating large language models to
10

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. Gated feedback recurrent neural networks. In Proceedings of the International Conference on Machine Learning, pp. 2067-20755 2075. PMLR, 2015.
Graves, A. and Graves, A. Long short-term memory. Supervised sequence labelling with recurrent neural networks, pp. 37-45, 2012.
Gu, X., Zhang. H., Zhang, D., and Kim, S. Deep api learning. Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 631-642, 2016.
Chung. J. Y., Kamar, E., and Amershi, S. Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions. arXiv preprini urXiv:2306.041 0306004140, 2023.
Gu, X., Zhang, H., and Kim, S. Deep code search. In Proceedings of the 40th International Conference on Software Engineering, pp. 933-944, 2018
Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., and Wei, F. Knowledge neurons in pretrained transformers. arXiv preprint ar wxxv:2214.008600 2021.
Gui, Y., Wan, Y., Zhang, H., Huang, H.,, Sui, Y., Xu, G., Shao, Z., and Jin, H. Cross-language binary-sourge code matching with intermediate representations. In Proceedings of the 2021 IEEE International Conference on Software Analysis. Evolution and Reengineering (SANER). IEEE, 2022.
Dai, H., Liu, Z., Liao, W., Huang, X., Cao, Y., Wu, Z., Zhao, L., Xu, S., Zeng, F., Liu, W., et Auggpt: Leveraging chatgpt for text data augmentation. IEEE Transactions Big Data, 2025.
Dekoninck, J. Fischer, M., Beurer-Kellner, L., and Vechev, M. Controlled text generation via language model arithmetic. In Proceedings of the Twelfth International Conference on Learning Representations, 2024a. URL https: hattps://openrevierr /et/forunt id id-SLw9fp4yIs
Guo, D., Zhu, Q. Yang, D.. Xie, Z. Dong, K., Zhang. W., Chen, G., Bi, X., Wu, Y., Li, Y., et al. Deepseek-coder: When the large language model meets programming-tie rise of code intelligence. arXiv preprint arXiv:2401. 14196, 2024a.
Dekoninck, J., Fischer, M., Beurer-Kellner, L., and Vechev, M. Understanding large language models through the lens of dataset generation. 2024b.
Guo, D. Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q. Ma, S., Wang, P., Bi, X., et al. Deepseek- rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501.12948, 2025.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A. et al. The Ilama herd of models. arXiv preprint arXiv:2407.21788, 2024.
Guo, Q., Cao, J., Xie, X., Liu, S., Li, X., Chen, B., and Peng, X. Exploring the potential of chatgpt in automated code refinement: An empirical study. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pp. 1-13, 2024b.
Fried, D., Aghajanyan, A., Lin, J., Wang. S., Wallace, E., Shi, F., Zhong, R., Yih, S., Zettlemoyer, L., and Lewis, M. Incoder: A generative model for code infilling and synthesis. In Proceedings of The Eleventh International Conference on Learning Representations, ICIR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Hartvigsen, T., Sankaranarayanan S., Palangi. H., Kim, Y., Sarranaraaaaannnn
and Ghassemi, M. Aging with grace: Lifelong model editing with discrete key-value adaptors. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Proceedings of the Advances in Neural Information Processing Systems, volume 36, pp. 47934 47959. Curran Associates, Inc., 2023.
Fu, M., Peng, Y., Liu, B., Wan, Y., and Chen, D. Livevqa: Live visual knowledge seeking, arXiv preprinl arXiv:2504.05288 2025.
Gao, C., Zhang, Q. Chen, D., Huang, Y., Wu, S., Fu, Z., Wan, Y., Zhang, X., and Sun, L. The best of both worlds: Toward an honest and helpful large language model. arXiv preprint 2xXiv:2406003366 2024.
Hong, J., Lee, N., and Thome, J. ORPO: Monolithic preference optimization without reference model. In Al-Onaizan, Y., Bansal, M., and Chen, Y.N.. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11170- 11 11189, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
GitHub. Github code search.
com/ wwwpuree/cooeeeerreeee Accessed: 2025-01-30.
GitHub. Github copilot: Your pair programmer, 2024. URL https: httpp::////hhab.///////p hom/ features/ copi lot. Accessed: 2024-12-14
Huang, S., Xu, Y. Geng, M., Wan, Y., and Chen, D. Wikipedia the era 1lms: Evolution and risks. arXiv preprint arXiv:2503.02879 2025.

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
Huang, Y., Sun, L., Wang, H., Wu, S., Zhang, Q. Li, Y., Gao, C., Huang, Y., Lyu, W., Zhang, Y., et al. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561, 2024.
Li, X., Wang, S., Li, S., Ma, Yu, J., Liu, X., Wang, J., Ji, B., and Zhang, W. Model editing for llms4code: How far are we? arXiv preprint arXiv:241 110e22411166638 2024b.
Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74- 2004.
Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Dang, K., et al. Qwen2r5-coder technical report. arXiv preprint arXiv:2409. /2186, 2024.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B.,, Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical report. arXiv preprint arXiv:2412.19437 2024a.
Iyer, S., Konstas, I., Cheung, A., and Zettlemoyer, L. Summarizing source code using a neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 2073- 2083, 2016.
Liu, R., Wei, J., Liu, F., Si, C., Zhang, Y., Rao, J., Zheng, S., Peng, D., Yang, D., Zhou, D., et al. Best practices and lessons learned on synthetic data. arXiv preprint arXiv:2404.07500 2024b.
Jandaghi, P., Sheng, X., Bai, X., Pujara, J., and Sidahmed, H. Faithful persona-based conversational dataset generation with large language models. arXiv preprint arXiv:2312.10007, 2023.
Liu, 7. I.., Pandit, S., Ye, X., Choi, E., and Durrett, G. Codeupdatearena: Benchmarking knowledge editing on api updates. arXiv preprint urXiv:2407.06249) 2024c.
Jinng J., Wang, F., Shen, J., Kim, S., and Kim, S. A survey on large language models for code generation. arXiv preprint wwXiv:2406.00515 2024.
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C. B.. Drain, D., Jiang, D., Tang, D., Li, G., Zhou, L., Shou, L., Zhou, L.,, Tufano, M., Gong, M., Zhou, M., Duan, N., Sundaresan, N., Deng, S. K., Fu, S., and Liu, S. Codexglue: A machine learning benchmark dataset for code understanding and generation. CoRR, arXiv preprint arXiv:2102.044: 02.006664 2021.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktüschel, T., et al. netrieval-augmenter generation for knowledge- intensive nlp tasks. In Proceedings of the Advances in Neural Information Processing Systems, 2020.
Luo, Z., Xu, C., Zhao, P., Sun, Q. Geng, X., Hu, W. Tao, C., Ma, J., Lin, Q. and Jiang, D. Wizardcoder: Empowering code large language models with evol- instruct. In Proceedings of the Twelfth International Conference on Leurning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Li, B., Sun, Z., Huang, T., Zhang, H., Wan, Y., Li, G., Jin, Z., and Lyu, C. Ircoco: Immediate rewards- guided deep reinforcement learning for code completion. In Proceedings of the 32nd ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ESEC/FSE '24, pp. 182-203, Porto de Galinhas, Brazil, 2024a. ACM.
Meng. K. Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Proceedings of the Advances in Neural Information Processing Systems, volume 35, pp. 17359- 17372. Curran Associates, Inc., 2022a.
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J.. Liu, Q.. Zheltomozhskii, E., Zhuo, T. Y., Wang, T., Dehaene, O., Davaadorj, M., Lamy-Poirier, J., Monteiro, J. Shliazhko, O., Gontier, N., Meade, N., Zebaze, A., Yee, M.. Umapathi, L. K., Zhu, J., Lipkin, B., Oblokulov, M.. Wang. Z. V. R. M., Stillerman, J. T., Patel, S. S., Abulkhanov, D., Zocca, M., Dey, M., Zhang, Z., Fahmy, N., Bhattacharyya, U., Yu, W., Singh, S., Luccioni, S., Villegas, P. Kunakov, M., Zhdanov, F., Romero, M., Lee, T., Timor, N., Ding, J., Schlesinger, C., Schoelkopf, H. Ebert, J., Dao, T., Mishra, M., Gu, A., Robinson, J. Anderson, C. J., Dolan-Gavit, B., Contractor, D., Reddy, S., Fried, D., Bahdanau, D., Jernite, Y., Ferrandis, C. M., Hughes, S., Wolf, T., Guha, A., von Werra, L., and de Vries, H. Starcoder: may the source be with you! Trans. Mach. Learn. Res., 2023, 2023.
Meng, K., Sharma, A. S., Andonian, A., Belinkov, Y., and Bau, D. Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229, 2022b.
Meng. Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference- reward. arXiv preprint arXiv:2405. 14734, 2024.
Mou, L., Li, G., Zhang, L.., Wang, T., and Jin, Z. Convolutional neural networks over tree structures for programming language processing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
12

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
Nguyen, D. M., Phan, T. C., Hai, N. L.., Doan, T.- T., Nguyen, N. V., Pham, Q., and Bui, N. D. Q- CodeMMLU: A multi-task benchmark for assessing code understanding capabilities of codeLLMs. In Proceedings of the Thirteenth International Conference on Learning Representations, 2025. htts:: openrevien..eeeee net/ htte://ommmimddhhaaeee -ww.iiidcaaaauuuggg
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. In Oh, A., Naumann. T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Proceedings of Advances in Neural Information Processing Systems, volume 36, pp. 53728- 53741. Curran Associates, Inc., 2023.
Nguyen, T. D., Nguyen, A. T., Phan, H. D., and Nguyen, T. N. Exploring api embedding for api usages and applications. In Proceedings of the 2017 IEEF/ACM 39th International Conference on Software Engineering (ICSE), pp. 438 449. IEEE, 2017.
Ren, S., Guo, D., Lu, S., Zhou, L., Liu, S., Tang, D., Sundaresan, N., Zhou, M., Blanco, A., and Ma, S. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009. URL https://azwe http:///aaxxw..//////////// wwwp///g///./ooooooeee/////////o 10297.
OpenAL. 2oxtemmbedding--largge
URL https:
Riabi, A., Scialom, T. Keraron, R., Sagot, B., Seddah, D. and Staiano, J. Synthetic data augmentation for zero-shot cross-lingual question answering. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. Wi-t (eds.), Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 7016-7030, Online and Punta Cana, Dominican Republic. November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/201. emnlp-main.55 URL https: htt:://ccaantooogg org/ 0rg/2021.0 emnlp ma main. 562//.
//plat openai com/ dwcc/modees text-embeddingg Baaiingg----rrgge 3-large.
OpenAL. Chatgpt: conversational ai model, 2023, URL https:/ //tp:://chaat iimeaii.om..
OpenAl. GPT-4 Turbo and GPT-4 documentation.
htt.://fww......ooooooooo/ooooda and-gpt-4, 2024.
Accessed: 2025-01-30.
OpenAL. Hello GPT-4o. https: httpe://pppnaaiic.
momnlp-main...
index/hello- 40/400//2200 2024.
Ristad, E. and Yianilos, P. Learning string-edit distance. IEEE Transactions on Puttern Analysis and Machine Intelligence, 20(5):522-532, 1998. doi: 10.1109/34. 682181.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 31.3116 311-318, 2002.
Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023.
Schick, T. and Schütze, H.
Generating datasets with pretrained language models. arXiv preprint arXiv:2 2021.
Peng, D., Zheng, S., Li, Y., Ke, G., He, D., and Liu, T.-Y.
How could neural networks understand programs? In Proceedings of the International Conference on Machine Learning, pp. 8476-8486. PMLR, 2021.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms.
Pu, S., Wang, Y., Chen, D., Chen, Y., Wang, G., Qin, Q., Zhang, Z., Zhang, Z., Zhou, Z., Gong, S., et al. Judge anything: Milm as a judge across any modality. arXiv preprint arXiv:2503.17489 2025.
arXiv preprint /7Xiv: 2017.
Sun, Z.. Wan, Y., Li. J., Zhang. H. Jin, Z., Li, G., and Lyu, Sifting through the chaff: On utilizing execution feedback for ranking the generated code candidates. In Proceedings of the 39th IEEF/ACM International Conference on Automated Software Engineering, ASE 2024, Sacramento, CA, USA, October 27 November 1, 2024, pp. 229-244 221- ACM, 2024.
httml, Python. ast - abstract syntax trees.
https:
//docs... www.so.ythooo.oooo prttthon.org//3///
a. Accessed: 2025-01-30.
Python. inspect- - inspect live objects. fttps://docss
python. httg////ibrar//nneee Accessed: 2025-01-30.
html,, b.
Tao, Y., Dang, Y., Xie, T., Zhang, D., and Kim, S. How do software engineers understand code changes? an exploratory study in industry. Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, FSE '12
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https::////. ht/p/w/wwmmmmmm.m//..///.......a github.
to/blogg/wen22.5/
13

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
New York, NY, USA, 2012. Association for Computing Machinery. ISBN 9781450316149. 2393596.2393656.
Yamashita, R., Nishio, M., Do, R. K. G., and Togashi, K. Convolutional neural networks: overview and application in radiology. Insights into imaging, 9:61 629, 2018.
doi: 10.1145/
Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., Silver, D., Johnson, M., Antonoglou, I,, Schrittwiser, J., Glaese, A., Chen, J., Pitler, E., Lillicrap, T., and Angeliki Lazaridou, .... O. V. Gemini: A family of highly capable multimodal models, 2024.
Ye, J., Wang, Y., Huang, Y., Chen, D., Zhang, Q. Moniz, N., Gao, T., Geyer, W., Huang, C., Chen, P.-Y,, al. Justice prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.0277 2024.
Zhang, J., Huang, W., Ma, Z., Michel, O., He, D., Gupta,
T., Ma, W.-C., Farhadi, A., Kembhavi, A., and Krishna, R. Task me anything. arXiv preprint 2024a.
VenkataKeerthy, S., Aggarwal, R., Jain, S., Desarkar, M. S., Upadrasta. R., and Srikant, Y. Ir2vec: Llvm ir based scalable program embeddings. ACM Transactions on Architecture and Code Optimization (TACO), 17(4):1-27, 2020.
Zhang, Q., Gao, C., Chen, D., Huang, Y., Huang, Y., Sun, Z., Zhang, S., Li, W., Fu, Z., Wan, Y., and Sun, L. LLM- as-a-coauthor: Can mixed human-writter and machine- generated text be detected? In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the Association for Computational Linguistics: NAACL 2024, pp. 409- 436, Mexico City, Mexico, June 2024b. Association for Computational Linguistics. doi: 10. 186533/1120224 findings-naacl.22 URL https: http:///acaaaatooa/gg// org 22g/
Wan, Y., Zhao, Z., Yang, M., Xu, G., Ying, H., Wu, J., and Yu, P. S. Improving automatic source code summarization via deep reinforcement learning. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, pp. 397-407, 2018.
Wan, Y., Shu, J., Sui, Y., Xu, G., Zhao, Z., Wu, J., and Yu, P. Multi-modal attention network learning for semantic source code retrieval. In Proceedings of the 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 13-25. IEEE, 2019.
Zhang. S., Xiao, G., Wang. J., Lei, H., Liu, Y., and Zheng,
Peart: Automated repair of python api parameter compatibility issues. arXiv preprint vrXiv:2406.0383 2024c.
Wan, Y., Bi, Z., He, Y., Zhang, J., Zhang. H., Sui, Y., Xu, G., Jin, H., and Yu, P. Deep learning for code intelligence: Survey, benchmark and toolkit. ACM Computing Survey, 56(12), October 2024. ISSN 0360-0300. doi: 10. 1441/ 3664597.
Zhang, Z., Zhu, H., Wen, M., Tao, Y., Liu, Y., and Xiong, How python framework apis evolve? an exploratory study. In Proceedings of the 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 81-92, 2020. doi: 10. 10.1109/AANE44425 200.50088000.0
Wang, W., Zhang, Y., Sui, Y., Wan, Y., Zhao, Z., Wu, J., Philip, S. Y., and Xu, G. settorremmeet--aaning-eeeee source code summarization using hierarchical attention. IEEE Transactions software Engineering, 48(1):102- 119, 2020.
Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. (inthe) wildchat: 570k chatgpt interaction logs in the wild. In Proceedings of the Twelfth International Conference on Learning Representations, 2023.
Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y., Fu, J., Chen, W., and Yue, X. Opencodeinterpreterr Integrating code generation with execution and refinement. arXiv preprini arXiv: 14658, 2024a.
Wang, Y., Le, H., Gotmare, A., Bui, N. D. Q., Li, J., and Hoi, S. C. H. Codet5+: Open code large language models for code understanding and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 1069-1088. Association for Computational Linguistics, 2023.
Zheng. Y., Zhang, R., Zhang, J., Ye, Y., Luo, Z. Feng, Z., and Ma, Y. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) Bangkok, Thailand, 2024b. Proceedings of the Association for Computational Linguistics. URL //ap:iv/////vvv////aa//aa otgg/bs//400 2403. 13372.
Wu, S., Huang, Y., Gao, C., Chen, D., Zhang, Q. Wan, Y., Zhou, T., Zhang, X., Gao, J., Xiao, C., et al. Unigen: A unified framework for textual dataset generation using large language models. arXiv preprint arXi::2406.1896. 2024.
14

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
A. Comprehensive Related Works
Deep Learning for Code Intelligence. Neural language models have made remarkable progress in code intelligence (Wan et al., 2024), encompassing a variety of tasks including code summarization (Wan et al., 2018; Wang et al., 2020), code search (Gu et al., 2018; Wan et al., 2019), and code generation (Bi et al., 2024; Sun et al., 2024; Li et al., 2024a). A central challenge in code intelligence is the effective representation of source code as vectors. Substantial effort has been devoted to this, primarily through the design of deep neural networks in three main categories: sequential code tokens (c.g.. plain text, intermediate representations, APIs), Abstract Syntax Trees (ASTs), and code graphs (such as control-flow graphs, data-flow graphs, and code property graphs). For sequential code tokens, approaches have employed Recurrent Neural Networks (RNNs) (Graves & Graves, 2012; Chung et al., 2015; Gu et al., 2018) and Convolutional Neural Networks (CNNs) (Mou et al., 2016; Yamashita et al., 2018) to process plain text (lyer et al., 2016; Allamanis et al., 2016), intermediate representations (Venkata ct al., 2020; Peng ct al., 2021; Gui ct al., 2022), and API calls (Gu et al., 2016; Nguyen et al., 2017) extracted from source code. For ASTs, prior research has either developed structural RNNs (Wan et al., 2018) and CNNs (Mou et al., 2016) to capture the hierarchical structure of the tree linearized the AST into sequential traversals (Alon et al., 2019; 2018) for processing with traditional RNNs or CNNs. To handle code graphs, various Graph Neural Networks (GNNs) (Chu et al., 2024; Allamanis et al., 2018) have been proposed, enabling more sophisticated representations of code structure and semantics. Recently, advancements in LLMs for text generation have spurred the emergence of specialized code-focused L.Mss including CodeT5++ (Wang et al., 2023), InCoder (Fried et al., 2023), StarCoder (Li et al., 2023), Code Llama (Roziere et al., 2023), WizardCoder (Luo et al., 2024), Qwen-Codet (Hui et al., 2024), and DeepSeek-Code Coder (Guo et al., 2024a). Despite recent advances, LLMs still struggle to keep pace with rapidly evolving programming knowledge. This paper explores methods for integrating dynamic knowledge, enabling LLMs to synchronize with the ongoing developments in programming languages, frameworks, and best practices.
LLMs for Code Generation. Recently, LLMs such as the commercial/black-on GPT- (OpenAI, 2024), Gemini (Team et al., 2024), and open-source models like Qwen-Coder (Hui et al., 2024), Code Llama (Roziere et al., 2023), and DeepSeek- Coder (Guo et al., 2024a), have demonstrated impressive capabilities in generating high-quality code. Building on these LLMs, several products, including Copilot (GitHub, 2024) and Cursor (AI, 2024), have been developed. However, the security risks posed by outdated APIs are often overlooked, and existing studies on the code knowledge update task have significant limitations. For example, the benehmark proposed by Liu et al. (2024c) generates API update pairs by prompting ChatGPT (OpenAI, 2024) rather than collecting authentic APIs. Li al. (2024b) construct an instruction benchmark where the subject and object of knowledge are vaguely defined, but apply knowledge model editing techniques to model tuning. In this paper, we aim to benchmark knowledge updating methods for real-world API updates using authentic GitHub releases.
Knowledge Updating for LLMs. LLMs often rely data from specific time period, leading to outdated knowledge that retraining can not easily fix due to its high computational cost. To address this, knowledge updating techniques offer more efficient way to integrate new information without sacrificing the model's current capabilities. One approach is supervised fine-tuning (SFT) (Liu et al., 2024c; Peng et al., 2023), which optimizes model parameters to integrate new knowledge directly. Other methods treat new knowledge as preferred behavior over outdated information, such as reinforcement learning from human feedback (RLHF) methods (Schulman et al., 2017; Meng et al., 2024; Rafailov et al., 2023; Hong et al., 2024), which is efficient for refining model behavior to align with new knowledge. Knowledge neuron theory (Dai et al., 2021) takes a further step by formulating knowledge as tuple {s. r, ,0,, where s, r, and represent the subject, relation, and object of knowledge, respectively. Based on this, knowledge model editing (Meng et al., 2022a; Hartvigsen et al., 2023; Meng et al., 2022b) emerge as more cost- cost-effective and time-efficien approach for updating knowledge. These methods first identify key neurons linked to the new knowledge and then optimize them, carefully preserving the language model's 's overall capabilities. However, Li et al. (2024b) reveal that many KME techniques struggle with effectiveness and fail to generalize.
Data Synthesized by LLMs. LLMs have demonstrated an impressive capacity for data generation, leading to their application in creating synthetic datasets for pretraining and finetuning, replacing the labor-intensive processes of manual data scraping and selection (Liu et al., 2024b). Distinct from earlier methods that focus traditional language models (Schick & Schütze, 2021), LLMS offer enhanced prospects for producing high-quality synthetic data across a wide spectrum of applications, such as multilingual QA (Riabi et al., 2021), chatbot conversation (Zhao et al., 2023; Zhang et al., 2024b), and data diversity augmentation (Dai et al., 2025; Chung et al., 2023; Chen et al., 2024a). The concept of synthetic benchmarks takes a step further by demanding that the LLM ALM-generated data be diverse, accurate and systematically challenging (Chen et al., 2025; Wu et al., 2024). Moreover, synthetic benchmarks have also been constructed in evaluating
15

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
LLM emergent capabilities such as trustworthiness (Huang et al., 2024; Ye al., 2024; Gao et al., 2024), persona-based conversation (Jandaghi al., 2023), and multimodal domain (Zhang et al., 2024a; Bao et al., 2024; Chen et al., 2024b). Our research advances synthetic benchmark for code generation by developing a paradigm that integrates three challenging code generation tasks. Recently, in response to concerns about the quality of synthetic datasets, Dekoninck et al. (2024b) conduct comprehensive experiments to evaluate the diversity and fidelity of synthetic data produced by LLMs, while Dekoninck et al. (2024a) introduce a new inference framework, model arithmetic, to control the generated content.
B. Detailed Experiment Setups
B.1. Dataset
B.I.1. API COLLECTION
The initial step of CODESYNC pipeline involves collecting APIs from various libraries. To achieve this, utilize the Python built-in module inspect, which enables us to navigate through library files and compile a comprehensive list of all available APIs. In this part, we will delve into the detailed process of how to collect APIs comprehensively from libraries.
C-extension APIs. C-extension methods and functions are a powerful feature in Python programming that are employed in many third-party libraries, (e.g.. NumPy, PyTorch), to accelerate execution efficiency. One of the key feature of C-extension functions and methods is their support for function overloading. Function overloading allows a single API name to be used with multiple different parameter lists, or signatures. This means to collect various versions of signatures for each API.
Inspect Module. Python built-in module, Inspect, provides several useful functions for introspecting live objects, such as functions, classes, and modules. allows us to retrieve information about source code of Python objects, such as signature, arguments and documentation.
Categories. Python offers diverse range of APIs, each designed for specific purposes and governed by distinct invocation rules. In this study, we focus on three primary types: function APIs, method APIs, and initializer APIs. These categories not only highlight Python's ' core capabilities but also exhibit unique characteristics and behaviors. Function APIs are standalone entities that can be invoked without requiring class instance context. In contrast, method APIs are inherently tied to class instances, leveraging encapsulation and object-oriented programming principles. The invocation rules for methods differ significantly from those for functions, reflecting their object-oriented nature. Additionally, Python provides several magic methods that are denoted by double underscore ("__") at the beginning and end of their names. Among these, initializers (i.e., "_init_"" ") are the most commonly used, serving as method for object creation and initialization. To evaluate and benchmark Python APIs evolution comprehensively, we select representatives from these three categories to construct our benchmark CODESYNCBENCIL
B.1.2. IDENTIFYING API UPDATES
Multiple Types of Parameter. parameter and positional & keyword parameter. The term 'positional' refers to parameters that can be passed only according to its position in definition. 'Keyword' is the name of parameter in the function signature, allowing passing parameter with marking it explicitly instead of position. There are two special symbols in API signatures (e.g., *, /). Parameters set before ** are positional-only parameters, which must be passed in order according to theirs positions in definition, and parameters located after ' are keyword-only parameters, requiring marking parameter name when used; otherwise, a syntax error will occur. Additionally, parameters can be also categorized according to default values into 2 types, required parameters and optional parameters. Therefore, changes of parameter types have impact invocation rules, which should be considered when determining API update operations.
The three fundamental types of parameters are positioonal-oly parameter, keyword-only
API Update Determination. How to determine API update operations? The most straightforward changes include the addition or deletion of parameters. A more nuanced level of analysis involves examining changes in parameter types as these alterations can significantly impact the rules for invoking APIs. Therefore, API updates can be categorized into 2 primary aspects, the addition of deletion of parameters and changes in parameter types. To effectively identify API updates, it is crucial to focus on parameter changes, including both the mapping relationships between parameters and modifications to their types. To systematically capture these changes, we construct parameter mappings for each pair of APIs, establishing connections between corresponding parameters the outdated and latest version of their signatures. Specifically, parameter
16

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
mapping enables categorize two distinct aspects. First, if a parameter mapping can be successfully constructed, it implies that all parameters consistently present in both versions of signatures, indicating no additions deletions. Following this, the next step involves detailed examination of each parameter pair within mappings, focusing on comparing their attributes to identify any modifications or differences. This approach enables clear and structured understanding of how APIS involve over time.
Parameter Renaming. Statie analysis, however, has inherent limitations, especially in cases where parameter renaming occurs. It is challenging to infer changes in functionality solely based parameter names. For example, in use auth token, whereas the keyword of this parameter was token in version transforme formers=:.. 25.1. 1. In spite of the same functionalityy renaming makes it impossible to recognize their equivalence solely by analyzing signatures. In this process, we assume that keywords of parameters are strongly connected to their functionality. The similarity between keywords suggests the similarity of their functionality. Instead of excluding all of name modification situations, we first set a threshold and compute the keyword similarity scores to account for some simple modifications. Based on this, we will then construct parameter mapping according to keyword mappings for further explorations.
.7.0, the API trans Coranfffrrerr pipeliness get_task has parameter named
Establishing Parameter Mappings. However, the inherent complexity of Python API signatures poses significant challenges in accurately establishing parameter mappings. To address this, we establish three rules that must be satisfied to determine whether no modification has occurred. Python introduces two special symbols (*'' and **'), which divide parameters into three categories, positional-only, keyword-only and positional-and weyood parameters. Specifically, we construct three individual parameters mappings for these types of parameters and establish three rules that must be satisfied determine whether no modification has occurred.
Rule 1: Successful Parameter Mapping. A valid parameter mapping must be constructed, ensuring that both the number of parameters and their corresponding keywords remain identical across different signatures.
Rule 2: Type-Specific Consistency, Each parameter type must follow specific rules:
For positional-only parameters, the order of parameters in the function definition must remain strictly unchanged across signatures.
For keyword-only parameters, the parameter names (keywords) must remain consistent to preserve their correspondence.
- For positional-nnd-keywrrr parameters, both the order requirement and keyword consistency must be satisfied simultaneously.
Rule 3: Required vs. Optional Parameters. Parameters can be further categorized into two types: required parameters, which must be provided when invoking APIs, and optional parameters, which have default values. While revisions to default values are not considered API updates, the type of a parameter must remain unchanged.
These rules collectively provide practical methodology for evaluating parameter modifications and determining API consistency, which is crucial part of CODESYNC implementing completely autoamted pipeline.
1.1.3. API INVOKING INSTANCES CRAWLING
After obtaining updated APIs along with corresponding information, it is necessary to crawl API invocations from ground truth which will be used to inject API knowledge into LLM for further exploration. Actually, directly feeding signature to models for tuning is unlikely to be effective, and limited reflect comprehensive information, such as invoking rules, which is hard to be formulated. Therefore, we collect a large dataset of invocation instances to implicitly reflecting relative knowledge.
WorldWorld API Invocation. Synthesizing invocation completely relied on L.LM is a convenient method for constructing dataset. However, this method exists inherent limitations. For example, information implied in context of generated code is insufficient and the contextual scenario restricted to LLMs' embedded knowledge. The inevitable bias therefore poses challenges to comprehensively reflect authentic invoking rules and habits. Instead of synthesizing invocations, we try to crawl code from GitHub with the help of GitHub Code Search, Code Search Engine developed by GitHub to effectively aggregate repositories or files using regular expression. Additionally, We involve search templates as shown in B.1.3, to enhance the effectiveness of invocation retrieval
Search Templates. Python allows aliases declaration of import statements to simplify usage of third-party modules and APIs. In the authentic programming scenario, directly invoking APIs with full name fails align with developers'
17

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
programming habits. We therefore design a of templates for each library to expand searching scope. For example, while the module torch.nn nn. ffnctiniiion is imported, these statements might exist:
as F 2. from torch.nn import functional as F
1. import
For any field in the API name (a segment separated by dots), an alias can be assigned and there are two formats: import as and from import. Based on these characteristics, we can generate series of searching templates. Templates of torch. nn. f onal. softmax are shown below:
1. soach..m.funciional.sfftmm (directly match)
2. "import torch as" + www.ffuciioaal.oott..a....
3. "from torch import nn" www.fnctionassootmaa...
"import torch.nn as" + sofuuctiioaal.sftmaaa""
"from torch.nn import functional" " ".softmax"
6. "import forch.nn.fun on.functional as" + ".softmax"
7. "from forch.nn.ffut www..toonnnonnalllmmmmmommo import softmax"
In the second template, we match "import torch as" instead of "import torch". This is because when the module is imported without an alias (e.g., simply "import torch"), the full path For function APIs and initializer APIs, the above patterns can be directly applied for decomposition. We next utilize GitHub Code Search retrieve code that contains all segments for each template (with an upper limit of 500 files).
will be directly used in the code,
Different from function and initializer, method APIs requires further step due to dynamic binding mechanism. A method API can be divided into two parts: class name and method name. For example, torch. Tensor and shape are class name and method name of torch. Tensor. shape, respectively. In the most programming scenario, Python objects lack explicit type definitions. To align with subsequent procedures, we only take one specific situation into consideration where both type declaration and API invocation exist in the same file simultaneously. Searching templates can be applied on method APIs retrieval well, while additional segments, f" (method name) (".should be included. For API torch. Tensor shape, each template will include shape (". Explicit type declarations will be clarified in Appendix B.1.4.
1.1.4. LOCATING VALID API INVOCATIONS
After retrieving a dataset of files that contain relative substring of target API invocation, further filtering is required to identify code that genuinely invokes the target API. The following illustration is divided into two parts: function iiiitiaiizer APIs locating and method APIs locating.
Function / Initializer APIs Locating. Initializer APIs share similar invoking rules with those of function APIs. We can abstract syntax tree(AST) to analyze crawled files for locating the target API invocations. Specifically, this part contains two steps: (1) Alias Mapping: We scan the import statements and construct mappings between original library/module name and aliases. (2) Invocation Analysis: Based on alias mapping, we traverse the AST of files and analyze each invocation statement to determine whether the target API are invoked. The start & end line number of invocations will recorded for subsequent process.
Method APIs Locating, Invocations of method APIs often associated with class instances. To determine method API invocations, need to infer the types of variables that invoke the methods. However, variables dynamically bound to types during program execution. We therefore focus situations where the types of variables can be statically inferred from the raw code. There are three situations:
Variables assigned by using initializer of target class.
Type annotations are provided in function definitions.
Function definitions provide return type annotations.
18

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
The first step is to scan the whole file record types of variables as well their scopes. We next traverse the AST, tracking target class instances in their own scope to identify methods they invoked.
Format Conversion. After locating and recording API invocations in each file, we perform two operations to split the data: (1) Segment Split: Treating the entire file single dataset item is inefficient and redundant, To better utilize the crawled files, we split each file into multiple segments based function definition. In other words, each segment corresponds to complete function definition and is treated as individual dataset item. (2) Metadata Convert: Each segment is then further divided into three parts: code context, target sequence and code suffix. The code context is the prompt in subsequent tasks. To avoid knowledge leaking, the target sequence is the first invocation of target API within the segment. These split operations allow for more efficient processing and better representation of the code's 's structure, ultimately improving the dataset's usability for subsequent tasks.
B.2. Models
Owen-225-7B-Instruet. A 7-billion parameter instruction-tuned model designed for general-purposs tasks, offering robust performance across various applications by following user instructions effectively.
Qven-2.5-Codee-----nsrruuttt specialized 7-billion parameter model tailored for coding-related tasks, excelling in code generation, debugging, and understanding programming languages through instruction-following capabilities.
Mlama-3-8--Insruuee An 8-billion parameter instruction-tunge model built for versatile applications, providing strong performance in natural language understanding and task execution based on user instructions.
Badel.lama-7--Insttree A 7-billion parameter model fine-tuned for coding tasks, optimized for generating, analyzing, and refining code while adhering to user-provided instructions.
A 6.7-billion parameter model specifically designed for coding and programming tasks, leveraging instruction-tuning to deliver accurate and efficient code-related solutions.
KeepSeek-Coder-
B.3. Knowledge Updating Methods
B.3.1. DIRECT PREFERENCE OPTIMIZATION (DPO)
Traditional reinforcement learning algorithms (e.g., PPO (Schulman et al., 2017)) introduce reward models to guide LLMs to align with human preferences. While these methods exhibit superior performance in many fields, they suffer from extremely high computational costs and require large amount of training data to optimize policy of reward models. To accelerate the process of training, DPO directly optimizes the model's policy to align with human preferences by leveraging pairwise comparison data. Each data pair consists of preferred sample y and a dispreferred sample y, for given input x,. DPO adjusts the model to increase the likelihood of generating preferred outputs while reducing the probability of dispreferred ones. By implicitly encoding preference rankings into the objective function, DPO climinates the need for explicit reward modeling or complex reinforcement learning pipelines, offering a simpler and more stable training framework.
The key insight of DPO is to reframe preference learning as supervised likelihood optimization problem. Given preference pairs (x,, .yF). the objective maximizes the log- bikeiiieldoood difference between preferred and dispreferred outputs:
log To (x:) Tee (xx)
log Toly, (xx, Ter (x,)
LDPO Loggo
where a denotes the sigmoid function and कee represents the reference policy. This formulation ensures the model assigns higher probabilities to preferred responses relative to the reference policy while maintaining generation diversity through implicit regularization.
B.3.2. ODDS RATIO PREFERENCE OPTIMIZATION (ORPO)
ORPO introduce Odd Ratio to quantify the preference learning. Specifically, it enhances preference learning by explicitly optimizing the odds ratio between preferred and dispreferred responses. The loss function combines log-odds maximization
19

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
with Kl-divergence regularization:
To (y;
x;) A KL (TTal
CRRO
Σ log
(
x;)
where \ controls the regularization strength. This dual objective encourages preference alignment while preventing excessive deviation from the reference policy, addressing the exploratin-exploitaiion trade-off inherent in policy optimization. ORPO's probabilistic framing improves sample efficiency in low-data regimes and enhances robustness noisy preference labels.
B.3.3. SIMPLE POLICY OPTIMIZATION (SIMPO)
SimPO extends the paradigm DPO through architectural simplifications that enhance both training efficiency and alignment precision.
core, SimPO reinterprets the alignment task as a margin maximization problem, where model learns maintain specified quality gap between preferred and dispreferred responses. This is achieved through two synergistic mechanisms:
Dynamic Length Normalization: Traditional probability-bass rewards inherently favor longer sequences due to multiplicative probability chains. SimPO counteracts this bias by computing rewards as length-normalized token probabilities:
3 lvv
Ro(y)x)
log oogoo,,e B<t).
where the normalization factor ly] (response length) ensures equal contribution per token, preventing length based reward inflation. This design choice proves critical tasks requiring concise yet high-quality responses, such as technical question answering or summarization.
Adaptive Margin Enforcement: Rather than relying on fixed hyperparameters. SimPO implements an intelligent margin threshold that interacts with the reward difference ARg Roly (x) Ro(y la:):
LSiPPO Σ max (0, m A Ro(x=))
The margin mechanism creates three distinct learning phases:
1. Active Learning: When ARg gradients actively push the model to widen the reward gap
2. Saturation Control: Once A 2 m, gradient flow ceases to prevent over-optimization
3. Implicit Regularization: The margin automatically scales with batch statistics, adapting to varying preference strengths
By eliminating reference policy computations and reward modeling, SimPO achieves faster convergence while maintaining competitive performance. The margin-bases based objective automatically suppresses gradient updates when preference distinctions become clear, preventing overoptimization and reducing computational overhead. This makes SimPO particularly effective for aligning LLMs with limited computational resources.
20

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
C. Prompts
C.1. Prompt to Update Code Legacy
I will provide a code snippet as the context, followed by calling statement that contains target API call and suffix. Additionally, the latest and outdated function signatures of the API are accessible(refereed to as latest latest. signature and outdated signature). Your task to update the calling statement according to both the latest and outdated API function signatures, producing two distinct answers: the "latest answer" and the "outdated answer".
You must adhere the following guidelines:
Calling Statement Updates: Only update the calling statement based the given signatures, ensuring the functionality and correctness of the calls.
2. Include Required Parameters: The updated calling statements should include only the required parameters from the API signatures. Optional parameters should only be included they are explicitly used or necessary based on the provided code context.
Avoid Unnecessary Defaults: not include default values optional parameters unless they are explicitly mentioned in the code or necessary for functionality.
Reflect API Updates: Clearly showcase the differences between the latest and outdated APT signatures through your modifications.
Latest API Signature: [innauee_
Outdated API Signature: [outdated signature] Context: [context]
Statement: [target.seq]
suffix: [suffix]
C.2. Prompt Generate Wrong Choices for MCQ
I want to create multiple-choice question where, based on specific code context, we identify the most appropriate parameter
list for the target API. I will provide you with the following information:
AP I I_path:: The full name of the API
updated.ssimatture. The APT's new signature
burddted.ssinaturee The API's old signature
import: The import statements in the code
context: The preceding code context, ending with the target API's
updated code: The correct answer that matches the new signature
outdated_ccoe: The incorrect answer that matches the old signature
I want construct multiple-choice question four options. Among these, updated.com will correct option, and outdatedd drtdddddddd is one incorrect option have already provided. need to create two additional incorrect options hased on the differences between the new and old gggnatures-speciffclll the old signature. In other words, if the model only knows the old signature, it might be inclined to select these incorrect answers. Here four possible approaches for crafting these additional incorrect options:
options that would he "misleading" if a model is still relying on
1. Remove optional parameters from the correct
(that updated_comm codde.
2. Add some incorrect optional parameters, such as parameters that existed in the old signature but do not exist in the new one, parameters that appear in neither signature (the name of these parameters should not be like extra_param which be judged to eгror very casily).
3. Rearrange the positions of any positional parameters based on updated.code
4. Change parameter names, for example changing add (x: int) to something like add (2-3).
WARNING: Your two new incorrect options MUST differ from both updated code and outda wutdated.com code that give to you, as well as from EACH OTHER.
Output Format:
Provide your new incorrect options as your answer, without any other output.
For example:
21

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
**Cfrn*ff*** Your output
Option 1: (parama, paramB-123)
Option 2: (paramk="hello")
API.path: [APLpath]]
updateed_sinnature: [uinated_signnauree]
outdated siigattd.sggnature:::
import: [import]
context: [context]
updated.comm code: [updated.coml code]
outdated foutdated
22

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
D. Experiment Settings
D.I. Metrics
D.1.1. BLEU METRIC
The BLEU score is used to evaluate the quality of generated text by comparing it to one or more reference texts. is based on the precision of n-grams (contiguous sequences of words) in the generated text, with brevity penalty to penalize overly short outputs. The BLEU score calculated follows:
N
BLEU = BP cxp
Σ
Wh log Pr.
where
*BP is the brevity penalty, defined as:
[1 ((0-//))
ifc>
BP
c
Here, is the length of the candidate (generated) text, and r is the length of the reference text.
Pa is the n-gram precision, calculated
Number of matching n-grams in candidate and reference
Total number of n-grams in candidate
Wn, is the weight for the n-th n-gram precision, typically set to for uniform weighting.
N is the maximum n-gram order (usually 4 for BLEU-4).
The BLEU score ranges from 0 to 1, where indicates a perfect match with the reference text and 0 indicates no overlap with the reference text.
D.1.2. ROUGE METRIC
The ROUGE metric is used to evaluate the quality of generated text by comparing to one or more reference texts. focuses on recall, measuring how much of the reference text is captured by the generated text. ROUGE has several variants, including ROUGE-N (n-gram overlap), ROUGE-L (longest common subsequence), and ROUGE-W (weighted longest common subsequence). In experiments, use ROUGE-L the metric.
The ROUGE-L score is based on the longest common subsequence (LCS) between the candidate and reference texts. It is defined as:
ROUGE-L. LCS(C, R) Length(
where
LCSS(C, R) the length of longest common subsequence between the candidate text and the reference text R.
Length(R) is the length of the reference text.
The ROUGE score ranges from 0 1, where indicates that the candidate text perfectly captures the reference text and O indicates overlap with the reference text.
D.1.3. RELATIVE EDIT DISTANCE METRIC
The Relative Edit Distance (RED) is a normalized metric used to measure the dissimilarity between two strings. is
calculated as the edit distance (c.g., Levenshtein distance) between the two strings divided by the length of the longer string. This normalization ensures that the metric caale-invariant and ranges between 0 and
23

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
The RED defined
RED EdinDistance(S,,55 max([Si]. (S2))
where
EditDistance(S,. S2) is the Levenshtein distance between strings S1 and S2, which measures the minimum number of single-character edits (insertions, deletions, or substitutions) required to transform S into S2-
|S1] and [Sg] are the lengths of strings S and S2, respectively.
max (S2l) is the length of the longer string, used normalize the edit distance.
The RED score ranges from 0 to 1, where O indicates that the two strings are identical (no edits are needed) and indicates that the two strings are completely dissimilar (every character needs to be edited).
1.1.4. PASS @ METRIC
The Pass@k metric is performance evaluation metric used assess the quality of code generation models. measures the probability that at least one correct solution is generated within the top k samples produced by the model. This metric is particularly useful for evaluating models in scenarios where multiple candidate solutions generated, and the goal is to determine how often the model produces correct solution within limited number of attempts.
Given a set of n generated samples for problem, the Pass@ k metric is calculated as follows:
Pass@k Number of problems with at least one correct solution in the top k samples
Total number of problems
Alternatvely, if the model generates l samples per problem, the Pass @ k metric can be computed as:
(n-c)
Pass@k
E
Prroblens
k
where
n is the total number of samples generated per problem.
c is the number of correct solutions among the samples.
("r" is the number of ways to choose h samples that do not contain any correct solutions.
is the total number of ways choose k samples from
The k metric ranges from 0 to 1, where indicates that at least one correct solution is always found within the top samples and 0 indicates that no correct solution ever found within the top samples.
D.2. RQ2. Experiment Settings
In the process of RQ2, we train five open-source models using five knowledge update techniques, and evaluate trained models on BNCCCII this section, we show the detailed experiment settings as follows.
1.2.1. MODEL TRAINING
Knowledge Update Methods. Supervised Fine-Tuning (SFT) is a widely used and traditional method for modifying and aligning model knowledge, relying labeled data train models. For the SFT training dataset, the context in metadata serves as the prompt, and the updated dala serves as the target sequence. We also evaluate three instruction tuning methods (c.g.. DPO (Rafailov et al., 2023), ORPO (Hong et al., 2024), SimPO (Hong et al., 2024)) to update the knowledge, relying on positive-negativy data pairs to train models. For their training datasets, we use updated ccdde and outdated data as the positive and negative target sequences respectively. We use LoRA for all instruction tuning experiments (?) based on LoRA SFT on A800 servers.
24

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
We adopt five knowledge update techniques: SFT, SFT (LoRA), DPO, ORPO, SimPO. Additionally, LoRA training requires less computation resources and possessed of high efficiency.
We train DPO, ORPO and SimPO using LoRA techniques, which is more efficient than that of full training. We use LLaMA-Factor (Zheng et al., 2024b), a user-friendly and reliable automated tuning framework.
Hyperparameter.
Table 7: RQ2. Hyperparameters for Qwen2.5-7B-Instruct
Techniques
Epoch
Learning Rate
Warmup Ratio
Preference Beta
SFT
1.0e-4
0.1
SFT (LORA)
1.0c-4
0.1
DPO
3,5
5.0c-6
0.1
0.1
ORPO
3,5
5,0c-6
0.1
0.1
SimPO
3.5
5.0e-6
0.1
D.2.2. EVALUATION ON HUMANEVAL
We utilize the open-source project Code Generation LM Evaluation Harness (Ben Allal et al., 2022) to assess models on the HumanEval benchmark (Chen et al., 2021b). This evaluation framework provides a standardized method for measuring the code generation capabilities of LLMs.
For each evaluation, we generate 10 independent samples per problem across all 164 programming tasks in the benchmark. We then compute the Pass@1 1, @ass@3, and P metries, which measure the probability of generating a correct solution within the top 3, or model outputs, respectively.
To further analyze model performance, calculate the Pass@5 ratio between the trained models and the reference models. This comparison, visualized in Figure 2, serves diagnostic tool to monitor the effectiveness of our training experiments, The results indicate that all models perform on par with the reference models, suggesting that catastrophic forgetting is minimal. Moreover, our approach successfully injects new knowledge into the models without degrading their existing capabilities.
This evaluation provides strong evidence that our training strategy effectively balances knowledge retention and expansion, ensuring that models maintain their baseline performance while learning new information.
D.3. RQ3-1. Experiment Settings
Retrieving invocation instances for each API presents challenges due the limited number of available instances, which complicates the scaling of both training sets and benchmarks. In most cases, only have access small number of instances. On the other hand, limited sample size may lead underfitting, while larger sample size does not necessarily equate to better performance. In this section, we evaluate the impact of sample size on model performance.
To address this, we prepare series of training sets, each containing the same APIs but varying numbers of samples per API. Specifically, we explore four different sample sizes: 5, 10, 20, and 50, representing different very high.
medium, high, and
We construct these training sets from the original dataset. To control the experimental conditions, all four sets are derived include APIs that have more than 50 samples. We then randomly select
from the same set of APIs. Consequently, fixed number of samples for each API. To reduce sample quality variance, we ensure that the sets overlap. For example, the 5-sample set is fully included in the 10-sample set, and so on.
Next, we train the model Qwen2.5-78 Brren.55-B-Innttruee IBIInttuttt (Qwen Team, 2024) on these sets. Due to the limited size of the subsets, we double the number of epochs (which was set to 3 in Appendix D.2, and thus set to 6 for this experiment). To ensure convergence of the loss value, we adjust the relevant hyperparameters, as shown Table 8.
25

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
Table 8: RQ3-1. Hyperparameters for Qwen2.5-7B-Instruue across different training datasets.
Counts
Technique
Eval Steps
Learning Rate
Preference Beta
SFT (LORA)
30
1.0e-5
DFO
30
5.0e-6
0.3
ORFO
30
5.0c-6
0.1
SimPO
30
5.0c-6
0.7
SFT (LoRA)
50
1.0e-5
DFO
50
5.0c-6
0.3
10
ORFO
50
5.0c-6
0.1
SimPO
50
5.0c-6
0.7
SFT (LORA)
200
1.0e-5
DFO
200
5.0c-6
0.3
20
ORPO
200
5.0e-6
0.1
SimPO
200
5.0e-6
0.7
SFT (LORA)
500
1.0e-5
DFO
500
5.0e-6
0.3
50
ORPO
500
5.0e-6
0.1
$ SimPO
500
5.0c-6
0.7
D.4. RQ3-2. Experiment Settings
LLMs demonstrate varying capabilities across different categories of APIs. To align with RQ2 (see Appendix D.2), we evaluate the trained models from RQ2 on different subsets of CCT within CODESYNcBENCE Specifically, we categorize CCT CODESYNCBENCH into three distinct groups based on API types: functions, methods, and initializers. Each trained model is assessed separately these subsets analyze its performance across different API structures.
To ensure fair and robust evaluation, set the temperature to 0.9 and generate five output samples prompt account for variability in model responses. The model outputs are then compared against reference answers using BLEU scores, which serve as a metric for measuring output accuracy. The results of this evaluation are presented in Figure 7, providing insights into how model performance varies across API categories.
This analysis helps us understand whether LLMs exhibit strengths weaknesses in handling specific API types, offering valuable guidance for improving future models and fine-tuning strategies.
26

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
E. Data Format
E.1. MetaData Format
MetaData
[API] opech.opti..swee AveragedModel
[Code Context] def _ea__model_fronsstttee (statte_ddd input_dim=None)
model =
AveragedMoee (SNN ianppptt_diiiippt_dddll model. lotd__oodd_stteieie dim)
num_
[Updated Code] astate_dict, strict=True, assign=False)
[Outdated Code] (state _icatediic,, strict=True)
E.2. Training Data Format
E.2.1. SET TRAINING DATA
SFT Training data [instruction]
Flease fill the parameter list of apí
load__
according to the given context.
[input]
def Lla ot input_dim-None):
model = optim..wa_uttlss.eee Averagehdee (SNN (ingut_dimsinput__g nur_hidden__unttshhdddddddddn
model.1oad_statee
[output] (state_dict, strict-True, assign-False)
27

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
E.2.2. DPO/ORPO/SIMPO TRAINING DATA
DPO/ORPO/SimPO Training data [conversations]
[from] system
[value] Please complete subsequent API calling statement.
[from] human [value]
def Load_model_frret staressttte___e state_dictt astate_dict,
model htt.:/sswwwwtttsseeeeeeeeeeeeeee (SNN siinput_dim=ipput dim, num_hioden__uuttshhhdddddddddd
[chosen]
[from] gpt
[value] sstate_dict, strict-True, aslien
[rejected] [from] gpt
[value] (state_dict, strict=True)
E.3. Code Completion Task Format
[APT_path] flask. flask.json... [question]
def rest_json [est_json_dump_____ to_ tt__ttoffeeeeeleeeeeeeee (self):
app flask.Flast ( name
test_data name' "Flask')
out = StringIO()
with app.app_context ():
flask. jaon.jsso.dumg
[answer] (test_data, out)
E.4. Error Correct Task Format
[API_path) flask. json. dump [question]
def test_json_dume @dump_to_filee (self):
app = flask.Flasse (_mee____________
best_data = namam' "Flask') }
out = StringIO()
with app app.app_co ():
flask jump (token_data) file, app=None)
[answer] (token_data, file)
28

---

CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
E.5. Multiple Choice Question Format
[API_path) flask. json. dump [question]
def zest__soon_at dump_tt_t to tt__ttt_feeeee__eeeeeetettteeeet (self):
app flask. Flask name________
test_data = { n 'Flask') out = StringIO
with
():
jlask.json.do
[A] (test_data, out, app=app)
[B] (test_data, out)
[C] (test_data, out, app=app, indent=4)
[D] (test_ moss__dtt,, out, app=None)
[answer] B
29


Reviews:
[{"review_id": "3BfnNWtDE3", "reviewer": "Reviewer_HLiK", "summary": "The paper introduces CodeSyncBench, a benchmark to evaluate LLMs’ abilities to invoke the most recent versions of Python APIs. The benchmark shows that LLMs struggle to invoke APIs in the benchmarks correctly as measured by the chosen metrics. Further, the authors use various alignment techniques such as SFT, DPO and SimPO to fine-tune the LLMs to invoke the APIs correctly.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer HLiK,**\n\nWe sincerely appreciate your suggestions and assessment of our work! Motivated by your feedback, we are committed to improving our manuscript and providing a more comprehensive benchmark and evaluations!\n\n---\n**Q1:** How do the results of the paper change if you use static analysis methods, such as AST matching, etc. instead of string-matching approaches for evaluation?\n\n**A1**: Thanks for the suggestion. Actually, we have considered CodeBLEU to measure the effectiveness of API invocation updating from the perspective of AST matching. However, we found that, in practice, many LLM completions contained invalid syntax or natural language, causing them to fail the AST parsing (e.g., \\~70% of Qwen2.5-Coder completions on our benchmark failed AST parsing).\n\nAs a result, we compute CodeBLEU using a fallback strategy: if an API invocation passes AST parsing, we compute CodeBLEU against the ground truth; otherwise, we assign a corresponding part score of zero. The results for Qwen2.5-Coder are shown below:\n|Method|CCT BLEU↑|CCT CodeBLEU↑|ECT BLEU↑|ECT CodeBLEU↑|\n|---|---|---|---|---|\n|Original|5.89|29.41|11.64|32.68|\n|SFT|15.44|31.17|19.20|36.03|\n|DPO|23.36|38.67|55.57|44.95|\n|ORPO|21.47|37.06|56.92|40.62|\n|SimPO|23.86|39.39|54.57|45.43|\n\n> The experimental results align with trends from the original metrics, indicating that our conclusions remain valid under structure-aware evaluation.  (Full results are available at: https://anonymous.4open.science/r/CodeSyncRebuttal-D65F ).\n\n---\n**Q2:** Why do you think current methods are limiting? What’s the expected upper-bound on performance? What kind of improvements are you suggesting will improve performance further?\n\n**A2:** Thanks for your comments. Current knowledge-updating methods are limited in helping LLMs fully internalize and recall updated APIs, as evidenced by suboptimal CodeBLEU scores (<40) in the CCT task. The upper bound of performance can be quantified by evaluating LLMs’ ability to invoke legacy APIs. The empirical upper bounds for Qwen2.5 and Qwen2.5-Coder are as follows:\n|Model|Original|Best Updating Method|Upper Bound|\n|---|---|---|---|\n|Qwen|30.21|39.67|**42.05**|\n|QwenCoder|29.41|40.70|**45.12**|\n\n> The results indicate a significant performance gap with upper bounds. To close it, we recommend exploring hybrid approaches, as discussed in response to Q3.\n\n---\n**Q3:** How does the simple RAG baseline compare with the methods discussed in the paper? When or when not should someone use your method as opposed to RAG to update API invocation knowledge?\n\n**A3:** As stated in lines 104–109, unlike RAG which increases inference overhead by retrieval for each query, our CodeSyncBench focuses on evaluating and improving LLMs’ ability to internalize API updates. FT methods allows models to update without extra costs while inference, making them preferable in latency-sensitive or offline scenarios. Moreover, RAG's reliance on retrieval quality is problematic, especially with large real-world API updates. In contrast, CodeSync tracks API changes and integrates them into LLMs, enabling more reliable recall of updated API knowledge.\n\nTo address your concern, we test a RAG baseline with a vector database of 5,056 updated API signatures from six libraries mentioned in the paper, indexed with OpenAI’s `text-embedding-3-large` model.\n\n|Method|CCT CodeBLEU↑|ECT CodeBLEU↑|MCQ P@1↑|\n|---|---|---|---|\n|Original|29.41|32.68|32.56|\n|SFT|31.17|36.03|35.16|\n|DPO|38.67|44.95|37.00|\n|RAG|35.17|42.26|34.26|\n|SFT+RAG|**40.70**|**51.35**|**36.89**|\n\n> Results show that RAG performs lower than FT methods (35.17), primarily due to a retrieval success rate of only \\~60% (665/1100). However, RAG achieves better results (40.70), while combined with FT methods, highlighting the potential of hybrid approaches. In summary, knowledge updating methods outperform RAG in scenarios requiring efficient inference and precise API knowledge tracking.\n\nWe will incorporate these results and analysis into the revised manuscript.\n\n---\n**W4:** One concern that stands out for me is the lack of commentary on the leakage of samples from CodeSyncBench into the training data of models being evaluated.\n\n**A4:** To demonstrate that there is no data leakage between training data and benchmark, we compute the n-gram similarity of all samples associated with each API. Lower n-gram (<15%) scores indicate minimal overlap and thus confirm the absence of leakage. \n|N-gram|N=5|N=10|N=15|N=20|\n|---|---|---|---|---|\n|Score|12.62|10.21|9.24|8.52|\n\nSpecifically, after retrieving data for APIs, we duplicate all data and then split them into two parts. This approach has a clean separation and prevents any potential data leakage.\n\n**W5:** The paper does not include contemporary works also focusing on API invocations and benchmarks developed.\n\n**A5:** Thanks for the suggestion. We will include all the mentioned works in the next version of the paper.", "claims_and_evidence": "The claims are more or less backed by concrete evaluations (see comments below). One concern that stands out for me is the lack of commentary on the leakage of samples from CodeSyncBench into the training data of models being evaluated.", "methods_and_evaluation_criteria": "1. The metrics chosen for checking the validity of API invocations are dated and should no longer be used. It is well-known that these metrics (string match based criteria) suffer from not capturing the nuances of code syntax and semantics. I recommend the authors to instead use static analysis to verify the correctness of API invocations [1, 2, 3]. Measures like edit distance can be very high despite the model prediction being semantically equivalent to the ground truth.\n2. There can typically be multiple ways of invoking the same API to achieve the same outcome, e.g., `np.sum([a, b]) <=> np.sum([b, a])` but the paper does not call these out or comment on how such edge cases are being handled during evaluations.\n\n[1] Patil, Shishir G., et al. \"Gorilla: Large language model connected with massive apis.\" Advances in Neural Information Processing Systems 37 (2024): 126544-126565.\n\n[2] Ding, Yangruibo, et al. \"Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion.\" Advances in Neural Information Processing Systems 36 (2023): 46701-46723.\n\n[3] Jain, Nihal, et al. \"On Mitigating Code LLM Hallucinations with API Documentation.\" arXiv preprint arXiv:2407.09726 (2024).", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "1. The evaluations conducted in the paper rely on alignment methods such as SFT, DPO, etc. While different methods improve performance over the baseline, the results among the methods appear largely mixed, and no concrete conclusion can be made about which method helps improve performance the most. \n2. I believe that RAG must be considered as a baseline and is missing from the paper. If the experiment is out of scope, the paper should discuss when their method must be considered over RAG. It is now common knowledge to use RAG to update LLM knowledge and it’s unclear when the methods discussed in the paper would be adopted solely to update models’ knowledge about a few APIs.", "relation_to_broader_scientific_literature": "This paper is related to generating updated API invocations with large language models. It provides a new benchmark to study this problem and surveys several post-training methods to address this problem. The paper’s findings that post-training methods slightly improve performance on the task is aligned with common knowledge about LLMs.", "essential_references_not_discussed": "The paper does not disregard contemporary works also focusing on API invocations [1, 2] and benchmarks developed [3, 4]. Further, the paper does not consider RAG, a popular and simple approach to updating API invocation knowledge [5], in their evaluations.\n\n[1] Jain, Nihal, et al. \"On Mitigating Code LLM Hallucinations with API Documentation.\" arXiv preprint arXiv:2407.09726 (2024).\n\n[2] Patil, Shishir G., et al. \"Gorilla: Large language model connected with massive apis.\" Advances in Neural Information Processing Systems 37 (2024): 126544-126565.\n\n[3] Kuhar, Sachit, et al. \"LibEvolutionEval: A benchmark and study for version-specific code generation.\" *arXiv preprint arXiv:2412.04478* (2024).\n\n[4] Islah, Nizar, et al. \"GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models.\" *arXiv preprint arXiv:2411.05830* (2024).\n\n[5] Asai, Akari, et al. \"Self-rag: Learning to retrieve, generate, and critique through self-reflection.\" *The Twelfth International Conference on Learning Representations*. 2023.", "other_strengths_and_weaknesses": "The text in Section 3 is largely a regurgitation of the results in the tables. The reader is not provided meaningful insights, if any, that can be drawn from the tables in the paper. Finally, the paper says that existing knowledge techniques “show limitations” without diving deep into these. It would be useful if the authors could comment on the “further refinements” that are needed. In fact, this should form the core focus of the paper as other details (such as SFT, DPO, etc. will improve performance somewhat) are well-known facts.", "other_comments_or_suggestions": "Line 149 - thirt → third", "questions_for_authors": "1. How do the results of the paper change if you use static analysis methods, such as AST matching, etc. instead of string-matching approaches for evaluation? Incorporating these evaluations in your benchmarking package will improve trust in your benchmark.\n2. Why do you think current knowledge updating methods discussed in the paper are limiting? What’s the expected upper-bound on performance using the current evaluations and metrics? What kind of improvements are you suggesting will improve performance further?\n3. How does the simple RAG baseline compare with the methods discussed in the paper? When or when not should someone use your method as opposed to RAG to update API invocation knowledge?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CCAIB8Npqu", "reviewer": "Reviewer_oxwH", "summary": "The paper proposes a new benchmark called CODESYNC to address the issue that in the real world, library functions evolve over time while LLM code generation models are not updated. They source the API function updates from 6 real-world repositories: pandas, numpy, scipy, tensorflow, torch, and flask. They collect the API function updates by comparing the function signatures between the latest version and the version from Jan 2023, which is the ChatGPT release date. They also source the invocations of these functions from GitHub and use LLMs to create old-version invocation code from the new-version invocation code. The resulting 1100 legacy-updated pairs are used to create 1100 code completion, error correction, and multiple-choice questions. The other 2200 pairs are used as training data for updating LLMs' library function knowledge. The results are evaluated across many open and closed LLMs. They also finetune open code LLMs with SFT, DPO, ORPO, SimPO, etc. to see if the finetuned models can improve performance on the CODESYNC benchmark.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer oxwH,**\n\nWe sincerely appreciate your thoughtful review and your recognition of our work’s contributions to the community.\n\n---\n**Q1:** Why is the HumanEval score for Qwen2.5-Coder-7B-Instruct so low? The results from the Qwen team show that it achieves 88.4% on HumanEval.\n\n**A1:** In fact, the HumanEval score we reported for Qwen2.5-Coder was obtained using the `bigcode-evaluation-hardness` framework, which applies stricter evaluation criteria than those used by the Qwen team. Notably, it does not automatically add import statements and enforces stop tokens, leading to a more rigorous evaluation.\n\nTo ensure fairness, we have updated our evaluation setup to align more closely with the official criteria and re-evaluated all base and updated models. The revised scores are as follows:\n\n|Model|Base|SFT|DPO|SimPO|ORPO|\n|--- |---|---|---|---|---|\n|Qwen|65.24|62.80|61.59|63.41|63.41|\n|Qwen-Coder|82.32|82.32|82.93|82.93|81.71|\n|Llama|62.20|60.98|58.54|62.20|60.37|\n|CodeLlama|38.41|36.59|36.59|35.98|35.37|\n|DeepSeek-Coder|72.56|71.34|70.12|68.29|68.29|\n\nThe updated results align with trends from the original settings, indicating that our conclusions remain valid.\n\n---\n\n**Q2:** How is the training data related to the benchmark? Does every function change in the benchmark problem have a corresponding data point in the training set?\n\n**A2:** As stated in lines 255–262, for each API update, we collected 15 legacy-updated invocation pairs—10 for training and 5 for evaluation—ensuring that every API change in the benchmark has corresponding data points in the training set. However, this does not imply data leakage, as the training and benchmark data points use distinct code contexts, as discussed in our response to **Reviewer HLiK (W4)**.\n\n---\n**Q3:** Is it possible to include RAG + prompting baseline?\n\n**A3:** Thank you for the suggestion. We have included RAG baseline results in our response to **Reviewer HLiK (Q3)**.\n\n---\n**Q4:** How much human effort is needed for creating the synthetic legacy update code from the latest API invocation data?\n\n**A4:** As stated in Sec. 2.3, to ensure benchmark quality, two authors manually verified the divergence between legacy and updated API invocations synthesized by DeepSeek-V3. On average, verifying and guiding re-synthesis required \\~1 minute per invocation pair.\n\n---\n**W5:** The paper uses BLEU, ROUGE, and relative edit-distance scores for the code completion and error correction tasks. Not including exact match score or semantic-aware matching score (exact match with respect to parameter ordering) is not ideal.\n\n**A5:** Thank you for the suggestion. We have addressed this concern by incorporating AST-based semantic matching in our evaluation in our response to **Reviewer HLiK (Q1)**.\n\n---\n**W6:** The paper only focuses on function signature changes, whereas API changes may also include changes to function semantics.\n\n**A6:** While API updates that involve purely semantic changes without signature modifications do occur, they are relatively uncommon and undocumented. Thus, our benchmark mainly focuses on API signature changes, as these typically reflect substantial semantic modifications and can be systematically tracked. \n\n---\n**W7:** Another limitation of the benchmark is the fixed legacy version (Jan 2023), which may favor LLMs with later knowledge cutoff dates.\n\n**A7:** While LLMs with later knowledge cutoffs may have greater exposure to recent API updates, this does not contradict the benchmark’s objective—evaluating LLMs' ability to access and adapt to real-time API updates. In fact, this setup reflects a realistic scenario for assessing up-to-date knowledge. Nevertheless, even models like DeepSeek-R1 with recent cutoffs (July 2024), achieve low BLEU scores (e.g., 19.32), indicating limited effectiveness in handling evolving APIs. This further supports the need for explicit knowledge updating methods.\n\n---\n**W8:** The paper only includes 6 popular Python libraries, which may be a limitation of the benchmark.\n\n**A8:** Although CodeSyncBench currently focuses on six widely-used Python libraries, our proposed CodeSync engine is generalizable. It can track API updates across any Python library over arbitrary time periods. Thus, the benchmark can readily be extended by broadening the version range and including more libraries as needed. We leave the extension to our future work.", "claims_and_evidence": "- The paper claims to propose a data engine that systematically collects real-time API updates from various Python third-party libraries, and they show that they can use the method to collect many real-world API updates.\n- The paper claims to propose a novel benchmark for evaluating the performance of LLM code generation models, which is indeed different from previous API-update benchmarks like CodeUpdateArena.\n- The paper claims to have comprehensive evaluation, and they did show model evaluations across many open and closed LLMs.", "methods_and_evaluation_criteria": "The paper uses BLEU, ROUGE, and relative edit-distance scores for the code completion and error correction tasks. Not including exact match score or semantic-aware matching score (exact match with respect to parameter ordering) is not ideal.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The design of the benchmark heavily sources from existing real-world repos instead of relying too much on synthetic data like the previous related work such as CodeUpdateArena, which makes it valuable to measure the real-world API update impact on LLM code generation. However, the paper only focuses on function signature changes, whereas API changes may also include changes to function semantics. Another limitation of the benchmark is the fixed legacy version (Jan 2023), which may favor LLMs with later knowledge cutoff dates. Finally, the paper only includes 6 popular Python libraries, which may be a limitation of the benchmark.", "relation_to_broader_scientific_literature": "The paper is related to previous work on API update benchmarks like CodeUpdateArena. It also relates to knowledge update work in the NLP literature.", "essential_references_not_discussed": "There are many many knowledge editing work in NLP which is not discussed.", "other_strengths_and_weaknesses": "The use of real-world API changes and invocations makes this a very valuable benchmark for evaluating the performance of LLMs in the midst of real-world API changes. However, the paper only focuses on function signature changes, whereas API changes may also include changes to function semantics. Another limitation of the benchmark is the fixed legacy version (Jan 2023), which may favor LLMs with later knowledge cutoff dates. Please also see the questions below.", "other_comments_or_suggestions": "Please see the questions below.", "questions_for_authors": "- Why is the HumanEval score for Qwen2.5-Coder-7B-Instruct so low? The results from the Qwen team show that it achieves 88.4% on HumanEval.\n- How is the training data related to the benchmark? Does every function change in the benchmark problem have a corresponding data point in the training set?\n- Is it possible to include RAG + prompting baseline?\n- How much human effort is needed for creating the synthetic legacy update code from the latest API invocation data?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p65v3CRR6e", "reviewer": "Reviewer_Jxkj", "summary": "This paper introduces CodeSync, a data engine for identifying outdated code patterns and collecting real-time code knowledge updates from Python third-party libraries. Building upon CodeSync, the authors further develop CodeSyncBench, a comprehensive benchmark for assessing LLMs’ ability to stay synchronized with code evolution, which covers real-world updates for 220 APIs from six Python libraries. Extensive experiments on 14 state-of-the-art LLMs reveal that they struggle with dynamic code evolution, even with the support of advanced knowledge updating methods (e.g., DPO, ORPO, and SimPO). This benchmark offers a strong foundation for the development of more effective methods for real-time code knowledge updating in the future.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer Jxkj,**\n\nWe would like to express our sincere gratitude for your thoughtful and constructive feedback. We have addressed all of the comments and thoroughly presented our most recent experimental findings.\n\n---\n**Q1:** The paper provides limited details on how 220 APIs are selected from the 6,036 tracked updates. What specific criteria are used for filtering? More details on building the dataset would be beneficial to assess the benchmark's representativeness and reliability.\n\n**A1:** As detailed in Appendix B.1, we filtered out APIs with insufficient real-world usage. Specifically, we selected 220 APIs that each had at least 15 retrieved real-world invocation examples.\n\n---\n**Q2:** Since libraries continue to evolve, how will CodeSyncBench remain up to date? Is there a plan for periodic updates, or will the dataset become outdated as newer API changes emerge?\n\n**A2:** Yes, we plan to maintain and periodically update CodeSyncBench to reflect ongoing API changes, ensuring it remains a relevant and reliable benchmark for evaluating LLMs' ability to stay synchronized with evolving API knowledge over time.\n\n---\n**Q3:** What is the \"Parameter Mapping\" that is referred to in Appendix B.1.2. Would this method include some APIs that are not updated?\n\n**A3:** As described in Appendix B.1.2, parameter mapping is a standard technique in software engineering for detecting API signature changes. For each pair of APIs across versions, we analyze:\n- Parameter types: positional-only, keyword-only, or both\n- Parameter attributes: required vs. optional\n\nWe then construct a mapping from old-version parameters to their corresponding new-version parameters. If no mapping can be established—for example, due to newly added or removed parameters—this indicates an update. Additionally, if mapped parameters differ in type or attribute, we classify the API as changed. We will clarify this in the next version of our manuscript.\n\n---\n**Q4:** Could the authors provide a clear and detailed explanation of the process for identifying valid method API invocations, given that method APIs often involve dynamic binding?\n\n**A4:** As stated in Appendix B.1, we initially collected 6,036 updated APIs and retrieved invocation samples for each. The next step is to identify target API invocations, as data retrieval relies on string match. We then analyze the code context to determine the actual API calls. For Method-Level APIs, precisely identifying their invocations through static analysis alone is extremely challenging. Therefore, we limit our focus on cases where parameter types can be inferred from the code context. Specifically, we track the dataflow and control flow of each instance. Then, we bind class type and identify method calls for each. This method makes sure to capture actual API calls. \n\n---\n**W5:** This paper assumes that LLMs should always be updated with the latest API versions. However, it may not align with real-world development needs, since many projects continue using older versions for stability and compatibility. If CodeSync only fine-tuned on the latest API changes, it might lead to over-specialization.\n\n**A5:** Thanks for your comments. We think there may be a misunderstanding here. Actually, *CodeSyncBench* selected the current time as a snapshot to highlight the importance of API evolution. However, our framework is highly flexible—users can specify any preferred version ranges of a library to generate customized benchmarks. As we have shown below, users can set different start times and deadlines of API evolution for each library. The training set generated by CodeSync helps LLMs to synchronize with target API knowledge. Therefore, in the real-world development scenarios, users can design their own training sets and benchmarks and fine-tune models using LoRA methods for further application and exploration.", "claims_and_evidence": "The paper introduces an interesting benchmark to evaluate the synchronized ability of LLMs for dynamic code evolution. The experiments conducted in Section 2.4 provide convincing evidence to support the proposal.", "methods_and_evaluation_criteria": "The pipeline of constructing the dataset and the evaluation criteria of the benchmark seem reasonable.", "theoretical_claims": "No", "experimental_designs_or_analyses": "I have checked the experimental designs and analysis", "relation_to_broader_scientific_literature": "The presented novel dataset and benchmark for dynamic code evolution can serve as a complement to the existing progress on LLMs for code intelligence.", "essential_references_not_discussed": "The related works are sufficiently discussed in this paper.", "other_strengths_and_weaknesses": "Pros:\n1. This paper is well presented with a clear structure.\n2. The studied problem of synchronizing LLMs with dynamic code evolution is well motivated and interesting.\n3. The experiments conducted in this paper are comprehensive, and several interesting findings are delivered in this paper.\n\nCons:\n1. This paper assumes that LLMs should always be updated with the latest API versions. However, it may not align with real-world development needs, since many projects continue using order versions for stability and compatibility. If CodeSync only fine-tuned on the latest API changes, it might lead to over-specialization.\n2. As LLMs generate legacy-updated API pairs, they may introduce errors. If the datasets include incorrect API updates, this could mislead LLM training and evaluation, compromising the reliability of the benchmark.\n3. Library evolution is a continuous process that not only involves modifying existing APIs but also includes introducing new functionalities and deprecating outdated APIs. The paper discusses API changes but does not explain how CodeSync deals with APIs that are removed without direct replacements (deprecated APIs).", "other_comments_or_suggestions": "No", "questions_for_authors": "1. The paper provides limited details on how 220 APIs are selected from the 6,036 tracked updates. What specific criteria are used for filtering? More details on building the dataset would be beneficial to assess the benchmark's representativeness and reliability.\n\n2. Since libraries continue to evolve, how will CodeSyncBench remain up to date? Is there a plan for periodic updates, or will the dataset become outdated as newer API changes emerge?\n\n3. What is the \"Parameter Mapping\" that is referred to in Appendix B.1.2. Would this method include some APIs that are not updated?\n\n4. Could the authors provide a clear and detailed explanation of the process for identifying valid method API invocations, given that method APIs often involve dynamic binding?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers agreed that this paper presents a useful dataset and positive results from its tuning experiments. They noted some concerns around the limited evaluation, in response to which the authors provided substantial expansions to the evaluations & results. This is appreciated and strengthens the work sufficiently to recommend its acceptance.
Decision: Accept (poster)