CursorCore: Assist Programming through Aligning Anything
Authors: Hao Jiang
Qi Liu
Rui Li
Shengyu Ye
Shijin Wang
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models for Code
AI-Assisted Programming
Primary area: applications

Abstract: Large language models have been successfully applied to programming assistance tasks, such as code completion, code insertion, and instructional code editing. However, these applications remain insufficiently automated and struggle to effectively integrate various types of information during the programming process, including coding history, code context, and user instructions. In this work, we propose a new framework that comprehensively integrates these information sources, collect data to train our models and evaluate their performance. Firstly, to thoroughly evaluate how well models align with different types of information and the quality of their outputs, we introduce a new benchmark, APEval (Assist Programming Eval), to comprehensively assess the performance of models in programming assistance tasks. Then, for data collection, we develop a data generation pipeline, Programming-Instruct, which synthesizes training data from diverse sources, such as GitHub and online judge platforms. This pipeline can automatically generate various types of messages throughout the programming process. Finally, using this pipeline, we generate 219K samples, fine-tune multiple models, and develop the CursorCore series. We show that CursorCore outperforms other models of comparable size. This framework unifies applications such as inline chat and automated editing, contributes to the advancement of coding assistants.

CursorCore: Assist Programming through Aligning Anything
Hao Jiang'
Qi Liu
Rui Li
Shengyu Ye Shijin Wang 3
Abstract
Liang et al., 2024; Yang et al., 2024). The technology mainly takes two forms. One form involves completing specified code snippet at the end or inserting corresponding code at designated position, typically accomplished by founda- tion models (Chen et al., ;021; Bavarian et al., 2022) that support relevant input formats. The other form involves gen- erating or editing code snippets based on natural language instructions or reflections through interaction with the en- vironment, usually carried out by instruction models that have been further aligned (Shinn et al., 2023; Cassano et al., 2023b; Muennighoff et al., 2024; Paul-Gauthier, 2024). Fig- ure shows simple examples of these forms.
Large language models have been successfully applied to programming assistance tasks, such code completion, code insertion, and instruc- tional code editing. However, these applications remain insufficiently automated and struggle to effectively integrate various types of information during the programming process, including cod- ing history, code context, and user instructions. In this work, we propose new framework that com- prehensively integrates these information sources, and collect data to train models and evaluate their performance. Firstly, to thoroughly evaluate how well models align with different types of informa- tion and the quality of their outputs, we introduce a new benchmark, APEval (Assist Programming Eval), comprehensively assess the performance of models in programming assistance tasks. Then, for data collection, we develop data generation pipeline, Programming-IIstuun Instruct, which synthe- sizes training data from diverse sources, such as GitHub and online judge platforms. This pipeline can automatically generate various types of mes- sages throughout the programming process. Fi- nally, using this pipeline, we generate 219K sam- ples, fine-tune multiple models, and develop the CursorCore series. We show that CursorCore out- performs other models of comparable size. This framework unifies applications such as inline chat and automated editing, contributes to the advance- ment of coding assistants.
However, in practical applications, neither the completion or insertion mode nor the instruction based mode per- fect. The completion or insertion mode generates based on the current code context, but in actual coding, tinuously editing the code rather than just completing and inserting. We prefer that the model predicts the upcoming edits, as neither completion nor insertion accurately reflects the coding process, and requires programmers to perform additional operations. The instruction- based mode allows for code editing, but it also has drawbacks, such as writing prompts for specific tasks may be slower challenging. The process is not automated enough, programmers would prefer model that can proactively predict future changes without needing extra prompts. In our view, the core is- sue lies in the limitations of the input and output in both forms of programming assistance. These forms either just align the output with the current code context, limiting com- pletion insertion instead of editing, or align the output with the user's natural language instructions. However, to effectively assist with programming, an AI programming assistant needs to utilize anything throughout the program- ming process. should be capable of aligning with the history of code changes, the current content of the code, and any instructions provided by the user, predicting the required responses and corresponding changes, reducing any actions required by users.
con-
1. Introduction
Since the rise of large language models (LLMs), Al-assisted programming technology has developed rapidly, with many powerful LLMS being applied in this field (Zan et al., 2022;
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China *Institute of Artificial Intelli- gence, Hefei Comprehensive National Science Center FiFLYTEK Co., Ltd. Correspondence to: Qi Liu <qiliuql@ caaiing@@utceediii edu.......u........
To solve these issues, in this paper, we introduce new framework of Al-assisted programming task: Assistant- Conversation to align anything during programming pro- cess. To comprehensively evaluate the alignment of models with different information in the programming process and the quality of the corresponding outputs, we propose new
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

CursorCore: Assist Programming through Aligning Anything
Complete
Assist (Ours)
function(s):
frequency[e] frequency[e]
Hi
History (H): Code editiig history
frequency = (c: 5. count(()) for c in set(s))
def function(1):
return frequency
for c in 5:
Frequency[c) Frequency[c]
Tnsert
Current C Context (C):
Function(s)::
def function(s):
Hs
Frequency s.count(c)
set(s)1A
Current code context
frequency
Frequency[c]
@ef Function(s):
ser Instruction (U): Prompt or feedback
Write a function that calculates the fregunccy each character in a string using Python.
Implement concisely.
def function(s)::
Here Python implementation:
frequency = (c: c.cont(()) for с in set(s))
Function(s):
return #requency
ssistant Output (A): Outgut of the model
Frequency [e: s.count(e) for set(s)) return frequency
We dict compechension to count the occurences.
is more concise readable compared before.
Figure Different forms of programming assistance. The common uses of current LLMs are shown on the left. Our framework shown on the right.
benchmark, APEval (Assist Programming Eval), com- prehensively assess the performance of models in assisting framework,
information during programming streamline work for programmers. By precisely defining various types of infor- mation and their formats. Assistat-Coonversation directly aligns with the input and output requirements of applications such as automated editing and inline chat. This framework facilitates model alignment, enabling fast and accurate gen- eration and parsing.
programming. For the Assistant-Conversann we build a data generation pipeline, Programming- Instructt to synthesize corresponding training data from various data sources. This data generation method can produce any types of messages throughout the programming process, without any additional human annotation and does not rely on spe- cific models. We use it to generate 219K data points and use them to fine-tune multiple models, resulting in the Cursor- Core series. These models achieve state-of-the-art results when compared with other models of comparable size.
2.1. Framework Formulation
We introduce the elements of Cosstaat-Conversationn Sys- tem (S), History (H), Current Context (C), User Instruction (U), and Assistant Output (A). A represents the output of the model, while the inputs consist of S, H, C, U. Figures I and 2 shows several examples of them. These definitions will be referenced throughout the rest of this work.
In conclusion, our main contributions are:
Sesistant-Conversalie A new framework to align
anything during programming process.
Programming-Instreen Data synthesis pipeline to pro- duce any types of messages throughout the program- ming process, and 219K data collected using it.
System S (Optional) The system instruction provided to the model at the beginning, which configures the answering style, overall task description and other behaviors. this work, we fix to a simple "You are a helpful programming assistant." " and omit it from the subsequent discussion.
APEval: A comprehensive benchmark for assessing the ability to utilize various types of information to assist programming.
History If (Optional) The program's 's editing history, con- sisting of multiple pieces of code. These may include several snippets or may not be present at all. We refer to them as H Hn,
CursorCore: One of the best model series with the same number of parameters for Al-assisted program- ming tasks.
Code, models and data are freely available at https:// wtt.hu/..oe/eeeeeeeeeeeerroooooo
Current Context C The code context currently being processed, along with temporary information like cursor position or selected code area.
2. Assistant-Convvrsationn New Conversation Framework for Programming Assistants
User Instruction U (Optional) User instructions related to the code, either written by the programmer or generated as feedback based on interactions with external environments (such code interpreter).
In this section, we introduce a new conversational frame- work, Aisistant-Conversation aimed at simplifying the pro- gramming process'. The framework leverages all available
In this work, "conversation" refers the common format used LLM generation, rather than multi-tumn dialogues.
Assistant Output A The output of the model, consists of modified code and chat-style interaction with the program-
2

---

CursorCore: Assist Programming through Aligning Anything
Model Input:
Model Outpot:
typing import
Hi
typing import
A
generate pange(starn::: int, int, step: int)
generate rangetttrnnntt:::::: int, int, stop: int) List]int]:
List[int]:
[] start
Ha
(stop
end) (step and end):
L.append(1) step
while True:
1. I.append(1)
return
Decide the condition break the loop
1f (step and end) (step 0 and end): break
Based on the previous revisions and the curvect code, it appeess loog condition (step and < and end), and there is an additional check for stop == inside the loop. However, this check redundat because loop condition almady ensures that the loop will not run if step zero.
(stap
while (step and 1 end) or (step 9 and end) H3
or step mn 8:
I.append(i)
Themfoo, the predicted modification is to remove the rodundant check for step inside the loog, simplifying the code without changing its functionllty. This will make the function more concise and easier understund.
1f step == 8: breakt
while (step and end) (stap 8 and > and):
J (.appen((1)
Model Output:
6f step == 0:7
break:
eodule exports function(cts)
+- step?
ctx. cooiciss returreetn,, "true' { maxige:
18 365 24 s 68 68 1000, httpönly: False })
Model Impot:
To make the cookie not HTTP-onhy need explicitly set, the hetponly option to false. This will allow the cookie to be accessed by client-side scripts.
aodule. exports Function(etx)
C
ctx. cookies, set( set(" returiig', 'true', {
nauige:
10 345 24 60 6o 2000 })
Here's the plan:
1. Locate ctx. cookies set method the current code.
Pix retariing user cookie to not be HTTP-only.
U
2. Ad the hetpūnly: false option to the options objact pessed to set method.
Figure Examples of Assistant-Conversaalee Conversation from our training data. The top example demonstrates predicting the corresponding edits and explanations based historical edits and the current code context. The bottom example demonstrates predictions based the current code and user instructions.
mer. In this work, we mainly focus on the prediction of modified code.
efficiency, and models may also lack alignment and proper training for such inputs.
2.2. Comparisons of Assistant-Conversation
Our framework addresses these issues We use multiple input sources to harness all relevant information from the programming process. For the output, we divide into two parts: modified code and chat-style communication with the programmer, aligning with the common practices of users. When the user only requires responses based on U, similar instruction models, we can omit H and C, suppress code modifications, and provide only chat output to ensure compatibility with past chat modes.
Completion and insertion modes face challenges when modeling both C and H Although they utilize C, they fail to capture H, limiting the modeling of future changes in C, and are incapable of deleting or editing code. Although user instructions and reflection information can be used through comments and assert statements, this capability is weak and unstable.
Chat models are not ideal for all programming assistance tasks These models focus on user input rather than the code content, while the input should primarily be centered on instead of just user instructions. In traditional conversa- tional frameworks, the sole input source is U, which works for chatbots but not for application assistants. Input sources should include H. and U, as both H and are related to represent the interac- tion history between users and assistants, they struggle to capture the historical changes in the application's 's content. Prompt engineering can integrate some of this information into existing models, but the impact is limited. Constructing prompts with numerous tokens increases cost and reduces
2.3. Specifications and Implementation
To represent a piece of code like C. we can either use it directly or wrap it in markdown code block. However, representing code changes, such as or changes in A, more complex. We can either use the whole code, patches that alter the code, or records of both the modification loca- tions and the specific changes. Some methods work well but experience issues when handling longer texts, such out- putting the entire modified code, which can be slow. Other methods output minimal content, like providing only the modification locations and changes. These are faster but still not optimal in terms of performance. We represent
C. Although instruction models
3

---

CursorCore: Assist Programming through Aligning Anything
code changes in the experiments of the main body using the whole code format, and investigate different ways to represent these modifications, as detailed in Appendix B. Additionally, we explore methods for compressing historical code changes in Appendix I.
tasks (CodeParrot, 2023; Muennighoff al., 2024), and multilingual generation (Zheng al., 2023b; Cassano et al., 2023a). We refer to these works and further extend it to comprehensively evaluate the model's ability to assist pro- gramming. We randomly categorize each task into one of the four types, then manually implement the functions and simulate the potential instructions that programmers might give to an LLM during the process, collecting all interactions. We invite programmers with varying levels of experience to annotate the data. After processing, we get the new benchmark, Assist Programming Eval (APEval), which contains approximately IK multilingual samples. Detailed statistics are shown in Table 1. Specific details regarding the collection process and examples of our benchmark can be found in Appendix C, which includes detailed human annotation rubric and results.
In some cases, programmers assign assistants to focus on specific areas of code. They might use the cursor to mark general location or directly select a range of code, as shown in Figure 2. We handle this by treating them special tokens (see Appendix F for further details).
We structure conversations in the order of S-H-C-U-A to match the actual workflow. This mirrors the chronologi- cal sequence in which information is generated during the programming process. By doing so, we maximize prefix overlap across multiple requests, utilizing prefix caching to reduce redundant kv-cache computations and improve efficiency (Zheng et al., 2023a). A is organized in code- chat order, prioritizing code edits due to their importance in real-time applications where speed is crucial.
3.2. Evaluation Process and Metries
In all tasks, we use the classic Pass@1 I metric to execute the generated code, which is the simplest version of the Pass @k metrie (Chen et al., 2021). Since APEval is extension of HumanEval, we evaluate its Python version using the test set created by EvalPlus (Liu et al., 2023) and assess its other language versions using bigcode- eercoodeevvlllation... -hanness (Ben Allal et al., 2022). We set the Python version as the de- fault version for evaluation, and report the results from both the basic and extra tests. We provide the model with relevant information during the programming process, and the model immediately returns the modified code. Some methods may improve performance by increasing the number of output tokens to model the thinking process; we discuss this further in Appendix G.
3. APEval: Benchmark for Assisted Programming
3.1. Benchmark overview
Past benchmarks assessing LLM code capabilities have ef- fectively evaluated tasks like program synthesis (Chen et al., 2021; Austin et al., 2021), code repair (Muennighoff et al., 2024; Jimenez et al., 2024), and instructional code editing (Cassano et al., 2023b; Paul-Gane Paul-Gauthier, 2024; Guo et al., 2024b). However, they fall short in fully assessing how models use various types of information to assist in pro- gramming. This gap calls for new benchmark.
4. Programming-Instrr Instruct: Collect any data during programming
Table 1. APEval Statistics and breakdown tasks by information type.
To align models with programming-relatte data, relevant training data must be collected. While large amounts of un- supervised code (Kocetkov et al., 2023) and instruction data (Wei et al., 2023b; Luo al., 2024b) have been gathered, there remains significant lack of data the coding pro- cess. Manually annotating the coding process is expensive, so we propose Programming-Instren a method to automate this data collection.
APEval Statistics
Python Multilingual
164 Samples
984 Samples
Language
Python, C++. Java, JavaScript, Go, Rust
Details
Mean
Max
Snippets (H)
2.8
10
Lines (HJClU)
21.7 8.4 3.2
139 31 19
Chars (H C(U)
0.6K 0.3K 0.2K 5.1K 1.4K 1.2K
4.1. Data Sources
As discussed in Section 2.1, programming assistance can involve different types of information, with H and U being optional. Thus, there are four possible combinations of infor- mation: H, C, U: H. C; C, U; and only C. HvaanEval (Chen et al., 2021) is a well-known benchmark for evaluating code completion. It has been extended to assess other tasks such as code insertion (Bavarian et al., 2022), instruction-bart baaed
To ensure both quality and diversity in the coding process data, we collect information from three different sources: Al Programmer, Git Commit, and Online Judge submission.
AI Programmer For each code snippet, we use LLMs to generate the corresponding coding history. Since human
4

---

CursorCore: Assist Programming through Aligning Anything
Al Programmer
Online J Judge Submission
query users
query users user
query users { name
query users id
Problem: Write function that accepts an integer and checks whether is odd or even. If the number is even, the function
Create History
1d
should retur true; odd, should return falso. Example Input: Example Output: true
function (nEnbe()
Function (nEnben) return number;
Git Commit
return number
def (an_aax(arr) return (arr)))), min
def nin_nax(arr); return min(arr), nar((rr))
R
?
function (nunber) return number
function (ounber) return number
Git commit: Change the order of return values
Figure 3. Samples from Al Programmer, Git Commit and Online Judge Submission.
@faster
pan(a,b): b):
-eFaster rdef b))
08
H
M
M
wH
W WHC
-&faster rdef pon(a,b): return a'b
def po: Q3
H M
return 21b
return arb H
Judge
Judge
haze
def pon(a,b):
a/b
@
M
M
def pon[a,b]: a*b
def ponCa,b): b):
return Am
M
-efaster
W HU
W VICUU
@Faster def pow(a,b): b]: retum a++b
eaaster pow(a,b):
return b 2
Create
Create
return a
Figure 4. Data processing pipeline. The randomly selected time point is the third, data type is H and C.
coding approaches vary widely, we utilize several LLMs, each guided by three distinct prompts, representing novice, intermediate, and expert programmers. The LLMs then return their version of the coding process. Prompts used are shown in Appendix O.
historical records H.
We then handle each type of data based on whether U is available. For cases without U, we segment the changes from C to F based on continuity, referring to them as M, and let LLMs analyze and then judge whether each segment of M aligns with user's purpose through principle-driven approaches (Bai et al., 2022; Sun et al., 2023; Lin et al., 2024). This approach accounts for ambiguity in user intent when inferring from H or C. For example, if a programmer actively adds some private information at the beginming of the code without it being mentioned in the previous records, LLMS should not predict this change. We discard segments deemed irrelevant, and merge the remaining ones out- puts that models need to learn to predict. For cases with U, we follow the instruction generation series methods (Wang et al., 2023b; Wei et al., 2023b; Luo et al., 2024b) by in- putting both the historical edits and current code into the LLM, prompting to generate corresponding instructions.
Some software can automatically track changes, such as Git. We use Git Commit data from Github, which captures users' code edits and modification histories.
Git Commit
Online Judge Submission Many online coding platforms like Leetcode and Codeforces allow users submit code for exccution and receive feedback. During this process, users continuously modify their code until it is finalized. We also make use of this data.
Through these sources, we obtain a large number of samples, each consisting of multiple code snippets. The last snippet in each sample is referred to as the final snippet (F). Examples of data sources are shown in Figure 3.
In addition the above, we model selected code regions, cursor positions, and make LLMs create chat-style interac- tions with users. Further details are provided in Appendix D.
4.2. Data Processing
After collecting programming processes, we process them to meet the requirements of shows the steps of data processing. First, we randomly select time point in the coding process, referred to C. As mentioned in Section 2, H and U are optional, we need to collect four types of data distinguished according to input data types: H, C, U: H, C; C, U; and only C. For each sample, we randomly designate one type. If the selected type includes H, We use the preceding edits of C as the
Figure 4
5. CursorCore: Fine-tune LLMs to align anything
5.1. Base models
We fine-tune existing base LLMs to assist with program- ming tasks. Over the past few years, many open-source foundation models have been trained on large code corpora
5

---

CursorCore: Assist Programming through Aligning Anything
Table 2. Statistics of our training data.
Sample Num
Language Num
History Snippets Mean/ Max
Input Length Mean Max
Output Length Mean / Max
AT Programmer
70.9K
2.0 / 17
0.6K 25K
1.0K / 5.2K
Git Commit
88.0K
14
1.5 / 15
1.5K/ 19.9K
1.4K / 5.2K
Online Judge Submission
60.5K
44
3.8 / 96
4.8K / 357.2K
1.9K / 35.
sourced from GutHub and other platforms, demonstrating strong performance in coding. We choose the base ver- sions of Deepseek-Codes (Guo et al., 2024a), Yi-Coder (AI et al., 2024) and @wene.5--Coder (Hui et al., 2024) series, as fine-tuning is generally more effective when applied to base models rather than instruction models. After training, we refer to them as CursorCore- DS, CursorCore- Yi and CursorCore- QW2. series. Deepseek-Coder has achieved state-of-the-art performance numerous coding-related benchmarks over the past year, gaining wide recognition. Yi-Coder and Qwen2.5-Coder are the most recently released models at the start of our experiments and show the best per- formance on many benchmarks for code now. These models are widely supported by the community, offering a good balance between size and performance, making them suit- able for efficient experimentation. For ablation experiments, we use the smallest version, Deepseek- Prepseek-Coder-11 1 1.3B, to ac- celerate the process. We use a chat template adapted from ChatML (OpenAI, 2023) to model Assistant-Coon Converttonniisstionnnnn during training, as detailed in Appendix M. Training details can be found in Appendix E.
30000
Alprogrammer Commit Online Submit
Sample Num
20000
10000
200
400
1700
1400
1600
1800
2000
2200
2400
2600
2000
1600
3200
4000
4000
Figure 7, Distribution of input lengths in the training data.
30000
Alprogrammer Commit
Sample Num
20000
On line Submit
10000
200
A00
B0G
000
1000
2000
1400
E600
1800
2800
2200
2400
2800
2000
3090
2200
Coder-1
Figure 8. Distribution of output lengths in the training data.
5.2. Training data
We use Programming- Instruct to collect data. For AI Pro- grammer, we gather code snippets from datasets such as the Stack (Kocetkov et al., 2023) and OSS-Instruct (Wei et al., 2023b), then prompt LLMs to generate the program- ming process. For Git Commit data, we collect relevant information from EditPackFT (Cassano et al., 2023b) (a fil- tered version of CommitPackFT (Muennighoff et al., 2024)) and further refine it through post-processing and filtering. Regarding Online Judge Submission data, we source the programming process from the Codenet dataset (Puri et al., 2021). First, we group all submissions by user for each problem, then exclude invalid groups without correct sub- missions to obtain complete programming processes. These are then fed into the processing pipeline to generate the final training data. total, we accumulate 219K samples, with detailed statistics and distributions shown in Tables 2 and and Figures 5 8. AI Programmer data has the shortest average length, while Online Judge Submission data has the longest. To ensure compatibility with previous chathot-style interactions and further improve model performance, we also incorporate the Evol-Instruct dataset (ISE-UIUC, 2023) collected using the GPT series (Ouyang et al., 2022), which has been widely recognized for its high quality during train-
100K
Commit Online Submit
Semple Num
10K
1K
10
pythen
ruby javascrpt
layo
php shall
rust
ca
go
typescript
scalo l
switt
hockall
kottin
nim
fortran
lua
julia
paoccl
pari other
Figure 5. Distribution of programming language in the training data.
100K
Alprggaamer Git Commit
Sample Num
1K
Online Submit
100
10
Figure 6. Distribution of history snippet counts in the training data.
6

---

CursorCore: Assist Programming through Aligning Anything
ing. Following StarCoder's data processing approach (Li et al., 2023), decontaminate our training data.
in Appendix L. It includes both the average results and the results across four different types of information within the benchmark, each item in the table is the score resulting from running the base tests and extra tests. We also report the evaluation results of other well-known models, which can be found in Appendix K.
During data collection, we randomly utilize two power- ful open-source LLMs: Mistral-Large-Inttuee (Mistral- AI, 2024b) and Deepseek-Coders -2-ls--r--nsttrett (DeepSeek-Al et al., 2024). These models have demonstrated performance comparable to strong closed-soure, models like GPT-4o across many tasks, and are currently the only two open- source models scoring over 90% the classic HumanEval benchmark at the start of our experiment. Additionally, they are more cost-effective and offer casier reproducibility than GPT-4o. For Mistral-Large-Iae Laatttt-targ--tsttrutt,,,, we quantize the model using the GPTQ (Frantar et al., 2022) algorithm and deploy it locally with aGla (Zheng et al., 2023a) and Marlin kernel (Frantar et al., 2024) on 4 Nvidia RTX 4090 GPUs. For Seepseek-Coder. (2-der-V--Innn V2-Instruct, use its official API for integration.
CursorCore outperforms other models of comparable size CursorCore consistently outperforms other models in both the IB+ and 6B+ parameter sizes. achieves the highest average score, with the best IB+ model surpassing the top scores of other models by 10.4%, and even by 11.5% when running extra tests. Similarly, the best 6B+ model exceeds by 4.3%. and by 3.0% in the case of extra tests. Additionally, across various information types, CursorCore consistently demonstrates optimal performance among all similarly sized models.
Instruction models mostly outperform base models For most model series, instruction-tuned models outperform their corresponding base models, instruction fine-tuning generally enhances model capabilities (Ouyang et al., 2022; Longpre et al., 2023). The only exception observed in our experiments is the latest model, Qwen2.5-Coder. Its base model achieves a very high score, while the instruction- tuned model performes worse. We attribute the base model's high performance to its extensive pre-training. which in- volved significantly more tokens than previous models (Hui et al., 2024). This training on wide range of high-quality data grants it strong generalization abilities, enabling it to effectively handle the newly defined APEval task format. In contrast, the instruction-tuned model is not specifically aligned with this task, leading to a decrease in its APEval score. This highlights the challenges of aligning models with numerous diverse tasks, especially small models.
Table 3. The proportion of four combinations of information dur- ing programming our training data.
C
H,
C
C,
U
H,
U
AI Programmer
24.1
22.2
25.4
28.3
Git Commit
25.9
20.0
28.0
26.1
Online Judge Submission
27.5
19.7
29.4
23.4
6. Evaluation and Results
In this section, we evaluate the CursorCore models. We begin by describing the experimental setup and then present and analyze the results.
6.1. Experimental setup
We conduct the data selection ablation and primary eval- uation on our APEval benchmark, and provide results on well-known benchmarks such as Python program synthesis, automated program repair, and instructional code editing, which are detailed in Appendix J. We choose prominent open-source and closed-sourds LLMs as our baselines. For all benchmarks, we use greedy decoding to generate evalua- tion results. CursorCore natively supports various inputs in APEval, whereas base and instruction LLMs require addi- tional prompts for effective evaluation. We design few-shot prompts separately for base and instruction models, as de- tailed in Appendix N. Data selection ablation can be found in Appendix H.
Performance difference between general and code LLMs is strongly related to model size In 1B+ parameter mod- els, general LLMs significantly underperform code LLMs. Even the best-performing general model scores over 10% lower compared to the best-performing code model, despite having more parameters. For models with 6B+ parameters, while general LLMs still lag behind code LLMs, the perfor- mance gap narrows considerably, with general L.Mss even surpassing in certain cases involving specific information types. When it comes to 10B+ models, the performance dif- ference between general and code LLMs becomes negligible. We think that smaller models, due to their limited parameter capacity, tend to focus on a single domain, such as pro- gramming assistance, while larger models can encompass multiple domains without compromising generalizahility.
6.2. Evaluation results on APEval
In Table 4, we present the results of evaluating CursorCore series models and other LLMs on the Python version of APEval. The results for multilingual versions can be found
Gap between closed models and the best open models is smaller Historically, open-source models significantly
7

---

CursorCore: Assist Programming through Aligning Anything
Table 4. Evaluation results of LLMSs APEval.
Model
H, C
C,U
H, U,
Avg.
Closed Models
GPT-4o
68.3 (63.4)
61.0 (56.1)
75.6 (75.6)
56.1 (53.7)
65.2 (62.2)
10B+ Models
Sodestral-V0.1-220
68.3 (56.1)
41.5 (41.5)
75.6 (73.2)
48.8 (46.3)
58.5 (54.3)
DS-Coder-33B-Inst
63.4 (56.1)
56.1 (48.8)
70.7 (63.4)
51.2 (48.8)
60.4 (54.3)
Qwen2.5-72BBImm
73.2 (68.3)
53.7 (51.2)
78.0 (70.7)
56.1 (56.1)
65.2 (61.6)
Mistral-Large-1233
65.9 (58.5)
56.1 (46.3)
73.2 (68.3)
48.8 (48.8)
61.0 (55.5)
DS-Coderr-V-226BB-mm
78.0 (65.9)
48.8 (43.9)
68.3 (61.0)
53.7 (48.8)
62.2 (54.9)
6B+ Models
L-ama-3.1-8B-Inst
24.4 (24.4)
31.7 (29.3)
53.7 (51.2)
39.0 (34.1)
37.2 (34.8)
Gemma-2-99-4
56.1 (53.7)
41.5 (36.6)
51.2 (46.3)
36,6 (29.3)
46.3 (41.5)
DS-Coder-6.77 B.rber--.7B-Baee
29.3 (24.4)
(22.0)
219. (31.7
29.9 (24.4)
DS-Coder-6.7B-Ing
56.1 (53.7)
41.5 (36.6)
70.7 (61.0)
34.1 (29.3)
50.6 (45.1)
Yi-Coder-9B
29.3 (26.8)
(22.0)
17.1 (17.1
29.3 (26.8)
25.6 (23.2)
Yi-Coder-9BB-Ch.
56.1 (51.2)
39.0
(30.7)
36.6 (36.6)
51.2 (48.8)
56.1 (53.7)
41.5 (36.6)
65.9 (56.1)
31.7 (29.3)
48.8 (43.9)
Qwen2.5 -e-eer--Bnsss Inst
22.0 (19.5)
46.3 (39.0)
41.5 (65.9)
46.3 (40.9)
CursorCore-DS-6..
68.3 (63.4)
41.5 (39.0)
68.3 (63.4)
(31.7)
53.7 (49.4)
CursorCore- Yi-9B
53.7 (53.7)
46.3 (43.9)
75.6 (68.3)
43,9 (36.6)
54.9 (50.6)
CursorCone-QW2.5-77
65.9 (61.0)
41.5 (39.0)
65.9 (63.4)
48.8 (43.9)
55.5 (51.8)
IB+ Models
Jlama-3.2-3B-Instrus
14.6 (14.6)
22.0 (19.5)
29.3 (26.8)
34.1 (31.7)
25.0 (23.2)
Gemma-2-2B-B
14.6 (14.6)
22.0 (19.5)
29.3 (26.8)
34.1 (31.7)
25.0 (23.2)
DS-Coder- 1.3--11.B-Baaat
(0.0)
12.2
17.1 (12.2)
(14.6)
12.2 (9.8)
DS-Coder- 1.3oder--.3B--11 1.3B-Inst
39.9 (36.6)
39.0
39.0 (29.3)
34.1 (34.1)
37.8 (34.1)
Yi-Coder-1.5B
(0.0)
22.4)
14.6
(2.3
7.9 (6.1)
Yi-Coder-
(31.7)
(4.9)
54.25
26.8 (22.0)
28.7 (25.0)
Qwen2.5-Coder-1 1.50
43,9 (36.6)
26.8 (26.8)
(41.5)
36.6 (34.1)
39.6 (34.8)
Qwen2.5-Coder-L..
14.6 (14.6)
17.1
43.9
31.7 (29.3)
26.8 (23.2)
CunsorCore-DS-L.33
36.6 (31.7)
(31.7)
53.7 (46.3)
26.8 (22.0)
39.0 (32.9)
CursorCore- Yi-ooC
46.3 (39.0)
34.1 (29.3)
(58.5)
36.6 (34.1)
46.3 (40.2)
CursorCore-QW2.551155
46.3 (43.9)
48.8 (43.9)
65.9 (61.0)
39.0 (36.6)
50.0 (46.3)
lag behind closed-source models, like those in the GPT se- ries, leading to a preference for closed-source models synthetic data generation and other applications (Taori et al., 2023; Xu et al., 2023). However, with the continuous ad- vancement of open-source LLMs, increasingly powerful models have emerged. On APEval, the best open-source mudels-such as Qwen2.5-72B-Insttee Instruct, Mistral-Large- Instruct, and Deepseek-Codee Sondddr-VV-Insruuct--eee -enstructtddmmosstaatt performance that closely approaches that the leading GPT series model, GPT-4o. This indicates that the perfor- mance gap between open-source and closed-source LLMs has considerably narrowed, encouraging the development of more interesting applications based on open-source LLMss. Despite this progress, GPT-4o remains more comprehensive than open-source LLMs. It utilizes I/ far more effectively than any other model, demonstrating its strong capability to
process and align with various types of information. This an area where open-source LLMs still need to improve.
7. Conclusion
This work explores how LLMs can maximize the use of any available information during programming process to assist coding, We introduce Assistan-Coonvrsaation to model the diverse types of information involved in programming. We present APEval, new benchmark that includes various historical edits and instructions, providing a comprehensive evaluation of the model's programming assistance capa- bilities. Additionally, which is designed to collect data for training LLMs to assist programming, along with their corresponding data sources. Furthermore, we train CursorCore. which demonstrate out-
propose Programming Instruct,
8

---

CursorCore: Assist Programming through Aligning Anything
standing performance in assisting programming tasks while achieving good balance between efficiency and cost. We also conduct extensive ablation experiments and analyzes. Beyond enhancing traditional approaches of programming assistance, we plan to extend this approach to support mod- els capable of assisting with repository-Ievel development as well as other applications.
Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey, C., Tworek, J., and Chen, M. Efficient training of lan- guage models to fill in the middle. arXiv preprint arXiv: 2207.14255, 2022.
Ben Allal. L., Muennighoff, N., Kumar Umapathi, L.,, Lipkin, B., and von Werra, L. A framework for the evaluation of code generation models.
Acknowledgments
bigcode elgcooee-valllaiionn-eeeee
2022.
This research was partially supported by grants from the Joint Research Project of the Science and Technol- ogy Innovation Community in Yangtze River Delta (No. 2023CSIZN0200), the National Natural Science Founda- tion of China (62337001), the Key Technologies R D Program of Anhui Province (No. 202423k09020399 and the Fundamental Research Funds for the Central Universi-
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps- Costin, L., Pinckney, D., Yee, M., Zi, Y., Anderson, C. J., Feldman, M. Q., Guha, A., Greenberg, M., and Jangda, A. Multipl-e: A scalable and polyglot approach to bench- marking neural code generation. IEEE Trans. Software Eng., 49(7):3675-3691 2023a. doi: 10.1T00/SSE 3267446. URL https: / http://o.....ro 10. 1109 / 322334267445
ties.
Impact Statement
Cassano, F., Li, L., Sethi, A., Shinn, N., Brennan-Jones, A., Lozhkov, A., Anderson, C. J., and Guha, A. Can it edit? evaluating the ability of large language models to follow code editing instructions. arXiv preprint arXiv: /2450, 2023b.
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavar- ian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert- Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code. arXiv preprint arXiv: 2107.03374, 2021.
References
i, Young, A.. Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., Yu, K., Liu, P., Liu, Q. Yue, S., Yang, S.. Yang, S., Yu, T., Xie, W., Huang, W., Hu, X., Ren, X., Niu, X., Nie, P., Xu, Y., Liu, Y., Wang, Y., Cai, Y., Gu, Z., Liu, Z., and Dai, Z. Yi: Open foundation models by 01 arXiv preprint arXiv: 2403.046 040500465,.200 2024.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H. Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., and Sutton, C. Program synthesis with large language models. arXiv preprint arXiv: 2108.07732. 2021.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D.,, Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J.. Ndousse, K., Lukosuite, K., Lovitt, L., Sellitto, M., Elhage, N., Schiefer, N., Mercado, N., DasSarma, N., Lasenby, R., Larson, R., Ringer, S., Johnston, S. Kravec, S., Showk, S. E., Fort, S., Lanham, T., Telleen-Lawton. T., Conerly, T., Henighan, T., Hume, T., Bowman, S. R., Hatfield- Dodds, Z., Mann, B., Amodei, D., Joseph, N., McCan- dlish, S., Brown, T., and Kaplan, J. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv: 22 22/2.08073, 2022.
CodeParrot. Instruct humaneval, 2023.
URL
https: httpss://hugginggfaccee co ctttaasssse/aaaass//e/sas codeaarroo/isstructuuaneee 2023-11-02.
Accessed:
Continue- Continue, 2024. URL https://gitthhb conm/continuudde//cttttta Accessed: 2024-3- 18.
Cursor-Al. Cursor, 2023. URL httpp:://ww...... com/. Accessed: 2023-12-24.
Dao, T. Flashattention-2. Faster attention with better par- allelism and work partitioning. In The Twelfth Interna- tional Conference on Learning Representations, ICLR

---

CursorCore: Assist Programming through Aligning Anything
2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL htttps://openreviieeeee wwt.:/wwwwwweeeeeeeeeeeeeeeeeeee forum? id id-m2n2Xyh99c.
Gulwani, S., Radicek, I., and Zuleger, F. Automated clus- tering and program repair for introductory programming assignments. ACM-SIGPLAN Symposium on Program- ming Language Design and Implementation, 2016. doi: 10.1 14111499999.911999999 19923877
DeepSeek-AI, Zhu, Q., Guo, D., Shao, Z., Yang, D., Wang, P. Xu, R., Wu, Y., Li, Y., Gao, H., Ma, S., Zeng, W., Bi, X., Gu, Z., Xu, H., Dai, D., Dong, K., Zhang, L., Piao, Y., Gou, Z., Xie, Z., Hao, Z., Wang. B., Song. J., Chen, D.,
Guo, D., Zhu, Q. Yang. D., Xie, Z., Dong. K., Zhang, W. Chen, G., Bi, X., Wu, Y., Li, Y. K., Luo, F., Xiong, Y., and Liang, W. Deepseck-coder: When the large language model meets programming the rise of code intelligence. arXiv preprint arXiv: 2401.14196 2024a.
Xie, X., Guan, K., You, Y., Liu, A., Du, Q. Gao, W., Lu, X., Chen, Q. Wang, Y., Deng, C., Li, J., Zhao, C., Ruan, C., Luo, F., and Liang, W. Deepseek-code coder--2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arviv: 2406. 1806.1192 11931, 2024.
Guo, J., Li, Z.., Liu, X., Ma, K., Zheng, T., Yu, Z., Pan, D., LI, Y., Liu, R., Wang, Y., Guo, S., Qu, X., Yue, X., Zhang, G., Chen, W., and Fu, J. Codeeditorbench: Evaluating code editing capability of large language models. arXiv preprint arXiv: 2404.03543, 2024b.
Ding, Y., Wang, Z., Ahmad, W. U., Ding, H., Tan, M., Jain, N., Ramanathan, M. K., Nallapati, R., Bhatia, P., Roth, D., and Xiang, B. Crosscodeeval: A diverse and multilingual benchmark for cross-ille code completion. Neural Information Processing Systems, 2023. doi: 10. 1S550/arXiv.2110.1111
Gupta, P., Khare, A., Bajpai, Y., Chakraborty, S., Gulwani,
S., Kanade, A., Radhakrishna, A., Soares, G., and Tiwari,
A. Grace: Generation using associated code edits. arXiv preprint arXiv: 2305. 14129, 2023.
Du, X., Liu, M., Wang, K., Wang, H., Liu, J., Chen, Y., Feng. J., Sha, C., Peng. X., and Lou, Y. Classeval: A manually-crafted benchmark for evaluating llms on class- level code generation. arXiv preprint arXiv:2308.01861, 2023.
He, Z., Zhong, Z., Cai, T. Lee. J. D., and He, D. REST: retrieval-basee speculative decoding. Duh, K., Gómez-Adomo H., and Bethard, (eds.), Pro- ceedings of the 2024 Conference of the North Ameri- Chapter of the Association for Computational Lin- guistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 1582-1595. Association for Compu- tational Linguistics, 2024. doi: 10.18653/V1/2024, NAACL-LONG.88, URL https: /doi 00. 18653/v1/2024
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arXiv: 2210.17323, 2022.
Frantar, E., Castro, R. L., Chen. J., Hoefler, T., and Alis- tarh, D. Marlin: Mixed-precision auto-regressive paral- lel inference on large language models. arXiv preprint arXiv:2408.1 1743, 2024.
-oon..oogg... 88.
Hsu, P.-L., Dai, Y., Kothapalli, V., Song, Q. Tang, S., Zhu, S., Shimizu, S., Sahni, S., Ning, H., and Chen, Y. Liger kernel: Efficient triton kernels for Ilm training. arXiv preprint arXiv: 10989, 2024.
Gao, W. Liu, Q., Li. R., Zhao, Y., Wang, H., Yue, L., Yao, F. and Zhang, Z. Denoising programming knowl- edge tracing with code graph-basee based tuning adaptor. In Proceedings of the 3 ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, pp. 354-365, 2025.
Huang, D., Qing, Y., Shang, W., Cui, H., and Zhang, J. M. Effibench: Benchmarking the efficiency of automatically generated code. arXiv preprint arXiv: 020377 2024.
Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L.. Liu, T., Zhang, J., Yu, B., Dang, K., Yang, A., Men, R., Huang, F., Ren, X., Ren, X., Zhou, J., and Lin, J. Qwen2.5-codee technical report. arXiv preprint arXiv: 2409.12186 12186, 2024.
Github-Copilod Github copilot your ai pair programmer, 2022. URL https: fiiithhbb... com/ features/ copilot. Accessed: 2022-1-22.
ISE-UIUC, 2023.
URL https://
Gu, A., Rozière, B., Leather, H. J., Solar-Lezama, A., Syn- naeve, G., and Wang. S. Cruxeval: A benchmark for code reasoning, understanding and execution. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL http::/ http:::///ppeeevieeeeee net forum? 1d=F U=FFfg52swwg.
huggingface wwt.t/tatetssseeeettiii...
Accessed:
2023-11-01.
Jain, N., Han, K., Gu, A., Li, W.D.,, Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Live- codebench: Holistic and contamination free evaluation
10

---

CursorCore: Assist Programming through Aligning Anything
of large language models for code. arXiv preprint arXiv: 2403.07974, 2024.
Li, R., allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C.,, LI, J., Chim, J., Liu, Q. Zheltonozhskii, E., Zhuo, T. Y., Wang, T., Dehaene,
Jiang H., Wu, Q.. Lin, C.-Y., Yang, Y., and Qiu, L. Limlin- gua: Compressing prompts for accelerated inference of large language models. arXiv preprint arXiv: 2310.05736, 2023.
O., Lamy-Poirier, J., Monteiro, J., Gontier, N., Yee, M.- H., Umapathi, L.. K., Zhu, J., Lipkin, B., Oblokulov, M., Wang, Z., Murthy, R., Stillerman, T., Patel, S. S., Abulkhanov, D., Zocca, M., Dey, M., Zhang, Z., Bhat- tacharyya, U., Yu, W. Luccioni, S., Villegas, P., Zh- danov, F., Lee, T., Timor, N., Ding, J., Schlesinger, C. S., Schoelkopf, H., Ebert, J., Dao, T., Mishra, M., Gu, A., Anderson, C. J., Dolan-Gavitt, B., Contractor, D., Reddy, S., Fried, D., Bahdanau, D., Jernite, Y., Ferrandis, C. M., Hughes, S., Wolf, T., Guha, A., Werra, L. V., and de Vries, H. Starcoder: may the source be with you! Transuc- tions on Machine Learning Research, 2023, ISSN 2835- 8856. URL ht tttpps/ opw.review..eeeeeeeoeee/e.roe.w. nnt / forum? id=KoFOga haaE. Reproducibility Certification.
Jiang H., Liu, Q., L., R., Zhao, Y., Ma, Y., Ye, S., Lu, J., and Su, Y. Verse: Verification baseatttoo--eeddddddee self play for code instructions. In Proceedings of the AAAI Conference on Artificial Intelligence. pp. 24276-24284, 2025.
Jimenez, C. E., Yang. J. Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. R. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna. Austria, May 7-11, 2024. OpenRe- view.net, 2024. URL https:h//////tttc /ttpss///ppeeevvieeeeeee net./ forum? Pid-:tnddd-iF 8YNK666.
Li, R., He. L., Liu, Q.. Zhao, Y., Zhang, Z., Huang, Z., Su, Y., and Wang. S. Consider: Commonalities and spe- cialties driven multilingual code retrieval framework. In Pruceedings of the AAAI Conference on Artificial Intelli- gence, pp. 8679-8687 2024b.
Kocetkov, D., Li, R., allal, L. B., J., Mou, C., Jermite, Y., Mitchell, M., Ferrandis, C. M., Hughes, S., Wolf, T., Bahdanau, D., Werra, L. V., and de Vries, H. The stack: 3 TB of permissively licensed source code. Transac- tions Machine Learning Research, 2023. ISSN 2835- 8856. URL /ttttps://opeeeee http://pppreviieeeeeee net forum? id=pxpbTdUBpD.
Li, R., Liu, Q., He, L.., Zhang, Z.,, Zhang, H., Ye, S., Lu, J., and Huang, Z. Optimizing code retrieval: High-quality and scalable dataset annotation through large language models. In Proceedings of the 2024 Conference on Em- pirical Methods in Natural Language Processing, pp. 2053-2065, 2024c
Kundu, A., Lee, R. D., Wynter, L., Ganti, R. K., and Mishra, M. Enhancing training efficiency using packing with flash attention. arXiv preprint arXiv: 2407.09105, 2024.
Li, R., Kang, J., Liu, Q., He, L., Zhang, Z., Sha, Y., Zhu, L., and Huang, Z. Mgs3: A multi- multi-graauuarity self-supervised code search framework. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, pp. 695-706, 2025.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H.. Gonzalez, E., Zhang. H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. Symposium on Operating Systems Principles, 2023. doi: 10.1 1414536000066663111
Liang. J. T., Yang, C., and Myers, B. A. A large-scale survey on the usability of ai programming assistants: Successes and challenges. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pp. 1-13, 2024.
Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong, R., Zettlemoyer, L., Yih, S., Fried, D., yi Wang, S., and Yu, T. Ds- 1000: A natural and reliable benchmark for data science code gen- eration. International Conference on Machine Learning, 2022. doi: 10.48550/arxiv.2211
Lin, Z., Gou, Z., Gong, Y., Liu, X., Shen, Y., Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., and Chen, W. Rho-1: Not all tokens are what you need. arXiv preprint arXiv: 040400765, 2024.
Li, J. Li, G., Zhang. X., Dong, Y., and Jin, Z. Evocodebench: An evolving code generation benchmark aligned with real-world code repositories. arXiv preprint arXiv: 2404.00599 2024a.
Liu, J., Xia, C., Wang, Y., and Zhang, L. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Neural Information Processing Systems, 2023. doi: 10.48550/ arXiv.2305.0122
Li, R., Yin, Y., Dai, L., Shen, S., Lin, X., Su, Y., and Chen, E. Pst: measuring skill proficiency in programming exercise process via programming skill tracing. In Proceedings of the 45th International ACM SIGIR Conference on Re- search and Development in Information Retrieval, pp. 2022.
Liu, Q., Huang, Z., Yin, Y., Chen, E., Xiong, H., Su, Y., and Hu, G. Ekt: Exercise-aware knowledge tracing for

---

CursorCore: Assist Programming through Aligning Anything
student performance prediction. IEEE Transactions on Knowledge and Data Engineering, pp. 100-115, 2019.
Muennighoff, N., Liu, Q., Zebaze, A. R., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., Werra, I.,, and Longpre, S. Octopack: Instruction tuning code large lan- guage models. In The Tweifth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024, URL https: / //openreviww net / forum? wwwmmwddddwwmmmmm wwwpwww.....
Longpre, S., Hou, L.., Vu, T., Webson, A., Chung, H. W. Tay, Y., Zhou, D., Le, Q. V., Zoph. B., Wei, J., and Roberts, A. The flan collection: Designing data and methods for effective instruction tuning. International Conference on Machine Learning. 2023. doi: 10.48550/arxi..2311. 13688.
OpenAL. Chat markup language, 2023. URL https: //github. blob/ gtlob/reeeasevv.. 28 0/ httpchhatml.... md. Accessed: 2023-8-29.
Lozhkov, A., R., Allal, L. B., Cassano, F., Lamy-Poirice J. Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., Liu, T., Tian, M., Kocetkov, D., Zucker, A., Belkada, Y., Wang, Z., Liu, Q. Abulkhanov, D., Paul, I., Li, Z., Li, W.-D., Risdal, M., Li, J., Zhu, J.. Zhuo, T. Y., Zheltonozhskii, E., Dade, N. O. O., Yu, W., Krauß, L., Jain, N., Su, Y., He, X., Dey, M., Abati, E., Chai, Y., Muennighoff, N., Tang, X., Oblokulov, M., Akiki, C.,, Marone, M., Mou, C., Mishra, M. Gu, A., Hui, B., Dao, T., Zebaze, A., Dehaene, O.. Patry, N., Xu, C., McAuley. J.. Hu, H., Scholak, T., Paquet, S., Robinson, J., Anderson, C. J., Chapados, N., Patwary, M., Tajbakhsh, N., Jernite, Y., Ferrandis, C. M., Zhang, L., Hughes, S., Wolf, T., Guha, A., von Werra, L., and de Vries, H. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv: 2412.1 2024.
OpenAL. URL
Learning to reason with 1lms, 2024. attps://opeet ctmp//ppaaaacmmooooaooooooocc com/ /in///nd/////// Learminggtt-rrssse-wttttt------- 2024-9-12.
Accessed:
Ouyang. I.., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instruc- tions with human feedback. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, vol- ume 35, pp. 27730-22222 27744.. Curran Associates, Inc., 2022.
Lu, S., Guo, D.. Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C. B., Drain, D., Jiang, D., Tang, D., Li, G., Zhou, L., Shou, L., Zhou, L., Tufano, M., Gong, M., Zhou, M., Duan, N., Sundaresan, N., Deng, S. K., Fu, S., and Liu, S. Codexglue: A machine learning benchmark dataset for code understanding and generation. NeurlPS Datasets and Benchmarks, 2021.
Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., and Gonzalez, J. E. Memgpt: Towards Ilms operating systems. arXiv preprint arXiv: 2310.08560, 2023.
Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv: 2305.15334, 2023.
Luo, Q., Ye, Y., Liang, S., Zhang, Z. Qin, Y., Lu, Y., Wu, Y., Cong, X., Lin, Y., Zhang, Y., Che, X., Liu, Z., and Sun, M. Repoagent: An lIm-powered open-source framework for repository- level code documentation generation. arXiv preprint arviv: 16667, 2024a.
Beul-rruthier. Aider terminal, 2024. URL nttps://githun: com/ Accessed: 2024-1-19.
pair programming in your
paul wtw.gggautheeraaaaaa
Luo. Z. Xu, C., Zhao, P., Sun, Q. Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q. and Jiang, D. Wizardcoder: Empowering code large language models with evol- instruct. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b. URL https: //openreview tttp://fouummiddnnnnSSsss
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., and Karri, R. Asleep at the keyboard? assessing the security of github copilot's code contributions. IEEE Symposium on Security and Privacy, 2021. doi: 10. 1091//9442214222222 9833571.
Puri, R., Kung, D. S., Janssen, G., Zhang, W., Domeniconi, G., Zolotov, V., Dolby, J., Chen, J., Choudhury, M. R.. Decker, L., Thost, V., Buratti, L., Pujar, S., Ramji, S., Finkler, U., Malaika, S., and Reiss, F. Codenet: A large- scale AT for code dataset for learning diversity of coding tasks. In Vanschoren, J. and Yeung, S. (,ds.), Proceed- ings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1. NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021.
Mistral-AI.
Codestral, 2024a.
URL
wttp:///o..siieee.eeee.eeee co/mistraall/
Codestral-22B-v06 Accessed: 2024-4-02.
Mistral-AI, 2024b. URL URLAattps://nuugiiggfeeee co/ misstrriaa//Mistaal-laag-----ee Accessed: 2024-8-01.
12

---

CursorCore: Assist Programming through Aligning Anything
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct preference optimization: Your language model is secretly reward model. NEURIPS, 2023.
Sun, Z., Shen, Y., Zhou, Q. Zhang, H., Chen, Z.., Cox, D., Yang. Y., and Gan, C. Principle-driven self-alignment of language models from scratch with minimal human supervision. NEURIPS, 2023.
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimizations toward training trillion parameter models. International Conference for High Performance Computing, Networking, Storage and Analysis, 2019. doi: 10.1 N0.1109/SC44405.. 200.000000000002244
Sweep-AI. Why getting gpt-4 to modify files is hard, 2024. URL https: sweep. dev dev/bleo /blog// gpt-4-modification Accessed: 4024-1-24.
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following Ilama model. https:/ htt///gitttue wwt.hh...oooteeeeooeeeeeeeeeeeee stanford_appace 2023.
Ren, J., Rajbhandari, S., Aminabadi, R. Y., Ruwase, O., Yang, S., Zhang, M., Li, D., and He, Y. Zero-offload: De- mocratizing billion-scale model training. In Calciu, I. and Kuenning, G. (eds.), Proceedings of the 2021 USENIX Annual Technical Conference, USENIX ATC 2021, July 14-16, 2021, pp. 551-564. USENIX Association, 2021. URL www.t:ttps... usw.ii/i.iw/............... www.ix. org/ http://oneeeee/c///eeooo at atc21 (tcel/ppresettation/reee
Team, C., Zhao, H., Hui, J., Howland, J., Nguyen, N., Zuo, S., Hu, A., Choquette- Choo, C. A., Shen, J., Kelley, J., Bansal, K., Vilnis, L., Wirth, M., Michel, P., Choy, P.. Joshi, P., Kumar, R., Hashmi, S., Agrawal, S., Gong, Z., Fine, J., Warkentin, T.,, Hartman, A. J., Ni, B., Korevec, K., Schaefer, K., and Huffman, S. Codegemma: Open code models based on gemma. arXiv preprint arXiv: 2406.11409, 2024.
Rozière, B., Gehring. J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Ferrer, C. C.,, Grattafiori, A., Xiong. W., Défossez, A., Copet, J., Azhar, F. Touvron, H., Martin, L., Usunier, N., Scialom, T., and Synnaeve, G. Code Ilama: Open foundation models for code. arXiv preprint arXiv: 2308.12950 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 2302.13971, 2023.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
Wang, F., Liu, Q., Chen, E.,, Huang, Z., Yin, Y., Wang, S., and Su, Y. Neuraled: A general framework for cognitive diagnosis. IEEE Trans. Knowl. Data Eng.. pp. 8312- 8327, 2023a.
Shazeer, N. M. and Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost. International Confer- ence on Machine Learning, 2018.
Shinn, N., Cassano, F. Berman, E., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal reinforcement learning. NEURIPS, 2023.
Wang. Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Align- ing language models with self-generated instructions. In Rogers, A., Boyd-Graber, J. L.., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, Pp. 34484-13508 Association for Computational Lin- guistics, 2023b. doi: 10. 10.1665/V122223AC1-1LLMM 754. URL https: /tl:://dooo http:://ooo.oggg///// wttp:////1.....00 //tp///.....r/0/.1133// 2023. aalaacllllnn. 754.
Shypula, A., Madaan, A., Zeng, Y., Alon, U., Gard- ner, J. R., Yang, Y., Hashemi, M., Neubig, G., Ran- ganathan, P., Bastani, O., and Yazdanbakhsh, A. Learning performance-improoing code edits. In The Twelfih Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
id= 10=xx7LLVHXYY.
Wei, J., Wang, X., Schuurmans, D.,, Bosma, M., Ichter, B., Xia, F.. Chi, E. H., Le, Q. V. and Zhou, D. Chain-of- thought prompting elicits reasoning in large language models. In Koyejo, S., Mohamed, S., Agarwal, A., Bel- grave, D., Cho, K., and Oh, A. (eds.), Advances in Neu- ral Information Processing Systems 35: Annual Confer- ence on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans. LA, USA, November 28. December 9, 2022, 2022.
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling Ilm test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint arXiv: 2408.03314, 2024.
Sun, W., Miao, Y., Li, Y., Zhang, H., Fang, C., Liu, Y., Deng, G., Liu, Y., and Chen, Z. Source code summarization in the era of large language models. arXiv preprint arXiv: $407.009559 2024.
13

---

CursorCore: Assist Programming through Aligning Anything
Wei, J., Durrett G., and Dillig, Coeditor: Leveraging contextual changes for multi-round code auto-editing. arXiv preprint arviv: 2305. 2023a.
Linguistics. doi: 10.1865sw/2222.fddnn--emmm 398. URL f findings mondings-emnl.... mmn1p.398.
Wei, Y., Wang. Z., Liu, J., Ding, Y., and Zhang, L. Magi- coder: Source code is all you need. arXiv preprint arXiv: 23/2.02/20, 2023b.
Zan, D., Chen, B.. Zhang, F., Lu, D., Wu, B., Guan, B., Wang, Y., and Lou, J.-G. Large language models meet nl2code: A survey. Annual Meeting of the Association for Computational Linguistics, 2022. doi: 10.18653/v1/ 2023.acl-long.44
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtow- icz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q,, and Rush, A. Transform- ers: Sfatc-of-the-art natural language processing. In Liu, Q. and Schlangen, D. (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, 45, Online, October 2020. Association for Computational Linguistics. doi: 10. 10.18663/w1/20220cmmm //acl antho 1 ogy. ooo.oogg////2/ emn pe--ddmo... 6.
Induusuusss,,s, 2025. URL https:/hhugiinffae co vo/datasets reeeiinddstrrssseeee. cessed: 2025-2-27.
Ac-
Zelikman, E., Wu, Y., Mu, J., and Goodman, N. D. Star: Bootstrapping reasoning with reasoning. Neural Informa- tion Processing Systems, 2022.
Zhang, F., Chen, B., Zhang, Y., Liu, J., Zan, D., Mao, Y., Lou, J.-G., and Chen, W. Repocoder: Repository-lvvel code completion through iterative retrieval and generation. Conference on Empirical Methods in Natural Language Processing, 2023. doi: 10.48555/arXiv.2333..3
demos.6. 6. URL https:
Xu, C., Guo, D., Duan, N., and McAuley, J. J. Baize: An open-source chat model with parameter-effiieen tuning on self-chat data. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods Natural Lnguage Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 6268-6278. Associ- ation for Computational Linguistics, 2023, doi: 18.18653/ V12003.EMNLP-MAIN....
Zhang, Q., Fang, C.,, Ma, Y., Sun, W., and Chen, 7. A survey of learning-based automated program repair. ACM Trans. Softw. Eng. Methodol., 33(2):55:1-55:69, 2024a. doi: 10.1145/3631974 URL htt.t::::/ddo.oogg//// 1145/363197
URL https: //doi. 0rg/10.18653/ /2023. emnlp-manlaee.. 385.
Zhang, S., Zhao, H., Liu, X., Zheng, Q. Qi, 7., Gu, X., Zhang, X., Dong, Y., and Tang, J. Naturalcodebench: Ex- amining coding performance mismatch on humaneval and natural user prompts. arXiv preprint arXiv: 2405.04520, 2024b.
Yang, K., Liu, J., Wu, J., Yang, C., Fung, Y. R., Li, S., Huang, Z., Cao, X., Wang, X., Wang, Y., Ji, H., and Zhai, C. If Ilm is the wizard, then code is the wand: A survey on how code empowers large language models to serve as intelligent agents. arXiv preprint arviv: 2401.00812, 2024.
Zhang. Y., Bajpai, Y., Gupta, P., Ketkar, A., Allamanis, M., Barik, T., Gulwani, S., Radhakrishna, A., Raza, M., Soares, G., al. Overwatch: Learning patterns in code edit sequences. Proceedings of the ACM on Programming Languages, 6(0OPSLA2):395 423, 2022.
Yang, N., Ge, T., Wang, L., Jiao, B., Jiang, D., Yang, L., Majumder, R., and Wei, F. Inference with reference: Lossless acceleration of large language models. arXiv preprint arXiv: 044874 2023.
Zhang, Z., Wu, L., Liu, Q. Liu, J., Huang. Z., Yin, Y.. Zhuang, Y., Gao, W., and Chen, E. Understanding and improving fairness in cognitive diagnosis. Sei. China Inf. Sci., 2024c
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In The Eleventh In- ternational Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRe- /
Zhao, Y., Huang, Z., Ma, Y., Li, R., Zhang, K., Jiang, H., Liu, Q. Zhu, L., and Su, Y. Repair: Automated program repair with process-based feedback. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 16415-16429. Association for Computational Linguistics, 2024.
view.net, 2023. URL pdf?id=WE_ vluYUL-X.
Ye, F., Fang, M., Li, S., and Yilmaz, E. Enhancing con- versational search: Large language model-aided infor- mative query rewriting. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Com- putational Linguistics: EMNLP 2023, pp. 5985-6006, Singapore, dec 2023. Association for Computational
Zheng, L., Yin, L., Xie, Z., Huang, J., Sun, C., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E.,, Barrett, C., and Sheng, Efficiently programming large language
14

---

CursorCore: Assist Programming through Aligning Anything
models using sglang. arXiv preprint arXiv: 2312.07104, 2023a.
Zheng, Q. Xia, X., Zou, X., Dong, Y., Wang. S., Xue, Y., Shen, L., Wang. Z., Wang. A., Li, Y., Su, T., Yang. Z., and Tang, Codegeex: pre-trained model for code generation with multilingual benchmarking humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6-10, 2023, pp. 5673-5684. ACM, 2023b, doi: 10. 1145/3580305.3599799., URL https: /ttp::/doooorroooo 10. 20.114553580305.35999
Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., Brunner, S., Gong, C., Hoang, T., Zebaze, A. R., Hong, X., Li, W.-D., Kaddour, J., Xu, M., Zhang, Z., Yadav, P., Jain, N., Gu, A., Cheng, Z., Liu, J., Liu, Q., Wang, Z.,, Lo, D., Hui, B., Muennighoff, N., Fried, D., Du, X., de Vries, H., and Werra, L. V. Bigcodebench: Benchmarking code genera- tion with diverse function calls and complex instructions. arXiv preprint arViv: 2406. 1577, 2024.
15

---

CursorCore: Assist Programming through Aligning Anything
A. Related Work
A.1. Al-Assisted Programming
Al-assisted programming has a long history, encompassing various tasks such as clone detection (Lu et al., 2021), knowledge tracing (Liu et al., 2019; Li et al., 2022; Gao et al., 2025), data mining (Wang et al., 2023a; Zhang et al., 2024c), code retrieval (Li al., 2024b;c; 2025), code summarization (Jiang et al., 2025; Sun et al., 2024), program synthesis (Chen et al., 2021; Austin et al., 2021), automatic program repair (Gulwani al., 2016; Zhao et al., 2024), code editing (Wei et al., 2023a), and code optimization (Shypula et al., 2024). These tasks attempt to incorporate wide range of information into their processes, such as historical edits (Gupta et al., 2023; Zhang et al., 2022) and user instructions (Cassano et al., 2023b), In the past, however, they were typically addressed by custom-built models, which were difficult to scale across different tasks and types of information. With the rise of LLMs, Al-assisted programming increasingly leverages LLMs to handle multiple types of tasks simultaneously. Numerous high-quality open-source and closed-source products, such as Continue (Continue- Dev. 2024), Aider (Paul-Gauthier, 2024), Copilot (Github-Copilot, 2022) and Cursor (Cursor-AI, 2023), are based on this approach.
A.2. Code Models
Recently, LLMs have attracted significant attention in the research community for their impact on enhancing various aspects of code intelligence. Open-source code L.MSs like Codellamm (Rozière al., 2023; Touvron et al., 2023), Seepseek-Coder (Guo et al., 2024a; DeepSeek-Al et al., 2024), StarCoder (Li al., 2023; Lozhkov et al., 2024), Codegemma (Team et al., 2024), Codestral (Mistral-AI, 2024a), Codegeex (Zheng et al., 2023b), Yi-Coder (AI et al., 2024), and Qwen-Codes Coder (Hui et al., 2024) have made substantial contributions by utilizing large code corpora during training. Some models, such as WizardCoder (Luo et al., 2024b), OctoCoder (Muennighoff et al., 2024), CodeL.lama-Instruce MagiCoder (Wei et al., 2023b), Yi-Coder- Chat, and Soven-Coder-Instruee have been fine-tuned using instruction data collected through methods like Self-Instruce (Wang al.. 2023b; Taori et al., 2023), Evol-Instruct, and OSS-Instruct. These models are specifically trained code-related instructions, improving their ability follow coding instructions. They have made significant breakthroughs in tasks like code completion and editing.
Spepseek-Coder-IIstiee
A.3. Code Benchmarks
HumanEval (Chen al., 2021) is one of the most well-known benchmarks in the code domain, featuring several variants that extend it to different programming languages, extra tests, and broader application scenarios. Other notable benchmarks include MBPP (Austin et al., 2021) for program synthesis, DS DS1000 (Lai et al., 2022) for data science tasks, SWE-Bench (Jimenez et al., 2024) for real-world software engineering problems, and CanItEdit/ / CodeEditorBench (Cassano et al., 2023b; Guo et al., 2024b) for code editing. Additionally, LiveCodeBench (Jain et al., 2024) focuses on contamination-free evaluations, while ClassEval((u et al., 2023), Bigcodebench (Zhuo et al., 2024) and Naturecodebench (Zhang et al., 2024b) provide comprehensive program synthesis assessments. CRUXEval (Gu et al., 2024) targets reasoning, ErossCodeEval (Ding et al., 2023) focuses on repository-Ieve code completion, and Needle in the code (Hui et al., 2024) is designed for long-context evaluations.
B. Code modification representation
As discussed in Section 2.3, there are various ways to represent code modifications. Many previous works have explored techniques for instruction-basee code editing (Wei et al., 2023a; Muennighoff et al., 2024; Paul-Gauthier, 2024; Sweep-AI, 2024). We build upon these works with the following formats, as shown in Figure 9:
Whole file format (WF) We use the entire code, allows for a straightforward representation of the modifications. However, when only small parts of the code are changed, this method leads to redundancy, especially for long code files. Certain mitigation can be achieved through technologies such as retrieval-basee speculative decoding (Yang et al., 2023; He et al., 2024).
Unified diff format (UD) The diff format is a common way to represent code changes, widely adopted for its efficiency and readability. Among various diff formats, unified diff is one of the most popular, as it efficiently shows code changes while reducing redundancy. It is commonly used in software tools such as git and patch.
16

---

CursorCore: Assist Programming through Aligning Anything
Location-and-change format (LC) To further reduce redundancy, consider further simplify the diff formats by showing only the location and content of the changes. The location is based on line numbers. Some reports indicate that LLMs often struggle with localization, insert line numbers into the code to assist them.
Search-and-replace format (SR) Another option is to eliminate the need for localization altogether by simply displaying the part to be modified alongside the updated version. This format eliminates the need for line numbers.
We conduct experiments using Deepseek-Code Sonreek-Cooer--eee 1.3B with these formats. For quick experiments, we train the model on data generated by AI Programmer. We then evaluate their performance APEval, with results shown in Figure 10. In programming assistance tasks, where real-time performance is critical, such in tasks like auto completion or editing, the generation speed becomes particularly important. The number of tokens in both input and output directly affects the model's speed, and the editing format greatly impacts the token count. Therefore, we also report the average input-output token count for each format in Figure 11.
1 def func(s):
s = :[::-1]
32
Extra tests
Base tests
Input
return 5
400
Output
350
WF
UD
@e -2,2 +2 00
Pass (as=
28-
def func(s):
$ s[::-1]
return s[::-1]
return
return s[::-1]
200
LC
SR s[::-1]
150
1,3
return 5
return s[::-1]
return s[::-1]
UD
WF
Ub
LC
Figure 9. Different formats for representing code modifi- cations.
Figure 10. Performance of models
Figure 11. Context length for models using different formats on APEval.
ing different formats on APEval.
The results show that using WF yields the best performance, followed by SR and LC, with UD performing the worst. In terms of token usage, LC uses the fewest tokens, followed by SR and UD, while WF uses the most. The average token count for SR and UD is only slightly lower than that of WF, as they are more concise for small code changes, when a large portion needs modification, they must include both versions, making them less efficient than using WF instead.
Recent research has pointed correlations and scaling laws between model input and output length, as well performance (OpenAI, 2024; Snell et al., 2024). Our results align with these findings. As the length increases, performance improves consistently across LC, SR, and WF. UD performs poorly in both token usage and performance, likely because it contains redundant information, such as both line numbers and content for the modified sections, where only one would suffice. This redundancy reduces the format's efficiency compared to the other three formats.
C. Details regarding the collection process of APEval
We inform the annotators about the function's entry point and its purpose, and allow them send instructions to the Al programming assistant at appropriate moments. We then use screen recording tools to capture the annotators' process of wrtining this function. Afterward, we manually analyze the recordings to construct our benchmark. The historical information, current code, and user instructions are all provided by annotators based on the specified function functionality, to cover various code editing scenarios.
During the process of creating the benchmark, in order to better evaluate the model's s ability to utilize historical edits and
17

---

CursorCore: Assist Programming through Aligning Anything
integrate this information with user instructions, we collected samples for the (H, C) and (H, C. U) types that required the use of relevant historical information accurately infer user intent. If a sample contained only single type of information (such only C or only U), it might be impossible to provide an adequate answer due to a lack of sufficient information.
In our benchmark collection process, we initially annotated one programming process for each task. For some tasks, the annotators consulted the programming assistant; for others, they did not. Similarly, some tasks involved complex editing histories, while others did not. Upon reviewing the data, we found that for certain tasks, it was nearly impossible to collect realistic programming processes containing specific types of information. For example. Some tasks are straightforward and can be completed with just few lines of code. Programmers who have undergone basic training can write these solutions quickly without needing to consult assistant repeatedly revise their code. Conversely, some tasks may involve calling specific libraries or algorithms that most annotators are unfamiliar with, leading them to rely on the programming assistant. It would be unrealistic and counterproductive to instruct annotators to "always consult the Al" or "edit your code repeatedly," as this would deviate from real-world scenarios and undermine our intention to use these reasons, we did not collect programming traces for the entire test set. While we still hope that the number of samples of four different combinations is at least balanced, At this stage, the number of samples for combinations involving all four data types was relatively similar. So we asked annotators to label additional programming process traces for combinations with fewer samples and collected the corresponding traces. Meanwhile, for combinations with slightly more samples, we discarded some of their traces. Subsequently, we manually translated them into different programming languages. Through this process, we established our final benchmark. Simplified examples of the annotated data is illustrated in Figure 12.
data. Considering
Example 1
Example 2
# Current def has_close_ (n t):
History 1
def iner_list(l: list):
for in range(len(n 1)): for j in range(i 1, len(n)):
return [x++ for x in 1]
Current
if n[i]- n[j]= < or n[j] n[i] < t:
def incr_lister list):
Figure 12. Simplified examples of APEval, which covering various code editing scenarios that require integrating multiple types of information to infer user intent. The left example checks any two numbers in list are closer than a given threshold. The current logic flawed and should verify if the absolute difference between two values is less than t. The model must detect this issue, fix the error, and generate the remaining code. The right example shows a programmer replacing incorrect code with corrected version. Without historical edits, the model cannot infer the function's 's intent. Thus, it must use edit history to make accurate code edits.
D. A Additional details about Programming-Instrree
In our code editing records, we place no limits on the granularity or number of edits. Changes between two code versions may involve anything from single character to multiple extensive modifications. However, data collected from various sources may be compressed, resulting in incomplete records. This compression can lead higher proportion of large-scale edits, particularly in Git Commit data. To address this issue, we propose decomposition strategy: when there are multiple changes between versions, we break them down into single-step modifications, with the steps ordered randomly. For Git Commit data, we apply this decomposition strategy with a 90% probability, while for AI Programmer and Online Judge Submission data, we apply it with a 50% probability.
We randomly select a time point from the records to represent C. In practice, we prefer the model to provide assistance at earlier stages. Thus, we implement a simple rule where the random selection follows an exponential distribution, with the probability of selecting each time point decreasing by 10% with each subsequent step. This biases the model toward choosing earlier time points.
In addition to generating H and U, as discussed in Section 4.2, we also simulate the programmer's 's specification of the target area and model interactions in a chat-style format. The target modification area is created using a random algorithm, as described in Appendix F, while the chat-style interaction is generated using LLMs which is similar to the generation of instructions. Prompts used for it are provided in Appendix O.
18

---

CursorCore: Assist Programming through Aligning Anything
E. Training details
Our models are trained for 2 epochs using the Transformers library (Wolf et al., 2020). We enhance memory efficiency and speed with techniques such as Deepspeed ZeRO3 (Rajbhandari et al., 2019), ZeRO Offiloa (Ren et al., 2021), FlashAttention2 (Dao, 2024), and triton kernels (Hsu et al., 2024). We calculate the maximum sequence length that can be processed per batch based the available VRAM. Using the First-Fit Decreasing algorithm (Kundu et al., 2024), pack training samples to ensure that each batch reaches its maximum sequence length, thereby optimizing training speed. The training process employs the Adafactor optimizer (Shazeer & Stern, 2018) with a learning rate of 5e-5, coupled with a cosine scheduler featuring 15 warm-up steps.
F. Target area representation
To modify code, programmers often specify the parts requiring changes, typically in one of two ways: either by clicking with the cursor to indicate general area or by selecting specific text range with defined start and end points. We model both cases using special tokens: "< target] > for cursor positions, and "< target_start| >> and "< target_end >> to mark the selected region's boundaries. While collecting training data, we determine modification locations based on the code differences before and after changes. real-world applications, the decision to provide explicit locations- and their granularity- varies among programmers. To account for this variability, we introduce randomized choices for determining the form and location, integrating this approach into the Programming- Instruct pipeline.
Extra tests
Pass
on APEval both with and without location information to assess its impact on performance. The results in Figure 13 show that including location information has minimal effect, likely because most APEval examples are relatively short, enabling LLMs to easily infer modification locations, much like humans do without a cursor. Previous works, such as those on automated program repair (Zhang et al., 2024a), have emphasized the importance of identifying the modification location. We believe this emphasis stems from traditional code completion and insertion paradigms, as well as the natural alignment of specifying modification points with human thought processes. However, with the advancement of LLMs, the benefit of providing location information diminishes when generating code at the function or file level. This may need further exploration in longer contexts, such as repository-leve editing tasks.
We evaluate CursorCore-DS.
32-
30
wio Target Target Area Area
Figure 13. With and without the use of location information on APEval.
G. Discussion about thought process
Incorporating reasoning processes in prompts has been shown to improve model performance, as demonstrated in various works like CoT (Wei et al., 2022) and ReACT (Yao et al., 2023). Some studies have even integrated these processes into the training phase to further enhance effectiveness (Zelikman et al., 2022). In this work, we also explore a self-taught approach, where we prompt LLMs to reverse- - the reasoning process from outputs and incorporate them into the model's output during training. Our model and data setup follow the same configuration as described in Appendix B enable quick experiments. The evaluation results are shown in Figure 14.
tests Base tests
34
800
Input Output
700
32
600
30
5c0
400
28-
After incorporating reasoning into training, the model shows slight performance improvements, but the output length increases significantly. The tokens used for reasoning often exceed those in the modified code. Since many programming- assist applications require real-time responses, longer reasoning times may be impractical, so we do not integrate this process into CursorCore. We believe that the decision to use reasoning processes should be based on a combination of factors, such as performance, latency, model size, and specific application requirements.
500
26-
200
100
24
Base
Thought
Base
Thouglc
Figure 14. Performance of models using thought process or not on APEval.
19

---

CursorCore: Assist Programming through Aligning Anything
H. Data Selection Ablation
train the smallest model Deepseek-Coder- on different combinations of datasets to determine the optimal data mix. The results of the ablation study are shown in Figure 15.
AI Programmer has the highest data quality Among the various data sources, the model trained on the Al Pro- grammer dataset achieve the best performance on APEval. We believe this is primarily because the data aligns well with the required format of APEval. Moreover, unlike other data sources such as Git Commit, the AI Program- mer data is almost entirely synthesized by LLMs, except for the initial code. As LLMs have advanced, the quality of their generated data has generally surpassed that of data collected and filtered from human-created sources.
Extra tests
Base tests
Online-Submit
Online-Submit (Py)
Git-Commit
Git-Commit (Py)
Alprogrammer
Alprogrammer
Commit (Py)
Importance of mixing data with different information types We find that using high-quality chat-style data alone, such as the Instruutt dataset, does not achieve the desired performance; it underperforms compared to the AI Programmer dataset. However, when combining both datasets, the model shows notable improvement. This indicates that to better align the model with a variety of data and information, it is necessary to use datasets containing diverse types of information.
Evol-Instruct
Alprogrammer
Evol-Instruct
Ours
5
10
25
40
Pass@ ((%)
Figure 15. Selection Ablation on APEval.
Our final selection We combine data from all sources for training. Since current research on Code LLMs primarily focuses on performance in Python, and training on multilingual data leads to slight decrease APEval scores, we use only the Python part of the Git Commit and Online Judge Submission datasets. As a result, we get CursorCore series models.
Conversation retrieval for Assistant-Conversatton
Not all code editing records necessary for inferring user intent and predict- ing output. Some past modifications, such simple typos corrected shortly after, offer little value to future predictions, and thus can be safely removed. Additionally, programmer continuously interacts with the model without deleting these records, the editing history will accumulate and grow until exceeds the model's maximum context length. This could negatively affect performance and speed.
Base tests
Pass@118)
To address this, it is essential to compress the editing history or retrieve only the relevant portions. Similar to how many conversation retrieval techniques, such as memory modules (Packer et al., 2023), prompt compression (Jiang et al., 2023) and query rewriting (Ye et al., 2023), are used to manage dialogues for chatbots, these methods can adapted for handling code editing records. In this work, we explore basic approach, sliding window, to investigate possible solutions. When the number of historical editing records surpasses predefined threshold, the model automatically discards the oldest entries.
2
Sliding Window Size
We evaluate this method on APEval, as shown in Figure 16. The impact of setting a sliding window a certain size on the results minimal, indicating that compressing the historical records effectively balances performance and efficiency.
Figure Performance models using dif- ferent sliding window sizes evaluated on APEval.
20

---

CursorCore: Assist Programming through Aligning Anything
Table 5. Evaluation results EvalPlus, CanltFdit and OctoPack.
EvalPlus
CanItEdit
OctoPack
Model
HE (+)
MBPP (+)
Desc.
Lazy
HE Fix
DS-Coder-6.7BB. 6.7-brr
47.6 (39.6)
70.2 (56.6)
34.3
27.6
23.8
1--Coder-6.7B-llee
74.4 (71.3)
75.1 (66.1)
41.9
31.4
42.1
CursorCore-DSS-6:1 (Chat)
78.0 (73.2)
74.1 (63.8)
45.7
31.4
43.3
CursorCorc-DS-6.75 (Inline)
73.8 (67.1)
71.2 (59.8)
38.1
32.4
32.3
CursorCore-DS-6e (Tab)
72.0 (65.9)
74.3 (63.0)
6.7
6.7
25.6
Yi-Coder-9B
(47.0)
69.6 (56.9)
47.6
34.3
32.3
Yi-Coder-9B-Chal
83.5
84.4 (71.4)
58.1
45.7
54.3
MursorCore-Yi-99) (Chat)
84.1 (79.3)
84.4 (73.5)
56.2
41.0
56.1
CursorCore- Yi-9B (Inline)
79.9 (72.0)
83.6 (69.6)
48.6
35.2
33.5
CursorCore- Yi-9B (Tab)
79.3 (71.3)
83.9 (72.5)
10.5
10.5
25.6
Qwen2.5-Coder-7B
61.6 (53.0)
76.7 (63.0)
49.5
40.0
17.1
Qwenn.55CCoder-77BBmms
87.2 (83.5)
83.5 (71.7)
53.3
44.8
54.3
CursorCore-QW2.5577 (Chat)
80.5 (75.6)
77.0 (64.3)
51.4
44.8
50.6
CursorCore-QW2.5-21 (Inline)
79.9 (73.2)
77.0 (64.0)
57.1
39.0
41.5
CursorCore-QW2.5-77 (Tab)
79.9 (74.4)
75.1 (64.3)
5.7
5.7
27.4
DS-Coder-1133BBBar
34.8 (26.8)
55.6 (46.9)
13.3
8.6
1.2
OS-Coder-1.3BB-ll
65.2 (59.8)
61.6 (52.6)
26.7
17.1
29.3
CursorCore 1.31 (Chat)
68.9 (63.4)
61.9 (49.7)
21.9
14.3
30.4
SeraaCCor-----.333 (Inline)
57.9 (53.7)
60.1 (51.1)
25.7
17.1
17.1
CursorCore-DDS11.3 (Tab)
63.4 (57.3)
65.6 (54.8)
2.9
2.9
8.5
Yi-Coder-1.5B
40.6 (34.8)
59.0 (50.0)
21.0
12.4
3.7
1.ht-hht
67.7 (64.0)
66.9 (56.6)
21.0
23.8
37.2
CursorCore-Yi-- Yi-1.5B (Chat)
68.9 (65.2)
65.6 (54.8)
27.6
24.8
38.4
CursorCore- Yi-1.5B (Inline)
60.4 (54.3)
65.6 (55.0)
28.6
24.8
22.6
Yi-- (Tab)
67.1 (59.1)
66.1 (56.6)
4.8
4.8
20.1
Qwen2.5-Coder-1.5B
43.9 (36.6)
69.3 (58.5)
31.4
22.9
4.9
Qwenn..5-Coder-1.5BB
70.7 (66.5)
69.3 (59.4)
28.6
21.0
32.9
CursorCore-QW225-11.88 (Chat)
71.3 (65.9)
69.3 (58.5)
31.4
22.9
36.6
CursorCone-QWW25551.55 (Inline)
66.5 (60.4)
68.5 (58.2)
23.8
20.0
36.6
CursorCone-QQ225511.38 (Tab)
64.0 (58.5)
67.2 (56.6)
1.0
1.0
13.4
J. Evaluation results of other benchmarks
We also evaluate CursorCore on other well-known benchmarks. We use HumanEval+ and MBPP+ (Liu al., 2023) to evaluate Python program synthesis, CanItEdit (Cassano et al., 2023b) for instructional code editing, and the Python subset of HumanEvalFix from OctoPack (Muennighoff et al., 2024) for automated program repair. All benchmarks based on their latest versions, and HumanEvalFix uses the test-based repair version as described in the original paper. To generate results, we consistently use vLLM (Kwon et al., 2023) due to its versatility and support for customized conversation formats. Evaluations are conducted within each benchmark's 's execution environment.
Unlike previous LLMs, CursorCore supports multiple input formats, and different formats may produce different results. To comprehensively showcase this, we categorize input formats based on specific assisted programming scenarios into three cases:
Chat: Similar to the chat format of ChatGPT (Ouyang et al., 2022), we wrap the query before passing it to the model, which returns a response in chat style. The final result obtained after post- post-processing.
Inline: Similar to Copilot Inline Chat (Github- (Github-Copilot,, 2022) and Cursor Command (Cursor-Al, 2023) scenarios, corresponding to the combination C and U Ansissaott Compared to the Chat mode, is more tightly integrated with the IDE and returns less additional content.
21

---

CursorCore: Assist Programming through Aligning Anything
Tab: Similar the use case of Copilot++ (Cursor-AI, 2023), it is the most automated of all scenarios. We provide only the to the model. For instructional code editing and automated code repair, explicit instructions are passed.
Evaluation results are shown in Table 5. Our model outperforms the corresponding Instruction-tunce and base models across several benchmarks. However, the performance of the 6B+ model, when compared to its corresponding models, is not as strong as that of the IB+ model. Notably, with the recent release of Qwen2.5-Coder-77 at the start of our experiments, we outperform it on only one benchmark, while other models achieve better performance across more benchmarks. We attribute it to the quantity of high-quality data: larger models require more high-quality data for training. While the current dataset is sufficient to train a highly effective IB+ model, additional data needed to train a more competitive 6B+ model.
We analyze the evaluation results of various input types defined in real-world assisted programming scenarios. The results of the Chat and Inline modes are comparable, with Chat mode showing a slight advantage. We attribute this to the flexibility of the Chat format, which allows the model to output its thought process and thus enhances output accuracy. The Tab mode shows comparable results on EvalPlus but underperforms on HumanEvalFix and struggles with CanltEdit, likely due to variations in the informational content of task instructions. For program synthesis based on doestrings, instructions like "complete this function" " provide minimal additional context. In contrast, program repair tasks provide crucial information by indicating the presence of errors. When only code available, the model must first determine correctness independently. Instructional code editing tasks clearly state objectives, such implementing a new feature, requiring the model fully understand the given information, as accurate predictions based solely on code nearly impossible.
To further evaluate the ability of CursorCore to leverage historical information for editing and its applicability to more general software engineering tasks, we additionally conduct experiments on Zeta Industrduss 2025), DS 100000 (Lai et al., 2022), and ClassEval (Du al., 2023), as shown in Table 6. For Zeta, report the average accuracy across all evaluated samples, with correctness judged by GPT-4o based on the associated assertion text. For DS 150000 and ClassEval, we choose to use the Inline and Tab modes, as they most closely resemble the original formats of them. We report the average score across all samples, using the subset of ClassEval that evaluates class-level generation. All generations are produced under greedy decoding. These results collectively demonstrate the strong effectiveness of CursorCore.
Table 6 Evaluation results Zeta, DS1000 and ClassEval.
Model
Zeta
DS1000
ClassEval
DS-Coder-1.3B-Bal
18.2
16.2
13.0
DS-Coder-1.3B--In
42.4
20.7
13.0
CursorCore-DS-1.33
45.5
21.2
17.0
K. Additional evaluation results on APEval
We also report the evaluation results of various versions of other well-known models on APEval, as shown in Table 7.
L. Multilingual evaluation results on APEval
We report the evaluation results on multilingual versions of APEval, as shown in Tables 8 to 13. CursorCore series achieve state-of-the-art performance across all languages, strongly demonstrating the effectiveness of our approach.
M. Chat template
Our model's chat template (OpenAI, 2023) is adapted from the ChatMI. template, where each message in the conversation is restricted to one of the following roles: system, history, current, user, or assistant. The assistant's output includes both code modifications and chat interaction with the user. To indicate code changes, we use two special tokens "< | nnxtt_ttar|| > effectively and
and "< next_end| > to wrap the code modification parts. This approach models Assistant-Conversiln compatible with standard ChatML templates and chatbot applications. Figure 17 illustrates an example of our chat template, while Figure 18 presents examples of the chat template when using the LC and SR modes described in Appendix B.
N. Prompts for evaluation
We report the prompts used to evaluate base LLMs on APEval in Table 20, while the prompts used for evaluating instruct LLMS are presented in Table 21.
22

---

CursorCore: Assist Programming through Aligning Anything
Table 7. Additional evaluation results of LLMs APEval.
Model
C
H,
C,
H, C,
Total
Llama-3.2-1B
0.0 (0.0)
14.6 (12.2)
2.4 (4.9)
14.6 (12.2)
7.9 (7.3)
-lama-32--B---- Instructt
7.3 (7.3)
14.6 (14.6)
19.5 (19.5)
22.0 (19.5)
15.9 (15.2)
Gemma-2-28
(7.3)
(2.4)
(12.
(4.6
9.8 (7.9)
Llama-3.2-3B
14.6
26.8 (9.8)
(17.1)
18.9 (15.2)
StarCoder2-3B
19.5 19.5 (19.5)
(17.1)
2190
217.1
20.7 (18.3)
Phi-3.5-3.8B-Inst
(14.6)
33.1
(3..1)
29.3 (26.2)
StarCoder2-7B
7.3 (7.3)
(12.2)
114.))
(17.1
15.9 (12.8)
Llamm-331188
12.2
17.1
(19.5)
217.1)
17.7 (15.9)
Gemma-2-99
22.0 (22.0)
17.1 (17.1)
(17.1
20.1 (18.9)
Codegeex4-Al1-99
(41.5)))5)
(3120) (31.7
34.1 (34.1)
46.3 (42.1)
SturCoder2-15B
26.8 (24.4)
(22.0)
(3696)
29.3 (24.4)
31.1 (26.8)
DS-CCoder-V2-16H-
(2..4)
(19.5)
(26.8
25.0 (17.1
DS-Coder--V-II6B-Ins
(41.5)
(31.7)
(63.4)
(31.7)
47.6 (42.1)
Gemma-2-27B
(3666)
(22.0)
56.1
26.8 (24.4)
36.0 (32.3)
Gemma-2-27B-I
(5.41)
48.8 (41.5)
(63.4)
41.5 (39.0)
55.5 (50.0)
OS-Coder-33BB-BB
(31.7)
26.8 (22.0)
(36.6)
24..4
31.7 (28.7)
Llama-3.1-700
(24.4)
(22.0)
(39.0)
29.3 (24.4)
31.1 (27.4)
Llama-3, 1lama-3.1-70B-mm. 111-7Bnnnnst
61.0 (56.1)
(46.3)
65.9 (58.5)
(51.2)))
57.3 (53.0)
Qwen2.5-72B
63.4 (61.0)
36.6(34.1)
75.6 (63.4)
39.0 (34.1)
53.7 (48.2)
DS-Coder- V2-236B-Baa.
41.5 (39.0)
36..631.7)
58.5 (56.1)
36.6 (34.1)
43.3 (40.2)
GPT-4o-Mini
17.1 (17.1)
36.6 (31.7)
78.0 (70.7)
53.7 (43.9)
46.3 (40.9)
O. Prompts for data collection
We design specific system prompts and few-shot examples to collect high-quality training data, as we find that many examples are very difficult to complete with current LLMs, and only a few of them can be successfully completed using rough prompts. For Programmer, we utilize LLMs to simulate programmers at three different skill levels, with each level using a distinct set of prompts as shown in Tables 14 16. Additionally, prompts used for evaluating whether the outputs align with user intent, generating user instructions, and facilitating chat interactions between models and users outlined in Tables 17 to 19. Partial few-shot examples shown in Figures 19 to 24.
P. Limitations and future work
Repo-level development assistance In this work, we focus on supporting the development of single files or function-level code. However, real-world development operates the repository level, involving multiple files and greater interaction with IDEs. Previous research has made notable advances in repository-Ievel tasks such code completion (Zhang et al., 2023), issue fixing (Jimenez et al., 2024), and documentation generation (Luo et al., 2024a). Repository-level code assistance deals with larger datasets, and achieving optimal performance and speed will require more effort. We leave the exploration of multi-file repository-level programming assistance and leveraging additional IDE interactions for future work.
More scenarios and criteria for evaluation Our benchmark is relatively small and based on a multilingual extension of FyaanFval, making insufficient cover all development scenarios. Beyond using the classic Pass@k metric to evaluate accuracy, other criteria should also be considered, such as evaluating the model's 's efficiency, security, and redundancy (Huang al., 2024; Pearce et al., 2021; Li al., 2024a).
Preference-basee optimization Methods like PPO (Schulman et al., 2017) and DPO (Rafailov et al., 2023), which optimize models based on human preferences, have been widely used in LLMs. In programming assistance, programmers can provide feedback on predicted outputs for identical or similar coding processes, further optimizing the model (Shinn et al., 2023). To enable this, a significant amount of feedback data from programmers using Al-assisted tools should be collected or synthesized.
23

---

CursorCore: Assist Programming through Aligning Anything
Table 8. Evaluation results
LLMs on the C++ version APEval.
Model
H,
C, U
C,C,
Avg.
6B+ Models
DS- AS-CCder-6..7 Baader-6.7B-Base
41.5
36.6
58.5
31.7
42.1
DS-Coder-6.7B--
29.3
34.1
61.0
39.0
40.9
Yi-Coder-9B
29.3
43.9
34.1
29.3
34.1
Yi-Coder-9B-Chir
34.1
36.6
56.1
39.0
41.5
Qwen2.5-Codde-77B
39.0
41.5
65.9
39.0
46.3
Qwen2.5-Codder-7B-
12.2
31.7
63.4
41.5
37.2
CursorCore-DS-61 7B
48.8
43,9
65.9
34.1
48.2
CursorCore- Yi-9B
53.7
46.3
68.3
39.0
51.8
CursorCore-QW2.5-78
46.3
43.9
63.4
43.9
49.4
1B+ Models
DS-Coder-1.33 1.3--BaseBeeeeeee
29.3
4.9
34.1
9.8
19.5
DS-Coder- 1.3B-rr-
31.7
39.0
41.5
36.6
37.2
Yi-Coder- 1.5B-Cht
24.4
26.8
17.1
22.0
22.6
Yi-Coder-1.5B
19.5
22.0
14.6
14.6
17.7
Qwen2.5-Coder-1.5
34.1
31.7
48.8
36.6
37.8
Qwen2.5- 1.55-d Inst
17.1
31.7
34.1
34.1
29.3
CursorCore -D---33
39.0
39.0
56.1
39.0
42.7
CursorCore- 1.-
46.3
34.1
56.1
34.1
42.7
CursorCore-QW2.5-11.55
41.5
39.0
58.5
36.6
43.9
Table Evaluation results LLMs on version ofAPEval.
Model
C
H,
C, U
U,
Avg.
6B+ Models
DS-Coder-6.777 (.recr--67--Basse
51.2
43.9
61.0
46.3
50.6
DS-Coder-6.77B
48.8
34.1
68.3
$1.2
50.6
Yi-Coder-9B
43.9
46.3
36.6
41.5
42.1
Yi-Coder-9B-Chn
43.9
41.5
56.1
39.0
45.1
Qwen2.5-Coder-77
63.4
53.7
70.7
46.3
58.5
Qwen2.5-Coder-77B
22.0
46.3
70.7
51.2
47.6
CursorCoreeDS-66.77
63.4
56.
65.9
48.8
58.5
CursorCore- Yi-9B
56.1
61.0
68.3
46.3
57.9
CursorCore-QW2.557B
65.9
58.5
78.0
51.2
63.4
1B+ Models
DS-Coder-1e 1.3ee-11.BBBBass
29.3
24.4
36.6
26.8
29.3
L-CCoder-L33B-Ins
39.0
39.0
48.8
43.9
42.7
Yi-Coder-1.5B
24.4
29.3
24.4
36.6
28.7
Yi-Coder- 1.5derr-
31.7
24.4
12.2
26.8
23.8
Qwen2.5-Coder-1.55p
39.0
43.9
53.7
48.8
46.3
Qwen2.5- 1.5BBInst
34.1
36.6
53.7
48.8
43.3
CursorCore- Saar-CCrr----11... DS31.3B
43.9
43.9
58.5
43.9
47.6
KursorCore-Yi-L1
53.7
48.8
53.7
46.3
50.6
CursorCone-QQW25-11.SB
53.7
51.2
53.7
51.2
52.4
24

---

CursorCore: Assist Programming through Aligning Anything
Table Evaluation results of LLMs the JavaScript version of APEval.
Model
C
H, C
C, U
H, C, U
Avg.
6B+ Models
B--Coodr--.7B-BBB
29.3
26.8
51.2
39.0
36.6
DS-Coder-6.7B-Img
48.8
41.5
58.5
36.6
46.3
Yi-Coder-9B
22.0
41.5
17.1
39.0
29.9
Yi-Coder-9B-Chat
48.8
34.1
56.1
41.5
45.1
Qwen2.5-Codder-78
51.2
36.6
68.3
43.9
50.0
Qwen2.5-Coder-7BBIn
22.0
34.8
63.4
43.9
40.9
CursorCore-DS-6.777 7B
48.8
41.5
61.0
36.6
47.0
CursorCore-Yi-op Yi-9B
56.1
51.2
70.7
43.9
55.5
CussoCoore-QW2.5577
51.2
36.6
70.7
46.3
51.2
1B+ Models
DS-Coder-1.3B 1.3B-Base
19.5
9.8
26.8
26.8
20.7
DS-Coder-113B--B
29.3
34.1
36.6
36.6
34.1
Yi-Coder-1.35
19.5
17.1
7.3
22.0
16.5
Yi-Coder- 1.5--Cha
17.1
19.5
9.8
12.2
14.6
Qwen2.5-Coder-1.5.
31.7
31.7
48.8
39.0
37.8
Qwen2.5-Codeee 1.5--nnst
19.5
26.8
43.9
39.0
32.3
CursorCore-DS-11
34.1
36.6
41.5
43.9
39.0
CursorCore-Yi-l...
34.1
34.1
41.5
41.5
37.8
CursorCore-QW22.511.338
34.1
34.1
51.2
41.5
40.2
Table 11, Evaluation results L..Ms version APEval.
Model
C
H, C
C,
C,
Avg.
6B+ Models
DS-Cooder-677B-Bar
26.8
24.4
31.7
39.0
30.5
DS-Coder-6.7BB
29.3
39.0
56.1
43.9
42.1
Yi-Coder-9B
39.0
29.3
39.0
36.6
36.0
Yi-Coder-9B-Chai
34.1
24.4
41.5
31.7
32.9
Qwen2.5-Coder-77
41.5
36.6
53.7
41.5
43.3
Owen2.5-Cooder-715
12.2
22.0
58.5
46.3
34.8
MunsorCore-D--6
56.1
34.1
56.1
46.3
48.2
CursorCore- Yi-9B
48.8
34.1
56.1
36.6
43.9
CursorCore-QW2.5-78
58.5
41.5
58.5
51.2
52.4
1B+ Models
DS-Coder- 1.3B-Base
22.0
7.3
26.8
29.3
21.3
DS-Coder- (ooderr-133...o I.3B-Inst
31.7
22.0
36.6
34.1
31.1
Yi-Coder-1.5BB
7.3
14.6
2.4
9.8
8.5
Yi-Coder- 1.55.er--..5B-hat
19.5
14.6
17.1
29.3
20.1
Qwen2.5-Coder-1..53
34.1
24.4
39.0
36.6
33.5
Qwen2.5- Coder-1dd5p 1.5B-Inst
22.0
22.0
36.6
34.1
28.7
CursorCore- BossoCCor--DS-133B
41.5
34.1
43.9
41.5
40.2
CursorCore- -i-oorr------.555 1i-1.BB
46.3
29.3
43.9
34.1
38.4
MursorCore-QW2.5-11
51.2
34.1
61.0
41.5
47.0
25

---

CursorCore: Assist Programming through Aligning Anything
Table 12. Evaluation results LLMs
Rust version APEval.
Model
H,
C,
C,C
Avg.
6B+ Models
DS JS-CCder-6., 6.ae-..677-BBea
26.8
26.8
31.7
39.0
31.1
DS-Coder-6.7BB
29.3
31.7
34.1
39.0
33.5
Yi-Coder-9B
29.3
34.1
29.3
39.0
32.9
Yi-Coder-9B-Chat
26.8
29.3
46.3
31.7
33.5
Qwen2.5-Coder-7B
46.3
34.1
48.8
36.6
41.5
Qwen2.5-Coxder-7B-Imm
9.8
22.0
51.2
36.6
29.9
CursorCore-DS-6.75
36.6
34.1
34.1
39.0
36.0
CursorCore-Yi-99
41.5
34.1
48.8
41.5
41.5
CursorCore-QW2.5-7B
48.8
39.0
51.2
39.0
44.5
1B+ Models
DS-Coder-1.3B--D
22.0
22.0
31.7
29.3
26.2
DS-Coder- 1.3B-Inst
19.5
26.8
36.6
31.7
28.7
Yi-Coder-1.5t
22.0
24.4
17.1
29.3
23.2
Vi-Coder-1.5B-CA
14.6
22.0
12.2
29.3
19.5
Qwen2.5-Coder-1155)
34.1
29.3
39.0
41.5
36.0
26.8
24.4
36.6
36.6
31.1
CursorCore-DS-1..
24.4
29.3
36.6
34.1
31.1
CursorCore- -i- Yi-1.5B
24.4
31.7
41.5
31.7
31.1
CursorCone-QQ22.5-11555
36.6
31.7
39.0
43.9
37.8
Table 13, Average evaluation results of LLMs across different language versions on APEval.
Model
C
H, C
C,
H, C,
Avg.
6B+ Models
SS-CCoder-677B-Bane
34.2
30.9
45.9
36.2
36.8
DS-Coder-6.7BB
40.3
37.0
58.1
40.6
44.0
Yi-Coder-9B
32.1
37.0
28.9
35.8
33.4
Yi-Coder-9B-Chat
40.6
34.2
54.9
36.6
41.5
Qwen2.5-Codde--78
49.6
40.7
62.2
39.8
48.1
Owen2.5-Cooder-715
16.7
33.8
63.8
43.5
39.5
CursorCore-DS-4e
53.7
41.9
58.5
40.2
48.6
CursorCore- Yi-9B
51.7
45.5
64.6
41.9
50.9
CursorCore-QW2.5-78
56.1
43.5
64.6
46.7
52.7
1B+ Models
DS-Coder- 1.3B-Base
20.3
13.4
28.9
23.6
21.5
DS-Coder- -ooder--13B.... I.3B-Inst
31.9
33.3
39.9
36.2
35.3
Yi-Coder-1.5BB
16.7
19.1
13.8
22.0
17.9
Yi-Coder- 1.50oerr-..5B-hat
22.4
17.9
19.5
23.2
20.7
Qwen2.5-Coder-1.553
36.2
31.3
46.8
39.9
38.5
Qwen2.5- 1.5B-Inst
22.4
26.4
41.5
37.4
31.9
CursorCore- Boser-CB--DDS
36.6
37.0
48.4
38.2
39.9
CursorCore- -i-oore------.555 1i-1.BB
41.8
35.3
50.8
37.4
41.1
CursorCore-QW2.55
43.9
39.8
54.9
42.3
45.2
26

---

CursorCore: Assist Programming through Aligning Anything
Integrated Development Environment (IDE)
Chat Template
< <|imsttart >system
Hi
You are a helpful programming assistant iimpettiimmmmmmme end
frequency[c] frequency[c]
<|im_start >-hstoory
ppython
frequency[c] frequency[c] +
Ha
"
>
def function(s):
<lim_start />history
2 for с in s: frequency[c] frequency[c]
"python
def function(s):
с in s:
frequency[c] = frequency[c] 1
H3
"

def function(s): for in frequency[c] +=
<I <|im_start| />history
python
def function(s):
for с in s:
frequency[c] +=
C
>

def function(s):
< din_start/>cumen
ppython
def function(s):
U
"
>
<lim_start/>use
Implement concisely.
"python
def (anction(s))
Implement it concisely.
" "<<lim_end]
def function(s):
sii__start/>assiitaan
frequency {c: s.count(c) for c in set(s))

 ppython def function(s):
return frequency
1 def function(s):
frequency = {c: s. c.cont((c) for c in set(s)}
frequency {c: count((() for in set(s)}
return frequency
return frequency
" "<<[nex__end > ><|ii_end]
Figure 17. Example of chat template and its corresponding demonstration in the IDE scenario.
Enhance performance with API calls We aim to integrate function calls (Patil et al., 2023) into the model to further enhance its capabilities, One potential application is incorporating function calls into the thinking process, such as retrieving information or executing partial code for feedback. Although our final models excludes this thinking step due to performance and speed considerations, we are exploring hybrid approaches to introduce this process while maintaining speed and combine it with other strategies for searching how edit. Another application leveraging function calls in output, where calling Python script for tasks like variable replacement might be more efficient than manually generating code blocks search-and cearch-and-replace strategies. For repository-lever changes, using terminal commands or IDE APIs could sometimes be more convenient solution.
Expand to other applications Our framework is designed for programming assistance applications, but the alignment approach can also be applied to other types of AI assistants. For example, in designing an art assistant, it should be able to predict the next drawing step based on the artist's previous drawing patterns, the current state of the canvas, and the artist's instructions. Extending this approach to design assistants for other applications is an interesting research direction.
27

---

CursorCore: Assist Programming through Aligning Anything
Chat Template (LC)
Chat Template (SR)
</im_start sssystmm
<I <lim_start) >>syttmm
You are helpful programming assistant:<| im__nd
You are a helpful programming assistant. | imm_edd|
<I <lim_start |>history
<I [lim_stae |>history
0,3
pyyhonn
python
frequency[c] frequency[c] +
frequency[c] Frequency[c]
< aerearch_and_replaaee
def function(s):
< <|im_end]]
for in 5: frequency[c] frequency[c] "<
<| <|im_start /
2,3
python
<|im_startt start//-hsstorry
frequency[c] = frequency[c]
" python


frequency[c] = frequency[c] 1
<I jm__ttar|| |>history
searchh_an__replace)
1,1
frequency[c] += 1
python
*<lim_end
for с in s:
[lim_starte start/>>istory
frequency[c] += 1
"python
"<lim_end|>>
def function(s): for s:
< siinsstar/><Cunge
python
frequency[c] += 1
def function(s):
aararchhanndreplace def function(s): "<|im_end|
<

<li__star|>
/im_start/>curen '. "python
pyython
Implement it concisely.
def function(s):
<<lim_end|>
'* "<|im_end| >>
<im_sttart |aassistant
<|im_start /
|nxt_staar|
"python
python
Implement it concisely.
frequency = {c: count((c) for in set(s)}
"><lim_end/>>
return frequency
<|in_start/>aasistarn
<<next_end]>>>
We uses dict comprehension to count the occurrences. is more concise and readable compared to before. im |

" python
def function(s):
< <|search_and_replacel/
def function(s):
frequency = {c: s. c.uutt((c) for c in set(s)}
return frequency
""<[next_end)>
We uses dict comprehension to count the occurrences. is more concise and readable compared to before.< <m__nd|>
Figure Example chat templates and SR modes.
28

---

CursorCore: Assist Programming through Aligning Anything
Table 14. Prompt designed to leverage LLMs for simulating the behavior novice programmer.
Please play the role of novice programmer. You required to write piece of code. Simulate the real process of repeatedly adding, deleting, and modifying the code. Please return the code block after each step of editing. While writing the code, make some mistakes, such as incorrect logic or syntax errors, etc.
Table 15. Prompt designed to leverage L.LSs for simulating the behavior of an ordinary programmer.
Please act as an ordinary programmer. Now, you need to write a piece of code. Please simulate the process of repeatedly adding. deleting, and modifying the code during the actual coding process. Please return the code block after each editing step. Try simulate the coding process of ordinary programmer much as possible.
Table 16. Prompt designed to leverage LLMs for simulating the behavior of expert programmer.
Please play the role of an expert programmer. You are now required to write a piece of code. Please simulate the process of repeatedly adding, deleting, and modifying code during the real coding process. Please return the code block after each step of editing. During the coding process, you should be professional possible.
Table Prompt designed generate user instructions.
You are programming assistant. The following content includes information related to your programming assistance, which may contain the record of the programming process, the current code, the git commit after all changes, relevant details about the problem, and your predicted modifications. Please generate an instruction for you to make the corresponding modifications, ensuring it resembles instructions typically given by a human programmer. The instruction may be detailed or concise and may or may not specify the location of the modification. Return the generated instruction the following format:
innstruction: {inttruction)
Table 18. Prompt designed to generate chat-style interactions between models and users.
You are programming assistant. The following content includes information related to your programming assistance, which may contain the record of the programming process, the current code, the user instruction, and your predicted modifications. Please provide the chat conversation for making the prediction. This may include analyzing the past programming process, speculating on the user's intent, and explaining the planning and ideas for modifying the code. Return your chat conversation in the following format:
# #chat:** {chat) ..
29

---

CursorCore: Assist Programming through Aligning Anything
Table 19. Prompt designed to evaluate whether the outputs align with user intent.
You are lasked with assisting programmer by maintaining record the programming process, including potential future changes. Your role discern which changes the pro- grammer desires you to propose proactively. These should align with their actual intentions and be helpful. To determine which changes align with programmer's intentions, consider the following principles:
Understand the C;ntext**; Assess the overall goal of the programming project. Ensure that any proposed change aligns with the project's objectives and the programmer's current focus.
**Maintain Clear C:mmunication** Before proposing changes, ensure that your sug- gestions clear and concise. This helps the programmer quickly understand the potential impact of each change.
* Prioritize Stability**: Avoid proposing changes that could introduce instability or significant complexity unless there is clear benefit. Stability is often more valued than optimization in the early stages of development.
Reeepettt the Programmer's Preferences Pay attention to the programmer's coding style and preferences. Propose changes that enhance their style rather than contradict it.
* Incremental Improvements** Suggest changes that offer incremental improvements rather than drastic overhauls, unless specifically requested. This approach is less disruptive and easier for the programmer to integrate.
**Consider Long-Term Maintenance**; Propose changes that improve code maintainability and readability. This includes refactoring for clarity, reducing redundancy, and enhancing documentation.
7. ** **Balacce Proactivity and Reactivity**: Be proactive in suggesting improvements that are likely to be universally beneficial (c.g., bug fixes, performance enhancements). However, be reactive, not proactive, in areas where the programmer's 's specific intentions are unclear where personal preference plays significant role.
For each potential change, return "Truc' if suggesting this change would beneficial to the programmer, return 'False' if the change does not align with the programmer's intentions if they do not want you predict this change. Give your decision after analyzing each change. Provide your response in the following format:
Analysis of change 1:**
Your analysis here.
Decision: ** "True' or 'False"
** Analysis of change ****
Your analysis here.
Decision: "True" or 'False'
30

---

CursorCore: Assist Programming through Aligning Anything
Table 20. Prompt used to evaluate base L.Mss.
Read the following messages during programming and return the modified code in this format:
< next_start | (modified code) next_end
messages_start Prrogramming process 1:
'python
b = 2
c= + ...
Current code:
'python
i=
b b=2 2 c a + ...
User instruction:
Please change variable names.< messages end
< next start >>
'python
i=1
j=2
k=i+j
< next_end
Read the following messages during programming and return the modified code in this format:
< next_start (modified code) < next_end
messaagss_start| > >Progrrmming process I:
Programming process }}
Programming process n:
(Programming process n}
Current code: [Current code)
User instruction:
{User instruction) messages_end
31

---

CursorCore: Assist Programming through Aligning Anything
Table 21. Prompt used evaluate instruct LLMs.
user
Read the following messages during programming and return the modified code in this format:
< next
>> (modified code next_end]]
Programming process
'python
a= b 2
c= c=a+ + b ...
Current code: "'yython
i= b= 2
c a
User instruction:
Please change variable names.
assistant
< next |tartstarrt |> >>>>>pp 'python
i=
j=2
k=i+j
> next_end
user
Read the following messages during programming and return the modified code in this format:
< start >> (modified code}< < next_end]]
Programming process
(Programming process }}
Programming process n:
(Programming process n}
Current code:
[Current code)
User instruction: {User instruction) }
assistant
32

---

CursorCore: Assist Programming through Aligning Anything
UsB::
modifications from codel through code2
**e final code:**
distl Levenshtein distance(codee code2)
dist2 Levenshtein distance (code2, code3)
def if F__.ontinuuu__nooffyccccoc ...
code2, code3);
dist3 Levenshtein distanneddteeeeeeeeeee (codel, code3)
Check if code3 is continuous modification of code1 and code2.
if dist3 == distl dist2:
return True
else: return False
Args:
codel (str): The code string.
code2 (str): The second code string.
Issucss:
code3 (str): The third code string.
The Levenshtein library is not imported.
Returns:
bool: True if code3 continuous
#еe 3: Importing the Necessary Library
modification codel code2, False otherwise.
A Ading the impott statement for the Levenshtein library.
Calculate the Levenshtein distance between codel and code2
"python import Levenshtein
distl = Levenshtein.co Levenshtein.dsstannccodddll code2)
Calculate Levenshtein distance
def
code2, code3):
code2 and code3
dist2 Levenshtein.disttaneceedd code3)
Checks code3 is continuation of modifications from codel through code2
Calculate the Levenshtein distance between codel and code3
una
dist3 Levenshtein. diaptteeeindssttneccodee,,, code3)
distl Levenshtein. disranncddttnncecccceeoee code2)
dist2 Levenshtein distance (coden code3)
# Check if code3 is continuous modification of and code2
dist3
Levenshtein. distance (codel, code3)
if dist3 = dist1 dist2:
dist3 distl dist2:
return True
return True
else:
else: return False
return False
*** Issues::**
assistant:
Could improve naming and documentation clarity.
A Aright, let's simulate the development process of this code, including typical errors and revisions. I'll start from the initial attempt and evolve the code step by step reach the final version you provided.
#ee Step 4: Remaming Function and Improving Documentation Final ad distteeents include renaming the function and enbancing the docstring for botter clartty.
#*# Step Initial temppt
python import Levenshtein
The initial attempt might be rough structure without importing necessary libraries and possible syntax emors.
def code3 is a continuous modification of codel and code2.
code2, code3):
"pytoon
def is_ contiinous__odifyyCcom. **.
code2, code3):
Checks if code3 is continuation of modifications fron codel through code2
Args:
codel (str): The first code string. code2 (str): The second code string.
**n
distl = "iventtein.distaneecddd doddance(codel, code2)
Levenstein. Levenstein.distaaceeecee. distance(code2, code3)
code3 (str): The third string.
dist3 Levenstein disdance(codel, code3)
Returns:
dist3 dist1 dist2
bool: True if code3 continuous
modification of codel and code2, False otherwise. una
return true
else: return False
distl Levenshtein destance code2)
dist2 Levenshtein. distance distance(code2, (code2, code3)
dist3 Levenshtein. distance (codel, code3)
**ssues::*
Misspelling of the function name Levenshtein' as Levenstein' ).
dist3 dist1 dist2:
return True
Syntax emrrr with missing colon at the of the statement.
else: return
Incorrect capitalization 'true".
### Step 2: Syntex Comections
A Addesssig the synlax erturs and some spelling issues.
This demonstrales typical progression in coding, from an initial rough draft with errors, through corrections and improvements, to final, refined code block.
"yy thon is coontinoous_oodifyycoo... **n Checks if code3 is continuation of
code2, code3):
Figure Few-shot prompts designed to leverage LLMs for simulating the behavior novice programmer.
33

---

CursorCore: Assist Programming through Aligning Anything
user: The final code:**
cos[position_ids] unsqueeze(1)
sin = sin[position_tds idd] unsqueezee(1
q.shape q = view(b, h, 5, d // 2, 2). transpose(4, reshape(b, h, s,
def poply_rotary-pos_eeme position_ide ids, unsqueeze_ _ii==)) cos cos[position. idd]].ussquezze(unsuuzze sin sin(position, ids] unssuueeeeuusssuuzzeeecee
cos, sin,
b, h, 5, d khape view(b, 3) reshapp h, d)
2). trranspose(4,
b, s, d shshape
= v.ve((b, h, d // 2, 2). transppose(4, 3) reshappee,, h, 5, d)
b, h, d s.shape view(b, h, 5, d // 2, 2). transspose 3) rbshapee
#*# Step 5: Define rotate_half and Compute Embeddings Next, they realize they need helper function rotate__aals and compute the final embeddings for q and 'k..
d)
q_embed [q cos) + (rotate_half(q) sin)
"python
k_embed (k # cos) + (rotate_half(k) sin) return q_enbed, kgembed
def @etate_half(x)>
return corch.cat([xx(.... 1::21, x[..., ::2]1, din=-1)
assistant:
def position_ids):
k, cos, sin,
Let's simulate coding process to reach the final version of the apply_ soopy__rotrr__osss _ function.
cos cos[position ids] unsqueeze(1) sin[position_idss iin.tiion_idl]]..suueeeeee
### Step Initial Function Setup
s, d q.shape view(b,
Initially, the programmer might start by defining the function and setting up the parameters they know they need work with.
d // 2). transpose(4,
3) reshape(b, h, s,
h, s, d k.spape k view(b, h, s, // 2, 2). transpose(4, reshape(b, 5,
"python def aoply_rotary_po_embbG k, cos, sin, position_ :
q_embed (q cos) (rotate_half(c) sin)
Initial setup pass
k_embed (k cos) (rotate_half(k) sin) return q_bed,, k_enbed
#* Step 2: A dd Basic Operations Next, they'll add the basic operations that essential for the function, such selecting the cosine and sine values based on position IDs.
#*# Step 6: Final A djustments for Flexibility Lastly, realizing the need for more flexibility, they introduce the unsqueeze dim parameter.
"python
"python def appty_rotaay_pos_embdCC k, cos, sin, position cos cos cos[position_ke ids] sin sinlposition_ie _dd]]
def htttly_rotary_pss_eee k, cos, sin, position_ids, unsqueeze_dinnt))
cas = cos[position_ics .unsuuuuseeeeeee (unsqueeze din) sin sin[position_ide
crsquuezzeuusqueezzee
b, h, s, d q.shape q view(b, h, 5, 2, transpose(4, 3) reshape(b, h,
#*# Step 3: Introduce Unsqueeze O Operation
Then, realizing they need to match the dimensions, they add the unsqueeze operation.
d k.shape
l k. view(b, h, d // 2, 2) transpose(4, reshape(b, h, 5, d)
"python
q_embed (q * cos) + (rotate (rotate_half(g) * sin) k_embed cos) (rotate_half(M) sin) return q_embed, k_enbed
def hoplyyrotary__os_oooo
cos, sin,
position_ids). cos cos[pasition_iies hddssssssuuezeeee...ee
sinlposition_ids unsquuezze)e)
This completes the journey the final version of the sopplyyootarrypos_eee function.
##4 Step Reshape and Process q and
A fter ensuring the dimensions of 'cos' and 'sin', the programmer starts processing q and kk. They need to
reshape these tensors to work with the embeddings correctly.
"python def hotyyrrotrr_poss.eee iisitiio_ids):
cos, sin,
Figure Few-shot prompts designed leverage LLMs for simulating the behavior of an ordinary programmer.
34

---

CursorCore: Assist Programming through Aligning Anything
User: "hhe code:*
std: std::string result;
size_t pos 0;;
size_t new_pos;
while ((neu_pos siinncsee,rhh, pos)) 1=
static void replace
replace_allsttd: all(std:::tting
s, const
:td::::tinn::::pos) {
std::string & search, const std::string & replace)
result += s.substr(pos, new_pos pos);
std: :string result;
result replace;
for (size_t pos
i pos
search. search.Length() Length())
pas new_pos search. Length:)
auto
= s. fenn(s, pos);
if (new_pos std: :strin::: :npos)
result s.substr(pos);
result += substr(pos, s.size() pos);
mtv::nooe(essult))
break;
result + s.substr(pos, new_pos pos)
replace;
### Slep Refine and Transition Static Function
pos new_pos;
}
ensure encapsulation and promote reusability, decide to make function static'.
5 std: mov:::ooversss
}
cpp static void replace_atl applace_al(stdd: :string & s, const std: :td:iitiing & search, const std::string & replace) std::string result;
assistant:
Let's simulate the development process of the seplace_all function in C++ step by step, showing how programmer might iteratively refine and improve the code.
size_t pos 0;
size_t new_pos;
while ((nem_pos Finndre,,,,,,, pos))
std: :ett:isirnn::ppp:
#c9 Step 1:
Basic Function Structure
result *= substr(poss, new_pos pos);
Initially, might start with simplest possible structure to replace occurrences a substring with another substring given string.
replace;
new_pos + search. Length))
result s.substr(pos); s std: :ode:rrie())))
cpp
void replace all(aal::::sstttaa :string s, const std::string & search, const std: istring & replace) {
// Initial approach find and replace
size_t pos sfinn(see)cch);
### Step Final Refinements
while (pos != std:: :npss r.epplaceppss nearch.lengthe. replace); f.nnddc,,ar,h,
Finally, realize the for loop structure be concise and less error prome, integrating the position updates right into the loop structure.
Aoplaace.Lngthh@O
}
cpp static void replace_ siltacc_all(std:::ss :string & s, const
std: std::string :sttiig & search, const std::string & replace)
#80 Step 2: Consider Performance and Correctness
std::string result;
realzze the initial approch be inefficient due to modifying the string in place. Also, it doesn't handle overlapping substrings well. Let's use buffer build the result.
for (size_t = 0;
+= search. (engch))))))())
auto new_pos s. find(saaarch, pas);
(nem_pos = std: sttring :npos)
result += s.substr(pos, s.size() pos);
break;
"cppp
void astiinn:::trinngg & s, const std::string & search, const std: istring & replace) {
result += subbstrrpos, new_pos pos) +
replace;
std:istring result;
new_pos;
size_t pos 0;
}
size_t new_pos;
std: move((uul))))
while ([new_pos (inarre, pos)) !=
std::string:::pos)
result s.substr(pos, new_pos pos);
result += replace;
pos new_pos search. Length();
This final version improves readability and maintains the optimized performance by constructing the result string in a single pass.
result += s.substr(pos);
5 result;
#$d Step 3: Optimize with ''sd:::moo
notice that the final assignment to 's' can benefit from std: nove' to avoid unnecessary copying.
void seplace_all(stie :sttrng & s, const std::string search, const std: :str:igtingg replace) {
Figure 21, Few-shot prompts designed leverage LLMs simulating behavior of an expert programmer.
35

---

CursorCore: Assist Programming through Aligning Anything
USEr::
@0verride
"ocordi:*
public void print.Json(finat StringBuffer sb, final
Revised code changes: diff
VoidResult o)
"D.append("(""))
+1,5
+/+*
magical invsqrt function Quake code
eDverride
see: http///ww.codeasstrrr// wwt.c//aaestto..oo///vevtaaa.aee con/reviens/9
public VoidResult Fron)son(Final Object
return VoidResult. INSTANCE;
+}
Current code:
/+*
Current code:
nagical invsqrt function Quake III
"jara
3 see: http://www..ooomesttee watoeeooooveiiieeeeeeeeeee con/reviews/9
*/
2
5
package con.google con.gogge..utjjonnpp...
public class WoidResult_.sonSeriaallee extends Js 3sonSerialierr/ooiReeee public static final Jaadfeesutt_usonserillleee INSTANCE
*hhange 1: diff
-3,3 +3,19 88
see: /
7
new Void@keautt.bbnserraliie)))))))
private VoldBesuut..sonsseiilleeerr
#Foat InvSqrt(float x)
11
Float xhalf = 9.5f+x;
80verride
int <(int+)6x;
13
public void print.Jsonffinal StringBuffer sb, final
Bx5f3759df (i>>1); ((float*)ā;; 6f1..5F-taa.fxet
VoidResult
sb ab.append("()))
15
return
17
80verride
public VoidResult fronJson(final Object o)
main(void)
19
return VoidResult. INSTANCE;
int result InvSgrt(O. (8856))
printf("Result; ad (should be 42)", result); return result 1= 42;
21
**change 1:*4
"dffff@@ -1.3 +1, @
assistant:
+// Copyright 2088 Google
+//
** Analysis of change 1:*+
+// Licensed
Apache License,
"License");
The change introduces the "InSSqrt function implementation and a basic tast within the main function. This change agpropriate and beneficial. The primary gool is to implement and test the InvSqrt" function, well- known gorithm from Quake III code. dding the function and basic
this file except compliance the License.
+// you nay
+// You obtain copy the License
+//
aligns directly with goal. The implementation clear and concise, following common C coding practices. The function's logic straightforward and well- documented, making easy to understand, Overall, proposod change is logical next step in developing the "IvSSrrt' function and ensuring works correctly.
+// http://wmw.. www.aaaacccce..ieee. wttp.r////eeeesss///ee//----
+//
+// Unless required by applicable lau or agreed to in weiting, software
/// distributed under the License is distributed on
"AS
+// MITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express inplied.
** Decision:** "True"
+// See the License for the specific language governing permissions and +// Linitations under the License.
USer::
**record:**
package won.googlle.gee wwwlllegggssssosc.cooom client;
Revised code changes: diff
+1,21 88
+package gpogle.gutjsonrpe.
assistant:
**Analysis change 1
*public class
extends
Witboul enough information to coafirm the use of Ube pache Liconse, predicting this change may not align with the programme's Intentions project's actual licensing requirements.
public static final Voiikesutt.btonSeriiliieer
INSTANCE
VoldBesult..soonSriillieee
private Voidfesuit..bansseriiliizer
Decision 'False'
Figure 22. Few-shot prompts designed to evaluate whether the outputs align with user intent.
36

---

CursorCore: Assist Programming through Aligning Anything
USer:
*iinstuccioon:**
Revised code changes:
Update the create_cmm _cnn_modet_ function to ensure that the RRLL'" activation function is added immodiately after the BatchNorn layer batch_norm' is enabled. A Adjust the function signature to remove default for the batch_norn parameter. The updated code should handle the addition the 'ReLU' layer conditionalyy based on the batch__eee norn parameter.
diff
-3,6 +3,10 ee
def create_ _______c_______ comn_node(([A)) (in_channels, config):
Layers []
conv2d = nn. www.coodd(inccaaneee
config,
kernel_size=3, padding=1)
layers += [conv2d, nn. n..ReeU(inplacceTrue)]
User::
1f norch_norm:
Layers += [canv2d, nn. (...BtcchMorn2e(comnig)))
Current code:
else:
"ubyy
Layers [conv2d]
frozen_string. true
layers += [nn. ReLUU[inlacc=-True]
nodule
nodel nn. Sequential
return nodel
N;
3 nodule hwtp://ww....eeeeoooooooooeeteee
Revised code changes: diff
Ad:ater; end nodule
:xtensions: http://wwww....oooooooooaeeeett.
68 -1,6 +1,6 80 import torch.nn as
W: ::a.Aad::: Deeere::DDDeereddcttiveeeeooo
extend activeSupport::
6
included de
-def create_ cnt__ccnn____ modelein_cchaeeneenennnn config): +def wttpecnooooeenenneeeee__
include Workflow: Actieee:
config,
8
include InstanceMethods
batch_norm=False):
end
layers =
conv2d = nn. www.coodiinncaannniin config, kernel_size=3, padding=1) if batch_norm:
11 nodule InstanceMMethods
def perrsist.worvfloousstttedd
13
arite_aatrributtcsel..co class www.ass.wwwloww.ooollooooooooo
new_value)
14 true
Current code:
end
16
end
1 inport torch.nn an as nn
17 end
3 def canateeccn__oeellinnccoo :ooh--oe==FFas)::
config,
4
Layers
Git commit message after all changes: Include WorkfooAA veroord in the state persistence extension.
5
conv2d nn. wwt.Cooodiinnchmmmne config,
kernel_sizee L_ize=3, padding=1)
6
if batch_norm:
layers [conv2d, (..Batchhorn22((onn1g]]]]
Changes in predictions: diff
8
else:
9
Layers += [convzd]
00 -1 +1, 60 # Frozen_string ww__string_iiterlll
Layers += Tmm.ReLU((inplaceeTrvee
literal: true
nodel am..Squuntiaa(AAayyrr
+require workkloowacctvvereoom
return nodel
nodule end nodule hwtp://wc.....o.oooooooooooettte
Changes in predictions: diff
Extensions:
-1,12 +1,11 import torch.nn as nn
w: A::Aperer;; end nodule
Extensions: D wtt.aa/re...eeeeeeeeoeottttteeee
def create_cmm cnn. con__cnneeeeeeee serch_nor=FFase): +def greate_ccn_noder (in_channecc,,,,,,,.aaaaaa config, batch_norn):
config,
:dapter:: : aeaaee::::BBeereeddttveeeerrr extend AetiveSupport:: included de
include Workflow: :Attteettttttiieoeeeooo include
Layers = []
conv2d = nn. wn..oonvd(in_chhnelss, confiig kereel_ padding=1)
Morkfloowctiverecorr :Aaaperr: Activeeeerstti
include InstanceMethods
1f batch_norm:
Layers [conv2d, Batchloorn2&((oomrig))
layers += [conv2d, mm.Batcchlorr22((coofig)) nn. R
else:
layers += [conv2d]
assistant:
Layers (n.RReL((nplaac=(rue))
** "sstrructoon:
layers +s [conv2d, nn. (n.ReUU(inllace-Tree))
At the beginning of the file, add the statement 'require
nodel www.seuueniia(=taayrss
sonnffooo_activerccordd On line change iinelude
return nodel
Workflow: Adapter: :Attvveeecord
'include
Workflowed www.ctivereccrrd.....ooooooo..d. :Adapter: :ActiveRecord; Ensure the final code reflects the necessary changes for including
assistant:
in the state persistence extension.
Figure 23, Few-shot prompts designed generate user instructions
37

---

CursorCore: Assist Programming through Aligning Anything
retum solutions;
LIST::
Revised code changes: " diff
vectonstrrngg geerrttllaaeeloeettrrrtt Squeens,
88 -14.3 +14,20
vectarsstrigg-C:
vectorsiiring> board gencoaassunn(cqees,
for (int i
solutions, push, backkaar))))
strinn(n, '."); ron[queens (i]] board, back_oacs(ro))
for (int (coluans.Finn(()
(coluens. FindCi) columns. continue;
return board:
int diagonall row 16 Allagnalst.&fnoodingpoe
+]:
diagonalst, and ())
continue;
Cureal code:
digonalss2 end O)
if (diagonals2, (ind(aaggmm)))) 1=
ist diagonat2 row
public:
class Salution [
ap
vectarveetso-strddng so solveniuueons(nnt n)
continue;
auto solutions vectesvectsr-eertnnn-0)) nito
qucens[ron]
salvessslutions,
vectorcintx<n, -13;
diagonalat, inw.rtt (dtppa(dtappaat);;
coluens, Insert(t);
solutions;
diagonalss (dspai())
diagonalsi, diagonals2);
queens(ron]
bacitrrackcsstttiion queens,
n)
10
11
vectorssteingg genesatebann(yecttrrittt Squaens, int
auto board vectarestring--O)
diagonalsi. ensse(diaaomaal)) diagonals2, anase((iaggnat2));
columns, crase(t);
12
for (int = 8: 1 n; i++)
13
string zow s-striingo, '.?); roul Quueess [i]] = '0';
14
15
board poshd. back(row);
16
}
17
return board;
vectorsstringe genenatebaardCveettriintt Aqueens, int n)
39
18
1:
Resiand code changes:
Prodictdd modification:
"
ddift
80 -3,41
vesteeaector-strings solveMRucens(int n)
00 +5, 23 vectorsint>(n, -1];
auto solutions vestorasstor-strinneee
colveesolutions, queens, return solutions;
colums unordered set<int=();; diagenals] uncrdered, set<<int>)); diagonals2 uncrdered. set-tint>)):
mncktraci(solutionn queens, 6, columns, diagonalsi, diagonals2]; retur solutions;
vold solu@vwetarvvettrcttiigg Asolutions, vectorsint> squeens, int n, int row, int coluns, int diagonals], int diagonals?)
1f [ron -- n)
auto board geverateloan((uuers, n);
void cenettaeeceettrveeerettrre Ssolutions, vectorkint> Squeens, int n, int row, undrdered setxints Rcolumns, unnnerred_settinn adiagenals), unarderee_seetcint Sdiagonalsz) {
slse
int availableoosstioes ((1 << n)
(-Ccoluens diagonalsi dlagonals2));
shile (wuxilablaaPsstt inns
3f Cron n)
vectoristring board = generateliarr(queenn, n);
wodtlaleePesttions)
position avaitabisfocitions
solutions. pash_aacckChaarrr)
CavailablePositions availableeostttees waiilabeeoosttions
8; for (columns.FinddC)) end() continue;
1++)
___uutidn.cttoosstiion)
queens(ron) colurn:
solve(solutions,
columns
position, (diagenals) position) (diagonats2
int diagonali (Anaomaat2.
position) 1);
diagonalst. en)
queens[roe] -1;
continue;
Int diagonal2 now
if
1=
vectorestrings georattdearr(tuettriiii Squeens, int n]
diagonals2.end1 {
1.
continue;
qroens[ron] = 1:
coluens. insart(k):
assistant:
diagonalst. (isert(daaappall)
diagonal x2, innert (d
**cbat
backtrackCsolution, queens, 8, row
The revised code transtons from using sets bibwise operations for the N-Queets prokken, timing optimize the alpoti by reducing dynamic menory allocalions improving spedd Inlially, ocigliaal ['unardered_setee setiirndddetttt>
coluans, diagenalsi, diagonals2):
queens[ion]] -1;
diliied
erase(t);
brckk columns and diagonals The likely seoks to eahance efficiency, prompting the replacemeti of these sets with integer bit masks. To achiove e this, the solve funcion is adjusted to use bibwise operations for conflict chicks, calculading avai Labl PPsiitiion using Nitwrio ND NOT opentions identfly salid queen ploomments signeficant
diagonal cnsse(Ciaagnlll)) diagonals2. ensse(Ciingnnal1)
to camopondinng to valid column, and the recussive 'solvve call updales kit masks for col amns and diagonals aftor placing queon. This maintaine the booktraking lagic while levernging bibwise operttins for performane. The generateBoard function mmains unchanged, simply formats the reslt. This approach effcciett szeemlined
builtineetz' function
vectorestrings gearateelssrnCeeetoriins Aqueens, int
Ravised oode changes: diff
Queens problem.
90 33,3 +3,17 88 vector auto soluti solutions vectoroectorsstrinne-))) auto queens vectorcint><e, -1);
solveeQucens(itt n)
snlveessnlations, queens, 6, 8, 0, a, 0);
Figure 24. Few-shot prompts designed generate chat-style interactions between models and users.
38


Reviews:
[{"review_id": "CnPbveG9pW", "reviewer": "Reviewer_g7Ss", "summary": "The paper introduces a new family of models called CursorCore, which enables handling of historical context while making code generation or assistant response predictions. The authors also propose Programming-Instruct which is a framework designed to collect data to train CursorCore with the historical code edit context. The authors evaluate that models on APEval, which is a modified version of the popular HumanEval benchmark to incorporate historical context, and show improvements over base models.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. Please see our response below:\n\n> Baseline of prompt-engineering to incorporate H is missing\n\nThe reviewer may have misunderstood our experimental setup. Our prompt engineering baseline does include H, as shown in Tables 19 and 20.\n\n> No empirical justification for incorporating H. Experiments show that (H, C) which includes historical context is poorer in performance compared to (C) which does not & Q1\n\nA direct comparison between the C and H+C subsets is impossible, as they involve different problem sets. Detailed APEval annotations are in Appendix C. The H+C subset is more challenging due to ambiguous or irrelevant history, making it harder for models to leverage and resulting in lower performance.\n\nWhile, as reviewer mFUu suggested, we can remove H from the H-included subset for direct comparison; please see our response to mFUu.\n\nHistorical information is essential in many cases. For example, during variable renaming, a model without access to edit history may mistakenly revert the name. Similarly, if a user writes and later deletes a draft, only the edit history reveals its intention.\n\n> No sufficient work to identify these cases\n\nFor the training data, we do not need to explicitly label which cases require historical information. In practice, the collected historical data naturally includes both helpful and unhelpful contexts. We must train LLMs to make predictions based on such inputs containing noise.\n\nFor the benchmark construction, we identify cases where historical edits are necessary to infer the intended changes, shown in Appendix C.\n\n> APEval does not thoroughly capture the relevance of historical user edits & It would not benefits from historical context\n\nAs noted above, we already considered this during annotation. We suspect the reviewer may have misunderstood our approach: we did not simply add historical edits to the original code. Since HumanEval docstrings include rich context like functionality and test cases, doing so would offer little benefit. Instead, we removed all docstrings and only showed annotators the signature and its purpose, making it significantly harder. In this setting, historical context becomes more helpful.\n\n> Collect user edits from GitHub and OJs is not reflective of data authors are seeking\n\nThe reviewer is concerned that the collected historical edits may already be “complete,” with no need for further edits. Our data collection pipeline has taken it into consideration. We adopt the following perspectives and methods:\n\n1. For data from OJs, we retain only the user’s first correct submission and submissions preceding it.\n\n2. As shown in Section 4.2, we use LLMs to judge whether code changes align with user intent based on historical and current context. If the current version is complete and changes are version updates (e.g., adding auther information), the LLMs are designed to filter out such updates, while meaningful edits are retained.\n\n> HumanEval-style of evaluations suffers from contamination & does not reflect real-world use cases & Q2\n\nThe base models’ technical reports confirm HumanEval contamination was addressed. We also cleaned our training data (Section 5.2). Besides, our benchmark inputs were significantly modified which further reduce contamination risk.\n\nWhile HumanEval-style benchmarks differ from real-world scenarios in data length (function/file vs. full repositories) and use of third-party libraries/tools, we still chose to extend this benchmark because:\n\n1. At the time of our experiments, no public benchmarks included all contextual elements like historical edits and user instructions. Thus, we chose to extend existing benchmarks as the most efficient way to assess models on basic tasks in this paradigm; directly creating a more complex benchmark might obscure performance differences.\n\n2. Repository-level benchmarks (e.g., SWE-Bench) target different goals than ours (see historical context for agent).\n\nFurthermore, a newly released dataset Zeta is constructed based on real-world engineering cases with historical edits; please see our response to mFUu.\n\nExamples in Figure 3 are from a real-world source; we chose the shortest one to aid clarity and fit space constraints.\n\n> Should also include human annotation rubric/results in Section 3.1\n\nThanks, we agree it is important. Due to space limits, it's in the appendix for now, but we will add the detailed discussion to Section 3.1 in the camera-ready version.\n\n> Historical context for agent & Q3\n\nIncluding such context could help the agent better understand user intent and adapt to coding style! While, our work focuses on single-call LLM usage with strict latency requirements, unlike agentic workflows, which involve multi-turn interactions and prioritize end-to-end performance. Integrating context into such workflows is a great direction for future work. \n\n> Typo\n\nWe have fixed it. Thanks for the careful review.\n\nWe appreciate your review and look forward to your response!", "claims_and_evidence": "There are many claims made in Section 2.2 such as:\n\n> Although they can utilize C, they fail to capture H, limiting the modeling of future changes in\nC, and are incapable of deleting or editing code. Although user instructions and reflection information can be used through comments and assert statements, this capability is weak and unstable.\n> \n\n> Prompt engineering can integrate some of this information into existing models, but the impact is limited.\n> \n\nHowever, these are not adequately supported in the evaluations conducted in Section 6. Specifically, the baseline of prompt-engineering to incorporate historical context is missing from the evaluations. Without this role of the target data curation and model training is unclear.\n\nFurther, the work focuses primarily on incorporating historical edit context into an LLM’s input. However, no empirical justification for this provided, and it is only assumed that this needed. In fact, experiments in Section 6 show that (H, C) which includes historical context is consistently poorer in performance compared to (C) which does not. Generally speaking, I agree that historical context would be needed under certain scenarios but it seems that not sufficient work was done to identify these cases; I don’t think that HumanEval-like evals would benefit from historical user context.\n\nFinally, it is known that HumanEval suffers from contamination due to overlap with public sources [1].\n\n[1] Matton, Alexandre, et al. \"On leakage of code generation evaluation datasets.\" *arXiv preprint arXiv:2407.07565* (2024).", "methods_and_evaluation_criteria": "1. I think APEval does not thoroughly capture the relevance of historical user edits for code generation tasks. So while the fine-tuned models are better than baseline models, it is not clear if the proposed methods are really relevant for real-world use cases. I have elaborated this concern above.\n2. The method used to collect user edits from GitHub and online judges is also not reflective of the kind of data that the authors are seeking. Specifically, the authors are looking to collect incomplete code snippets (as in Figure 1) but the sources for training data collection do not contain incomplete code snippets but rather earlier versions of complete code snippets.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Please see my other comments.", "relation_to_broader_scientific_literature": "The paper attempts to present extend the traditional code completion task [1, 2, 3] by introducing historical user edit context. I think this is an interesting direction, though the treatment of this problem in this work is not very thorough as I have discussed above.\n\n[1] Chen, Mark, et al. \"Evaluating large language models trained on code.\" *arXiv preprint arXiv:2107.03374* (2021).\n\n[2] Athiwaratkun, Ben, et al. \"Multi-lingual evaluation of code generation models.\" *arXiv preprint arXiv:2210.14868* (2022).\n\n[3] Guo, Daya, et al. \"DeepSeek-Coder: When the Large Language Model Meets Programming--The Rise of Code Intelligence.\" *arXiv preprint arXiv:2401.14196* (2024).", "essential_references_not_discussed": "The inclusion of historical user context will also benefit agents for code [1, 2]. I think that the work will greatly increase in value by also discussing their contributions in this context.\n\n[1] Jimenez, Carlos E., et al. \"Swe-bench: Can language models resolve real-world github issues?.\" *arXiv preprint arXiv:2310.06770* (2023).\n\n[2] Wang, Xingyao, et al. \"Openhands: An open platform for ai software developers as generalist agents.\" *The Thirteenth International Conference on Learning Representations*. 2024.", "other_strengths_and_weaknesses": "1. The paper is mainly well-written and easy to follow.\n2. The authors should consider moving away from HumanEval-style of evaluations as this does not reflect real-world use cases. In fact, I’d like to know if the code examples in Figure 3 are actually from real-world sources; I’d be surprised if they are.", "other_comments_or_suggestions": "Line 21 - collect → collects; evaluate → evaluates", "questions_for_authors": "1. Under which scenarios do you think it is absolutely necessary to include historical user context? Have you considered experiments to demonstrate the value of historical context in the input to LLLMs?\n2. Can you comment on the distribution gap between real-world scenarios and HumanEval-style of benchmarks? These days most code generation evaluations follow either software-engineering (SWE-Bench), repository-level coding (CrossCodeEval) or very hard programming problems (LiveCodeBench). As such HumanEval is an outdated benchmark, and you should consider moving away from this, especially given the focus on historical user context.\n3. Do you think it would be useful to conduct a study on utilizing historical user context for agentic workflows?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JTxW0a1jId", "reviewer": "Reviewer_mFUu", "summary": "Current code LLMs typically use only the code context and, optionally, user instruction as input, without considering the code’s development history. In this paper, the authors propose training a model to integrate various types of the information - particularly editing history - along with the current context and user instruction to predict future code edits. For evaluation, they introduce a new benchmark, APEval, to evaluate code capabilities using different combinations of information. For training data, they generate 219K samples via Programming-Instruct, which uses LLMs to automatically synthesize training data from GitHub commits and online judge platforms. Using the collected data, they fine-tune multiple state-of-the-art LLMs and develop the CursorCore model series, which outperforms models of comparable size.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. Please see our detailed response below:\n\n> Claims and Evidence & Q1\n\nThanks for the suggestion. We can provide the results of removing H from the samples in APEval that contain H, and compare them with the results obtained without removal. This ensures a fair comparison under the same conditions. The evaluation results are as follows:\n\n|w/o H|w H|\n|-|-|\n|35.4 (29.9)|39.0 (32.9)|\n\nThe settings are consistent with those used in Appendix I. Additionally, the results presented in Appendix I further support it, as they provide evaluation results under different window lengths of H.\n\n> “let the LLMs judge whether each segment of M aligns with the user's purpose through principle-driven approaches” is ambiguous\n\nThanks for the detailed review. We did prompt the LLMs to first analyze the user's intent before making a judgment, as shown in Table 18. We have corrected this in the revised version.\n\n> It would be better if the benchmark included more complex tasks\n\nIncluding more benchmarks is certainly beneficial! While, the prompts in the DS1000 and ClassEval lack historical context. Recently, a new dataset called Zeta has become available on HuggingFace (released just last month, so it was not possible to include it before the ICML deadline). Zeta is constructed from real engineering use cases, and better reflects the distribution of tasks in real-world scenarios. It include H (though not U). We have now conducted evaluations on this benchmark, and some results are as follows:\n\n|Model|Metric|\n|-|-|\n|DS-Coder-1.3B-Base|18.2|\n|DS-Coder-1.3B-Inst|42.4|\n|CursorCore-DS-1.3B|45.5|\n|Qwen2.5-Coder-7B|51.5|\n|Qwen2.5-Coder-7B-Inst|54.5|\n|CursorCore-QW2.5-7B|60.6|\n\nThe reported metric is the average accuracy over all evaluation samples. We use GPT-4o to assess the correctness of the generated results based on the assertion texts.\n\nOf course, we can also evaluate DS1000 and ClassEval, to assess the model's ability to leverage U and C. We choose to evaluate the performance of CursorCore on these benchmarks using the Inline and Tab modes, as they most closely resemble the original formats of them. Some results are as follows:\n\n||DS1000|ClassEval|\n|-|-|-|\n|DS-Coder-1.3B-Base|16.2|13.0|\n|DS-Coder-1.3B-Inst|20.7|13.0|\n|CursorCore-DS-1.3B|21.2|17.0|\n\nThe reported metrics are averaged over all samples, with class-level generation evaluated using ClassEval. All generations are performed under the greedy decoding setting.\n\nAppendix J also includes another open-domain code editing benchmark (CanItEdit). These results collectively demonstrate the strong effectiveness of CursorCore.\n\n> Performance of Qwen2.5 and Concerns about bias in APEval (Q2)\n\nWe have included a discussion of this issue in lines 1130 to 1135. Moreover, while instruction-tuned models are effective at aligning with prompts for program synthesis and code repair, they struggle to align with various forms of contextual information, such as historical context, which is commonly encountered in programming workflows. Therefore, the strong performance of Qwen2.5-Coder-7B-Inst on the synthesis and repair tasks, coupled with its  weaker performance on APEval, is expected.\n\nRegarding your concern about bias in APEval, we have already taken this into consideration during the annotation process. As described in Appendix C, we ensure that the current code is sufficient to indicate the user's next intent.\n\n> Explanation of user instruction (Q3)\n\nFor \"user instruction,\" we refer to general prompts or feedback related to the code. Alternatively, from another perspective, if we consider the historical context as representing changes in the internal state, then the user instruction can be viewed as an external signal. The term is defined broadly—any type of prompt or feedback that may assist programming can be regarded as a user instruction, such as \"write a quicksort algorithm,\" \"translate this code into Java,\" or feedback messages like \"Traceback...\". In normal conversation templates of instruct models, this is commonly labeled as \"user\" or \"instruction.\" To maintain compatibility and in consideration of a previous reviewer's suggestion, we choose to use it.\n\nFor HumanEval, we employed different modes, as described in lines 1093 to 1128. In the Tab mode, we leave the docstring as part of the code context; in contrast, we transform the docstring into a user instruction. This setting helps align the evaluation inputs more closely with those encountered in real-world applications.\n\n> Typo\n\nWe have fixed them. Thanks for your careful review.\n\nWe appreciate your review and look forward to your response!", "claims_and_evidence": "My main concern is that the paper’s central claim - that integrating editing history into LLM training data helps models to learn more effectively from extensive editing data and ultimately become better programming assistants - was not clearly demonstrated in the evaluation. While Table-4 shows that CursorCore models outperform their unadapted counterparts in H+C and H+C+U settings, this improvement could be attributed to the unadapted models’ unfamiliarity with the format of history (H).\n\nIf I understand correctly, a direct comparison between C+U and H+C+U is not possible because they involve different subset of problems. Therefore, to show the benefit of incorporating edit history in both training and testing, it is necessary to compare the CursorCore model’s performance with and without H on the same set of problems. For example, if we construct H for the C+U problems and use the new H+C+U for prediction, would this improve CursorCore’s performance over C+U?", "methods_and_evaluation_criteria": "Overall, the proposed methods and evaluation criteria make sense to me, except for the following issues:\n\n**Unclear data processing**\n\n(page 5) The statement, “let the LLMs judge whether each segment of M aligns with the user's purpose through principle-driven approaches”, is ambiguous. How can the LLM determine whether a code change aligns with user intent without first inferring that intent? Does this mean the LLM only filters out simple and obvious cases, such as private information, based on predefined principles? The prompt in Table 18 suggests that the LLM is instructed to first assess user intent. If that is the case, the statement should be revised for clarity.\n\n\n**Limited scope of benchmark**\n\nThe APEval benchmark consists mostly of simple function-level problems, as they are extended from HumanEval and contain on average 139/31/19 lines of H/C/U. Since part of the training data come from GitHub commits, I would expect that the fine-tuned model to perform well beyond function-level code generation. It would be better if the benchmark included more complex tasks, such as those from DS1000 (data science tasks) or ClassEval [a] (class-level program synthesis).\n\n[a] ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation. (ICSE 2024)\n\nupdate after rebuttal: The authors have conducted further experiments on Zeta, which is a non-contaminated, more realistic dataset.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Overall, the experimental design and analysis are sound and valid.", "relation_to_broader_scientific_literature": "Key contribution:\n\n- Integrating code commit and edit history into code LLM training using a carefully-designed format, making a novel contribution. Prior work primarily focuses on using a snapshot of current code and user instruction as input, while the best way to represent and utilize edit history remains understudied.\n- A training dataset with edit history input collected from an automated data synthesis pipeline. The large dataset (219K) requires significant computation resources to construct and can benefit future research.\n- Extensive evaluation across models and edit representation formats. It conducts a comprehensive evaluation across a wide range of models (including close-source and open-source, various sizes), along with extensive ablation studies to support its technical choices (such as different representations of code changes, integrating reasoning traces, and data selection ablation).", "essential_references_not_discussed": "The paper discusses essential related works.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "* Please ensure the correct capitalization of tool names:\n  * Line 326 and later paragraphs: “stack” should be “Stack”, “oss-instruct” should be “OSS-Instruct”, “editpackft” should be “EditPackFT”, “evol-instruct” should be “Magicoder-Evol-Instruct” (as per the citation), “sglang” should be “SGLang”, and similar adjustments for other tool names.", "questions_for_authors": "1. Can you provide further empirical evidence demonstrating that adding history indeed improves the model’s ability to assist with programming? (see Claims And Evidence)\n\n\n2. From Appendix J, it appears the CursorCore models trained from Qwen2.5-7B perform worse than Qwen2.5-Coder-7B-Inst on EvalPlus and HumanEvalFix. This contradicts CursorCore’s strong overall performance on APEval, which also extends HumanEval. This is surprising to me - could the authors provide further discussion? It also makes me wonder if APEval has some inherent bias. For instance, the C-only setting might be problematic, as the current code alone may not sufficiently indicate the user’s next intention.\n\n\n3. Could you please clarify what is meant by “user instruction”? For example, in the context of HumanEval completion, do you transform the docstring into user instruction, or do you leave the docstring as the code context? Also, on page 2, it mentions “User instructions ... generated as feedback based on interactions with external environments (such as a code interpreter)”. Can you clarify what type of feedback can be considered as user instructions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "quBAcFwkqH", "reviewer": "Reviewer_qAD7", "summary": "The core problem this paper addresses is that existing coding benchmarks are incongruent with human development processes. The paper argues that an effective coding assistant should be able to use various types of information available to humans to make edits, rather than simply respond to constrained prompts. \n\nTo this end, the paper proposes a new benchmark, APEval that provides. a testbed for AI assistants that can utilize a richer context stream and propose / predict modifications to the code. The benchmark is created by human annotation, and based on HumanEval. The paper proposes a pipeline called Programming-Instruct that uses an LLM parameterized by a persona to simulate a human coder. SLMs trained with this pipeline outperform other SLMs and approach the performance of frontier LLMs (GPT-4o).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. We sincerely appreciate your recognition of our work.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "I checked the details of the experiments in Section 6.", "relation_to_broader_scientific_literature": "The primary novelty of the paper is the task formulation and the dataset. To my knowledge, there are no datasets that provide the edit stream amid a more naturalistic evaluation of coding assistants. In comparison, datasets like HumanEval (which APEval is based on) and SWE-Bench focus on a more restricted setting in which a single prepared instruction is provided and the final product is evaluated.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The paper provides a resource of significant utility to the community (a testbed for more naturalistic coding assistants), and the evaluation of the proposed pipeline is reasonable.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents CursorCore, a new series of models that are fine-tuned on Programming-Instruct data points that include different code context including code editing history, current code context and the user instruction. The paper also proposes a new benchmark APEval for assessing model capability to incorporate different types of contexts. All reviewers found the overall idea of incorporating history and the framework to collect such edit history context interesting and useful. But there were some concerns around the generalizability of the current APEval benchmark to more real-world coding contexts, ablations around necessity of history, limited datasets. The new results in the response on the Zeta dataset as well as the additional results on DS1000 and ClassEval helped with the generalizability of the results beyond just APEval. It would be great to incorporate the additional experiments, rebuttal clarifications and suggestions from the reviews in the final version of the paper.
Decision: Accept (poster)