RePair: Automated Program Repair with Process-based Feedback (2024)

Abstract: The gap between the trepidation of program reliability and the expense of repairs underscores the indispensability of Automated Program Repair (APR). APR is instrumental in transforming vulnerable programs into more robust ones, bolstering program reliability while simultaneously diminishing the financial burden of manual repairs. Commercial-scale language models (LM) have taken APR to unprecedented levels. However, the emergence reveals that for models fewer than 100B parameters, making single-step modifications may be difficult to achieve the desired effect. Moreover, humans interact with the LM through explicit prompts, which hinders the LM from receiving feedback from compiler and test cases to automatically optimize its repair policies. In this literature, we explore how small-scale LM (less than 20B) achieve excellent performance through process supervision and feedback. We start by constructing a dataset named CodeNet4Repair, replete with multiple repair records, which supervises the fine-tuning of a foundational model. Building upon the encouraging outcomes of reinforcement learning, we develop a reward model that serves as a critic, providing feedback for the fine-tuned LM’s action, progressively optimizing its policy. During inference, we require the LM to generate solutions iteratively until the repair effect no longer improves or hits the maximum step limit. The results show that process-based not only outperforms larger outcome-based generation methods, but also nearly matches the performance of closedsource commercial large-scale LMs. 1

heading: 1 Introduction
text: Since the birth of the program, its reliability has been a primary concern. The capacity of large language models to auto-generate code has further
*Corresponding author 1Code and data are publicly available at https://github.
com/TnTWoW/RePair
intensified these concerns (Khoury et al., 2023), sparking discussions on possible solutions. Program repair accepts vulnerable programs as input, enhancing them through locating, correcting, and testing - a process that is often challenging within software development and programming competition. The confrontation arises when these defective programs encounter errors only when they receive rare and unexpected inputs. Moreover, it requires experienced programmers to invest substantial time in identifying and rectifying errors. A sophisticated Automated Program Repair (APR) system can reduce the expertise threshold and time commitment required, significantly enhancing the robustness of the program.
Prior works viewed APR as a sequence-tosequence task, then LLMs, with their massive model parameters and training data, can address this issue through zero- or few-shot learning (Jiang et al., 2021; OpenAI, 2023; Ouyang et al., 2022; Chen et al., 2021; Zeng et al., 2022; Touvron et al., 2023; Hoffmann et al., 2022; Anil et al., 2023; Chen et al., 2022; Le et al., 2022; Rozière et al., 2024; Guo et al., 2024; Wei et al., 2023; Zhang et al., 2023). Whether it’s traditional methods or
16415
LLMs, they achieve the repair by making a single modification. These outcome-based supervision methods pose a stringent challenge to the model due to the substantial edit distance between input and output. Simultaneously, the one-step modification approach does not align with human behavioral patterns: To be specific, when facing complex tasks, programmers usually repair the buggy program step by step in a cycle of modification, testing, and feedback. This inspires us to pay attention to the exploration of modification processes in APR tasks. Unlike outcome-based supervision, this process-based method requires guidance and supervision at each modification step.
Figure 1 depicts the typical program repair process in programming competitions, which is guided by compilers and test cases: Competitors initially construct a sketch of the program based on the problem’s intent and other constraints. They then complete the repair through a continuous process of submission, feedback, and interaction. Specifically, they initially attempt to solve “Find the majority element” problem by sorting and selecting the middle element. However, when they encounter time constraints, they shift to calculating each element’s frequency. But this method still requires O(n) space complexity. After receiving “Memory Limit Exceed” status, they ultimately use the Boyer-Moore majority vote algorithm with O(1) space complexity.
This example illustrates the characteristics of process-based program repair, which necessitates ongoing interaction with the compiler and test cases for feedback-guided repair. However, applying this type of process-based feedback on LMs appears unfeasible. First and foremost, the primary obstacle hindering related research is the absence of processbased datasets in practical scenarios. Second, the supervision method for intermediate processes in APR tasks are still under exploration. Previous work explored process-based supervision in mathematical problem solving and reasoning tasks (Lightman et al., 2023; Liu et al., 2023a), which required the intermediate steps to be correct. However, the intermediate steps in APR tasks serve as incorrect supervision signals and cannot be directly used for training. Last, interaction with LMs can be achieved through explicit prompt engineering. Attempts to use prompt engineering instead of compiler and test cases for feedback hinder the precise construction of human intentions by LMs.
In this work, to the best of our knowledge, we
conduct the first few comprehensive exploration of process-based feedback with LMs in APR task. For this, we first establish a multi-step program repair dataset called CodeNet4Repair. Following that, we introduce a process-based feedback APR framework called RePair. RePair includes two models: a reward model and a repair model. We start by training a reward model to mimic the compiler as a virtual tool. It takes program text as input and gives assertions about the program’s status. The repair model works on buggy programs, completing one repair in a step. It then offloads the assessment of the program’s status to the virtual tool and waits for feedback to adjust the strategy for the next modification. Finally, we use pass@k as a metric to objectively assess the quality of program repair.

heading: 2 Data Collection
text: As far as we know, there has not been a dataset established for program repair tasks that includes processes. Thus we attempt to develop a procedural dataset specifically for these tasks. Overall, the dataset includes problem description, memory and time limit of the problem, repair process (a series of programs from error to correctness) and resource usage during execution (memory usage, CPU time, and code size). We derive our data from the large-scale programming competition dataset CodeNet (Puri et al., 2021). We make every effort to cleanse and filter this data to guarantee its quality. We clarify problem description, clean up the program from preliminary to fine, and collect additional high-quality test cases. Finally, we organize the program into a procedural format. In Table 1, we compare other excellent datasets and showcased the unique aspects of CodeNet4Repair (Gupta et al., 2017; Huq et al., 2020; Tufano et al., 2019).

heading: 2.1 Problem Description Collection
text: Even professional programmers need to understand the problem that the program is intended to solve when fixing it. CodeNet provides unprocessed HTML for storing the description of each question. We utilize regular matching to extract these descriptions from the CodeNet’s HTML files. Finally, we manually supplement any descriptions that were challenging to recognize with Optical Character Recognition.

heading: 2.2 Program Preliminary Filtering
text: We chose Python as our benchmark language. We design the following rules for preliminary filtering.
(1) No duplicate submissions. Casual or copy submissions do not aid in program repair process.
(2) No malicious submissions, which include automated disruptive attempts, invalid code aimed at attacking the platform’s codebase, and destructive codes designed to interfere with system files.
(3) No privacy breach submissions. Some IDEs automatically generate comments that could reveal the author’s identity.
(4) Ensure each repair process includes at least one acceptable commit. The processes without any accepted outcomes will be excluded.
After this preliminary filtering, we obtain 1,227,259 submission programs. All non-English characters from the code comments are removed prior to fine filtering.

heading: 2.3 Program Fine Filtering
text: Although the original CodeNet dataset provides the program execution status (e.g., Wrong Answer or Accepted), these may vary due to differences in Python versions and environments. We set up a standard Python 3.11.3 environment and are meticulously testing each of the 1,227,259 execution results using test cases to ensure consistent status. Ignoring extra expenses such as process switching and assuming an average execution time of 4 seconds per solution, we would require a total of 4×1227259
60×60 = 1364(Core · hours) CPU time. Finally, we obtain 278,408 consistent programs. We arrange the answer records of each user for each question in chronological order, ensuring that the last program is “Accepted”.
To enhance evaluations precision, we expand the number of high-quality test cases collected online and annotated by hand. We organize these programs with high-quality test cases into a procedure-based dataset called CodeNet4Repair. CodeNet4Repair is an information-leak-free program repair dataset with Apache-2.0 License.

heading: 3 Method
text: Our methodology follows that of Ouyang et al., which aligned GPT-3 with human. We start from StarCoderBase (Li et al., 2023), a 15.5B code foundation pre-trained LM with 8K context length. It is trained on 1 trillion tokens from The Stack (Kocetkov et al., 2023). Figure 2 illustrates the techniques used to train RePair.

heading: 3.1 Supervised Fine Tuning on APR Task
text: To ensure that the LM can understand program repair tasks, we use the prompt templates in Appendix A.1 for Supervised Fine-Tuning (SFT) (Zhang et al., 2022; Huang et al., 2024). Given a vulnerable program sequence x = {x1, x2, ..., xN}, LM is expected to output a robust program y = {y1, y2, ..., yN ′}, yt ∈ V that can pass all test cases, where V is vocabulary. During the training phase, the model parameters θ are learned by maximizing the likelihood of the output and the ground-truth (Zhang et al., 2021). The training objective is to minimize the following loss:
Lce(θ) = E[− logP (y | x; θ)]
= − N ′∑
t=1
logP (yt | y<t,x; θ) , (1)
where E is the expectation over entire dataset, and y<t is a partial sequence before time-step t.

heading: 3.2 Process-based Feedback
text: After supervised fine-tuning, we generally obtain a fine-tuned LM suitable for the APR tasks. However, this model only understands how to provide a possible solution under the condition of given x, lacking process supervision and feedback from environment like compilers and test cases. To ensure that the LM can gradually refine the program through interaction, we introduce reinforcement learning (RL). In this context, we treat the finetuned LM as an actor. Given a state x, its output ŷ is considered an action. Guided by feedback from the reward model, it iteratively refines program towards a final possible result.

heading: 3.2.1 Reward Modeling
text: Instead of using direct feedback from the compiler and test cases, we train a reward model (RM) to assess program quality. The reward model serves as a virtual tool, while the LM optimizes repair strategies by interacting with it. The main reasons boil down to two points: (1) During training, direct interaction with the environment significantly blocks the training pipeline. The batch-processed tensors in cuda must first be transferred to memory for decoding, undergoing syntax check and case testing sequentially; when providing feedback, these results are re-encoded and sent back to cuda. This process drastically reduces the throughput - a cost that is unbearable. (2) During the inference phase,
Algorithm 1 Generate Repaired Program with Process-based Feedback, Actor-Critic Style Input: Trained critic rϕ(·); Trained actor’s policy πθ; Vulnerable program x0; Max iterations T ; Max Patience P Output: Repaired Program
1: t = 0 ▷ Counter of timestep 2: p = 0 ▷ Counter of unimproved patience 3: r0 = rϕ(x0) ▷ Initialize the reward 4: while t < T & p < P do 5: xt+1 = πθ(xt) ▷ Actor generate 6: rt+1 = rϕ(xt+1) ▷ Critic review 7: ∆ = rt+1 − rt ▷ Evaluate the boost 8: if ∆ ≤ 0 then ▷ If no boost 9: p = p+ 1 10: else 11: p = 0 ▷ Counter reset 12: end if 13: t = t+ 1 14: end while 15: return xt−p ▷ Roll back to p steps ago
the inability to access test cases for vulnerable programs compromises generalization performance.
Unlike the methods used by (Christiano et al.; Ziegler et al.; Stiennon et al.; Ouyang et al.), our approach doesn’t require extensive investment in collecting human preferences to train a reward model. We can substantially reduce costs by sourcing program execution preferences from automated compiler and test cases. Specifically, we first empirically define a non-strict partial order based on program quality from high to low:
AC > PE > WA = TLE = MLE > CE > RE.
AC signifies that the program was “Accepted” by all test cases. PE stands for “Presentation Error", which indicates that the output data is correct but not properly formatted. WA, TLE, MLE, CE, RE are abbreviations for “Wrong Answer”, “Time Limit Exceed”, “Memory Limit Exceed”, “Compile Error” and “Runtime Error” respectively. This partial order defines the severity of program errors in an ascending order, from AC (lowest) to RE (highest). We prioritize fixing RE over CE because RE typically indicates more serious issues such as logical errors which pose a greater risk than CE.
Then, we optimize the reward model parameters ϕ using pairwise ranking based on the above partial order. Given K programs with status, all aimed
at solving an identical problem, there is a partial order set D containing ( K 2 ) pairs of partial orders. After shuffling D, we train the reward model by minimizing the following pairwise ranking loss:
Lpr(ϕ) = − 1(K2 ) E(yw,yl)∼D [log (σ (rϕ (yw)− rϕ (yl)))] , (2) where rϕ (·) is a reward model, which takes a program as input and outputs a scalar reward. In practice, we use another smaller pre-trained autoregressive model, and replace the original model’s non-embedding layers with a projection layer to output a scalar value. The pair (yw, yl) is a partial order where yw is the preferred program (for instance, the status of yw is AC while yl is WA).

heading: 3.2.2 Reinforcement Learning
text: In the previous two sections, we fine-tuned a LM and developed a reward model capable of realistically evaluating program states by optimizing loss Lce(θ) and loss Lpr(ϕ) respectively. In this section, we will utilize reinforcement learning (RL) to finalize program repair through multiple interactions with the fine-tuned LM and well-trained reward model (Qian and Yu, 2021). We adopt Proximal Policy Optimization (PPO) algorithm (Stiennon et al., 2020; Ouyang et al., 2022; Schulman et al., 2017) to fine-tune our LM. The LM functions as an actor, generating a repaired program (action) based on the input program (state) and the policy π of the LM (i.e., LM parameters θ). The trained reward model plays as a critic, assessing the effectiveness of repairs and rewarding the LM accordingly. Our objective is to optimize the LM parameters θ by maximizing these rewards.
To be more specific, we supervise the process by maximizing the the following objective function:
Lrl(θ)=E(xi,xi+1)∼Dπ θ ′
[ rϕ(xi+1)−rϕ(xi)−βKL ( θ, θ ′ )]
= E(xi,xi+1)∼Dπ θ ′
[ ∆rϕ(xi+1, xi)− β log
πθ(xi+1 | xi) πθ′ (xi+1 | xi)
] ,
(3) where xi+1 is the refined output based on the input xi and learned LM’s policy πθ′ , rϕ(·) is calculated as the learned reward model, the KL reward coefficient β control the strength of the KL penalty, ∆rϕ(xi+1, xi) = rϕ(xi+1)− rϕ(xi) means the evaluation of the degree of improvement.

heading: 3.3 Multi-step Generation Under RM Supervision
text: We achieved process supervision with feedback by maximizing reward function. In this section, we will demonstrate how the language model interacts with virtual tools during the generation phase to achieve alignment during both training and generation stages. We continuously request the LM to refine solutions until either of two conditions is met: (1) there is no improvement (∆ ≤ 0) in P consecutive steps; (2) the maximum number of iterations has been reached. The Algorithm 1 elucidates the specific process of generation.

heading: 4 Experiments

heading: 4.1 Data Preparation
text: We sample repair records that corrected after twostep modifications from CodeNet4Repair, resulting in three versions, and calculate the edit distance between these versions. In Figure 3 (a), the findings align with our hypothesis: The edit distance required to complete the repair in one step is longer than that required in multiple steps. Despite an increase in overall cost, step-by-step repairing can reduce the complexity of tasks.
We ensure evaluation quality by using only questions with high-quality test cases for our training and testing sets. We manually annotate and collect additional high-quality test cases from the internet to serve as hidden tests for each problem. This process resulted in test cases for 794 questions.
Figure 3 (b) shows the distribution of these test cases. To prevent data leakage, we divided CodeNet4Repair based on the problem ID in a ratio of 9:1. The training set consists of 94,062 repair processes, while the testing set comprises 10,144 repair processes. Table 2 provides the statistical information about filtered dataset.

heading: 4.2 Experimental Setup
text: In the fine-tuning stage, we train our model using mixed precision training. We use AdamW optimizer with 2e-5 learning rate. To prevent training instability caused by an excessively high learning rate, we utilize a cosine LR schedule down to 10% of the original learning rate with learning rate warmup. We use ZeRO++ to distribute model tensors across accelerators (Wang et al., 2024).
In reward modeling, we adopt the LR of 9.6e-6 and cosine learning rate schedule. We randomly select K = 9 programs in same problem and rank them based on their execution status. Each batch contains 64 units, thus a single gradient backpropagation involves 64× ( 9 2 ) = 2304 comparisons.
In the process-based feedback stage, we initialize RL policies from the fine-tuned LM. We train the LM for 32k episodes with 512 batch size. We assign a KL penalty factor, β, of 0.02 and establish a learning rate of 9e-6. We perform nucleus sampling with top_p=0.95 and top_k=50, and set the temperature to 0.2.

heading: 4.3 Baselines
text: We compare the latest state-of-the-art LM that possess few-shot learning and code comprehension capabilities. These models include PaLM (chatbison-001) (Anil et al., 2023), GPT-3.5 (gpt-3.5turbo-0613), Claude2 and ChatGLM-Pro (Zeng et al., 2022). We also compare some open-
sourced models such as StarCoderBase/Chat (Li et al., 2023), CodeGen2, CodeGeeX2 (Zheng et al., 2023), LLaMA2/-chat (Touvron et al., 2023).

heading: 4.4 Evaluation
text: Previous program repair datasets lacked annotated test cases and stable test environment (Dinella et al., 2020; Hendrycks et al., 2021; Li et al., 2022; Wang et al., 2023), leading to a shortage of automatic evaluation methods for program repair based on execution results. We contend this issue and advocate for the evaluation using execution outcomes.
Following previous code generation’s evaluation method, we use pass@k as our evaluation metric (Hendrycks et al., 2021; Chen et al., 2021; Nijkamp et al., 2023). Here we use the unbiased estimator proposed in (Chen et al., 2021), which is defined as follows:
pass @ k = E [ 1− ( n−c k ) ( n k ) ] . (4)
In this work, we report k ∈ {1, 3, 5} as the final result. The reason is that in pass@k, a larger k provides a more comprehensive assessment of LM’s ability to generate code. However, in practice, users won’t request multiple candidates repeatedly; therefore, reporting a small number of k suffices for program repair evaluation.

heading: 4.5 Main Results
text: We use the pass rate of all problems in 14 billion submission records as a standard to measure the difficulty of the problems. These problems are categorized into three levels - easy, medium, and hard - based on their pass rates divided into tertiles. The model’s performance at various difficulties is detailed in Table 3. The results of the experiment
indicate that our model outperforms all other opensource models. Even when compared to sophisticated commercial LLMs, we achieve competitive results. Starting with an analysis of the open-source models: We acknowledge that some performance gains come from supervised fine-tuning. However, as evidenced by StarCoderBase/Chat results, our foundation model lacks zero- or few-shot program correction capabilities and struggles with repair tasks without supervised fine-tuning. In comparison to LLMs designed for code downstream tasks, our repair model generates programs that better meet task requirements and outperforms them at every difficulty level. Moreover, in LLaMA2-Chat we found that multiple rounds of dialogue-based supervised fine-tuning do not degrade but rather enhance its understanding of human instructions and repair abilities. While our model’s performance is not yet on par with commercial LLMs such as ChatGPT and Claude2, considering the resources they’ve invested - including compute, data collection, and manual annotation among others - which are significantly greater than ours, we have still achieved competitive results. This validates the effectiveness of our process-based feedback method and provides guidance for future research.

heading: 4.6 Model Analysis
text: We explore the rationality of the model including process supervision, feedback, and reward function design. We design different variants from those three perspectives. First, we arrange and combine process supervision and feedback to create four different approaches. Then, we discuss the design of reward functions within the process-based feedback framework.
Impacts of Process Supervision. We attempt to eliminate process supervision signals, enabling
the model to depend solely on single-step supervision signals. Without feedback, the model degenerates into a fine-tuned single-step program repair model. While with the feedback, the fine-tuned model’s adjustment of its repair policies is limited to the final signal.
Impacts of Feedback. We remove feedback from the reward function during process supervision, which prevents the language model from using reinforcement learning for policy adjustment. When given a repair sequence, we force the language model to use the subsequent repair step as a supervision signal for learning.
Impacts of Reward Function. We introduce two variants to verify the effectiveness of pairwise ranking in reward modeling. We use two ranking methods: point-wise, and list-wise. Point-wise ranking scores individual program states without considering their order relationship; while list-wise ranking compares multiple programs together (beyond pairs) to determine their order relationship.
The results of the experiment are shown in Table 4. We can draw the following conclusions from the results: (1) Regardless of the supervision method used, introducing feedback to adjust LM’s repair policies is necessary. Performance may significantly decrease due to additional noise if feedback is lacking in the process-based supervision method. (2) Pair-wise ranking proves more effective as a reward model compared to point-wise or list-wise ranking. Using point-wise ranking solely requires the model to score each program based on compiler and test case, without comprehending the program’s preferences and modification direction. Conversely, providing K rankings simultaneously
in a list-wise task can be overly complicated and potentially hinder performance.

heading: 4.7 Process-based Feedback Can Bring Performance Benefits
text: Someone might question the superiority of RePair over current open-source models due to our use of SFT on the training set, while other models employed a few-shot setting. It is important to clarify that SFT was utilized to enhance the model’s comprehension of the repair task, rather than being the primary factor for performance improvement. To substantiate this, we also fine-tune several opensource models with similar parameter counts using the same dataset. The results, presented in Figure 4, demonstrate a performance enhancement following SFT. However, this improvement is largely due to the model’s increased understanding of the task and the standardization of output formats. Furthermore, when combined with SFT, process-based feedback method significantly amplifies the model’s potential, leading to even greater performance gains.

heading: 4.8 Smaller Models May Struggle to Receive Effective Explicit Feedback
text: As mentioned in the abstract and introduction, understanding explicit prompts for small-scale LMs remains challenging. We directly interact with LMs using the prompt template provided in Appendix A.1. We fine-tune two open-source models, StarCoder and CodeGen2, within 5 steps and present the experimental results in Figure 5. The experimental results indicate that: (1) The performance of the open-source model peaked after three
steps of fine-tuning. (2) ChatGPT’s performance was further enhanced after receiving feedback from compilers and test cases, demonstrating better results than one-step repair. (3) Small-scale models struggle to comprehend feedback information through explicit prompts.

heading: 4.9 Qualitative Results
text: As shown in Figure 6, our model through step-bystep modification, have effectively simplified the complexity of each repair step and achieved satisfactory “Accepted” results. For more examples, please refer to the Appendix A.3.

heading: 5 Related Works

heading: 5.1 Large Language Models
text: The recent years have witnessed a substantial evolution in the field of large language models. language modeling is the prediction of the probability distribution of the next token given a context. The scaling law makes it possible to enhance performance by increasing the size of the model and data (Kaplan et al., 2020). The emergence of largescale language models with over 100B parameters has consistently resulted in record-breaking performance (Srivastava et al., 2023; Hendrycks et al., 2021; Chen et al., 2021). Larger models begin to solve problems that smaller models cannot handle, a phenomenon known as emergence. Typical phenomena of emergence include In Context Learning (ICL) and Chains of Thought (CoT) (Brown et al., 2020; Wei et al., 2022). The core of ICL is to draw knowledge from analogies. After providing demonstrations, the query problem is presented alongside prompts to form the input. A CoT is a prompt formed through a series of logical thinking steps, guiding the LLMs to answer in a thoughtful manner.
This method can significantly improve performance on reasoning tasks (Liu et al., 2023b). Our work is dedicated to using small-scaled language models to enhance program repair performance.

heading: 5.2 Outcome- and Process- based Approaches
text: The outcome-based approaches supervise the final results, while the process-based approach focuses on each step leading to the result. Both have their strengths and weaknesses: outcome-based requires fewer supervision signals, but process-based approaches align more with human thinking patterns. In (Uesato et al., 2022), researchers conducted experiments on math dataset and found that both approaches produced similar error rates, but due to cost considerations, the process-based method performed better. However, recent research (Lightman et al., 2023) has proposed a different view; after building larger datasets and annotating more processes, the process-based method showed clear advantages. Process-based can help machine learning systems accurately understand humans’ way of thinking and regulate potential hallucination.

heading: 5.3 Learning from Feedback
text: Through learning from feedback, language models can effectively receive signals beyond labels (Gou et al., 2024). Some NLP tasks rely on automated evaluation metrics such as BLEU, ROUGE to get feedback from ground truth. However, this kind of automatic feedback that solely depends on exact matching is not an effective assessment criterion (Colombo et al., 2022). Therefore, researchers are attempting to train language models using human preferences as supervisory signals. In some open text generation tasks, learning from human feedback has significantly improved performance (Jaques et al., 2019; Bahdanau et al., 2016; Lawrence and Riezler, 2018; Zhou and Xu, 2020; Stiennon et al., 2020). Following this line, (Ouyang et al., 2022) applies human feedback to GPT-3 to achieve alignment with humans, thereby mitigating the issue of generating untruthful or toxic text. Our work focuses on obtaining feedback from the compiler as a virtual tool, integrating signals from the tools into the training process of language models.

heading: 6 Conclusion
text: In this work, we proposed an automated program repair method that utilizes process-based feedback. This approach is inspired by strategies used in programming competitions and incorporates process
supervision into the repair process. Due to the unavailability of suitable datasets, we first created a multi-step repair dataset called CodeNet4Repair. This dataset not only includes the initial faulty programs and the final accepted ones but also records the intermediate repair process. On this basis, we implemented supervision over the process with feedback. Specifically, we fine-tuned a pre-trained model StarcoderBase through to make it understand program repair tasks. Then, we empirically defined a partial order relationship for program states and trained reward models using pairwise ranking. This reward model would serve as a critic, assessing each attempt by the LM to polish the program. Meanwhile, the LM plays the role of an actor, constantly adjusting its policies for repairing the program based on received rewards. During the inference stage, LM iteratively repaired the program until the exit conditions are met. The experimental results indicated that the process-based feedback method can demonstrate excellent performance. With smaller model sizes, RePair still can rival or even surpass closed-source commercial large models. We believe that process-based feedback is currently underexplored, and will continue to explore more universal methods.

heading: 7 Limitation
text: We contributed a dataset called CodeNet4Repair, which focuses on program repair in real competition scenarios. We acknowledge that CodeNet4Repair still has limitations in evaluating model repair capabilities. Due to time and resource constraints, we were unable to collect all repair processes across different languages and software engineering domains. But there are significant similarities between competition and engineering scenarios. This implies that our model can achieve a level of usability for engineering project purposes, and our dataset can partially evaluate its repair performance in an engineering context.

heading: 8 Acknowledgement
text: This research was partially supported by grants from the National Natural Science Foundation of China (No. 62106244), the Anhui Provincial Natural Science Foundation (No. 2308085QF229), the Fundamental Research Funds for the Central Universities (No. WK2150110034), and the CIPSCSMP-Zhipu.AI Large Model Cross-Disciplinary Fund.

heading: A Appendix
text: A.1 Prompts Used in Experiments
We performed supervised fine-tuning and explicit feedback experiments following the prompt template shown in Figure 7. For explicit feedback, we also input optional standard test cases.
A.2 Ethics Statement As creators and users of LLMs, we recognize the profound ethical responsibilities that come with their development and deployment. We will strive to avoid creating or perpetuating harm, including the reinforcement of bias, misinformation, and harmful stereotypes. We advocate for the use of open-source LLMs in potential ethical risk research. We believe that a public LLM can reveal its internal state, thereby avoiding possible risks.
A.3 Detailed Case Study In the next page, we provide six examples of repairs. Through the examples shown, we found that the edit distance for single-step modifications (directly from Step 1 to Step 3) is very high, resulting in tasks being too complex (e.g., having both compilation errors and semantic errors) to be completed in one step. However, process-based modifications can decompose complex tasks into simple ones and solve each problem gradually, thereby producing good repair effects.

heading: Future Work
text: This validates the effectiveness of our process-based feedback method and provides guidance for future research