AuPair: Golden Example Pairs for Code Repair
Authors: Aditi Mavalankar
Hassan Mansoor
Zita Marinho
Mariia Samsikova
Tom Schaul
Venue: ICML 2025 poster (2025)
Keywords: LLM
Coding
In-context learning
Primary area: deep_learning->large_language_models
TL;DR: Prompting with high-diversity examples of fixes boosts code-repair performance.

Abstract: Scaling up inference-time compute has proven to be a valuable strategy in improving the performance of Large Language Models (LLMs) without fine-tuning. An important task that can benefit from additional inference-time compute is self-repair; given an initial flawed response or guess, the LLM corrects its own mistake and produces an improved response or fix. We leverage the in-context learning ability of LLMs to perform self-repair in the coding domain. The key contribution of our paper is an approach that synthesises and selects an ordered set of golden example pairs, or AuPairs, of these initial guesses and subsequent fixes for the corresponding problems. Each such AuPair is provided as a single in-context example at inference time to generate a repaired solution. For an inference-time compute budget of $N$ LLM calls per problem, $N$ AuPairs are used to generate $N$ repaired solutions, out of which the highest-scoring solution is the final answer. The underlying intuition is that if the LLM is given a different example of fixing an incorrect guess each time, it can subsequently generate a diverse set of repaired solutions. Our algorithm selects these AuPairs in a manner that maximises complementarity and usefulness. We demonstrate the results of our algorithm on 5 LLMs across 7 competitive programming datasets for the code repair task. Our algorithm yields a significant boost in performance compared to best-of-$N$ and self-repair, and also exhibits strong generalisation across datasets and models. Moreover, our approach shows stronger scaling with inference-time compute budget compared to baselines.

AuPair:
Golden Example Pairs for Code Repair
Aditi Mavalankar Hassan Mansoor'
Zita Marinho
Masha Samsikova'
Tom Schaul
Abstract
1. Introduction
Recent progress in the field of Large Language Models (LLMs) has resulted in models that keep getting better at generating responses to user queries. When providing these already powerful models with more inference-time compute-increasing number of LLM calls-methods that sample different responses and then select the best among them, such as best-of-N (Stiennon et al., 2020) self- consistency (Wang et al., 2023b), have shown clear ben- efits. While these approaches are more breadth-focused, another way to leverage inference time compute is to im- prove or repair the LLM's initial guesses by generating better fixes (Olausson al., 2024). We propose combining the benefits of both these approaches to generate wide set of repaired solutions for poor initial LLM responses, out of which the highest-scoring solution is the final answer.
Scaling up inference-time compute has proven to be a valuable strategy in improving the perfor- mance of Large Language Models (LLMs) with- out fine-tuning. An important task that can benefit from additional inference- time compute is self- repair; given an initial flawed response or guess, the LLM corrects its own mistake and produces an improved response or fix. We leverage the in- context learning ability of LLMs to perform self- repair in the coding domain. The key contribution of our paper is approach that synthesises and selects an ordered set of golden example pairs, or AuPairs, of these initial guesses and subse- quent fixes for the corresponding problems. Fach such AuPair is provided as a single in-context example at inference time to generate repaired solution. For an inference-time compute budget of N LLM calls per problem, N AuPairs are used to generate N repaired solutions, out of which the highest-scoring solution is the final answer. The underlying intuition is that if the LLM is given a different example of fixing an incorrect guess each time, can subsequently generate a diverse set of repaired solutions. Our algorithm selects these AuPairs in manner that maximises complementarity and usefulness. We demonstrate the results of our algorithm on 5 LLMS across 7 competitive programming datasets for the code repair task. Our algorithm yields a significant boost in performance compared to best-of-N and self-repair, and also exhibits strong generalisa- tion across datasets and models. Moreover, our approach shows stronger scaling with inference- time compute budget compared to baselines.
To generate a wide range of repaired solutions for each initial LLM response, we exploit the in-context learning capability exhibited by LLMs. The main contribution of our paper is an algorithm that, given an inference-time compute budget of N LLM calls, produces an ordered set of up N golden example pairs, AuPairs', Each such AuPair contains the initial guess and consequent fix for the corre- sponding coding problem, along with their respective unit test scores. An example AuPair is illustrated in Fig. 1. At inference time, the contents of an AuPair are concatenated as described A.10, and provided generate fix for the test problem. This is done for each of the N AuPairs: of all the fixes generated, the one that gets the highest score on the unit tests is selected.
1-shot example to
A core ingredient of our proposed algorithm involves the selection of these AuPairs. We propose a submodular ap- proach that selects AuPairs based on the ability of each pair solve different problems in held-out validation set. Since the AuPairs are selected such that each subse- quent AuPair solves different set of problems than the ones solved by its predecessor AuPairs, by design, we get complementary AuPairs. Also, as the list of AuPairs is con- structed by taking the greedy pair at each step, only those
'Google DeepMind. Correspondence to: Aditi Mavalankar <mavalankarde @wweemind.com..........e.
'The name AuPair is coupling of Au, the chemical symbol for gold, and Pair, jointly referring to golden pairs that are produced by algorithm. The high-level interpretation is that like "au pair", the approach guides the LLM towards better behaviour.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

AuPair: Golden Example Pairs for Code Repair
2. Approach
The Lattie Elashant loves nettrr, passtiie intaree The little Elephant eancs nuebor have Juast common (the digit in their decioal espessetattons. Help
Lecssers
the k,
The goal of our proposed approach in the coding domain is to improve code repair performance on unit tests at in- ference time by curating list of pairs that can be provided as in context examples. The code repair prompt includes an in-context example, followed by text description of the problem to solve and the initial guess generated by the LLM. The LLM then generates revision, or a fix that im- proves performance on the unit tests for that problem. In the prompt, we include the scores achieved by the guess and fix on the unit tests, but no other execution feedback.?
A Single line contados single Integee x (1 a x.K 1879),
output
stngle iine print an intaegee the ancwer to the probiiee
exmples
Inout
Output
Insut
Output
def sollede: Lnt]:
score: 6.6
def solvete: int):
count
count pangers, 28 antted Sche33 for dipie sn 13 breas (=Int(couut))
Guass
Lf XB8) In SErEE) or Steelaga)
31
count
In order to disentangle repair performance from the quality of initial guesses, we first curate composite datasets consist- ing of initial guesses for all the coding problems. Given a dataset consisting of problems and their corresponding tests, we generate an imitial guess for each problem and compute its score on the unit tests. If the guess passes all the unit tests for that problem correctly, no further improvement is required and we discard that problem. If not, we add this guess along with its corresponding score and problem as datapoint to our curated dataset. This dataset is then di- vided into training, validation, and test datasets. We use the training dataset Dirain = DD for pair generation (Fig. 2), and the validation dataset Daat for AuPair extraction. The test dataset Dest is used in the final testing phase.
scone: 2.0
An example AuPair The --M-eenerated guess and fix, along with their respective scores for the corresponding CodeForces problem (problem description at the top). To provide this AuPair in context at inference time, the problem description, guess, and fix, are concatenated as described in Fig. 18.
Figure I:
pairs that lead to increase in the fix scores are selected, resulting in usefil AuPairs.
In this paper, we address the code repair task: given a cod- ing problem, an initial guess which is L.M--genrated code, and a set of test cases that are used only to evaluate the correctness of the generated code, the LLM has to generate an improved fix for the problem. We show that the fixes generated by AuPair are significantly more useful and di- verse than those generated using best-of-N ($3) for the same inference-tine time compute budget. We also show that AuPair outperforms self-repair (Olausson et al., 2024), which gener- ates intermediate verbal feedback before generating repaired solutions for a problem. The key contributions of our paper are the following:
Following the creation of these three datasets, we now dis- cuss our approach, which consists of two main phases: 1) Pair Generation ($2. I), and 2) AuPair Extraction ($2.2).
2.1. Phase I: Pair Generation
In this phase, we generate a large set C of pairs that are potential candidates for our final set of AuPairs. This is done in the following manner: a problem along with its initial guess is sampled from the training dataset D. The LLM generates a fix for this guess, If this generated fix has higher score on the unit tests for that problem than the initial guess, this guess-fix pair is added to C. Furthermore, if this fix is imperfect, i.e. it does not pass all the unit tests. it becomes a potential guess with further scope for improvement, so it is added as a new guess to the training dataset D. This is repeated several times to collect a large set of such candidate pairs.3 3 A visual illustration of this phase is provided in Fig. 2. While we include the pair generation phase for completeness, it is important to note that in several
An inference-time algorithm, which constructs golden set of code repair examples, or AuPairs, that boost performance significantly when used as in- context examples ($2).
Reliably outperforming best-of-N (Stiennon et al., 2020) and self-repair (Olausson et al., 2024) across 5 different models: Gemini-1.5-Pro. GPT-4o-mini, Gemini-1.55Flaat Gemma-278 Gemma-99 ($3.1).
"The repair prompt is composed using the prompting strategy shown in Fig. 18.
Strong scaling performance with inference time com- pute, with far less diminishing returns than oest-of-N and self-repair ($3.3).
"Since the aim is to collect large set of pairs, we want the L.M to generate wide variety of fixes. For this, we compose a k-shot prompt for repair, which the in-context example pairs are randomly sampled from the existing set C for each sampled problem. As the LLM generates more fixes, C gets populated, subsequently resulting in more diverse prompts.
Robust out- distriibittt generalisation, across both model sizes and datasets ($3.4).
2

---

Au Golden Example Pairs for Code Repair
vsl
AuPairs
train
sample guess
repair
Fix
candidate pairs
candidate pairs
repair
submoduler selection
k-shot prompts
fx-quality
I-shot prompts
matrix
sample
We collect a large set guesses for coding problems and their fixes yielding candi- date pairs D that will later be used to get AuPairs. A problem with its guess is sampled from the training dataset, with & randomly sampled few-shot pairs from the candidate pair buffer. This k-shot prompt is passed through an LLM to generate a fix, which is evaluated on the unit tests. If this fix is better than the guess, this (guess, fix) pair is added to the set of candidate pairs. Imperfect fixes are added as new guesses the training dataset (See $2.1).
Figure 2: Pair Generation:
Figure 3: AuPair Extraction: Given a large set candidate pairs the next step is to extract AuPairs, Each pair D is provided as a 1-shot in-context example in the prompt for each problem and its guess from the validation dataset. These prompts are passed to the LLM which generates fixes that are evaluated on unit tests to populate a fix- fix-quality matrix, as described in Algorithm 1. Next, submodular selection mechanism is applied on this fix-quality matrix to obtain the list of AuPairs (Algorithm 2).
Next, use this fix quality matrix M extract the AuPairs by taking the following steps: 1) Select the pair that gets the highest mean score across all problems in D say Ck. and add it to the list of AuPairs A A + AU ck- This is a greedy way of selecting the best pair given all previous AuPairs and produces an ordered set of AuPairs. 2) Subtract the row score Mj (i.e. score on all the problems in Dval) of this newly added pair from all the rows in the fix quality matrix with update: M M2. This ensures that redundant AuPairs are not produced by the approach. The updated fix quality matrix is clipped to (0, 1) since any negative value in the matrix M, say Mi,j implies that the problem x, cannot be improved further by pair cs. Without clipping, we would not get an accurate estimate of the improvement in the next step of submodular extraction. 3) Repeat this process until improvement falls beyond a certain tolerance
other domains, paired data may already be available. In such cases, the set of candidate pairs C that we curate in this step, can simply be replaced by the given paired data.
Algorithm Fix quality matrix computation
LLM
large language model
C
candidate pairs validation dataset code eval function
Require:
Dval
score
1: init fix quality matrix M +- x1]
2: for pair G. problem z; E C x Val do
3: build 1-shot prompt: p + Q Φ
4: generate fix: <- (L)
5: evaluate fix: M < score
6: end for
7: return M
Algorithm 2 Submodular AuPair extraction
{
M fix quality matrix
Require:
C candidate pairs
2.2. Phase 2: AuPair Extraction
e tolerance
Now that we have large set C of candidate pairs, the next step is to determine which of these will actually help boost performance, i.e., which of these are AuPairs. We do this in a submodular fashion by making use of the validation dataset Dval- For each pair-problem combination (c,, 2,) e Dyal, we build 1-shot prompt p using the prompting strategy described in Fig. 18. This 1-shot prompt p is given as input to the LLM, which generates a fix for the given problem xy. The fix generated by the LLM is then evaluated on the unit tests and stored in the fix quality matrix M E R|C) x Daal at index j). This part of AuPair extraction is outlined in Algorithm
1: initialise AuPairs A +
repeat
3: per-pair scores:
row-mean(M)
4: get best pair: C2 argmax.
5: append to AuPairs: A + A + Ck
6: update M 4- clip(M Mr, 0, 1)
7: until max(m) < return A
This process of iteratively constructing the set of AuPairs ensures that they improve performance on disjoint parts of
3

---

AuPair: Golden Example Pairs for Code Repair
the problem space. The AuPairs that we obtain from this phase are used in the same manner at inference time, as 1-shot examples, to improve code repair performance. The compute budget N determines the number of AuPairs that we can use at inference time. Since the AuPairs form an or- dered set, the first N AuPairs are used for compute budget of N LLM calls. The final solution for each problem is the one that passes the most test cases, among all generated so- lutions. This submodular extraction of AuPairs is shown in Algorithm 2. Fig. 3 has a joint diagram depicting fix quality matrix computation and submodular AuPair extraction.
out of N responses generated by the LLM, the test pass rate for a test dataset with P problems is calculated as:
P Σ
ITyl
1
Σ 1 (eeaal(coeee : Tp.)) == 13
max
P p=1
[) [Tpl
where Tp refers to the umit tests for problem p. and codep. is the code generated by the LLM for problem in the thh LLM call. The innermost loop computes the percentage of unit tests passed by the LLM output codep, Next, we select the code output that has the highest value for percentage of unit tests passed, indicated by the max operation over all LLM calls i E (1,.... N... The outermost loop averages this across all the problems in the test dataset.
3. Experiments
Datasets: We use 7 datasets that contain problems and test cases from competitive programming contests: 1) Code- Forces (8.8k problems), 2) AtCoder (1.3k problems), 3) HackerFarth (1.2k problems), 4) CodeChef (768 problems), 5) LiveCodeBench (400 problems), 6) CodeJam (180 prob- lems), and 7) Aizu (2.2k problems) (Li et al., 2022b; Jain et al., 2024). We choose CodeForces and AtCoder, sepa- rately, for in-distribution testing, and use the rest exclusively for out-of-distribution testing. Our training/ / validation test split proportions for the CodeForces and AtCoder datasets are 3........55555 505% Some datasets have difficulty levels as part of the problem: for those we maintain the same strati- fied distribution of questions in the training, validation and test datasets.
We also report the results of strict accuracy (Hendrycks et al., 2021), which is the percentage of generated solutions that pass all test cases (definition and results in Appendix $A.2). Note that cannot provide fair results for the pass @ metric because that involves making an assumption that all k responses from the LLM are i.idd generated, whereas our approach produces fixes in specific order. In our case, the first AuPair is more useful than the second, which is more useful than the third, etc; so their success probabilities monotonically decrease as a function of k:
Baselines: We compare the effectiveness of our proposed approach with best-of-N (Stiennon et al., 2020) and self- repair (Olausson et al., 2024). Best-of-N is strong baseline to improve model performance by allowing multiple LLM calls at inference time. Of the N generated responses, the highest-scoring response is selected. To ensure the sampling of high-quality diverse responses in best-of-N, we set the temperature 1.0 (Renze & Guven, 2024). Self-repair, on the other hand, uses the compute budget of N LLM calls to either generate verbal feedback or repaired code. Our compute budget is N = 32, of which 4 LLM calls are used to generate verbal feedback and 7 L.LM calls to generate repaired code for each verbal feedback.
Models: We demonstrate the superior code repair capability of AuPair 5 different models: Gemini-1.5-PPo GPT-4o- mini, L.5-ai- Gemma-27B, and Gemma-9B In addition to using these models for dataset curation and pair generation, we also look at the transfer capabilities of our method with respect to different models in Section 3.5.
Evaluation: We conduct two types of evaluation: in- distribution and out-of-distrbbttion For in-distribution eval- uation, we use the test split from the same dataset as the one used for pair generation and AuPair extraction. This ensures that the format of questions and test cases in the test questions matches that of the AuPairs. Out-of-distribution evaluation uses a different coding dataset; this means that the test samples have different format of questions, difficulty, types of problems and test cases than the AuPairs. Another axis of out-of diitrii-iitiittionnn evaluation is the model axis: we report performance obtained using AuPairs produced by different model than the one used for inference.
The remainder of this section will discuss a plethora of empirical results, on overall and ablated performance 1 and 3.2), scalability and generalisation ($3.3 to 3.6), and diversity ($3.7 to 3.10).
3.1. Significantly Boosted Code Repair Performance
Metrics: Our primary metric is the commonly used test pass rate, also called test case average (Hendrycks et al.,
The first step to assess code repair performance is to measure in-distribution performance; namely generating and select- ing AuPairs on the training and validation sets that match the test dataset, and using the same model for evaluation as AuPair construction. We do this for 2 datasets (Code- Forces and AtCoder) and all models. Fig. shows the
2021; Ouyang et al., 2024; Wu & Fard, 2024), which we compute as the average percentage of test cases passed, as done in prior work (Tang et al., 2024; Ding al., 2024; Chen et al., 2023). In our setting, since we choose the best
4

---

Au Golden Example Pairs for Code Repair
CodeForces score @ 32
AtCoder score @ 32
Initial
0.6
0.4
Best-of-N
Self-repair
0.5
0.3 AuPair
0.4
0.3
0.2
0.2
0.1
Gemma-98
Gemma-278
Gemini-1.3 Siaiiii-..--Faaat 5-F-shh
GPT-4o GPT-4o-mini
Gemini-1.5-Pr
Gemma-98
Gemma-278
Gemini-1 1.5-Flash
GPT-4o-mini
Gemini-1.5-Pro
Figure 4: In-distribution code repair performance: with N 32 LLM calls inference time and the same train / val test data distribution, we compute the test pass rate. The same model is used for generating the initial guesses and fixes and the AuPair extraction. CodeForces (left, 8.8k problems) and AtCoder (right, 1.3k problems), see $3.1 for more details.
resulting comparison between the best-of-N and self-repair baselines and AuPair, for budget of N = 32 LLM calls inference time.* AuPair clearly superior to best-of-N and self-repair (matching in few cases) on all models and datasets, sometimes by wide margins. This clearly estab- lishes that proposal of providing different in in-context example of code repair in each LLM call can significantly boost performance.
3.2. Selection Matters: AuPairs are More Effective than Random Pairs
We design an ablation to disentangle the two possible sources of improvement that our approach demonstrates, namely 1) in-context learning and 2) the choice of AuPairs. It is implausible for the boost in performance re- sult from the LLMs' in-context learning ability, and that the same result could be achieved by including any set of pairs. On the other hand, our approach specifically targets complementarity during construction of AuPairs in that sub- sequent AuPairs are selected based on their ability to solve problems that previous AuPairs were unable to solve. To resolve this, compare the full method to a random-pan pair bascline that randomly seleets pairs from the full candidate set (the result of Phase 1), deduplicating the problems that the random pairs solve (which makes it a stronger baseline). Fig. 5(a) shows that AuPair significantly outperforms the random-pan pair bascline for N = 1, ..., 32, saving 2.5 3x more compute by achieving the same score with 12 AuPairs as the random pair baseline gets with AuPairs. Note that for any fixed candidate set, N grows toward the size of the full set of pairs, the performance of the random-pan baseline will equal that of AuPair.
An interesting side-result visible in initial performance, ie., the performance of the initial guesses, which have to repaired. Gemini-1.5 1.ooPr, despite being superior model to Gemini--155-Flane shows worse initial performance. Since the code generated has certain conditions that allow suc- cessful execution, we observe that many initial guesses of generated code fail because they do not obey these con- ditions (see Appendix $1.1 In such cases, code repair with either best-of-N or self-repair is unlikely to give us high boost in performance since the initial solution is badly formatted. This is one clear where having AuPair in context significantly improves performance. Thus, using AuPairs in conjunction with high performing models leads to large performance improvements despite poor initial per- formance, as we can see for both CodeForces and AtCoder with the Gemini- 1.5-Pro model Fig. 4. This also miti- gates the need for more sophisticated prompt engineering to large extent.
3.3. Better Scaling with Inference- Compute
At fixed budget of N 32 LLM calls, our results look promising. In this section, we investigate whether and how performance scales with the compute budget N. Fig. 5(b) plots the score as a function of N using Gemini-1.5-Pro on the CodeForces dataset (additional scaling curves for Gemini-1.5-Flaas and strict accuracy metric in the Ap- pendix, see Figs. 10 and 13). For each additional L.M call, we use the next best AuPair produced by the algorithm
*Since our algorithm yields a variable number of AuPairs, datasets with fewer generated pairs, this number can be less than 32. To have a fair comparison, we set the same compute budget N for best-of-N and self-repair. This is the case for AtCoder (Fig. right), where our algorithm yields 14, 15, and AuPairs for Gemma-9B, GPT-4o-mini, and Gemma-277 respectively. So the corresponding baselines also use the same compute budget.
5

---

AuPair: Golden Example Pairs for Code Repair
Pairs v/s AuPairs
Scaling inference compute
0.45
Initial guess
Random Pairs
0.5
Best-of-N
0.40
AuPair
Self-repair
0.4
0.35
0.25
Test
0.2
0.15
0.1
10
15
20
25
30
10"
10a Inference compute budget N
10a
Number LLM calls
Figure 5: (a) AuPairs vs. random pairs: AuPairs (green) are far (~ 2.5 3x) more compute-efficien than random pairs (red); it takes only 12 AuPairs to reach the same performance as 32 random pairs; (b) Scaling inference-time compute: the score increases with compute budget at much steeper rate using AuPairs (CodeForces, Gemini-1.5-Prr
and provide in context to generate the LLM response. Our algorithm produces 144 AuPairs for the CodeForces dataset using Gemini-1.5-Pro, and achieves test pass rate of 51.32% and strict accuracy of 39.73% (see $A.3) at 144 LLM calls. The results shows a clear scaling trend with a consistent log-linear performance increase as function of compute, without any sign plateau. More impor- tantly, the increase is substantially sleeper than best-of-N and self-repair (which achieve test pass rate of 16.05% and 15.79% and strict accuracy 12.04% and 12.23% respec- tively); in other words, prompting with in-context comple- mentary AuPairs makes more efficient use of compute than either repeated sampling given fixed repair prompt, or repair with model-generated verbal feedback.
3.5. Decent Cross-Model Transfer
Now that we have seen that our approach can exhibit very good -ut-o--distribution generalisation along the data axis, evaluate it on its ability to generalise the model axis, i.c., we look at the performance of AuPairs collected using different model. We evaluate this cross-model transfer capability for several model combinations on CodeForces. The resulting ablations are shown in Fig. 7(a), and help disentangle the impact of the AuPairs versus the code re- pair capabilities of the inference model. key takeaway is that the Gemma models exhibit worse performance, gardless of the quality of AuPairs used at inference time, indicating that they are inferior at the capability of code repair. Gemini- 1.5-Fashh performs much better at code re- pair, and its sensitivity to the source of AuPairs is negligible: it is equally performant for each source. Gemini- 1.5-Pro on the other hand, is sensitive to the source of AuPairs; in particular, when Gemini-1.5-Pro uses AuPairs collected by the same model, it achieves the best performance by large margin. With AuPairs selected using other mod- els, Gemini-1.5-Pro achieves comparable performance to Gemini- 1.5-Fash One reason for the standout performance when using Gemini-l.5-PM AuPairs seems that those ex- amples result in substantially more diverse generations, as shown Section 3.7. However, Fig. 7(a) whole sug- gests that there is an ordering in terms of performance: 1) the model used at inference time has to have good code repair capabilities, and 2) the stronger the model is at code repair, the more improvement we can expect from it with a higher quality of AuPairs. We also show additional results for cross- model transfer on GPT-4o in $A.4.
3.4. Strong Generalisation to Out-of-distribution Datasets
The aim of this experiment to determine whether our approach exhibits out-of-distribution generalisation, i.e., given AuPairs collected on different dataset, see if we retain the performance improvements that obtain in- distribution. To test this, evaluate the AuPairs collected using Gemini-1.5-Pir on CodeForces on the other datasets and compare them with the corresponding baselines. Fig. 6 shows that across datasets, our approach outperforms base- lines by large margin, despite having -itt---diittibttion AuPairs. This implies that the process of collecting AuPairs may only be needed on one dataset and the benefits can be reaped across wide range of problems (from other datasets, or users) at inference time.
6

---

AuPair: Golden Example Pairs for Code Repair
Out-of-distributioo Generalisation Performance
0.6
0.5
Test pass rate
0.3
0.1
0.0
AcCoder
LiveCodeBench
HackerEarth
Codejam
CodeChef
Aizu
Initial
Best-of-N
Self-repair
CodeForces AuPairr
Figure Out-of-distributioo code repair performance: AuPairs extracted on the CodeForces dataset show strong generalisation performance across the other six datasets (model: Gemini-1.5-PP metric: test pass rate).
Quality vs. Diversity
Gemma-98
3.2
0.18
0.17
0.188
0.45
Repair model
Gemma-278
0.19
0.21
0.19
0.21
0.35
0.34
0.35
0.35
0.35
0.30
Gemini-115-Pro
0.32
0.35
0.35
0.44
0.20
Gemma-90 90
Gomma-270
Gemini-1 Flash
Gemini-1 1.50mi11.... 1.5-Pro
0.15
Source of AuPairs
0.02
0.04
0.06
0.08
0.10
0.12
Diversity
Figure 7: Cross-model transfer: AuPair shows good cross-
98
1.mmnn-1.5-Flaas Gemini-1.5-Po
AuPair
Gemma-278
model transfer capabilities on CodeForces.
Figure 8: Diversity-Score plot: AuPair (*) generates more diverse responses than best-of-NN (O) for Gemini-1.5-Flaa and -Pro; this trend reversed for Gemma models. AuPair always generates higher-scoring fixes than best-of-N.
3.6. Better scaling with validation data
We conduct experiments with smaller validation sets cu- rate AuPairs and report the results in Table 1. The takeaway is that the larger the validation set, the more distinct com- plementary improvements can be observed, and hence the larger the maximal set of AuPairs that can be discovered. So, larger validation sets make it possible to effectively scale up to more inference compute. However, even just looking the top 32 AuPairs (which
and the guess AST, and normalise by the maximum number of subtrees. We plot this diversity metric against the score to gauge how diverse and useful the AuPairs are, compared to best-of-N, in Fig. 8. $A.9 contains details on the diversity score computation. The results show that while AuPairs always increase performance, they yield more diverse fixes when given to the more competent models (Gemini- 1.5-Pro and -Flash), and less diverse fixes for the Gemma models. The superior performance of AuPairs produced by Gemini- 1.5-Pro corresponds to highly diverse fixes (Fig. 8, top right).
fair comparison for vary- ing validation sizes), we find that their quality increases monotonically with the size of the validation set.
3.7. High Code-specific Diversity
We use Abstract Syntax Trees (ASTs) to study the fixes generated using AuPairs. Since there are N fixes for each problem as the number of unique changes made to the guess over all N fixes for that problem. The diversity score calculated as the average number of unique ASTs generated per problem. We perform the set difference all subtrees in the fix AST
3.8. High Repair Diversity
problem (N = 32), we measure the diversity
In addition to code diversity, we also analyse the repair di- versity of the solutions generated by AuPair compared to the baselines. In this, we report the following values: 1) percentage of problems in which the code was reformatted

---

AuPair: Golden Example Pairs for Code Repair
Validation set size
# of AuPairs
Test pass rate
test pass rate, i.e., the increase in this score achieved by AuPair CodeForces. The key observations are (a) AuPair helps significantly at all difficulty levels across mod- els, and (b) there are larger improvements on easier levels; this trend is consistent across models. Note that the initial performance of Gemini- 1.5-PPr is low because the initial guesses do not adhere to the instruction (elaborated in Ap- pendix $A.11); however since this the strongest model and has the best overall performance, the increases in score are significantly higher than other models.
Random
N/A
0.383
10%
32
0.403
25%
52
0.418
100%
144
0.438
Table I: Scaling with validation data: We see an increase in 1) the number of AuPairs discovered, and 2) performance using the top 32 AuPairs on scaling validation data.
problems with formetting composite
3.10. Using AuPairs may lead to worse responses
50
Best-ehi
40-
Selt-paat
While AuPairs have been shown to significantly boost perfor- mance in best-of-N setting, they occasionally generate worse repaired response compared to the initial response in some cases. Table contains the percentage of CodeForces problems in which some fixes were worse than their initial guesses. Note that the performance gains shown earlier still hold, since for measuring performance, the best scoring response is selected. As can see from Table 3, in most cases, using AuPair results in increase in the number of problems for which fix is worse than the initial guess. This is to be expected since AuPair is an algorithm that in addition to boosting performance also boosts diversity of the generated responses.
10
% problems with logical fixes
15
20
% problems with unchenged fix score
103
4. Related Work
Locrangges
Automatic Program Repair (APR) has been longstanding area in machine learning (Devlin et al., 2017; Bhatia & Singh, 2016; Chen et al., 2019; Feng et al., 2020; Berabi et al., 2021; Chakraborty et al.. 2022: Yuan et al., 2022). Most methods rely on supervised fine-tuning (SFT) to adapt LLMs the task of code generation using labeled pairs of broken / fixed code pairs, which is costly obtain and often task- and problem-specific (Hu et al., 2022; Jiang et al., 2021; Xia & Zhang, 2022; Dinella et al., 2020). On the other hand. unsupervised APR is challenging since it requires syntactic and semantic understanding of code, and most automatic code breaking approaches tend to be out-of- distribution with real samples. Yasunaga & Liang (2021) train breaker and fixer to learn to propose code fixes that realistic, and compiler verify correctness. Our work uses imitial broken code to perform repair.
Figure 9: Repair Diversity across models and approaches: In most cases, AuPair outperforms baselines by performing a diverse set of fixes like formatting, composite, logical fixes. This diversity also reflected in the lower number of problems where the fix is equivalent to the guess.
to increase the score by obeying the formatting constraints described before (higher is better), 2) percentage of prob- lems in which the fix generated an improved solution by just changing the logic (higher is better), and 3) percentage of problems in which the fix score remained unchanged with respect to the guess (lower is better). Note that fixes that involve multiple changes, such as formatting changes and logical changes, are counted under formatting fixes. Fig. 9 provides a visual illustration of these results.
partial or complete fixes on
More recently, a few unsupervised approaches have been proposed based on the capability of L.MS to generate code (Chen al., 2021; Nijkamp et al., 2023; Chowdh- ery et al., 2024; Li al., 2022a; Fried al., 2023; Li et al., 2023). APR still remains challenging, are better at generating code (Olausson et al., 2024; Chen et al., 2023). Zhao et al. (2024) use a step-by-step method to repair code using a reward model as a critic, providing
3.9. Improvement on All Difficulty Levels
Coding datasets have heterogeneous difficulty. We con- duct additional analysis to determine which problem levels are most helped by AuPair, compared to the quality of ini- tial guesses. Table 2 shows the absolute improvement in
though models
8

---

AuPair: Golden Example Pairs for Code Repair
Difficulty level
A (671)
B (675)
(
(666)
(649)
F+ (537)
Gemma-9B
0.34 (40.16)
0.23 (+0.13)
0.19 (2)
0.15 (+0.09)
0.14 (+0.08)
0.12 (+0.07)
Gemma-211
0.28 (+0.1)
0.25 (+0.12)
0.20 (+0. 12)
0.19 (+0.1)
0.17 (+0.1)
0.20 (+0.11)
Gemini-1.5 1,5--llss
0.54 (+0.2)
0.39 (+0.18)
0.34 (+0.15) 15)
0.18 (+0.11)
0.26 (+0.12)
0.28 (=0.11)
Gemini-1.5-PP
0.62 (+0.42)
0.52 (+0.4)
0.43 (40.35)
0.38 (+0.32)
0.32 (+0.28)
0.35 (+0.29)
Table 2: Difficulty-wise analysis: test pass rate using AuPairs, categorised by difficulty level from easy (A) hard (F+), accompanied by number of problems, absolute improvement in parentheses.
Model
Approach
% problems
ability to generated better code may be connected to the ability to identify its faulty behaviour. Welleck et al. (2023) decouple generation and correction by independently train- ing a corrector with scalar and natural language feedback to correct intermediate imperfect generations. Ding et al. (2024) teach code-competent models self-refine by factor- ing in the execution results from previous incorrect code and updating the pretrained model using this feedback. Tang et al. (2024) provide stochastic approach for choosing which code node to expand by using a Thommsso sampling approach; they leverage an exploitation-exploaae nism for evolving the tree of code refinements.
Gemini-1.5-Pro
Best-of-N
10.52
Self-repair
7.62
AuPair
11.63
GPT-40-mini
Best-of-N
20.09
Self-repair
11.87
AuPair
15.28
Gemini-1,.5Flass
Best-of-N
9.47
Self-repair
22.28
AuPair
11.79
mecha-
Gemma-27B
Best-of-N
14.86
Self-repair
9.72
AuPair
15.21
Yuan & Banzhaf (2017) propose a multi-objective evolution- ary algorithm search over possible correct code patches; Romera-Parede et al. (2023) use an island-based evolution- ary method to encourage exploration of diverse programs, and perform iterative best-shot prompting to improve the quality of the generated code. We use a generative approach; closer to the work of Shirafuji et al. (2023), we make use of ICL abilities of LLMs generate improved code repairs, but we provide an extra submodular process to select the samples, encouraging diversity.
Gemma-9B
Best-of-N 13.16
Self-repair 9.38
AuPair 13.09
Table 3: Failure analysis: increase in the % of problems in which using AuPairs resulted in a lower-scoring fix.
feedback to finetune an LLM. Shypula et al. (2024) propose a retrieval based few-shot prompting approach with Chain- of Thought (CoT) reasoning traces, and use SFT to finetune model using self-play. The main disadvantage of using SFT comes from the need to finetune the model to the task, which becomes more costly with increasing model sizes. In recent years, in-context learning (ICL) (Brown et al., 2020) has emerged as a flexible and compute-efficien adaptation approach to new tasks (Von Oswald et al., 2023; Akyürek et al., 2023). Le et al. (2022) use an LLM to generate code and critic predict functional correctness of the gener- ated program with zero-shot transfer new tasks. Gou et al. (2024) employ tool use to provide feedback for the LLM to self-correct via additional calls to evaluate its own output. Wang et al. (2023a) also make use of external tools and use an LLM in a learner teacher role to provide a chain of repairs to fix the code.
5. Conclusions and Future Work
We propose an algorithm, AuPair, which produces an or- dered set of golden example pairs each of which can be provided an in-context example using 1-shot prompting with an inference compute budget of N LLM calls to im- prove code repair performance at inference time. AuPair is highly scalable, showing significantly better outcomes than best-of-N and self-repair, both of which are known to improve performance as inference compute is scaled up. In addition this, the AuPairs generated using our algorithm show strong out-of-disttibution generalisation and thus can be reused inference time solve wide range of prob- lems. While in this paper we have explored repair in the coding domain, future work can look at using it in other settings in which an initial solution generated by an LLM can be improved via repair. Additionally, the choice of coding implies that all our feedback grounded, but us- ing ungrounded feedback from reward models to build the AuPairs might be another direction worth exploring.
Yin et al. (2024) propose an automated self-repair approach with fhw-shot prompting using CoT and execution feedback. Agarwal et al. (2024) also use CoT rationales but remove them from context when using few-shot prompting. Olaus- son et al. (2024) show that using an LLM as a feedback source for self repair has limitations when compared with independent model calls for the same problem since the

---

AuPair: Golden Example Pairs for Code Repair
Acknowledgements
Chakraborty, S., Ding, Y., Allamanis, M., and Ray, B. Codit: Code editing with tree-based neural models. IEEE Transactions Software Engineering, 48(4): 1385-13988199 April 2022. ISSN 2326-3881. doi: 10. 109/tte.2000.. 3020502. URL. http: : wwd..do..oooo g 10/0 1/19/ 1109/ TSE. 2020. 3020502..
The authors would like to thank Dan Calian for valuable feedback and suggestions on the paper, David Silver for his sponsorship: Wojtek Czarnecki, Kate Baumli, Jakub Sygnowski, Victor Cärbune, Volodymyr Mnih, Mina Khan, Georg Ostrovski, Shantanu Thakoor, Lei Zhang, Disha Shri- vastava, Feryal Behbahani, the RL team, and the wider DeepMind community for helpful discussions.
Chen, M., Tworek, J., Jun, H., Yuan, Q. Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03:: 2021.
Impact Statement
The goal of our paper is to advance capabilities of LLMs in the domain of code repair. This work may have potential societal consequences, which we do not feel the need to highlight here.
Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023.
Chen, Z., Kommrusch, S., Tufano, M., Pouchet, L., Poshy- vanyk, D., and Monperrus, M. Sequencer: Sequence-to- sequence learning for end-to-end program repair. CoRR, abs/1901.01808, 2019. URL abs/1901.01880
References
Agarwal, R., Singh, A., Zhang, L. M., Bohnet, B., Rosias, L., Chan, S., Zhang, B., Anand, A., Abbas, 7., Nova, A., Co-Reyes, J. D., Chu, E., Behbahani, F. Faust, A., and Larochelle, H. Many-shot in-context learning, 2024. URL utthttss: /ttp://axxvv..//o orgo www.ooggg//////oo.ooo.oooeooesss /000////44444 11018.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B.. Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fedus, L., Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, S., Pel- lat, M., Lewkowycz, A., Moreira, E., Child, R., Polozov, O., Lee, K., Zhou, Z., Wang, X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei. J., Meier-Hellstern, K., Eck, D., Dean, J., Petrov, S., and Fiedel, N. Palm: scaling language modeling with pathways. J. Mach. Learn. Res., 24(1), March 2024. ISSN 1532-4435.
Akyürek, E., Schuurmans, D., Andreas, J., Ma. T., and Zhou, D. What learning algorithm in-context learn- ing? investigations with linear models. In The Eleventh International Conference on Leurning Representations, / http///ppeerevveeeeee net/ forum?
2023. URL https: id=0goX4HByN4I.
Berabi, B., He, J., Raychev, V., and Vechev, M. Tlix: Learning to fix coding errors with a text-to-text trans- former. In Meila, M. and Zhang, T. (eds.), Pro- ceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 780-791. PMLR, 18-24 Jul 2021. URL. https:/ htttttt:s:::p/eeeeeeeeeeee http://pooccedeeeee mlr..mmr....... wwr....ss././/...m v139 1139/berabi2la. html.
Bhatia, S. and Singh, Automated correction for syntax errors in programming assignments using recurrent neural networks. CoRR, abs/1603.06112 2016. URI. http: //arxiv..o org. abs/ /ass//663.000 06129.
Devlin, J., Uesato, J., Singh, R., and Kohli, P. Seman- tic code repair using neuro-symbolic transformation net- works. CoRR, abs/1710.110544 2017. URL http:
//arxiv.org/abs /atrg//bb///11... .11514
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding, 2019. URL hwtps:t //axxiv. org/ abs /1100. 04805.
Askell, A., Agarwal, S.. Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. CoRR, abs/2005.14165, 2020. URL https:://///. /ttpss://axxvv.rrr// 201.2/0001..155
Dinella, E., Dai, H., Li, Z.,, Naik, M., Song, L., and Wang, K. Hoppity: Learning graph transformations to ddetect and fix bugs programs. In International Conference on Learning Representations, 2020. URL https: openreview. wwt.:/fooonnndddddsssssee.
10

---

AuPair: Golden Example Pairs for Code Repair
Ding, Y., Min, M. J., Kaiser, G., and Ray, B. Cycle: Learning self-refine the code generation, 2024, URL https: / /arxiv.vvvv org/ g/ /bs/ 2403. 18746.
Le, H., Wang, Y., Gotmare, A. D., Savarese, S., and Hoi, S. C. H. Coderl: Mastering code generation through pre- trained models and deep reinforcement learning, In Ad- vances in Neural Information Processing Systems, 2022.
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M.. Shou, L., Qin, B., Liu, T., Jiang, D., and Zhou, M. CodeBERT: A pre-trained model for programming and natural languages. In Cohn, T., He, Y., and Liu, Y. (eds.), Findings of the Association for Computa- tional Linguistics: EMNLP 2020, pp. 1536-1547, On- line, November 2020. Association for Computational Linguistics. doi: 1..18663/11/20000 finpe/..nnnnigs-mmmny 139. URL https:://///oo / ac lanthology org/2020. findings-emnly 139.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., tau Yih, W., Rock- täschel, T., Riedel, S., and Kiela, D. Retrieval-augmente generation for knowledge-intensive nip tasks, 2021. URL. https: https://arxii.orr org
wwtprriw.v...r//////e/p wew.g////////oooo///// 11401.
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C.,, J., Chim, J., Liu, Q. Zheltonozhskii, E., Zhuo, T. Y., Wang. T., Dehaene, O., Davaadorj, M., Lamy- Poirier, J., Monteiro, J., Shliazhko, O., Gontier, N., Meade, N., Zebaze, A., Yee, M.-H., Uma- pathi, L. K., Zhu, J., Lipkin, B., Oblokulov, M.. Wang, Z. Murthy, R., Stillerman, J., Patel, S. S., Abulkhanov, D., Zocca, M., Dey, M., Zhang, Z., Fahmy,, N., Bhat- tacharyya, U., Yu, W., Singh, S., Luccioni, S., Villegas, P., Kunakov, M., Zhdanov, F., Romero, M., Lee, T., Timor, N., Ding. J., Schlesinger, C., Schoelkopf, H., Ebert, J., Dao, T., Mishra, M., Gu, A., Robinson, J., Anderson, C. J., Dolan-Gavitt, B., Contractor, D., Reddy, S., Fried, D.. Bahdanau, D., Jernite, Y., Ferrandis, C. M., Hughes, S.. Wolf, T., Guha, A., von Werra, L.. and de Vries, H. Starcoder: may the source be with you!, 2023. URL https: //tp://aaxivv../ .rg aaabs/2/300 06161.
Fried, D., Aghajanyan, A., Lin, J., Wang. S., Wallace, E., Shi, F., Zhong, R., Yih, W.-t,, Zettlemoyer, L., and Lewis, M. Incoder: generative model for code infilling and synthesis. In International Conference on Learning Rep- resentations, 2023.
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., and Wang, H. Retrieval-augmented generation for large language models: A survey, 2024. URL hwtpstpp://// //tp://arxxt /araiv.vvvv// www.ltv..rggbbb/b/e//e/ wtw.or/////......////
Gou, 7.,, Shao, Z., Gong, Y., yelong shen, Yang, Y., Duan, N., and Chen, W. CRITIC: Large language models can self-correct with tool-interactive critiquing. In The Twelfth International Conference Learning Represen- tations, 2024. URL https: //pp://vee//////ee//eeemeeee net/ forum? i dxd3xxx0xxxje..
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eceles, T., Keeling, J., Gimeno, F. Dal Lago, A., Hubert, T., Choy, P., de Masson d' Autume, C., Babuschkin, I., Chen, X., Huang, P.S.,, Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J.. Suther- land Robson, E., Kohli, P., de Freitas, N., Kavukcuoglu, K., and Vinyals, O. Competition-levell code generation with alphacode. Science, 378(6624) ber 2022a. ISSN 1095-9203. URL http: //dx /dx doi. abq1158.
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., and Steinhardt, J. Measuring coding challenge competence with apps. In Advances in Neural Information Processing Systems, 2021.
Decem-
ww1//1scciiiii org 10.. org/10.
Hu, Y. Shi, X., Zhou, Q., and Pike, L. Fix bugs with trans- former through neural-symbolic edit grammar, 2022. URL
Li, Y. Choi, D., Chung, J., Kushman, N., Schrit- twieser, J., Leblond, R. Eccles, T. Keeling, J., Gi- meno, F., Lago, A. D., Hubert, T., Choy, P., de Mas- son d' A'ttum, C., Babuschkin, I., Chen, X., Huang, P.-.,, Welbl, J., Gowal, S., Cherepanov, A., Mol- loy, J., Mankowitz, D. J., Robson, E. S., Kohli, P., de Freitas, N., Kavukcuoglu, K., and Vinyals, O. Competition-level code generation with alphacode. Sci- ence, 10966664):002-10999 2022b. doi: 10.1 abence.abbl118.. URL https: //tpp:::://wwwee http///wwcieec.... org/doi/ /bss 100 111.1.12110 wwt.//cciece..aagl.1...
absp/////////////////////// 06643.
Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F,, Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free eval- uation of large language models for code. arXiv preprint arXiv:2403.07974 2024.
Jiang, N., Lutellier, T., and Tan, L. Cure:
neural machine translation for automatic program repair. In 2021 JEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, May 2021. doi: 10.1 icse43900..0211.00000 URL http: /dxx .og. 292.
Nijkamp, E.,, Pang, B., Hayashi, H., Tu, L., Wang. H., Zhou, Y., Savarese, S., and Xiong, C. Codegen: An open large language model for code with multi-turn program synthe-
10. 100.109/ICEE439222220..0000

---

AuPair: Golden Example Pairs for Code Repair
sis. In International Conference on Learning Representa- tions, 2023.
Wang, H., Liu, Z.,, Wang, S., Cui, G., Ding, N., Liu, Z.,, and Yu, G. Intervenor: Prompt the coding ability of large language models with the interactive chain of repairing. CoRR, abs/2311.09868, 2023a. URL www.bbpp.ui-ttittee. de/pee//ddd////onaal//// hww.:htstsss-a........m
Olausson, X., Inala, J. P., Wang, C., Gao, J., and Solar- Lezama, A. Is self-repair silver bullet for code genera- tion? In International Conference on Learning Represen- tations, 2024.
Wang. X., Wei, J., Schuurmans, D., Le. Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency im- proves chain of thought reasoning in language mod- els, 2023b. Utthttps: /ttp/aaavvv////////////////aaa gorg/bbs/2000 11171.
Ouyang, S., Zhang. J. M., Harman, M., and Wang. M. An empirical study of the non-determinism of chatgpt in code generation. ACM Trans. Softw. Eng. Methodol.. Septem- ber 2024. ISSN 1049-331X. doi: 10. 145/3697010. URL Just Ac-
https: / /ooi org/ / 10. 1145 / 369 3691000 cepted.
Welleck, S., Lu, X., West, P., Brahman, F,, Shen, T. Khashabi, D., and Choi, Y. Generating sequences by learning self-correct. In The Eleventh In- ternational Conference Learning Representations, 2023. URL https: http::://pppeee http://pppeeeeeeeeeeeeee uorr/orr id=hH36JeQZDaQ.
Renze, M. and Guven, E. The effect of sampling tempera- ture on problem solving in large language models, 2024.
htts: / aa//v.xxvvvv........ org/ abs /20...24....0 05201.
Romera-Paredes, B., Barekatain, M., Novikov, A., Ba- log, M., Kumar, M. P., Dupont, E., Ruiz, F. J. R., El- lenberg, J. S., Wang, P., Fawzi, O., Kohli, P., Fawzi, A., Grochow, J., Lodi, A., Mouret, J.-B., Ringer, T., and Yu, T. Mathematical discoveries from program search with large language models. Nature, 625:468 - 475, 2023. URL https: https://www.ntt http://wwture.cc.mmm articles/s41585 020-00232220 06924-6
Wu, J. J. and Fard, F. H. Benchmarking the communication competence of code generation for Ilms and Ilm agent. arXiv preprint 2rXiv:2406.002// 2024.
Xia, C. S. and Zhang, L. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, pp. 959-971, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 978 7781450394130. doi: 10. 1145/5540250.5549101. URI. https: htpps://doff //o////o.../////// org/ 10. N0.1145/35402555 3549101.
Shirafuji, A., Oda, Y., Suzuki, J., Morishita, M., and Watanobe, Y. Refactoring programs using large lan- guage models with few-shot examples. In 2023 30th Asia-Pacific Software Engineering Conference (APSEC). IEEE, December 2023. doi: 10. 1009//psc660848.22222 00025. URL http: dx. doi org 10. 1109/ APSEC60848 20025
Yasunaga, M. and Liang, P. Break-it-fix-it: Unsupervised learning for program repair. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11941-11952 PMLR, 18-24 24 Jul 2021. URL htRl:: /ppcee/ddgs./oeeeenngngg.. mmr. press / v139//19////// wat.:///yyuunagaaa..a. html.
Shypula, A., Madaan, A., Zeng, Y. Alon, U. Gardner, J., Hashemi, M., Neubig, G., Rangamathan. P., Bastani, O., and Yazdanbakhsh, A. Learning performance-improoingg code edits, 2024. URL https:: http::///axxvv.orre/ aab/ 2302. 07867.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize from human feedback. CoRR, abs/2009.01325 2020. URL https: abs/2009.0133
Yin, X., Ni, C., Wang, S., Li, Z., Zeng, L., and Yang, X. Thinkrepair: Self-directed automated program repair. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, pp. 1274-1286, New York, NY, USA, 2024. Asso- ciation for Computing Machinery. ISBN 97984007061177. doi: 10. 1411145552222366000000 org/ / 10.
Tang, H., Hu, K., Zhou, J. P., Zhong, S., Zheng, W.-L.,, Si, X., and Ellis, K. Code repair with llms gives exploration-exxloitation tradeoff, 2024. URL. https: // //arxiv.orgg aww.a///........ 17503.
URL https:///oii. 3686359..
Yuan, W., Zhang, Q., He, T., Fang, C., Hung, N. Q. V., Hao, X., and Yin, H. Circle: continual repair across programming languages. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. ISSTA 2022, pp. 678-690, New York, NY, USA, 2022. Association for Computing
Von Oswald, J., Niklasson, F., Randazzo, F., Sacramento, J. a., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M. Transformers learn in-context by gradient descent. In Proceedings of the 40th International Conference on Machine Learning, ICMI.'2. JMLR, 2023.
12

---

AuPair: Golden Example Pairs for Code Repair
Machinery. ISBN 9781450393799. doi: 10.1145/ 3533767.3534219. URL https://doi.oom orgg/1.org/... 1145/3533767 3534219.
Yuan, Y. and Banzhaf, W. Arja: Automated repair of java programs via multi-objective genetic programming. IEEE Transactions Software Engineering, 46: 46:1000-1067,
2017. URL https: / api. semanticscholar. org/ / /CoppusSD::5222211
Zhao, Y., Huang, Z., Ma, Y., Li, R., Zhang, K., Jiang, H., Liu, Q. Zhu, L., and Su, Y. RePair: Automated pro- gram repair with process-based based feedback. In Findings of the Association for Computational Linguistics ACL 2024. Association for Computational Linguistics, August 2024. URL https: //aclanntoooogyy org/ 2024.
aindings-acl.999 .733.
13

---

AuPair: Golden Example Pairs for Code Repair
A. Appendix
Scaling inference compute
Initial guess
A.1. Pair Generation
Best-af-N
AuPair
In this section, we discuss the specifics of the pair genera- tion phase and provide results pertaining this phase. The approach that we for pair generation is provided in Al- gorithm 3. Note that this is way to generate pairs; they can be generated in other ways, or be available beforehand. Studying the impact of using pre rre-generated pairs for ex- tracting AuPairs could be an interesting avenue for future work.
Selfirepair
Test pass rate rate
0.30
0.20
We
in this algorithm. The reason for this is that during pair generation, want diverse pairs be generated, and using different set of k examples with the same problem could give us different fixes.
100
10: Inference compute budget N
102
Figure 10: Scaling up inference compute the CodeForces dataset with Gemini-1.55 1.5--lash. Scores correspond to aver- age pass test rate on all the test problems.
For the AtCoder dataset, we set a budget of 10,000 LLM calls for pair generation. Since the CodeForces dataset is larger, we set budget of 35,000 LLM calls to maintain a good balance between having enough LLM calls per prob- lem and maintaining the affordability of the overall approach in terms of computational resources. We report the number of pairs generated both of these datasets across all 5 models: Gemini-1.5-PP GPT-4o-mini, Gemini- Mimnini-1.5-Flase 1.5-Flash, Gemma-27B. 27B, and Gemma-9B in Table 4. Here we provide some additional results that we were unable to include in the main text.
ITy II 1 L(eva peovallcoeep.... Tp.) == pass] j=1
Σ max P (L..NN
where Tp refers to the unit tests for problem p, and codep.i is the code generated by the LLM for problem p the 7t LLM call. Here, the innermost loop. like test pass rate, computes the percentage of unit tests passed by the LLM output codep.i. Following this, select the code output that passes all tests (max over binary values yields if any such output exists, otherwise O). The outermost loop averages this across all the problems in the test dataset.
CodeForces
# Pairs
# AuPairs
Gemini-1.5-Prrs
1560
144
GPT-4o-mini
1192
94
Gemini- 1.5-Flash
1327
110
Gemma-27B
509
77
Gemma-9B
556
122
We see that AuPair outperforms all other baselines on all models across the board, with results for CodeForces and AtCoder shown in Fig. 11.
AtCoder
# Pairs
AuPairs
Gemini-1.5-PP5
927
64
We also show the results for sation on this strict accuracy metric in Fig. 12; again, the results clearly indicate that AuPair outperforms all baselines on this metric as well across all datasets.
generali-
GPT-4o-mini
378
15
Semini-115--laan
397
64
Gemma-27B
295
27
Gemma-9B
147
14
A.3. Scaling Inference Compute
Table 4: Number of pairs collected during phase of the algorithm (# of pairs) and number of AuPairs extracted in phase 2 (# AuPairs): CodeForces (left) and AtCoder (right) for all models.
In addition to the scaling experiment we performed using Gemini- 1.5-Pro (results in Fig. 5(b), we also perform same scaling experiment using Gemini-1.5- the results Fig. 10. Moreover, the same scaling experiment the strict accuracy metric in Fig. 13. The trend is similar to what we observed before: best-of-N plateaus after a certain number of LLM calls, while our approach scales as the compute budget increases, delivering an improvement in performance for each newly included AuPair. The self-repair baseline performs better
and show report the results of
A.2. Measuring correctness in terms of solved problems
In addition to pass rate of unit tests, we also report the percentage of fully solved problems, for which the generated code passes all cases. This the strict accuracy metric:
14

---

AuPair: Golden Example Pairs for Code Repair
Algorithm 3 Pair Generation
LLM
large language model training dataset
D Drain
Require:
number of few-shot examples
N
total number of LLM calls
score
code eval function
1: init candidate pairs C 0
2: for i NN do
3: sample problem from dataset: z ~ Dirain
4: sample pairs to use in-context: C1,..., Ck ~ C
5: build k-shot prompt: p 4 C1 ||... Ck 1
6: generate fix: 9 < (LLM
7: evaluate fix: 8g < score (j)
8:
if 89 8. then
9:
create pair: c (a, 0) add to candidate pairs: C + CUc
10:
11:
if Sy then
12:
create new problem with guess add new problem to dataset: Duain +- Duain U else
13:
14:
remove all instances of problem from dataset: Duain Derain {x} end if
16:
17: end if
18:
end for
return
CodeForces score 32
AtCoder score @ 32
Initial
0.30
Best-of-N
Self-repair
0.25
AuPair
0.15
0.10
0.1
0.05
0.0
Gemma-98
Gemma-278
Gemini-1. 5-FFlsh 5-Flash
GPT-4o-mini
Gemini-1.5-PP
Gemma-9B
Gemma-278 -27
Gemini-1.5 1emiai-1.5-Flash 5-Fssh
GPT-4o-mini
Gemini-1.5-PP Pro
Figure 1I: In-distribution code repair performance for the strict accuracy metric with 32 LLM calls inference time. CodeForces (left) and AtCoder (right).
15

---

AuPair: Golden Example Pairs for Code Repair
Out-of-distributioo Generalisation Performance
0.6
0.5
Strict accuracy
0.4
0.3
0.2
0.1
0.0
AtCoder
LiveCodeBench
HackerEarth
Codejam
CodeChef
Aizu
Initial
Best-of-N
Self-repair
Codeforces AuPairs
Figure 12: Out-of-distribution code repair performance for the strict accuracy metric: CodeForces dataset show strong generalisation performance across the other six datasets (the above results are obtained using Gemini-1.5-Pro
AuPairs extracted on the
with the Gemini- 1omini--..5-Fase 1.5-Flash model than with the Pro model; hypothesis is that since the initial guesses for the Pro model were worse because of formatting issues, self-repair did not yield significant improvements. However, when the initial guesses are better. the self-repair baseline shows stronger scaling result. Our algorithm yields 110 AuPairs and achieves a final test pass rate 37.83% and strict ac- curacy 24.1 14%. Best-of-N,, on the other hand, given the same budget of 110 L.M calls, has test pass rate of 21.8% and strict accuracy 11.93%. Self-repair with the same pute budget has final test rate of 34.1 1 and strict accuracy 22.39%. Since our AuPairs are selected submodu- larly, the initial pairs yield high returns performance and these returns start diminishing slowly, but notably, perfor- mance does not plateau yet. Thus, it is abundantly clear that using AuPairs has distinct advantage over currently used approaches like best-of-N and self-repair in improving performance at inference time as compute budget increases.
Approach
Test rate
Initial
0.244
Best-of-N
0.100
Self-repair
0.374
Gemini-1.5-Pro AuPairs
0.486
GPT-4o-mini AuPairs
0.573
Table 5: Transfer to GPT-40: We see strong cross-model transfer using GPT-4o-mini and Gemini-1.5-Pro AuPairs with GPT-4o, with 11% and 20% absolute performance improvement respectively over the strongest baseline.
A.5. AuPair v/s RAG
A.4. Cross-model Transfer to GPT-40
Our approach yields a fixed ordered of examples, of which the first N examples used at inference time, de- pending on the inference compute budget N. Retrieval- Augmented Generation (RAG) techniques (Lewis et al., 2021; Gao et al., 2024), on the other hand, retrieve examples that are closest proximity to a particular test problem in the embedding space. As result, for each test problem, different set of N examples retrieved. Table shows comparison between RAG and AuPair for inference pute budget N = 32 on the CodeForces dataset. The RAG baseline retrieves the top N examples from the candidate set of all pairs (cardinality of each set of candidate pairs per model given in Table 4). We use BERT (Devlin et al., 2019) embeddings to retrieve the top pairs.
In this section, study the ability of the model gener- alise to GPT-4o. We report results using AuPairs collected on GPT-4o-mini, since it is a weaker model than GPT-4o but from the same model family, and Gemini-1.5-Pro since is the strongest model from a different model family. The results in Table validate our previous finding that in spite of using AuPairs from other models, there are significant performance gains (1 11% and 20% absolute performance gains) over the strongest baseline.
16

---

AuPair: Golden Example Pairs for Code Repair
Scaling inference compute: Gemini-1.5-4
Scaling inference compute: 1omini-1.33-lashh
0.40
Initial quess
0.250
Initial quess
Best-ofNN
Best-of-N
0.35
AuPair
0.225
AuPair
Seftrepair
Self-ropalr
0.30
0.200
Strict accuracy
Strict accuracy
0.25
0.1.5
0.20
0.15
0.100
0.05
0.075
10a
10a
10a
Inference budget
Inference compute budget
Figure 13: Striet accuracy when scaling inference-time compute: with N 144 for Gemini-1.5-Pro and N 110 for Gemini-1.5-Flash
Model
RAG score
AuPair score
AuPairs and out-of-distribution CodeForces AuPairs on the AtCoder dataset.
Gemini-1.5-Pro
0.379
0.438
GPT-44-mini
0.361
0.378
Another interesting result in Fig. 14 is that both metrics, the test pass rate and strict accuracy, are comparable when using in-distribution AuPairs from LiveCodeBench and out- of-distribution AuPairs from CodeForces. This reinforces the insight mentioned earlier that extracting AuPairs on one dataset could lead to significant improvements over baselines even on other datasets.
Gemini-1 1.5-Flash
0.318
0.352
Gemma-271 0.178
0.214
Gemma-99
0.156
0.198
Table 6:
Comparison with RAG: We do an apples-to- apples comparison between AuPair and RAG the Code- Forces dataset, and the results conclusively show that AuPair outperforms RAG across models.
A.7. Coverage of Problem Categories is Preserved
The CodeForces dataset is richly annotated with category labels for each problem. A problem may have multiple tags, for instance, str strings ings and two pointers. We use these fine-grained tags to study how the problem distribution is affected by Phase and Phase 2 of our method separately. Fig. 16 shows the proportions of these categories observed in the initial dataset, the full set of pairs generated during Phase 1, and the final AuPairs. The high-level result is encouraging, namely that the starting diversity is approximately preserved. Phase yields pairs for every single category, even those that lie at the tail. Furthermore, the (sparser) distribution over categories for the AuPairs after Phase still shows several problems from rare categories. This additional result consolidates insight that AuPairs are highly diverse in the types of problems they contain.
A.6. Code Repair with LiveCodeBench
Generalisation of AuPair prompting is important improve code repair of smaller datasets. We posit that the AuPairs contain diverse code changes that transfer meaningfully across datasets, which may be important to those with scarce data, since out-o--distribution generalisation becomes espe- cially relevant when we have small datasets, on which it can be quite difficult obtain many different AuPairs.
We now show some the results obtained for smaller dataset (400 problems) LiveCodeBench (LCB) (Jain et al., 2024). We generate the same train/val/test split (37.5/12.5/50%) over 400 problems and apply our AuPair approach to obtain in distribution AuPairs for LCB.
Fig. 14 shows that even with smaller number of selected AuPairs we still obtain gain over best-of-N prompting. We obtained AuPairs with the submodular extraction in Algorithm 2 for all the models except Gemma-9B which ob- tained only 3 AuPairs. The results indicate that performance with CodeForces AuPairs matches or exceeds that of us- ing in-distribution AuPairs from LiveCodeBench. We also observe similar performance for in-distribution AtCoder
A.8. Lineage
Here look the lineage of each pair generated during phase of our algorithm, pair generation. The key idea here is to see the set of all pairs collected during the pair generation phase are deeper i.e., they generate iteratively better solutions for a smaller set of problems, or broader i.e., they generate solutions for a larger set problems but
17

---

AuPair: Golden Example Pairs for Code Repair
LiveCodeBench: pass
LiveCodeBench: strict accuracy
0.35
Initial
Best-ofNN
0.25
AuPair
0.25
CodeForces aupairs
0.20
0.20
0.15
0.15
0.10
0.10
0.05
0.05
0.00
Gemma-98 9B
Gemma-27B
Gemini-1 Gemini-1.5-Flaase 5-Fllshh
Gemini-1.5-Pro
Gemma-96 988
Gemma-278 278
Gemini-1 Gemini-1.5-Flash 1-5-flash
Gemini-1.5-PM
Figure 14: LiveCodeBench results: using AuPairs from the CodeForces dataset matches or outperforms in in-distribution AaPai from LiveCodeBench (left: test pass rate, right: strict accuracy).
1290
Soee off
Ster
Scer
1000
Figure 15; Visualising the lineage of the of all pairs as the first phase of the algorithm, pair generation, progresses.
18

---

AuPair: Golden Example Pairs for Code Repair
Distribution of problems across categories in CodeForces
graphs
geometry
matrices
All pairs
sortings
*special
graph matchings
AuFairs
dfs and similer
dsu
string suffix structures
implernentatinn
binary search
probabilities
30
shortest paths
expression persing
greedy
strings
divide and conquer
ternary search
dp
number theary
games
meet-in-the middee
% questions
date structures
combinatorics
hashing
2-sat
two pointers
interactive
remainder
constructive algoritmms
bitmasks
floss
schedules
15
Figure 16: Category-wise analysis: analysing the distribution of AuPairs across different categories and comparing it with the distribution of problems in the dataset.
those solutions may not necessarily be perfect. The last plot in Fig. 15 (pairs generated on the CodeForces dataset us- ing Gemini-Pro-I. 1.5) indicates
Prompting Strategies
the pairs collected have shallow lineage: a large proportion of guesses that had score of had corresponding fixes with perfect score at depth 1. We also that the number of fixes decreases depth increases (as seen from the size of the circles), indi- cating that several problems could not be improved beyond a certain point, or that they were not resampled during the pair generation phase. both these cases, one solution is to allow more L.LM calls during phase allow each problem be sampled for repair more times. The takeaway here is be
instruction and score
with instruction
naive
10
20
Performance
that more sophisticated fixes for diffieult problems increase the budget of LLM calls during the pair generation phase. The entire evolution of this lin- eage at different points during pair generation is illustrated in Fig. 15.
discovered
Figure 17: Ablations over repair prompt
A.9. Code Diversity
Dess
8 = N lest
S
(2)
We compute the code diversity score in Fig. 7(b) based on the number of different abstract syntax subtrees that each code instance produces. Algorithm deseribes how this diversity score computed. As first step, for each guess in the dataset and its corresponding fix generated by the LLM, we compute the respective abstract syntax subtrees. Next, we compute their corresponding set difference to get the unique subtrees for each code diff. This is done N times for a compute budget of and the number of code diff subtrees across pairs and problems is averaged and normalised in the following manner to yield the diversity score 8:
Smaxx
1=1
where Sthil the set of all code diff subtrees generated with compute budget N for problem i, and the normalising factor Smax corresponds to the highest number of subtrees that are present in any such set of code diff subtrees.
A. A.10. Prompting
There are 2 types of prompts that we use: 1) guess gener- ation prompt, and 2) repair prompt. The guess generation prompt used during dataset creation, for obtaining the initial guesses for all problems in the dataset. The repair prompt is used throughout the rest of the paper: the Pair
19

---

AuPair: Golden Example Pairs for Code Repair
Algorithm 4 Diversity score computation
N
inference compute budget
Diest
test dataset
Require:
Y
fixes for test problems
JAST
abstract syntax subtree computation function
1: init set of code diff' subtrees: Schir +-
2: for problem E Diest and its corresponding fixes 1j E 5' do
3: compute abstract syntax subtrees for guess: Stuess + fAST fAST(I (I.ess)))
4: init code diff subtrees for this problem: 8 +-
5: for j E NN do
6: compute abstract syntax subtrees for fix: spix + (ust(Us)
7: update code diff subtrees: +- u (stxx (Suusss
8: end for
9: append to code diff subtrees: 5 4- Sdiff +
end for
11: compute normalising factor: Smax f- argmax, Soliff
[Deul
12: compute diversity score: 8 + Ni S
Σ
Sdiff
return 8
Generation (Phase 1, $2.1 with h: = 32 random examples) and in the AuPair Extraction (Phase 2. $2.2) and during inference, with & = The function signature indicates that the function expects a string as an input. The instruction specifies that the final answer meant be printed inside the function, and that the main function is not meant to be written.
cases. This is the prompt that we finally use and the one that gives us better results when compared using the same set of pairs with the previous 2 strategies. An important thing to note here is that we prompt the model with desired fix score of 100 for the test problem.
We test the three strategies described above subset of the CodeForces dataset and report their performance in terms of number of problems solved, in the figure on the right. The results clearly indicate that the final prompting strategy that includes the instruction and score is the best strategy and so we choose it to compose the repair prompt.
The structure of our repair prompt is as follows: there is an instruction at the top, followed by the few-shot examples in the format: question, guess, fix. We also add the score achieved by the guess and the fix for the in-context example pairs. Following this, add the text and initial guess for the problem and the LLM then has to generate better fix. Note that we do not provide any extra execution feedback in the form of execution traces; this could potentially be explored by future work.
A.11. Code Execution
When the LLM generates a fix for any problem, we call the solve() function for each test case associated with that problem. We then compare the output with the ground truth and give partial score corresponding to the proportion of test cases passed by this fix.
Our aim is clear: the pairs indicate a certain type of change and we provide these pairs in context to aid the LLM in gen- erating an improved solution for the given problem. Some different prompting strategies that we tried out were the following:
An important point to note is that the so lveev) () function has to take as input a string, which is then parsed into the correct variables. This formatting requirement is a key reason for the poor initial performance of Gemini-1.5. 1.5-Pro in Fig. 4. Since the instruction for generating the initial guess is not correctly followed by the model, lot of guesses end up invariably having incorrect parsing of the input, leading to low scores. A lot of AuPairs extracted using these models, as a result, contain this formatting fix, as we will see in Section A.
Naïve prompting: only include the problem, guess and fix for the pairs, followed by the problem and guess for the test problem.
Prompting with instruction only: include the header instruc- tion followed by the components of the naïve prompting strategy.
Prompting with instruction and score: include the elements of 2 above, but in addition, also include the score that each guess and fix received on the corresponding problem's 's test
20

---

AuPair: Golden Example Pairs for Code Repair
Guess Generation Prompt
<problem text>
Complete the function definition below. Print the final answer in the function. Do write main. Do not write anything outside the solve () function.
def solve(s: str):
Repair Prompt
You are experienced software developer. Look the question (Q) and solutions below (A).
The main objective is to improve the solve () function to answer the question.
Example 1:
(Q:
Bad solution code A (bba)):
def solve(s: str):
The score of this code is score (A(bad)) <example_guess_score
Good solution code (good):
The score of this code is score (A(good)) sxxaple__ixxscorre
def solve(s: str):
The main objective is to improve the solve () function to answer the question. (Q):
Bad solution code (bad):
def solve(s: str):
The score of this solution is score (A(ad))) <guess_score>
Good solution code A (good):
The score of this solution is score (A(go)))) 1000
Figure 18: Composing the repair prompt.
21

---

AuPair: Golden Example Pairs for Code Repair
A.12. Types of Fixes in AuPairs
We now show some examples of AuPairs and highlight the differences between the guess and fix for each pair. These mix of CodeForces pairs collected using different models. The scores achieved by the guess and fix on the corresponding problem's 's test cases are specified at the top right corner for each example in Fig. 18. We also provide a short description for each type of fix in the caption for ease of understanding. The types of pairs discovered using algorithm cover large of potential fixes that can be made to an initial buggy piece of code: from smaller ones like parsing, fixing logical bugs pertaining to indexing errors, variable initialisations, etc., to larger changes like rewriting parts the code, or even suggesting alternate routes to solve the same problem.
22

---

AuPair: Golden Example Pairs for Code Repair
score: 8.33
score: 8.67
def solve(n: str): int(n)
def solve(n: str): int(n)
dp [0] (n
[[0 for in range(4)] for in range(n 1)] dp[0][e] = for 1 range n 1): for in range(4):
dp[0] 1
range(1, n 1): dp[i] dp[1
print(dp(n])
for in range(j, 4)
dp[il[k] (dp[i][k] dpli 11(1) x
1000000007
print(sum(dp(nn))
Fix: converts 1-D dynamic programming solution 2-D.
score: 0.0
score: 1.0
def solve(s: str, str, int):
solve(s: str):
len(s)
map(int, s.split("\n")]0))
len(t)
$ s.split( [1] = s.split( (n [2]
count
range(n
1):
match True
for i in range(n - match True for in range(m): Found False for in range(max(1,, 1 1+ k), min(n, 1):
for j in range(n):
if abs(i + f.iin((([]), i, i m)) ki
match False break
match: print(count)
5(p
T[]]:
found True
break found:
match False break 1f match:
print(count)
Fix: composition of 1) input parsing correction, and 2) logical bug fix.
score: 8.5
score: 1.0
def solve(s: str):
def solve(s: str):
n, a, b, с map(int, s.split())
n, a, b, c map(int, s.split())
if b:
print(n
print(n // a)
else:
else:
glass_bod battles
blass_bottles
remaining_moner
basssle +
remaining
glass bottles glass_botttles
plastic_bottles bottles remaining_mone // print(glass_bottle, plastic_bottles)
plastic.bottle bottles = remaining homaining_noneyyy //
if rewaininggmoone money
glass_ bottles
remaining_mone Boaiing__ooeyyeeeee
plastic bottles remaining money //
print(glass_bbottie plastic_bottle)
Fix: add an extra condition for edge cases.
23

---

AuPair: Golden Example Pairs for Code Repair
score: 0.5
score: 1.0
def solve(s: str):
def solve(s: str):
Lnt(sssppltt"''"
n int(s.splitt""n"))
1ist(nap(1nt, s.split( ()))(s))))))))(( count
count list (sp)it((""")))..pptt))
for in range(n):
for i in range(n);
sum for i in range(s, n):
sum for in range(i, sum += a[j] sum 8:
sum += a[]] if sum != 8: print(count)
else:
break print(count)
Fix: add loop exit condition.
score: 0.0
score: 0.25
def solve(s: str):
solve(s: str):
(nt(inutt))
snttssssplit""'
(iat(nn,, input() splitt))))
= list(na,,int, (.s)lit('m''AA)ssp)pl
sun(a) 8:
if sum(a)
print("NO")
print("NO")
return
else:
print("VES")
print ("YES")
a[0] 1= print(1)
print(n)
for in range(n):
print(1, n)
print(1 + 1, i 1)
else:
print(n)
for i in range(1, 1): print(i, i)
Fix: composition of 1) input parsing correction, and 2) logical bug fix.
score: 0.0
score: 9.67
def solve(n: int, d: int, list(int]);
def solve(s: str)
balance =
n, map(int, list(nap(Int, s.split( (aattt"""n))())seeea (11)))))
bank_ visits for i in range(n): balance += a[1]
balance
bank bank_ vinktvisits-a for in range(n):
balance 0:
balance
balance a[i]
print(-1)
if balance d:
return
print(-1)
bank visits *-
return
balance balance
if a[i] and balance bank visits +=
if balance > d:
print(-1) return
balance =
print(bank_visitss
print(bank_ugg visktk__iiss)
name main
n, d mint snput().split())
(iap input().split()))
solve(n, d, a)
Fix: composition of function signature correction, 2) input parsing correction, and 3) logical bug fix.
24

---

AuPair: Golden Example Pairs for Code Repair
score: 0.9
score: 0.5
def solve(s; str):
def solve(s: str);
n int(Input())
int(s.split('\n [[0])
ans list(map(int, "sppit((""""[
(x)t(x) for s] ans
for in range(n):
for 1 in range(n):
ans ans p[i] (10t29 + 7)
p[i] (10**9
print(ans)
print(ans)
Fix: input parsing correction.
score: 0.2
score: 0.6
def solve(s: str):
def solve(s: str):
map(int, s.split())
cnt p map(int, s.split())
dp = [float('inss dp[0] for range(1,
(n 1)
= 0
if cnt +- p >>=
for j in range(32);
if (1 << 1) 1:
if 1 dp[i] min(dp[i), dpli (1 f)] 1)
cnt (n // print(cnt)
dp[i] - min(dp(i), dpli p] 1)
if dp[n] float( ('inf')
print(-1)
else: print(dp(n))
Fix: solve problem using bit manipulation instead of dynamic programming.
score: 8.9
score: 0.5
def solve(s: str):
def solve(s: str):
int(ss.sllit""""'))0
graph = [[] for in range(n 1)]
graph = [[] for in range(n + 1)]
range(1, n):
for i range(1, n):
nap(int, S.split("\n")) ("))tt(("""())_ssllttt)) graph(u).appenddv))
u, map(int, 5.spltt("vn)
graph[u]
visited [False]
def dfs(node, visited);
visited[node]] True for neighbor in godeh if not visited[[eiighbr]] dfs(neighbor, visited)
for in range(1, n 1): if not visited[i];
dfs(i, graph, visited) print(count - res = for in range(1,
count =
visited = [False] (n 1) for i in range(1, n 1):
if not visited[i]:
if not visited[i]: append)))d print("res)
count dfs(i, visited)
print(count: 1)
def dndde, graph, visited): visited[node] = True for neighbor graph[node]; not visitee[neighhbr:::
visited = [False] (n 1)
for 1 in range(1, n 1):
if visited[ dfs(i, visited) (1s.append print(*res)
dfsighg,,,,,, graph, visited)
Fix: partial correction to depth-fire search graph algorithm.
25

---

AuPair: Golden Example Pairs for Code Repair
score: 0.0
score: 1.0
def solve(s: str):
def solve(s: str):
n, k map(int, (.sp)it()(e:3))
map(int, s.split([(8:3])
list(nap(int, (.5p31t((((:3+nn)))
list(map(int, (.)plit(((3:3+mn)
[list(nap(int,
[list(map(int,
(oglit(((3mmmtp::33mm(4111111113 for i in range(n)]
(ooit((((nmatpp:3mm(())))))))))) for in range(n)]
people sorted(enumerate(a) key-lambda x: x[1], reverse-True)
people sorted(enumerate(() key-lambda x: x(1], reverse-True)
jo__strength_.2
max_strength
for i, (person_ iinrsoo_inddx, audience audience_strengtee strength) in enunerate(people))
for i in range(k);
person_indes people[i][e]
max_strength a[person_ index]
break
for j in range(p):
max pox_strength.o +* audience stheencesttrenthh for j in range(p):
best best_player_ime _en___idexx____ee
best gyst_player_stregee for i in range(n);
strengtt for ror_position-
max max(max_strrr staaa_sstrenngh_for.... for_foo__pooe_ _or_pooition, s[person_ (isson__dddx(((])
if 1 not in [person[e] for person in
max strength += max sox_sstrength_ooe for for_position
]k]ple[:k]]:
(rint(mma_streengt))
1f best sest_player_steegthh s[i][]] best_ gest_player_strenthh sttrrsttttthhhtt s[i][i]
best_player index =
@clayer_indexx
nax strength best player strength
(rint(max__treegth))
Fix: rewrite partial solution to pass all test cases.
Figure 18: Examples of AuPairs produced by our algorithm (multiple models represented above)
26


Reviews:
[{"review_id": "sytXOR6MYc", "reviewer": "Reviewer_g7mK", "summary": "Paper introduces AuPair, an inference time algorithm to improve code repair capabilities of LLM. The core idea lies in first a diverse of generating golden pairs (guess, fix) using an LLM and then using a submodular selection algorithm to identify and generate an ordered set of golden example pairs. During inference, these pairs are used sequentially as 1-shot in-context examples to guide the LLM. The paper illustrates extensive experiments across 5 LLMs and 7 code repair datasets, demonstrating that AuPair consistently and significantly outperforms best-of-N and self-repair", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Including a more diverse set of models would strengthen evaluation\n\nWe have included 5 models spanning 3 model families across 7 datasets in the existing results – Gemini, GPT, Gemma; the results clearly indicate that AuPair works across models. We understand the reviewer's concern and agree that showing results on more models would strengthen evaluation, but getting the full set of results with another model within the short rebuttal period would not be feasible. What we can do, however, is include new transfer results with GPT-4o (suggested by reviewer) using AuPairs from GPT-4o-mini and Gemini-1.5-Pro.\n\n**Approach**|**Score on GPT-4o**\n---|---\nInitial|0.244\nBest-of-$N$|0.100\nSelf-repair|0.374\nw/ Gemini-1.5-Pro AuPairs|**0.486**\nw/ GPT-4o-mini AuPairs|**0.573**\n\nEven when using AuPairs from other models, we see **11% and 20% absolute performance gain** over baselines, thus solidifying our claim that even for more code-competent models such as GPT-4o, AuPair gives large performance gains.\n\n2. RAG over the candidates dataset pairs is not considered as a baseline\n\nWe implemented a RAG baseline, choosing the top 32 pairs for each problem, retrieved from the candidate pair dataset, as the reviewer suggested:\n\nModel|RAG score|AuPair score\n---|---|---\nGemini-1.5-Pro|0.379|**0.438**\nGPT-4o-mini|0.361|**0.378**\nGemini-1.5-Flash|0.318|**0.352**\nGemma-27B|0.178|**0.214**\nGemma-9B|0.156|**0.198**\n\nWe see that across all models, AuPair, which uses a fixed set of in-context examples, outperforms RAG, which requires the entire set of candidate pairs to be compared with each test problem to choose the in-context examples.\n\n3. Another analysis on the diversity aspect which includes the type of repairs these pairs help in\n\nWe show a breakdown of the repairs generated into 4 categories: 1) %problems with improved fix score, 2) %problems in which code was reformatted to obey constraints, 3) %problems in which fix was improved by just changing the logic, 4) %problems in which score remained unchanged. Composite changes (formatting + logical) are reported in formatting fixes.\n\n**Model**|**Approach**|**Improvements**|**Formatting fixes**|**Logical fixes**|**Unchanged**\n---|---|---|---|---|---\nGemini-1.5-Pro|Best-of-$N$|9.67|1.37|8.45|81.99\n&nbsp;|Self-repair|8.50|0.83|7.91|82.48\n&nbsp;|AuPair|47.14|50.32|14.11|44.82\nGPT-4o-mini|Best-of-$N$|8.66|2.48|7.39|61.92\n&nbsp;|Self-repair|12.98|2.71|10.81|60.24\n&nbsp;|AuPair|22.34|25.41|6.70|56.46\nGemini-1.5-Flash|Best-of-$N$|8.27|0.05|8.24|82.92\n&nbsp;|Self-repair|18.52|0.12|18.52|54.77\n&nbsp;|AuPair|24.21|16.12|16.88|65.99\nGemma-27B|Best-of-$N$|11.57|0.49|10.78|77.09\n&nbsp;|Self-repair|9.35|0.44|9.13|79.55\n&nbsp;|AuPair|18.97|13.90|11.32|69.24\nGemma-9B|Best-of-$N$|16.76|7.48|12.77|72.66\n&nbsp;|Self-repair|10.60|0.42|10.50|79.85\n&nbsp;|AuPair|20.00|18.16|14.71|69.73\n\nSome insights:\n\n- Since Gemini-1.5-Pro guesses have more formatting bugs, % problems with formatting fixes using AuPair is high.\n- AuPair also helps the model repair solutions with logical errors.\n- AuPair yields responses that are more diverse than baselines in the test case scores, indicated by the lower value of \"Unchanged\".\n\nExample of composite fixes:\n\nGuess:\n```\ndef solve(s: str):\n   n = len(s)\n   a = int(input()) - 1\n   b = int(input()) - 1\n    cost = 0\n   for i in range(a, b):\n     if s[i] != s[i + 1]:\n       cost += 1\n   print(cost)\n```\n\nFix:\n```\ndef solve(s: str):\n   n, a, b = map(int, s.split('\\n')[0].split())\n   companies = s.split('\\n')[1]\n   cost = 0\n   if companies[a - 1] != companies[b - 1]:\n     cost = abs(a - b)\n   print(cost)\n```\n\n4. Does AuPair utilize a slightly higher computational budget compared to baselines?\n\nThis is partially correct: there is a one-time amortised cost that our algorithm incurs to construct the AuPairs, but the cost at test-time is identical. Note that the same fixed set of AuPairs boosts performance across models and datasets (Fig. 6, 7 show out-of-distribution generalisation), indicating that the upfront cost is easily amortised.\n\n5. The phrase \"in conjunction with high performing\" in line 260 is somewhat confusing\n\nWe explain this in line 250: \"since the code generated has to adhere to a specific format to allow successful execution, we observe that many initial guesses of generated code fail because they do not obey these conditions\". Moreover, our experiments on GPT-4o above also show that the insight stands, since we observe absolute improvements of 11% and 20% using AuPair compared to self-repair (strongest baseline).\n\n6. The necessity of the subtraction step is not intuitively clear\n\nThe fix quality matrix contains the score for each candidate pair for each validation problem. After picking the pair with the best mean score, we subtract this score from the matrix because only after removing the previous best pair can we find the complementary next best pair.\n\n7. Missing references\n\nWe will include the mentioned references in the Related Work section.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "While the paper focuses on code repair this can have impact on contributing to more robust and reliable software.", "essential_references_not_discussed": "1. Code Repair with LLMs gives an Exploration-Exploitation Tradeoff (Tang et al. NeurIPS 2024)\n2. CYCLE: Learning to Self-Refine the Code Generation (Ding et al, OOPSLA 2024)", "other_strengths_and_weaknesses": "Strengths:\n1. Extensive experiments section covering various aspects of the approach (scaling, generalization, multiple datasets)\n2. The approach is interesting for improving the program repair capabilities of LLMs without any fine-tuning/training. \n\n\nWeakness:\n1. The evaluation primarily focuses on models from the Gemini family and GPT-4o-mini. It omits other models recognized for strong coding performance (and high rankings on relevant benchmark leaderboards), such as Claude 3.5 Sonnet, GPT-4o, and DeepSeek-V3. Including a more diverse set of models would strengthen the evaluation.\n2. All experiments are conducted within the domain of competitive programming. Checking if AuPair could be used for other programming tasks, such as programming-by-example (e.g., using datasets like those explored by Tang et al. in \"Code Repair with LLMs Give an Exploration-Exploitation Trade-off\"), would be valuable and help generalize the approach.\n3. While the ablation study in Section 3.2 using random pairs provides insights into AuPair quality, further ablation experiments could help assess the effectiveness of the greedy selection approach. This could include comparing against randomly ordered AuPairs or exploring alternative selection criteria, such as a curriculum learning-inspired approach (e.g., selecting pairs in increasing order of problem difficulty).\n4. Is there a reason why RAG over the candidates dataset (guess, fix) pairs is not considered as a baseline?", "other_comments_or_suggestions": "See questions for authors", "questions_for_authors": "1. Is it accurate to state that AuPair utilizes a slightly higher computational budget compared to best-of-N and self-repair baselines? AuPair involves LLM calls in two preprocessing stages (dataset creation and pair selection) before the budgeted N inference calls. The budget for the creation phase, ranging from 10,000 to 35,000 calls, is quite substantial. \n2. The phrase \"in conjunction with high performing\" in line 260 is somewhat confusing, given that Gemini-1.5-Pro's initial performance is reported as low.\n3. Section 2.2, describing the fix quality matrix, requires a more detailed explanation as it is central to the approach. The necessity of the subtraction step, in particular, is not intuitively clear and needs further justification.\n4. Figure 7 suggests a performance drop when AuPairs are generated by a different model, even if the generating model is generally superior. More insights could help explain this phenomenon and the underlying reasons for this cross-model performance variation.?\n5. Could there be another analysis on the diversity aspect which includes the type of repairs these pairs help in? For instance, incorrect formatting, syntax, semantic, different types of bugs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j1vFjzZcKG", "reviewer": "Reviewer_oGUo", "summary": "The paper introduces AuPair, a novel algorithm designed to improve Large Language Models' (LLMs) performance on code repair tasks through inference-time computation. AuPair leverages in-context learning by synthesizing an ordered set of example pairs (called \"AuPairs\") consisting of initially incorrect code (\"guesses\") and subsequent corrected code (\"fixes\"). At inference time, the method provides each AuPair as a one-shot in-context example to guide LLMs toward generating diverse and improved code fixes. The inference-time algorithm to construct highly effective \"golden example pairs\" for code repair can achieve significant performance boosts over traditional approaches like best-of-N and self-repair.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Limited exploration of failure cases\n\nWhile AuPairs have been shown to significantly boost performance, they can occasionally have unintended impacts as well. The following table contains the percentage of CodeForces problems in which some fixes had a decrease in fix score compared to the initial guess. Note that this does not affect any of the performance results in the main paper, since for measuring performance, the best scoring response is selected.\n\n**Model** | **Approach** | **% problems w/ decreased fix score**\n--- | --- | ---\nGemini-1.5-Pro | Best-of-$N$ | 10.52\n&nbsp; | Self-repair | 7.62\n&nbsp; | AuPair | 11.63\nGPT-4o-mini | Best-of-$N$ | 20.09\n&nbsp; | Self-repair | 11.87\n&nbsp; | AuPair | 15.28\nGemini-1.5-Flash | Best-of-$N$ | 9.47\n&nbsp; | Self-repair | 22.28 |\n&nbsp; | AuPair | 11.79\nGemma-27B | Best-of-$N$ | 14.86\n&nbsp; | Self-repair | 9.72\n&nbsp; | AuPair | 15.21\nGemma-9B | Best-of-$N$ | 13.16\n&nbsp; | Self-repair | 9.38\n&nbsp; | AuPair | 13.09\n\nAs we can see from the above table, in most cases, using AuPair results in an increase in the number of problems for which a fix is worse than the initial guess. This is to be expected since AuPair is an algorithm that in addition to boosting performance also boosts diversity of the generated responses.\n\n2. The evaluations focus predominantly on competitive programming tasks. It remains unclear whether the impressive results would translate directly into other code repair domains.\n\nCompetitive programming tasks are a canonical domain for code repair for two reasons: these tasks are **rigorous** (can be precisely evaluated, cannot be gamed) and they are **hard** (supposed to differentiate between top human coders). Both of these make them appropriate to research and test new methods, the crisp, reliable evaluation of challenging tasks means that even differences of a few percent are meaningful capability improvements. But indeed, competitive programming tasks are not the bulk of what users might request. While we see no reason to doubt that our method will translate to broader, less well-defined, or easier tasks, we do not want to make a strong claim about this transferability; producing that evidence is out of scope for this paper. We will expand on this in the final version of the paper.\n\n3. The paper's method strongly relies on predefined test cases and evaluations on these tests to judge fix quality.\n\nIndeed, in this work, we have used test cases for assessment of fix quality and for guiding the exploration phase, such as has been done in prior work [1,2,3]. However, contrary to other approaches for code repair, where failed test cases are given to the model in the prompt, we do not include test cases in the prompt; we use test cases solely to build the AuPairs.\n\nFurthermore, we would like to highlight that there is nothing fundamentally in the approach that prevents the use of other feedback mechanisms, such as reward models, or other sources of feedback to boost its performance. We leave this set of experiments for future work since we believe it is orthogonal to the key ideas presented.\n\n4. How sensitive is AuPair's effectiveness to the size and diversity of the validation set used for AuPair selection?\n\nThis is a great question; we conducted experiments with smaller validation sets to curate AuPairs and report the results below. We also include the results of the random baseline for calibration:\n\nSize of Validation | # of AuPairs | Score (inference budget = 32)\n-----------------------|------------------|-----------------------------------------\nRandom\t    | N/A\t   | 0.383\n10%\t\t     | 32\t\t   | 0.403\n25%\t\t     | 52\t\t   | 0.418\n100%\t\t     | 144\t   | 0.438\n\nThe larger the validation set, the more distinct complementary improvements can be observed, and hence the larger the maximal set of AuPairs that can be discovered. So larger validation sets make it possible to effectively scale up to more inference compute. However, even just looking at just the top 32 AuPairs (which is apples-to-apples for varying validation set sizes), we find that their quality increases monotonically with the size of the validation set. We have also conducted additional analysis on the impact of smaller datasets, for which we point the reviewer to section A.4 in the paper.\n\n---\n\n[1] Code Repair with LLMs gives an Exploration-Exploitation Tradeoff, Tang et al. 2024\n\n[2] Cycle: Learning to self-refine the code generation, Ding et al. 2024\n\n[3] Teaching LLMs to self-debug, Chen et al., 2023", "claims_and_evidence": "- The authors claim their proposed approach (AuPair) significantly improves the code-repair capability of Large Language Models (LLMs) by leveraging carefully selected golden example pairs for in-context prompting.\n- AuPair demonstrates strong generalization capability\n- AuPair significantly outperforms traditional inference-time methods like Best-of-N and Self-repair methods.\n- AuPair scales notably better with increasing inference-time compute budgets, yielding substantially higher performance improvements per unit of compute than baselines.\n- AuPairs result in more diverse code fixes compared to best-of-N approaches\n\nThe claims are well supported.", "methods_and_evaluation_criteria": "AuPair consists of two phases:\n\nPhase 1: Pair Generation\n- Data Collection: Starting with a dataset of coding problems and initial LLM-generated guesses (potentially flawed code), the LLM iteratively generates fixes.\n- Candidate Pair Generation: For each sampled problem and its initial flawed guess, the LLM produces improved fixes using a k-shot prompt (with randomly selected pairs from existing candidate set). Generated fixes that outperform the original guess are added to a pool of candidate guess-fix pairs. Imperfect fixes become new guesses to generate additional candidate pairs.\n\nPhase 2: AuPair Extraction\n- For each candidate pair and problem in the validation set, the algorithm constructs a 1-shot prompt to evaluate how effectively each pair guides the LLM to repair other problems.\n- The LLM-generated fixes are scored against provided unit tests, forming a fix-quality matrix.\n- AuPairs are selected in a greedy manner from the candidate pairs based on their incremental contribution to solving distinct problems, ensuring complementarity and diversity.\n- This selection iteratively picks pairs that yield maximum additional performance on unsolved problems until further improvements fall below a predefined tolerance threshold.\n\nThe paper evaluates the proposed method using seven competitive programming datasets, using multiple models.\n\nThese methods and metrics allow for a rigorous evaluation of the algorithm's effectiveness.", "theoretical_claims": "The paper does not make explicit theoretical claims or provide formal theoretical analysis", "experimental_designs_or_analyses": "The experimental designs and analyses are sound.", "relation_to_broader_scientific_literature": "The paper builds upon the broader literature which has shown that increasing inference-time computation, without additional fine-tuning, significantly enhances Large Language Model (LLM) performance (e.g., best-of-N, self-consistency, etc). The paper effectively integrates ideas from multiple recent lines of research—leveraging in-context learning capabilities, automated code repair without supervised fine-tuning, and strategic prompting—to deliver significant performance gains, and positions itself well within and advances the state of the literature in automated self-repair for code generation.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- The paper introduces a practical and impactful approach\n- Extensive experiments across multiple competitive programming datasets (CodeForces, AtCoder, CodeChef, etc.)\n- Empirical results convincingly show substantial improvements\n- Detailed ablation and additional analysis\n\nWeaknesses:\n- Limited exploration of failure cases\n- The evaluations focus predominantly on competitive programming tasks. It remains unclear whether the impressive results would translate directly into other code repair domains, such as debugging production software (or non-code related domain), where test coverage or complexity might differ significantly.\n- The paper's method strongly relies on predefined test cases and evaluations on these tests to judge fix quality.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "How sensitive is AuPair's effectiveness to the size and diversity of the validation set used for AuPair selection?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I6sGnoYjZX", "reviewer": "Reviewer_2Ysj", "summary": "The paper introduces Aupair, which is a customed method that generates golden example pairs for enhancing code repair performance. In their work, each pair contains an initial guess and its fixes, which are used as in-context example at inference time to generate a repaired solution. During inference call, the fix with best score are selected as the output. The authors claim that this approach not only greatly outperforms methods like best‐of-N and self-repair on 5 LLMs and 7 datasets, but also scales better with on inference-time compute.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. discuss related work that uses adversarial approaches for reference.\n\nWe will include references to adversarial approaches such as [1,2] in the Related Work section.\n\n2. Only unit test scores as the metric to evaluate correctness of code repairs\n\nIn this work, we have used unit test scores for assessment of fix quality and for guiding the exploration phase, such as has been done in prior work [3,4,5]. However, there is nothing fundamentally in the approach that prevents the use of other feedback mechanisms, such as reward models, or other sources of feedback to boost its performance. We leave this set of experiments for future work since we believe it is orthogonal to the key ideas presented.\n\n3. Does not has good analysis on failure cases or situation that the method not working well\n\nWe refer the reviewer to point 1 in our rebuttal address to reviewer oGUo: we provide concrete failure modes in which using AuPairs can lead to fixes that are potentially worse than the initial guesses, and this is observed more frequently for AuPair than other baselines on average, since AuPair is a diversity-boosting algorithm. \n\n4. Is the unit score the only metrics you used?\n\nNo, in addition to the test pass rate for unit tests, we also report the strict accuracy metric [6], which is the percentage of test problems that are fully solved. We report this metric for every single ablation in the paper; we refer the reviewer to Section A.2 in the Appendix, as well as Figs. 10, 11, 12 for plots showing this metric.\n\n5. Consider discuss potential avoid-situation that may cause failures or lower performance.\n\nWe refer the reviewer to Section A.4 of the Appendix, where we have a detailed discussion on the impact of smaller datasets on the efficacy of our proposed approach. Our results indicate that even though AuPair works well in the small dataset regime, curating the AuPairs on a larger dataset and then applying them to a smaller dataset, even though out-of-distribution, recovers in-distribution performance; it can also potentially lead to stronger scaling, since there will be fewer AuPairs extracted on smaller datasets. Furthermore, we have additional results for this in point 4 of our rebuttal to reviewer oGUo.\n\n---\n[1] Adversarial patch generation for automated program repair, Alhefdhi et al. 2023\n\n[2] Learning to repair software vulnerabilities with Generative Adversarial Networks, Harer et al. 2018\n\n[3] Code Repair with LLMs gives an Exploration-Exploitation Tradeoff, Tang et al. 2024\n\n[4] Cycle: Learning to self-refine the code generation, Ding et al. 2024\n\n[5] Teaching LLMs to self-debug, Chen et al. 2023\n\n[6] Measuring coding challenge competence with apps, Hendrycks et al. 2021", "claims_and_evidence": "Claim: Aupair significantly improves code repair performance on different models and datasets, as well as scaling.\nEvidence: Via experiments on the 7 datasets and the 5 LLMs, where the aupair shows better performance on different comparing metrics.", "methods_and_evaluation_criteria": "The method is about generating guess-fix pairs and select a diverse subset by using their method. The evaluation seems good as it covers repair quality as well as diversity. So it make sense to me.", "theoretical_claims": "There seems no theoretical proof.", "experimental_designs_or_analyses": "The design of experiments seems reasonable to me, as it contains a good validation frame and across different LLMs. However, a more detailed discussion of parameters selection could benefit the paper more.", "relation_to_broader_scientific_literature": "The paper's Aupair is based on best-of-N, self-repair, and in-context learning. The main contribution seems to be a prompt-engineering method to improve the LLM's performance", "essential_references_not_discussed": "No.\nHowever, regarding the code generations, it might be good if the authors can discuss related works that uses adversarial approaches for reference.", "other_strengths_and_weaknesses": "Strong empirical results based on the datasets and LLMs the author presented.\n\nWeakness: \nOnly unit test scores as the metric to evaluate correctness of code repairs.\nDoes not has good analysis on failure cases or situation that the method not working well\nLimited theoretical part", "other_comments_or_suggestions": "Consider include a more comprehensive ablation studies so we know the impacts with more details.\nConsider discuss potential avoid-situation that may cause failures or lower performance.\nMain concern: You didn't mention about the code availability, please ensure to make them available for reproducibility.", "questions_for_authors": "Have you tested the sensitivity of the input? Is the unit score the only metrics you used? have you considered better assessment of code quality?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents an approach to synthesize an ordered set of AuPairs (golden example pairs) of an incorrect and subsequent corrected version of the code. The key idea of the approach is to select the AuPairs that are complementary to each other and maximize their usefulness based on a held-out validation set. These AuPairs are then used at inference-time for 1-shot in-context code repairs, and the results show that it yields a significant improvement over best-of-N and self-repair approaches. All reviewers found the overall idea of synthesizing an ordered set of golden example pairs (AuPairs) and then using them at inference time to compute diverse repair solutions  quite interesting. But there were also concerns around generalizability of the approach on other models, actual notion of diversity, RAG baseline, only using test-based metrics, failure analysis, and limited evaluation dataset of only competitive programming. The author response with additional experiments with GPT-4o, RAG baseline, further analysis of diversity and failure cases, and robustness of selection helped quite a lot. It would be great to include the additional results and discussions in the rebuttal response as well as incorporate feedback from the reviews in the final version of the paper.
Decision: Accept (poster)