Towards Low-Resource Automatic Program Repair with Meta-Learning and Pretrained Language Models
Authors: Weishi Wang
Yue Wang
Steven Hoi
Shafiq Joty
Venue: EMNLP 2023 Main (2023)
Keywords: Low-resource APR
TL;DR: We are the first to investigate the low-resource automatic program repair (APR) based on error types, and propose a simple yet effective framework (Meta-APR) that extends CodeT5 with an efficient first-order meta-learning algorithm.

Abstract: Automatic program repair (APR) has gained increasing attention as an essential technique in software development to reduce manual debugging efforts and boost developers' productivity. Recent advances in deep learning (DL) based models have demonstrated promising results by learning from large-scale bug-fix examples in a data-driven manner. However, in practical scenarios, software bugs have an imbalanced distribution, and the fixing knowledge learned by APR models often only capture the patterns of frequent error types, making it inapplicable to handle the rare  error types. To address this limitation, we investigate  a novel task of low-resource APR, and propose Meta-APR, a new meta-learning  framework integrated with code pretrained language models to generate fixes for low-resource bugs with limited training samples. Our Meta-APR learns better error-specific knowledge from high-resource bugs through efficient first-order meta-learning optimization, which allows for a faster adaptation to the target low-resource bugs. Besides, while we adopt CodeT5, a pretrained code-aware encoder-decoder Transformer, as the backbone model for Meta-APR, it is a model-agnostic framework that can be integrated with any neural models. Extensive experimental results on three benchmarks in various programming languages verify the superiority of our method over existing DL-based APR approaches.

Towards Low-Resource Automatic Program Repair with Meta-Learning and Pretrained Language Models
Weishi Wang!? Yue Wang',, Steven C.H. Hoi', Shafiq Joty
SSalesforce AI Research
#Nanyang Technological University, Singapore (weishi wang, wang y, sjoty, shoi hesalleeaaaleecccooo com
Abstract
High-Resouroi Mata-Training
Low Resource Adapiaaion
Automatic program repair (APR) has gained increasing attention an essential technique in software development reduce manual debug- ging efforts and boost developers' productivity. Recent advances in deep learning (DL) based models have demonstrated promising results by learning from large-scale bug-fix examples in data-driven manner. However, in practical scenarios, software bugs have an imbalanced distribution, and the fixing knowledge learned by APR models often only capture the patterns of frequent error types, making it inapplica- ble to handle the rare error types. To address this limitation, we investigate novel task of low-resource APR, and propose Meta-APR, a new meta-learning framework integrated with code pretrained language models to generate fixes for low ow-resource bugs with limited train- ing samples. Our Meta-APR learns better error- specific knowledge from high-resource bugs through efficient first-order meta-learning opti- mization, which allows for faster adaptation to the target low-resource bugs. Besides, while we adopt CodeT5, pretrained code-aware encoder-decoder Transformer, as the backbone model for Mcta-APR, it is aggoosic framework that can be integrated with any neu- ral models. Extensive experimental results on three benchmarks various programming lan- guages verify the superiority of our method over existing DL-baaed APR approaches.
Optimization
Support
Loop
e= -
Outer Loop
Low Rosourse APR
changes " payload, gec ""hangge",, 413 change changes:
Enooder
Decoder
cowddastaagget... qww.t"
Bugay code
comtat semaatteechangg.... gett "'cannttt"
Figure Illustraiion of Meta-APR framework with CodeT5 for low-resource error-specific automatic pro- gram repair. We first meta-tran veeta-train the CodeT5 with high- resource bugs ( where the backbone model is updated via gradient descent with respect to 0. After that, Mcta- APR is finetuned on the target low-resource bugs (4,A) using few-shot examples (10, 50, 100).
With the advances in deep learning (DL) mod- els (Vaswani al., 2017) and accessibility large code corpora (Tufano al., 2019: Lu et al., 2021), neural approaches to APR have achieved remarkable performance via exploiting existing code patches (Chen et al., 2021b; Zhu et al., 2021). These models are typically trained and evaluated on datasets that comprise mix of various error types, which are diverse in nature: they vary in terms of the number of bug-fix pairs per error type, and are typically imbalanced. Moreover, the perfor- mance gaps across different error types are tremen- dous (Berabi et al., 2021), which can significantly impair the APR models' performance.
1 Introduction
Program repair is critical to improving the pro- ductivity and stability of software development. However, it is resource-consuming and cost- prohibitive (Weiß et al., 2007; Planning, 2002: Jør- gensen and Shepperd, 2007). A reliable automatic program repair (APR) system is thus crucial to re- duce manual debugging efforts and development time (Gazzola et al., 2019; Winter et al., 2023).
These observations motivate to consider the idea: rather than training the model jointly on all error types, could we train model that is quickly adaptable to any low-resource error-specific APR task? Inspired by the success of meta-learning on low-resource NLP tasks like machine transla-
"Corresponding author: wang yesalesforccc.comm

---

tion (Gu et al., 2018; Park et al., 2021) and dialogue generation (Mi al.. 2019; Lin al., 2021), in this work, we drive pioneering efforts in formalizing low-resource APR, and propose an effective meta- learning framework that utilizes a code pretrained model to enhance APR performance.
We show that Meta-APR significantly outperforms the standard transfer-leaming method in all settings, As our Meta-APR
model-agnostic framework that can be integrated with any other DL models, we compare its performance when integrated with other pretrained models like UniXcoder (Guo et al., 2022). Our results demonstrate that Meta-APR con- sistently enhances performance. Further analysis confirms that Meta-APR is a more robust and ef- fective approach in fixing bugs with various buggy patch lengths and error types.
Low-resource APR formulation: Unlike tradi- tional APR approaches that jointly learn a model from a mix of error types, our formulation consid- ers each rare error type as a low-resource target task. Accordingly, we create datasets specifically to support the evaluation of low-resource error- specific APR, based on three practical APR bench- marks in various programming languages: TFix in JavaScript (Berabi al., 2021), ManySStuBsA] in Java (Karampatsis and Sutton, 2020), and TSSB- 3M in Python (Richter and Wehrheim, 2022). We observe diverse and imbalanced error type distributions in these benchmarks, e.g. TFix, ManySStuBs4I, and TSSB-3M respectively have 31, 8, and error types that are of low-resource' along with 21, 6, and 14 high-resource enror types.
We further compare with closed-sourced lan- guage models such as ChatGPT (OpenAl) in fix- ing these low-resource bugs. We find Meta-APR achieves much better performance than ChatGPT under zero-shot/few-shee settings. Besides, we ob- serve that ChatGPT often predicts "no bugs", which is probably because it does not well capture the fix- ing patterns of these low-resource bugs due to their data scarcity issue.
Related Work
Meta learning for low-resource APR: To bet- ter address the task distribution issue while adapt- ing the model to low-resource (synonymously, few- shot) tasks, we propose novel meta-learning ap- proach integrated with pretrained code language models. To the best of our knowledge, this is the first work study the low-resource error- specific APR. We build Meta-APR with code- aware pretrained encoder-decoder Transformer model CodeTS (Wang et al., 2021) ) and an efficient first-order meta-learning algorithm Reptile (Nichol et al., 2018) for the challenging low-resource APR tasks. Fig. illustrates the overview of our Meta- APR approach. Specifically, we first meta-train CodeT5-based APR model on high-resource bug- fix pairs to learn a better model initialization that captures srror-specific knowledge, which enables faster adaptation to the target low-resource bugs via finetuning on few-shot examples. In our experi- ments, we show that Meta-APR effectively aligns the representations between high-resource and low- resource bugs that they have closer distance in the representation vector space.
Automatic Program Repair (APR) Recently, there is growing body of APR research that aims to automate the rectification of software defects with less human intervention. In general, conven- tional APR approaches can be divided into three categories (Zhang et al., 2023, 2022), which are 1) heuristic-based (Goues et al.. 2012; Qi et al., 2014; (Martinez
Jiang et al., 2018), 8), 2) constraint-bac and Monperrus, 2018; Nguyen et al., 2013; Xuan et al., 2017), 3) template based approaches (Liu et al., 2019; Koyuneu et al., 2020).
Besides, learning- approaches (Chen al., 2021b; Lutellier et al., 2020; Jiang et al., 2021) have shown to achieve promising results by learn- ing the fix patterns from previous bug-fix pairs in an end-to-end manner. Motivated by the success of Neural Machine Translation (NMT) in the NLP domain, one notable lcarning- based APR method is formulated sequence-to-sequees tion problem (Tufano al., 2019), which aims translate a buggy code into its correct version. This technique is further enhanced by using pretrained models such as T5 (Raffel et al., 2020) in Berabi et al. (2021). In this work, we propose to exploit a pretrained code-aware CodeT5 (Wang et al., 2021) following Bui et al. (2022); Wang et al. (2023a).
genera-
We extensively evaluate Meta-APR on three cu- rated low-resource multilingual APR benchmarks with different degrees of low-resource settings. i.e. different numbers of training samples (10, 50, 100).
Meta- Learning for Low-Resource Tasks Meta- lcarning has been well studied for few-shot learning as a learning-to-leam learn approach, which attempts to
'ee select low-resouree sccnarios based on the numher of examples per error type for each datasct.

---

learn new concepts based on past experiences (Ben- gio et al., 2013; Vilalta and Drissi, 2002). Recently, optimization-buser techniques yield substantial im- provement in many low low-resource NLP tasks (Zhao et al., 2022). Among them, Model-Agnostic Meta- Learning (MAML) (Finn et al., 2017) has been widely used to tackle low-resource NLP tasks such machine translation (Gu et al., 2018; Park et al., 2021), dialogue generation (Mi al., 2019; Lin al., 2021), and text-to-speech alignment (Lux and Vu, 2022). MAML has shown exceptional effi- cacy in learning good parameter initialization for a fast adaption with limited resources.
Algorithm 1: Meta- Teainnn for APR
Require:
A set of high-resouree emror types
Ti [Ti, Ta...., T.., VT; Ti, it pairs
with associated bug-fix pairs that
D, = {(8,, F2) ")) APR model fo. inner loop learning rate outer loop learning rate B, meta update step size M
Initialize: Initialize 0 from the APR model fe
Output: Optimal meta-trained APR model fo while not done do
Dr. = b.
forall T с Tr. do Append the training dataset of TT into Dr. end
Randomly divide the merged training dataset Dj into hatches B.
forall B, do
Recently, meta-learning approaches have been adapted to solve low-resource software intelligence tasks such as code summarization (Rauf et al., 2022; Xie et al., 2022), and code search (Chai et al., 2022). To the best of our knowledge, we are the first to formulate the low-resource error-specific APR task based on the error type distributions and investigate the effectiveness of meta-learning methods. In addition, unlike prior approaches that mostly use the second-orde meta-learning algo- rithm MAML, we exploit a more efficient first- order meta-lcarning method Reptile (Nichol et al., 2018). In the ablation studies, show that it out- performs MAML in various low-resource settings.
Obtain Boupport and Bguerry as in $3.1 Evaluate the inner loop eross entropy loss Linner (fo. Beupport)
Update error- error-specific model parameters with gradient descent:
8' = A aVaoinner (fo, B,uppor))
if current step mod M 0 then
Update global model parameters with
estimation: 0 <- 0 3(0' 0) end
Evaluate the inner loop cross entropy loss Cinner (Jo, B8 Bquerry)
end
end
ages a code pretrained model for low-resource error-specific APR. We first formulate the task of low-resource error-specific APR in $3.1. Then, we describe our error-specific meta-APR dataset creation in $3.2 and our Meta-APR method in $3.3.
Programming Language Models Inspired by the success of pretrained language models (LMs) such as BERT (Devlin et al., 2019), GPT (Rad- ford et al., 2018), and T5 (Raffel al., 2020) in NLP tasks, there many recent attempts for code pretrained models that can be classified as three categories: 1) Encoder-only approaches like CodeBERT (Feng et al., 2020) and GraphCode- BERT (Guo et al., 2021): I); 2) Decoder-only methods such as CodeGPT (Lu et al., 2021); and 3) Encoder- decoder models like CodeT5 (Wang et al., 2021; Le et al., 2022: Wang al., 2023b). Besides, UniX- coder (Guo et al., 2022) adopts UnikM archi- tecture (Dong et al., 2019) with various attention masks. Recent studies further explore the use of very large LMs such as Codex (Chen et al., 2021a) for APR tasks (Prenner and Robbes, 2021; Joshi et al., 2022) in a zero-shot/few-shot setting, where there is still clear gap between Codex and the domain-specific finetuning methods.
3.1 Task Formulation
Assume that we have a set of error types T = (Ti
For each error type T,, it as- sociates with collection of bug-fix fix pairs D; {BB),F)) F where (B,, Fy) denotes the j-th bug-fix pair. For the error types in T, we define their resourceness based on the total number of bug-fix pairs [Dz]. Considering the actual data dis- tribution across three benchmarks, select em- pirical cutoff value of 1000 instances. This thresh- old value established to identify an error type as low-resource it has less than 1000 samples. Otherwise, we treat it as high-resource.
Formally, our proposed framework com- prises a neural sequence- pruuencc-to-ssquence (seq2seq) model (Sutskever et al., 2014) fo as base APR model. Given a set of error-specific bug-fix pairs D. = [(,, F
3 Approach
Fig. illustrates the overview of our proposed Meta-APR, a meta-learning framework that lever-
[P, 11 fo generates F, based on B,

---

no-extra-Oind
SWAP SWAP_ BOOLEAN, LITERAL
SAME FUNCTION WRONG, CALLER
class PushEvan tilook (Hase weetttott
mlockedEnodeFathh snodePath nInode7ses. LooePul entry..getddl.
changee
sarr.paplodd.ges /
difference
zaturo nagch (arr, nemols
(11. 11.get|| ccaaggee"
(thim)
InodeTree. LockMode MBI131)
for change filter (None,
bind (thisl vatercb)
wattteilbutototernall (inodeRath, Folac, entry ge petopTimeMs (), options)
commi
+). xatercb);
target. get cooriits". (1)
(bbidd(thia)
sechttributannternd (inoell(innnnddi (inodePath,
commit
untzy.getopCinaka opticns);
change.
(1)
not cormi
continue
zeturo napCb (aZE, namoly bind(thiin (thio) vaterCb) (thia),
vetittributeIiterraaainna (nooss false, eaczy. wwt.:/ggtttttnsss... V1, optiona)
conmita tazget. "ccomiiaa,
return maptb (ert, nemol: })r
sedhstributeIrttrnn.. (LnosePath, true, comnita entzy. geteypiaatpp optional;
change.get ("comni ta",, []]
watertb1: biadddddddd (this),
(a) TFix (JavaSeript)
(b) ManySStuBs4J (Java)
(c) TSSB-3M (Python)
Figure 2: Bug fix examples three low-resource error-specific APR tasks from one particular error type (ET), where our Meta-APR successfully fixes bugs while the transfer-learning (TF) approach fails to do so.
in an autoregressive manner. Formally,
based on the actual counts of its associated bug- fix pairs (< 1000). To construct more challenging low-resource scenarios, we randomly select 10, 50, and 100 samples from each low-resource error type. Following the common practice (Gao et al., 2021), we repeat this few-shot sampling process with five different random seeds (13, 21, 42, 87, and 100). For evaluation, we report the averaged results over the five seeds to rule out the random noises.
F( II P(FAAlB,,Fs. F3a Fjk-1,
where Fý.1 : FJ,--1 1 is the previous sequence at k-th token with denoting the total number of tokens in the target sequence Fj.
During the meta-training stage, we ran- domly sample batch of bug-fix pairs B, |8.1 {(B,, F.)) s=1 from high-resource error types. Each batch B, is further divided into Boupport and Bquery equally. Then, we apply the first-order meta- learning algorithm Reptile (Nichol et al., 2018) update fo via gradient descent. After that, the model fe is finetuned on a low-resource error type with few-shot examples. The underlying idca of Meta-APR is to meta-train a model on high- resource error types such that it is quickly adapt- able to low-resource types with few-shot examples.
3.3 Model-Agnostic Meta-APR Framework
Base APR Model CodeT5 (Wang et al., 2021) a unified code-aware encoder-decoder Trans- former model pretrained from large-scale source code corpus in eight different programming lan- guages. CodeT5 has been shown to achieve SoTA performance in many code understanding and gen- cration tasks such as defect detection and program refinement. In this work, we propose to adapt CodeT5 as the base model of our Meta-APR leverage its better code understanding capability.
3.2 Low-resource APR Dataset Construction
High- RighhRReoouree APR Meta-Traa Cata-Trainnig During the meta-training phase, each mini-batch of data simu- lates the low-resource scenarios. In our Meta-APR approach, we iterate through set of high-resource eтror types private training task to update fe. We first merge all high-resource error-specific training dataset Dirain and randomly segment Diroin into N batches BB,, Br) equally. 3 Then, each B, is further split into B soupport and Bquery to form a local error-specific meta-learning
As there are no available low-resource APR bench- marks for evaluation, we curate three low-resource APR datasets in various low-resource settings from three existing APR benchmarks with error type an- notations, which are TFix in JavaScript (Berabi al., 2021), ManySStuBs4J in Java (Karampat- sis and Sutton, 2020), and TSSB-3M in Python (Richter and Wehrheim, 2022). As mentioned in $3.1, we define the low-resource error types

---

task update the global APR model fe using gra- dient descent:
High-resource
Low-resource
Benchmark
#Error
#Train
#Error
Few-shots
#Test
ManySSSuBs4J
6
20,225
(0,10.50,100)
569
0' = 0 10--00 (o B5 B
TFix
21
75,998
31
(0, 10,50,00,100
1,087
TSSB-3M
14
66,384
00,10,50, 100)
538
- 0 3(0 0)
Table I: Data statistics of 3 error-specific low-resource APR benchmarks. During low-resource finetuning, we randomly sample (10,50, 100) shots for each error type to construct various low-resource settings.
where 0 is the global model parameters, and 6' is the local error-specific model parameters, and 3 denote the learning rate of the inner loop and outer loop respectively, L denotes the cross en- tropy loss function. The error-specific local gradi- ents are grouped by every M steps to update the global APR model parameters A. The meta-training procedure of our Meta-APR is summarized in Al- gorithm 1. In the low-resource setting, we set the size of support and query sets 10 and lever- age support sets for the inner loop update. The query sets are used to track the meta-loss and not involved in parameter updating.
TSSB-3M (Richter and Wehrheim, 2022) is a dataset of over 3 million isolated single statement bug fixes across 23 error types. Each bug fix is as- sociated with commit in an open-sourced Python project that does not modify source code in other files or statements. We randomly down-sample by 10% for each error type.
To facilitate future research in this new field, we release our curated error-specific low- resource APR datasets https://github. com/ wang-weishi MaR See Appendix A.l for more detailed statistics.
Low-Resource APR Adaptation After the meta- training, we adapt Meta-APR to the target low- resource APR tasks via directly finetuning the meta-learned global APR model on few-shot train- ing samples. Such meta-learned APR model is expected to capture error-specific knowledge by providing a better model initialization, which en- ables faster adaptation to fix low-resource bugs. In finetuning, the objective is to minimize the cross-entropy loss between model predictions and ground-truth fixes.
Data Preprocessing As discussed in $3.2, we process all three benchmarks to create high- resource and low-resource APR tasks based on the number of bug-fix in each error type. The data statistics are reported in Table 1. We further pro- vide bug-fix examples for each benchmark in Fig. 2. To prepare the source input to Meta-APR, we fol- low Berabi et al. (2021) to combine error type, error message, and error context into single piece of text in the following format:
4 Experimental Setup
4.1 Error-Specific APR Dataset
fix ferror type) (error message) (error context)
ManySStuBs4J (Karampatsis and Sutton, 2020) has small and large versions comprising 10,231 and 63,923 bug-fix pairs respectively in Java. It is orga- nized at the level of the single statement changes for each bug-fix pair. We consider ManySStuBs4] large with 14 error types in this work.
where error context consists of the given localized error line and its two neighboring code lines form buggy code patch. The corresponding fixed line is used as the target sequence.
4.2 Metrics and Baselines
TFix (Berabi et al., 2021) is a large-scale pro- gram repair dataset that consists of ground truth repair code patch for each buggy patch in JavaScript. It focuses syntax and stylistic errors from open-source GitHub commits, which com- prise 104,804 fug-fix pairs. Among them, 52 error types are detected by a static analyzer ESLint (Tó- masdóttir et al., 2020).
Metrics Following the common practice (Berabi et al., 2021), we use the Exact Match (EM) accu- racy to measure the APR performance. Specifically, EM requires the prediction to be identical to the ground-truth fix, which can reflect how well model predictions are aligned with historic correct fixes from human developers. EM is commonly utilized to uphold correctness standards, especially in cases where static analyzers or unit tests are not available.
"https://eslint /ttps::///lnit.rggg//

---

Baselines We compare Meta-APR with three learning settings: 1) only finetuning on low- resource bugs; 2) transfer-learning from high- resource to low-resource bugs; and 3) multi-task learning on both high-resource and low-resource bugs with or without upsampling strategies. Specif- ically, for the transfer-leariing baseline, we first finetune the model on the high-resource training data and then have another stage of finetuning on the low-resource training data. Under the multi- task learning setting, we jointly finetune our models on a mix of both high-resource and low-resource training data.
Method
Shot = 100
Shot = 50
Shot = 10
Shot =
Low remoucee finetuning
CodeBERT
3.64
UniXender
42.28
33.25
18.24
0.00
CodeTS
42.74
36.10
9:24
0.00
Tearnie--earning
CodeBERT
43.34
37.08
34.02
15.82
UniXeoder
53.43
47. 7666
34.87
18.10
CodeT3
55.23
49.91
38.49
18.28
Mult-iask learning
Codel1
53.78
46,75
35.81
CodeT5 upsampling
$8.98
52.73
41.09
Mett-learning
Meta-APR
59,44
54.34
42.04
22.50
-MAML
58.10
$3.00
41.69
23.02
-UnXoodder
$4.48
48.22
36.77
19.68
Table 2: Results low-resource ManySStuBs4J.
Besides, compare with other code pre- trained models the backbone model, which in- clude encoder-only CodeBERT (Feng al., 2020), decoder-only UniXcoder (Guo et al., 2022), and encoder-decoder CodeT5 (Wang et al., 2021). l). For our Meta-APR method, we perform two abla- tion studies by replacing either the Reptile meta- learning approach to MAML or replacing the back- bone model C5 into UniXcoder to verify the effectiveness of our design choices.
Method
Shot 100
Shot 50
Shot 10
Shot
Low-re finctuning
CodeBERT
3.05
0.00
UniXcoder
33.79
28.14
16.39
0.00
Codel'S
35.61
29.07
13:05
0.00
Transfer-eorriing
CodeBERT
21.80 19
10.89
3.35
UniXonder
40.15
35.28
24.98
15.80
CodeT5
46.91
43.05
31.23
13.57
Multi-ius leurning
CodeT5
46.47
41.26
30.30
Codel3 upsampling
46.84
41.38
29.41
Metu-learning
4.3 Implementation Details
Meta-APR
47.77
44.83
35.28
24.72
-MAMI.
47.03
44.09
34.95
20.82
We implement Meta-APR based on the deep leamn ing framework PyTorch3. We employ CodeT5- base* with 220M parameters as our backbone model. All of our experiments are conducted on a single NVIDIA A100-40GB GPU. We use the library Higher (Grefenstette et al., 2019) to meta- train the model on high-resource error types for 50 epochs with batch size of 10, where the first 5 instances work the support set and the remaining 5 instances are query set. For inner loop gradient updates, we use the SGD optimizer with an inner loop learning rate a of le-4. For the global gradi- ent updates, we use the AdamW (Loshchilov and Hutter, 2019) optimizer and set the outer loop learn- ing rate 3 to 5c-5. Moreover, in the meta-traine voca-training stage, we warm up the first 1000 steps with linear decay. The meta update step size M is set to 150, 20, 150 for TFix, ManySStuBs4J, and TSSB-3M respectively. For low-resource APR adaptation, finetune the meta trained model for 50 epochs on low-resource error types with batch size of 25 and a learning rate of 5c-5. For testing, we select the checkpoint which has the best EM on held-out validation set.
-UmXeoder
40.85
35.58
25.58
15.61
Table Results on low-resource TSSB-3M
Experimental Results and Analysis
In this section, we compare Meta-APR with other code pretrained models different training settings on set of our curated low-resource error-specific APR tasks from three benchmarks ($5.1), followed by detailed analysis on the effects of different er- ror types and token length ($5.2), and pilot study to compare with closed-sourced large language model such ChatGPT in fixing these challenging low-resource bugs ($5.3).
5.1 Low-Resuurce APR Performance
Tables 2 to 4 present the results of exact match (EM) accuracies on ManySStuBs4J, TSSB-3M, and TFix benchmarks respectively at different low- resource settings. We can observe that Meta-APR consistently outperforms other baselines in various few-shot settings across 3 benchmarks in different programming languages. Among different mod- els, we find that CodeT5 achieves consistent per- formance gains over CodeBERT and UniXcoder in most cases, demonstrating that it can serve as
hhttps://github. www.:/ttthuucoosaaaaaaeaeeeeeaae wtt.s///wwoo/e///aaaaaaoooo

---

Method
Shot 100
Shot 50
Shot 10
Shot
Low resounce jinetuning
Col:BERT
13.15
Cumulettve Traction
03
UniXeoder
45.11
41,53
27.51
0.00
CodeTS
45.85
40.18
24.58
0.00
Tramnie--looning
CodeBERT
42,80
38,86
26.66
16.38
UniXooder
46.64
44,89
32.75
18.31
02
Exoct Match
Code13
51.02
34.26
21.44
Necxupps connet
Metx-PP2 Wrong
Mult-tosk learning
CodeT5
$0,60
47.29
34.41
Number of tokiens
CodeTS + upsampling
51.39
47.58
36.28
Mets-learnig
Meta-AP
51,63
48.06
39.50
24.38
-MAML
30.56
47.38
37.09
21.71
-+ -UniXeoder
47.45
44.98
34.37
17.66
Exeuct Match (S)
30
Table 4: Results on low-resource TFix.
better backbone model for APR tasks with encoder-decoder architecture.
Among different learning paradigms, we find that transfer-learning from high-resource to low- resource bugs and multi-task learning on both bugs yield much better results compared to di- rectly finetuning low-resource bugs, validating our assumption that low-resource APR can benefit from the bug-fixing knowledge learned from high- resource bug-fix data. These two approaches gener- ally exhibit comparable performance across differ- ent benchmarks, and the upsampling strategy often proves to be helpful in multi- task learning. Over- all, our Meta-APR further improves the adaptation from high-resoure to low-resource bugs, thereby leading to superior APR performance. Notably, the performance gain of Meta-APR over other learning paradigms becomes more significant when there are fewer or even no low-resource training samples available. This implies that Meta-APP is able to learn a better model initialization that captures the error-specific knowledge, thereby enabling faster adaptation to the target low-resource error types.
Number of tollns
(b)
Figure (a): cumulative fraction of programs by num- ber of tokens in the source buggy patch, grouped by whether Meta-APR can have a correct fix. (b): distri- bution of correct fix over number tokens for low- resource finetuning and transfer-learning from high- resource to low-resource bugs and our Meta-APR.
also more efficient than MAML's second-order meta-learning approach.
5.2 Further Analysis
We proceed to analyze the model predictions better understand our Meta-APR behaves in fixing various bugs compared to other approaches. All results in this section are under 100-shot setting.
Effect of Bug Sequence Length We analyze how Meta-APR performs in fixing low-resource bugs with varying numbers of bug tokens. Fig. 3a shows the cumulative fractions of bugs by their number of tokens, grouped based on the Meta-APP repair out- come (EM). Comparing the blue and orange lines. we observe that the blue one consistently above the orange one, and if we select a fixed cumulative fraction based on the y-axis, the blue line (correct fixes) will have fewer tokens (i.e. shorter) than the orange one (wrong fixes), indicating the bugs successfully fixed by Meta-APR tend to be shorter than the ones that are incorrectly fixed. We further compare Meta-APR with other training strategies, based on the same backbone model of CodeT5, fixing bugs with various lengths in Fig. 3b. We
We consider variants of Meta-APR verify the design choices in our proposed framework, where " "*MAML" means that we replace the first-order meta-learning algo- rithm with second-order meta-learning approach MAML, and "nninixcoder" means that we change the backbone model CodeT5 to UniXcoder. From the results, find that both CodeT5 and the first- order meta-learning algorithm are important in en- hancing low-resource APR performance, observed by a consistent performance drop from these two variants in most settings across 3 benchmarks. Note that our Meta-APR's 's first-order meta- meta-learning is
Ablation Study

---

Trandeclemmnng
Nete-A7R
codda ClientTestãase (unittast: (as.tttttt..ettsreee PestCiaee
100
self. assertTrue clainetanos (modals,
Inpur Promppt
List (xodels k)ys
Eloct Match Score (Nu
celf. asserttrue (len(keys)
Check zandon. (rodels a"a13ss waa.aas_ssa1aaCa_ (""ten") "corponett"" celf
self bupgy "aki356_5a""
Ourput
s0lE. a2900tinn "ax1355 Sa
keys)
Figure 6: One example of low-resource TSSB-3M.
(1000))
1531
E.124)
F.2
(144)
Type
Figure 4: Distribution of correct fix on 9 low-resource error types from ManySStuBs4J. Number of bugs for each type is included the parentheses at the x-axis, The details of error type A-H can be found in Table 5.
tion through error-specific meta-training compared to the default transfer-leariing approach, we visu- alize the embeddings of both high-resource and low-resource bugs after the high-resource finetun- ing from transfer-learnig and Meta-APR in Fig. 5. We observe that Meta-APR representations between high-resource and low- resource bugs so that they are distributed in a closer distance in the embedding vector space, en- abling faster adaptation from high-resource to low- resource bugs with limited training samples.
better align the
Case Study We provide qualitative examples from our multilingual low-resource APR bench- marks in Fig. 2. We find that our Meta-APR able to fix the bugs using various fix operations such as deletions, boolean conversion, and identifier renam- ing, while the standard transfer-leaming approach fails fix bugs by simply copying the buggy line as the fixed line. This indicates Meta-APR, can enable faster and better adaptation to low-resource APR scenarios.
(a) Transterieamning 2-D embedsing plot
(b) Mota APR 2:0 embodding piot
Figure 2-D visualization of embeddings for high- resource bugs (blue) and low-resource bugs (red).
observe a monotonous performance decline when the bug sequence length increases for models, suggesting that shorter bugs are easier to be fixed which might be due to their limited complexity.
Effect of Error Type We further analyze how Meta-APR performs in addressing various error types. Fig. presents breakdown of results by error type for ManySStuBs44J, where we can find a notable variance in performance across different error types. For instance, Meta-APR achieves more than 50% EM score on types A, C, and E, while it achieves around 10% more EM score on type B. Overall, we observe that Meta-APR is more robust APR method, consistently outperforming other finetuning strategies. Interestingly, finetuning only on low-resource bugs achieves comparable performance to Meta-APR fixing bug type F (add throws exception) and G (delete throws excep- tion). These bugs relate to the decision to add or remove 'throws' clause in a function declaration, implying that such error types comprise easy-to-fix bugs and require only few training samples.
5.3 Comparison with ChatGPT
Recent studies (Prenner and Robbes, 2021; Joshi et al., 2022) have shown that large language models (LLMs) are capable of bug fixing zero-shot/few- shot settings. In order to investigate their perfor- mance in fixing challenging low-resource bugs, we use ChatGPT (GPT-3.5-TTurbb) and evaluate it on 80 randomly sampled test bug- fix pairs for each benchmark. As illustrated in Fig. 6, we construct the shot prompt to provide the code context and its buggy line, together with instruction "fix the buggy line:"" Besides, we randomly select one bug-fix fix pair from the same error type to design the one-shot prompt for in-context learning.
We report the comparison results in Fig. 7. We observe that Meta-APR significantly surpasses
ChatGPT in both zero-shot/one-she settings across
3 tasks. This shows that ChatGPT is still lim-
Representation Visualization To understand how Meta-APR learns a better model initializa-
http:://chat..peeee com/chat cttcaccccccccccccaatt.t.e..a

---

Zerosbat
ChaIGPT Greehut
NetavlRR
Limitations
50
As we are the first to investigate the low-resource APR tasks, we curated 3 datasets with different low-resource degrees (i.e., shot=10/50/100 from existing APR benchmarks to support our study. Such data construction will have data quality de- pendency issue from those original APR datasets. Besides, the low-resource sub-sampling may in- troduce some randomness issues. To mitigate this issue, we performed multiple rounds of random sampling with different seeds and reported the av- erage results. Furthermore, to evaluate the APR performance, we employ exact match scores the metric to compare the predicted fixes with the ground-truth fixes written by developers, which might fail capture other correct fixes with differ- ent formats and styles.
Numbat of conroit
40
30
20
22
ManySStulbba)
Trix
1XMD-34 SM4
Figure 7: Evaluation results of correct fixes on subset of 80 bugs from the test. data across three benchmarks.
ited to handle the challenging low-resource bugs as it did not see much such bug-fix data during training due to the data scarcity issue. Addition- ally, we find that the one-shot example is not al- ways beneficial for low-resource APR and might introduce some noises compared zero-shot ting. substantially improves the performance on TFix but leads to some performance degrades on ManySStuBs4J and TSSB-3M. By inspecting the predictions, we find that ChatGPT often pre- dicts "no bugs" as it might require more semantic information for decision decision-making. Besides, Chat- GPT performs pretty well in fixing bugs related to syntax errors such as the error type of "no unsafe negation"", which is fix the bug by simply adding parentheses to an expression after negation oper- ator. This is probably due to the fact that ChatGPT has been pretrained on a large-scale code corpus and can understand the program syntax well.
Ethics Statement
Our work complies with ACL Ethics Policy. this work, we construct our datasets using pub- licly available APR benchmarks, which are widely used to examine the program repair performance. We provide detailed procedures to create our low- resource APR datasets and provide proper citations their source benchmarks. We will publicly lease our curated datasets with the same licenses their source datasets. As an APR tool, one potential risk of Meta-APR is that the predicted fixes from Meta-APR cannot be guaranteed to be correct, and directly adopting them without manual checking could cause security risks to the software develop- ment. We suggest that all the fixes should have manual check from experts before real adoption.
6 Conclusion
this work, we present Meta-APR, a simple yet effective framework that extends CodeT5 with meta-learning for low-resource APR. It is model- agnostic framework that can be integrated with any learning-based models. To the best of our knowl- edge. we the first to investigate APR in the low- resource setting and curate error-specific datasets different low-resource degrees from three APR benchmarks in Python, Java, and JavaScript. Com- prehensive experiments have verified the superior- ity of Meta- APR over other learning strategies with various code pretrained models. More analysis shows that Mcta-APR can better align the represen- tations of high-resource and low-resource bugs, and fix bugs with various sequence lengths and error types. pilot comparison with ChatGPT further shows that our Meta-APP is still more capable of fixing these challenging low-rew rewrresurce bugs.
References
Samy Bengio. Yoshua Bengio, Jocelyn Cloutier, and Jan Gescei. 2013. On the optimization of synap- tic learning rule. In Optimality in Biological and Artificial Networks?, pages 281-303. Routledge.
Berkay Berabi, Jingxuan He, Veselin Raychev, and Mar- tin T. Vechev. 2021. Thi:: Learning to fix coding entors with text-to-text transformer. In Proceed- ings of the 38th International Conference on Ma- chine Learning, ICMI. 2021, 18-24 July 2021, Vir- tual Event, volume 139 of Proceedings of Machine Learning Research, pages 780-791. PMLR.
Nghi Bui, Yue Wang, and Steven C. H. Hoi. 2022. Detect-loccaliz--repai: A unified framework for leam- ing debug with codet5. Findings of the Associ- ation for Computational Linguistics: EMNLP 2022,

---

Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages $12-823. Association for Computational Linguistics.
of Findings of ACL. pages 1536-1547, Association for Computational Linguistics.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the 34th Inter- national Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Re- search, pages 1126- 1 1135. PMLR.
Yitian Chai, Hongyu Zhang. Beijun Shen, and Xiaodong Gu. 2022. Cross- Cross-domain deep code search with few- shot meta learning. CaRR, abs/2201.00150..
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov. Heidy Khlaaf, Girish Sas- try. Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Vosss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji, Shantamu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Nya Sutskever, and Wojciech Zaremba. 202 la Evaluat- ing large language models trained on code. CoRR, .3.277
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Longuage Processing, ACI/J/NLPP 2021, (Volume I: Long Papers). Virtual Event, August 1-6, 2021, pages 5816-3830 Association for Computational Linguis- tics.
Luca Gazzola, Daniela Micueci, and Leonardo Mariani. 2019. Automatic software repair: A survey. IEEE Trans. Software Eng., 45(1):34-67.
Claire Le Goues. Thanh Vu Nguyen, Stephanie Forrest, and Westley Weimer. 2012. Genprog: A generic method for automatic software repair. IEEE Trans. Software Eng., 38(1):54-72 72.
Edward Grefenstette, Brandon Amos, Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, and Soumith Chin- tala. 2019. Generalized inner loop meta-leaming. CoRR, abs/1910.01727.
Zimin Chen, Steve Kommrusch, Michele Tufano, Louis- Noël Pouchet, Denys Poshyvanyk, and Martin Mon- perrus. 2021b. Sequencer: eqquence-to-sequenae learning for end-to-end program repair. IEEE Trans. Software Eng.. 4799):1443-199..
Jiatao Gu, Yong Wang, Yun Chen, Victor O. K. Li, and Kyunghyun Cho. 2018. Meta-learning for low- resource neural machine translation. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 3622-3631. Association for Computational Linguistics.
Jacob Devlin, Ming-Wi Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLT 2019. Minneupolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang. Ming Zhou, and Jian Yin. 2022. Unixcoder: Unified cross- modal pre-training for code representation. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7212-7225 Association for Computa- tional Linguistics.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xi- aodong Liu, Yu Wang. Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Pro- cessing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019. December 14, 2019, Vancouver, BC, Canada, pages 13042 13054.
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svy- atkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin B. Clement, Dawn Drain, Neel Sundare- san, Jian Yin. Daxin Jiang, and Ming Zhou. 2021. Graphcodebert: Pre- traiiiaan code representations with data flow. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xi- aocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Code- hert: A pre-trained model for programming and nat- ural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020
Jiajun Jiang, Yingfei Xiong, Hongyu Zhang, Qing Gao, and Xiangqun Chen. 2018. Shaping program repair

---

space with existing patches and similar code. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018, Amsterdam, The Netherlands, July 16-21, 2018, pages 298-309. ACM.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Li- dong Zhou, Linjun Shou, Long Zhou, Michele Tu- fano, Ming Gong, Ming Zhou, Nan Duan, Neel Sun- daresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021. Codexglue: A machine learning bench- mark dataset for code understanding and generation. In Proceedings of the Neural Information Process- ing Systems Track on Datasets and Benchmarks I, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual.
Nan Jiang, Thibaud Lutellier, and Lin Tan. 2021. CURE: code-aware neural machine translation for automatic program repair. In 43rd IEFF/ACM Inter- national Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021, pages 1161 1173. IEEE
Thibaud Lutellier, Hung Viet Pham, Lawrence Pang, Yitong Li, Moshi Wei, and Lin Tan. 2020. Coconut: combining context-aware neural translation models using ensemble for program repair. In ISSTA '20: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, USA, July 18-22 2020, pages 101 101-114.440 ACM.
Magne Jørgensen and Martin J. Shepperd. 2007, A systematic review of software development cost esti- mation studies. JEEE Trans. Software Eng., 33(1):33- 53.
Harshit Joshi, José Pablo Cambronero Sánchez, Sumit Gulwani, Vu Lc, Ivan Radicek, and Gust Verbruggen. 2022. Repair is nearly generation: Multilingual pro- gram repair with llm.. CoRR, abs/2208. 1640.
Florian Lux and Ngoc Thang Vu. 2022. Language- agnostic meta-learning for low-resource text-to- speech with articulatory features. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), ACI. 2022, Dublin, Ireland, May 22-27, 2022, pages Association for Computational Linguis-
Ratael-Michael Karampatsis and Charles Sutton. 2020. How often do single-statement bugs occur?: The manysstubs4j dataset. In MSR '20: 17th Interna- tional Conference Mining Software Repositories, Seoul, Republic of Korea, 29-30 June, 2020, pages 573-577. ACM.
tics.
Matias Martinez and Martin Monperrus. 2018. Ultra- large repair search space with automatically mined templates: The cardumen mode of astor. In Cearch-Basee Software Engineering 10th Interna- tional Symposium, SSBSE 2018, Montpellier, France, September 8-9, 2018, Proceedings, volume 1036 of Lecture Notes in Computer Science pages 65-86. Springer.
Anil Koyuncu, Kui Liu, Tegawendé F. Bissyandé, Dong- sun Kim, Jacques Klein, Martin Monperrus, and Yves Le Traon. 2020. Fixminer: Mining relevant fix patterns for automated program repair. Empir. Softw: Eng. 25(3): 1980-1990-222 2024.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu-Hong Hoi. 2022. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. In NeurIPS.
Fei Mi, Minlie Huang, Jiyong Zhang, and Boi Falt- ings. 2019. Meta-learning for low-resource natural language generation in task-oriented dialogue sys- tems. In Proceedings of the Twenty-Eihhth Interna- tional Joint Conference on Artificial Intelligence, IJ- CAI 2019, Macao, China, August 10-16, 2019, pages 3151-3157. ijcai.org.
Shuai Lin, Pan Zhou, Xiaodan Liang, Jianheng Tang, Ruihui Zhao, Ziliang Chen, and Liang Lin. 2021. Graph-evolving meta-lcarning for low-re wwwresource med- ical dialogue generation. In Thirty-Fifth AAAI Con- ference on Artificial Intelligence, AAAI 2021, Thirty- Third Conference on Innovative Applications of Arti- ficial Intelligence, IAAI 2021, The Eleventh Sympo- sium on Educational Advances in Artificial Intelli- gence, FAAI 2021, Viriual Event, February 2-9, 2021, pages 13362- 13370. AAAI Press.
Hoang Duong Thien Nguyen, Dawei Qi, Abhik Roy- choudhury, and Satish Chandra. 2013. Semfix: pro- gram repair via semantic analysis. In 35th Inter- national Conference on Software Engineering, ICSE 113, San Francisco, CA, USA, May 18-26, 2013, pages 772-781. IEEE Computer Society.
Kui Liu, Anil Koyuneu, Dongsun Kim, and Tegawendé Bissyandé. 2019. Tbar: revisiting cemplate-based automated program repair. In Pro- ceedings of the 28th ACM SIGSOFT International Symposium on Softwre Testing and Analysis, ISSTA 2019, Beijing, China, July 15-19, 2019, pages 31 31-42. ACM.
Alex Nichol, Joshua Achiam, and John Schulman. 2018. On first-order meta-learning algorithms. CoRR, 803.02999.
OpenAL. Chatgpt. 2022.
Cheonbok Park, Yunwon Tae, Taehee Kim, Soy- oung Yang, Mohammad Azam Khan, Lucy Park, and Jacgul Choo. 2021. Unsupervised neural ma- chine translation for low-resource domains via meta- learning. In Proceedings of the 59th Annual Meeting
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization, In ICLR (Puster). Open- Review.net.

---

of the Association for Computational Linguistics and the 1ith International Joins Conference on Natural Language Processing, ACI/JJCNPP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pages 2888-2901 Association for Computational Linguis- tics.
Ricardo Vilalta and Youssef Drissi. 2002. A perspective view and survey of meta-learning. Artif. Intell. Rev. 18(2):77-95.
Weishi Wang, Yue Wang, Shafiq Joty, and Steven C. H. Hoi. 2023a. Rap-gen: Retrieval-augmented patch generation with codet5 for automatic program repair. CoRR, abs/2309.06057.
Strategie Planning. 2002. The economic impacts of in-
adequate infrastructure for software testing. National Institute of Standards and Technology, page I.
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023b. Codet5+: Open code large language mod- els for code understanding and generation. CoRR, abs/2305.07922.
Julian Aron Prenner and Romain Robbes. 2021. Auto- matic program repair with openai's 's codex: Evaluat- ing quixbugs. CoRR, (bs/2111.03922,
Yuhua Qi, Xiaoguang Mao, Yan Lei. Ziying Dai, and Chengsong Wang. 2014. The strength of random search automated program repair. In 36th Inter- national Conference on Software Engineering, ICSE '14, Hyderabad, India May 31- June 07, 2014, pages 254-265. ACM.
Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code under- standing and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 8696-8708. Association for Computa- tional Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, Nly Sutskever, et al. 2018. Improving language under- standing by generative pre-training.
Cathrin WeiB, Rahul Premraj, Thomas Zimmermann, and Andreas Zeller. 2007. How long will it take to fix this bug? In Fourth International Workshop on Mining Software Repositories. MSR 2007 (ICSE Workshop), Minneapolis, MN, USA, May 19-20, 2007, Proceedings, page 1. IEEE Computer Society.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn Res., 21:140:1-140:67
Moiz Rauf, Sebastian Padó, and Michael Pradel. 2022. Meta learning for code summarization. CORR, abs/2201,0$$ 10.
Emily Winter, Vesna Nowack, David Bowes, Steve Counsell, Tracy Hall, Sæmundur Óskar Haraldsson, and John R. Woodward. 2023. Let's talk with devel- opers, not about developers: A review of automatic program repair research. IEEE Truns. Software Eng., 49(1):415 (111):4491111 436
Cedric Richter and Heike Wehrheim. 2022. TSSB-3M:
mining single statement bugs at massive scale. In 19th JEEE/ACM International Conference on Mining Software Repoxitories, MSR 2022, Pittsburgh, PA, USA, May 23-24, 2022, pages 418 422. ACM.
Rui Xie, Tianxiang Hu, Wei Ye, and Shikun Zhang. 2022. Low-resources spejecc-specific code summa- rization. In 37th IEEF/ACM International Confer- ence Automated Software Engineering, ASE 2022, Rochester, MI, USA, October 10-14, 2022, pages 68:1-68:12 12. ACM.
Ilya Sutskever, Oriol Vinyals, and Quee V Le. 2014. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27.
Kristín Fjóla Tómasdóttir, Mauricio Finavaro Aniche, and Arie van Deursen. 2020. The adoption of javascript linters in practice: A case study on eslint. IEEE Trans. Software Eng., 4668).863-891.
Jifeng Xuan, Matias Martinez, Favio Demarco, Maxime Clement, Sebastian Lamelas Marcote, Thomas Durieux, Daniel Le Berre, and Martin Monperrus. 2017. Nopol: Automatic repair of conditional state- ment bugs in java programs. IEEE Trans. Software Eng., 43 15:34-55.
Michele Tufano, Cody Watson, Gabriele Bavota, Massi- miliano Di Penta, Martin White, and Denys Poshy- vanyk. 2019. An empirical study on learning bug- fixing patches in the wild via neural machine transla- tion. ACM Trans. Softw, Eng. Methodol., 28(4):19:1- 19:29.
Quanjun Zhang, Chunrong Fang, Yuxiang Ma, Weisong Sun, and Zhenyu Chen. 2023. A survey of learning-base based automated program repair. CORR. 0a//2301.03270
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, 7, Long Beach, CA, USA, pages 5998-6005 -608..
Quanjun Zhang, Yuan Zhao. Weisong Sun, Chunrong Fang, Ziyuan Wang, and Lingming Zhang. 2022. Program repair: Automated vs. manual. CoRR, abs/2203.05166.
Yingxiu Zhao, Zhiliang Tian, Huaxiu Yao, Yinhe Zheng. Dongkyu Lee, Yiping Song, Jian Sun, and Nevin L.

---

Zhang. 2022. Improving meta-learning for low- resource text classification and generation via mem- ory imitation. In Proceedings of the 60th Anmual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), ACL 2022, Dublin, Treland, May 22-27, 2022, pages 583-595 Associa- tion for Computational Linguisties.
A
Appendix
A.1
More Dataset Statistics
We provide the detailed statistics of our curated low-resource APR benchmarks in Table 5, Table 6, and Table for ManySStuBs4J, TSSB, and TFix respectively. We can observe very imbalanced error type distribution across these benchmarks.
Qihao Zhu, Zeyu Sun, Yuan-an an Xiao, Wenjie Zhang, Kang Yuan, Yingfei Xiong, and Lu Zhang. 2021. A syntax-guided edit decoder for neural program repair. ESEC/FSE '21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, August 23-28, 2021, pages 341-353. ACM.

---

Error Type
Train
Valid
Test
CHANGE IDENTIFIER
10.175
1,250
1,250
12,675
OVERLOAD METHOD OVERLOAD.METTOO SECTHOD_MORE_ARGE
2,734
338
339
3,411
CHANGE NUMERAL
2,694
336
336
3,366
High-resource
CHANGE MODIFIER
2,580
322
322
3,224
MORE SPECIFIC IF CHANGE OPERATOR
1,028
128
128
1,284
1,014
126
127
1,267
LESS SPECIFIC IF (E) SWAP BOOLEAN LITERAL (B)
968
121
121
1,210
871
109
109
1,089
OVERLOAD METHOD DELEEEDD ARGS (C)
790
98
97
985
CHANGE, CALLER IN FUNCTION CECCTIONNCALLLLCC (A)
712
89
89
890
Low-resouree
CHANGE_ GEANGE_UNARY LORRRROROROTORRRR (D)
511
64
63
638
DELETE. THROWS EXCEPTION (G)
328
41
41
410
SWAP_ARS AAA_MERNSSNTS (H)
304
38
37
379
ADD THROWS EXCEPTION (F)
98
12
122
Table 5: Statistics of ManySStuBss4] benchmark.
Erro Type
Train
Valid
Test
All
SINGLE STMT STMM
24,346
3,044
3,043
30,433
CHANGE, BRANGE_STRING STRING LITERAL
14,285
1,786
1,785
17,856
CHANGE IDENTIFIER USED
5,035
630
629
6,294
CHANGE BINARY OPERAND
3,434
430
429
4,293
SAME FUNCTION MORE, ARGS
3,061
383
383
3,827
WRONG FUNCTION NAME
2.813
352
351
3,516
CHANGF NUMERIC LITERAL
2,480
310
3,100
High-resource
ADD FUNCTION AROUND EXPRESSION
2.318
290
290
2,898
CHANGF, ATTRIBUTF_USSS
2,206
276
2,758
SINGLE_TOKEN TOKEN ADD METHOD, CALL
1,895
237
2,369
1,290
1,612
MORE_SPECITE SPECIFIC IF
1,101
1,376
ELEMENTS ITERABLE SAME_ FWNCFUFNCTONNOO
1,070
134
1,337
1,050
131
1,312
CHANGE BOOLEAN LITERAL
907
114
113
1.134
ADD ATDRATUBBUEEEAAC ACCESS
716
90
89
895
CHANGE BINARY OPERATOR
681
85
851
SAME FUNCTION, WRONG CALLER
$58
Low-resource
CHANGE KEYWORD ARGUMENT USED
470
59
588
LESS LESS_ SPECIFIC_IF
382
48
CHANGE_ JUNNGE OPERATOR
318
SAME FUNCTION, SWANN
SWAP_ARGS ARGS
150
187
CHANGE CHANGE_CONSTIOE CONSTANT TYPE
118
Table Statistics of TSSB benchmark.

---

Type
Train
Valid
Test
no-invalid-this
13,101
1,456
1,609
16,166
no-undef
8,614
958
1.064
10,636
no-umused-vars
6,289
699
777
7.765
comma-style
5,180
576
639
6,395
no-redeclare
5,167
575
639
6,381
no-extra-semi
4,834
5,969
no-unreachable
3,826
4,725
prefer-rest-pe rertrrraat--ssss
3,675
454
4,534
no-debugger
3,372
no-throw-literal
3,300
408
4.075
High-resource
guard-for-in
2,616
3,231
no-console
2,484
3.067
no- co-useless-cscape
2,364
293
2,920
prefer-spread
2,001
221
244
2,466
no-dupe-keys
1,765
197
219
2,181
no-emply
1,665
184
206
2,055
no-process-exit
1,225
137
152
1,514
no-cond-assipp
1.194
132
146
1,472
no-extra-boolean-cas
1,180
132
146
1,458
gemerator-staar-spccin
1,130
126
140
1,396
no-constant-conditton
1,016
112
123
1,251
no-array-construccor
793
98
no-inner-decllrations
671
no falfallthoough
601
743
no no-case-de
no-case-declane
dellarttttsss
584
73
723
no-extra-bind
547
no-self-assign
494
61
610
valid-typeof
436
49
54
539
constructor-super
375
42
47
464
no-new-object
360
41
45
446
no-caller
360
41
45
446
no-extend-native
358
40
45
443
require-yicld
347
39
43
429
no co-unsafe-nen negation
342
38
43
423
no-this no-this-befoor--spper
333
38
42
413
no-new-oo -e--ne-wwwappee wrappers
291
33
36
360
Low-resoures
go-global-assign
257
29
32
318
no-const-ussign
224
25
28
277
no-sparse-arrass
191
22
24
237
getter-retum
163
19
21
203
du-dplicccte-csee no
157
18
20
195
ln-unused-labels
151
17
19
187
e--ept-pptteett
144
18
178
no-func-assign
118
147
no-dupe-clas- members
94
117
no-class-assign
89
10
111
use-isnan
56
no-unsafe-finally
50
63
for-direction
40
50
no-ex-assign
32
no-compare-neg-zero
9
13
no-new-symbol
18
10
Table 7: Statistics TFix benchmark.


Reviews:
[{"review_id": "nSrfCJwfyL", "reviewer": "Reviewer_rer7", "summary": "", "questions": "- In Fig 1, what do high and low resources represent? I think it is a bit confusing, as the high-resource is on top of the meta-learning framework. And the right part contains both high and low resource data.\n- Could you please explain in more detail how you “find that the bugs successfully fixed by Meta-APR tend to be shorter than the ones that are incorrectly fixed.” from Fig 3. a?\n- In 4.2, the details of the baselines are provided. However, it is not clear what exactly these baselines are. In Particular, it is not clear what exactly transfer-learning and multi-task learning are.\n- I appreciate that you also considered ChatGPT as the baseline. Is there any particular reason that you did not check the results with two or more shots (examples)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the appreciation and support for our work. We are glad that you recognize our motivation of investigating the pre-trained code models’ capabilities within a meta-learning framework in dealing with low-resource APR tasks, and find our proposed approach technically sound and backed with detailed evaluation results. \n\nWe address your concerns as follows:\n# Q1: “In Fig 1, what do high and low resources represent? I think it is a bit confusing, as the high-resource is on top of the meta-learning framework. And the right part contains both high and low resource data..”\nThanks for flagging this. In Figure 1, the high-resource one refers to the more abundant and common error types that are used to meta-train the backbone model, while the low-resource ones represent the more rare error types that our model adapts into. \n\nIndeed the right part of the figure contains both high (green circles) and low resource (triangles) test data, which is meant to highlight their different distributions. Specifically, the target low-resource bugs are highlighted with ellipses and arrows to represent the transferability from high-resource to low-resource scenarios. We will make it more clear in our revised version.\n\n# Q2: “Could you please explain in more detail how you “find that the bugs successfully fixed by Meta-APR tend to be shorter than the ones that are incorrectly fixed.” from Fig 3. a?”\n\nThe plot in Fig 3 (a) presents a cumulative fraction of programs by the number of tokens in the source buggy patch, grouped by whether Meta-APR can correctly fix it or not: the blue line for the correct fixes and the orange one for the wrong fixes. This aims to showcase the distribution differences of the buggy patches between which Meta-APR can and cannot fix.\n\nAs illustrated in Fig 3 (a), we can observe that the blue line is consistently above the orange one, and if we select a fixed cumulative fraction (say 0.5 in the y-axis), the blue line (correct fixes) will have fewer number of tokens (i.e. shorter) than the orange one (wrong fixes).\n\n# Q3: “In 4.2, the details of the baselines are provided. However, it is not clear what exactly these baselines are. In Particular, it is not clear what exactly transfer-learning and multi-task learning are..”\nThanks for bringing this to our attention. Let us further explain these baselines here and we will make it clearer in our revised version.\nTransfer-learning from high-resource to low-resource bugs refers to that we first finetune the model on the high-resource training data, and then have another round of finetuning on the low-resource training data.\nMulti-task learning on both high-resource and low-resource bugs refers to that we mix these both data and have only one round of finetuning. Under the multi-task learning setting, we further explore ablations of whether to upsampling the low-resource bugs at a similar size of the high-resource ones.\n\nAfter the training, the model will be evaluated on the low-resource test data to examine its low-resource APR capabilities. By considering all these training strategies for comparison, we aim to demonstrate the consistent benefits of our meta-learning strategy.\n\n# Q4: “I appreciate that you also considered ChatGPT as the baseline. Is there any particular reason that you did not check the results with two or more shots (examples).”\n\nThanks for your appreciation. We indeed explored using two and three examples for the few-shot in-context learning with ChatGPT. Specifically, we randomly selected bug-fix examples from the same error type and concatenate them as the prefix prompt for guidance. However, we did not observe significant improvements compared to the one-shot prompt. Besides, we had a similar observation that few-shot examples are not always beneficial for low-resource APR and might introduce some noises compared to zero-shot settings. We will add more discussion in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper investigates the pre-trained code models’ capabilities in dealing with low-resource automatic program repair (APR) tasks. - Furthermore, it proposes to use a meta-learning framework to deal with these tasks.\nIt provides detailed evaluation results to show the effectiveness of the meta-learning framework in dealing with low-resource APR tasks.", "reasons_to_reject": "- There is not a major novelty in the meta-learning framework (However, I think that is a minor issue).\n- There are some presentation issues in the paper, including unclarities in some of the Figs.\n- Some of the baseline details are not clear (Please see “Questions For the Authors”).", "questions_to_authors": "- In Fig 1, what do high and low resources represent? I think it is a bit confusing, as the high-resource is on top of the meta-learning framework. And the right part contains both high and low resource data.\n- Could you please explain in more detail how you “find that the bugs successfully fixed by Meta-APR tend to be shorter than the ones that are incorrectly fixed.” from Fig 3. a?\n- In 4.2, the details of the baselines are provided. However, it is not clear what exactly these baselines are. In Particular, it is not clear what exactly transfer-learning and multi-task learning are.\n- I appreciate that you also considered ChatGPT as the baseline. Is there any particular reason that you did not check the results with two or more shots (examples)?", "paper_topic_and_main_contributions": "**Summary:**\n\nThe paper investigates low-resource automatic program repair (APR) based on the different types of error. It mainly shows that different types of software bugs have an imbalanced distribution, and the APR models often only capture the frequent patterns (error types), and have difficulty to deal with low-resource error types. To deal with this issue, the paper proposes a meta-learning framework to repair the low-resource software bugs. This Meta-APR framework first learns the knowledge from high-resource bugs, and then fine-tuned on the target low-resource bugs.\n\n**Contributions:**\n\n- The paper investigates the pre-trained code models’ capabilities in dealing with low-resource automatic program repair (APR) tasks.\n- The paper proposes a meta-learning framework to generate fixes for low-resource bugs with limited training data.\n- The provided results show that the meta learning framework consistently enhances the performance of the model in low-resource APR tasks. \n- Furthermore, the paper shows that the proposed model perform better than large close source models such as ChatGPT.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NmvVs0YbmL", "reviewer": "Reviewer_yvFS", "summary": "", "questions": "1. In the meta-training process, error types are made visible to the model where it can learn to distinguish between types. Is this setting consistent with your baselines? How would your method work if this information were not available?\n\n2. After fine-tuning the LLMs to carry out Automatic Program Repair, the programs provided are expected to contain one error, which is the default setting of APR LLMs. Have you tried the setting where ChatGPT was explicitly informed about this—\"the following program contains one error\"?\n\n3. Since your low-resource definition is under \"1000 samples\", why not use the naturally-formed low-resource set: all samples as long as they don't exceed 1000? Why have you excluded UniXCoder and CodeBert's performances for multi-task learning?\n\nComments on rebuttal\n\nThe author's response clarified various issues that made it clear the baselines used are fair comparisons and addressed some of our concerns with the paper. However, the task still seems quite artificial where the data had to be artificially limited to illustrate the advantages of the system, and the paper does not introduce any new algorithms or datasets, so I changed the scores from 2,2 to 3,3", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments. We are glad that you acknowledge our novelty to draw attention to the largely unexplored area of low-resource APR, and find our approach of combining LLMs with a meta-learning framework skillful and efficient. We are further encouraged that you recognize our work as a timely and significant contribution to explore LLMs for an interesting and important task of APR.\n\nIt seems that your concerns on soundness derive from the study’s design. We will make every effort to address these points and provide clarification in our revised manuscript. If your confusion or concern is addressed, we request your kind support by changing your stance. \n\n# Q1: “The study's design is somewhat artificial. The authors artificially restrict the number of data points for tasks with more than 100 data points, creating an arguably unrealistic low-resource scenario. For the application of their task, the bug type must be discernible during both the fine-tuning/training and testing phases, deviating substantially from real-world conditions. Testing on a benchmark where the available data is naturally limited, rather than artificially limiting the available training data, would provide better motivation for the method to address a real-world problem.”\n\nWe appreciate your feedback regarding the study's design. We acknowledge your concerns about the potential artificial nature of our study and would like to provide further clarification to elaborate on the rationale behind our methodology. We will add more discussion on the applicability and generalizability of our approach in our revised version.\n\n## Rationale behind the formulation of low-resource APR by controlling the number of data points \nAs you aptly noted, our work seeks to address the novel problem of low-resource APR tasks. In this pioneering endeavour, we faced a significant challenge: the absence of existing datasets that accurately capture the intricacies of this particular problem domain. Given this limitation, we embarked on a methodical exploration of available datasets to establish a foundation for our study.\n\nOur decision to create artificial low-resource scenarios by restricting the number of data points for tasks was born out of necessity. This approach allowed us to simulate low-resource conditions and evaluate our proposed method's effectiveness in situations where data is scarce at different levels (i.e. 10/50/100 instances)—a crucial aspect of our research objective. By focusing on error types and strategically formulating these scenarios, we aimed to capture the essence of low-resource APR tasks while working within the constraints of the data at hand. \n\nAlthough we acknowledge the potential artificiality introduced by this approach, we believed that this was a pragmatic way to initiate the study of low-resource APR tasks in the absence of suitable datasets. Our intention was to spark discussion and exploration in a relatively uncharted research area, paving the way for future investigations that can refine and expand upon our initial findings. We are encouraged to see an explicit recognition on this point from Reviewer A13G that our curated datasets could be useful to the research community in further studying this task.\n\n## Rationale behind the use of error types\nThe rationale behind our use of error types stems from our aim to delineate the distinctions between high-resource and low-resource bug scenarios. We agree that relying solely on error types might introduce limitations, but it is essential to emphasize that this categorization is just one facet of our approach. We firmly believe that the underlying principles of our method are adaptable to other forms of categorization, such as project domains, thereby increasing the applicability of our approach across diverse scenarios.\n\nAdditionally, while error types are not always readily available, we want to highlight that our study was designed to demonstrate the feasibility of obtaining error types from various sources, including static rules, analyzers, dynamic interpreters, and compilers.  In our work, we have demonstrated the effectiveness of our approach across multiple APR datasets in 3 programming languages (PLs) with different definitions of error types. While the availability of error types might vary, we believe that our work underscores the potential utility and generalizability of this approach in addressing low-resource scenarios across various programming contexts, thus mitigating the concern about over-reliance on specific error types and data sources.\n\n# Q2: “While the authors' method proves effective for tasks with minimal data, the improvements over multi-task transfer learning on 100 samples are minor, potentially curbing its efficacy for bug types with larger sample sizes”\n\nOur approach is rooted in the goal of addressing low-resource APR tasks effectively, where data scarcity is a key challenge. We acknowledge your observation that, while our method demonstrates strong performance in scenarios with minimal data, the gains observed over multi-task transfer learning on 100 samples may appear relatively minor. We appreciate your attention to this aspect, and we would like to offer the following insights to provide a more comprehensive understanding of our approach:\n\n* **Focus on Low-Resource Scenarios:** The primary objective of our work is to excel in low-resource conditions, where traditional approaches may struggle due to a lack of sufficient training data. While the improvements in scenarios with 100 samples might appear less pronounced, it's important to note that our method's efficacy becomes increasingly significant as the data scarcity intensifies.\n\n* **Consistent Gains across Multiple APR Tasks:** First, the minor improvements observed in tasks with 100 samples should not overshadow the substantial gains our method demonstrates when operating under extremely low-resource scenarios, such as those with fewer than 50 samples. These scenarios are where our method truly shines, offering solutions in situations where other methods might fail. Besides, we have demonstrated consistent performance gains in low-resource scenarios across various programming contexts and observed similar performance patterns across different numbers of data samples. We believe these findings would be insightful for the research community to further explore this problem.\n\n# Q3: “The experiments are not sufficient. The lack of exposure to high-resource data for low-resource fine-tuning makes the comparison less balanced. The paper does not present performance outcomes for UniXCoder under a multi-task and upsampling scenario. Additionally, the authors only use the Exact Match metric for evaluating the generated repair, which overlooks the potential correctness of fixes that do not exactly match. When available, the authors should also present the Error Removal results.”\n\nYour observations highlight crucial aspects that warrant further clarification and enhancement, and we would like to address each of these concerns comprehensively.\n\n## The lack of exposure to high-resource data for low-resource fine-tuning: \nWe would like to clarify that we did, in fact, conduct the experiment you mentioned. Our methodology involved a comprehensive evaluation under various scenarios, including exposure to high-resource data during the fine-tuning phase in different strategies (see transfer learning and multi-task learning w/ or w/o upsampling in Tables 2,3, and 4). We understand the importance of a balanced comparison, and therefore, this experiment was an integral part of our research design. \n\n## The paper does not present performance outcomes for UniXcoder under a multi-task and upsampling scenario. \n We did indeed assess UniXcoder in multi-task learning and observed its consistent inferior performance compared to CodeT5, echoing findings from other learning scenarios (see finetuning-only and transfer learning in Table 2,3,4). We omitted these results for clarity in presentation. We value your suggestion and will incorporate these results back for a more comprehensive paper.\n\n## Limitations of using Exact Match metric\nWe recognize your concern regarding the limitations of Exact Match. Your suggestion to include Error Removal results is indeed valuable. We plan to explore the Error Removal metric to make our evaluation more comprehensive. For Exact Match, we note that while is a kind of too strict metric that necessitates that fixes precisely match the ground-truth developer-written fixes, it is commonly utilized to uphold correctness standards, especially in cases where static analyzers or unit tests are not available. From the TFix paper, we also find that the Exact Match metric aligns well with the Error Removal metric, providing further validation to the soundness of our results based on the Exact Match metric. \n\n# Q4: The novelty of the work is somewhat limited as it primarily relies on applying the existing Reptile algorithm to the APR task; there are no new algorithms presented.\n\nYour observation about our reliance on the existing Reptile algorithm is valid. We want to take the opportunity to provide a comprehensive understanding of our contributions.\n\nOur novelty indeed extends beyond merely applying the Reptile algorithm. We take pride in formulating a novel task within the realm of APR: the challenge of low-resource APR. This formulation presents a distinctive and meaningful contribution to the field (acknowledged by both you and Reviewer A13G). It entails curating datasets that not only serve our research but also have the potential to benefit the broader research community in further exploring this novel task (acknowledged by Reviewer A13G). These acknowledgments reinforce our belief that our approach is both innovative and valuable. Leveraging Reptile meta-learning with strategic adaptation to address this task presents another contribution of our work. Our focus is on exploring the potential of established methods in novel contexts, rather than presenting entirely new algorithms.\n\n# Q5: In the meta-training process, error types are made visible to the model where it can learn to distinguish between types. Is this setting consistent with your baselines? How would your method work if this information were not available?\n\nYes, the input format (see Line 350) including error types are consistent with all baselines and our methods. We follow the TFix work to employ such input formats and aim to explore how different training methods affect the knowledge transfer from fixing high-resource bugs to low-resource bugs.\n\nWhile we did not explicitly explore an ablated setting where error type information is absent, we appreciate your suggestion which prompts us to further consider whether the model itself can potentially implicitly cluster error types. We will explore this setting and incorporate more ablation results to our revised paper.\n\n# Q6: After fine-tuning the LLMs to carry out Automatic Program Repair, the programs provided are expected to contain one error, which is the default setting of APR LLMs. Have you tried the setting where ChatGPT was explicitly informed about this—\"the following program contains one error\"?\n\nIn our prompt to ChatGPT, we explicitly provide the buggy line and ask the ChatGPT to fix the buggy line. A brief example can be found in Figure 6.\n\n# Q7: Since your low-resource definition is under \"1000 samples\", why not use the naturally-formed low-resource set: all samples as long as they don't exceed 1000? \n\nWe appreciate your insightful suggestion to include a setting of a naturally-formed low-resource set. We want to provide clarification on our choice.\n\nOur decision to use different instance thresholds (10/50/100) for creating low-resource scenarios across multiple datasets is rooted in the desire to control the degree of low-resource more accurately. As we include 3 different datasets, it doesn't account for the varying data distributions within each dataset if directly using the low-resource set cut by the \"1000 samples\" threshold. By defining the low-resource scenarios based on different instance thresholds, we aim to maintain a consistent level of data scarcity across datasets with distinct characteristics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper innovatively draws attention to the largely unexplored area of low-resource automatic program repair, presenting criteria to form low-resource datasets. \n\n2. The authors skillfully utilize the meta-learning method Reptile Algorithm for fine-tuning large language models (LLMs) on high-resource automatic program repair tasks. This enables efficient adaptation to low-resource tasks with a range of 10 to 100 examples.\n\n3. APR is an interesting and important problem and exploring how LLMs can help address it is timely and significant.", "reasons_to_reject": "1. The study's design is somewhat artificial. The authors artificially restrict the number of data points for tasks with more than 100 data points, creating an arguably unrealistic low-resource scenario. For the application of their task, the bug type must be discernible during both the fine-tuning/training and testing phases, deviating substantially from real-world conditions. Testing on a benchmark where the available data is naturally limited, rather than artificially limiting the available training data, would provide better motivation for the method to address a real-world problem.\n\n2. While the authors' method proves effective for tasks with minimal data, the improvements over multi-task transfer learning on 100 samples are minor, potentially curbing its efficacy for bug types with larger sample sizes.\n\n3. The experiments are not sufficient. The lack of exposure to high-resource data for low-resource fine-tuning makes the comparison less balanced. The paper does not present performance outcomes for UniXCoder under a multi-task and upsampling scenario. Additionally, the authors only use the Exact Match metric for evaluating the generated repair, which overlooks the potential correctness of fixes that do not exactly match. When available, the authors should also present the Error Removal results.\n\n4. The novelty of the work is somewhat limited as it primarily relies on applying the existing Reptile algorithm to the APR task; there are no new algorithms presented.", "questions_to_authors": "1. In the meta-training process, error types are made visible to the model where it can learn to distinguish between types. Is this setting consistent with your baselines? How would your method work if this information were not available?\n\n2. After fine-tuning the LLMs to carry out Automatic Program Repair, the programs provided are expected to contain one error, which is the default setting of APR LLMs. Have you tried the setting where ChatGPT was explicitly informed about this—\"the following program contains one error\"?\n\n3. Since your low-resource definition is under \"1000 samples\", why not use the naturally-formed low-resource set: all samples as long as they don't exceed 1000? Why have you excluded UniXCoder and CodeBert's performances for multi-task learning?\n\nComments on rebuttal\n\nThe author's response clarified various issues that made it clear the baselines used are fair comparisons and addressed some of our concerns with the paper. However, the task still seems quite artificial where the data had to be artificially limited to illustrate the advantages of the system, and the paper does not introduce any new algorithms or datasets, so I changed the scores from 2,2 to 3,3", "paper_topic_and_main_contributions": "This paper proposes to apply Reptile algorithm for meta-learning to low-resource automatic program repair (APR). They construct a low-resource training set by sampling 10, 50 and 100 datapoints for various error types across three APR benchmarks. Their results demonstrate that their method is effective with 10, 50, 100 samples per error type and achieves better performance than transfer/multitask learning. Their findings also show that the first-order (Reptile) algorithm outperforms the second-order algorithm (MAML).", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qS4nQSq1Z5", "reviewer": "Reviewer_A13G", "summary": "", "questions": "A) My understanding is that in Reptile, SGD is performed on each task one at a time. That is, the \"Update error-specific APR parameters with gradient\" step in Algorithm 1 would correspond to a specific error type. However, it seems that the batches that are used are randomly segmented from all high-resource error types (Line 284-286), suggesting that each batch could have examples corresponding to many different error types. If this is the case, then which \"error-specific\" parameters are updated?\n\nB) In Lines 112-116, the authors claim that their approach is model-agnostic and generalizable to UniXcoder. In Tables 2-4, I see that is the case when comparing to the low-resource and transfer-learning settings. However, the multi-task learning results are not included for UniXcoder. Could you comment on how the multi-task learning performance compares to the meta-learning setting for UniXcoder?\n\nC) In Section 5.3, the authors claim that ChatGPT performs poorly on APR for low-resource error types. However, it is not clear whether this is specific to low-resource cases or whether ChatGPT is performing poorly with respect to APR in general. Could you comment on how ChatGPT (and Meta-APR) perform with respect to high-resource error types? (Also, it would be nice to see how Meta-APR compares to the other baselines in Tables 2-4 on these high-resource cases.)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the appreciation and support for our work. We are glad that you find our formulated low-resource APR task novel, and our curated dataset useful to the research community in further studying this task. We are further encouraged that you champion our clever design to leverage meta-learning for low-resource APR tasks, and find our work both technically and empirically sound, and backed with very thorough experiments and extensive analysis.\n\nBelow we make every effort to address your questions. If you believe that our rebuttal is satisfactory, it would be great if you could consider increasing your soundness score. \n\n# Q1: “It is unclear to me whether the results in this work will generalize to more realistic settings with software bugs \"in the wild.\" Namely, the authors assume that the error type is labeled, and in fact, this is part of the input to the model (Line 350). In real-world settings, such a label will likely not be available. While one could argue that a classifier could be trained for determining the error type, that may also not be possible in realistic settings in which you have very complex and diverse bugs. ”\n\nWe appreciate your concerns and insights, which have prompted us to provide a clearer perspective on the versatility and applicability of our approach.\n\nThe rationale behind our use of error types stems from our aim to delineate the distinctions between high-resource and low-resource bug scenarios. Regarding the concern about the availability of error type labels in real-world settings, we want to clarify our approach's versatility. While our study demonstrates the feasibility of obtaining error types from various sources, including static rules, analyzers, and interpreters/compilers, we acknowledge that complex and diverse bugs may not always fit neatly into predefined categories. We want to emphasize that the categorization of error type is just one facet of our approach.\n\nWe believe that the underlying principles of our method are adaptable to other forms of categorization, such as project-specific attributes or patterns, thereby increasing the applicability of our approach across diverse scenarios. In our evaluation, we have showcased the effectiveness of our approach across multiple APR datasets in 3 programming languages with different definitions of error types. This reinforces the potential generalizability of this approach in addressing more complex real-world bug scenarios by embracing different categorization strategies and data sources.\n\n# Q2: “Additionally, it seems that one would have to finetune the model separately for each low-resource error type. This is not that feasible in practice, especially as the scale of models increases and the number of error types increase.”\n\nWe did NOT have to finetune the model separately for each low-resource error type. Across our methods and all the baselines, we always combine the low-resource data with different error types together for joint training. We just need to inject the error information into the input.\n\n# Q3: “The meta-training algorithm is a bit confusing and not explained clearly…it seems that the batches that are used are randomly segmented from all high-resource error types (Line 284-286), suggesting that each batch could have examples corresponding to many different error types. If this is the case, then which \"error-specific\" parameters are updated?”\n\nWe will provide more clarifications in the revised paper to address this. Your understanding of batches being randomly segmented from all high-resource error types is correct. For the “error-specific” parameters, they are error-specific in a sense that the error type information is embedded in the input, and the meta-learning algorithm implicitly captures this categorization information when updating the model parameters. In summary, our method design does not require updating the parameters for each type in a batch separately but models multiple types jointly, thereby being more efficient.\n\n# Q4: “Could you comment on how the multi-task learning performance compares to the meta-learning setting for UniXcoder?”\n\nThanks for pointing this out. We did indeed assess UniXcoder in multi-task learning and observed its consistent inferior performance compared to CodeT5, echoing findings from other learning scenarios (see finetuning-only and transfer learning in Tables 2,3 and 4). For UniXcoder’s performance across different learning settings, we found that multi-task learning with upsampling is mostly comparable to the transfer learning baseline and consistently worse than meta-learning across 3 benchmarks. We will incorporate these results back for a more comprehensive paper.\n\n# Q5: “In Section 5.3, the authors claim that ChatGPT performs poorly on APR for low-resource error types. However, it is not clear whether this is specific to low-resource cases or whether ChatGPT is performing poorly with respect to APR in general. Could you comment on how ChatGPT (and Meta-APR) perform with respect to high-resource error types?”\n\nWe appreciate the suggestion to evaluate our methods and other baselines like ChatGPT on high-resource error types. We will evaluate ChatGPT on high-resource error types to provide clarification on this. We want to emphasize two findings of ChatGPT’s common behaviour in the context of APR: 1) ChatGPT often predicts “no bugs” as it might require more semantic information to understand the context; 2)  ChatGPT often performs pretty well in fixing bugs related to syntax errors.\n\nFrom our preliminary evaluation of Meta-APR on high-resource bugs, we observed that both Meta-APR and other baselines such as multi-task learning can achieve good results with comparable performance, indicating that both methods can perform reasonably well when there is abundant training data. However, we want to emphasize that our approach is rooted in the goal of addressing low-resource APR tasks effectively, where data scarcity is a key challenge. The primary objective of our work is to excel in low-resource conditions, where traditional approaches may struggle due to a lack of sufficient training data. From our comprehensive evaluation (Table 2,3,4), we have showcased that our method's efficacy becomes increasingly significant as the data scarcity intensifies.\n\n# Q6: Presentation improvement: more explanations on Reptile and MAML\nThanks for the suggestion! We will elaborate these two meta-learning methods in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors formulate a novel task relating to low-resource automatic program repair (APR) and curate datasets that could be useful to the research community in further studying this task.\n- Leveraging meta-learning for this task is quite clever in my opinion. By demonstrating the advantage of meta-learning over simple finetuning, transfer learning, and multi-task learning, the authors also empirically highlight the value of this approach.\n- The authors conduct very thorough experiments with many different baselines across different learning settings, and they even compare to ChatGPT at a small-scale to demonstrate how their approach stacks up against some of the most recent LLMs.\n- The authors conduct extensive analysis of their results, with respect to length and also the distribution of the latent space. Figure 5 is especially useful in demonstrating how meta-learning helps align representations close together.", "reasons_to_reject": "- It is unclear to me whether the results in this work will generalize to more realistic settings with software bugs \"in the wild.\" Namely, the authors assume that the error type is labeled, and in fact, this is part of the input to the model (Line 350). In real-world settings, such a label will likely not be available. While one could argue that a classifier could be trained for determining the error type, that may also not be possible in realistic settings in which you have very complex and diverse bugs, unlike the single-statement and syntax/stylistic errors that are considered in this work. Additionally, it seems that one would have to finetune the model separately for each low-resource error type. This is not that feasible in practice, especially as the scale of models increases and the number of error types increase. \n- The meta-training algorithm is a bit confusing and not explained clearly. This is one of the main contributions of this work, so this should be further clarified. I hope the authors can answer Question A below during the rebuttal period to provide further clarity on this.", "questions_to_authors": "A) My understanding is that in Reptile, SGD is performed on each task one at a time. That is, the \"Update error-specific APR parameters with gradient\" step in Algorithm 1 would correspond to a specific error type. However, it seems that the batches that are used are randomly segmented from all high-resource error types (Line 284-286), suggesting that each batch could have examples corresponding to many different error types. If this is the case, then which \"error-specific\" parameters are updated?\n\nB) In Lines 112-116, the authors claim that their approach is model-agnostic and generalizable to UniXcoder. In Tables 2-4, I see that is the case when comparing to the low-resource and transfer-learning settings. However, the multi-task learning results are not included for UniXcoder. Could you comment on how the multi-task learning performance compares to the meta-learning setting for UniXcoder?\n\nC) In Section 5.3, the authors claim that ChatGPT performs poorly on APR for low-resource error types. However, it is not clear whether this is specific to low-resource cases or whether ChatGPT is performing poorly with respect to APR in general. Could you comment on how ChatGPT (and Meta-APR) perform with respect to high-resource error types? (Also, it would be nice to see how Meta-APR compares to the other baselines in Tables 2-4 on these high-resource cases.)", "paper_topic_and_main_contributions": "This paper formulates automatic program repair (APR) for low-resource error types and proposes a meta-learning framework to address this. More concretely, Meta-APR is a CodeT5-based model which is first meta-trained on high-resource bug-fix pairs using the Reptile algorithm and then fine-tuned on low-resource error types, in 0-shot, 10-shot, 50-shot, and 100-shot settings. For training and evaluation, three existing datasets with labeled error types (TFix-JavaScript, ManySStuBs4J-Java, and TSSB-M-Python) are curated, such that any error type with less than 1,000 examples is considered low-resource while all others are considered high-resource. Meta-APR is evaluated against baselines in low-resource finetuning, transfer learning, and multi-task learning settings, as well as an alternative meta-learning baseline. Overall, their approach outperforms baselines. Moreover, they conduct a small-scale study against ChatGPT, and they find that their approach outperforms 0-shot and 1-shot ChatGPT for low-resource APR.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- APR parameters -> model parameters\n- Explain Reptile in more detail since that is the backbone of your approach.\n- Explain the MAML baseline more clearly and highlight the major differences."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have contributed to an important and challenging area of program repair, and have raised the important issue of the lack of data in the area. Their evaluations create an artificial scarcity of data which can be critiqued, but their evaluations demonstrating the value of their method are comprehensive. Some writing and figure captions can be clarified to improve presentation and readability.
Decision: Accept-Main