Improving Transformer-based Program Repair Model through False Behavior Diagnosis
Authors: Youngkyoung Kim
Misoo Kim
Eunseok Lee
Venue: EMNLP 2023 Main (2023)
Keywords: Transformer
Program Repair
False Behavior

Abstract: Research on automated program repairs using transformer-based models has recently gained considerable attention. The comprehension of the erroneous behavior of a model enables the identification of its inherent capacity and provides insights for improvement. However, the current landscape of research on program repair models lacks an investigation of their false behavior. Thus, we propose a methodology for diagnosing and treating the false behaviors of transformer-based program repair models. Specifically, we propose 1) a behavior vector that quantifies the behavior of the model when it generates an output, 2) a behavior discriminator (BeDisc) that identifies false behaviors, and 3) two methods for false behavior treatment. Through a large-scale experiment on 55,562 instances employing four datasets and three models, the BeDisc exhibited a balanced accuracy of 86.6\% for false behavior classification. The first treatment, namely, early abortion, successfully eliminated 60.4\% of false behavior while preserving 97.4\% repair accuracy. Furthermore, the second treatment, namely, masked bypassing, resulted in an average improvement of 40.5\% in the top-1 repair accuracy. These experimental results demonstrated the importance of investigating false behaviors in program repair models.

Improving Transformer-based Program Repair Models through False
Behavior Diagnosis
Youngkyoung Kim Department of Electrical and Computer Engineering, Sungkyunkwan University agnes66@skku.c edu
Misoo Kim Department of Artificial Intelligence Convergence, Chonnam National University misoo. aisoo.ii@@juu kim@jnu.ac kr.
Eunseok Lee" College of Computing and Informatics, Sungkyunkwan University leeesêskku. edu
Abstract
Ahmad et al., 2021; Mashhadi and Hemmati, 2021:
Prenner and Robbes, 2021).
Research on automated program repairs using transformer-hase models has recently gained considerable attention. The comprehension of the erroneous behavior of a model enables the identification of its inherent capacity and pro- vides insights for improvement. However, the current landscape research on program re- pair models lacks investigation of their false behavior. Thus, we propose methodology for diagnosing and treating the false behaviors of transformer-basee program repair models. Specifically, we propose 1) behavior vector that quantifies the behavior of the model when it generates output, 2) behavior discrimina- tor (BeDisc) that identifies false hehaviors, and two methods for false behavior treatment. Through large-scale experiment on 55,562 in- stances employing four datasets and three mod- els, the BeDisc exhibited balanced accuracy of 86.6% for false behavior classification. The first treatment, namely, early abortion, success- fully eliminated 60.4% of false behavior while preserving 97.4% repair accuracy. Furthermore, the second treatment, namely, masked bypass- ing, resulted in average improvement of 40.5% in the top-1 repair accuracy. These ex- perimental results demonstrated the importance of investigating false behaviors in program re- pair models.
Because existing APR studies have primarily focused on developing and improving the APR model, the internal prciision-maiing process behind patch generation is often overlooked. Investigating the inner workings of a model can offer valuable in- sights by identifying its capabilities and limitations (Mohammadkhani et al., 2023; Palacio et al., 2023; Borowski et al., 2020; Gunning and Aha, 2019; Voita et al., 2019; Ribeiro al., 201 6a.b; Katuwal and Chen, 2016). Specifically, false behaviors of the model can provide a direction for improvement. For example, Kim al. improved bug localization accuracy by eliminating the input tokens that the model considers when it fails to localize (Kim et al., 2022c).
Similarly, an examination of the internal behav- ior of the model during the generation of incor- rect patches can provide direction for improvement. Current studies on incorrect patches primarily fo- cus on distinguishing incorrect patches by rely- ing primarily on the embedding vectors of the in- put buggy code (Phung et al., 2022; Csuvik et al., 2020). However, because of the repeated reformu- lation of these vectors within the inner layers of the transformer-based model, the embedding vector lacks sufficient information to discern the process that causes incorrect patch generation. Moreover, once an incorrect patch is identified, determining directions to improve becomes challenging. There- fore, research must be conducted with focus on identifying and addressing false inner workings (that is, false behaviors), which the model gener- ates incorrect patches.
1 Introduction
Automated program repair (APR), a technique that has attracted substantial attention from both academia and industry, can mitigate the costs asso- ciated with bug fixing during software development and maintenance by automatically generating fix- ing patches. Transformers have received enormous attention and have been applied to APR, showing effectiveness in fixing defects, bugs, errors, and vul- nerabilities (Fu et al., 2022; Pearce et al., 2022;
Applying the methods and findings of previous studies that analyzed the model behavior may ap- pear like an intuitive solution. However, directly utilizing them becomes challenging because of the disparity between our target task, program repair, and the tasks investigated in previous studies. Code
corresponding author We refer to these as bugs in a broud sense.

---

fixing is generation task that poses difficulties when the results or methodologies typically used in classification tasks are applied (Kim et al., 2022c; Ribeiro et al., 2016a), Among the various genera- tion tasks (Voita et al., 2019), code fixing focuses specifically on eliminating bugs from the source code.
Our first treatment filtered 60.4% of the incorrect patches, significantly reducing the patch verifi- cation time. Additionally, our second treatment, namely, masked bypassing, increased the gener- ation of correct patches by a minimum of 5.8% and up to 130.4%.
Preliminaries and Related Works
To ensure consistency in the input/output lan- guage and preserve the correct structure and se- mantics of the code, the majority of the inputs must remain unchanged. Furthermore, as most existing studies focus on comprehending the successful or overall behaviors of a model (Rabin et al., 2021; Voita et al., 2019; Ribeiro et al., 2016a), a defi ciency exists in understanding false behaviors and identifying their characteristics for automatic detec- tion. Therefore, is essential conduct research to comprehend false behaviors, along with their iden- tification and subsequent treatment for the APR domain.
2.1 Attentions in Transformer Architecture
Attention heads are the core elements of the trans- former architecture (Hao et al., 2021; Wang et al., 2020; Voita al., 2019). The transformer exten- sively utilizes an attention mechanism, representing the input tokens in various context vectors. Con- text vectors are used to compute the likelihood each vocabulary as the next token. In a transformer model, each attention head AHI (h-th head of I- th layer) reformulates context vector based on the tokens in input string S {#1, 22, .x., In the first layer (block), embedding vector X is pro- jected onto the query, key, and value spaces, with projection weight W WK WV, respectivel, as shown in Equation 1. Notably, previous studies on incorrect patches have employed the initial embed- ding vector X before projection to evaluate patch correctness.
This study proposes methodology for diag- nosing and treating false behaviors in transformer- based program repair models. We propose be- havior vector to represent the internal behavior of the model. The proposed behavior vector gener- ated by extracting the attention weights and value vectors from each attention head inside the trans- former model. This approach enables us to capture the model behavior in terms of the way it consid- ers neglects the input tokens during the patch generation steps. Additionally, we introduce a be- havior discriminator (BeDisc) that distinguishes the patterns between successful and false behaviors ex- hibited by the model. Based on this, we introduce two treatments I) early abortion and 2) masked bypassing. Our contributions are as follows:
(1) Q and K are used to calculate attention map A that stores the relevancies between tokens. In Attention is all you need, where the trans- former is first proposed, relevancies calculated with scaled dot product in Equation 2 (Vaswani et al., 7). They proposed scale the dot prod- uet value by /dg, where dg represents the size of the key vector, to prevent the dot product value from becoming excessively large.
-
.WVV
We introduce a novel perspective for improving program repair tasks by analyzing the internal behavior of a transformer-based program repair model.
QKT
A = Softmart
(2)
Vdg
The relevance of tokens 2; and æy is stored as Ai in matrix A, which is real number osj E IR denoting the attention that x; pays to Tz. Owing to the use of multiple attention heads with distinct weight matrices W W WV in each A H1 rel- evance in diverse perspectives is stored in each map A of the Al Finally, context vector matrix C is derived by applying Equation 3. Consequently, the context vector corresponding to each token 2; can be expressed as shown in Equation by decompos- ing Equation 3 with respect to C,.
We present methodology for representing the internal behavior of a transformer-ban program repair model and propose approach for diag- nosing and mitigating false behaviors exhibited by models for the APR task.
The results of large-scale experiment on 55,562 instances using seven pairs of models and datasets showed average identification rates of 86.9% for true patches and 86.4% for incorrect patches by a BeDisc.

---

[Xp No X
C= = AV
Embed
Iayers]
[heads]
AHf AR
AB
AVG
C--
(4)
AHP
AN
AH
RSP
RS
RS
RSP
RS
RS
RSP
RSP
RSP
Within attention block, multiple AH concurrently compute context vector matrix Cth, which is subsequently treated as a new embedding vector and propagated through the feed-forward layer. The reformulation process is repetitively it- erated in successive blocks. The behavior of the model is influenced by these AH The AH make the model focus on different parts of the in- put sequence for each token, considering their rel- evance from various perspectives. In this study, we propose behavior vector to quantify model behavior by analyzing the focus on target token it during the internal processes of each AH
AHP:
AND
AH
heads]
Figure Illustration of generating behavior vector rep- resentation for the target token.
QuixBug benchmarks (Prenner and Robbes, 2021) and was able to address security vulnerabilities (Pearce et al., 2022). Overall, transformers have ex- hibited promising results for program repair. How- ever, improving the model without accounting for its behavior during patch generation may result in ineffective improvements (Mohammadkhani et al.. 2023).
2.2 Transformer for Program Repair
The effectiveness of the transformer-based program repair model has been experimentally demonstrated in both encoder-decoder families (Li al., 2022: Kim et al., 2022b; Wang al., 2021; Berabi et al., 2021) and decoder-only familics(Jesse et al., 2023; Joshi et al., 2022; Prenner and Robbes, 2021), with their correct patch generation accuracy. The pro- gram repair model is trained to transform the in- put buggy code into fixed code (that is, patch). The generated patches are verified through man- ual test case evaluations to filter out incorrect patches. The developers upply a patch that success- fully passes the verification.
2.3 Model Behavior: Existing Approaches
To understand and interpret the behavior of the model, researchers in the fields of NLP and soft- ware engineering have explored explainable ar- tificial intelligence (XAI), specifically focusing on the model consideration of input tokens and the influential role of specific neurons in deter- mining the model output(iarpakdee et al., 2020; Wattanakriengkrai et al., 2020; Rabin al., 2021; Suneja et al., 2021). The attention weight, a com- mon medium for interpreting the behavior deep learning models with attention mechanisms, pro- vides insight into how the model refers to specific tokens (Li et al., 2016). Prior studies on program repair models demonstrated their code-fixing abil- ity by utilizing attention maps, highlighting the focused words during patch generation et al., 2020, 2019; Jiang al., 2021). However, simply tracking the attention weight may be insuffi- cient to capture the model behavior. This limitation arises from neglecting the size of the token vectors and misaligning their actual impacts (Kobayashi et al., 2020). In Equation 4, the effect of the target token 2, when calculating C is related attention weight ou and vector Vi of token I7, Considering all these factors, we propose behavior vector that indicates the reference level of the model for each word. By leveraging this vector, we can identify false behaviors and apply further treatments.
In recent studies, repair model such as TFix, demonstrated the repair ability of T5-based repair model on 52 eттor types of [SLint(Berabi et al., 2021), and Fu et al. utilized the T5 model for vulnerability repair (Fu et al., 2022). PLBART fixed 19.21% and 8.98% of bugs, respectively, in the small and medium datasets of Patches in wild (Ahmad al., 2021). CodeT5 fixed 22.59% and 14. 18% of the bugs, respectively, for the same datasets by incorporating a code-specific tokenizer (Wang et al., 2021). Mashhadi et al. showed that CodeBERT could fix simple bugs in Java with an accuracy of 19 72% (Mashhadi and Hemmati, 2021). Kim al. empirically demonstrated 20% bug-fixing accuracy for Kotlin bugs in an indus- trial environment using TFix (Kim et al., 2022b). Codex successfully resolved 23 bugs out of 40
Shtnss/sinngnniiicccooooostttter summary

---

3 Methodology
Buggy Code
3.1 Behavior Vector Representation
We propose a behavior vector b, |ilaayess- that represents the internal process of the trans- former model on a buggy imput token 2t when generating fixing code (that is, patch). The proposed behavior vector reflects the extent to which the model considers 2t during the generation of patch Sout {In+1. Fm for an input with buggy code Sin = {x1, (.)
Model
BVG Vector Generation
BeDisc
Patch
False
Generation
Detection
No
Patch
IsFalse
Yes
neglects token
Model
No
Patch Generation
Stop?
With Mask
New
Yes
Patch
Figure illustrates the proposed methodology to
T2
obtain the behavior vector for target token from a transformer-basee repair model with behavior vector generator (BVG). To generate behavior vec- tor by, the BVG iterates each AH and calculates the relative stake (R of 2t in computing the context vectors used to predict an output token at each generation step. the subsequent description, note that the target token is referred to as Ty, the in- put tokens are denoted as Ij. and the output tokens are designated as
Human/TC Patch Validation
Abort
Invalid
Valid
Fixing
Patch
Figure Program repair with false behavior diagnosis and treatment.
Recalling Equation calculating context vector C for token z; in each AIl can be interpreted weighted sum of tokens Vay C Sax- In other words, each input token Z has different stake in calculating C, for generation, and the values that determine this stake are attention weight Oij and size of the value vector [VV, Hence, the stake of target token It in calculating VC; E C' in each A AH can be represented by Equation 5.
the generation step concatenated into the final attention map A IR. (//Suu/x|Ss..
Consequently, a scalar value RS representing the manner each 1Hth considers 2; during the gen- eration step can be obtained. The obtained RS values for each AH are then concatenated into behavior vector b; E [llayersl-[heade) of the model for target token Fy. The behavior vector for a multi- token set S. Sin can be represented in two ways, First, to represent the behavior a fine-grained manner, by for all tokens can be concatenated as B, E all tokens, Vx; E Sa, can be averaged and used as a behavior vector to capture the behavior of the model on tokens of the specific type as collective entity.
slake(æ1) s/ aal
(5)
Second, the vectors
The relative stake of model reference to target token on compared with other input tokens Vzz E S, during generation, denoted R.Sth in Figure 1, is calculated as Equation 6.
3.2 Program Repair with False Behavior Diagnosis and Treatment
au VIIl
R.S
(6)
5 a IVSI
S D
Figure 2 shows the program repair process using the proposed false behavior diagnosis and treat- ment. The uncolored portion in the figure corre- sponds to the typical program repair method de- scribed in Scction 2.2. Instead of directly passing the model-generated patch to the subsequent valida- tion step (indicated by the dotted line), which can be significant bottleneck, the proposed method incorporates the process of diagnosing and treat- ing false behaviors, as depicted in the gray boxes.
For caleulation, we collect the Euclidean norm of value vectors for every Zÿ in AH We process at- tention maps at each generation step to construct an attention map 4th | | where attention weights ouj for every iv and Ty are stored. From each generation step where token I; is selected as the output token, we can extract the attention weight that each zj received from token I; and con- catenate them as E Ri Then, every a from

---

Treatment methods T1 and T2 can be selected based on the development environment.
Test
Validution
Idx
H
Model
Data
Alaput
#Fix
Hingut
#Fix
8
TFix8
1SL.nt
10,504
4,670
9,454
4,245
False Behavior Diagnosis. The box labeled D in Figure 2 represents the diagnosis of the false behav- iors when the model generates a patch. The BeDisc is a binary classifier that uses the behavior vector (BV) of the model as input and determines whether it corresponds to a behavior that could result in gen- erating incorrect patch (that is, false behavior). The generated patch is returned if the BeDisc pre- dicts that the behavior true. Otherwise, the next treatment step follows. The validation (or training) data used to train (or validate) the patch generation model can be used to train the BeDise. The label of the behavior vector is assigned value of if the repair model successfully generates a correct patch for the input buggy code; otherwise, it is set to O, Treatment 1: Abortion. An effective treatment for false behaviors is early abortion, as denoted by the T/ box in Figure 2. This method immedi- ately aborts the patch before the validation stage. After generating patch, the APR model under- goes a validation process to assess the patch. This can be conducted manually by the developer or by employing a test suite to verify the compliance with all test cases. Manual validation can be labo- rious, and test suites can incur significant costs to build and compile the entire software to execute test cases. Therefore, aborting an incorrect patch without proceeding to the validation stage can save considerable time.
2
12
12
TFixB
1SLint
10,504
5,004
9,454
4,480
12
CT5
Ws
5,835
1,246
5,835
1,253
WM
777
6,546
12
CT5
wSt
5,835
1,513
5,835
1.424
20
16
CGen
WS
5,835
853
5,835
755
20
16
Total JSLint
CGen
10,504
3,258
9,454
2,937
55,562
17,321
52,413
15,359
Table 1: Experimental datasets and models. TFixS indi- cates TFix-small, TFixB indicates TFix-base, CTS in- dicates CodeTS-base, CGen indicates CodeGen-350M- multi, WS indicates Wild-small, WM indicates Wild- medium. Validation data the APR model are used to train the BeDisc.
ment could be preferable for maximizing the num- ber of correct patches, prioritizing repair accuracy over saving validation time.
4 Experimental Setup
4.1 Models and Datasets
We conducted experiments using four mod- els (TFix-small, TFix-base, CodeTS-base, and CodeGien-350M-muih on three publicly available datasets (Table 1). Pairs 1-4 were validated in previous studies (Berabi et al., 2021; Wang et al., 2021), whereas pairs 5-7 were added for our ex- periments. To analyze the effect of the proposed method with and without token abstraction, we introduced the fifth pair, Wild-small"* using the Wild-small dataset without abstraction. The sixth and seventh pairs involved the CodeGen model (Ni- jkamp et al., 2022), which is designed specifically for software-related text generation. The [layers| and hcads of each model are listed in columns L and H. The #Input and #Fix columns represent number of inputs and the number of inputs exhibit- ing true behavior (correct fixing code generated top-1), respectively. Instances with false behaviors outnumbered those with true behaviors. Input and output for each model are available in Appendix A.2.
Treatment 2: Masked Bypassing. This method prevents the model from referencing a suspicious target token that might have caused the false be- havior when generating the patch. We use an at- tention mask to assign value of zero to the target tokens, thereby avoiding their influence. The atten- tion mask can indicate the model in which tokens should be attended to (Hugging Face, 2020; Kim et al., 2022a). For example, when we want to disre- gard the token b for a token sequence (a b c), we can use the masking array (1, 0, 1). Then, based on the given masking array, the model will use (a _c} as the input instance. To bypass the effect of the target token without altering the positional information of the remaining tokens, use an attention mask instead of replacing the token with alternative tokens such as <MASK>.
4.2 Metries
BeDisc Accuracy. To evaluate the BeDisc, we used classification evaluation metrics including the Fl score, sensitivity (true positive rate; TPR), speci- ficity (true negative rate; TNR), and balanced accu- racy (Brodersen et al., 2010) where higher values indicate better performance.
Treatment recommended for software with numerous test cases or large-scale systems to streamline the validation process. Otherwise, Treat-
Top-k Fixing Accuracy. To evaluate the repair performance, we examined the top-k generated

---

Idx
neuron
thres
TPR
TNR
F1
Bal Ace
ther studies are required to effectively distinguish false behavior patterns. Nevertheless, the proposed single-layer BeDisc achieved a classification aceu- racy of 36.61% % by balanced accuracy (Bal_Ace) and 79.61 % by Fl score on average, surpassing 50% Bal_Acc of the random classifier.
420
0.64
94.87
70.78
82.28
82.82
410
0.61
90.26
80.64
85,54
85.45
3
440
0.67
86.65
91.13
80,86
88.89
460
0.84
89.16
95.22
81,56
92.19
5
480
0.60
80,34
90.79
78.78
85,57
430
0.66
80.76
87.94
67.34
84.35
7
390
0.60
85,90
88.08
80,95
86,99
The first pair, which showed the second best FI score and best TPR, had the worst TNR and Bal Acc. Overall, a higher TPR corresponded to lower TNR performance. Classifiers with high TNR can better filter the false behavior but might result in degradation in the fixing ability. Therefore, the hyperparameters for the classifier should be selected based on the development environment, considering factors such as the software scale.
Average
86.85
86.37
79.62
86.61
Table 2: Diagnosis accuracy of the BeDisc with the proposed behavior vector. The hest performance of each metric is marked in bold.
patches for each instance and considered them to be a successful fix if any patch within the top-k was correct. In accordance with previous studics (Wang et al., 2021; Berabi et al., 2021; Kim et al., 2022b), an exact match with the developer's patch was con- sidered as correct patch. For &e > 1, an effort to determine the first correct patch was evaluated using a mean reciprocal rank (MRR).
The BeDisc demonstrated its effectiveness in both the encoder-decoder and decoder-only mod- els, albeit with certain distinctions. When com- paring the model architectures, the order of perfor- mance based on the Fl score was TFix CodeT5 CodeGen, whereas that based on the Bal Bal_Acc standard was CodeT5 > CodeGen > TFix. When comparing the two pairs with the same wild-small dataset but different architecture (the third and sixth pairs), CodeT5 exhibited better classification accu- racy across all four evaluation metrics. However, for the same JSLint dataset (the first, second, and seventh pairs), CodeGen exhibited better Bcc_Acc.
4.3 Hyperparameters
We employed single hidden layer MLP as BeDisc assess the performance achievable using simple classifier. We selected need-fix fix token sets among the input tokens and used the average vector as the final behavior vector; these are crucial tokens among the buggy codes for fixing. We evaluated the results on the test dataset using the hyperparameters obtained from three-fold validation process. The hyperparameter setting with the highest average value across the three-fold validation was selected. The model was then trained on the entire validation dataset and used to make inferences on the test dataset to obtain the final results. Further details on hyperparameters are available in Appendix 1.
The behavior vector size for the first and sec- ond pairs, TFix-small and TFix-base, were 48 and 144, respectively, with TFix-base being three times larger than Tmil-small. However, they displayed only a marginal difference of approximately 3p% for both Fl and Bal_Ace The seventh pair, which utilized the JSLint dataset, had a behavior vector size of 320, exceeding TFix-small by more than six times. However, the difference in the performance was less than 5p%. The results indicated that the BeDisc could effectively discriminate false behav- iors using the proposed behavior vectors across transformer models of different sizes.
5 Results
5.1 Accuracy of Behavior Discriminator
Table 2 lists the classification performances of the proposed BeDise. We selected the hyperparameters with Fl score. In the experimental data, a threshold of 0.6 or higher was selected for all pairs because those with false behaviors outnumbered instances with true behaviors. The neuron size of the opti- mal model can indicate the complexity of the target task because larger models have more expressive capacity (Hu et al., 2021). The selected neuron range (390~480) was relatively large within the hyperparameter search space, suggesting that learn- ing behavioral patterns is a non-trivial task. Fur-
The third and fifth were pairs with and without text abstraction on the wild-small dataset, and the BeDisc classified better for code that performed abstraction than for raw code with marginal dif- ference. The sixth and seventh pairs used differ- ent data on the same model. The JSLint data that had more instances for the BeDisc to learn, per- formed better than the wild-small data. This trend remained when comparing the first and second pairs that used JSLint and the third-fifth pairs that

---

Intended Tolerance for Search
0G
BeDise, Mask
BeDisc,
Mask
-1%
-2%
-3%
-4%
-5%
#laput
#Fix
#Fix
#Fix
10,504
4,670
3,062
-34,4%
5,017
7.4%
RealTol
-2.2
-2.8
-4.2
-9.2
-11.7
2
10,504
5,004
1,748
65.1%
5,272
5.4%
TNR
61.1
67.0
71.0
78.7
81.1
5,835
1,246
871
-30.1%
1,543
23.8%
RealTol
-1.5
-3.1
-4.8
-6.4
-7.5
6,545
777
1,640
111.1%
1,790
130.4%
TNR
51.5
56.3
68.9
72.2
75.0
5,835
1,513
1,271
1,900
RealTol
-0.9
-1.5
-0.8
-2.6
-3.1
5,835
853
1,438
82.8%
3
7
10.504
3,258
2,407
-6..1%
3,469
6.5%
TNR
61.6
62.6
58.9
74.0
74.4
Total
55,562
17,321
12,437
-28.2%
20,550
18.6%
RealTol
-0.6
-3.0
-2.2
-5.5
-6.9
TNR
48.9
82.0
81.7
90.5
91.5
Table 4: Fixing accuracy with and without the proposed
RealTol
-1,4
-4.3
-5.1
-7.0
-13.0
treatment, highlighting the best repair accuracy for each
5
TNR
42.6
69.9
73.2
78.0
85.9
pair in bold.
RealTol
-2.0
-2.4
-2.9
-12.6
-9.7
TNR
35.7
42.7
50.2
72.8
72.8
Intended Tolerance for Search
7
RealTol
-1.1
-1.1
-2.7
-10,6
-12.8
-1
-2%
-3%
-4%
-5%
TNR
39.1
42.0
56.7
74.9
79.9
1
8.4%
8,9 %
8.7%
7.9%
7.4%
Average
RealTol -1.4
-2.6
-3.2
-7.7
-9.2
2
5.8
5.2%
5.8%
5.5%
5,4%
TNR
48.6
60.4
65.8
77.3
80.1
3
19.2%
19.0%
18.1%
23.8%
23.8%
74.1%
117.9%
116.2%
128.6%
130.4%
Table 3: Early abortion performance by different false negative tolerances.
5
16.2%
24.9%
26.0%
26.6 %
25.6%
46.1%
49.1%
58.6%
84.3%
82.8%
7
6.9%
7.1% %
7.9%
6.7%
6.5%
Avg.
25.2%
33.2%
34.5%
40.5%
40.3%
used the Wild dataset.
Table 5: Repair performance improvement regarding BeDise hyperparameter search tolerance. The hest im- provement for each model is presented in bold.
5.2
T1: Early Abortion
This section presents the results of the first treat- ment, namely, early abortion. We selected the hy- perparameters based on the highest FNR of the val- idation dataset for each tolerance, aiming to max- imize the early abortion of false behaviors while meeting the degradation criteria. Therefore, we examined the effectiivnees of the early abortion treatment in achieving accurate abortion while tol- erating performance degradation within the range -1% to -5%. Table lists the obtained experi- mental results.
incorrect patches. Conversely, we recommend a tolerance range of -1% to -3% for the opposite scenario.
5.3
T2: Masked Bypassing
5.3.1 Top-1 Accuracy
This section presents the results of the second treat- ment, that is, masked bypassing. We used an identi- cal BeDisc from the -5% column in Table 3. Table 4 compares the fixing accuracy of the original model (OG column), the proposed method (w BeDisc, w Mask column), and an ablation case in which the target tokens are masked irrespective of the false behavior judgment (w/o BeDisc, w Mask column) of the model.
Disparities can exist between the intended toler- ance and the real tolerance (RealTol), that indicates the performance deterioration rate. Despite per- formance degradation (-7.7% and -9.2%), BeDises successfully aborted a substantial portion of false behaviors, with success rates of 77.3% and 80.1% 1% for tolerance levels of -4% and -5%, respectively. When aiming for a tolerance range -1% to -3%, only a slight additional performance degradation (average of -0.4p% and maximum of -2.3p%) oc- curred than intended and 48.6% to 65.8% of false behaviors were successfully aborted. Additionally, between -3% and -4%, sharp increase noted in the performance degradation. In summary, recommended setting the tolerance for hyperpa- rameter search to -4% to -5% for large-scale soft- ware that requires significant time for building and testing to minimize the time spent on validating
The proposed treatment method (w BeDise, Mask) resulted in an average improvement of 18.6% in the repair accuracy across all seven pairs, with the best case generating 130.4% more cor- rect patches. In contrast, when masking was ap- plied to all instances without considering false be- haviors (w/o BeDisc, w Mask), the performance deteriorated in five of the seven combinations, re- sulting in average decrease of -28.2%. Com- pared with the proposed method, -65.2% (~ (20,350 12,437)x100 decrease was observed in re- 12,437

---

MRR
Ranks Original Model
This pair incorporates the largest model and text input, posing computational capacity issue when generating top-50 patches with a beam size of 50, 104
0G
OX+T2
Imp.
2
3
4
5
6~-50
N/A
0.50
0.55
7%
100
32
13
174
0.52
0.55
5%
80
13
5
97
155
0.31
0.35
12%
0
36
18
5
8
108
39%
36
26
This caused the model to fail in some cases due to an out-of-memory error, making it unsuitable for comprchensive comparison. In this section, we used a BeDisc with a tolerance of 1%.
5
0.32
0.36
13%
0
20
13
8
4
74
167
0.27
0.32
19%
63
30
12
8
143
157
16%
64 894
Table Investigation result false behaviors with top- 50 accuracies. N/A denotes the scenario in which the correct patch is not found among the generated patches.
We compared the effort required to find the first correct patch with (OG+T2 column) and without (OG column) the proposed treatment, using the MRR metric. contrast to OG, where developers receive the initial top-50 patches from the original model, OG+T2 employs subsequent process to restructure the patch rankings. Following the gen- eration by the original model, the model's behavior is evaluated using BeDisc. If the behavior is deter- mined erroncous (i.e., when the current top-I patch identified as incorrect), two steps are taken. In the first step, a patch is generated using masked attention, employing T2's treatment. Then, in the second step, the original top-1 patch is replaced with the top-1 patch from T2. On the other hand, if the behavior is determined as accurate (i.e., the cur- rent top-1 patch is considered correct), changes are made the top-50 patches, and they remain unchanged.
pair accuracy because the ability of the proposed method to assess true/false behaviors on target to- kens enabled their preservation when they led to correct patch generation.
'he repair performance of CodeGen (the sixth pair) improved from 853 to 1,559 on the wild-small dataset, whereas that of CodeT5 (the third pair) im- proved from 1,246 to 1,543. Before the proposed treatment, CodeT5 showed better repair accuracy; however, with the proposed treatment, CodeGen could fix more defects. The target token caused more false behaviors in CodeT5 models than in CodeGen, which is more recent model. Even without BeDisc (w/o BeDisc, Mask), 68.6% more fixes were possible in the sixth pair. Fur- ther investigation is required because the type of input that leads to significant false behaviors can vary across models.
The results are presented in the MRR columns of Table 6. The OG+T2 column shows the MRR of the top-50 ranked patch list following the applica- tion of treatment T2. This evaluates the efficiency of finding correct fixed patch within the ranked patches. A higher MRR indicates a quicker iden- tification of the correct fixed patch, consequently reducing the effort required for validation. The higher the MRR, the lcss effort is required until the developer encounters the correct fixed patch. For example, if the correct patch is ranked 10th, vali- dation of the preceding nine patches is necessary. Conversely, if the correct patch is ranked top- no effort will be consumed for validation. The Imp. column denotes the performance improvement rate of OG+T2 compared with that of OG. Top-50 re- pair attempts on the APR model revealed that the original model achieved an MRR of 0.35. However, by avoiding false behaviors, a 16% higher MRR was achieved. This improvement was achieved by lowering the ranks of incorrect patches, facilitating the prompt identification of correct patches. The experimental results shed light on the impact of false behaviors on patch generation.
Table 5 presents the fixing accuracy improve- ment rate corresponding to BeDises with differ- ent tolerance. For each tolerance, the same model shown in Table 3 was used. Irrespective of the tol- crance we selected for the hyperparameter search, the second treatment improved the repair accuracy. Across the first, second, and seventh pairs, for each model using the JSLint dataset, the variation in im- provement between the least and the most was less than 2p%. Pairs using the wild dataset exhibited greater variation, yet consistently achieved optimal performance with tolerances of -4% and -5%.
5.3.2 Top-k Accuracy
We assessed the ranking of correct patches within the top-k generated patches of the model. This eval- uation demonstrated the efficacy of the proposed treatment method and highlighted the negative im- pact of false behaviors on accurate patch generation. Considering the capacity of the experimental envi- ronment, we set as 50 and used beam size of 50. The seventh pair was excluded because of the ca- pacity limitations of the experimental environment.
To conduct a thorough analysis, we examined

---

2,572 instances that were identified as false be- haviors but could be fixed generate the correct patch as top-1 by applying the second treatment. We investigated the rankings of the corresponding patches in the original model.A small number the examined instances (3% R
et al., 2021). Although certain repair models may not have been trained with optimal hyperpa- rameters, we used the CodeTT and TFix models with the same performance as in previous studies, suggesting the anticipated effectiveness of the proposed method in typical cases.
15x were initially 2572
generated with the correct patch the top-1 but were classified as false behavior owing to judgment errors in the BeDisc. However, the majority (97% 1007x100 were cases where the patch could not 2572
BeDisc Architecture. The BeDisc used in the experiment may not be the best architecture for identifying false behaviors. However, despite its simplicity, the model identified 60.4% of false behaviors with -2.6% of tolerance. Further studies on advanced model architectures for identifying false behaviors can lead to additional improvements in the BeDisc.
be generated as the top-1 owing to false behaviors. (894=954)x100) of the in- 2372
Specifically, 72% (
stances that attained the correct patch the top-1 result after applying the proposed treatment were initially unable to generate the correct patch within the top-5 ranks owing to false behavior the orig- inal model. Additionally, 50% them, which corresponded to 894 instances, failed to generate the correct patch even when the search progressed up to the top-50 patches. This suggested that false behaviors could significantly hinder the ability of model to sort the necessary vocabulary for gen- erating correct patches. Given that the proposed treatment method simply aims mitigate the im- pact of false behaviors, further refinement of the false behavior treatment method is expected to en- able a higher rate of correct patch generation.
Tolerance Setting, The selection of appropriate tolerance is a human-driven process because it may vary across developmental environments, potentially leading to challenges in determining tolerance settings. However, because we confirmed that the proposed method was consistently effective within a tolerance -1% to -5%, we recommended deciding based on the experimental findings.
Behavior diagnosis target token type. Accurately identifying the location of the target token (that is, need-fix token) can pose challenges, particularly in the present experiment, in which assumed perfect localization of the need-fiv fix token. However, this can be supplemented by techniques that locate fixed locus (Phung and Lee, 2020; Chandra et al., 2011). 1). We plan to explore wider range of tokens to address this issue in future studies.
6 Conclusion
This study is the first to examine the inner workings of program repair model failures and to introduce a behavior vector for diagnosing false behaviors in transformer-based program repair model. We proposed methods for diagnosing and addressing the false behavior of the program repair model and validated their effectiveness across three models and four datasets. The proposed BeDisc and treat- ment strategies improved the APR performance by 18.6% on average and up to 130.4%. The ex- perimental results demonstrated the necessity for additional investigations into the false behavior of program repair models. In future studies, we intend expand the proposed method to wider range of models and bugs. In Addition, intend investi- gate false behaviors across broader target tokens to develop an advanced treatment approach.
Acknowledgements
This work supported by the National Research Foundation of Korea(NRF) grant funded by the Korea ggwermeen//NSST///oooRR//AAAAA
References
Wasi Uddin Ahmad. Saikat Chakraborty, Baishakhi Kai-Wei Wei Chang. 2021. Unified pre-training for program understanding and generation. arXiv preprint arXhv:2103.06333.
Ray, and Kai-
Nikolich Alexandr, Osliakova Irina, Kudinova Tatyana, Kappusheva Inessa, and Puchkova Arina. 2021. Fine- tuning gpt-3 for russian text summarization. In Data Science and Intelligent Systems: Proceedings of Sth Computational Methods in Systems Software 2021, Vol. 2, pages 748-757. Springer.
Limitations
Repair Model. We experimented with three representative models, excluding the encoder-only models owing to their sub-optimal nature (Wang

---

Berkay Berabi, Jingxuan He, Veselin Raychev, and Mar- ti Vechev. 2021. Tfx: Learning to fix coding er- rors with text-to-text transformer. In International Conference on Machine Learning, pages 780-791. PMLR.
Xia Hu, Lingyang Chu, Jian Pei. Weiqing Liu, and Jiang Bian. 2021. Model complexity deep learn- ing: A survey, Knowledge and Information Systems, 63:2585-2619.
Junjie Huang, Duyu Tang, Wanjun Zhong. Shuai Lu, Linjun Shou, Ming Gong, Daxin Jiang, and Nan Duan. 2021. Whiteningbert: An easy unsuper- vised sentence embedding approach. arXiv preprint arXiv:2104.0177.
Judy Borowski, Roland S Zimmermann, Judith Schep- ers, Robert Geirhos, Thomas SA Wallis, Matthias Bethge, and Wieland Brendel. 2020. Exemplary natural images explain emn activations better than state-of-the-art feature visualization. arXiv preprint arXiv:2010. 12606.
Hugging Face. 2020. Hugging face transformers glos- sary. Accessed October 13, 2023.
Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, and Joachim M Buhmann. 2010. The bal- anced accuracy and its posterior distribution. 2010 20th international conference on pattern recognition, pages 3121-3124. IEEE.
Kevin Jesse, Touîique Ahmed, Premkumar T De- vanbu, and Emily Morgan. 2023. Large language models and simple, stupid bugs. arXiv preprint arXiv:2303. 11455
Satish Chandra, Emina Torlak, Shaon Barman, and Rastislav Bodik. 201 I. Angelic debugging. In Pro- ceedings of the 33rd International Conference on Software Engineering, pages 121-130.
Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023. Impact of code language models on automated program repair. arXiv preprint arXiv:2302 05020.
Nan Jiang, Thibaud Lutellier, and Lin Tan. 2021. Cure: Code-aware neural machine translation for automatic program repair. In 2021 IEEE/ACM 43rd Interna- rional Conference on Software Engineering (ICSE), pages 1161-1173, IEEE.
Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos, Jum Shem Chan, Samuel R Bow- man, Kyunghyun Cho, and Ethan Perez. 2023. Im- proving code generation by training with natural lan- guage feedback. arXiv preprint arXiv:2303.1677
Jirayus Jiarpakdee, Chakkrit Kla Tantithamthavom, Hoa Khanh Dam, and John Grundy. 2020. An em- pirical study of model-agnostic techniques for defect prediction models. IEEE Transactions on Software Engineering, 48(1):166-185.
Hyunjin Choi, Judong Kim, Seongho Joe, and Youngjune Gwon. 2021. Eyaluation of bert and albert sentence embedding performance downstream nlp tasks. In 2020 25th International conference on pat- lern recognition (ICPR), pages 5482-5487. IEEE.
Harshit Joshi, José Cambronero, Sumit Gulwani, Vu Le, Ivan Radicek, and Gust Verbruggen. 2022. Repair is nearly generation: Multilingual program repair with llms. arXiv preprint arxiv:2208. 1/640.
Viktor Csuvik, Dániel Horváth, Ferenc Horváth, and László Vidáes. 2020. Utilizing source code embed- dings to identify correct patches. In 2020 IEEE 2nd International Workshop on Intelligent Bug Fixing (IBF), pages 18-25. IEEE.
Gajendra Jung Katuwal and Robert Chen. 2016. Ma- chine learning model interpretability for precision medicine. arXiv preprint arXiv:1610.0904;
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: generative model for code infilligg and synthesis. arXiv preprint arXiv: 2rXi:22044005999
Urvashi Khandelwal, Kevin Clark, Dan Jurafsky, and Lukasz Kaiser. 2019, Sample efficient text summa- rization using single pre-trained transformer. arXiv preprint 1rxxiv/905.088.00
Michael Fu, Chakkrit Tantichamthavorn, Trung Le, Van
Nguyen, and Dinh Phung. 2022. Vulrepair: a 15- based automated software vulnerability repair. In Proceedings of the 30th ACM Joint European Soft- ware Engineering Conference and Symposium on the Foundations of Software Engineering, pages 935- 947.
Misoo Kim, Youngkyoung Kim, Jinseok Heo, Hohyeon Jeong, Sungoh Kim, and Funseok Lee. 2022a. Im- pact of defect instances for successful deep learning- based automatic program repair. In 2022 IEEE Inter- national Conference on Software Maintenance and Evolution (ICSME), pages 419-423. IEEE
David Gunning and David Aha. 2019. Darpa's explain- able artificial intelligence (xai) program. Al maga- zine, 40(2):44 58.
Misoo Kim, Youngkyoung Kim, Hohyeon Jeong, Jin- scok Heo, Sungoh Kim, Hyunhee Chung, and Eun- scok Lee. 2022b. An empirical study of deep transfer learning- learning-based program repair for kotlin projects. Proceedings of the 30th ACM Joint Furopean Soft- ware Engineering Conference and Symposium on the Foundations of Software Engineering, pages 1441- 1452.
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2021. Self- attention attribution: Interpreting information interac- tions inside transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12963-12971.

---

Youngkyoung Kim, Misoo Kim, and Eunseok Lee. 2022c. Tracking down misguiding terms for locat- ing bugs in deep learning-based software (student abstract). In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 12983- 12984.
Quang-Ngoo Phung, Misoo Kim, and Eunseok Lee. 2022. Identifying incorrect patches in program re- pair based meaning of source code. IEEE Access, 10:12012-12030.
Quang-Ngoc Phung and Eunscok Lee. 2020. Incremen- tal formula-based fix localization. Applied Sciences, 310:33033
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is not only a weight: Analyzing transformers with vector norms, arXiv preprint (xXx:20004101000
Julian Aron Prenner and Romain Robbes, 2021. Auto- matic program repair with openai's codex: Evaluat- ing quixbugs. arXiv preprint arXiv:2111.0392.
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Un- derstanding neural networks through representation crasure. arXiv preprint arXiv: arXiv:66/2.088 162200820..
Md Rafiqul Islam Rabin, Vincent J Hellendoorn, and Mohammad Amin Alipour. 2021. Understanding neural code intelligence through program simplifica- tion. In Proceedings of the 29th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineer- ing, pages 441-
Yujia Li, David Choi, Junyoung Chung. Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, al. 2022. Competition- code generation with alphacode. Science, 378666244 1097.
Marco
Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 201 6a. why should trust you?" explain- ing the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and duta mining, pages 1135- 1144.
Thibaud Lutellier, Lawrence Pang. Viet Hung Pham, Mashi Wei, and Lin Tan. 2019. Encore: Ensemble learning using convolution neural machine transla- tion for automatic program repair. arXiv preprint 10Xiv:7906.0800
Marco Tulio Ribeiro. Sameer Singh, and Carlos Guestrin. 2016b. Nothing else matters: Model- agnostic explanations by identifying prediction in- variance. arXiv preprint 16xiv
Thibaud Lutellier, Hung Viet Pham. Lawrence Pang. Yitong Li, Moshi Wei, and Lin Tan. 2020. Coconut: combining context-aware neural translation models using ensemble for program repair. In Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis. pages 111-114.
Sahil Suneja, Yunhui Zheng, Yufan Zhuang, Jim A Laredo, and Alessandro Morari. 2021. Towards re- liable ai for source code understanding. In Proceed- ings of the ACM Symposium on Cloud Computing. pages 403-411.
Ehsan Mashhadi and Hadi Hemmati. 2021. Applying codebert for automated program repair of java simple bugs. In 2021 IEEE/ACM 18th International Confer- ence on Mining Software Repositories (MSR), pages 505-509. IEEE.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need, Advances in neural information processing systems, 30.
Ahmad Haji Mohammadkhani, Nitin Sai Bommi, Mariem Daboussi, Onkar Sabnis, Chakkrit Tan- tithamthavorn, and Hadi Hemmati. 2023. A system- atic literature review of explainable ai for software engineering, arXiv preprint ar arXiv:2302.06065.
Elena Voita, David Talbot, Fedor Moiscev, Rico Sen- nrich, and Ivan Titov. 2019. Analyzing multi- head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint 1rX5v:1905.094:1
Erik Nijkamp, Bo Pang. Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv: 13474.
Wenhui Wang. Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural In- formation Processing Systems, 33:5776-5788.
David N Palacio, Nathan Cooper, Alvaro Rodriguez, Kevin Moran, and Denys Poshyvanyk. 2023. Toward theory of causation for interpreting neural code models. arXiv preprint arXiv:2302.03788
Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code un- derstanding and generation. arXiv preprint arXiv:2109.00859
Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavil Gavitt. 2022. Ex- amining zero-shot vulnerability repair with large lan- guage models. In 2023 IEEE Symposium Security and Privacy (SP), pages 1-18. IEEE Computer Soci- ety.
Supatsara Wattanakriengkrai, Patanamon Thongta- nunam, Chakkrit Tantithamthavorn, Hideaki Hata, and Kenichi Matsumoto. 2020. Predicting defective

---

lines using a model-agnostic technique. IEEE Trans- actions on Software Engineering,
Input:
48(5))1448888 48(5):1480-1499)
fix no-undef Undefined variable. = foo.)
A Implementation Detail
bar(5);
Z obj.fooo;
The code and data used for our experiment can be found in the supplement. We modified Transform- ers 4.12.33 to extract the normalized attention map. The experiments were conducted with one GPU (GeForce RTX 3090) with 24 GB RAM.
Output:
bar(S);
var = foo0)
Figure 3: Input and output format of the TFix model
A.1 Hyperparameter Settings for training Models
A.2 Prompt and Post-process
BeDisc. The hyperparameters for BeDisc included a neuron range of 0 to 500 and a class determination threshold range of 0.01 to 1. We used the default settings of seikit-learn for other hyperparameters. It important note that hyperparameters are searched using validation data and subsequently applied to test data. Consequently, the performance the proposed method can be further enhanced by utilizing optimized hyperparameters specific to the test dataset.
Special tokens (e.g., cos, sos tokens) were excluded from the string output of the model and employed as the final output for evaluation. The input/output settings for each model are follows:
TFix. The JSL.nt dataset used in the TFix model followed the prompt template of the orig- inal study. As shown in Figure 3, the datasets trained the model to output fixed code for the in- put in the form fix (rule id) (error message)
(buggy line) AAn (buggy code)
APR Models. We followed previous studies (Chen al., 2023; Alexandr et al., 2021; Khan- delwal et al., 2019) to train the CodeGen model it for one epoch because large lan- guage models (LLMs) such as CodeGen are suscep- tblle to overfitting (Jiang et al., 2023; Fried et al., 2022). We trained CodeGen-3300Mmmult a learning rate of 5e-5 and batch size 4, with max length of 512. The pretrained tokenizer does not have pad_token, as token The repair model is trained with Transformers library 3. the JSLint dataset, there were 84,846 instances for training, 9,454 for val- idation, and 10,504 for the test set. For the WS dataset, the corresponding numbers were 46,680 for training, 5,835 for validation, and 5,835 for the test set. The default values of PyTorch were used for unspecified hyperparameters. For the CodeT5 model, the pre-trained CodeT5 model is fine-tuned with small and medium datasets of Patches in wild (Ahmad et al., 2021), independently with a learning rate initialized to 10 with batch size 32. TFix was trained with a learning rate initialized to 10-4 with batch size 32. The pre-trained T5-base model is trained with ESLint javascript defects.
CodeT5. Wild datasets use buggy code imput, with no additional prefixes or postfixes. Unlike TFix, dataset, the dataset used only buggy code as an input to output fixed code
and
CodeGen. The CodeGen model is decoder- only model trained for causal language modeling, predicting the next token in a sequence. Therefore, fine-tuning was performed learn the sequences of (foriiinal input format of the dataset] Fixed: (ixed code] " For inference, the original input was provided to the model to generate the text. The text returned by the model was split with "Fixed:" and posterior tokens were treated as a generated patch.
with
we have eos_toke token
B Effectiveness of Treatments in Mitigating False Negatives of BeDisc
False negatives of BeDisc are the cases where the repair model's 's correct behavior is wrongly classi- fied as false behavior. As our experimental results in Table 2, whose hyperparameter search criteria was FI-score, a false negative rate (FNR) of classi- fication was 13.63% (ss 100 86.37) This dimin- ishes the efficacy of the repair model by wrongly discarding the correct patch. We introduced a In- tended Tolerance for Search to address this issue. By modifying the classifier hyperparameter search criteria from Fc-score to best TNR within speci-
"hups://pypi. agg/proecc/ttaansoomeess// "hups://scikit-learr leraooo.riiieaaaaaooorroo/ /2/ "htts://hugginggaae cossieefooccoodggen-550000

---

Pair Impact Repair Model
-1%
-2%
-3%
-4%
-5%
Vector Type
TNR
Wrong Rejection (PN)
95
123
180
398
504
Last token
43.171
59.324
77.076
41.572
Additional Fix (T2px)
39
60
98
243
322
Avg. of all tokens
43.17
59.71
74.04
45.39
Additional Fix (T2px)
449
477
486
522
529
Avg of NeedFix tokens
52.36
67.25
66.53
67.96
Wrong Rejection (FN)
56
65
90
167
Behavior Vector (Ours)
78.78
85.57
80.34
90.79
Additoonl Fix (12px)
16
21
63
Additional Fix (T2
261
417
437
460
490
Table 8: Classificatton accuracy comparison with other
Wrong Rejection (FN)
35
87
344
416
Additional Fix (T2px)
17
43
174
212
vector representations.
Additional Fix (T2pw)
248
249
303
388
415
Table 7: Effectiveness of T2 recovering the negative impact of FNs on the repair model.
Token vectors are the input/result of inner workings, while our behavior vector encap- sulates the process of inner workings. For instance, context vector C2 for token 2; in each AH is weighted sum of tokens Væz C Sin (please refer to Equation 3 and Equation 4 in Section 2). Such tokens may capture contextual information of in- put tokens, yet they cannot express how the model handles the tokens. This inherent ambiguity poses challenges in distinguishing between the correct and incorrect approaches the model adopts for to- ken handling. However, our behavior vector offers quantitative measure of the model's utilization of each token during its internal computations, conse- quently providing more explicit representation of the model's behavior.
Explicitness.
fied Intended Tolerance for Search, we can find a classifier that rejects correct patches within an endurable range. As a result, our treatment T1 was able to reject 48.6% to 80.1% of false behaviors within -1.4% -9.2% of repair aceuracy (corre- sponding to the average row in Table 3). Hence, TI offers significant time savings for developers with a minor trade-off. Given the subjective nature of a "minor trade-off", developers have the flexibility to establish an intended tolerance in accordance with their requirements.
Regarding our second treatment (T2), both TNs and FNs of the classifier trigger the repair model to regenerate patches with masking. Regenerating the patch for FNs may potentially compromise repair accuracy as their correct patch has already been generated. Nonetheless, T2 was able to recover the negative impacts of FNs by successfully generat- ing accurate patches for both FNs and TNs. We present specific numbers and impacts on the repair model across the 1st, 5th, and 7th pairs (one pair per model) in Table For the Intended tolerance for search setting ranging from -1% to -5,, the first rows of each pair show the number of wrong rejections due to FNs. However, as shown in sub- sequent rows, our proposed treatment T2 enabled substantial additional fixes for both FNs (T2 and TNs (T2rN). Notably, the combined number of T2rN and T2FN surpassed the number of FNs, resulting in an overall improvement of the repair model.
Compactness. Our behavior vector depicts the internal model behavior in a layer-wise and head- wise manner. Achieving a similar representation using token vectors significantly inflates the vector size to [layers] x heads embedding_ instance, C5deT5-base requires vector size of 110, 593(= 12 12 768), whereas our approach entails a vector size of just 144(= 12 x 12). Uti- lizing only the final context vector of the encoder could reduce redundancy, but it would result significantly lower explicitness.
For
In table 8, we report additional experimental re- sults on the CodeT55base model with the W Wn dataset to compare the effectiveness of behavior vectors against token vectors for false behavior classification. We employed the final context vec- tor(last hidden state of the encoder) of the last ken, which was either ":" or "}". Specifically, we selected the token that consistently appears in the same location. Therefore, the last token, which was either ":" or "J" was chosen. Hyperparame- ters for the classifier were selected with validation data that maximized the Fl-score, which the same method we used in Table 2. The classifi cation accuracy was 43.17 17 of Fl-score and 59.32 of Bal Bal_Ace, which are 47% (z/ 80.86-43.17) and 88.89
C Comparing Vector Representations for Behavior Classification
For the purpose of behavior classification, employ- ing the input/output vector (e.g., embedding vector, context vector) of the model itself is feasible. How- ever, our behavior vector offers an advantage in terms of explicitness and compactness, resulting in improved classification accuracy.
80.86
33% (Rs 09.89-59.32, lower than our behavior vec-
tor. For further comparison, we have conducted

---

experiments using the average vector of all tokens, which is another way to retrieve sentence embed- ding (Choi al., 2021; Huang et al., 2021). How- ever, resulted in lower accuracy; 43.17 for and 59.71 for Bal_Acc. Additionally, we have con- ducted experiments using NeedFix tokens, result- ing in 52.36 for F1 and 67.25 for Bal_Acc. The third and fourth rows in Table 8 summarizes the results. These results show the need for novel ap- proach to detect false behaviors of the model upon token vector embedding.


Reviews:
[{"review_id": "MnlrLApm1G", "reviewer": "Reviewer_1EGD", "summary": "", "questions": "A. Can you please clarify impact of false negatives on the system?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for acknowledging our main contribution. As for concerns about comprehensibility for a broader audience, we will enhance clarity by providing more detailed explanations, including supplementary details in the appendix. Below, we address the raised question.\n\n__Q1. Can you please clarify impact of false negatives on the system?__  \nFalse negatives of BeDisc are the cases where the repair model's correct behavior is wrongly classified as false behavior. As indicated by the reviewer's feedback, Table 2, whose hyperparameter search criteria was F1-score, displays a false negative rate (FNR) of 13.63% (100% - 86.37%) in the classification outcomes. This makes the system discard the correct patch, diminishing the efficacy of the repair model. In our paper, we addressed this by modifying the classifier's hyperparameter search criteria from F1-score to TNR with \"Intended Tolerance\". Therefore, we use the hyperparameter that maximizes false behavior detection within the specified \"Intended Tolerance\", which is determined by FNR. This adjustment reduces FNs and prevents the rejection of correct patches. \n\nAs a result, our treatment T1 was able to reject 48.6% ~ 80.1% of false behaviors with -1.4% ~ -9.2% of repair accuracy (corresponding to the average row in Table 3). Hence, T1 offers significant time savings for developers with a minor trade-off. Given the subjective nature of a \"minor trade-off,\" developers have the flexibility to establish an \"intended tolerance\" in accordance with their requirements.\n\nRegarding our second treatment (T2), negatives (Ns) of the classifier trigger the repair model to regenerate patches using masking. This occurs even if the correct patch has been generated without masking (FNs), potentially causing inefficiencies. Nonetheless, T2 successfully generates accurate patches for both FNs and true negatives (TNs). We present specific numbers and impacts on the repair model across the 1st, 5th, and 7th pairs (one pair per model) in Table 1. For the \"Intended tolerance for search\" setting ranging from -1% to -5%, the initial rows of each pair show the count of false negatives (FNs). This might reduce the repair model's accuracy due to incorrect rejections. However, T2 enables substantial additional fixes for both FNs (T2_FN) and TNs (T2_TNs). Notably, the combined count of T2_TN and T2_FN surpasses the number of FNs, resulting in overall improvement of the repair model.\n\n\nTable 1. Impact of False Negatives on Repair Model and Impact of T2 Despite the Presence of FNs.\n| Pair | `Impact on Repair Model` | -1% | -2% | -3% | -4% | -5% |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| 1 | *Wrong Rejection (FN)* | 95 | 123 | 180 | 398 | 504 |\n|  | Additional Fix (T2_FN) | 39 | 60 | 98 | 243 | 322 |\n|  | Additional Fix (T2_TN) | 449 | 477 | 486 | 522 | 529 |\n| 5 | *Wrong Rejection (FN)* | 18 | 56 | 65 | 90 | 167 |\n|  | Additional Fix (T2_FN) | 2 | 16 | 21 | 33 | 63 |\n|  | Additional Fix (T2_TN) | 261 | 417 | 437 | 460 | 490 |\n| 7 | *Wrong Rejection (FN)* | 37 | 35 | 87 | 344 | 416 |\n|  | Additional Fix (T2_FN) | 14 | 17 | 43 | 174 | 212 |\n|  | Additional Fix (T2_TN) | 248 | 249 | 303 | 388 | 415 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Along with a good overall presentation of the paper it has some strengths. Treatment 1, Abortions has good potential false behavior diagnosis. Also Treatment 2, masked Bypassing eliminates suspicious target tokens responsible for false behaviour while producing patches.\nOverall improvements seem to be significant. Qualitative and quantitatively significant datasets are used.", "reasons_to_reject": "One of the weakness is that as per table 2. still there are some false negatives. It is not clear how significant can such false negatives be for such systems.\nAnother weak point is overall the paper is not easy to read for general audience.", "questions_to_authors": "A. Can you please clarify impact of false negatives on the system?", "paper_topic_and_main_contributions": "Automated program repair is an widely discussed topic and has potential to reduce costs associated with bug fixing in software development\nand maintenance procedure. This paper proposes some novel techniques to improve performance of such program repairs.\nAuthors proposed a method to diagnose and treat false behaviors of transformer-based program repair models with the help of a behavior vector, a behavior discriminator (BeDisc) that identifies false behaviors, and two methods for false behavior treatment.\nTheir novelty is in improving program repair tasks by analyzing the internal behaviors of transformer-based models by demonstrating internal behavior of a transformer-based program repair model.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JjFp24TQD4", "reviewer": "Reviewer_aJHn", "summary": "", "questions": "- It is not clear if the proposed behavior vector is the best option to provide representation for classifying the patches. Could you please justify why we can not use the model itself to classify the patches? Or adding a special token (e.g., [CLS] token) to the inputs and employing that as the behavior vector representation?\n- It is not clear how the CodeGen model was trained (fine-tuned). Could you please provide the training details, including the learning rate, input structure, and input size?\n- Do you think the output of BeDisc can be used as an implicit uncertainty estimation? \n- Table 6 is a bit vague. What does T2 stand for (I know it refers to the treatments. However, it would be better to provide the details)? Why does the 7th pair cause the computational capacity issue?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the insightful comments and acknowledgment of our main contribution. If we get the chance to submit the final version of the paper, we will include supplementary details in the appendix. Here, we address the concerns and questions raised.\n\n__Q1.  Could you please justify why we can not use the model itself to classify the patches? Or adding a special token (e.g., [CLS] token) to the inputs and employing that as the behavior vector representation?__  \nEmploying the model itself based on the vector (e.g., embedding vector, context vector) of a specific token for the purpose of patch classification is feasible. However, our behavior vector is more explicit yet compact in depicting the model's behavior.\n\n - Explicitness: Token vectors are the *input/result* of inner workings, while our behavior vector encapsulates the *process* of inner workings. For instance, the context vector of a target token results from contextual adjustment involving a weighted vector sum of input tokens. However, we cannot know *how* it was used for contextual adjustment. This inherent ambiguity renders it less explicit than our behavior vector, which estimates the degree to which the model utilized the token during context adjustment. Thus, our behavior vector offers a more explicit representation of the inner workings, making it easier to identify the false behavior of the model.   \n\n- Compactness: Our behavior vector depicts the internal model behavior in a layer-wise and head-wise manner. Achieving a similar representation using token vectors significantly inflates the vector size to |layers| * |heads| * embedding size. For instance, CodeT5-base requires a vector size of 110,593 (12 * 12 * 768), whereas our approach entails a vector size of just 144 (12 * 12). Utilizing only the final context vector of the encoder could reduce redundancy, but it would result in significantly lower explicitness.\n\nAlso, we have conducted additional experiments on the CodeT5-base model with the wild-small-na dataset to compare the effectiveness of behavior vectors against token vectors for false patch classification. We employed the final context vector(last hidden state of the encoder) of the last token, which was either \";\" or \"}\". Hyperparameters for the classifier were selected with validation data that maximized the F1-score, which was the same method we used in Table 2. The classification accuracy was 43.17 of F1-score and 59.32 of Bal_Acc, which are 47% (=(80.86-43.17)/80.86) and 33% (=(88.89-59.32)/88.89) lower than our behavior vector.\n\n__Q2. It is not clear how the CodeGen model was trained. Could you please provide the training details, including the learning rate, input structure, and input size?__  \nFor input and output structure we have described it in the last paragraph of Appendix A.2. \n>...fine-tuning was performed to learn the sequences of\"{original input format of the dataset} Fixed: {fixed code}\". For inference, the original input was provided to the model to generate the text. The text returned by the model was split with “Fixed:” and posterior tokens were treated as a generated patch.\n\nWe trained CodeGen-350M-multi [1] with a learning rate of 5e-5 and batch size 4, with a max length of 512. The pretrained tokenizer does not have pad_token, so we have set eos_token as pad_token. The repair model is trained with a transformers library[2]. The specific modeling code of Codegen we used for our experiments can be found in *\"Supplements\\transformers.zip\\transformers\\models\\codegen\"*. In the JSLint dataset, there were 84,846 instances for training, 9,454 for validation, and 10,504 for the test set. For the WS dataset, the corresponding numbers were 46,680 for training, 5,835 for validation, and 5,835 for the test set.\n\n\n__Q3. Do you think the output of BeDisc can be used as an implicit uncertainty estimation?__  \nThis question is intriguing. Since the behavior vector represents how the model regard/disregard the token, using it for uncertainty estimation appears feasible. We should explore how model uncertainty relates to 1) the Deviation of RS^{lh} across layers and heads, in addition to 2) the overall value of RS^{lh} value for uncertainty estimation. Although uncertainty estimation is not our expertise, we speculate that behavior vectors could aid in estimating token-level uncertainty in language models. We would like to further investigate this topic in the future.\n\n\n__Q4. Table 6 is a bit vague. What does T2 stand for? Why does the 7th pair cause the computational capacity issue?__  \nThe objective of this experiment, reported in Table 6, was to investigate how severe the false behavior of the original model was, which can be treated by T2. If T2 is applicable to only moderate false behaviors, it will lift the rank of patches that were already in high rank to Top1. Also, we wanted to check if the remedy is possible by simply enlarging the beam size. However, our investigation shows that T2 was applicable to severe false behaviors, whose fixing was impossible in the Top-50, even with enlarged beam size. The \"Ranks in Original Model\" column shows the number of instances that T2(with beam-size 5 generation) can improve its rank from x-th-> 1st (x<=2). The MRR column shows its impact overall. The higher the MRR, the less effort is required to validate the patch. The MRR improvement was possible because T2 was able to move the rank of the correct patch to Top-1, whose rank was even out of Top-50 (with beam-size 50 generation). \n\nThe 7th pair incorporates the largest model and the largest text input. This pair caused the computational capacity issue when generating top-50 patches with beam size 50, which is the minimal beam size we need to generate top-50 patches. Consequently, the model failed to generate patches in some instances because of an OOM error, even with inference batch size one. Therefore, we did not investigate this pair as the result might be an incomplete comparison.  \n\n[1] https://huggingface.co/Salesforce/codegen-350M-multi    \n[2] Transformers: State-of-the-Art Natural Language Processing (https://github.com/huggingface/transformers)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposed a novel perspective to improve program repair tasks by considering the false behavior of the models.\n- It provides extensive experimental results on three different models and four different datasets.\n- It shows the effectiveness of the approach in identifying correct and incorrect patches. Furthermore, it employs false behavior diagnoses to improve the APR models. \n- The paper is well-written and easy to follow.\n", "reasons_to_reject": "- The novelty of the behavior vector is questionable and it is not clear if it works better than using the model itself to classify the patches (e.g., using a specific token like [CLS] as the classification vector representation, see the “Questions For The Authors”) .\n- The training details of the CodeGen model are not provided. E.g., the input size, learning rate, etc.\n", "questions_to_authors": "- It is not clear if the proposed behavior vector is the best option to provide representation for classifying the patches. Could you please justify why we can not use the model itself to classify the patches? Or adding a special token (e.g., [CLS] token) to the inputs and employing that as the behavior vector representation?\n- It is not clear how the CodeGen model was trained (fine-tuned). Could you please provide the training details, including the learning rate, input structure, and input size?\n- Do you think the output of BeDisc can be used as an implicit uncertainty estimation? \n- Table 6 is a bit vague. What does T2 stand for (I know it refers to the treatments. However, it would be better to provide the details)? Why does the 7th pair cause the computational capacity issue?\n", "paper_topic_and_main_contributions": "**Summary:**\n\nCurrent Automatic Program Repair (APR) studies mainly focus on improving these models to generate correct patches for the faulty program. However, investigating the false behavior of these models was not addressed in the previous works. To address this, the paper proposes a methodology for diagnosing and treating the false behaviors of the transformer-based program repair models. The paper mainly proposes a behavior vector together with a behavior discriminator to quantify the behavior of the models and employ them to improve the general APR models’ performance.\n\n**Contributions:**\n\n-The paper introduces a novel perspective to improve program repair tasks by diagnosing and treating the false behaviors of transformer-based program repair models.\n-The paper proposes a behavior vector based on transformer-based models to diagnose and mitigate false behaviors of the model in generating the patches for the given buggy program. \n-The evaluation results for 55,562 instances of different datasets showed the effectiveness of the approach in identifying correct and incorrect patches (It correctly classified 86.9% of correct and 86.4% of incorrect patches).\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- At 2.1 add what is d_k (Eq 2)? I suggest to provide one or two sentences about that.\n"}, "extra_text": {}}, {"review_id": "ZlFFZiAbte", "reviewer": "Reviewer_GLov", "summary": "", "questions": "- Is the \"Intended Tolerance for Search\" in Table 5 derived under the same parameter settings as the \"Intended Tolerance for Search\" in Table 3?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments regarding the readability of our paper. In response to the reviewer's comments, we will enhance the paper's clarity by providing more specific explanations. Below is the response to the reviewer's question.\n\n__Q1. Is the \"Intended Tolerance for Search\" in Table 5 derived under the same parameter settings as the \"Intended Tolerance for Search\" in Table 3?__\n\nYes, the results in Tables 3 and 5 utilized the same hyperparameters for each tolerance.  We apologize for the inconsistent column names, which could have led to confusion. We will modify \"Intended Tolerance\" in Table 3 to \"Intended Tolerance for Search\" to align with Table 5.  \n\nThe BeDisc's hyperparameters were selected based on validation data from the dataset for each pair. Therefore, the same hyperparameters are selected as long as the pair and \"intended tolerance\" are identical. We introduced the concept of \"Intended Tolerance\" to confine FNs to a limited scope, thus preventing the rejection of accurate patches. The classifier's FNs can erroneously discard correct patches, thereby reducing program repair accuracy. To address this, we searched hyperparameters that maximize the TNR (False patch correctly identified as False) while maintaining program repair accuracy within a specified standard. \n\n__Concern about description on attention mask:__  \nFor further clarification, we would like to add an explanation about the attention mask.  \n\n> Attention mask can indicate the model in which tokens should be attended to [1][2]. For example, when we want to disregard the token *b* for a token sequence {*a b c*}, we can use the masking array {1, 0, 1}, and the model uses {*a _ c*} as the new input instance based on this masking array.\n\n[1] https://huggingface.co/transformers/v3.5.1/glossary.html#attention-mask  \n[2] Impact of Defect Instances for Successful Deep Learning-based Automatic Program Repair (ICSME'22)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper enhances the performance of the program repair model by analyzing its internal generation process from a novel perspective. This direction of improvement represents a promising path worth exploring.", "reasons_to_reject": "- The writing and readability of this paper leave room for improvement. Some key points in the article lack detailed descriptions and explanations, making the content unclear and difficult for readers to comprehend.\n    - The paper fails to clearly explain the difference between the Attention MASK introduced in section 3.2. There is a lack of example illustrations.\n    - The meanings of each row and column in the results tables are not clearly explained, and the purposes of some experiments are not explicitly stated. Furthermore, it is unclear whether the parameters of the models remain consistent across different experimental results. It would be beneficial to clarify whether the observed improvements result from parameter variations or methodological enhancements.\n    - The distinctions between Balanced Accuracy and the F1 score are not clearly explained, nor are the respective functions of these two metrics.\n", "questions_to_authors": "- Is the \"Intended Tolerance for Search\" in Table 5 derived under the same parameter settings as the \"Intended Tolerance for Search\" in Table 3?\n", "paper_topic_and_main_contributions": "- This paper explores the diagnosis and treatment of False Behavior in the Patches generation process within the Program Repair Model.\n- This paper proposes a highly compatible framework for diagnosing and treating False Behavior and accordingly introduces the Behavior Vector for diagnosis. Additionally, two methods for treatment are proposed: abortion and masked bypassing.\n- Experimental results on numerous datasets show that simply adding the modules proposed in this paper can effectively increase the number of correctly repaired programs, without changing the baseline.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In line 12 of section 2.1, \"wight\" should be corrected to \"weight\"."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Program behavior diagnosis is a very challenging and important problem, and the authors have made an interesting and impactful contribution to the problem space. While their approach has false positives and negatives, these seem comparable even to some static analysis methods I have seen for detecting bugs.
Decision: Accept-Main