An interpretable error correction method for enhancing code-to-code translation
Authors: Min Xue
Artur Andrzejak
Marla Leuther
Venue: ICLR 2024 poster (2024)
Keywords: Computation and Language; Programming Languages; Software Engineering; Machine Learning
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Transformer-based machine translation models currently dominate the field of model-based program translation. However, these models fail to provide interpretative support for the generated program translations. Moreover, researchers frequently invest substantial time and computational resources in retraining models, yet the improvement in translation accuracy is quite limited. 
To address these issues, we introduce a novel approach, $k\text{NN-ECD}$, which combines $k$-nearest-neighbor search with a key-value error correction datastore to overwrite the wrong translations of TransCoder-ST. This provides a decision-making basis for interpreting the corrected translations. Building upon this, we further propose $k\text{NN-ECS}_{m}$, a methodology that employs a distributed structure with $m$ sub-datastores connected in series,  utilizing $m$ diverse experts for multi-round error correction. Additionally, we put forward a unified name rule, encouraging the datastore to focus more on code logic and structure rather than diverse rare identifiers. Our experimental results show that our approach improves the translation accuracy from 68.9\% to 89.9\% of TransCoder-ST (for translation from Java to Python). This error correction method augments program translation, overcoming the inherent limitations of Transformer-based code translation models, such as resource-intensive retraining requirements and uninterpretable outcomes.

Published as a conference paper at ICLR 2024
AN INTERPRETABLE ERROR CORRECTION METHOD FOR ENHANCING CODE-TO-CODE TRANSLATION
Min Xue* Artur Andrzejak Marla Leuther Heidelberg University, Germany (mi xue, andrzejj,, Leuther)@uni -eiieunni-hiiiee hew.:/eereeiieiiieeeeeeeeeeeeeee
ABSTRACT
Transformer-basee machine translation models currently dominate the field of model-based program translation. However, these models fail to provide inter- pretative support for the generated program translations. Moreover, researchers frequently invest substantial time and computational resources in retraining mod- els, yet the improvement in translation accuracy is quite limited. To address these issues, we introduce a novel approach, &NN-ECD, which combines k-nearest- neighbor search with a key-value error correction datastore to overwrite the wrong translations of TransCoder-ST (Roziere et al. 2022). This provides decision- making basis for interpreting the corrected translations Building upon this, we further propose &NN-ECSm. methodology that employs distributed structure with m sub-datastores connected in series, utilizing m diverse experts for multi- round error correction. Additionally, we put forward unified name rule, encour- aging the datastore to focus more on code logic and structure rather than diverse rare identifiers. Our experimental results show that our approach improves the translation accuracy from 68.9% to 89.9% of TransCoder-STT (for translation from Java Python). This еттог correction method augments program translation, over- coming the inherent limitations of Transformer-basee code translation models, such as resource-intensive retraining requirements and uninterpretable outcomes.
INTRODUCTION
Transformer- basse large language models (LLMs) (Vaswani et al. 2017). such as BERT (Devlin et al. 2018), XL. (Lample & Conneau 2019) and XLNet (Yang et al. 2019). have been widely used in the field of program translation. In recent work, researchers have attempted enhance Transformer- baadddd program translation by incorporating compiler intermediate representations, but the improvements remain limited (Szafraniec al. 2023 Rozière al. al.| age large amounts of data collected from public repositories such as GitHub and GitLab, combining unsupervised and self-supervised learning to overcome the need for parallel corpora. Neverthe- less, these techniques often require substantial investments in computational resources for retraining translation models, while yielding only marginal improvements return. Compared to retraining the L.MMs, Error Correction emerges as a more efficient alternative, enhancing accuracy through repairing wrong translations produced by the program translation model. This approach holds sig- nificant practical value, making it possible to enhance program translation accuracy at low cost.
[2221). ). These methods lever-
The internal workings Transformer-basee models are relatively complex, making challenging to track and identify which snippets in the training dataset contribute each output token. In other words, the Transformer-basee for the output. A common approach is exploit knowledge neurons and causal effects to explain the output of the transformer model (Dai et al., 2022 Vig et al.) 2020). In the recent research, ford-forwwrd modules in storing factual associations, but it has been so far applied only to simple peapiiettpredicat-objeeee sentences. In con- trast, the t-nearest-neighbor machine translation (kNN-MT) combined with large-scale datastore has demonstrated remarkable capacity in optimizing and interpreting natural language transla- tions (Khandelwal et al. 2020 Zhang al., 2018 Tu al. 2017). Compared with traditional
program translation model cannot provide an intuitive interpretation
Meng et al. (2023) revealed the crucial role of middle-layed
"Corresponding author.

---

Published as a conference paper at ICLR 2024
Transformer- based models, datastore possesses inherent interpretability. The kNN retrieval method can provide a clear inference path and recision-making basis through tracking and identifying which snippet in the training dataset contribute to each generated token, without the need to analyze com- plex neural network hierarchies. Based on this, we consider integrating the ANN-MT method with error correction, which exhibits great potential in providing an interpretable correction analysis.
In modern source code datasets, among millions of unique identifiers, only less than 1% of the identifiers appear frequently (Karampatsis et al. 2020). Diverse rare identifiers, such as function names, variable names, and parameter names, often increase perturbations during the model training and inference phases (Chirkova & Troshin, 2020b). This phenomenon is akin introducing noise into the process of information transmission, making it difficult to focus on the key information. Notably, although these different rare identifiers tend to introduce noise in program translation and error correction, only a few researchers have paid attention to this issue. For this problem, we propose a unified name rule to replace diverse rare identifiers during the training and testing phases. minimizing the emphasis on rare identifiers and focusing more on code logic and structure (as shown in Figure 2
Our work builds upon the Transformer-based code translation models proposed in the TransCoder (Roziere et al., 2020) and TransCoder-ST (Roziere et al. 2022) projects. TransCoder employed self-supervised learning across multiple programming languages (between Java, C++, and Python), and then TransCoder-S" extended it by introducing a rigorously tested parallel corpus. In this paper, propose extract error correction information from TransCoder-ST guide the error correc- tion model in learning repair knowledge. More specifically, we create unit tests for the Java source dataset, and then use TransCoder-ST to generate multiple Python functions with unit tests for each Java function. Subsequently, we conduct unit testing on the Python functions, and then extract the first failed Python function and the first successful Python function to form an error correction language pair for each Java function. Based on this, we establish two alternative error correction models, kNN-ECD and &NN-ECS,,, to improve the translation accuracy of TransCoder-ST by cor- recting wrong translations. Overall, our contributions are as follows:
We propose ANN-ECD, which leverages &NN search to retrieve correction information from the error correction datastore, thereby repairing wrong translations in TransCoder-ST Notably, the error correction process is interpretable, overcoming the opaque defects of Transformer-based program translation.
We introduce &NN-ECSm, approach that employs distributed structure with m small datastore units, capturing comprehensive repair information by using multiple datastore variants for multi-round error correction. This approach promotes the traditional ANN-MT technique, effectively improving the data retrieval capabilities by trying diverse data flows.
We introduce a unified name rule, which ignores the differences in rare identifiers and focuses more on the logic and structure of the code, thus avoiding interference caused by irrelevant identifiers on similarity retrieval.
We evaluate our approach, showing that the translation accuracy of TransCoder-ST (Java-PPython) significantly improves from 68.9% to 89.9% after employing kNN-ECD/ ANN-ECS,p. Without the necessity of retraining Transformer-based models, our approach still achieves substantial improvements in program translation.
RELATED WORK
Transformer- Program Translation. In recent years, Transformer-buus based unsupervised learn- ing methods have become the mainstream approach in the field of program translation (Shi et al.) 2022 ZZhng et al. 2023). TransCoder (Roziere et al. 2020), as the first work that combined an unsupervised model with programming translation, pioneered automatic program translation in the field of software development. Benefitting from the structural characteristics of programming lan- guages, Rozière et al. [202l) introduced new pre-training objective, DOBF, to recover the original version of the obfuscated source code by pre-training a model. Based on this, Roziere et al. (2022) developed TransCoder-ST which utilizes automated unit testing system to filter out invalid trans- lations, and then uses a well-tested parallel corpus to fine-tune the unsupervised model (Radford
et al. 2018: Yang al. 2019 Raffel al. 2019 Pan al. 2023). Subsequently, (Szafraniec et al.
2

---

Published as a conference paper at ICLR 2024
Error Correction Dotaset with original somes
Xbb ferec
NEW LINE DEPENT
Wrong Source
Trmesaatons
Corrected Trandations
sallied ni
LNTTCH LBC
Corrected Translatins
Erroe Curration Dutasce with unified nanes
Tallec
Phase 3: KNN-ECD
Filter
Frrse
Wreng Source
Carmentd Transtations
Transhsions
Ifbuld BCS,?
Decoder
FCD
Key bazs
excrmid Roner Tradston
Small Datnset
AAN ANNSLL M.D.
Curnected Translations
Faltin Tol
Small Dataset;
(Proces erver cervchis Seszaoge pain so -aay valec poire)
Sol.
Daraset)
Test
Generated token
Translaions
Natterreed Wrrr Seams Trandaten
Query Aoc
Small Datsvel,
Ialasture
23% Neh TCl.
Teal
Transhatinss
Cornected
Torget token
Fallar
Phase 1: Data Processing
Phase 2-2: Perlorm KNN Retrieval
Phase J: &VN-ECSS
Training process: Construction workflow of &NN-ECD and kNN-ECS,.. The work- flow consists of three phases. (1) After deduplication, we can get the error correction dataset. If we intend to build &NN-ECD, we use the еπоо correction dataset directly; if we intend to build kNN-ECSm we divide the error correction dataset into m small datasets. (2) For the error cor- rection dataset/small dataset<e[1.m) we process it into corresponding datastore and introduce the ANN retrieval. Here, we can get the corresponding KNN-ECID/NNN-sUL BaiNNsuu__DCDDeiaeeee FCD; respectively. (3) For kNN-ECD, we directly conduct unit testing on the output; for &NN-ECS,. connect Lub_ Sub,ECD,,.... SuD_ m in series and carry out unit testing on the output each sub_ECD mm
Figure I:
2023) proposed TransCoder-IR, which leverages lower-level compiler intermediate representations
to advance code translation. Due to the success of pre-trained language models, Feng et al. (2020) introduced CodeBERT, a neural architecture combining BERT (Devlin al. 2018), RoBERTa (Liu et al.) 2019) and a bidirectional Transformer (Vaswani et al. 2023). trained with a hybrid objec- tive function that incorporates both natural language and programming language. Further, in order to accommodate multilingual representations, Ahmad et al. (2021) employed bidirectional and au- toregressive transformer (Lewis al. 2019) for pre-training on unlabeled natural language and programming language data. Currently, almost all program translation models are based retrain- ing the Transformer model. However, the complex internal workings Transformer models leads to uninterpretable translations. Furthermore, this process usually consumes significant amount of computational resources, yet the gains in translation accuracy are quite limited.
ANN-MT Method. As method for providing interpretable results, the datastore- &NN trieval approach promising alternative to Transformer based translation models. Khandelwal et al. (2020) pioneered the concept of ANN-MT, which augments natural language translation by retrieving (key, value) pairs from an external datastore without updating the model. However, large- scale datastores often suffer from high-latency data retrieval. To address this problem. Wang et al. (2021b) introduced a hicrarchical clustering strategy (Kanungo et al.) 2002), aiming to improve re- trieval efficiency by approximately querying the distance between data points in datastore. On this basis, Wang et al. (2022) proposed to use the cluster-basse based compact network and cluster- pruning solution to compress datastore, thereby reducing the ANN retrieval latency. Based on the traditional &NN-MT method, researchers have initiated the exploration of more adaptive &NN-MT paradigms. Zheng et al. (2021) introduced the concept of adaptive ÀNN-MT, dynamically customiz- ing the number of nearest neighbors for each target token. Furthermore, in order to achieve more interactive and efficient learning method, Waan et al. (2021a) proposed the ANN-over-KNN (KoK) method plug-and- aldyply solution for online learning combined with human feedback. However, existing works only focus on the retrieval speed and context adaptation while overlooking the limited search capability, failing to capture global information in large datastore space.
Diverse Rare Identifiers. Various rare identifiers often introduce noise in program translation, dis- tracting attention from critical information such code structure and logic. Modern source code
3

---

Published as a conference paper at ICLR 2024
Test ingut: wrong transiatio
Wrong translation unified names
def meseee
det nundjads
min e
onpeennnueeeern sumbentit
Unified Name Rule (nepoocemen, name)
Unit Test
esc
min mnbommbber)]] nusneeed] +
(6206 u000 20
20
Corrected translution with recovered numes
output: corrected trandation
KNN-ECD
de'
der nundad] ed
WNN-ECSm
Hommmmarra we miin numbencii
Recover Name
optocally
. M011
(uumbeng rumbert
Jui midddjdd 4000
Figure 2: Testing process: Implementation workflow for KNN-ECD and kNN-ECSm. First, we apply the unified name rule on wrong translations and store (replacement, name) pairs. Subse- quently, we perform the unit testing on the processed wrong translations, and then feed the failed translations to the eπтоr correction model. If necessary, we can recover the masked names in the corrected translations using the (replacement, name) pairs.
datasets contain millions of unique identifiers, but less than 1 1% of them occur more than 5 times, despite using open vocabulary approaches like BPE (Karampatsis et al.. 2020). To address this issue. [Chirkova & Troshin (2020b) presented an identifer-anonyyizatoon based approach to han- dle Out-of-Vocabulary identifiers, which significantly improves Transformer's 's performance in code completion (Svyatkovskiy et al. (020) and bug fixing (Gupta et al. 2017 Li et al. 2020). Addition- ally,,Xu et al. (2019) also recogmized the limitation of Out-of- Oft-of-Vocabulaayy terms, proposing to replace all class/variable names with corresponding placeholders. For unfamiliar identifiers, AAhmed et al. 2018)) proposed the TRACER method, which handles these cases by replacing all identifiers with their recommended abstract types. Furthermore, by incorporating more versatile value anonymiza- tion process, Chirkova & Troshin (2020a) explored the effectiveness of anonymization across a broader range of tasks based on the Transformer architecture. In previous works, researchers have ignored the possibility of unifying diverse identifiers in both training and test datasets. Also, existing methods are extremely simple, often breaking the code structure and neglecting to avoid replacing special identifiers such as built-in functions.
3 APPROACH
In this paper, we aim to enhance code translation through error correction, rather than retraining the program translation model. As shown in Figure [ following data processing, create the corre- sponding datastore and perform ANN retrieval on two variants of error correction dataset, construct- ing two alternative error correction models to enhance TransCoder-ST: ANN-ECD and kNN-ECS,..
3.1 CREATION OF ERROR CORRECTION DATASET
Generating error correction language pairs. We intend to extract error correction language pairs from TransCoder-ST. guiding &NN-ECD and KNN-ECS,, in learning correction knowledge. First, we download the Java source dataset from Google BigQuery and process it using the TransCoder- ST preprocessing pipeline. Subsequently, we employ EvoSuite (Dinella al. 2022) to create high- quality Java unit test cases, and then feed the processed Java functions with the test cases into TransCoder-ST (Java - Python, beam size- N). For each Java function, we can get N Python functions with their corresponding unit test cases, followed by executing the Python unit tests. If both 'success" and 'failure' oceur N test results, we combine the first failed Python function with the first successful Python function to form error correction language pair.
Unified name rule. Inspired by OOV anonymization method (Chirkova & Troshin 2020b), we propose unified name rule to standardize the diverse rare identifiers in the training dataset and test dataset, thus reducing noise in the retrieval process. First, we categorize non-built-in identifiers into three groups: variable names, function name, and parameter names, where each function has only one function name. Then, we replace the identifiers initially used in the code with identifiers following homogeneous schema of naming, as follows:
4

---

Published as a conference paper at ICLR 2024
Unified name rule: For non- non-built-in identifiers in function, we sequentially replace parameter names with ag, d1..... aj, function name with func, and variable names with bu, bi....,du by. During this process, the (replacement, name) pairs of each function recorded. If needed, the original name can be recovered.
In Figure 2 give example of unified name rule used for identifier replacement. By imple- menting the unified name rule, we can filter out extraneous information from rare identifiers, thereby channeling attention toward essential code elements such as logic and structure. Besides, this rule fa- cilitates preliminary error correction within the test dataset, repairing errors resulting from identifier confusion in the code, such as the overlap between function names and variable names.
3.2 ERROR CORRECTION USING &NN-ECI
Creating error correction datastore (ECD). The BCD is generated from the error correction dataset, which mainly includes two primary storage components. The first component is designed to store error correction knowledge in the format of (key, value) pairs. where the generation of (key, value) pair follows ANN-MT method (Khandelwal et al., 2020). The second component is used store source functions and target function prefixes (src June, tgl func pref), ; as well as their corresponding ground truth target tokens tgt tok. More specifically, we use the pre-trained Transformer model sentation of (src.func, igt-ffnc.prre value h(tgt_tok) is the tokenization of tgt_tok (Ferrando et al., (022). For each token gener- ated from (key, value) pairs, users can obtain a detailed explanation by accessing the corresponding e func, tgt pref) + tgt tok, where ferc.fune, tgt funcepref serves the decision- making busis for the generated token Igl. lok.
coding tool, where key = f((sre_func, obtained from the last hidden layer of the decoder, and
sott_func.pee]l
is the repre-
Performing N:N retrieval on ECD. Combined with the ECD, we utilize &NN retrieval to correct the wrong translations of TransCoder-ST.. The kNN method conducts the similarity search on large-scale datastore, retrieving relevant (key, value) pairs to generate error correction results. In the code correction process, when generating the next token y; each step, we utilize the representation fæ, (1:-1) as a query, based on the test input x. Following this, we retrieve the nearest neighbors to the query from the еттог correction datastore, where タ represents the generated token. Then, calculate the distance d() between the key and the query as a weight to regularize the probability of the value. On the basis of KNN-MT (Khandelwal et al., 2020), we define the probability distribution of the next token as follows:
1 exp ( (=d(ky,fe f(x, 014-1))
p (y: Ir, D14-1) =
Σ (NG
T
where 7 is the temperature, used to prevent overfitting in the retrieval context. When the temperature value 7 > 1, it tends to make the distribution more uniform, preventing deviations towards the most similar retrieval results and ensuring diversity in the retrieved data.
3.3 ERROR CORRECTION USING ANN-ECS
Creating error correction system (ECS,.). As shown in Figure I ECS,,, consists of m sub-datastores, which are connected in sequential order. To build ECS... we first randomly divide the eтгor correction dataset into equal parts (staal small dattasttmm where each small_dataee dataettdtas contains the same number of error correc- tion language pairs It means that ECD and ECS,, are generated from the same emor cor- rection dataset. Then, we follow the ECD creation process to generate the correspond- ing (sub ECD1..... sub ECD,.... sub ECD,,.. where these sub ECDs are linked sequentially. Here, each sub FCD contains two primary storage components. The first components dedicated to storing (key, value) pairs, while the second components is designed for storing fsrc.func, fggtffunc.. func.pref)
....smllldatasee
tok records.
Performing &NN retrieval on ECS,.- Within the framework of ECS.., we repeatedly feed incor- rected wrong translations into subsequent sub Lub_ECDie[1_.m and perform kNN retrieval, using diverse data flows for multiple rounds of error correction. Specifically, for each sub ECD)e(1.m)) we employ ANN retrieval to generate outputs and conduct unit testing the generated results. If the unit testing
5

---

Published as a conference paper at ICLR 2024
result is successful, it means that the wrong translation has been corrected. In this case, output the corrected code. However, if the unit testing result failed, means that the wrong translation has not been adequately corrected. In response, sub-datastore. By adopting a distributed structure with multiple sub-datastores, INN retrieval can capture more comprehensive repair information, effectively enhancing the kNN search capability.
re-enter the wrong source translation into the next
EXPERIMENTS
4.1 EXPERIMENTAL DETAILS
Model architecture. The implementation of ANN-ECD/ANN-ECS, is conducted under the guid- ance oracle (Loney & McClain 2004), where oracle represents a group of manual unit testers. By reviewing the output and providing feedback, oracle assists us in determining whether the corrected results make sense. As shown in Figure 2 we implement the unified name rule on the test input for preliminary error correction before feeding it into ANN-ECD/KNN - Then, we execute unit testing on the processed test dataset. If successful, it means that the wrong translation is due to identifier confusion. If failed, we feed the failed test samples into ECD/ECS,- (Here, we define the model as &NN-ECD" ANN-FCDD/ANNN-FCSS; /ANN-ECS m when the unified name rule is not employed during the training and testing phases.)
Training dataset & test dataset. We download the Java source code from Google BigQuery and implement the TransCoder-ST preprocessing pipeline for dataset filtering. Next, we set the maxi- mum runtime of 20 seconds for each process and employ EvoSuite to create Java test cases. Here, the unit test cases are created based on two criteria: mutation score over 0.9 and at least two assertions. In this process, we collect 82,665 Java functions with unit test cases.
Subsequently, we feed the above processed Java function into TransCoder-ST (beam size = 10), and set the target language as Python. In this step, 82, 665 x 10 Python functions with unit test cases are generated. For each Java function, we perform unit testing on the corresponding 10 Python functions. If 'success' and 'failure' both appear in 10 test results, we merge the first failed Python function and the first successful Python function into an error correction language pair. Ultimately, can get preliminary error correction dataset with 26,230 language pairs for TransCoder-ST.
After that, we employ the unified name rule to standardize diverse rare identifiers within the prelim- inary eπror correction dataset. By removing duplicate entries from the processed dataset, error correction dataset with 21,385 correction pairs is built. Then, we divide the error correction dataset into training dataset and test dataset with ratio of 9 : 1. Among them, the training dataset consists of 19,368 error correction language pairs, and the test dataset consists of 2,017 wrong translations, each with unit test cases and (replacement, name) pairs]]
Training details & fine-tuning details. During the training phase, we introduce the previous ver- sion of TransCoder-ST as coding tool for error correction language pairs, mainly because its en- coder and decoder can simultancously process codes in the same programming language, which conforms to the data properties of error correction language pairs (Qi al., 2018). Based on this, we feed the error correction language pair into the coding tool, where we treat the correct translation as targer and the wrong translation as source. Then, we extract the cross-attentio output as key, and the tokenization of the ground truth target token as value.
During the fine-tuning phase, we evaluate the performance of ECD and ECS,, with different numbers of sub-datastores, where m E {3, 6, 9, 12, 15, 18). We consider the following param- eters: neighbor (po) and temperature (p1). When more neighbors are retrieved, noise may be introduced, which can lead to worse results. Meanwhile, properly adjusting the temperature pa- rameters can prevent excessive bias towards a single neighbor, thus ensuring the diversity of re- sults. In this case, we test the performance of ECD and ECS,,, under Po E {1, 4, 8, 12, 16, 32), Pi E [1, 10, 100, 1000}. and select the optimal parameter combination. Experiments show that for ECD, when po: Pi = (4, 10). the best error correction rate can be achieved. For ECS,, with m = 3, 12, 15, the optimal parameter combination is Po, P1 {8, 10). ECS,,, with m = 6, 18 attains the highest effectiveness with the parameter combination Po. = {16, 10). As for ECS,.
with m = 9, the most effective parameter combination is Po. P1 = {2, 10).
http://githuhhcom/minnnncco 1/Fmrr Correction

---

Published as a conference paper at ICLR 2024
4.2 RESULTS AND DISCUSSION
Translation performance. In Table we compare the translation performance of TransCoder-ST after combining ECD/ SCD/ECS,,, or ECD /ECS In the comparison, we focus on the translation from Java to Python. where the performance of our models is measured against j2py TransCoder, DOBF, TransCoder-ST (TC-ST). As shown in Table[]] is clear that when TransCoder-ST utilizes the error correction model, the translation performance improves significantly, increasing from 68.9% to a range of 82.4% 89.9%. This improvement stems from the datastore, which stores a large amount of error correction information generated from TransCoeer-ST. By systematically retrieving relevant (key, value) pairs, we can correct relative errors in the wrong translations. Moreover, comparing KNN-ECD BCD/KNN-ECSS,, with ANN-ECD' /KNN-ECS, the former shows superior performance. This difference is attributed to the unified name rule, which reduces the interference caused by rare identifiers during datastore construction and implementation.
Table 1: Translation performance of TransCoder-ST with ANN-ECD/ANN-ECC, m (Java - Python).
KNK ECS1s/
ENN DCSt0/
ANN.FCS1a/ ANN-ECS,
ENN /CS ANN-ECS,
ENN ECSa/
ANN ECSa /
ECD/
ANN-ECS
ANN-ECS,
ANN-ECS
ANN-ECS,
ANN-ECD
p2py
38.3%
38.3%
38.3%
38.3%
38.3%
38.35
38.3%
TranSooder
49.0%
49.0%
49.0%
49.0%
49.0%
49.0%
49.0%
DOBF
52.7%
52.7%
52.7%
$2.7%
52.7%
52.7%
52.7%
Pure TC:ST (Online)
68.9%
68.9%
68.9%
68.9%
68.9%
68.9%
68.9%
TC-ST EEDD /BCS;
89.4%
89.1%
88.7%
87.9%
87.1%
85.15:
82.4%
TC-ST + FCD/ECS,,,
89.9%
89.6%
89.3%)
89.1%
87.9%
86.5%
84.5%
We direcily compare our
research, TransCidk-IRR IR, because it not suitable translating
Python.
Interprefability of error correction model. During the testing phase, the traditional Transformer model cannot track and identify which snippets in the training dataset contributed to each gener- ated token. However, ANN-FCID/ANN-ECSS, can address this issue, providing an intuitive and readable decision-making basis for each output token. When building the datastore, we store the (key, value) pairs and the corresponding snippets (sre_fune, tgt__f pot_ffnc_preff) + tgg_tok from the training dataset, where the coding of (src_f unc, fgt_ffnc..rref" is key, the coding of tgt tgk_ is value, and (src_f fnc,, fgt-ffnnc.pre tyt.tok. Consequently, when generating each output token, will return both (key, value) pair and (src. func, tgt func pref) - tgt tok. In APPENDIX T we show a detailed decision- making process for generating corrected translation. For clarity, we only focus on the correction process of the wrong token '/' + '//" in Table 2 where the (key, value) pair used to correct the a string of numbers, making it challenging to extract valid intuitive infor-
serves as the decision-making basis for the generated token
wrong token - '// mation. In such cases, the uncoded form (.--) + '/'" of (key, value) pair can provide more intuitive information.
Table 2: Interpretable error correction process for repairing incorrect tokens in wrong translations
Wrong translation
Corrected translation
Decision making busss
dac coturn 50
daf suno roturn a0 /7
func : LINE INDENT return
22
NEW LINE DEDENT', 'def func (aD): NEW.IINE
INDENT retum ab") - '
det Funo Hist
tuno ( a0
fune (50): NEWJ..NN INDENT seet () NEW J.INE return les all 2 J
. eert )
b cart [ return b len b //
DEDENT'. 'def func a0) NEW LINE INDENT a0.
return b I len 1 /
son () NEW NFW.LINE return al len (=0 + '2"'
def Funo =0
suno
{ "del Fune (:0): NEW LINE INDENT =
0
INDENT (30 10 NEW LINE a0 a0 10 NEW
al
while al
whi le 50 D.:
if a0 10 b 1-| z0
12 a0 $ 10 -- 2
DEDENT relum sum NEW LINE DEDENT". 'del func NEW.LINE while all INDENT +s (0 10) **2
NEW LINE
20
20 10
>> Q:
return
NEW LINE a0 a0)) - '
(.. 3) -y WT: (. represents the mocisoon-making basis of the conrection peocess Y - WP, 6 indicates (are. fune, Egt, fgt.ffunc.rrr..c and W indicates Egt tok.
Generalizabilityy analysis. To explore the generalization of the error correction model, first, we analyze the overlap of code text between the test dataset and the training dataset. Following data preprocessing, we observe that there are duplicated codes between the two datasets. Addition- ally. comparing the wrong code fragments (i.c., code fragment containing the wrong token) in the
7

---

Published as a conference paper at ICLR 2024
test dataset with those in the training dataset, we still cannot find any identical wrong code frag- ments. It means that the MUN-ECD/ANN-EEE accidentally correcting wrong tokens due to the test dataset containing duplicate 'codes' or "wrong code fragments' with the training dataset. Second, we show the interpretable decision-making basis for wrong token fixing in Table 2 where (sre_fune, figt_fnnc-preee
can learn how to correct wrong tokens, rather than
tgt__ok is retrieved to fix the wrong token '/' - '/f", We lind that the text of the wrong translation is quite different from the text of (sre_func, the knowledge learned from the error correction dataset to new samples. In summary, the above error correction behavior demonstrates that the error correction model can extend the repair knowl- edge acquired from the training dataset to new wrong translations, rather than merely relying on straightforward comparisons of similar code texts.
It indicates that the error correction model can well apply
Error correction performance of &NN-ECD" and &NN-ECS".. correction performance of &NN-ECD and ENN-ECS, .n It is worth noting that &NN-ECD" and ANN-ECS, are generated from the same error correction dataset, which means that both contain the same error correction information. Surprisingly, in this case, kNN-ECS" m achieves 65.8% error correction rate, showing a substantial 22.3% improvement compared to ANN-ECD". Besides, as shown in Figure 4[a] with the increase in the number of sub-datastores under the error correction system, we observe an improvement in emror correction performance. The main reason is that tradi- tional KNN methods usually suffer from insufficient retrieval capabilities in a large datastore, which tends focus on high-density regions while failing to capture potential correlations in low-density regions. In contrast to KNN-ECD', the improvement of KNN-ECS", m is primarily attributed to its dis- tributed structure, which includes diverse datastore variants. By employing different data flows for multiple rounds of error correction, it can capture more comprehensive error correction information.
m- In Table 3 we compare the error
Table Correction performance of &NN-ECD' DAAEECDD/ANNNCCEEE /ENN-ECS on TransCoder-Sy wrong translations
ENN-ECS
ENN-ECS
ENN-ECS
ENN-ECS,
ANN-ECS,
ENS:
ENN ECD
[
25.4%
26.4%
27.3%
28.5%
30.7%
33,9%
43.5%
Sub_ECD
26.5%
27.2%
30.0%
33.0%
36.3%
sub FCD
26.5%
27.4%
28.08
28.7%
32.0%
35.9%
sub FECD
26.1%
27.9%
30.4%
29.3%
30.9%
Bub
25.0%
26.2%
29.5%
28.0%
31.4%
sub BCD
26.0%
26.6%
28.6%
29.5%
32.1%
ECD,
25.4%
29.1%
28. 7%
28.4%
sub_ECD,
24.1%
26.7%
26.8%
26.9%
sub BCD,
25.9%
27.0%
26.8%
30.2%
sub .DDEED LD
27.9%
27.6%
27.3%
JubLECD;, 11 sub ECD
26.9%
26.8%
28. 28.
25.7%
24.9%
27.85
sub ECD is sub_ECD, sub ECD is
27.7%
26.0%
27.1%
27.0%
26.0%
27.6%
sub ECD
26.2%
sub_ECD 17 sub BCD
26.0%
26.3%
sub ECD ECD
26.1%
27.0%
28.2%
28.8%
31.7%
35.4%
43.5%
ECD" /BCS
65.8%
64.9%
61.1%
58.4%
52.1%
43.5%
Impact of the unified name rule. In Figure 4. we show the error correction performance of KNN-ECD and KNN-ECS,, after applying the unified name rule. Comparing Table 3 and Table 4 we observe that implementing the unified name rule can effectively enhance the error correction rate, ranging from 1.7% ~ 6.5%. The main reason is that, during the training and testing phases, using the unified name rule can ignore the diversity of function names, parameter names, and variable names as much possible, while paying more attention to the logic and structure of the code. By adopt- ing the unified name nule, we can minimize the emphasis on rare identifiers, thereby eliminating the interference produced by diverse rare identifiers during the process of datastore construction and implementation. Furthermore, it is worth mentioning that employing the unified name rule alone can achieve 5.1% preliminary error corrections before feeding the wrong translation into ECD/ECS,.. where the errors mainly arise from identifier confusion within the code.
Ablation analysis. In Table 5 we investigate the importance of adjacent tokens in the process of identifying and correcting wrong tokens within the input. Our approach is to progressively remove the adjacent tokens near the target wrong token in the input. This systematic approach provides
8

---

Published as a conference paper at ICLR 2024
Table 4: Correction performance of ENN-ECD/KNN-EON on TransCoder-ST wrong translations
15S15
1CS
1NN-
ENN-ECSS
ENN-ECS,
RNN
sub_BCD:
29.0%
29.4%
30.4%
32.1%
35.5%
42.5%
48.7%
sub DCD
26.3%
27.1%
29.9%
32.8%
37.1%
37.8%
sub BCDs
26.0%
26.1%
27.3%
33.8%
33.1%
37.1%
sub BCD
25.3%
28.4%
27.9%
29.0%
33.7%
sub BCDs
25.3%
29.7%
30.9%
30.4%
33.2%
sub BCDs
32.0%
30.0%
sub_ECD,
28.3%
26.9%
31,0%
28.4%
sub BCDs
26.3%
29.9%
28.0%
30.9%
sub FCDs
25.0%
29.4%
29.8%
29.2%
sub ECD
25.7%
28.5%
28.1%6
sub BCD,1
26.7%
28.0%
28.1%
FCD12
25.8%
26.7%
sub BCD
29.1%
27.1%
sub BCD,4
27.8%
29.39
suh 5
27.3%
27.2%
25.2%
sub BCD1y
25.8%
suh FCDrs
sub_ECDavs
26.8%
27.9%
29.2%
30.7%
34.2%
39.25%
48.7%
Unified name nale
5.1%
5.1%
5.1%
5.1%
5.1%
5.1%
5.1%
ECD/ECS..,
67.5%
66.7%
65.7%
64.8%
61.2%
56.5%
50.0%
deeper understanding of how adjacent tokens contribute to the identification and correction of wrong tokens, depending whether the wrong token is successfully corrected in the output. In Table [ the wrong token '/' the wrong translation should be corrected to '//" Initially, when we feed the original input to the error correction model, it can effectively correct the wrong token '/. However, as we systematically remove the adjacent tokens one by one, we encounter challenges in rectifying the wrong token. means that adjacent tokens are key identifying and correcting wrong tokens. The identification and correction of wrong tokens relies on the internal relationships among adjacent tokens, rather than directly pinpointing the wrong tokens. In essence, the error correction method corrects wrong translations by learning and understanding the internal relationships between tokens within a function.
Table 5: The impact of adjacent tokens in the error correction process
Input (original format) def
Input (remove lea ())
Input (remove b) def Func (aC):
Input (remove return)
Funu: (a0):
def (aC)
def Func (a0)
Input
in w0:
in
return b (a0)
returr b / a0
return / Jentadi
b / len (a0)
def (at):
def func (ac):
def func (ac):
del Eunc (a0)
b
Ouipul
80;
a0s b return / Jenla0)
b C
len (20) b
return à // (on(a0)
reture / 60
CONCLUSION
In real-world scenarios, the interpretability of outputs plays a crucial role in gaining users' trust. Currently, Transformer-based models are widely used for program translation. However, even with researchers investing significant time and computational resources in retraining models, the im- provement in translation accuracy remains relatively limited. Furthermore, due to the complex internal workflow of the Transformer model, it is difficult to track and identify which snippet in the training dataset contribute to each output token. In this paper, we employ the &NN retrieval on an error correction datastore to enhance the translation capability of TransCoder-ST through code correction. This approach provides a decision-making basis for each generated token, laying a solid research foundation for the subsequent improvement error correction. Importantly, by simply integrating additional error correction datastore, the datastore- based &NN retrieval approach significantly enhances the translation performance of TransCoder-S without the need to consume significant computational resources to retrain the Transformer-based model.

---

Published as a conference paper at ICLR 2024
REFERENCES
Hindle A. 2015 Aggarwal K, Salameh M. Using machine translation for converting python 2 to python code. PeerJ PrePrints 3:e/459v1, 2015. URL
7287/
1etprinnns.119v1 1459v1
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai- Wei Chang. Unified pre-training for program understanding and generation. CoRR, abs/2103.06333, 2021. URL https://
arxiv. wtt.:/rrgabbb.......... 06333
Umair Z. Ahmed, Pawan Kumar, Amey Karkare, Purushottam Kar, and Sumit Gulwani. Compi- lation error repair: For the student programs, from the student programs. In 2018 IEEE/ACM 40th International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET), pp. 78-87,20 2018.
Xinyun Chen, Chang Liu, and Dawn Song. Tree-to-tree neural networks for program transla- tion. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL httt::::/prooceedggs..eeeee http:////.eppeeeea....ooaaeaaaaa 775375cesabb.0022263335551333333
.df
Nadezhda Chirkova and Sergey Troshin. Empirical study of transformers for source code. CoRR, abs/2010.07987, 2020a. 2020a.
wtt.a/awwwww.g.a///aaaaaaaaaa/// 07987
Nadezhda Chirkova and Sergey Troshin. A simple approach for handling out-of identi- fiers in deep learning for source code. CoRR, abs/2000..06.... 2020b. URL /tRppttss://axxiv
org/ abs/
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. pp. 8493-8502, 01 2022. doi: 10.18653/1/00222.al long.581..
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep
bidirectional transformers for language understanding. CoRR, abs/1 110.0880,, 2018. URL.
abs wtt./1110.0 04805
Elizabeth Dinella, Gabriel Ryan, Todd Mytkowiez, and Shuvendu Lahiri. TOGA. Proceedings of the 44th International Conference on Softwrre Engineering. ACM, may 2022. doi: 10.1145/ 5510003.3510144
htt.tttss//d....o/11111111111 011.2222335500033311111111 3510141
Zhangyin Feng. Daya Guo, Tang. Nan Duan, Xiaocheng Feng. Ming Gong. Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. Codebert: pre-trained model for programming and natural languages. CoRR, 0bs/2002.08155. 2020. URL http://xxxi..orrra///0/000
08155
Javier Ferrando, Gerard I. Gállego, Belen Alastruey, Carlos Escolano, and Marta R. Costa-jussà. Towards opening the black box of neural machine translation: Source and target interpretations of the transformer, 2022.
Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish K. Shevade. Deepfix: Fixing common lan- guage errors by deep learning. In Satinder Singh and Shaul Markovitch (eds.), Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, Cal- ifornia, USA, pp. 1345 1351. AAAI Press, 2017. URL http://aaai: http://aaaaa.roocss/ooo.ooo wwt.oo////oo/ooooooooooooooooooo /indxx.
php/ AAAI/ AAA.AAAAAAAAAAAAA.AAAAAAAAAATTTT /ttp//AAAAAALL1////////////////
Tapas Kanungo, David M. Mount, Nathan S. Netanyahu, Christine D. Piatko, Ruth Silverman, and Angela Y. Wu. An efficient k-means clustering algorithm: Analysis and implementation. JEEE Trans. Pattern Anal. Mach. Intell., 24(7):881 892, 2002. doi: 10.1 2..1109//AMM22200.10000
URI. htw.:htts:///o httt:s::/dooo.oo/1111 000. 1109 / AMTM
Rachel-Michael Karampatsis, Hlib Babii, Romain Robbes, Charles Sutton, and Andrea Janes. Big models for source code. CoRR, abs/2003.07914 2020. URL htttps://aaatt http:///eexxi....oraa///////////
code 1= big vocabulary:
07914
10

---

Published as a conference paper at ICLR 2024
Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer. and Mike Lewis. Nearest neigh- bor machine translation. CoRR, abs/2010.00710, 2020. URL https///aaxvvvoog///o//
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. CoRR, abs/1901.07291, 2019. |ttpp//////rxxi www.oogggg/.o/oooeee/.oooeeeeeae 1901. 07291
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to-seqqeenc pre- training for natural language generation, translation, and comprehension. CoRR, abs/1910.13461, 2019. htRpbttp://aatt
org/ abs 1anbs//9911 13461
Yi Li, Shaohua Wang, and Tien N Nguyen. Difix: Context-based code transformation learning for automated program repair. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pp. 602 614, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
Kevin Loney and Lisa McClain. Oracle Database 10g The Complete Reference. McGraw-Hill, Inc., USA, edition, 2004. ISBN 0072253517.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt, 2023.
Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. Under- standing the effectiveness of large language models in code translation, 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics ACL '02. Association for Computational Linguistics, 2001. doi: 0..111//1773083.0733333. URL http://pp../.o...r//10.....///1/
Ye Qi, Devendra Singh Sachan, Matthieu Felix, Sarguna Janani Padmanabhan, and Graham Neubig. When and why are pre-trained word embeddings useful for neural machine translation? CoRR, abs/1 00418844006,,,,. 2018. URL http:
abs 1804. 0118800663233 06323
Alec Radford, Karthik Narasimhan, Tim Salimans, and llya Sutskever. Improving language under- standing by generative pre-training. 2018
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.1065. 2019. http::/t:///attt www.wrwwwwooro...oooo/oerooo 910. 10683.
Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. Unsupervised translation of programming languages. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 20601- 20611. Curran Associates, Inc., 2020.
Baptiste Rozière, Marie-Anne Lachaux, Mare Szafraniec, and Guillaume Lample. DOBF: A deob- fuscation pre-training objective for programming languages. CoRR, abs/2102.07492, 2021. URL
/ 07492
https://aaxiie / wttp:r///p........../..
Baptiste Roziere, Jie M. Zhang, Francois Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. Leveraging automated unit tests for unsupervised code translation. urXiv:2110.0677 [cs], Feb 2022. URL
06773 arXiv: 2110.06773.
Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida Wang. Natural language code translation with execution, 2022.

---

Published as a conference paper at ICLR 2024
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. Intellicode compose: Code generation using transformer. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium the Foundations of Software Engineering, pp. 1433-1443, 2020.
Marc Szafraniec. Baptiste Roziere, Hugh Leather, Francois Charton, Patrick Labatut, and Gabriel Synnaeve. Code translation with compiler representations. 2023.
Zhaopeng Tu, Yang Liu, Shuming Shi, and Tong Zhang. Learning to remember translation history with continuous cache. CoRR, /0//1711.09361 2017. URL
1711.09367
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762,, 2017.
http::////axxix.
abs/1706..000 03762 03762
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, 2023.
Jesse Vig, Sehastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 22388-22401 Curran Associates, Inc., 2020. URL
neurissss.nn.nnnospper........ ctpppprr fttleeppeeeppeeee//pepppppaeeeee Paper.
Dexin Wang, Kai Fan, Boxing Chen, and Deyi Xiong. Efficient cluster-based k-nearest-neighbor machine translation, 2022.
Donggi Wang, Haoran Wei, Zhirui Zhang, Shujian Huang, Jun Xie, Weihua Luo, and Jiajun Chen. Non-parametric online learning from human feedback for neural machine translation. CoRR, abs/2109.11136 2021a. BRRppsts:://axx http://...rg/bb////////// 11136
Shuhe Wang, Jiwei Li, Yuxian Meng, Rongbin Ouyang, Guoyin Wang, Xiaoya Li, Tianwei Zhang, and Shi Zong. Faster nearest neighbor machine translation. CoRR, abs/21 12.08152, 2021b. URI.
https: /ttp://aavvvvv/.. org/ abs 2212. 08152
Shengbin Xu, Yuan Yao, Feng Xu, Tianxiao Gu, Hanghang Tong, and Jian Lu. Commit message generation for source code changes. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pp. 3975-3981. International Joint Conferences Artificial Intelligence Organization, 7 2019. doi: 10. 10.24963fijcai221195552
URL https:
7/doi. org/00 24w11//..223/........
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. CORR,
abs/1906.082377, 2019. URL BRLthttt:///axx http:://wwxwvvo../......
absr/ 08237
Jingyi Zhang. Masao Utiyama, Eiichiro Sumita, Graham Neubig, and Satoshi Nakamura. Guiding neural machine translation with retrieved translation pieces. CoRR, abs/1804.02559, 2018. URL
/ org/ abs 804 0t////8004000559999
Jiyang Zhang, Pengyu Nie, Junyi Jessy Li, and Milos Gligoric. Multilingual code co-evolution using large language models, 2023.
Xin Zheng, Zhirui Zhang, Junliang Guo, Shujian Huang, Boxing Chen, Weihua Luo, and Jiajun Chen. Adaptive nearest neighbor machine translation. CoRR, 10/22105.1302.. 2021. URL
//arxiv// http:////awvv.oo///////////
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 EVALUATION
Most program translation studies employ BLEU scores assess the quality of the results (Papineni et a 2001 Chen et al., 2018 Aggarwal KJ 2013). However, wrong translation typically involves only few wrong tokens, which means that the BLEU score tends to remain relatively high, regard- less of the correctness of the translation. Inspired by computational accuracy in TransCoder (Roziere) et al. 2020). we introduce functional equivalence to measure the quality of code correction.
Functional equivalence: Given the same input, the source code and the corrected code produce the same output, i.., both the source code and the corrected code succeed under the same unit test cases.
This metric emphasizes the correctness and functionality of the code, rendering it better suited for evaluating the quality of corrected translations. For example, some wrong translations occur not due to bugs in the code, but because the source code and the translated code fail to produce the same output when given the same input.
A.2 GENERATION OF PYTHON UNIT TEST DATASET
Why do we need to use translated Python unit tests? Firstly, in the process of generating the error correction training dataset, we translate a Java function to multiple Python functions (beam_size > 1), and then combine the first failed Python function with the first successful Python function to form an error correction language pair. Therefore, it is crucial to ensure that the Java function and its corresponding multiple Python functions have equivalent unit test dataset, as well as multiple Python functions under the same Java function have the same unit test dataset. Secondly, during the testing phase, we need to verify that the wrong Python function, the corrected Python function, and its source Java function have equivalent unit test dataset, as well as the wrong Python function and the corrected Python function have the same unit test dataset. Based on the above two cases, directly translating Java unit tests into Python unit tests is a simple and effective approach.
How to ensure the feasibility of the translated unit test dataset? For the translation process from Java unit test dataset to Python unit test dataset and the screening process of source Java functions, we refer to the work of TransCoder-ST (Roziere et al., 2020). In Table 6 we show the process of using the error correction model to rectify the wrong Python function generated by TransCoder- ST, where Python unit tests are the translations of Java unit tests. In our training dataset and test dataset, each wrong Python function has matching correct function (ground truth) that passes the corresponding unit test dataset, and almost every unit test dataset contains 3~6 unit tests. This ensures that the training dataset, the wrong function and the correct function under the same error correction language pair use the same unit test dataset. It also guarantees that during the testing phase, if the wrong function can be corrected, the corrected function will successfully pass all unit tests. More importantly, multiple unit tests under each unit test dataset assure the reliability of the final unit test result.
Table 6: Translation process from Java unit tests to Python unit tests
Type
Details
Source Iwna function
public atatie double Idouble value, double min, double
10 (value mini
retuzn min;
10 (value retuen max;
uun values
Continued on next page
test oet.epaaaeeeiiiiiieeeeee.ererrrn
13

---

Published as a conference paper at ICLR 2024
Type
Details
Woong Python funcrios
doc
zunc (a0, al, 52) 18
12 30 52:
retutn max
Corrected Python flanction
daf Zuna (aC, 92)
setuzn at
1r a0 a2:
cotuan a0
Java test Untaset
was socionatiillll genezaled by EvoSuite
This Jan 26 21:55:92 GMT 2023
import org.junit. Test;
import statls org. jtn.j..esert.........
import rrg.evosultee te. runt ine Evollrrner
import erg.evoruite. reniitttrnutii runtine ErokornerPerametesss import arg- org-junit runner.RunMithy
ERunkita (EvoRannee class) #evouunrarFarameterr inoouutttenneterri.... true, useVES
true, uSeVNET 21A85_64<$55037755
"azetStat inState separatec waseeeelaasseseeeeeeee truel
@Test it ineout 40001
public vold
Throwable
deable9
(-92868,4857075) 47848-991;
double0, 1.0E-1);
#Test (tinesui 80001
public vold test throve Throwable
doubleo (ts..e.210003111..mmm..eeee. 2000rtEqa.. doublet, 1.0E-1);
1-02604.81, (-1.01)7
public
souble deable0 (ttp:003....ccmmm333..11........
94690, 010166346, (-1.0)1;
doubled, 1.02-4])
eTest (Cinesut 6000)
publis vold
Throwable souble deablev can.i 0.0, (-1.01);
a0cartEquaia (1-1.9)) doublet, 1.0E-4))
erost (cinoout 4080)
public void test throvs Throwable
double doubled 2rr._
0.0, 0.033
102artEqua110.00 doublet, 1.0E-())
(cinesuu 40001
public void LeatS1) throws Thronable
CLA88_6ke250032711 L288_660200033700 CLASs_6ker300377C(0
Continued on next page
14

---

Published as a conference paper at ICLR 2024
Type
Details
Python lest dataset
import ntppy
import seport
import collections
from collections import
import itartsols
import random
import syd
import unittast
#2OFILL
class hat.5....eee00221//.ttttttttoto.
test0 (selCi:
doublcd [_C_1164(((12275.77777 (171111122777777774455555))) 1-72969.48570735 (2848.99)
agcert abo1(-125.
double0) <- 1.05-6
def testl
doublee L_E-1lee1116640 (111866000000066666 (-12506.0), (-1.03)
agsort absl(-1.0) doublet) <a 1.08-4
def Lest2 (selCi:
doubied [L (110--222DD.1111111010003 66600.110665355, (-1.0)1
ascert absid8590.
coubicU) 1.96-6
def (ast3(
Soublcd E_E.11e610000 C.O, 1-1.031
ansort aball-1.0) doublec) <= 1.08-4
def L
doubled
0.0,
agSsEt absi0.0 - doublet) 1.02-4
maiin's woittasc.mainjf
A.3 ITERATIVE CODE CORRECTION
To verify the error correction capability of ANN-ECD, we conduct multi-round experiments &NN-ECD, trying to perform iterative code correction on the same datastore. We carry out ex- periments from the following two aspects. On one hand, iteratively input the wrong source translations into ANN-ECD and compare the overlap of the outputs. The results reveal significant overlap in corrected translations across multiple trials, with only minor differences in error correc- tion rates ranging from 0.012% to 0.047%. On the other hand, we also attempted to repeatedly feed the wrong output of KNN-ECD back into &NN-ECD for multiple rounds. The experimental results indicate that only 2.5% of wrong functions are re-corrected in the first round. In subsequent rounds, no more than 0.4% of wrong functions are re-corrected each time. The above experiments illustrate that kNN-ECD usually corrects all errors in wrong translation at once, with little additional gain from iterative error correction.
A.4 STABILITY ANALYSIS.
We analyze the stability of the error correction model terms of both construction and implementa- tion. In the construction phase, we randomly and evenly divide the error correction training dataset into m sub-datasets, and then generate the corresponding sub [C. In Figure 3 we sepa- rately test the independent error correction performance of sub B.D,ECD [I. under the ECS,., where (sub_ECDD aabb_HHD2....... sub_ sab__EDDmm) show close error correction rates within the same system. Meanwhile, comparing sub_ Lrb_EEDDaor avg of different &NN-ECS, m in Figure 5] we find that the larger the sub-datastore, the higher the error correction performance. The above observations indicate that the datastore can stably learn correction information from the error correction dataset during the con- struction process. Moving to the implementation phase, we repeatedly feed the same test dataset into &NN-ECD, and then compare the overlap the outputs. The results demonstrate significant overlap in the corrected translations across multiple trials, with only slight differences in error cor- rection rates ranging from 0.012% to 0.047%. This implies that the error correction model exhibits strong stability during the implementation process, enabling users to trust the output of the model.
15

---

Published as a conference paper at ICLR 2024
40%
40%
KNN-ECS 18
KNN-ECS' 15
Error Correction Accuracy
35%
KNN EC518
Frror Correction Accuracy
35%
KNN-EC515
30%
30%
25%
20%
15%
1.5%
10%
10%
516
0%
14
15 16
4
7
9
10
12
14
15
sub_ _CDD
Sub_ECDI
40%
KNN-ECS KNN-ECS12
KNN- 9
35%
35%
Error Correction Accuracy
Error Correction Accuracy
30%
25%
20%
20%
15%
10%
5%
5%
0%
0%
1
2
4
5
7
8
10
11
12
4
6
sub
ECDI
sub_ECDI
50%
KNN-ECS*6
45%
KNN-ECS6
Error Correction Accuracy
35%
30%
25%
15%
10%
5%
0%
6
sub_
Figure 3: Independent error correction performance of MIN-sub_ECD. (e) under ANN- ECS,- In the same emror correction system, where sub ECDs have similar memory storage, we compare their independent error correction performance respectively. We find that sub ECDs with approximate information storage capacity exhibit close error correction capabilities. This indicates that datastore can stably learn and apply error correction information from error correction language pairs.
16

---

Published as a conference paper at ICLR 2024
Table 7: The decision-making basis for each generated token in the corrected translation.
Type
Details
function
public static void im but for (int i-
length (i bur [urli]-1
Wrong Python function
det fune ( ): NEW LINE INDENT for in all NEW LINE INDENT b 7 NEW LINE DEDINT DEDENT
Corrected function
func NEW LINE INDENT
raage (len )) NEW LINE INDENT a J- NEW LINE DEDENT
Deviiion-makingg bosis
DEDENT
Vee fune (10) al ): NEW JINE INDENT for (h (h.b) enumerate ( (10) NEW,LINE INDENT 10 bJ -a NEWJ.INE
DEDENT DEDENT,
"dee"
{ Wel fine (a0) al NFWJ INF INDENT for b ringe (al) ): NEWJ.INF INDENN yield aD NEW DEDENT
DEDENT' "def") - "fune"
"deef [ime [a0. _a NEW,LINE INDENT for b range (al) NEWLLING INDENT ziehd ab NEW LINE DEDENT
DEDENT 'def fune")) ->
{ def fimc (a0 ): NEW.LINE INDENT b NEW LINE for c in aO: NEW LINE INDENT = + NEWLIIE DEDENT
return b NEW
DEDENT "def (") - "10"
filef (+00) NFW J. INE INDENT sum O N.W LINE for h in a0: NEW J. INE INDENT sum 4= I NFW.LIWWW LIIEE
DEDENT
summ NEW
INDENT
NEW.LINE
d: NEW
INDENT return
NEW J.IN DEDENT + NEW LINE DEDENT return I LEWLLINE DEDENT" "def fume (s0') '
{ "dief fune: [x0) ): NFWJ INE INDENT b O NEW J INE for e #0: NEW J. INE INDENT is 4m NFW NFW,LINE DEDENT
sum NEW LINE DEDENT" "def func a0 13 ":3
y: LINE INDENT if c. isalmum O:: NEW LLIEE INDENT retum I LLINE DEDENT DEDENT retum len a0) )- NEW LINE DEDENT', "def func (al) )
[une [=0]: NEW LINE INDENT (b.c e)
(1) - 'NEW LLINEE
"deff fune (x0. .al NEWLLINE INDENT for b a0: JEWJ INDENT b al NEWLIINE DEDENT DEDENT' 'def
func
NEW LIIEE) - 'INDENT"
[ deef fume (a0) aal ): NEW.LINE INDENT for b a0: NEWLINE INDENT + al NEWLINE DEDENT DEDENT" 'def
(i00 al 3: NEW LINE INDENNT) - "for"
NEW LINE INDENT for a0: NEW LINE INDENT al NEWLIINE DEDENT DEDENT' 'ee (ame ): NEW LINE INDENT for') ) "b'
"dee Eunc (a0.
[ "def fime (a0) all 3: NEW INE INDENT for a0: NEWJ..NE INDENT b al NEWJIINE DEDENT DEDENT" 'def
func a0_ al 3: NEW LLINE INDENT for + 'in'
"def [une (a00 aal NEW LIIN INDENT for b a0: LINE INDENT b al NEWLIINE DEDENT DEDENT' 'def
func
1: NEW LLINE INDENT for in') - "range"
{ deef fume (a0 aal ): NEW.LINE INDENT for b in a0: NEW LINE INDENT += al NEWLIINE DEDENT DEDENT', 'def
fine 60 all NEW LINE INDENT For b ronge' ) - '("
'def Eune
NEW.LING INDENT
NEW LINE for
LINE INDENT
NEW .LINE
DEDENT = sum NEW LINE
INDENT NEW.IINE if d: NEW LINE INDENT return
NEW LINE DEDENT +H b NEW.LING LINE DEDENT return- NEW LINE DEDENT" ddef [ume (x0) ): LINE INDENT
sum 0 NEW LINE for in raanee (') - 'Jen'
"def fune a0 NEW LINE INDENT
NEW LINE b in ab: LINE INDENT + b NEW LINE
DEDENT
sua NEW LINE for in NEW LINE INDENT NEW.LINE if d: NEW LINE INDENT return
NEW LINE DEDENT tHe b NEW LINE DEDENT return 1 NEWLLINE DEDENT', ddef fune (a0) ): NEWLLINE INDENT
sum O NEW LINE for in ramge Ien')
fune
NEW.LINE INDENT for aD: LINE INDENT b NEWLIINE DEDENT DEDENT', 'def LINE
fune
runge leen { ddef fume (a0) 3: NEW LINE INDENT sum = NEW LINE for in a0: NEW INDENT sum += NEW LINE
DEDENT' o di summ NEW LLINE For b in ao: LINE INDENT d -nD NEW.LINE if -- d: NEW INDENT I return NEW LINE DEDENT return NEW LINE DEDENT", ddef fune (10 NEW LINE INDENT
NEW LINE DEDENT
sum D NEW LINE for in image
"1"
[ ddef Func at) ): NEW.LINE INDENT sum NEW .LINE For in al: NEW LINE INDENT sum +- b NEW LLINE
DEDENT o =d sum NEW LINE for b in a0: NEW LINE INDENT d = NEW LINE of =E d: NEW LINE INDENT return NEW
NEW
DEDENT for raage
DEDENT
NEW LINE DEDENT", 'def
NEW LIIE INDENT
sum O NEW
Y)
[ ddef func aO al ): NEW LINE INDENT For b in aQ: NEW LINE INDENT += al LINE DEDENT DEDENT', 'def
Fune 80. al 3: NEW LINE INDENT FOR b range IIo G0 Y) '.7
Eunc
NEW LINE INDENT For
LINE INDENT al NEW LINE DEDENT DEDENT', 'def
hunc d0. 3: NEW LINE INDENT
range Ien a0) 1) y)-) "NEW
[ ddef Fune ad) al ): NEW LINE INDENT for b in DO: NEW LINE INDENTE 6- A1 NEWLLINE DEDENT DEDENTT, de
func a0. al 3: NEW LINE INDENT FOr b runge Ien G0 ): NEW LINE') ) 'INDENT'
fune
NEW.LINE INDENT for
LINE INDENT'
NEW LINE DEDENT DEDENT'
fune a0. 1: NEW LINE INDENT FOr range Ien aO D: NEW LINE INDENT") - '2D'
[ def func a0) aa ): NEW LINE INDENT for b in JQ: NEW LINE INDENT'S b tH al NEW LINE DEDENT DEDENT', de
fune 60. a1 NEW HEW.LINE LINE INDENT FOR b range Ion G0 ): NEW LINE INDENT all) 'I' LINE INDENT for
NEW fune (0 al ): NEW LINE INDENT for rango len (0) 3 NEW LINE INDENT all (')
tune aall..
LINE INDENTTS
DEDENE DEDENT',
{ 'def fune ad) al y: NEW. INDENT for b in 10: NEW LINE INDENT -- al NEW LINE DEDENT DEDENT', 'def
fune a0. al 1: NEW LINE INDENT LOC b runge Ien G0 ): NEW LINE INDENT all (T) -
fune [x0): NEW LINE INDENT [a0] J- NEW
DEDENT" deef
NEW LINE INDENT #0 ((O])
[ def Func at) 3: NEW LINE INDENT b af And BY " NEW LINE = NEW LINE return | e: b NEE LINE
DEDENT (19) "def func Call) J: NEW L TIEE INDENT aD find (" L'L Y NEW LINE c- 3 '7" NEW LINE INDENT
tune all
If TO None die NEW
DEDENT", 'def fune (all):
NEW
INDENT retum all
else 7') 'NEW LINE"
{ "dief fune (00) al 3: NEW LINE INDENT for b in 20: NEW LINE INDENT al NEW LINE DEDENT DEDENT" 'def
Fune a0, o1 NEW LLINE INDENT For h range ( (ion ( 60 )): NEW NEW_LINE INDENT all [1 4- al NEW LINEE)
"DEDENT"
{ ddeef fune (a0), .ll y: Fune a0. o1 LEWW LINE INDENT for b in range ( lion 60 1): NEW_LINE INDENT all b 4- LIWE DEDENT') - 'DEDENT'
INDENT for i IF: NEW
INDENT al NEW LINE DEDENT DEDENT"
JEIE DEDENT DEDENT' fane a0. al NEW ,LINE INDENT for range (len, a0 1): NEW LINE INDENT all 4- NEW LINE DEDENT
'ddef flnne [10) al JEWJ. INDENT for aO: JIWN INDENT
DEDENT'S - 'FOOS"
(...) ia mer (..)) represents the decision basis for each generated token 1> in the corrected Python function. NEWLLINE, INDENT, DEDENT represent newline, indentation, dedent in code formatting, respectively. EOS: and senience.
17

---

Published as a conference paper at ICLR 2024
92%
ECD*/ECS"m ECD/ECSm
TC-ST ECD /
TC ST + ECD/ECSm
Error Correction Accuracy
Translation Accuracy
60%
88%
55%
86%
84%
45%
82%
40%
80%
1
18
18
m
(a) Correction performance
(b) Translation performance
Figure 4: The influence of diverse datastore distributed structures. We find that the distributed architecture can effectively enhance the eтror correction performance by efficiently learning and applying massive error correction information, thereby improving translation accuracy. Meanwhile, we also explore the impact of the unified name rule on the eтror correction model. This approach exhibits a positive gain effect, effectively improving the model's error correction capabilities.
50%
sub ECD*avg
sub Lub_ECDavg
Error Correction Accuracy
45%
40%
35%
30%
25%
1
3
6
9 m
12
15
18
Figure 5: The influence of datastore capacity on translation accuracy. We observe positive correlation between the sub-datastore capacity and error correction performance. The larger the sub-datastore, the higher the error correction capability of the sub-datastore. This finding suggests that the datastore can robustly acquire and integrate correction information from the correction language pairs.
18


Reviews:
[{"review_id": "l3nGOeaQDS", "reviewer": "Reviewer_Uzpj", "summary": "The paper proposes an error correction method, KNN-ECD, which is based on KNN-MT and improves the performance of code translation.  Building upon this, the paper further propose $kNN-ECS_{m}$, which divides the data store to $m$ sub-datastores. In addition, the paper proposes a new unified name rule to encourage the datastore to focus more on code logic and structure rather than diverse rare identifiers. The experiments show the the proposed methods largely improve the translation accuracy.", "questions": "1. For unified name rule, how to identify the variable name or function name in a program and replace them? Is it replaced by some automatic tools?\n\n2. How do the method judge the wrong translations of TransCoder-ST? Is the error correction only applied the wrong programs?\n\n3. What does the method have better interpretability? The key value pairs in the datastore is still based on neural networks.\n\n4. Is there any fine-tuning stage of the TransCoder model?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper applies the $kNN-MT$ to the code translation and obtain a significant improvement of the translation accuracy. Using functional equivalence as evaluation metrics instead of BLEU better reflects the true  code translation quality. And the proposed method increase the accuracy by about 20%.\n\n2. The paper proposes a novel $kNN-ECS_{m}$ framework, which further improves the translation accuracy of program.\n\n3. The paper performance extensive empirical analysis of the proposed method.", "weaknesses": "1. $kNN-ECD$ is very similar to $kNN-MT$. Therefore, the technical contribution of the paper is limited.\n\n2. The motivation of applying $kNN-MT$ is not very clear. Although $kNN-MT$  is useful for natural language translation, is there some particular reasons that it will be more effective for programming languages.\n\n3. The presentation is experiment results is hard to read, especially for Table 3 and Table 4. I would suggest the authors to use Figures to present this results and put the detailed numbers in the Appendix.\n\n4. The paper does not show the proposed method can perform error correction for OOD errors. The paper uses model $A$ to build the pair of incorrect programs and correct programs. Therefore, the error is specifically related to model $A$ itself. For a new model $B$, it may make different kinds of error, does the proposed method with learning datastore for model $A$ can fix the error of model $B$. If not, the method requires building datastore for every new method, which largely limiting the application of the proposed method.\n\n\nMinor:\n\n\"Uncorrect\" should be changed into \"Incorrect\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CKvDpjTU8U", "reviewer": "Reviewer_oueh", "summary": "To address a need for code-to-code translation accuracy improvements, the authors propose to combine a transformer model, specifically TransCoder-ST, with an error correction model that optionally overwrites the output of the transformer model. They do so in two ways. Initially, they consider the error correction model to be a single error-correction datastore in which they perform kNN (kNN-ECD). Later, they improve on the initial model by dividing the dataset into m-sub-datasets and they construct a distributed datastore (kNN-ECD$_m$). To make the dataset more uniform, the authors also employ a \"unified name rule\", to perform $\\alpha$-renaming while keeping certain type information. They show that this full-pipeline can improve TransCoder-ST performance on Java to Python translation from 68.9% to 89.9%.", "questions": "In the multi-round error correction(kNN-ECD$_m$), does each subsequent ECD get the mis-corrected version from the previous module or does each ECD get the original wrong translation?\nIf the former, does kNN-ECD$_m$ perform partial corrections that build on top of each other? \n\nOn the dataset construction, have PyTest translations been manually or otherwise been validated to be correct/faithful to the EvoSuite output?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Simple framework both in the single and multi-round error correction. \n- Shows generalisation of fixing patterns (and the authors check for data leakage).\n- Extensive ablation to understand which pipeline components contribute to the overall performance (single- vs multi-round error correction, sub-datastore performance, unified renaming)", "weaknesses": "- Interpretability feels like an after-thought, it is left to the reader to infer it from the use of kNN-MT derived methods. Indeed, the discussion in S4.2 focuses more on the ability of the model to generalise from the examples (which is interesting and significant), but the showing the mapping to a software engineer would do little to explain the model decision and gain trust in the model. To build on the S4.2 example, a user explanation would be that the final values should be \"int\", and it is difficult to derive this insight from the mapping.\n\n- On dataset construction, EvoSuite will overfit to the current implementation, which means there is an underlying assumption that the Java version is bug free. Further, translation of the PyTests from Java Unit tests can also be erroneous.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CyxZE2UbHF", "reviewer": "Reviewer_7Q1Z", "summary": "This paper focuses on improving Java $\\rightarrow$ Python translation using error correction, rather than retraining the underlying translation model. They devise two error correction techniques (kNN-ECD and kNN-ECS) based on kNN-MT, which entails retrieving from datastores. To build this datastore, they first collect 82,665 Java functions and generate high-quality unit tests for them using EvoSuite. Then, they use TransCoder-ST to translate the Java functions paired with the unit tests to Python. From these they extract pairs of the form (failed Python function, successful Python function), which are then used to build (a) datastore(s). The datastore is organized based on two components: (1) (key, value) pairs and (2) (key, value) $\\rightarrow$ token. The query to this datastore is formed by using the last decoder hidden states corresponding to the full source input (i.e., failed Python function) and partial target (i.e., possible correction generated so far). To reduce noise caused by diverse rare identifiers during retrieval, they apply the unified name rule. In kNN-ECD, only one round of correction is performed. In kNN-ECS_{m}, they perform $m$ rounds of correction, with m smaller datasets (after segmenting the large datastore into $m$ parts). Results show that kNN-ECS outperforms kNN-ECD as well as a vanilla TransCoder-ST with no error correction.", "questions": "1) Based on what is written in the paper, 10 Python functions with unit test cases are generated for each Java function. So, you have $(func_1, tests_1), (func_2, tests_2), (func_3, tests_3)... (func_{10}, tests_{10})$. Success is measured by executing the tests in some Python environment, where $func_i$ is considered a success if it passes all tests in $tests_i$. By this definition, suppose $func_1$ fails $tests_1$ and $func_2$ passes $tests_2$.  The paper states \"we combined the first failed Python function with the first successful Python function to form an error correction language pair.\" Based on this, it seems that $(func_1, func_2)$ would be considered an error correction pair. However, there is no guarantee that $tests_1 = tests_2$, meaning that the two functions could be executed against different test suites. Therefore, $func_2$ may not actually correspond to a correction of $func_1$. Could this please be clarified? \n2) The \"interpretable decision-making\" idea is not clear to me. It seems that you are suggesting that the reasoning for predicting a specific token at a timestep $t$ can be attributed to the source and partial target function predicted so far. This is also the case for transformer-based decoders, so it is not clear to me how your approach can be considered more interpretable than a transformer as they claim.\n3) In 3.2, you state that the hidden representations from the last layer of the decoder are used to build the (key,value) and query. My understanding is that the (key ,value) and query correspond to (failed Python function, partial Python function). It is not clear to me how there would be a decoder state corresponding to the failed Python function since that is passed into the encoder (Figure 1). Or is (failed Python function, partial Python function) meant to actually only represent the representation of the partial Python function generated so far, as labeled as \"Key\" in Figure 1? \n4) You claim that the improvement of ECS over ECD is \"primarily attributed to its distributed structure, which includes diverse datastore variants.\" However, you do not seem to have multi-round experiments with ECD in which you repeatedly perform retrieval/correction on the same large datastore up to $m$ times. Therefore, isn't it possible that the advantage is actually from doing iterative code correction rather than the distributed nature of it?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed approach successfully corrects errors to a certain extent, without retraining the model or re-sampling the model many times, which is usually done in self-repair.\n- The idea of multi-round error correction and the analysis done with this, varying the number of rounds, and analyzing the performance for each of these, is quite interesting and may inspire future work.", "weaknesses": "- Evaluation is based on translated unit tests generated by the same model that the authors are trying to correct translation errors for. Therefore, the unit tests that are generated could be wrong, and so the evaluation is unreliable. Evaluation should be performed based on a separate, high-quality set of unit tests. Possibly datasets like HumanEval-X would be better alternatives here.\n- The experiments and results are fairly limited. First, the authors focus on only Java $\\rightarrow$ Python and fail to consider other languages or even the reverse direction of Python $\\rightarrow$ Java. Next, Table 1 seems to be missing many baselines and other models to which their approach should be compared. Namely, the only baseline is the pure TransCoder-ST model, which is only the starting point of their approach. The authors discuss that the main advantage of their approach is that no retraining is required, so it would be important to see how their approach performs relative to a retraining-based one. For this, they could have simply fine-tuned TransCoder-ST on the error correction pairs they collected for building their datastore. Next, latency is not measured, even though the authors discuss latency in related work. It seems that retrieving from a large datastore or retrieving multiple times from smaller datastores could take a long time, so it would be important to understand how the overall latency compares to other approaches. Finally, the authors do not report results on state-of-the-art code models, so it is difficult to assess the true value of their approach.\n- The authors present the unified name rule as a novelty; however, I do not find this to be that novel, given the work the authors discussed in the \"Related Work\" section.\n- There are multiple aspects of the paper that are not clear.  Please see the \"Questions\" section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)