SUT: Active Defects Probing for Transcompiler Models
Authors: Mengnan Qi
Yufan Huang
Maoquan Wang
Yongqiang Yao
Zihan Liu
Bin Gu
Colin Clement
Neel Sundaresan
Venue: EMNLP 2023 Main (2023)
Keywords: Program translation
LLM Evaluation
Unit Test
Syntax Error Analysis
TL;DR: We propose a challenging code translation evaluation dataset and provide supporting analysis tools to help developers find the weak points of the model in translating syntax elements

Abstract: Automatic Program translation has enormous application value and hence has been attracting significant interest from AI researchers. However, we observe that current program translation models still make elementary syntax errors, particularly, when the target language does not have syntax elements in the source language. Metrics like BLUE, CodeBLUE and computation accuracy may not expose these issues. In this paper we introduce a new metrics for programming language translation and these metrics address these basic syntax errors. We develop a novel active defects probing suite called Syntactic Unit Tests (SUT) which includes a highly interpretable evaluation harness for accuracy and test scoring. Experiments have shown that even powerful models like ChatGPT still make mistakes on these basic unit tests. Specifically, compared to previous program translation task evaluation dataset, its pass rate on our unit tests has decreased by 26.15%. Further our evaluation harness reveal syntactic element errors in which these models exhibit deficiencies.

SUT: Active Defects Probing for Transcompiler Models
Mengnan Qi Yufan Huang"* Maoquan Wang' Yonggiang Yao
Zihan Liu
Bin Gu Colin Clement' Neel Sundaresan'
Microsooft Cloud and AI
Shanghai Jiao Tong University
3 School of Artificial Intelligence, Jilin University
Mohamed bin Zayed University of Artificial Intelligence
(ienggannqj,uffaahuunn,,, maoquanwang, longgiangyao wetemicooooft.cmmm com
1zh1 jhh123ssjuu tu. edu. cn
Abstract
2021; Clement et al., 2020; Rozière et al., 2020, 2021). These innovations have resulted in impres- sive improvements neural transcompilation, but most models still make elementary syntax errors, especially when the source and target languages have different sets of syntactic structures.
Program translation, i.e. transcompilation has been attracting increasing attention from re- searchers due to its enormous application value. However, observe that current program translating models still make elementary syn- lax errors, particularly when the source lan- guage uses syntax elements not present in the target language, which exactly what devel- opers are concerned about while may not be well exposed by frequently used metrics such as BLEU, CodeBLEU and Computation Accu- racy. In this paper, focus on evaluating the model's ability to address these basic syntax eroos and developed an novel active defects probing suite, the Syntactic Unit Tests (SUT) and highly interpretable evaluation harness in- cluding Syntax Unit Test Accuracy (SUT Acc) metric and Syntax Element Test Score (SETS), to help diagnose and promote progress in this area. Our Syntactic Unit Test fills the gap in the community for fine-grained evaluation dataset for program translation. Experimental analysis shows that evaluation harness is more accurate, reliable, and in line with human judgments compared previous metrics.
Existing datasets are often criticized for their lack of granularity, which mask deficiencies in basic syntactic elements that humans care. Their data are derived from randomly sampled code stores or programming contest websites, and the code they contain may not even be executable. As a result, researchers have had to rely on sequence similarity metrics like BLEU, ROUGE, or Code- BLEU to evaluate the quality of code translation. However, these metries have been shown to be in- adequate for objectively assessing the quality of code translation.
In this paper, we propose Syntactic Unit Tests(SUT), which leverage basic syntax struc- tures of each language along with unit tests to verify the correctness of the translations. Tablel presents comparative analysis of SUT against other code translation evaluation datasets. Our ex- periments show that SUT performance of state-of- the-art transcompilers like ChatGPT has definite room for improvement, demonstrating the value of the metric for judging future model improvements. Further, we propose the Syntax Unit Test Accuracy (SUT Acc) and Syntax Element Test Score (SETS), using statistical inference to analysis individual syntax element performance, which is more in line with translation quality that humans concern about. In summary, we make the following contributions:
1 Introduction
Program translation or transcompilation aims to automatically convert source code from one pro- gramming language (e.g., C++) into another (e.g.. Java), meanwhile the results should preserve the program function and ideally follow the target language conventions. The recent proposal of transcompilation datasets such CodexGLUE(Lu al., 2021) benchmark, GeeksforGeeks dataset for computation accuracy(Rozière al., 2020) and XLCoST(Zhu al., 2022) benchmark has pro- pelled related research in the program translation domain (Feng et al., 2020: Kanade et al., 2020; Ah- mad et al., 2021a; Guo et al., 2021; Lachaux et al.,
We build SUT, new set of unit tests for pro- gramming languages C++, Java, Python, and C#, interpretable syntactic categories for each, and harness environment for compiling and executing these unit tests given hypoth- esis translations from any model, which fills the gap in the field of a fine- tie-ggrined evaluation
"Joint first author

---

Table 1: Comparison against other code translation eval datasets (Py Python, JS JavaScript).
Support Languages
Code Form
Arg. tokens
Executable
by Parallel
Syntax Element Analysis
CodeXGLUE
Java, Ct
methods
43,1
XLCoST
Py, PHIP.
code snippets
AVATAR
Jann Python
programs
147.9
V
TranCoder GedksforGeeds
Python,
programs
SUT
Java, Python. C++ C#
unit tests
28.15
dataset for program translation.
source code. TransCoder (Rozière et al., 2020) test whether the translated program outputs the same results as the source program for each test case. They defined Computational Accuracy (CA @ K) to evaluate the probability that a given program can be translated and produce the correct output in K attempts. The results of the CA meet human expectations, but it is too expensive to build test environment for code with complex dependencies.
We propose more accurate reliable, and in line with human judgments evaluation harness, including SUT Acc and SETS, which focus on analysing model performance on fine-grained elements.
We evaluate a variety of the most advanced program translation models in unified man- ner. Our experiments reveal syntactic element errors which these models exhibit deficien- cies, and help researchers working in this field to get potential avenues for improvement.
Syntactic Unit Tests Suite
3.1 Syntactic Unit Tests (SUT)
2 Related Work
2.1 Program Translation Datasets
Inspired by how humans learn programming, we note that it is essential for a program translator to master the syntax elements to learn how to write programs in language. To evaluate whether the translation model has solid understanding of lan- guage characteristics, tests designed for each syn- tax element are necessary. Similar to Computa- tional Accuracy by the authors of TransCoder (Roz- ière et al., 2020), we introduce scenario programs and unit tests which must be satisfied for each syn- tax elements of the languages under consideration. The idea is that, if the model fully understands a syntax element, it should translate the test func- tion correctly and thus pass the unit tests. Further, failures are explicitly associated with syntax ele- ments, allowing us to interpret the performance the model and plan for improvements which target the most impactful elements.
Both CodexGLUE (Lu et al., 2021) and XLCoST (Zhu et al., 2022) benchmark provide complete par- allel pair functions for program translation tasks, while XLCoST also provides random code snippet level data pairs. However, a major issue occurs that they do not account for the environmental de- pendencies of running the program. Transcoder (Rozière et al., 2020) extracts code from the Geeks- forGeeks website and provides test environment, then AVATAR (Ahmad et al., 2021b) collects more parallel datasets for Java-Python translation. How- ever, the generated code must be completely cor- rect to pass the test, while developers often want to clearly locate which part of the code generates errors to facilitate quick debugging.
2.2 Metric for Program Translation
For evaluation of Transcompilation, early studies followed the NLP analogy and mainly employed BLEU (Papineni et ul., 2002) to evaluate the gener- ated translations. But the n-gram overlap does not directly reflect the quality of the resulting transla- tions. CodeBLEU (Ren al., 2020) improves upon BLEU by using features from the abstract syntax tree (AST) and data flow of variables. Neverthe- less, these static metrics still cannot tell whether the translated code has the same behavior as the
The program in each test case accepts one or more parameters as input and generates the ex- pected output. Each case from SUT has four prop- erties: 1) the name and category of the syntax el- ement 2) the souree program 3) the input of the function 4) the expected output of the function. We manually craft SUT datasets for four languages: C++, Java, Python, and C#, following program- ming language instruction sets and show some spe- cific cases in Appendix A.

---

Syntax Unit Table
Source & Target Code Pair
Golden Results
select one example
transcompller
Outputs
Runtime Environment
Translated Code
Source Code
Inputs
Python Execution
replace
plarohaldor
transcomplier
Inputs
Pytind/lpype
wrappled by
cross-language interaction library
</>
Pybind/lpype
Figure I: Comparison the construct process between SUT Acc(the left) and Computation Accuracy(the right). Our method wraps the translated target language code with cross- tross-language interaction python library, and executes it uniformly in the python environment. We only compare final execution results, so target language does not need to provide the corresponding golden code.
3.2 Evaluation Harness
3.2.2 Syntax Element Test Score (SETS)
3.2.1 Syntax Unit Test Accuracy
Based on the results of SUT Acc, we can get some sense of translation performance on the syntax el- ements. However, sometimes SUT case contains multiple syntax elements as some syntax elements are not by themselves able to render a valid pro- gram. Therefore, we further disentangle the statisti- cal effects of multiple syntax elements in each SUT case. For each SUT test case i, tool Tree-sitter' to obtain the set of syntax elements, and construct matrix A., which if that test pro- gram contains syntax element Then when we get the results y; of each unit test, which may empiri- cally be or 0 the test passed or failed. We are in- terested in the fail rate of syntax element though, and so we model the relationship as y = Az, where Y/ log P(fail test,) is the empirical log fail rate of test i, 2y log P(fail syntax element,) unknown fail rate of syntax element j. and A is the known relationship between test i and syntax element j. We then use Lasso regression to get con- sistent results for the estimated log fail rate of each syntax element. The higher the score, the worse the model is at translating such basic elements.
Transcoder (Rozière al., 2020) randomly selected coding problems from the GeeksforGeeks website and used Computation Accuracy as an evaluation metric. Our syntax unit test accuracy metric eval- uates the model's pass rate on disentangled test samples just containing basic elements. Figure compares the calculation process of Our SUT Acc and Transcoder's CA. Our method does not rely on parallel data pairs, which not only reduces poten- tial data noise, but also can be expanded upon with relative ease. The translated function is wrapped Python !nittest! function. Leveraging lan- guage interoperation tools such as JPype* and py- 1 we can use Python to compile and execute the programs in the other languages under consid- eration. Combined with the above unit tests, the system injects each input of the input list to the unit test function and compares the output with the ex- pected output, defining
the parsing
the
passing if they match. Syntax Unit Test Accuracy calculates pass rate of execution results and measures the various basic syntactic correctness of the model.
'https://docs.pytte 3tt http://whhooooor/////rrrrtreenee
html
"nttps://rre-siitter. www.ggiii./t.eeeeeeeeeeeeeeeeeee wwwpooiiettreetteeeeeeeeeeeeeeee

---

Table 2: Comparation the translation performance of these program translators on our SUT and test dataset Transcoder. "To X" "From X": average performance when translating to & from language X. The SUT Ace line shows the pass rate on our SUT. The CA line and the CA filter line respectively show the pass rate on the test dataset Transcoder GeeksforGeeks and after the data errors are filterd.
Model
EvalSet
To C++
From C++
To Ja
From Ja
To Py
From Py
To C#
From C#
Avg.
TransCoder
SUT Ace
35.94%
$5.94%
34.74%
42.58%
45.16%
27.32%
38.61
CA 56.20%
56.10%
50.85%
64.40%
48.05%
34.60%
51.70%
CA filter 61.04%
60.85%
55.08%
69. 69.96%
52.22%
37.53%
56.
TransCoder-ST
Ace 42.02%
49.5y%
43.67%
57.42%
53.28%
31.95%
46.32%
CA 70.65%
64.65%
63.10%
76.75%
65.10%
57.45%
66.28%
CA filter 76.73%
70.13%
68.35%
83,38%
70.74%
62.31% %
71.94%
text-davinci-003
SUT Acc
55.99%
65.31%
58.89%
63.54%
65.60%
41.05%
58.08%
68.67%
59.64%
CA 77.84%
74.72%
72.09%
79.79%
74.14%
69.57%
74.69%
CA filter 84.54%
81.06%
78.09%
86.82%
80.71%
75.45%
81.11%
gpt-3,5-turbo
SUT Acc 60.62%
70.45%
62.50%
67. 67%1
78. 78.99%
50.31 %
63.53%
77.70%
66.41
CA
88.55%
85.21
80.60%
88.71%
85.67%
80.89%
84.94%
CA filter
96.17%
92.45%
87.31%
98.61%
93.09%
87.74%
92.56%
4 Experimental Results
4.1 Evaluation of Program Translators
Devidtii croentert
Datu
Table 2 shows the translation performance of in- fluential program translators with our SUT and the unit test proposed in TransCoder (Rozière al., 2020). For a detailed introduction to these models, please refer to Appendix B. The test data in TransCoder still remains some data impu- rity(mainly on the input the test sample and the mismatch of the given translation pair logic). Table 2 demonstrates that after these errors were addressed, certain translation tasks almostly longer confused the SOTA models, while the SUT remained challenging for them. It is worth noting that although most test functions are elementary, models still fail to translate in many cases. Our test results indicate crucial bottleneck in most existing program translators: these models focus on achieving program mappings in a macroscopic manner, without specially designed tasks to learn code snippet mappings associated with each indi- vidual syntax. This may limit the further improve- ment of model performance.
Figure 2: Error Distribution of These Tested Models.
searchers in identifying the model's weak areas and addressing these issues to quickly improve the model's performance. In the appendix D, we pro- vide brief demonstration of how our reports be utilized enhance model performance.
Conclusion
In this paper, we introduce an interpretable bench- mark via Syntactic Unit Test and associated Syntax Element Test Score, which diagnose the spe- cific weakness of given translation model. It fills the gap the field of fine-grained evaluation dataset for program translation. By our designed evaluation suites, researchers can actively probe deficiencies in their models about specific basic syntactic elements and improve their performance in targeted manner.
4.2 Performance Syntactic Element
In this section, we present the translation perfor- mance of our models the base clements using SETS scores. For the convenience display at fig it is summarized into 5 categories, Variable & Con- stant, Datatype, Data Structure, Operator, and Loop & Branch. More detailed analysis can seen in Appendix C. The diagnose report can assist re-

---

Limitations
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR).
While our work provides an evaluation tool for exploring code translation models on fine-grained underlying semantic elements. However, limited by time and manpower, our data set is still not comprehensive enough, and currently only supports four languages: Python, Java, C++ and C#. This article only uses the S-Tuning experiment (shown in the appendix D) to show the prospect of fine- grained data sets for improving the model effect. Regarding how to use these diagnostic information, more effective solutions need to be proposed by the follow-up community.
Marie-Anne Lachaux, Baptiste Rozière, Marc Szafraniec, and Guillaume Lample. 2021. Dobf: A deobfuscation pre-training objective for program- ming languages. In Advances in Neural Information Processing Systems (NeurlPS), pages 14967-14977
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang. Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, and Duyu Tang al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. In Proceed- ings of the Neural Information Processing Systems Track on Datasets and Benchmarks.
OpenAL. 2022. Chatgpt: Optimizing language mod- els for dialogue. https: //openai ctm/ccoo////a/ blog/ chatgpt. Blog post.
References
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai- Wei Chang. 2021a. Unified pre-training for program understanding and generation. In Proceed- ings of the 2021 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies (NAACL- HI.T), pages 2655-2668. Association for Computa- tional Linguisties.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-2777 -77744.
Kishore Papineni. Salim Roukos. Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation, In Proceedings of the 40th Annual Meeting of the Association for Compu- lational Linguistics, pages 111-318. ACL.
Wasi Uddin Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, and Kai-We Chang. 2021b. Avatar: A parallel corpus for aava-python program translation. arXiv preprint 2 100.0/099000 /1590.
Shuo Ren, Daya Guo. Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. CoRR, abs/2009.10299
Colin Clement, Dawn Drain, Jonathan Timeheck, Alexey Svyatkovskiy, and Neel Sundaresan. 2020. Pymt5: multi mode translation of natural language and python code with transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9052- 9065.
Baptiste Rozière, Marie-Anne Lachaux, Lowik Chanus- sot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. In Advances in Neural Information Processing Systems (NeurIPS).
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xi- aocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Code- bert: A pre-trained model for programming and nat- ural languages. In Findings of the Association for Computational Linguistics. volume EMNLP 2020 of Findings of ACI. pages 1536-1547. Association for Computational Linguistics.
Baptiste Rozière, Jie M. Zhang, François Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. 2021. Leveraging automated unit tests for unsuper- vised code translation. CoRR, abs/2110.06773.
Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravin- dran, Sindhu Tipimeni, and Chandan K Reddy, 2022. Xleost: A benchmark dataset for cross- monss-lingual code intelligence. arXiv preprint 2tt:2206.08477.
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, and Shengyu Fu et al. 2021. Graph- codebert: Pre-training code representations with data flow. In International Conference on Learning Rep- resentations (ICLR). OpenReview.net.
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating con- textual embedding of source code. In Proceedings of Machine Learn- ing (ICML), volume 119 of Proceedings of Machine Learning Research, pages 51 10-5121, PMLR.
the 37th International Conference

---

A Examples of SUT Cases
1) C++
name
SEC
input
expected output
int (int 9, b)
division operator
int / bi return
(),
14,3,21
footint x, int (x y) ? Y
conditional operator
(1. (L. -2), (-2, (2.-1).(0.00
121.
return
int Foo int int for (int 1; x; ti)) {
break in for statement
if (s > y) break; +- 1;
(3.26.))
13,3,61
return 8;
int foo x) {
int a[31;
a[0]
amay index
a[l] 1;
[1,2,3]
[0, 1,
a[2] 2;
return a[x-1];
Figure 3: Examples of C++ SUT cases.
2) Java
name
src
Imput
expected output
int foo(String s) { return s. length;
string length
"abcde"
[5]
int foo(int b)
return 1:
else if (a -- b) {
if-else if-else condition
(4)
[0]
return -1;
int foo a)
int 8;
do-while statement
itt;
[3]
[4]
while a); return 1;
int Foo(int a)
inti] n (1,2,3,4); for n) 1;
for cach statement
[8]
[18]
return
Figure 4: Examples of Java SUT cases.

---

3) Python
input
expected output
expected_output
def Foo
raw string
a r'\n"
[]
["[]a"]
return
conditional expression
def foo b):
[44.9]
[4]
return if olse
def Foo(x):
(ist(range(1e)
continue in for state-
ment for in a: if
12,31
2,5 27
return
def Foo Q::
f3
stack top
131
s append (?)
5 append (3)
return s[-1]
Figure Examples of Python SUT cases.
4) C#
imput
expected output
string (double return ToString():
double string
13.14
13.14", "-1.5"
int foolint a) {
int b; switch
break: 1:
switch statement
break:
[1.233.455
[,
case 3:
break;
fafull:
4;
break;
return b;
string (na
char 12 ('a", 'b', 'e'):
initialize string with array
string
string(a);
('abe"]
return b;
int foo() [
inti] arr intrT(L, 33;
int
aray traversal
arr)
[]
[6]
+ 2;
Figure 6: Examples of C# SUT cases.

---

B Models for evaluation
statements. The model sometimes translates it incorrectly.
TransCoder (Rozière et al., 2020) is an unsu- pervised program translation model that can translate functions between C++, Java, and Python with high accuracy.
iv. do- do-while statement: "do-while" statement is exclusive usage write loop in C++, and the model keeps it in the translation which causes syntax errors.
TransCoder-ST (Rozière et al., 2021) en- hances TransCoder by leveraging an auto- mated unit-testing system to filter out invalid translations in the back-translation process.
switch-case statement: "swi tcwhcasc"se" state- ment is also exclusive usage write branch in C++, and the model translates it into a serial "try-except" expression.
text-davinci-003 originated from Instruct- GPT (Ouyang al., 2022), the OpenAl team continues use Instruction Learning and Re- inforcement Learning from Human Feedback (RLHF) based on the GPT series models to guide the training of the model.
vi. conditional expression: C++ ternary operator ("?:") is used to write branch in the sim- plest way, and one-line "if-else" is used in Python for the same syntax. The model cannot understand its correct translation.
TransCoder-ST gpt-3.5-turbo originated from Chat- GPT (OpenAL, 2022). it is similar to the training method of InstructGPT, and adding more high-quality training data, making it be the currently one of the most competitive code generation models.
TransCoder-ST improves the syntactic de- fects in TransCoder. However, it still cannot fully understand the defect iii above. text-davinci-003
text-davinci-00 can easily solve many funda- mental problems that are prone to errors in the Transcoder series model, but it will still be con- fused by some of the carefully designed problems.
C Detailed Syntax Elements Probing
According to the translations and error messages of failed cases, we manually probe the syntactic de- fects for these models. For simplicity, we analyze one language pair for each model since method- ology can easily be generalized other languages. Specifically, focus on the C++-PPython trans- lation task for Sannsoooee/TaanCCodee the Cython-+++ translation task for text-davinci- 003/gpt-3.5-turfe t.btuub.. The probed syntactic defects for each model are summarized as follows: TransCoder
preserved words: There are many system reserved word in python, c++, c# and java, which are not allowed to be used as variable names, but the reserved word in different lan- guages only partially overlap, which makes the variable names in the source language may be unacceptable in the target language, and models often copied directly for use.
and
i. variable initialization: Initialization in differ- ent languages often has different implementa- tion methods. In some scenarios, it is random initialization, while in others, default values are assigned, which often confuses the current powerful LLMs(large language models).
i. update expression: C++ increment and decre- ment operators ("++",""" "=") have different exe- cution logic when they are used as prefix and postfix. The model is totally confused about its syntax.
ii data structure operation: There some spe- cial data types relatively unpopular opera- tions among them, and the text-davinei-000 model cannot be well understood and sup- ports mapping to corresponding implemen- tation methods in other languages, such as the del method of list in Python.
ii. long keyword: The model naively copies "long" the translation, while Python has no support for this type.
iii. comma expression: C++ comma expression is a way simplify code. can be used in an assignment of several variables (e-g.. "a,b=1, 2"), or used in the init/loop expres- sion of "for" statement to contain multiple
ii dynamic data rype: Python can disregard the type requirements strongly typed languages

---

for assignment, but runtime, it will dis- tinguish the type of value that the variable points to, which is still difficult for static lan- guages such as C++ to implement automatic type derivation.
Step
Code
int foo (int
int b +ta; return b;
Source
int foo (int a)
a+= I:
gpt-3.5-turbo
Logically equivalent modification
gpt-3.5-turbo can sometimes cleverly identify scenarios that are prone to confusion in text- davinci-003, especially in the ii, but still cannot perfectly translate all types of code correctly, espe- cially when the input code structure complex.
return b;
1
Translation the modification
= return
D Syntax Fine-Tuniin (S-Tuning)
Figure 7: An example of creating a S- S-Tuning sample of C++ prefix increment for C+t-oythhn translation. The source C++ function and translation of the modification are combined as parallel sample.
In this section, we present solution for improving the weak points of model translation based on our diagnosis report. It is also very convenient to use this report to expand the model's capabilities in other ways.
each syntax element, the generation of its S-Tuning dataset consists of three steps:
D.1 S-Tuning Dataset
1. Modification: For each source language pro- gram x that contains the syntax element, we generate a logically equivalent function a' with syntax as close as possible syntax of the target language.
Through the analysis report, we found that the model is relatively weak in the mapping ability of the special syntax structure of each language. which is especially reflected in Transcoder and Transcoder-st. We introduce an unsupervised data augmentation method called Syntax Fine-Tuming (S- (S-Tuning) that leverages syntactic defects probed our SUT dataset to generate syntactically cor- rect parallel samples for supervised fine-tuning. All syntactic structures in a source language which not present target language can be transformed to cquivalent but more elementary structures which are closer in appearance o structures the target language. For example, when translating from C++ to Python, int x=i++; can be transformed into int x=i; i=i+1;, which is much closer to a cor- rect Python implementation. This can be viewed analogous to classical syntax-tree transforma- tion transcompilers, but is heuristic to bring the source and target closer together to improve the ability of statistical machine learning models to learn to map patterns. With the set of automatic syntax transformations in hand, we augment the existing training data, which we call the S- S-Tuning dataset, to modify the source language code snip- pets and fine-tune any existing model to improve its ability translate elementary syntactic structures. The S-Tuning dataset built based on a large-scale monolingual dataset each source language under consideration, we show some case in Fig. 8 For
Assuming the model will under- stand the commonly used syntax structures, we use an existing trained transcompilation model like TransCoder to translate the modi- fied function x' into the target language pro- gram y.
Translation:
3. Reinforvement:
We then fine-tune the transcompilation model like TransCoder to translate source language program æ into the translated target language program y, help- ing the model associate the previously poorly understood syntax elements with their proper translation in the target language.
Let us take an example to illustrate the process of S-Tuning dataset generation. The TransCoder- ST (Rozière et al., 2021) model achieves strong performance on translation among C++, Java, and Python. However, it cannot always correctly trans- late some basic expressions such as prefix incre- ment operation. "++" operator has subtle syntax in C++. The prefix and postfix "++" have different semantics depending on the program context. The process of creating the S-Tuning dataset for prefix increment is shown in Fig. 7.

---

Table Statistics of the S-Tuning datasets in our exper- iment. Note that WransCoder/TransCoddee requires monolingual data for fine-tuning,
Table 4: S-Tuning results of TransCoder and TransCoder-ST C+++PPyhon models.
TranoCader Ciecks
SUT
Model
Parallel
Monolingual
CodeR1
Model
TranCCoder
9.6
68.8
69.9
47.1
48.0
source-target
# source
# target
TransCoder (S-Tuning)
TransCoder TransCoder-ST
120k
200k 200k
200k
TransCodee ST
9.8
70.4
70.1
61.3
52.0
IransCode:ST (S-Tuning)
70.5
60k
200k
in one NVIDIA Tesla V100 GPU.
D.2
S-Tuning Experiment
D.3 S-Tuning Results and Discussion
D.2.1 S-Tuning Dataset Generation
Table shows the S- S-Tuning results of TransCoder and TransCoder-ST C++-Pyyhhn models. After S- S-Tuming, TransCoder improves with 15.9% SUT accuracy and 3.5% CA@ 1, and TransCoder-ST im- proves with 18.4% SUT accuracy and 2.19 % CA@1 We believe that if we fine-tune the models for more defects, the model performance can be further en- hanced. We note that static metrics (EM, BLEU, CodeBLEU) have marginal improvements after S- Tuning, which is consistent with our motivation that these metrics cannot assess the syntactic cor- rectness via program execution. Programs with subtle literal differences will have high EM/BLEU/- CodeBLEU scores while they could lead com- pletely different computation results.
IBM Project CodeNet dataset used our mono- lingual data source. is a large-scale curated dataset with diverse implementations where those defects can be easily matched. For each defect, we first search in the monolingual dataset to match the function r that contains the syntax. Then, the func- tion is transformed into a logically equivalent form of that is familiar the model based the mod- ification rules shown in Fig. 8. Next, the vanilla model used to generate the correctly translated function y. Finally, the original function æ and the translated function y are paired as a parallel sam- ple in the S-Tuning dataset. During the S-Tuning of
we note that fine- tuming solely with parallel functions has negative effects model performance, and other training steps with monolingual functions are also indis- pensable.
Morcover, we observe that our S-Tuning pro- cedures cannot be easily applied to some syntax elements. For example, template functions in C++ are highly flexible and hard to use rules to describe logically cquivalent modifications. Therefore, we skip such syntactic defects even though we have probed them. How patch those highly flexible syntactic defects remains an intractable task that is worth exploring in the future work.
D.2.2 Fine-tuning Details
Dataset For match 20,000 C++ functions for each defect to generate parallel S-Tuning dataset, and we also tract 200,000 C++ and 200,000 Python functions as monolingual datasets for other training steps in vanilla TransCoder (i.c., masked language mod- eling, denoising auto-encoding, back-translation). The statistics of the S-Tuning datasets in our exper- iment are shown in Table 3.
we
Training We initialize the three models with their best checkpoints released on GitHub. The models are optimized with the Adam (Kingma and Ba, 2015) optimizer, a learning rate of 2 x 10 Other settings are the same as the original. We fine-tune the models for maximum of 100 epochs, and save the model parameters in every epoch. The final models are selected mainly based on the SUT ac- curacy. The fine-tuning experiments are conducted

---

Language
Syntactic Defeet
Example
Logically Equivalent Modification
Foo 42 { a +- 1;
int foo(int 2)
i. update expression
return b;
int b +ta;
int b 2:2;
return
Foo () [
roo
i. long keyword
long a11);
int (1));
return 8;
return a;
int feo
int 1:
i, j:
for (1 8, 5 8:
8;
5; (+))
1 == 0:
tii. comma expression
for G 1 5: i-+)
+2 1;
return
return j:
int fco() (
int (na [
int 6;
int do while (a 5); return o:
while 1;
do-while statement
1:
C++ PPythn
return a:
int feo
int foo() {
switch (a)
8)
case 8:
b 1:
}
b 1: break:
else if (a 1)
switch-case statement
2;
break;
fault:
else (
b = 3:
3:
return b;
return
feo (
int 1:
int (no [
int b;
int int -1
if (a e) {
vi. conditional expression
) -1;
return b;
return b;
Figure 8: The manually probed syntactic defects for the models studied. For each defect, we show a code snippet example and its corresponding logically equivalent modification.


Reviews:
[{"review_id": "5XS8cK16eZ", "reviewer": "Reviewer_cbZo", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 1, "soundness": 1, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you time and energy to review our manuscript! We read your opinions carefully and we want to explain further about these points of opinions.\n\nThe research objectives of natural language processing not only include languages that have evolved with culture, such as English and French, but also various languages created by humans specifically for certain purposes, such as Esperanto, code languages, tables, etc. Their common feature is that they both contain rich information and are tools used by people to communicate and collide with each other's thinking.\n\nThe following are relevant papers on programming languages (or code languages) that have been published in top-level natural language processing conferences such as ACL and EMNLP in recent years, for your reference, thank you!\n\n**ACL2023:**\n- **[ReCode: Robustness Evaluation of Code Generation Models]**\n- **[Self-Edit: Fault-Aware Code Editor for Code Generation]**\n- **[Code4Struct: Code Generation for Few-Shot Event Structure Prediction]**\n- **[Multi-target Backdoor Attacks for Code Pre-trained Models]**\n- **[DIP: Dead code Insertion based Black-box Attack for Programming Language Model]**\n- **[Backdooring Neural Code Search]**\n\n**EMNLP2022:**\n- **[CoCoa: An Encoder-Decoder Model for Controllable Code-switched Generation]**\n- **[CodeRetriever: A Large Scale Contrastive Pre-Training Method for Code Search]**\n- **[Exploring Representation-level Augmentation for Code Search]**\n- **[Towards Compositional Generalization in Code Search]**\n- **[Language Models of Code are Few-Shot Commonsense Learners]**\n\nOur paper, as well as the aforementioned papers, both discuss tasks related to understanding and generating code. We present a new dataset and evaluation scheme in our paper, which can objectively evaluate the performance of current large-scale language models in code translation tasks. The experiments in our main text and appendix also explore how to use our evaluation tools to improve the performance of these language models. Please review our paper again based on our rebuttal. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* probably the experiments may interest some software engineers at emnlp", "reasons_to_reject": "* programming languages are not natural languages\n* there's not much content related to conference topic at all and paper makes no explanations to bridge the gap", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents testing approach for programming code made by neural translator. The experiment seems interesting to me as a computer scientist but I fail to see the relation of the proposal to natural language processing which is the main topic of the conference.", "excitement": 1, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ByP3qXhqMT", "reviewer": "Reviewer_aNwM", "summary": "", "questions": "- How can we scale the proposed method? (How can we automatically generate the data for the target programming language?)\n- In Lines 187-188: “as some syntax elements are not by themselves able to render a valid program.” It would be good to add a few examples of these syntax elements (From your dataset items)\n- In Lines 205-206: “We then use Lasso regression to get consistent results for the estimated log fail.” Could you please elaborate on why using Lasso regressions and provide a relevant reference(s) on that?\n- In Lines 219-221: “The test data in TransCoder still remains some data impurity(mainly on the input of the test sample and the mismatch of the given translation pair logic).” Could you please provide evidence for that? E.g., reference(s) or example(s).\n- In Fig 2, what the different colors of the “Syntax Unit Test Table” refers to? \n- In Table 2: “To X” and “From X” are the averages over different target program languages? Please specify that in the paper. “Average performance” was a bit unclear.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you time and energy to review our manuscript! We read your opinions carefully and we want to explain further about these points of opinions.\n-   How can we scale the proposed method? (How can we automatically generate the data for the target programming language?)\n \nThe basic syntax element test cases are manually annotated by expert programmers and successfully compiled and run. The dataset we propose is an evaluation dataset, so we value the evaluation quality of the model more than the scale. Subsequently, more types of test cases can be collected and cleaned by crawling programming learning websites to expand the dataset. At present, these websites often have inspection type tags, which are convenient for us to label specific unit tests when further processing.\n\n- Some of the evaluation details are not clear. E.g., how were the text-davinci-003 and gpt-3.5-turbo where prompted?\n\nSorry, we forgot to mention in the paper, the prompt template we used is\n > // Convert this from X code to Y code\n\n > // X code\n\n > %s\n\n > // Y code\n \n X and Y respectively represent the source code language and target code language in the translation task, such as (Java, Python, C++, C #), %s represents the code ontology of the source code language.\n \n-   In Lines 187-188: “as some syntax elements are not by themselves able to render a valid program.” It would be good to add a few examples of these syntax elements (From your dataset items)\n\nSure. Here is the example.\n```\nint func(int x)\n{\n\tint i;\n\tfor(i=0;;++i){\n\t\tif (x^i=0){\n\t\t    return ++i\n\t\t}\n\treturn 0\n}\t\n```\nAlthough we have made every effort to construct simple and independent unit tests for each syntactic element, some cases still struggle to achieve true decoupling and merging. For the loop and branch parts, their judgment conditions partially rely on logical expressions, while the block part may also have variable assignments, operator operations, and other parts. When such a case makes an error, it is actually difficult to define which part of it went wrong.  So these examples will be repeatedly labeled under different categories, and further decoupling operations (such as SET scores) need to be relied on to evaluate their more detailed differences under different categories.\n\n\n- In Lines 205-206: “We then use Lasso regression to get consistent results for the estimated log fail.” Could you please elaborate on why using Lasso regressions and provide a relevant reference(s) on that?\n\nGiven that we explained in the previous question that even though we have made every effort to build code for relatively independent modules, there are still unit tests that are coupled together. We use Lasso regression here to make the regression coefficients more sparse and further decouple the final coupled unit test, making it easier to locate problematic grammar parts.\n\n-   In Lines 219-221: “The test data in TransCoder still remains some data impurity(mainly on the input of the test sample and the mismatch of the given translation pair logic).” Could you please provide evidence for that? E.g., reference(s) or example(s).\n\nSure. We evaluate the transcoder CA data from the official code library, [CodeGen/data/transcoder_evaluation_gfg at main · facebookresearch/CodeGen (github.com)]\n\nFor example, this question SMALLEST_OF_THREE_INTEGERS_WITHOUT_COMPARISON_OPERATORS_1\nthe java code\n```\nstatic int f_gold ( int x , int y , int z ) {\n  if ( ( y / x ) != 1 ) return ( ( y / z ) != 1 ) ? y : z ;\n  return ( ( x / z ) != 1 ) ? x : z ;\n}\n```\nthe c++ code\n```\nint f_gold ( int x, int y, int z ) {\n  if ( ! ( y / x ) ) return ( ! ( y / z ) ) ? y : z;\n  return ( ! ( x / z ) ) ? x : z;\n}\n```\n( y / x ) != 1 in Java not equal  ! ( y / x ) in C++\n\nTheir logic does not match, which results in the model not passing the case even if it is translated correctly.\n\n-   In Fig 2, what the different colors of the “Syntax Unit Test Table” refers to?\n\nIn  Fig 2, our ablation experiment explored the error distribution of these tested models. The meanings represented by these colors are displayed in the upper right corner of the table, where we categorize errors into five categories for specific analysis. (Variable&Constant corresponds to purple, Datatype corresponds to orange, Data Structure corresponds to green, Operator corresponds to red, and Loop &Branch corresponds to purple.), There are still more detailed categories under these five categories, which are marked in the name column of the table. These more detailed error analyses are detailed in Appendix C.\n\n - In Table 2: “To X” and “From X” are the averages over different target program languages? Please specify that in the paper. “Average performance” was a bit unclear.\n \nDue to space limitations in the short article, we converted the subtasks in the experiment to this format to accommodate the submission template. Actually we have briefly explained the meanings of To X and To Y in Table 2 of the paper, \"To X\"&\"From X\": average performance when translating to&from language \". Here, let's give an example to explain in detail that our current evaluation method supports four languages: Python, C#, C++, and Java. So, To Python represents the average pass rate of the three translation tasks: C# to Python, C++ to Python, and Java to Python. From Python represents Python to C #, Python to C++, and Python to Java. Average performance refers to the average of all To X and From X effects at the front of the table.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper provides a set of test units that can be used to evaluate the transcompiler models and diagnose the issues of these models at the syntax element level.\n- The paper proposes two metrics to evaluate the models' abilities in translating the syntactic element in different programming languages.\n- The provided test units and the proposed metrics were used to evaluate the state-of-the-art models.", "reasons_to_reject": "- There are some issues in the presentations of the paper. (E.g., Fig 2 and Table 2, please refer to “Questions For The Authors”)\n- It is not clear how exactly the dataset was constructed and how it can be extended to a new programming language.\n- Some of the evaluation details are not clear. E.g., how were the text-davinci-003 and gpt-3.5-turbo where prompted?", "questions_to_authors": "- How can we scale the proposed method? (How can we automatically generate the data for the target programming language?)\n- In Lines 187-188: “as some syntax elements are not by themselves able to render a valid program.” It would be good to add a few examples of these syntax elements (From your dataset items)\n- In Lines 205-206: “We then use Lasso regression to get consistent results for the estimated log fail.” Could you please elaborate on why using Lasso regressions and provide a relevant reference(s) on that?\n- In Lines 219-221: “The test data in TransCoder still remains some data impurity(mainly on the input of the test sample and the mismatch of the given translation pair logic).” Could you please provide evidence for that? E.g., reference(s) or example(s).\n- In Fig 2, what the different colors of the “Syntax Unit Test Table” refers to? \n- In Table 2: “To X” and “From X” are the averages over different target program languages? Please specify that in the paper. “Average performance” was a bit unclear.", "paper_topic_and_main_contributions": "**Summary:**\nThe paper observes that the current transcompiler (code translation) models still make different elementary syntax mistakes, and the current metrics like BLEU and CodeBLEU may not expose that in an interpretable way. To deal with this issue, the paper proposes a new set of unit tests and metrics to investigate the model abilities in translating the elementary syntax element.\n\n**Main contributions:**\n- The paper release a set of unit test in different programming languages (including C++, Java, Python, and C#). The unit tests are categorized into different syntax groups and can be used to have a fine-grained evaluation at the elementary syntax level.\n- The paper proposes Syntax Unit Test Accuracy (SUT Acc) and Syntax Element Test Score (SETS), and claims that the proposed metrics are more accurate and in line with human judgment.\n- The paper evaluates a set of transcompiler models and shows the issues of these models in dealing with different syntactic element categories.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 081: “We propose a more accurate reliable” -> “We propose a more accurate and reliable” ?"}, "extra_text": {"Missing_References": "- In line 045: “Existing datasets are often criticized for their lack of granularity”, Could you please provide relevant references?(The papers criticized the lack of granularity of the previous datasets)"}}, {"review_id": "03NrRYh3tl", "reviewer": "Reviewer_ZEWW", "summary": "", "questions": "1. The authors try to collect their test suites in a top-down approach, i.e., the test cases are organized by different categories. However, I could only find the aggregate accuracy (Table 2). Could authors also show the per-category error rate?\n\n2. Some numbers in Table 2 are wrongly computed (i.e., 59.64% and 66.41%). I could not get these numbers by either averaging the first 6 columns or 8 columns.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you time and energy to review our manuscript! We read your opinions carefully and we want to explain further about these points of opinions.\n\n -   The authors do not clearly explain how translating programs from the source language to the target language could be useful. If the program translation is not a real problem, how could improving the testing of such models be useful?\n \n I don't quite understand your concerns here. My understanding of your question is as follows, you probably don't quite understand how our evaluation tool can improve the performance of the model. Here, you can refer to Appendix D of our paper, Syntax Fine Tuning (S-Tuning) section, which is actually tailored to the situation. For open-source models such as Transcoder, we can mine similar samples with high error rates in our current evaluation samples as training data. After fine-tuning, we can improve the model's performance in this area.\n\n-  The authors claim that previous metrics like BLEU, ROUGE, and CodeBLEU are not good enough but do not provide detailed explanations or citations (L050-L056).\n\nSorry, this was our oversight. The shortcomings of these n-gram statistical indicators (BLEU, ROUGE, and CodeBLEU) have been analyzed in Transcoder's paper. Here are the original words of the paper,\n\"A limitation of  these metrics is that they do not take into account the syntactic correctness of the generations. Two programs with small syntactic discrepancies will have a high BLEU score while they could lead to  \nvery different compilation and computation outputs. Conversely, semantically equivalent programs  \nwith different implementations will have low BLEU scores.\"\nWe will add it in the official version.\n\n- The authors try to collect their test suites in a top-down approach, i.e., the test cases are organized by different categories. However, I could only find the aggregate accuracy (Table 2). Could authors also show the per-category error rate?\n Sure. Here are the per-category error rate\n \n| |Variable&Constant |Datatype|Data Structure|Operator|Loop&Branch|\n|------|------|------|------|------|------|\n|TransCoder|27.09%|12.31%|44.33%|7.3%|8.86%|\n|TransCoder-ST|22.17%|12.35%|46.3%|6.17%|7.41%|\n|text-davinci-003|49.3%|7.04%|28.17%|7.04%|8.45%|\n|gpt-3.5-turbo|55.56%|6.67%|22.22%|6.67%|8.89%|\n\n- Some numbers in Table 2 are wrongly computed (i.e., 59.64% and 66.41%). I could not get these numbers by either averaging the first 6 columns or 8 columns.\n\nLet me calculate here.\n\nThe average performance of the text-davinci-003 model on SUT Acc is calculated as follows:,\n55.99%+65.31%+58.89%+63.54%+65.60%+41.05%+58.08%+68.67%=477.13%,\n477.13%/8=59.64125%,\n taking the first two places is 59.64%\n\nThe average performance of the gpt-3.5-turbo model on SUT Acc is calculated as follows:,\n60.62% +70.45% +62.50% +67.18% +78.99% +50.31% +63.53% +77.70%=531.28%,\n531.28%/8=66.41%,\n taking the first two places is 66.41%", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed approach is technically sound and simplifies the processing of testing code generators because of Python unit tests.\n- The idea to collect test suites in a top-down approach (i.e., first defining test categories and then collecting test cases) is novel in the code domain. This practice is reminiscent of a series of behavioral testing works in NLP models (for example, \"Beyond Accuracy: Behavioral Testing of NLP models with CheckList\"), which also try to organize test cases by specifications. \n- The idea of decomposing a unit test into syntax elements and computing the element failure rate (Section 3.2.2) is novel and inspiring. The NLP community could benefit from a similar approach when testing models to generate non-code texts. ", "reasons_to_reject": "- The motivation for this work is not strong enough. Specifically\n   - The authors do not clearly explain how translating programs from the source language to the target language could be useful. If the program translation is not a real problem, how could improving the testing of such models be useful?\n  - The authors claim that previous metrics like BLEU, ROUGE, and CodeBLEU are not good enough but do not provide detailed explanations or citations (L050-L056).\n\n- The authors do not provide a detailed validity analysis of the test suites (L153-157). Specifically, are these test cases validated to be correct by held-out annotators?\n", "questions_to_authors": "1. The authors try to collect their test suites in a top-down approach, i.e., the test cases are organized by different categories. However, I could only find the aggregate accuracy (Table 2). Could authors also show the per-category error rate?\n\n2. Some numbers in Table 2 are wrongly computed (i.e., 59.64% and 66.41%). I could not get these numbers by either averaging the first 6 columns or 8 columns.", "paper_topic_and_main_contributions": "The authors propose a new test pipeline and two metrics to improve the testing of program translation models. The new test pipeline has no complicated dependencies and is easier to configure. The authors collect a test suite of 4 programming languages to test their method. The experimental results show that the proposed approach could incur more model failures for both open-source and closed-source models (i.e., `text-davinci-003` and `gpt-3.5-turbo`).", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902–4912, Online. Association for Computational Linguistics.\n- Guanqun Yang, Mirazul Haque, Qiaochu Song, Wei Yang, and Xueqing Liu. 2022. TestAug: A Framework for Augmenting Capability-based NLP Tests. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3480–3495, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.\n- Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress Test Evaluation for Natural Language Inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353, Santa Fe, New Mexico, USA. Association for Computational Linguistics.\n- Marco Tulio Ribeiro and Scott Lundberg. 2022. Adaptive Testing and Debugging of NLP Models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3253–3267, Dublin, Ireland. Association for Computational Linguistics."}}]

Author rebuttal:

Meta-review: The main conclusions of the reviews and the post-rebuttal discussions:

2/ 3 reviewers consider the paper sound (scores 1, 3, 4)
2/ 3 reviewers find the paper exciting (scores 1, 4, 3)
From reading the rebuttal and seeing the scores above, I find that the reviewers consider strong points for soundness the following:

- The paper observes that the current transcompiler (code translation) models still make different elementary syntax mistakes, and the current metrics like BLEU and CodeBLEU may not expose that in an interpretable way. 
- To deal with this issue, the paper proposes a new set of unit tests and metrics to investigate the model abilities in translating the elementary syntax element.
- The idea of decomposing a unit test into syntax elements and computing the element failure rate is novel and inspiring. The NLP community could benefit from a similar approach when testing models to generate non-code texts.

Reviewers had questions regarding dataset construction, evaluation, and how it could be extended to a new programming language. The authors answered these questions in the rebuttal and the reviewers accepted their answers and gave constructive feedback for improving the paper.
Decision: Accept-Main