Explain-then-translate: an analysis on improving program translation with self-generated explanations
Authors: Zilu Tang
Mayank Agarwal
Alexander G Shypula
Bailin Wang
Derry Wijaya
Jie Chen
Yoon Kim
Venue: EMNLP 2023 Findings (2023)
Keywords: code generation
machine translation
program translation
large language model
prompting
chain-of-thought
TL;DR: We analyzed how useful self-generated natural language explanations of code can be in the setting of program translations

Abstract: This work explores the use of self-generated natural language explanations as an intermediate step for code-to-code translation with  language models. Across three types of explanations and 19 programming languages constructed from the MultiPL-E dataset, we find the explanations to be particularly effective in the zero-shot case, improving performance by 12\% on average.  Improvements with natural language explanations are particularly pronounced on difficult programs. We release our dataset, code, and canonical solutions in all 19 languages.

An Analysis on Improving Program Translation with Self-generated Explanations
Zilu Tang', Mayank Agarwal', Alex Shypula', Bailin Wang',
Derry Wijaya'.5, Jie Chen', Yoon Kim
Boston University "IITIIBM Watson AI Lab, IBM Research
UPenn MIT Monnsh University Indonesia zilutangêbu.c edu
Abstract
that Explain Sesllaintheen-Trannee Translate improves zero-shot per- formance consistently in 18 Python-to-X transla- tion directions, but much less so in the few-shot setting. We observe detailed explanations to be more useful when translating into high-resource PL.s and from low-resource into other low-resource PLs. In contrast, translating from high- to low- resource PL's benefits from more abstract expla- nations. To aid future research in code-to-code translation across diverse language, we release our evaluation system, as well as canonical solutions all languages, providing a 19-way parallel program translation evaluation set.
This work explores the use of self-generated natural language explanations an intermedi- ate step for code-to-code translation with lan- guage models. Across three types expla- nations and 19 programming languages structed from the MultiPL-E dataset (Cassano al., 2022), we find the explanations be par- ticularly effective in the zero-shot case, improv- ing performance by 12% on average. Improve- ments with natural language explanations particularly pronounced on difficult programs. We release our dataset, code, and canonical solutions in all 19 languages."
1 Introduction
Explain-then- Srsplannthen--Transtteee for Code Translation
Program translation (i.e., translating code from one language another) has significant value real- life applications, including in legacy software mod- ernization and enabling programmers to quickly adapt to new languages. Within prompt-based ap- proaches to code translation, Chen et al. (2023b) recently found that simply prompting an LLM to generate explanations of the source program before generating the target program can improve perfor- mance. However, this conclusion drawn on single translation direction from C++ to Python (Lachaux et al., 2020), and lacks evaluation on a broader set of programming languages includ- ing low-resource languages-a a key component of code-to-code translation tasks in software mod- emnization setting.
In code translation, we are given code x in a source language and must generate a program y in a tar- get language that functionally equivalent to x. In this paper we are interested whether a self- generated natural language explanation 2 can be used to improve this translation process.
2.1 Prompt Variations
Fig shows an example of our prompts for program translation. In addition to the direct translation baseline (Fig 1, left), we experiment with 3 types of explanations (full prompts are given in Apx C); exp: We ask the model to explain the source program few sentences (Fig I, right). exp-lbl: ask the model to explain the source program line by line. This roughly mirrors the setup in Chen et al. (2023b).
This paper systemtically evaluates this "Explain- Tren--ranslate" approach code translation through the MultiL-EE dataset (Cassano et al., 2022). As the original dataset was constructed for the NL-to-code setting, we repurpose this dataset into a code-to-code, "MultiPL-C2C"" dataset. We analyze our results in 36 different translation direc- tions over different types of explanations. We find
3. exp-Ibl-d: We ask the model to explain the source program line by line in additional detail. In particular if an individual line is complicated, ask it to break it down,
"While we focus on natural language explanations in our main experiments, in Apx J we show results with other types of "oxpllaaaation"" such as another pivot language, pseudocode, etc.
/tap explain-then--ranslleee translate

---

Standard Translation (baseline): direct
Explain-then- Transllae (ours): exp
Please translate the following Python code
Can you explain what this Python program does with a couple of sentences? The goal with the explanation, is that reader easily rewrite the program in ### Python version def Largest_divisorde for i in reversed(range(n)) if n % == 0:
to R?
**# Python version
def
int) -> int: for in reversed(range((n)) if n 8: return 1
int) -> int:
return
##* Explanation
This function takes an integer 'n' and returns the largest divisor of 'n'. It does this by iterating over all integers from "n-1" down to 'I' and checking if 'n' is divisible by each integer. If it is, the function returns that integer. If no divisor is found, the function returns '1', иия version
MNR R version
largest_divins divisor function(n) {
for (1 rev(seq(n)) if (n 8% 8)
return(1)
largest _dvvisor targest_divisor for (i in rev (seq 1))) { if (n 3% == 0) return(i)
function(n) {
1/ Translation is incorrect!
return(1)
// Translation is correct!
Figure Compared to direct code translation prompt, exp (ours) prompts models to explain the code before translating. Blue highlights are model completions, and red highlights point out the crucial difference between the two translations, Example prompts and explanations for exp-lbl and exp-lbl-d in Apx C. E.19
plain individual fragment of the line, and then summarize the purpose for the entire line. This prompting method allows us to decom- pose compositionally difficult fragments of the code down, re-use individual fragments of explanation before explaining the whole line, similar to Zhou et al. (2022).
Additional details on language-specific stop tokens and how few-shot programs are selected are de- scribed in Apx E and Apx F, respectively.
2.2 Dataset: MultiPL-C2CC
MultiPL-E (Cassano al., 2022) benchmark that was recently introduced in an effort to evalu- ate NL-to-code generation capabilities of language models in 19 different programming languages.? Cassano et al. (2022) groups these languages by resource level:
When generating explanations, we treat the token sequence \n# as a stopping sequence in order to prevent models from generating target translations (since we condition target program with translated signatures in addition to explanations). Sometimes, model might generate larget-anggugge-ppeee details (equivalent classes, attempted translation, etc.). In order to control for inconsistencies caused by the ungge-laaguagg-speciin explanations, we re-use the same explanations (from Python-Java) for all Python-to-X translation experiments (Sec- tion 3.... Before reusing, we also remove any target specific information with programmatic rules be reused across experiments. For complete- ness, in Apx G.1 we show the impact of removing larget-language-speiffcc details for the exp experi- ments: the effects are generally insignificant, but are more pronounced low- now-resource languages.
High-resource: JavaScript (js), Python(py), Java (jv). C++* (cpp). TypeScript* (ts)
Medium-resource: PHP (php). Ruby (rb). C#* (cs), Go (go)
Low-resource: Perl (pl). R (r), Rust* (rs),
Scala* (sc), Swift* (sw)
Extremely-loo-rrsoorre Bash (sh), Lua (lua), Racket (rkt), Julia (jI), D* (d)
'Concretely, taking the original HumanEval (Chen 2021) MBPP (Austin al., 2021) datasets (where models are prompted with problem description erate Python program that solves the problem). MultiPL-E built transpilers
are tasked to gen-
unit tests as well as code generation prompts such that models can be evaluated from NL-to-code 19 different languages (Python + 18 additional languages). de indicates statically typed language (vs. dynamically)

---

Res
Hligh
Medium
Extremely Low
cpp
jv
plp
20
g4
dirces(0)
76.6
76
79.9
42.4
58.3
40.4
64.4
51.3
56.2
31.3
expe())
62.9
55.4
exgxth020)
84-3
80.3
82.5
54.7
77.5
65.3
4B
75.1
56.8
61.5
exp-Ibl-sk0)
83.2
79.1
84
63.2
77.6
82.7
$2.7
65.8
62.9
45.5
76.2
73.8
74.8
36.4
59.5
36.1
dircco(4)
95
56.5
52.4
56.4
72.2
74.8
expe3
77.8
$2.1
69.8
81.7
83.3
51.5
73.2
35.6
75
73.7
67.1
75.1
67.9
64.7
89.0
45.9
exp-Ib004)
87.6
82.4
$5.3
$7.1
81.3
80.1
$4.7
50.8
72
36.9
74,9
73.9
70.1
71.9
68.6
65.6
72.1
46.2
exp tbi- 41-5)
87.8
8339
36.3
83.5
82
80.7
84.3
30.0
70.9
57
76.3
73.5
70.9
72.3
69.4
43.5
71.1
45.1
expt14)
88.9
80.8
85
83.3
78.6
81.4
15.]
52.2
71.6
57.3
74.5
76.1
71.3
73.1
67.9
46.2
72.9
50.7
exg-Ihildh(4)
87.9
84.6
$7.8
88.5
81.5
80.6
65.3
$1.6
70.1
57
78.7
77.7
71.9
72.1
69.4
46.3
71.6
Table 1: Translation pass 1 from Python to X. uses heuristically selected explanations (Sec 3.4). Parenthesis in
trial indicates shots, Best within same-shot (no heuristics) underscored and overall best bold.
To repurpose MultiPL-E into a code-to-coe codde trans- lation dataset, we change the task formulation by including canonical Python programs in the prompt and removing the NL problem descriptions. We dub this version of the dataset as MultiPL-C2CC, and release it for future work in this area.
Experiments and Discussion
In our study we focus on two main sets of transla- tion directions: Python-to-X, where we translate from Python to 18 other target languages ranging from high extremely-low--esouee to-X, where we target a representative set of transla- tion directions varying source and target language resource levels and typing characteristics ($3.2).
($3.1), and X-
2.3 Metrics
We evaluate our methods using unit test pass rate (Chen et al. 2021: Cassano et al. 2022) as string match-based evaluations do not capture the diverse ways in which program can be translated and still be functionally equivalent to the source. We calculate the pass rate as:
We analyze translation improvements across mod- els of 4 different sizes ($3.3) and discuss improving individual explanations through heuristics ($3..4). Finally we show our method improves more on aeffcull--t-trrnnllie examples ($3.5) and provide ablations to understand what NL explanations im- proves performance and whether alternative self- generated contexts could help ($3.6).
p = E
Problems
3.1 Python-to-X Translation
where is the total number of generations, and C is the number of correct generations. The best sampling temperature & (or top-p) (Holtzman et al., 2020) is often dependent on n/k, where smaller temperatures are best for small n/k, while larger temperatures increase the generation diversity (bet- ter recall) and can improve pass rate with large n/k. We prioritize precision and calculate pass @ with = 20, t = 0.2, and top-p= 0.95 following Cassano et al. (2022).
In Table we present results of the Python-to-X experiments in the zero- and four-shot settings with GPT-3.5. Results with open-source models results show similar trends and are shown in Apx G.9.
Natural language explanations improve perfor- mance in the zero-shot setting, and this effect is more pronounced in low-resource languages. Providing explanations improves relative perfor- mance by 11.5% on average across 18 target lan- guages. Regardless of the target language resource level, the best explanation improves translation with average relative improvement of 6.2% in high- resource languages and 14.5% in extremely loww resource languages. There is no significant dif- ference between improvements on translating into statically vs. dynamically typed languages. Self- generated explanations even slightly outperform human-written doc-string instructions that are part of the original HumanEval dataset (see Apx J).
2.4 Models
We evaluated four models of varying sizes. We main report the results from GPT-3.5 w.--turbo-0000 in the main paper un- less otherwise specified, and defer the results from open source models (CodeGen2-19 CodeGen2- 16B, and Llama2Codelnstruutttaa 2023; Rozière et al., 2023)) the appendix.
(Nijkamp et al.,
"We also considered into CodeXGLUE (Lu et al., 2021) and TransCoder (Lachaux et al., 2020) for the unit tests eval- uations, but initial studies suggested significant number of examples (more than 25%) contain mistakes in gold programs or inadequate unit tests (see Apx A, B).
High-resource target languages benefit from de- tailed explanations while low-resource alterna- tives benefit from abstract explanations. We

---

hypothesize that high-resource languages benefit from more detailed explanations due to higher co- occurrences of NL and PL the pretraining cor- pora; whereas in low-resource languages we spec- ulate the additional detail may introduce spurious correlations. Since we re-use explanations across translation directions, the translation performance difference can be attributed only to the code gener- ation step.
corect (4 shorty -IINGI (too)) Faeeigh
OraType0c1p
Natural language explanations less helpful in the few-shot setting, but good few-shot ex- amples are crucial. the four-shot setting, the
0.66
JendSonpp
average improvement is much less at 1.1%, al- though some language pairs observe as much as a 1% improvement. Average improvement in high-resource languages (1.2%) is smaller than that languages (3.4%). The most detailed explanations perform the best in 12 out of 18 language directions amongst explanation types. This is likely due to the carefully curated few-shot examples, which are semantically and syntactically complex enough to benefit from de- composition and explanations (see in Apx FF for more details).
0.00
0.00
0.20
0.05
0.10
(0
Figure 2: Zero-shot exp improvements correlate with few-shot improvements over baselines (r² 0.151).
in
analysis of explanations (Apx E.19), pass (Apx G.7), correlation between NL-to-code vs. translation results (Apx G.8). and improvements vs. translation difficulty (Sec 3.5, Apx I).
3.2 Alternative Translation Directions
To understand whether findings only hold for Python (or only high-resource languages), ex- periment on additional translation directions from different resource groups and typing characteristics, and present our results in Table 2. Since source lan- guages are different, we do not re-use explanations. In the four-shot explanation, we use zero-shot gen- crated explanations (3.1). ). In the following section, we have High-high-resource languages and Ext- Low=extremecl--oowrrtee resource languages. Results from open-source model results in Apx H.
Few-shot explanations result in worse perfor- mance than zero-shot explanations. The most abstract explanation (exp) performs the worst (best in only 3 out of 18 directions) in the few-shot set- ting. Since we source the few-shot explanations from minimally modified zero-shot explanations, including these self-generated explanations simply restricts the model's explanation to follow stylistic patterns and decreases the diversity of the expla- nations. In Apx G.2, we disentangle the effect of target specific explanation and zero/four-sher set- ting provide further evidence to this point.
High-to-Extl Low and High-to-High follow a simi- lar patterns as Python-to-X. In zero-shot, High- to-High has varied performance across different ex- planation types, whereas High-to-ExtLow benefits mostly from simple explanations (exp). In four- shot, there is little improvements in High-to- High, but some improvements
Improvements in the zero-shot setting correlate with improvements in the few-shot setting. Ex- cept for a few outliers, Fig shows a good cor- relation. This interesting because few-shot is manually curated and written in PL, while expla- nation is self-gen generateedd and written in NI. In our ablation 3.6 and Apx we further analyze to what extent the source of information provides the struc- ture of the output, and whether the correctness of the sequence actually matters.
tethhto-EEtLLo..
SetLow-to-Highh Models are poor at explaining low-resource language programs. The improve- ment in ExtLow-to- High trials is limited in both zero- and four-shot directions. Across explanation methods, can see general decrease in perfor- mance from high-level (exp) detailed (exp-Ibl-d) explanations. We speculate that LLMs generally struggle to understand and explain lower-resse resoure PLs; more details may introduce more errors which
A Addtiional analyses. In the appendix we pro- vide the breakdown error types (Apx G.4, G.5), source program lengths (Apx G.6), qualitative

---

shot
4 shot
Resource
Type
src-tgt
direct
exp-lbl
exp-Ibi-d
direct
cap-Ibl
exp-lbl-d
D-D
Ps Ps-j8
85.5
85.7
84.3
83.2
86.6
86.
87.6
87.8
D-S
71.3
82.5
86.7
High-to-High
9pp py
92.3
90.2
90.5
93.9
89.9
89.4
89.2
8-8
JV-epo
77
79.8
83
79.8
82.3
81.1
78.4
78.7
D-D
J8 tki
30.2
41.9
41.3
37.9
44.7
51.8
49.8
46.3
High-to-Fxrl.io .ow
D-S
Py
42
42.7
44.4
45.6
48.5
46.2
45.1
opp
71.9
72.5
79.2
77.2
75,1
S-S
jV
60.2
75.4
68.8
72.2
72.7
72.9
72.4
71
D-D
Iua
89.5
85.9
86.5
84
38.5
89.3
88.4
83.2
D.
rkt jv
65.9
77.1
74.4
71.7
86.4
84.2
80.8
79.4
ExiLow-to-HHl
82
87.1
S
d cpp
88.4
81.4
80.5
88.4
86.3
81.7
85.5
D-D
lua tki
29.6
45.2
38
37.7
49.2
49
50
48.7
D-S
rkt-jl
63.3
64,3
67.7
62.1
71.7
70.4
73
70.5
ExiLow ExiLow
d 68.4
69.4
66.8
69.9
71.7
70.3
72.5
73.5
j1-d
43.6
43.8
Table 2: Translation pass@1 between 16 different pairs of languages. Resource indicates the language re-
source levels of the source and target. Type indicates the source and target language typing characteristics (DIS=dynamicallySsaticallll typed). The best runs within the same-shot setting are in bold.
Exp Type
sre-tgt
random
len
line-e
line
frag
logprob
oracle
pass 6
exp
Py-jv
83.0 +
84.2
84.8
84,2
81.0
85.4
93.7
85.4
exp-Ibl
Py-jv
85.1 + 1.0
84.4
84.4
84.4
85.1
83.8
90.3
exp-Ibl-d
jy-jv
86.1 +
86.4
85.1
88.3
87.0
92.2
87.6
exp
py-rkt
41.1 1 2.0
41.0
39.8
39.8
41.6
43.5
70.2
40.7
Table 3: Explanation selection heuristics performance. We estimate heuristics performance (pass@1 1, n=1) and validate the best method by generating 20 programs and calculate pass (n = 20). No heuristic is able to outperform baseline with exp-Ibl did not verify with p
Assolute morovement
Relarive morevement
Better explanations transferable and lead to better translations. We also experimented with CodeGen2-1B 1B by using GPT-3.5 generated explanations (Fig 3) and found to improve per- formance further, outperforming self-generated ex- planations in 12 out of 18 directions. Comparing absolute improvements against CodeGen2-1B IB with self-explanations, we find that GGT-335--generaae explanations improve
350-
coderoee
120
when generating higher resource than lower resource languages, indicating that smaller models are less sensitive to improve- ments. More analyses are given Apx G.9.
Figure 3: Py-to-X translation (pass zero-shot) im- provements (best explanation over baseline) across mod- els grouped by target language resource level.
may compound into the translation phase.
3.4 Explanation Selection with Heuristics
3.3 Comparisons Across Different LMs
In the context of thain-of-thouuht prompting, Wang et al. (2022) demonstrate the importance of sampling diverse "reasoning" paths. is diffi- cult to ensemble sampled programs from language models, but we find sampling diverse explanations, where we first sample 20 explanations and then sample program each, improve recall for correct programs (pass@ 10) than sampling 20 pro- grams from 1 explanation, direct translation in zero/four-shot settings. This indicates that there is significant room for improvement we are to be able to select the best explanation that can gener-
Improvements are robust across models. From Fig 3 and 9, we can see that in general, the larger the model, the larger the absolute improvement with self- reffeeeeeeteed explanations. In terms of im- provement over resource levels, our method im- proves low-resource language generations more with larger models, while improving high-resource languages more with smaller models. See detailed result tables in Apx H and G.9. CodeGen2-16RR the only model that does not improve consistently with explanations.

---

ate correct program with the highest probability (oracle column Table 3).
Passi1 Civel Pass
Pike
Motivated by the potential of diverse explana- tions improve translation results, explore five explanation re-ranking heuristics: 1) length of ex- planation (in characters) excluding code; 2) lines of source code explained (line-e); 3) number of lines explanations; 4) number code fragments en- closed < <.6 5) with logprob (Zhang et al. 2022; Min et al. 2022a), ranking the explanations with a weighted combination of a** p(code) explanation)) +
.60-
Coant
100
90
(1 (1-al a)= p p(explanation code) using CodeGen2 (Ni-
0.0 0.2
04
05
0.5
jkamp al., 2023) (more details in Apx L).?
Filer Faaeg1
For each explanation type, we generate 20 ex- planations and I program from each explanation (train set). We estimate each hcuristics' perfor- mance by averaging the pass rates of its selected (argmax) explanations for each individual problem in the train set. For random baseline, we select explanation for cach program randomly;" and for oracle, we select the explanations with the highest pass rates in the train set. For each explanation type, we pick the heuristics with the best estimated pass@1 1 (n = 1). and generate 20 programs from these explanations for pass @1 (n 20) score (right most column in Table 3). We use zero-shot explanations for exp (see See 3.1) and four-shot for exp-Ibl and exp-Ibl-d. Our main results are shown in Table 3, from which we observe the following.
Figure 4: We count explanation improvement cases over direct pass@1 1. Results include all trials between Python- to-X and X-to-X directions. For better contrast, all problems with the same exp pass @ and direct pass @ are removed.
Some heuristics generalize across explanation types. Only frag and logprob perform robustly. Intuitively, frag makes sense because data contain- ing a piece of code and an explanation more likely to be correct if the author refers to the code more frequently. With logprob, since we are effec- tively measuring the mutual information between codes and explanations (Zhang et al., 2022).
There is still ample room for improvement. As can see in the difference between oracle and pass 1, no heuristics able to come close this oracle upper bound. This gap is much larger in high-to to-low-resource translation direction (py-rkt). Qualitatively, we could not distinguish a good ex- planation from a bad one manually (Apx L.3 and L.4), suggesting that the distinction between "good" and "bad" explanations may be hidden due stylis- tic noise (wording, spacing, etc.), potentially due to chance.
Heuristics can improve performance, and this is robust across different target languages. With exp. logprob improves upon random by abso- lute 2.54% (p 0.055), 10 and frag improves explain-Ibl-d upon random baseline by absolute 2.2% (p- = 0.033) with simulation. Both improve- ments can be reproduced with pass@1, so we in- clude these heuristically selected explanations as two additional rows in Table 1. With logprob se- lected exp. we improve or match performance in 15/18 directions, with average improvement of 1.7% (p < 0.001). With frag selected exp-Ibl- simp, improve match performance in 13/18 directions, averaging 0.48% (p- 0.022). See Apx G.3 for more comparisons.
3.5 Which programs benefit from explanations?
To understand where natural language explanations benefit most, we investigate how exp improvement varies across problem hardness, which is approxi- mated through direct translation pass@) 1. Through Fig 4, we discovered that explanation improves dif- ficult problems (left of x-axis), but could hurt casy problems (right of x-axis). This potentially sug- gests we could determine which program explain using a hardness threshold, and improve perfor- mance further. We verified the validity of such ap- proach through our oracle metric (direct pass@ l),
'Markdown pattern for referencing code
"We tried scoring with GPT-3.5 directly as well but found it to not outperform the random baseline (Apx K)
"We discuss the trade-offs of alternative settings (sampling explanations and programs from each) in Apx M.
"Repeated 100 times to obtain mean and variance. The resulting v-values are from a one-tailed paired t-test.

---

and show the full results in Apx I. We found selec- tive explanation to improve performance over di- rect and exp 35/36 directions. We leave building such difficulty-basee problem selector for future work.
jv
php
sw
rkt
exp
6.9
8.6
6.1
10
exp-Ibl
6.5
9.1
10.7
7.7
swap.s (exp)
7.9
4.5
7.6
swap-s (lb))
9.2
9.3
3.5
obf-exp (exp)
9.3
7.9
7.3
ohf-exp (lbl)
9.8
9.6
5.2
3.6 Ablation Studies
del-s-0.25 (exp)
5.3
7.7
4
5.4
del-s-0.5 (exp)
5.6
7.2
4
7.1
We perform additional ablation studies to under- stand what aspects of the explanations improve translation ($3.6.1), whether explanations are ro- bust to variations in surface semantics/readabll ity of the source code ($3.7, Apx N), and if self- generated context in PL could help few-shot exam- ples ($3.8, Apx J). Additionally, we explore the relationship between context length and improve- ments in Apx G.6.
del-s-0.25 (lbl)
6.7
7.7
8.5
6.2
del-s-0.5 (lbl)
5.2
7.7
7
4.3
del-w-0.25 (exp)
1.8
7.7
3.4
6
del-w-0.5 (exp)
0.1
3,3
-1.1
5,8
del-w-0.25 (lbl)
4.5
6.5
5.1
3.3
del-w-0.5 (lbl)
0.8
6.5
3.2
3
rand-exp (exp)
3.8
6.1
3.1
5.1
ret-exp (exp)
-0.2
5.3
1.8
-0.2
no-exp (exp)
-1.5
-54.1
-10
3
rand-cxp (lbl)
7.7
7.4
ret-exp (lbl)
0.8
5.2
5.4
-1.2
no-cxp (lhl)
-2.7
-48.2
-3.9
0.4
3.6.1 Explanation Ablation
Table 4: Performance improvements over the baseline with various explanation variations from Python-to-X (see $3.4). IbI=exp-lbl. del-w-0.5-deletin 50% of words at random. Best (ablated) exp in bold and best (ablated) exp-Ibl underlined.
We select target languages of different resource levels where explanations provide the most im- provements (zero-shot) for Python-to-X. With each explanation, we ablate in following ways:
(del-w). Lastly, when models receive completely irrelevant explanations (rand-exp), they are able to recover performance to some extent; but if the explanations are convincingly mislcading (ret-exp) performance deterioates.
swap-s: We randomly reorder sentences (exp) or code-explanation segments (exp-Ibl) to test explanation provides structural supervision.
obf-exp: We obfuscate source programs (see amples in Apx N), where function and variable names are replaced systematically with templates (FUNC_ O, VAR_1, etc.). This tests whether an ex- planation uses specific variable references (struc- tural supervision at loken level).
Models rely on semantics of explanations less when generating lower-resource languages. Different types of ablations affect lower-resource languages more uniformly than higher-resource lan- guages. Relative to exp/exp-lbl, ablations that com- pletely alter the semantics of the explanations (del- w) decreases improvements less in lower-resource languages than higher counterparts, while abla- tions that keep overall semantics of the explanation (swap-s) decreases improvements less in higher- resource languages.
ret-exp, rand-exp, no-exp: We replace the ex- planation with another program's explanation ran- through retrieval (ret-exp. details Apx O), or with an empty string (no-exp) to verify if explanations need to be correct/relevant.
domly (rand-exp),
del-w: We randomly remove some of the words and see if fluency (i.e. high logprob) is necessary.
Semantics of explanation not the only picture. Despite explanations having completely wrong se- mantics (rand-exp. ret-exp), models still improve from the added context. CoT self-generated reason- ing has been found to follow unmentioned/hidden biases within the context (Turpin al., 2023). It would be interesting investigate further what remaining biases (if any) contribute to the improve- ments program translation.
del-s: We randomly remove a percentage of sen- tences (exp) or code-explanation paragraphs (exp- Ibl) to see the dependency of the translation on the completeness of the explanation.
Explanation needs to be coherent, relevant, and observe in Table 4,
correct. From what explanations do not provide much structural guid- ance (swap-s), and models do not overly rely on their completeness (del-s). Models do not rely on the surface form to redirect attention as much (obf- exp). but they do require explanations to fluent
3.7 Source Program Ablation
To test whether our explanation methods work with a different distribution source programs, we ob-

---

quality
jv
php
pl
When the few-shot program is similar, verifica- tion is important. In Table 5, we observe that when the retrieved one-shot is paired with a wrong target program, the decrease in performance is much more significant than in the fixed-shot set- ting." 11 In other words, curated few-shots are ro- bust to label noise. This is consistent with the ear- lier conclusion in Table 4 that an "almost-correat" explanation (ret-exp) could influence generation more than when it is obviously incorrect (rand- exp). If verification is available, retrieve-gold shows that a formally correct (similar) program is more useful than a natural language explanation. However, on average, self-generated unverified ex- planations (exp) still outperform one-shot in all directions (fixed-unk by 0.7-4.5%; retrieve-unk by 0.5-2.0%), indicating that NL generations often have higher quality than programs and can serve better medium for intermediate reasoning step.
direct
76
68.4
58.3
31.3
fixed
gold
82.1
78.5
70.1
41.7
fixed
unk
82.2
74.3
63.7
39.1
fixed
had
78.4
75
67.2
40.8
# problem
158
161
161
161
retrieve # problem
gold
83.5
78.9
68.4
46.7
158
161
161
161
retrieve # problem
unk
81.9
76.5
67.2
38.3
138
80
132
154
retrieve
bad
67.2
61.3
54.3
36.1
exp
82.9
77
68.2
41.3
Table Translation performance using different source programs and quality of target one-shot example. fixed indicate fixed example for one-shot, and retrieve uses BM25 retrieved program as one-shot example. quality indicates the correctness of the target program in the one-shot. unk is any program output sampled randomly from GPT-3.5 and bad is programs sampled from the incorrect pool. Since not every retrieved prob- lem has incorrect (or correct) generations, we report the # problems evaluated for each retrieval setting.
To further compare NI. explanations with other formal/non- formal reasoning steps, in Apx J, we experiment with translating to pivot program- ming language before translating to the target lan- guage (e.g. Python-Java-Phe By controlling the pivot language correctness, we can more closely verify the model's translation performance sensitiv- ity to correctness in context. The result indicates mistakes intermediate PL steps corrupt transla- tion performance more than imperfect NL expla- nations. This indicates that using seff-generated NL as an intermediate step often helps more than relf-generated PL., and reasoning in the NL space is advantageous for language models.
fuscate variables and funciton names source pro- grams, removing all surface form semantic informa- tion (Apx N). When comparing direct translation vs. exp. in Table 18, we find explanations to be robust regardless of surface semanties of the code. In half the trials, relative improvements using ex- planation are even higher for obfuscated source code than original code. This is potentially due to the fact that explanations become more reliant on actual syntax of the program, rather than hallucinat- ing on the program semantics from surface variable names. This is promising because when using mod- els in the real world, such for app modernization, there is no guarantee of having readable code.
Related Work
3.8 Few-shot Ablations
Explanation in deep learning. Many works have explored using explanations to improve lan- guage models. Hase and Bansal (2022) investi- gate various ways explanations can be introduced in modeling and find it most useful for retrieving similar data. Joshi et al. (2022) find explanation regularization to improve OOD performance. Most works in LLMs generate explanation using zero- shot, few-shot, or finetuning. before generating the target response (Ling et al., 2017; Nye et al., 2021; Wei et al., 2022: Mukherjee et al., 2023; Hsich et al., 2023). A few works have also explored post- hoc explanations (Lampinen et al., 2022; Krishna
Since few-shot improvements correlate with expla- nation improvements ($3.1) we conduct ablations check how sensitive the models are to the correct- ness of few-shot examples, and whether unverified self-generated few-shots can also improve as well as explanation does. Here, we replace our correct few-shot examples with naturally generated pro- grams from GPT-3.5 (high logprob, but formally unverified (unk) or incorrect (bad)), and observe how much self-generated few-shots improve trans- lation and models' sensitivity towards their correct- ness. We experiment with a fixed one-shot example as well as retrieval one-shot to observe the improve- ment/sensitivity when the exemple program is sim- ilar or different from the testing program.
"retrieval-bad (Table 5) should be taken lightly since the subset of problems evaluated for had also likely contains harder problems.

---

al., 2023). Wiegreffe al. (2021) and Chan et al. (2022) propose metrics to quantify rationalc-label association. We refer readers with further interest to surveys (Miller, 2019; Hartmann and Sonntag, 2022; Zhao et al., 2023).
NL steps from NL instructions before generating the code. Zelikman et al. (2023) and decomposes complex problems in NL and generated/verified subproblems to achieve high performance in NL- to-code. Chen et al. (2023a) finetune policy mod- els to correct code given human critique. Wang et al. (2023c) searches multiple hypothesis in NL before generating PL targets. Our method uses self- generated context without overly relying on feed- back, few-shot examples, or complicated frame- works, and is targeting code-translation specif- ically instcad of NL-to-code generation. Chen et al. (2023b) briefly mentions in their ablation studies that explanation improves translation in Python- C+++, but our analysis reveals more nu- anced settings in which explanations improve code translation.
Language model for code. Much work has been dedicated to applying transformer-based language models to NL and PL generation (Brown et al., 2020; Ahmad et al., 2021; Chen et al., 2021; Li et al., 2022; Ouyang et al., 2022). TransCoder leverages unsupervised pretraining and supervised finetuming to build one of the first neural transpilers (Lachaux et al., 2020, 2021: Roziere al., 2021). Later works obtain impressive zero and few-shot learners by simply pretraining on NL and PI. data with language modeling or derivative training ob- jective (Ahmad et al., 2021; Nijkamp et al., 2022; Chen et al., 2021; Scao et al., 2022; Xu et al., 2022; Nijkamp et al., 2023; Allal et al., 2023; Li et al., 2023; Rozière et al., 2023; Athiwaratkun et al.. 2022).
Conclusion
This work conducts thorough analysis of the performance of large language models in pro- gram translation by using different types of self- generated explanations as an intermediate step. Models generate higher quality detailed explana- tions for high-resource languages, while still gen- erating good enough abstract explanations for low- resource languages. With simple heuristics, we have also demonstrated the potential to improve explanation quality and consequently translation quality. We identify key requirements for explana- tion and find that on average, mistakes in NI. are less detrimental to performance, and do not require verification to perform well compared to using PL as self-generated contexts.
Intermediate state prompting. The Explain- Then-Translate approach is an instance of chain- of-thought prompting (Wei et al. 2022; Nye et al. 2021), where the model is prompted generate reasoning steps before the final answer. Follow-up works have found it to be useful on numerous tasks outside of niche reasoning tasks (Wang et al., 2022: Zhou et al., 2022; Chowdhery et al., 2022; Suzgun et al., 2022; Yao et al., 2023). In our setting, we find most improvements to come from the zero- shot setting (Kojima et al., 2022). Different from previous works, our task focuses program trans- lation, with significant token level correspondence between the source and target. Ghazvininejad et al. (2023) and Lu al. (2023) improve NL translation by augmenting prompts with dictionary transla- tions, but their contexts are not self-generated. would be interesting to explore whether other forms of "explanations" (e.g., BNF grammars (Wang et al., 2023a)) could further improve performance, especially on low-resource languages which may not have been encountered frequently during pre- training.
Limitations
There are several limitations to our work. First, while we focus on the (adapted) MultiPL-E bench- mark due to its widespread use, it is unclear whether programs in this benchmark are repre- sentative of programs that are targets for code-to- code translation. Second, while we saw our con- clusions to largely hold across GPT-3.5 and other open-source models, it is unclear whether they will hold for more powerful LLMs such as GPT-4. Fi- nally, somewhat disappointingly we found natural language explanations to be not helpful in the few-shot setting, and insofar obtaining several demonstration examples for each language pair is quite practical, natural language explanations for code-to-code translation may not actually be useful for many applications of interest.
Code prompting and feedback. In the code- generation space, Zelikman al. (2022) incor- porate model-generated rationales given question- answer pairs as part of fine-tuning to improve model reasoning capabilities. Jiang et al. (2023) use few-shot examples to teach models to create

---

Acknowledgements
Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan, Samuel R Bow- man, Kyunghyun Cho, and Ethan Perez. 2023a. Im- proving code generation by training with natural lan- guage feedback. arXiv preprint Siii::2303.164499
We thank all anonymous reviewers. We also thank Justin Weisz, Afra Feyza Akyürek, Najoung Kim, Aditya Yetetore, Garry Kuwanto, Reuben Tan, and Yuwen Pu for their helpful discussions and sugges- tions. This project is supported by the MIT-IBM Watson AT lab and DARPA HR001 I 8S0044 (the LwLL program). ZT is supported by Lu Lingzi Fellowship from Boston University. Any opin- ions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the view of the sponsor.
Mark Chen, Jemry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- tr, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. 2021. Evaluat- ing large language models trained on code. arXiv preprint arXiv:2107.03374.
References
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified pre-training for pro- gram understanding and generation. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, pages 2655-220 2668.
Loubna Ben Allal, Raymond Li, Denis Kocetkow, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. 2023. Santacoder: don't t reach for the stars! arXiv preprint arXiv:2301.03990
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou, 2023b. Teaching large language mod- els to self- arXiv preprint 2rXfv:2304.05/28
Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang. Xiaopeng Li. Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, et al. 2022. Multi-lingual evaluation of code generation models. In The Eleventh International Conference on Learning Representations.
Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. 2023c. Exploring the use of large lan- guage models for reference-free text quality evalua- tion: A preliminary empirical study. arXiv preprint arXiv:2304.0072.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. 2021. Program synthesis with large language models. arXiv preprint 100.0:2200..7.....
Tom Brown Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Jinlan Fu, See-Kiong Ng. Zhenghao Jiang, and Pengfei Liu. 2023. Gpiscore: Evaluate as you desire, arXiv preprint arXiv:2302.04116
Marjan Ghazvininejad, Hila Gonen, and Luke Zettle- moyer. 2023. Dictionary- hhased phrase-level prompt- ing of large language models for machine translation. arXiv preprint arXiv:2302.0788.
Federico Cassano, John Gouwar, Daniel Nguyen, Syd- Costin, Donald Pinckney, Ming Ho Yee. Yangtian Zi, Carolyn Jane Ander- son, Molly Q Feldman, et al. 2022. A scalable and extensible approach to benchmarking n12code for 18 programming languages. arXiv preprint 2XXi:2200.00827..
ney Nguyen. Luna Phipps-Costin,
Mareike Hartmann and Daniel Sonntag. 2022. A survey improving nip models with human explanations. In Proceedings of the First Workshop on Learning with Natural Language Supervision. pages 40-47.
Aaron Chan, Shaoliang Nie. Liang Tan, Xiaochang Peng, Hamed Firooz, Maziar Sanjabi, and Xiang Ren. 2022. Frame: Evaluating rationale-label consis- tency metrics for free-text rationales. arXiv preprint arXiv:2207.00779.
Peter Hase and Mohit Bansal. 2022. When can mod- els learn from explanations? a formal framework for understanding the roles of explanation data. In Proceedings of the First Workshop on Learning with Natural Language Supervision, pages 29-39.

---

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text de- generation. In International Conference on Learning Representations.
et al. 2022. Competition code generation with
alphacode. Science, 378(6624):1092-10097
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pages 158-167.
Cheng-Yo Yu Hsich, Chun-Liang Li, Chih-Kuan Ych, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprin! arXiv:2305, 02.00..0.......
Hongyuan Lu, Haoyang Huang, Dongdong Zhang, Hao- ran Yang. Wai Lam, and Furu Wei. 2023. Chain- of-dictionary prompting elicits translation in large language models. arXiv preprint arXiv:2305.065575
Xue Jiang. Yihong Dong, Lecheng Wang, Qiwei Shang, and Ge Li. 2023. Self-planning code gen- eration with large language model. arXiv preprint arXiv:2303.066m 06689.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).
Brihi Joshi, Aaron Chan, Ziyi Liu, and Xiang Ren. 2022. Er-test evaluating explanation regularization methods for nip models. In Proceedings of the 2nd Work- shop Trustworthy Natural Language Processing (TrustNLP 2022). pages 1391
Thirty-
Tom Koemi and Christian Federmann. 2023. Large language models are state-of-the-arr evaluators of translation quality. arXiv preprin! arXiv: 14520.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 8086-8098.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In JCML 2022 Workshop on Knowledge Retrieval and Language Models.
Tim Miller. 2019, Explanation in artificial intelligence: Insights from the social sciences. Artificial intelli- gence, 267:1-38.
Satyapriya Krishna. Jiaqi Ma. Dylan Slack, Asma Ghan- deharioun, Sameer Singh, and Himabindu Lakkaraju. 2023. Post hoc explanations of language mod- els can improve language models. arXiv preprint arXiv:2305.1142:
Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022a. Noisy channel language model prompting for few-shot text classification. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 5316-5330.
Marie-Anne Lachaux, Baptiste Roziere, Lowik Chanus- sot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. arXiv preprint arXiv:2006.03/11
Sewon Min, Xinxi Lyu. Ari Holtzman, Mikel Artetxe. Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022b. Rethinking the role of demonstra- tions: What makes in context learning work? arXiv preprint arX2v:2202. 12837.
Marie-Anne Lachaux, Baptiste Roziere, Marc Szafraniec, and Guillaume Lample. 2021. Dobf: A deobfuscation pre-training objective for program- ming languages. Advances in Neural Information Processing Systems, 34: 14967-14979.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawa- har, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orea: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707.
Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. 2022. Can language models learn from explanations in con- text? In Findings of the Association for Computa- tional Linguisties: EMNLP 2022, pages 537-563.
Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Sil- vio Savarese, and Yingbo Zhou. 2023. Codegen2: Lessons for training llms on programming and natu- ral languages. arXiv preprint arXiv:2305.023099
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! arXiv preprint 23X5vv2305.06166.
Erik Nijkamp. Bo Pang. Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingho Zhou, Silvio Savarese, and Caiming Xiong, 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203, 2003.13344344 13474.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schritwieserr Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago,

---

Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, al. 2021. Show your work: Scratch- pads for intermediate computation with language models. arXiv preprini wrXiv2212..01114
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint UrXiv:2001.11900
Sarah Wiegreffe, Ana Marasović, and Noah Smith, 2021. Measuring association between labels and free-text rationales. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 102844
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. Advances Neural Information Processing Systems, 35:27730-27744,
Frank F Xu, Uni Alon, Graham Neubig, and Vincent Jo- sua Hellendoorn. 2022. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium Machine Progrumming, pages 1-10.
Baptiste Rozière, Jonas Gehring. Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code lama: Open foundation models for code. arXiv preprint arXiv:2308. 12950.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao. and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.1000
Baptiste Roziere, Jie M Zhang, Francois Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. 2021. Leveraging automated unit tests for unsupervised code translation. arXiv preprint 1rXiv:2110.06733.
Eric Zelikman, Qian Huang, Gabriel Poesia, Noah D Goodman, and Nick Haber. 2023. Parsel: A (de-) compositional framework for algorithmic rea- soning with language models. arXiv preprint 2IXiv.2212.10561
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ili,, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Good- man. 2022. Star: Bootstrapping reasoning with rea- soning. In Advances in Neural Information Process- ing Systems, volume 35, pages 15476- -15488. Curran Associates, Inc.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Se- bastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, al. 2022. Challenging big-bence tasks and whether Shain-of-thoughe can solve them. arXiv preprint arXiv: 200.02.2..
Tianyi Zhang, Tao Yu, Tatsunori Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried, and Sida I Wang, 2022. Coder reviewer reranking for code generation. arXiv preprint arXiv:2211. 1.1490..
Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. 2023. Language models don't 't always say what they think: Unfaithful explana- tions in -fain-of-thought prompting. arXiv preprint arXiv:2305.0433 043388.
Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang. Dawei Yin, and Mengnan Du. 2023. Explainability for large language models: A survey. arXiv preprint wrXiv22309001299
Bailin Wang, Zi Wang. Xuezhi Wang, Yuan Cao, Rif A Saurous, and Yoon Kim. 2023a. Grammar prompting for domain-specific language generation with large language models. arXiv preprini 2rXiv:2305.192..
Denny Zhou, Nathanacl Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoe Le, and Fdd Chi. 2022. Least-to-most prompting enables complex reason- ing in large language models. arXiv preprint arXiv:2205.1000 10625.
Jiaan Wang, Yunlong Liang. Fandong Meng. Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou 2023b. Is chatgpt good nig evaluator? preliminary study. arXiv preprint arXiv:2303.0e 04408.
A Transcoder Evaluation Clean-up
TransCoder (Lachaux et al., 2020) and other fol- lowups (Chowdhery et al., 2022; Chen et al., 2023b) evaluated their translation model using mined programs from "GeeksforGeek", an online programming practice website. Within the evalu- ation (valid and test) set, we find more than 25% of the data points were erroneous due to mistakes in gold programs or inadequate unit tests. After
Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D Goodman. 2023c. Hy- pothesis search: Inductive reasoning with language models. arXiv preprint XXXi:2339.05666
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Lee, Ed Chi, and Denny Zhou. 2022. Self-consistency im- proves chain of thought reasoning in language mod- els. arXiv preprint 200.

---

cleaning, we find the dataset be too easy for our baseline model GPT-3.5 (Apx B) and seek more challenging benchmarks (MultiPL-E).
(porammmare 16e7) and (f_filled (*parameters_ 10e7)): n_success*=1
Within Transcoder validation and test set, in the direction of C++ to Python, and we find several type of errors that are hindering existing evalua- tions:
There are about 12 programs for such cases.
Inadequate test: non-sensible test inputs Orig- inal unit tests sometimes contain non-sensible test inputs. For instance, some programs are intended to process string representation of numbers, but the unit test input would contain non-digit strings, This is especially harmful for translation based on semantics because extreme edge-cases like this should not be used as regular tests. Therefore, we replace these tests inputs with values from the ap- propriate space. There are 7 instances of such pro- grams.
Inadequate test: exact comparison Python tests on integers and float types are often inadequate in establishing equality. Depending on the library and specific function called, integers or floats can be rounded with different precision. Because tests in TransCoder evaluation dataset work by comparing gold Python program's 's output with that of the gen- erated Python program, if the generated program does not use the exact function call, exact equality (==) is not sufficient. Thus, we changed all cases of such violation from checking exact equality to approximate equality like below:
Wrong source program Sometimes, C++ pro- grams are incorrect non-standardized (such returning 0/1 returning true/false, or miss- ing return statements). We standardized all C++ programs. There are around instances of such program we fixed.
Old way of evaluating exact equality ( par poramteessset). n_successi=1
f_filled(*
Wrong gold program Of all errors, this is the most devastating type of error. Without a correct gold program to compare output with, no matter what the generation is there will be no chance of success. Errors range vastly from in-correct range value, wrong indentation, missing return statements, to using wrong/undefined variables. One of the most frequent errors that spans more than 60 programs is the in-correct translation of integer division from C++ to Python. Often it is translated to / when it should be //. We found 100+ examples of this type of error.
# New way of evaluating approximate equality, borrowed from other instances of float comparison tests in TransCoder.
if a1 (8.0000001 + abs (rrameters_set)) / sabs(f_filledd parameters_set) 0.0000001)) < 0.001: n_success+=1
There are 40+ examples of such case.
Inadequate test: default large number In C++ programs, integer variables are often initialized with INT INT_MAX and INT_MIN as default values when comparing with other values in loop. When such values are not replaced and returned, compar- ing the exact values of such placeholders does not make sense, especially when there is not single correct way of translating it into Python. Hence, to accommodate such programs, we check for approx- imate equality between input-output or check that the output of gold and generated program both re- turn an extremely large/small placeholder number.
We stress that have only fixed existing is- sues have noticed in C++ to Python direction. With sampled programs from Java, we have also ob- served errors gold programs and tests frequently.
Leakage to training data One of the other rea- sons that we do not formally evaluate our meth- ods on TransCoder dataset is because that the TransCoder dataset is very likely to have been in- cluded in GPTs' training corpora. We are often able to generate the entire program in test/valid split by only providing the signature. The unique detok- enized program pattern makes easy for the model to detect and regurgitate from training sequences.
old way
abs(1 (8.0008001 abs(f_gold(t par (awaameeessstteeeeeeeeeeeessssee
) / (fbslfffilled( filled parameters_set) 0.0000001) < 0.001: n_success*=1
# New way of evaluating approximate equality, as well as including cases where outputs are extremely large/small value placeholders.
B TransCoder Evaluation
if abs(1 - (0. (0.0008901 + (f__gold(* parameters_seti / (f fflleed parameters_set)) 0.0009001)) < 0.001 or ((
We report the full evaluations on TransCoder eval+test set with GPT-3.5 in Table 6. The model

---

Original
Fixed
test and compiler feedback. When unit tests or gold programs are wrong, relying on these feedback in- formation is the only way to improve translation performance. In another word, translation mod- els start to deviate from a faithful translation and "over-correct" itself to pass unit tests. Our methods, however, assumes such access to such feedback signals, and slightly under-perform.
Other work
Codex
80.4
?
self-debug
92.5
?
Our work
direct (no-sig)
80.8
90
exp (no-sig)
69.1
77
direct (gold-sig)
81
88
When looking at gold-sig trials, we see that by asking model to explain the program and then trans- late consistently improve over baseline. In no-sig trials, we see consistent under-penn snder-performance of our methods. Most of the errors are result of generated program having incorrect signature (incorrect num- ber of input variables). Often, these source C++ programs contain non-intuitive redundant input variables. For example, in the following exam- ple EENMEENTT LEEEEEETTAARRAA.._.. variable n non-intuitive. find the most frequent element in a array, should normally process the whole input array arr.
exp (gold-sig)
81.7
89.9
direct (typed-sig)
80.6
90.8
exp (typed-sig)
80.2
90.6
Table 6: TransCoder evaluation with the original and fixed dataset, C++ Python, total of 567/566 programs). Entries under our work are pass @ (n=l))
improves around (absolute) 10% on the fixed dataset, indicating the lack of difficulty in the pro- gram. Through manual inspection we find the syntax of the programs to rudimentary. We conducted three type of evaluations regarding the amount of target program signature specification we provided:
__T___RRA_____ the input program were to
progr am generated through direct trans lation
def f_gold arr
Hash dict )
for i range ( ( )
no-sig: we only prompt the beginning of Python program generation with def. This signals the beginning of python program with- out specifying any part of the signature. This is what we use compare with Self-debug (Chen al., 2023b) baselines.
arr [ i ]
fash.keys (
Hash [ arr [i] +=
else :
Hash [ arr [i]]=1
max_count res =
for i in Hash if maxx_count < Hash [i]]
res =
the gold programs signature to prompt the rest of Python generation. Some of the programs are translated incorrectly after being prepended with signature because a few Python gold programs contain non-equivalent program name and input variable name. This is the same evaluation setting in which we conducted other experiments in HumanEval, and is main source of comparison.
gold-sig:
max_count Hash [i]
return res
program generated with explanation def f_ f_filled arr hash for i in arr if i in hash hash [ i +=
else hash [i] =1
max max_count = = =-1
typed-sig: In addition to prompting target pro- gram with gold programs, which do not con- tain Python type hints, built rule-based transpiler translate the C++ program signa- tures Python with type hints, and prompt the rest of the program.
for key value in hash. items (
if max_count value key return max_count value
By explaining before translating, the model aligns the generation to the explanation, which of- ten follows the distribution of "typical programs" in the wild.
We do not have direct measurement of the self- debug method on the fixed dataset. We can, how- ever, infer that self-debug method's improvements over baseline is from the additional access to unit
In typed-sie we see general improvements over other trials. This expected because we can

---

provided more accurate information regarding the input-oulput types. The slight perforperf of exp compared to direct could be due to small experiment trial size.
return tostringnnn).eeuals new StringBuilder (Logg tostring
(e))_ (oo).ren to))rrng();;
}
Full Prompts
*#* Python version
C.1 Python-Java direct (four-shot)
from typing import List
def is_sorted(lst: List[int]) -> bool:
You are a helpful and faithful compiler that transpiles Python code Java lst]) code. Please translate the following Python code to Java?
count_digit = dict([(i, 8) for i
for in lst:
count count_digit(i]--
if any mnycount_ddgit(i] > 2 for in
### Python version
lst):
return False
from typing import Tuple
if all([st[f-1] <= lst[i] for in
range(1, len(lst))):
def even_odd_ awve__ddd_alindroone padd_palindroee(( int) Tuple(int, int]: def (n_palindrome(m))
return True
else: return False
return str (n) str(n) [::-1]
even_pallnddrome. wwwalliddoeecouuuee odd bad_ppalindrme_comm
even_pa
N** Java version
import java.util.*; import java. lang. reclect.;;
for in range (1, n+1): if 152 == and is__alindrome{1
import org.javatuples.n;
import java.
):
odd_palinde conndrome_eoo ame_count 1 elif 182 == and is_palindrome
import java.math.*;
import java.io.*;
(i):
import java.util.stream..
even_pal even_palindromeee wealliiddoe_couum += 1 return (even_pal wamiinomme_ommmtt odd add_palinee rnd_ooooccuutt
class Problem
public static boolean isSorted ArrayList<Long> lst) {
HashMapp<Lnng, Long> count_digit = new HashMap<Long, Long> (); for (long i lst)
#*# Java version
import java.util.x;
if (count_digit..conteissss
import wwv.llang.reflese
(1)) { count_ cunt_digitt.ut((1 juut_digitt.ee(((
import org. org.javatuples.
import java.
import java.math.x;
import java.io.*;
else cunt_diggit.ptth: 1L);
import java.util.stream..
class Problem [
public static Pair<Long, Long> evenOddPal indrome (long n)
for (long lst)
long LednPalindromeComm indr
0;
if (count_ (ccunt_digii..ee(()) 2)
long addPalindromeConm omeCount = ;;
{ return false;
for (long = <= i++) {
if (i % = isPalindrome oddPalindromeCountes
for (int = lst.size();
i++)
} else if (i & == e
(lst.get(f - 1) > lst. get(i)) return false;
isPalindrome (1)) {
evenPal evenPalindromee
return wair.with( evenPaliindromeContt oddPalindromecountt)
return true;
}
Python verston
private static boolean isPalindrom (long
from typing import List

---

}
def
str) -> List[str]:
result = current_string = current_depth =
return result;
for in paren_string:
Python version
if = '(':
current_ depth +=
from typing import List
perrent_string.apeeee
elif == ')':
def port_array(arrayy List[int]) -> List[int]:
current_depth -=
return [] if len(array) == 0 else (array, reverse= (array[0] tarray [-1]) % 2 == 6)
if current_depth
result.append("
current_string))
Java version
current__tring.cllerr
import java.util.*;
result
import java. lang. reflect:;
import org.javatuples.e;
### Java version
import java.
import java.math.
import java.util.x; import java. lava.lang.-ae reflect.:)
import java.
import java.utíl.streem..
import arg.javatuples.;;
class Problen
import sava.security..; import java.math.x;
public static ArrayList<Long> sor (orttrrra(Arraayiisttoooo array) {
import java.io.*;
1f (array.size() == 0) { return new ArrayList<Long >(;
import java.util.strramm.t
class Problem
public static ArrayList<Stringg separateParendrouussttrrtgn paren_string)
else {
boolean reverse = (array. get(0) array.get(array,
ArrayList<string> result new
size() 1)) 2 == 0; collections. sort(array); if (reverse) Collections. array);
ArrayListestringe():
ArrayListcCharacter>
current_string new ArrayList< Character>>); int current_depth 8;
return array;
for (int 0; paren_string. length(); 1++) {
char larppaaaarriinee string. charAt(i);
}
if (c == (()
### Python version
current_depth += 1;
current_string current_string..dd(c)
<insert Python program here>
} else if (c == ")
current_dens perrhtt_eppthha - 1;
### Java version
current_strinn current_string..dd(())
<insert Java completion herex
1f (current_depth ==
StringBuilder sb new StringBui (der
C.2 Python-Java exp (four-shot)
current_string.st size ());
for (Character ch current_ current_string)
You are an helpful AI assistant who understands all programming languages and can translate between them at ease. Can you explain what this Python program does with couple of sentences ? The goal with the explanation, is so that reader can easily rewrite the program in Java?
ab.append (ch);
(sb toString()); current_string. clear();

---

### Python version
long oddPal indrommCunt 8;
from typing import Tuple
for (long = <= i++)
if (1 % & (sPalindrome(t))
def oven_odd__allfndromm (n: int) Tuple(int, int]: def "s_paliinromme()):
oddPalindroomcooumiii
else if (i % ==
return str (n) == str(n)
isPalindrome (i))
evenPalindroomeCou
gonn_pallndrome_couuu cocurrotoeeccottt/e// odd hod__aalindrom_coom
return Pair.with(
for 1 in range(1, n*1):
evenPalinndromecount, oddPalindrromeoonnt))
if 1%2 == and is_palindrome(
):
odd_ps add_palindrome _omue_tuttttt
elif 182 and is_palindrome
private static boolean isPalindrome (long n)
(1):
return even_palindrome
return Long, toString(n) equalss( StringBui Ider (Long, toString reverrev toString());
Leven_palindree wenddone_oooous
### Explanation
}
This function takes integer 'n' input and returns tuple of two integers. The first integer is the number of and 'n', and the second integer is the number of odd palindromes between 1 and
### Python verston
from typing import List
pal indreess between
def (1st:: List[int]) -> bool: count_digit dict([(1, 9) for 4 in lst])
'n', palindrome is number that reads the
forwards and backwards.
for i lst:
dount_digit[i]==11
The function first defines helper function "is_palindrome" that takes number and returns 'True' if palindrame and 'False* otherwise. This done by converting the number a string and checking equal to its reverse.
(ccoooon_digii(i))) for in lst):
False
all([st[1-1] Ist[i] for in
range len(lst))): return True
string
else: return False
The function then initializes two counters, "eeven_palindrone counts odd hodd_palindrme_comee cotrrmmmeccuut'..// to 0. then loops through all numbers between 1 and 'n' (inclusive) and checks if each number is a palindrome and even or odd. If the number is palindrome and odd, it increments 'odd_palin pel_palindrome_ccunt If the number is palindrome and even, it increments
#** Explanation
This function takes in list of integers and returns boolean indicating whether the list sorted in non-descending order and has than occurrences of any integer. The function first creates dictionary where the keys are integers in the List and the values times that integer appears in the list. It then checks if any integer appears more twice in the list, and returns False if so. Finally, checks if the list sorted non- descending order, and returns True so, and False otherwise.
c
the number of
Finally, the function returns tuple of the two counters.
#*# Java version
import java.util.x; import www.llang.reflecee
import arg. org.javatuples.:;
import java.
Java version
import java.math.
import java.
import java.util. import java. lang, reflect.*; import org. arg.javatuples... import secaritcu) import java.math.s;
import java.util.stream.
class Problem [
public static Pair<Long, Long> evenOddPalindroon (long n)
long evenPal indromeCount
import java.1o.*;

---

import Java.util.stream
The function works by iterating over each character in the input string, and keeping track of the current depth of nested parentheses. When an opening parenthesis is encountered, the depth is incremented and the character is added to the current string. When closing parenthesis is encountered, the depth is decremented and the character is added to the current string. If the depth reaches complete group has been found, the current string is added to the result list and cleared for the next group. The function returns the list of groups
class Problem {
public static boolean isSorted(
ArrayList<Long> 1st) {
HashHap<Long, Long? count_digit new HashMap<Long, Long> ();
for (long i : lst) {
if (conm__cigtt..oottrsssee (i))
fuunt_digit.putie count_digit.gett((
it means that
;
} else
count_digit.put(( 1t);
}
for (long i lst)
if sooutt_digit.get(( 2)
##* Java version
false;
import java.util.*;
}
import java. lang. reflect.;;
}
import org.javatuples.:;
for (int 1; lst.size(); i++) {
import java.secur
import java.math.*;
if (1st.get(1 1) lst.
import java.io.*;
get(i)) {
import java.uti1.stream.r
return false;
class Problem
}
public static ArrayList<string>
}
seprrateearenrruusstttrrnnn
return true;
paren_string)
ArrayList<String> result
ArrayListcStrrnng((}
ArrayListsCharaccter current_string new ArrayList< Character> int current_ deppth 8;
### Python version
from typing import List
for (int 0; paren_string. length(); 1+4) char с paren_string. charAt(i);
def
str) -> List[str]:
result []
current_string []
(c == '() {
current_depth
current_depth 1;
current_string.adee else (c == ')') {
for с in paren_string:
if c == '(":
current_depth 1;
current_depth +=
current_string.ade
surrent_string.appete
elif == ")';
if (current_depth == 0)
current_depth -
{
current_string.
StringBuilder sb
new StringBuilder(( current_ street_sttringsize ());
if current_depth == 6:
result..apendd"..jonnn
current_string)
for (Character ch current_strine {
current_string.clear
(bhap
return result
} addst. toString()): surrent_string. clear();
инR Explanation
This function takes a string of parentheses and returns list of strings, where each string is a group of parentheses. A group is defined as a set of parentheses that are balanced, meaning that each opening parenthesis has corresponding closing parenthesis
return result;

---

### Python version
from typing import List
def sort_ aarraa List[int]) >> List[int]:
return [] if Len(array) == else
cortedaarayy reverse= (errayl0]+arpe [-1]) % 2 == 0)
#*# Explanation
This function takes list integers "array" as input and returns a sorted list integers. The function first checks the input Hst is empty. If it is, it returns an empty list. Otherwise, it sorts the list using the 'sorted' function. The 'reverse' parameter is set to 'True' if the sum of the first and last elements of the array is even (and the array will be sorted in descending order), otherwise, it is set to 'False' (and the array will be sorted in ascending arder).
#*# Java version
import java.util.x;
import
import org.javatuples.t;
import java.security.
import java.math.*;
import java.io.*;
import java.util.stream.r:
class Problem [
public static ArrayList<Long>
sor (erttrray(rrryyissccloogg array) if (array.size() == 8) { return new ArrayList<Long
>0;;
else {
boolean reverse (array.
get(a) + verra..ettaarene
size() 1)) &
if (reverse) { Collections.reveet array);
return array;
### Python version
<insert Python program here>
### Explanation
Thristinsert Python program explanation here>
#*# version
<insert completion here>

---

C.3 Python-Java exp-Ibi (four-shot)
elif 182 == and is_ even_pal wven_palindrome_cm Within the for-loop, the program checks if each number is a palindrome and even or odd. If the number is Balindrome and odd, it increments hod_palindroog
+=
You an helpful AI assistant understands programming languages and can translate between them you explain program does line by Line? The goal with explanation, so reader can easily rewrite the program Java?
this Python
If the number is palindrome and even, it increments
return liven_palindrome_.cmm edd_ aad__palindrom__co _emu_tt Finally, the function returns tuple of the two counters.
#*# Python version
from typing import Tuple
def even_odd, palinddoom oeee__dd_palindrrme (n: int) +>
##* Java version
Tuple[int, int]:
def is_ is_palinnroom(():
import java.util.:;
return str (n) == str [::-1]
import java.lang lang. reflect.;;
import org.javatuples.:;
import java.security.""
odd_palinndrom__com
import java.math.*;
import java.io.*;
for i in range(1, n+1); if 182 == and is_
import java.util.streem..
class Problem
):
public static Pair<Long, Long?
odd pww.paaaaoaaeeammmmmmmmmmmma couut += elif 182 == and is is_palindrome
evenoddPalindrome(( n)
long evenPal long oddPal roneonot 0;
(i):
even_pal indrome_ddeemmmommmm return Liven_palindrome_coom odd_pal_indrome_count))
for (long i =;; <= n; i++)
if 2 oddPalinndromCCounnt+ (1 isPalindrome (1))
isPalindrome (i))
иNa Explanation
The code an implementation of finding the number of even and odd palindrome numbers between 1 and 'n' inclusive). It takes an integer input returns a tuple of two integers: even and odd palindrome number counts. A palindrome is number that reads the same forwards and backwards.
as
return Pair.with( evenPaliindromeconnt, oddPalindromeCount)
private static boolean isPalindrome (long n) {
def is.palindromeen):
return Nong.toSSting(n .arpggnn)euuussssss.s.sa equals new Stringgurilder toonttirin toString
return str(n) == [n: The function first defines a helper function "is_palindromm" that takes a number and returns 'True! if it is pal indrome and 'False" otherwise. This is done by converting the number to string and checking if the string is equal to its reverse.
(n)) re))_ree toString());
}
}
Python version
from typing import List
dige_paatiddromeccmmme odd_ hod_palindrome The counter variable
def (sstt: List[int]) -> bool: count_digit dict([(1, 8) lst]) for lst:
=
caan_palindrammeccomm count' and odd_ hid_pallindrme__ee record the result.
are used to
count_digft(f])=11 if any any(count_digit(i] for in
for in range
lst):
The loops goes through numbers between and 'n' (inclusive)
return False
if all([st(i-1] <= lst[i] for in range(1, len(lst))): return True
is_palindrome (1):
odd add_ppalnndrome_r count
else:

---

return False
for (int i 1; 1 lst.size(); i++) if (lst.get(1 - 1) > lst. get(i)) { return false;
#*# Explanation
The code an implementation of checking whether List is sorted non don-descnnding order has no more than two occurrences of any integer. takes in list integers and returns a boolean.
return true;
}
count_digit = dict([(i, 0) for in lst 1)
### Python version
for in lst:
from typing import List
countddigttii]+=1
The function first creates a dictionary where the keys are the integers in the list and the values are the number of times that integer appears in the list.
def catrrrat__preengrruppppepee str) <> List[str]:
result = []
current_string []]
if [nyccount_digft[[]
for i in lst)
current_depth
return False
for in paren_string:
It then checks if any integer appears more than twice in the list, and returns False 1f
if == '(": current_depth +=
current_string.
elif == ")';
if (1sstst(1-1] lst[i] for i in r1nge (en(ts)))): return True
current_depth current_string. append
else:
if current_depth result. jooin(
return False
Finally, checks if the list is sorted non-descending order, returns if so, and False otherwise
current_string)
current.string.cllarr
result
### Java version
#*# Explanation
import Java.util.r: import lava. reflecttt)
The code is implementation of function 'separate_ Battrrate_ppre__gree groups* which takes string of parentheses and returns list of strings containing the groups of parentheses.
import org.javatuples.e;
import secu.yycu
import java.math.x;
import java.io.*;
import java.util.stream.tt
result [] current_string
class Problem
public static boolean ( ArrayList<Long> lst) {
current_depth
The function initializes empty list the groups of
HashMap<Long, Long? count_digit new HashMap<Long, Long> O);
'result' to parentheses. also initializes empty list cuurrentsttring' the current group of parentheses being processed. variable wurrent__epthh is initialized to to keep track depth the current group of parentheses.
(long i lst) if tooutt.digt..conttanssyyy
store
(i))
count_digitt putt,, count_digit.get(ii ;
} else count_digft.putt/ 11);
for с paren_string:
}
The function iterates through each character 'c' in the input string paren_string".
}
for (long i lst) {
if doount_iggit.ett//e 2)
if == '(':
return false;
current_ depeth += current_string.. append(c) If the character is an opening parenthesis ((',, the function
}
}

---

increments the 'current, ' variable and appends the character to the current_string' list.
for (Character ch
current current_string) {
sb.append (ch);
} (sd
clif == ")":
current_depth - surrent_string-appendd If the character is closing parenthesis "*)', the function decrements the 'current_depti variable and appends the character the current_string list.
toString());
current_string clear();
return result;
if current_depth == 0: r"sult.append
current_string))
If current_.tring.cclerr)) 'current_ depth variable reaches it that the current group parentheses been fully processed, function joins the current_string' list into string and appends it 'result' list. The current_string" list then cleared to start processing the next group of parentheses.
### Python version
from typing import List
def sort_array(arrayy List[int]) -> List[int]:
return [] if len(array) == 0 else (array,,,,,, reverse= +arraay [-1]) % == 0)
Explanation
return result
The function returns the 'result' list containing the groups of parentheses.
The code is an implementation of sorting an array of integers. It takes a list of integers 'array' as input and returns a sorted list of integers.
### version
import java.util.x;
return [] if len(array) == 0 else (array, reverse= [array[0] +array [-1]) % 2 == 0)
import java. www.llang.reflectt.
import org.javatuples.:;
import securitc
The function first checks if the input list is empty. If it is, it returns an empty list. Otherwise, it sorts the list using the 'sorted" function. The reverse" parameter is set to 'True' if the of the first and last elements of the array is even (and the array will be sorted in descending order), otherwise, it is set to 'False' (and the array will be sorted in ascending order).
import java.math.*;
import java.
import java.uti..
class Problem [
public static ArrayList<String separatePPreedrouus(ttrriig
paren_string) {
ArrayList<String> result new ArrayList<strring>3
ArrayListsCharacter:
current_string = new ArrayListe Character> int current_depth. 8;
##* Java version
import java.util.:;
for (int i 0; sei.nsstrrnn..ennttttn. char parennstring. charAt(i);
import java. lang, reflect.
1++) {
import org.javatuples.;;
import java.security.
import java.math.*;
if (c == '() (')
import java.io.*;
current_depth += 1;
import java.util.stream.
current_string.adde } else if (c == ")')
class Problem
public static ArrayList<Long>
cifrent_deptti - 1;
tArray tArray (Arraytskorrt array) {
current_string.
if (array.size() ==
return ArrayListcLong
if (current_depth 0)
>();
else
itriinuuilder sb StringBui Ider(
boolean
(array.
get(B) Peray-get(arra))
current_string.s size ());
size() 1)) 8;

---

if (reverse) { rellers array);
integers: even and odd palindrome number counts. A palindrome is a number that reads the same forwards and backwards.
}
return array;
def is_palindrome (n):
return str(n) == str (n) [::-1]
The function defines helper function. Let's break it down step step: str(n) (n)
#*# Python version
The function first converts the input from integers to strings, because easily reverse integer by its digits. str [::-11
<insert Python program here>
Python does not allow
#*# Explanation
The code is an implementation offinsert Python program am explanation here>
It then string manipulation '[::-1]' to reverse the string.
str(n) == str(n) It then checks if the string version the integer equal its reversed string version, thereby determining whether the input integer palindrome not.
#*# Java version
<insert Java completion here>
Hence, the helper function' is "s_ppaindrome takes number and returns 'True* it is a palindrome 'False' otherwise.
C.4 Python-Java exp-Ibi-d (four-shot)
You are an helpful AI assistant who understands all programming languages and can translate between them at Can you explain what this Python program does Line Line? If Line is long too complicated, simplify it and explain what individual parts of first before explaining the whole line. The goal with the explanation,
even_paa aad__palindromeeem counteeetttcceeeeeeeoeeeeeeeeeee The counter variable even_pal odd_pal w_l.paandooee_comee are used to record the result.
htt__palindromee_oom
even_palindrome_ aelliindreeeuuut and
the line
for i in range(1, n+1):
that reader can easily rewrite the program am Java?
The loops goes through all numbers between 1 and 'n' (inclusive)
if 1&2 == and is_palindrome : @ad_palindrome_com count += elif 152 == and is_pal'ndromee (t):
### Python version
from typing import Tuple
even_palindrome_comm Within the for-loop, the program defines set of if-else statements. Let's break it down step by step: i%2 ==
+=
def even_odd _eceeodd_dplleeaee Tuple[int, int]:
_ar___ddd_pp__doeoee paa.aanaaooeeeeeec (n: int)
def is_palinde (n_palindrrmm(n))
return str (n) (tr(n)
This checks if the number is odd by checking if the mod 2 of the number has remainder of is_palindromedf
twen_palindrome count odd add_palindrome__r
Here the program uses previously defined he helper function to determine if the number is palindrome not. if i&2 == and is_ppalindrome(i): odd_ hau_palliddrmme_comm Putting together, both of above components check
for in range(1, n+1): if 192 == and is_palindromee1
):
aad__plliddron conddoomecount elif 182 and is_palindrome
even_palindrome__
number is add and palindrome, and if so, increment counter fidd_palindrmme_comm same logic applied to even palindrome numbers.
return Leven_palindrome__m count, odd_pallindrome_coont)
#*# Explanation
Hence, the if-else conditions checks each number palindrome even/ and increment respective counters.
The code impeemtnatiion of finding the number of and odd palindrome numbers between and 'n' inclusive). takes integer input and tuple of
return (even_pallindrom__comm)

---

Finally, the function returns a tuple of the two counters. Even pal pndrnne count first, odd palindrome count next.
count_digis dict([(i, 0) for i in 1st })
for i in lst:
count_diy cuut_digittii]=al
#*# Java version
The function first creates dictionary where the keys are the integers in the list and the values are the number of times that integer appears in the list.
import java.util.*;
import java. reng.eeflttt.t;
import org- org.javatuples.r;
import java.
if )ny(count.digit((]) > for in lst)
import java.math.x;
import java.io.;;
return False
import java.util.streem...
It then checks if any integer appears more than twice in the list, and returns False if so.
class Problem {
public static Pair<Long, Long>
wvenOddPalindromee(tnn n)
Long evenPaliindrmeeoons
if all(lst[i-1] <= Ist[i] for in range(1, len(lst))): return True
long oddPalindromeCoom = 8;
for (Long 1 = 1; <= 1++) {
else:
if (f % 2 == isPalindrome oddPalindrommCouu
False
Let's break down step by step: [st[f-1] lst[t]
} else if (i & == 0 &
Within the list comprehension, this checks whether the current element at index 'i' is greater or equal previous element at index 'i-1', if (ls((s[(--1] <= lst[i] for in range(1, len(lst))):
isPalindrome {
return (air.with( ovenPalindromeCoun,,
The whole expression then checks if every element is greater or equal to the element before the list. In another word, this checks if the list is sorted in non-descending order. return True
codPatiindromeCount)]
private static boolean isPalindrome (long n)
return Long. (oString()) ( StringBuilder (Long. tostring (n)) (e))_reverse() toString());
e else:
return False
If the list is sorted in non-descending order the function returns True, otherwise it returns False.
Hence, the if-else statements returns True if the list is sorted in non- descending order, and False otherwise.
UNa Python version
from typing import List
##* Java version
def is (1sttt List[int]] -> bool: count_digit dict([(i, 8) for in lst])
import java.utíl.*;
import java. lang. rellett.; import org.javatuples.r;
for in lst:
count_ digit[i]+==1
import java.security.
if any(count_digittt]] for lst):
import java.math.*;
import java.io.*;
return False
import java.util.stream..
all([st[i-1] Ist[i] i range(1, len()))))):
class Problem
public static boolean isSorted ArrayListcLong> lst)
return True
else: return False
HashiMap<Long, Long> count_digit = new HashMap<Long, ()ng>(); for (long i lst)
if ccunt_ddiitt.contaissk (i)) { count_ count_digit.ppt(( count_digit.get(11
#*H Explanation
The code is implementation of checking whether List is sorted in non-descending order and has than occurrences of any integer. It takes list of integers and boolean.
else duntt 1L);

---

for (Long 1 : lst) {
if ccontt_dggit.gett. 2)
if == '(:
current_depth += current_strings surrent_strnnggapend(c]] If the character is an opening parenthesis "("", the function increments the 'current_depth" variable and appends the character to the current_string' list.
return false;
}
}
for (int i 1; lst.size(); i++) [
if (lst.get(i 1) lst.
get(t)) { return false;
elif c 1)':
}
current_ _enth -= porrent_string.appee closing If the character parenthesis ")"s, the function decrements the 'current_depth" variable and appends the character to the current_string" list.
return true;
#*# Python version
from typing import List
if current_ result.append (** Jjoin( current_string))
def separate__paneee Batpstteeppeengrrreeeee_ grouupsspprrenssrrrnng
current_strinngcller
str) -> List[str]:
Let's break down step by step: w.joon(cureet__srring program joins the "current_string (which is represented as list) into string
result []]
current_string
current_depth
for in paren_string:
Hence, the if-statement does the following: if the puureen__eppthe variable reaches 8, it means that the current group of parentheses has been fully processed, so the function joins the peurret_ssring list into a string and appends it to the 'result' list. The current_string list is then cleared to start processing the next group of parentheses.
if == '(':
current_depti +=
surrent_string.appent
elif = ')';
current_depth surrent_string.appeeee
if current_depth
result.append("" joijo
current_string))
current__tringg.clar
return result
The function returns the 'result' list containing the groups of parentheses.
return result
и** Explanation
Java version
The implementation of function 'separate_ ceatte_aaengggruuee which takes string of parentheses and returns list of strings containing the groups of parentheses.
import java.util.*; import java. lang. reflect;; import org.javatuples.r;
import seur
import java.math.*;
import java.io.*;
result current_string []]
import java.util..trrem..
class Problem
current_depte
public static ArrayList<string
The function initializes an empty list 'result' to store the groups parentheses. It also initializes an empty List 'current_string" to store the current group of parentheses being processed. The variable 'current_deptee is initialized to 0 to keep track of the depth of the current group of parentheses.
separateearendroosstttrrrr
paren_string)
ArrayListcString> result
ArrayList<stringe())
ArrayList<Character>
current_string new ArrayListr
Character> int current_depth =
for (int i paren_string length(); f++)
for in paren_string
The function iterates through each character 'c' in the input string paren_string'.
char paren_string. charAt(1);
if == (() {

---

current_depth += 1;
input array is even. If the sum of the first and last elements of the input array is odd, the reverse parameter of the sorted function is set to False. Hence, the else statement sorts the input array. If the sum of the first and last elements of the input array is even, it is sorted with reverse=True descending order), otherwise, it is sorted with reverse=false (ascending order).
current_strae (trrent_srring.add(()) } else if (c == )) {
current_depth - 1;
current_string.add(c)
if (current_depth ==
StringBuilder sb new StringBuilderd
perrent_string.stie ()); for (Character ch ourrent_string
Hence, the entire inline if-else statement (and the return statement) returns empty array if the length of the input array zero. Otherwise, if the of the first and last element of the array even, the array is sorted with reverse parameter True. the
sb.appedd
result.add(sb. toString()) current_string. clear();
the first and last element the array is odd, the parameter is False.
##* Java version
return result;
import java.util.*;
import java. lang. reflect.*;
import org.javatuples.:;
import java.
### Python version
import java.math.";
import java.fo.*;
from typing import List
import java.util.streem..
class Problem
def anrtaaraay List[int]] List[int]:
public static ArrayListsLong>
CorttrrayyArrayyisttcooo array) {
return [] Len(array) else sartedaarra, reverse= [array[0] [-1]) 0)
(array.size() == 8)
return ArrayList<Long
>0;
else {
### Explanation
boolean reverse (array.
get(0) array.get(array
The code is implementation of function that sorts array of integers. The function takes array of integers input and returns sorted array of integers.
size() - 1)) 0; soreectioos.ssttarraeeee (reverse) collections.reverse array);
return if Len(array) else
return array;
(array, reverse= (array[0] + arraay [-1])
This statement contains
inline if-else statement. Let's break down step by step:
Python version
[] if Lan(array)
This inline if statement checks if the length the input array is zero. If the length the input array is zero, the function returns an empty array. If the length of the input array not zero, the function proceeds to else statement.
<insert Python program here>
Explanation
The code is an implementation ofkinsert Pytnon program explanat ion here>
carraay reverse= +a] [-1]) % == 0)
### Java version
This function sorts the list 'array', Let's break it down step by step: (array(0)+array(-1]) & == The reverse parameter of the sorted function is set to True if the sum of the first and last elements of the
kinsert Java completion here>

---

D Computational Resources
translator is__nd__ddddeee function ffunctoo
1)
All completion queries are made to GPT-3.5 (gpt- 3.5 (.--turbo-3010 between March June 2023. We use the Azure completion endpoint (as opposed to the OpenAI chat complction endpoint). Compute credits are provided by MIT-IBM Watson AT Lab. The average query time for single experiment (e.g. exp, Python -> Julia) takes around 10-40 minutes with one API key (of which we only used one). For Tables and 2, the total query time between 21 84 hours. The experiments for CodeGeen211 are conducted on BU SCC and MIT Satori clus- ters with NVIDIA RTX V100, A100, and A6000, Each single experiment takes 12-24 hours so the whole compute time for Tables 1 and 2 comple- tions is around 72- 144 GPU days. Experiments for CodeGen2-168 and Jlama2Cooeennstrrr:::4444 are done through IBM hosted model inference APIs. Query time is about 2-3X to that of GPT-3.5. Code exccution/evallutioo is done locally on the same MacBook Pro 2015.
in function False
else: if we are not in function,
check if Line is signature
(en(L.ttrip() != 8:
if translator.
is us_ffuctiionsignan signttuurnss
in_function True
else:
lines = lines[:i]
break
truncated_.ompletiion "\n" .oinjonioinnss)
return truncated_ complettiioo
additional implementation get_ function_name and additional stops property. This function is used deal with languages that are difficult to determine the end of functions without using lexers (hard to implement of
requires
this case, we put additional stop tokens that indicate for certain that the line is not inside function (usually \n with every single letter in the alphabet along with special symbols such An#, \n!.) We additionally add the main completion function's name to make sure the completion doesn't call the function itself at the base level.
E Language Specific Stop Token and Post-Processing
To stop model generations and extract the rele- vant code needed for the problem, use few language-specific tokens. In the case of OpenAl APIs, such token size limit is In order to accom- modate multi-function and remove irrelevant gener- ations, we modify the stop tokens from MultiPL-E and add post processing to some languages. There are two types of post-processing we use to truncate completions:
E.1 JavaScript
original stops: console. log
Anfunction, /*, 11,
modified stops: \n}. /*, 11-
console. log. Instead of stopping at the beginning of the next irrelevant code segment, we stop right before where the func- tion ends, and add closing bracket back with test strings. This speeds up experiments by removing all possible additional un-expected completions.
truncate_ss truncate_after. aateee_ffer___ee www.ffunnnnnnnneeee ends re- quires two function implementations from each language translator: _s__nd_o_ ss___e____ffuciiii______________ function and is wwwctiinnsiigaattteeu
The function works by greedily search the completion line-by- line. As soon as we encounter end of function, start looking for the next con-empty line. We remove the rest the next non-empty line is not function signature. Otherwise, keep the line and continue. Below is the Python implementation srunnate_aater_funntiioo...
E.2
C++
original stops: An}.
no modification.
E.3 Java
original stops: An
An
def Souncate_afterr Sruncate_atter_funtttio..cm ott.aeeeeeeuuuttttoooeoosscsoooo translator):
Lines completiinn.split("""") in_ffuction True
modified
stops:
public static void main. ###
for 1, 1 in enumerate (lines):
\n). Nested functions are illegal in Java. order to allow models to generate multiple functions, instead of stopping at the end
if ii_ function:
# if encounter end of functioon, start looking out for next non empty line

---

function, stop right before the main function, or end of class.
E.10 R
original stops: An#, Ansse
E.4 TypeScript
modified stops: Anj.
Same reason as
original stops: console. log
Anfunction, /*,
11,
JavaScript.
E.11
Rust
modified stops:
\n3.
/*,
11.
original stops: An}.
console. log. Same reason JavaScript
no modification.
E,5
PHP
E.12
Scala
original stops: \nfunction, An?>, An//.
original stops: \n
An.
An#
no modification.
modified stops: \n}, \n?>, \n//, An#. Same reason as JavaScript
E.13
Swift
original stops: \n}.
E.6 Ruby
no modification.
original stops: \n|n
Anclass, Andef, An#,
E.14
Bash
original stops: An}
modified stops: \nclass, Andef, \n#, Anputs. The original stop \n is problem- atic because it prematurely stops translation as soon as the source program has an extra empty line the program. Additionally, Anputs to ensure we don't have extra generations (self- calls) at the end.
modified stops: An#, \nAnswer, \necho, \n..< Although nested functions are techni- cally allowed, it is quite conventional to write helper functions in separate function. We also post-processing truncate additional unwanted generations.
post-processing:
post- post-processing:
E.15
Lua
E.7
C#
original stops: \nlocal, \nfunction. \n--.In|n
original stops: An
modified
stops:
modified stops: \n--, \n#, \nend. The original stop \n\n is problematic because it prematurely stops translation as soon as the source program has an extra empty line the program. We add base indentation level \nend to truncate after the function ends.
public static void Main, static void Main, \n#, \n). Same reason as Java. Although nested/local function is allowed in C#, this increases the variety of generations which can be accepted by unit tests.
E.16 Racket
E.8 Go
original stops: \n An#l, An;, An(
original stops: Anfunc, struct, \n/
modified stops: (ndefiine \n#l, An;.
modified stops: Anfunc, struct, \n//
\n(
\n]. Same reason JavaScript.
additional stops: An#", Ancce and "\n" with all letters in alphabet.
E.9 Perl
original stops:
\nsub, \n#, \n\n.
post-processing
modified stops: \nsub, \n#, \n}. Same reason as Ruby.
truncate_ Sittcate_aftereeee afweeefadrdddiinaaaaa s stops

---

E.17 Julia
The function takes a list of strings as input and returns the string with the maximum number of unique characters.
original stops: \nfunction, \nmacro, \n)n,
gorrdds key lambda x: (seen(set (x)), x)) [e]
modified stops: Anend, An#. Same as JavaScript: Anend is stricter way of stop- ping multi-function generation since Julia al- lows nested functions.
This statement sorts the input list words'. Let's break it down step by
step:
set(x)
The 'set' function returns set of unique characters in the string 'x' len(se((x))
E.18 D
original stops: \n\n, Anvoid, \nbool, Anint
The 'len' function returns the number of unique characters in the string 'x'. -len(set(x))
modified stops: Anvoid, \nbool, \nint, \n}. More strict stopping at the end of func- tion, and remove problematic \n)n
The :-r sign used sort the list in descending order of the number of unique characters.
E.19
Explanation Qualitative Analysis
The parameter is used list in ascending order of the string itself.
the
Hence, the 'key' parameter of the sorted function set to lambda function that returns a tuple of values: the negative length the set of unique characters the string itself. The 'sorted'
и*я Python version
def find_max (words: List[str]) -> str: return sorted (words, key lambda x (-len(set(x)), [e]
the string and
functian sorts the input 1ist based on this tuple. The '[0]* at the end of the statement returns the first element the sorted list, which is the string with the maximum number of unique characters.
### Explanation (exp)
This function receives list of strings and returns the string with the highest number of unique characters, and if there are ties, the string that first in lexicographic order.
### Explanation (exp-lbl)
Above is an example explanation of a program in Humaneval Laumaaneval_15__0 find_max).. With exp. the explanation is very high-level and does go into details with the Python implemen- tation. With exp-lbl, each line of the program is referenced along with an explanation. However, we can see that in the second paragraph exp-Ibl explains that words sorted "by fewest unique char- acters first" when it should be the other way. This led to the wrong translation where the negative sign is ignored. On the other hand, with exp-Ibl-d, com- plicated line like ones above is decomposed first, explained separately and then combined together. This particular explanation emphasizes the negative sign and improves translation pass rate.
The code an implementation of finding the longest words that has the fewest unique characters, It takes a list of strings 'words' as input and returns the longest string in the list that has the fewest unique characters.
list of
sarted swrrdd (words, (x)), [0]
key lambda x: (set
The function first sorts the list of words using the 'sorted" function. The 'key' parameter is set to lambda function that returns tuple of two values: the negative length of the set of characters in the word (to sort by fewest unique characters
first), and the word itself (to break ties by alphabetical order).
The '[0]' at the end of the line returns the first (i.e., longest) word in the sorted list.
F
How did we select with few-shot programs and write explanations
To be consistent across trials, and due the fact that not every program has gold translations, we use fixed few-shot our main experiments. In this case, the few-shot examples we pick is crucial to the performance: they need to be representative
#*# Explanation lbl-d
The code an implementation of function that finds the word with the maximum number of unique characters.

---

the dataset characteristics and contain features that may demonstrate the usefulness of explanations. Once we select the few-shot programs, we simply use the model to generate zero-shot explanations, and modify for correctness and structural prefer- ences (c.g. code lines followed by explanations with exp-lbl).
semantically complicated line. This is because the model has to spend paragraphs explaining these lines, and often produces confusing/wrong state- ments. Many of the remaining 17 assertion errors (semantic errors) in four-shot exp-Ibl are due to in-corretiinsufficient explanation). Section E.19 shows an example program.
These emrors, however, can be effectively miti- gated if they were further decomposed into smaller chunks of code statements. Due to the inherent tree-like structure (AST) of program, asking mod- els to decompose is another way of learning fine- grain parsing of the code. Surprisingly, models like GPT-3.5 is able to parsing very well. Hence, we developed our third explanation method exp-lbl- d: we ask the model to explain line-by-line, and breakdown the line into smaller parts if the line is too long or complicated. To demonstrate the use- fulness of such method, we pick program that is short but contains a long and complicated line (humaneval_ 2er__sor__arrre
Our "development" translation direction is Python - Java. We quickly notice that GPT-3.5 is bad translating nested functions, which occur many times in the canonical solutions in HumanEval. Compounded with the fact that the original MulúPL-E stop tokens do not allow models to generate multiple functions and that Java does not allow nested functions, the only way for GPT-3.5 to generate correct translation of these program to use lambda expressions, which can be extremely convoluted if the nested function is long. Therefore, after loosening the constraint of single function only (see Java section Apx E), we decide to use the first few-shot example numaneval__07__een__oor woooddppalinddooo indr ome demonstration show the model how to translate these type of functions (by adding private helper function after the main function).
See E.19 for the result of explanations generated with few-shots across 3 explanation methods. With exp-lbl-d four- shot, we find the explanations generated for almost all 17 previously failing programs (in exp-Ibl) to be correct, and the remaining 12 assertion errors are all results of wrong program generation given correct explanations.
Second and third few-shot examples were selected quite arbitrarily. The only criteria we had in-mind was that these programs need to be somewhat difficult, on the longer side length-wise, and that the semantics of the program is not immediately clear after looking at the function name first sentence explanation of the function. Lastly, the programs also should not be close to each other the problem sequence (just in case when designing the dataset, (Chen et al., 2021) decided to stress-test different aspect of code-generation in batches). Hence, we picked numaneval_126__1 1aap____2___isss____ is_sorted and humaneval_l_seeaeee hat_____pprrtteeee pettrrrttt__arreggruuuss
G Python-to-X Detailed Result Investigation
All experiments detailed here are conducted with GPT--3.5 unless otherwise specified.
G.1 Removing target language specific information in zero-shot explanation
#*# Python verston
def strlen(string: str) int: return len(string)
Last example is often the most influential due to the proximity to test examples (Lu et al., 2022). We decide on this program after careful error anal- ysis of exp and exp-Ibl. With few-shot exp-Ibl, we found the explanations to be of great qualities already: the explanation chunks the model into several semantically independent segments and ex- plain each of them separately. We notice that the quality of the explanations (through manual in- spection) are worse when the segments are longer or more complicated. In many cases, these coin- cides with programs that contain one extremely
### Explanation
This function takes string as input and returns the length of the string. It does this using the built-in 'len function in Python. The function is defined with the name 'strlen' and takes single argument, which is string. The return type of the function is integer.
rewrite this program Java, you would need define function with same name and argument type,

---

Java
Lua
G.2
Four-shot explanation variations
To observe the difference between the Python- to-Java four-shot explanation, four-shot target language-specifie explanation, and re-using zero- shot larget-language specific explanation in four- shot translation, we compare their Python-to-X. translation performance in 18 target languages (Ta- ble 9). Just by observing best trials across lan- guage directions in Table 9, exp (java) wins 6/1 directions, whereas exp (tgt-specific) and exp (zero-shot) each win 5/18 and 11/18 directions. 2-tail paired t-test indicates that both alternative tri- als are significantly different from re-using 4 shot examples from Java (Ptgt_specific = 0.036) and Pzero-shot 0.024)). Four-shot explanations worse than zero-shot generated explanation. This is intuitive because the explanations with exp method in the zero-shot setting are good enough. By incorporating mostly its own explanation in a few-shot setting, the model is not obtaining more information, but restricting its potential to generate diverse types of explanation.
exp
0.829
0.593
exp (remove)
0.829
0.592
exp-lbl
0.825
0.615
exp-Ibl (remove)
0.825
0.615
exp-Ibl-d
0.846
0.595
exp-Ibl-d (remove)
0.84
0.595
Table 7: Comparing results for different explanation types before and after removing target-ppecific informa- tion zero-shot setting
use the ')ength() method of the "String' class to get the length of the string. The return type would also need to be changed to 'int'. Here example of what the Java code might look Like:
# aavaa
public static int strlen(String string)
return string. (ength())
G.3 heuristically selected explanations
For more fair comparisons, we include here the heuristically selected explanations with respective baselines. Since exp selection zero-shot explanations,
Above is an example of what target specific in- formation in generated explanations looks like.
done over compare exp (logprob) against exp (zero-shot). There is still a mean im- provement of 0.73% with standard deviation of 1.8%.
Since different types of explanations generate different types of kind of information, we remove all of them in Python-to-Java direction to disen- tangle the effects of target-specific information vs the level of detail in the explanation. The removal is done through script, where we remove from the first sentence that mentions anything about the target language.
G.4 Error types breakdown in Python-to-X
For cach program, addition to determine unit tests pass rate, we also use string matching on stderr and stdout to categorize the error type. In order to generalize across different languages, we group the errors into the following types:
Here, we compare the pass @ass@1 before and af- ter removing such information for all explanation types. As seen, there is only slight decrease in the performance. When translating into low-resource languages, the occurrence of target-specific infor- mation is much less frequent, leading much less difference between regular explanations vs. expla- nations with target information removed.
Type Error includes all errors related in- teractions between variables with the wrong types. For example, in Python, floats cannot be used to index list, and a string cannot be used to multiply with another string.
Undeclared Error includes all errors calling methods or variables that do not exists. It ranges from undeclared variable, to unable to find equivalent built-in function such as string. swapcase () in Python
In addition, we compare using target- independent explanations (exp (java)) vs. target-dependent explanations (exp (tgt-speciffc)) in Table 8. Target specific explanations do impact performance significantly zero-shot (p = 0.090). Target- explanations tend to decrease performance in loworreesource languages.
Assertion Error catches all cases where the function output does not match the expected

---

High
Medium
Low
Exuenely-Loo
Iniol
App
6x
phpp
pl
4h
lus
eap (joca)
84.8
82.1
82.9
65.1
77.0
78.3
83.6
45.7
68.2
46.9
74.8
14.7
70.5
55.4
60.0
41.3
70.5
44.
exp (ig-speolic))
65.7
80.7
82.
83.7
76.4
76.4
85.
45.0
669
47.5
75.0
74.6
70.3
53.2
59.3
19.5
71.2
42.
Table 8: Comparing zero-shot target-independent explanation vs target-specific explanations across 18 languages. 2-tail paired test p 0.090
High
Medium
Low
Extemely-Low
Infol
upp
jv
6x
phpp
19
pl.
16.
sh
lua
rk
csp (jara)
$7.3
77.8
92.1
58.4
76.4
80.4
83.3
51.4
73.2
55.6
73.0
13.7
61.1
71.1
67.9
44.7
09.4
45.9
esp (igt-sweilic))
65.1
79.0
82.1
$7.4
78.2
80.4
828
52.7
73.1
891
76.4
75.7
68.7
72.9
6K.6
45.3
69.2
68.7
12cro shoo)
86.1
80.5
81.6
85.7
78.4
80.4
8337
52.7
73.1
57.5
76.8
75.8
70.0
73.8
68.8
47.0
08.3
43.5
Table 9: Comparing different four-shot translation with different explanations: Python-Java explanation, target- specific 4 explanation and target-specific shot explanation across 18 languages. 2-tail paired t-test between first and second row is 0.036 and between first and third row is p 0.024
output. This indicates that the program runs, but is not functionally the same as the source.
translation step to generate appropriate method call. Surprisingly, there is no sign of decrease se- mantic error. This is likely due to the fact that by resolving syntactic errors, those programs switched to having semantic errors. In Fig 6 we look-into this phenomenon specifically.
Runtime Error generally includes all errors that do not occur for every unit tests. For instance, index out of bound error may only occur with input of long lists.
No significant difference in four-shot Errors seem to be distributed very similarly across all trials. There are two exceptions. First, in high- resource target languages, other syntax errors seem to drop significantly in exp-lbi-d than the other expl axplanations, which both contain than direct baseline. In extremely low-resource target language, there also seem to be a somewhat significant drop in other syntax errors.
Other Syntax Errors includes all other type of errors not captured by specific groups from above.
Unhelpful includes cases where the generated program contain exclusively comments like TODO, Your Code Here.
error
For better generalization, we also combine as- sertion error and unhelpful into semantic error and Type, Undeclared and other syntax errors into synfactie error. Here are some of our main conclusions:
G.5 Error conversion between direct translation and with explanations
To observe the program status with and without using explanations, we track each problem's 's status in direct and explanation trials. In Fig 6, we plot direct status on the x-axis and corresponding status with explanation on the y-axis. Here are some key take-aways:
Less syntax error across the board in zero-shot In general we see decrease of syntactic error across all target language resource level (Fig 5). Specifically, there significant decrease in helpful generations in trials with explanations. This similar to effects of having few shot ex- amples Min et al. (2022b), except in this case we do not actually provide the actual format of target translation. Other than reducing unhelpful gener- ations, self-generater explanations also decrease undeclared and type error (more so in higher resource directions). This intuitive because as model reasons through program explanations, may generated sequences that specify variable type or specific methods used within source program, which term provides more information for the
More detailed explanations decrease semantic error rate In the top two rows of Fig 6, we can observe the differences between three explanation methods. In zero-shot setting (row I), we can see that exp-Ibl converts more semantic errors in di- rect to pass, and slightly semantic error, which are both indication of im- provements. In four-shot setting (row 2), with more detailed explanations, we see consistent decrease in -ssmantiitticc error (cxplanation misleading translation), semantic error->pass but an increase
syntactic errors to

---

High
Mediuma
Low
Extremely-LLom
tpp
jo
phop
gn
pl
cap (ccro-hhot)
$6.1
80.5
81.6
85.7
78.4
80.4
83.7
52.7
73.1
57.5
76.8
15.8
70.0
73.8
68.8
47.0
68.3
43.5
cxp"
$5.9
80.8
65
88.3
78.6
61.4
85.1
52.3
71.5
57.3
24.5
76.
71.3
73.1
67.9
46.3
72.9
50.7
It
57.6
K2.4
85.3
87.1
$1.5
90.1
86.7
30.5
72
36.9
74.9
73.9
70.1
71.9
68.6
45.6
72.1
66.2
esp-tt INi-d"
87.8
83.9
86.5
58.3
80.7
84.3
30.6
70.9
26.3
75.5
70.9
72.5
69.4
45.3
71.1
45.1
87.8
8533
72.3
69.4
Table 10: Four-shot translation with explanations respective heuristically selected explanations. exp* represent heuristically selected exp using logprob. exp-Ibl-d represent heuristically selected exp-lbl-d using frag
Differant types of Erron@1 Pythan-X (Toa: 0-shct, Batiom: 4-shat)
dee
High
Medium
Low
Extremay Low
0.5
brondr
0.2
02-
0.1
Semantic Erirr
Syntactic Enrer
Semanti Enor
Synladio Error
Semands: Enar
Syntecii Errr
Semardic Eror
Syrtacii Enor
0.20
04
0.c
0.15
0.25
0.18
0.10
0.1
0.9
0.0
Semmentc Eror
Synlecii Error
Somantio Enor
Eymlaxti Gror
Bomanis Eror
Syntasii Errrr
Semurdic Eror Syntacto Enor
Different types of Emoong in PythonX (Top: 0-shot, Bottmm 4-snct)
0.20
hight
Madou
UMM
Ediomdd Lose
04-
6.10
500
0.1
B
20
Toue Sior
Alverion fou mostee Eino
Lnteopie
Dver Austatin Goor Bieiine Sinse Lxtieinn
Sydte Ausectin Cim
Earthoa firu
Undecieed
Spinn
Tipe
Undeeceene
Type
Undoutooen
Thee Unddectond
Syste Alturboo
0.0
1.5-
2.10
01
31
Toe Orne
Undettond Gone
Ooee Coss
Ausition Cose
thndime Come
Unbesnnd
Tyme Bres
Unveeddene Stret
Other Sprina Dees
Type Cote
Unmendoned Cora
Other Sypter Cane
Carmine Some Ualovsta
Oyen Boes
Good Ooler dscidioa Biee
Busioon Cone Lanaiped
Unoo docod
Figure 5: Python-to-X X translation Errrr@ rate (of different types) across different target language resources. The top plot displays semantic syntactic while the bottom displays detailed error types. We string simple string matching to determine error type and use the same formal as Pass to calculate Error@

---

Python-X program
from baseline explanation (Top: 0-shot, Bottom: 4-shot)
O-shot statuo explenetion
direct exp
direct exp-Ibi
direct -> exp-ibl-d
15
Semantic Error
2.4
8.6
3.9
10
Semantic Error
2.4
8.2
4.3
10
Semantic Error
2.5
8.3
41
10
5
5
Syntactic
Syntactic
Pass
Semantic Enar
Syntactic Error
pass
Semantic Error
Syntactic Error
pass
Semantic Enror
Syntactic Error
4-shot statuse explenation
10
Semantic Error
3.9
9.3
1.5
Semantic Error
3.1
9.2
1.6
Semantic Error
2.2
15
5
Syntactic
Syntactic
Semantic Emar
Syntactic Error
Semantic Error
Syntactic Error
Semantic Error
Syntactic Error
statupg1 direct
statugg1 direct
statusg1 direct
Python-X program execution status conversion from baseline to explanation (Top: 0-shot, Bottom: 4 4-shot)
0-4hot statungi W explanation
High
Medium
Low
Extremely-Low
Pass
76
3.5
Pass 64
3.4
4.5
15
Pass
55
2.2
9.9
Pass
42
4.1
6.4
10
-5
Syntactic Emor
0.94
0.69
2.8
Syntactic Error
1.1
1.5
14
Syntactic Erao
1.6
0.68
15
Syntactic Erro
2.5
3.4
23
Pass
Semantic Emror Syntacti Error
Pass
Semantic Error Syntacti Error
Pass
Semantic Error Syntactic Error
Pess
Semantic Error
Syntactic Error
stanueg1 w/ explanation
15
15
10
20
10
10
15
Semantic Error-
3.2
7.9
0.59
Semantic Error
2.5
7.3
1.3
Semantic Error
3.5
11
3.2
Semantic Erro
2.9
9.6
2.8
10
Syntactic
Syntactic
1.4
Syntactic
Syntactic
2.5
5
Pass
Semantic Emrrr Syntacicc Emrr
Pass
Somantic Error Syntactic Error
Pass
Semantic Error Syntacicc Error
Pass
Semantic Error Syntactic Erro
Matusg1 direct
Malusg1 direct
Malusg1 direct
status@1 direct
Figure 6: Python-to-X translation status conversion between baseline and explanation. X-axis indicate baseline status and y-axis indicate translation status with explanations. In the top figure, results are aggregated across target languages, In the bottom figure, results are aggregated across exp, exp-Ibl, and exp-Ibl-d

---

of pass +syntacie error. These all indicate that a more detailed explanation indeed decreases the amount of semantic errors
program in free-form natural language, and ab- stract explanation provide the least constraints on generating a diverse set of programs (better recall)
Higher target language resource, proportion- ally more improvement with explanation, less misleads In the bottom two rows of Fig 6, we can see the effectiveness of explanation across target languages of different resource level. In zero-shot majority of the improved cases
Four-shot better/detailed explanation leads bet- ter coverage In the bottom 4 row of Table 11, we can see that either direct translation, exp-Ibl-d, or heuristically selected explanation wins. Indicating that with good quality explanation, obtain improvements in few-shot setting.
(row 3), (not pass->pass) come from improving syntactic error. However, if we count the improvements of syntactic error->semantic error, the effeet be- comes similar. Proportionally, high-resource bene- fits the most in improving syntactic errors. In lower resources, there's proportionally more chance of explanation misleading the translation (pass->no pass. In four-shot (row 4), the effect of explanation is much smaller (pass->no pass or no pass->pass)
still
G.8 Python-to-X translation performance NL-to-Code performance
To investigate whether NL-to-code performance correlates python-to-X translation performance, compare our zero-shot results with Cassano al. (2022) with can see a strong correlation between the two. On top of direct translation, explanations (best explana- tion for each target language) improve translations (absolute difference) uniformly across source lan- guages, and higher relative improvements in lan- guages which are hard for NL-to-code task (lower- resource languages)
In Fig 8, we
G.6 Translation pass rate with different program lengths
Typically, generating longer programs is harder. We look into the success rate each our trials with respect to the source program length to observe there are any patterns. We find that explanation affects translation across length uniformly, with better performance in high-resoen resoures long pro- grams. In the top left box plot of Fig 7, we can more significant improvement for longest set of programs with explanation. This effect damp- ens slightly as we translate lower-resource lan- guages.
G.9 Python-to-X for Opensource models
H X-to-X Opensource Model Results
All experiments detailed here are conducted with GPT-3.5 unless otherwise specified.
There are still improvements with self-generated explanations across most directions In weaker opensource models CodeGen2-11 improves more consistently (than 16B) using self ceff-generated explanations (baseline is the worst in all X-to-X directions, and in 17/18 Python-to-X directions), much as 300%+ improvement (12, Lua->Python In Aython-RRub model with explanations obtains a pass rate of 5.9, while baseline does not generate any single correct trans- lation (pass rate of 0). CodeGen2-16B shows weaker results, with baseline outperforming 10/18 directions in Table and 2. Perhaps has weaker alignment between natural language and programming language, resulting in worse expla- nations generated for each problem. The major- ity of errors from translation with explanation are syntactic. For Llama22Codelnstrrct--334 there are consistent absolute improvements of 5%-10% and maximum relative improvements of up to 40% (Java - C++ in table 2).
G.7 Python-to-X translation Pass@ @ 10 (n=20)
the main result table we presented Pass@ results with GPT-3.5. For convenience and cost, we also report @ 10 results from the same trial (Table II), but note that for more accurate and optimal estimation @ 10 should be estimated with n 200 and 0.8 (Cassano et al., 2022).
Python-+JavaSeripn
Less relative improvements than pass@1 Over- all, from Table 11, we can see less improvements in @ 10. This is reasonable because ultimately adding explanation restricts the generation space and lowers the diversity of the output generations. Still, we see consistent improvements with expla- nations.
Zero-shot exp provide best coverage In the top rows of Table 11, we can see exp outperforms the rest in the most directions (9/19). This proba- bly because there are countless ways explaining

---

Pass over program cifferent lengths (Top: 0-shot, Bottom: 4-shot)
dreck
High
Modium
Low
Extiamaly Low
0.0
sap-itidd
0.5
0.7
6.6
08-
07.
0.0
0:4
0.2
100
0.80
0.9
0.8-
0.6
08-
0.6
0.4
Figure 7: Python-to-X translation pass @ rate across programs binned in quartiles. On x-axis, 25=shortest, 100=longest programs
Ros
High
Modium
Low:
Esteemely- Law
Trial
js
phpp
:b
sh
luz
rit
jl
diecctO))
40.3
83.3
83.1
$4.4
75.8
83.7
85.4
50.5
67.2
51.9
92.2
75,3
75.2
70.3
66.8
45.3
69.6
29.1
exget)
89.
87.3
88.6
58.7
50
81.8
87.5
52.5
74.4
57.4
81.3
90.8
78.3
71.9
70.2
58.1
25.8
$1.5
exp-Ibtr)
88.6
87.8
87.8
53.3
70.4
83.8
87.4
50.5
73.4
59.2
83.4
30.5
S0.6
72.6
21.6
52.2
75.5
$1.7
exp-Ibl-dt))
87.7
83.7
87.9
58.3
78.7
85.4
85.9
49.1
71.4
55.8
82.3
78.9
81.7
67.9
49.3
73.8
$0.4
direct (1)
92
88.4
38.6
90.9
64
86
86.1
56.9
73,9
65.1
84.7
$1.2
79.6
83.8
73.7
59.7
71.5
52.5
(4)
90.3
81.2
86
55.9
78.7
85.
85.4
56.5
70.8
61.1
82.4
50.5
78.4
82.3
25.1
$7.6
24.5
$2.9
19
85.3
$4.5
86
77.9
exp-Ibl-d(44)
86.3
83.9
61.6
(d)"
89.8
85.8
90.1
83.2
86.6
88.1
61.8
91.4
82.6
02.5
73.1
51.9
(4)-
89.8
84.9
86.5
Table 11: Translation pass @ 10 from Python to X. indicate heuristically selected explanations (Section 3.4).
Parenthesis in trial indicates of shots. Res-target language resource level. Best within same shot setting (no heuristics) underscored and overall best bold
Res
High
Medium
Low
Exteemlyy Low
j+ epp
php
20
1C XWV
j
4.6
6,4
2.6
2.4
3.8
expeltl
3.3
exp-Ibi-d
13.7
11.2
7.9
4.7
CudeGGen2 explasations
15.5
11.2
16.4
0.1
3.7
17.2
9.5
0.1
exp-Ibi-d
4.9
12
9.5
7.2
CodeCien2
48.4
41.2
46
20.6
29.6
22
11.7
14.4
expelt)
24.9
exp-tbi-d
43.2
38.4
37.6
47.4
32.6
30.5
24.3
24.8
Llama2Cooetsssoeet
64
52.1
55.5
45.6
30.8
43.5
26.5
expeli)
exp-lbi-d
60.4
69.3
42.6
46.1
26
23.7
85.1
Table 12: Translation p from Python to (0-shots) with open source models. Best within same the same model bold. Between CodeGen2-1B and CodeGen2-1B with explanation from ChatGPT, best amongst using same model's explanation is underlined.

---

CodeCien-188
1odeGen2-168
Resource
Type
stc-tgt
direet
exp
exp-ibl
exp-Ibl-d
direct
exp
exp-Ibl
exp-lbl-d
direct
exp
exp-Ibl
exp-Ibled
D-D
py-js
4.6
14,9
13.9
15.1
48.4
45.1
45.1
43.1
69.9
69.3
69.4
High-to High
D.S
jy
7.8
11.2
9.6
10.1
47.3
44.3
42
40
67.1
69.1
70.4
66
9p- py
8.3
13,9
12.5
55.1
84.4
86.1
S-S
cpp
14.2
15.6
14.8
15,9
52.7
49.2
47.7
46.9
43.3
60
60.3
59.2
D-D
0
0
0
0
2.1
2
1.7
21.3
26.6
23.7
19
my -d
7.9
High-to-ExiLLow
S4D
app loa
8.9
11.8
10
9.8
33.6
35.2
33,2
33.5
34.8
4.3
38.9
36.7
je jl
3.2
4,9
3.9
4.4
26.4
27.9
25.8
25.9
$1.5
62.2
58.1
py
1.9
47.8
46.7
D.S
rit-jv
7.9
7.6
8.1
7.9
28.4
27.8
24.7
24.8
61.2
59.4
58.2
57.3
ExiLoow-to-High
S-D
jl-3s
3.7
6.9
3.9
5.1
40.1
39.7
40
42.9
55.1
67.7
66.9
68.8
S-8
d-cpp
11.7
12.7
12.6
14.1
53.1
47.6
47.1
50.4
65.3
59.8
D.D
lua rkt
0
0
0
0
0.8
0.9
1
0.9
18.7
23.5
19.4
18
D-S
fk1-
2
2.1
1.5
0.8
13.1
14.6
14.6
14.4
52.4
58.3
50.2
47.8
ExiLow to ExiLow
S.D
lua
7.7
8.2
6.3
7.2
29.3
28.1
28.8
28.9
27.6
35.3
31.1
S-8
3l-d
0
0.007
0
10.1
7.8
5.4
5.3
24.9
26.5
28.2
28.3
Table 13: O-Shot translation pass @ between 16 different pairs of languages for open source models. Resource
indicates the language resource levels of the source and target. Type indicates the source and target language typing characteristics (DISadynamicallysatcaaly typed). The best runs within the model bold.
Better explanation leads to better translation
even in smaller model cases. We compare CodeGen2-1B performance given self-generated explanation vs. GPT-3.5 generated explanation and see that the better explanation outperforms self- generated explanation in 12/18 Python-to-X direc- tions, with maximum improvement of 400%+ in Python JavaScript.
In smaller/weaker models, detailed explanations (exp-Ibl or exp-Ibl-d) do not improve as much as exp does Often this is due lower quality explanations generated when the model is asked to do something it is not capable of. The line-by- line explanations often lead to repetitive content when source programs contain several repetitive lines (library import in C++ Python direction, with CodeGen2-1BB
ChatgPTiar ToopT(pppaao 5 s-turo (301) (1801)yPooo Pytron Transiation passog (o=20)
NL-Code vs. translation/exp improvement
0.8
06-
/'2/ML-Cade, relefive improv]]-0:25
Explanation's effectiveness at improving down- stream translation is transferable between mod- els Explanations that lead to higher pass rates in GPT-3.5 also tend to lead to higher success rates in CodeGen2-1b.. we compare Table 12 exp. exp-Ibl, exp-Ibl-d trials with CodeGen2-1b with GPT-3.5 against the same three rows in Table 1, we see the best explanation type in cach translation direction (e.g. Python - TyypeScript) between two models. This an indication of the "robustness" of the explanations.
0.-
0.0
M22/N codo, improvejm0.01
Codex)davinc)) NL-Code pasa@1m==0)
Figure 8: NL-to-code generation vs. Python-to-X trans- lation performance (zero-shot)
typically the same
Explanation Improvement correlation with problem difficulty
To understand whether self eeff-genertted explanation improve more difficult problems (or vice versa), we

---

Aksoluto Improvemnt
Relaive Incovemeent
Alternative latent language guidance
Ulmuscoonn out
In addition to asking the model to generate expla- nations, we experimented with various forms latent languages (in the order of more structurally formal to more free-form natural language). We report here their pass @ (n=1) for Python-Java
Optidlent los
Aprosdg0
Cruon fodeeoo
Pivot language: Instead of generating target program language directly, we also asked the program to translate to pivot language and then translate to the target language. For ini- tial experiment, we take the first generation from direct translation to the pivot language as intermediate step regardless of their accu- racy. pass +e 0.732, pass@ Bash 0.81, pass@ i = 0.703. More experiments in the next section.
Figure 9: X-to-X translation with GPT-3.5 (pass@) I, zero-shot) improvements (best explanation over base- line) across models grouped by source-target resource level
plot per-problem direct pass@1 1 and whether exp improves over direct (Fig 4). direct pass@) rate serves as a good approximation of how difficult the problem given the model. In main text we discuss that the exp improves difficult problems more often than easy problems. For easy problems (the right-most column), explanations can often decrease performance. Perhaps this is result of redundant or inconsistent information leading to confusion. This indicates that potential way to improve performance further is to automatically pick the difficult problems to explain.
Pseudocode: An intermediate form of pro- gram sketch described with a mix of mathe- matical operations and natural language. To ensure the format of the pseudocode, we prompt with W algorithm) ) and use (algorith)) stop token. pass@l 0.861
CoT: In Chain-of-thought (CoT) prompting, we break down the the input program space and translate each sub-components before combining all results together whole. In the decomposition phase, try decomposing through model's perception of "steps" within algorithm, as well as programmatically ex- tracting function calls within source programs that are often hard to translate (especially low resource languages) pass@l 1 = 0.731
To show that such strategy works, we assume ac- cess oracle metric (direct pass @ 1) and leverage cached generations from direct and exp trans- lations. For each problem, if the direct pass @ rate smaller than threshold (i.e. difficult prob- lem), we use explanation, otherwise we use direct translation. We repeat this for all 36 translation directions in Python-to-X and X-to-X and present full results in Table 14. Immediately we can see that 1) low-resource languages typically require more explanations. 2) select almost always out- perform direct and exp (only lost in case with D++C++). In best case scenario (Racket -+ Julia), we see as much as 9.6% relative improvement over exp with select, while explaining less than half of the problems.
Steps: ordered list of natural language steps describing major steps of the program follow- ing work by Jiang al. (2023). pass@l 0.824
Summary: free-form natural language sen- tences summary of what the program does. pass@1 = 0.854
This is however still impractical for inference during test time. Having to approximate hardness through direct translation requires more computa- tion than generating single explanation. Ideally, one could build classifiers or use heuristics to se- lect programs to translate. We leave this for future directions.
Gold summary: We the original hu- man written docstring instructions (from Hu- manEval) as gold summaries for the program and ask the model to translate given the pro- gram and summary. pass@1 0.813

---

direction
# exp
total
direct
select
d-lua
48
116
68.4 4
69.4
5.6
py-d
100
156
42
44.4
45.4
py-sh
115
158
47.9
55.4
55.8
0.4
cpp-py
19
149
92.3
90.2
93.6
1.3
lua-py
22
144
89.5
85.9
89.9
0.4
py-rb
47
161
78.3
78.3
81.7
3.4
py-scala
77
160
63.7
74.7
75.8
1.1
py-swift
69
161
64.4
70.5
71.4
0,9
pу-ts
47
159
78.9
85.1
85.9
0.8
рy-epp
49
161
76.6
82.1
83.1
java-cpp
43
155
77
79.8
80.9
1.1
py-jl
71
159
61.6
70.5
71.2
0.7
sh-java
39
138
71.7
73.9
76.8
2.9
jl-d
90
137
41.6
43.4
43.7
0.3
rkt-java
58
136
65.9
77.1
82.6
5,5
js-rkt
117
154
30.2
41.9
44.3
2.4
rkt-jl
67
137
63.3
64.3
70.5
6.2
java-ji
72
154
60.2
75.4
76.1
0.7
py-go
96
154
42.4
45.7
47.5
1.8
java
50
158
76
82.9
84.2
1.3
py-r
113
161
40.4
46.9
48.6
1.7
py-pl
83
161
58.3
68.2
71.2
py-rkt
131
161
31.3
41.3
41.9
0.6
Js-java
43
152
77
77.3
81.3
ji-js
29
138
83.1
83.5
85.8
2.3
py-php
59
161
68.4
77
77.8
0.8
py-lua
86
161
56.2
py-rs
58
156
0.8
d-epp
21
116
88.4
81.4
87.8
epp-lua
58
py-cs
45
158
79.2
lua-ıkt
121
45.2
py-js
36
161
85.5
84.8
Table 14: Pass@1 I for direct, exp, and select trails, where select picks only hard problems with direct pass@ I less than 0.9 to explain. # exp records the number of problems explained. is calculated as select max(direct, exp)

---

J.1 model's dependency on pivot program accuracy
an easier subset. This is intuitive because higher source language generations in general are of better quality, and if the benefit of obtaining more infor- mation and generation length out-weights the noise, this a valid way of boosting performance. Nat- ural language can be thought of an extreme case of this with highest level of resource, with high probability of quality self- self-generated context.
Within pivot program experiments (Python-Cai ++++ Java, Bython-Bash-Java, Python-R-Java), we fur- ther analyzed the Java accuracy by measuring the subset accuracy: we split the set of source prob- lems into those with a correct pivot translation, vs those with an incorrect pivot translation. Here is the result:
Formal intermediate steps are highly umpre- dictable. By glancing the difference between Correct Pass 1 and Incorrect Pass 1, see incorrect pivot programs lead drastically worse performance. we observe the breakdown of the errors, we see a lot of Incorreet Unhelp- ful @ 1, indicating that the pivot programs them selves are unhelpful. Even if we assume mod- els do not generate any unhelpful generations and combine Incorrect Unhelpful@* @ 1 with Incorrect Pass @ 1, we still see significant gap between in- correct and correct pivot programs. Specifically, Incorrect Semantic Error@) tends to be much higher than Correct Semantic Error@1. In ab- lation studies Table we Iearned that when the wrong intermediate step is highly related to the source program in semantics, it decreases the trans- lation performance more. In this experiment, since the semantics of source program and pivot program is almost identical, the mistakes in pivot program can have deleterious effects on translation.
As seen in Table 15, there is clear differences between the subset in which the pivot language passes or fails:
For C++ and R, regardless of the pivot ac- curacy, Java translation accuracy drops with pivot.
For Bash, regardless of the pivot accuracy, Java translation accuracy improves with pivot.
Although the improvement given pivot language seem monotonic not to say that pivot language has no effects on translation accuracy because the programs that are correctly translated in pivot language likely has some characteristic that can confound the transla- tion accuracy.
aggregated level, this is
To further investigate whether can change in- dividual behavior in an individual problem setting, we pick either only correct incorrect programs sampled from ChatGPT and observe translation per- formance (Table 16). If we do not have such a cor- rect/incorrect pivot program we discard that prob- lem. Since there could be bias in the dataset where for specific language, harder problems might have more likelihood of having incorrect problems than correct problems, experiment with various high- /low resource combinations of target language and pivot, to be able to make conclusions overcoming such bias.
Natural language mistakes taken less seri- ously To compare the effect of having mistakes in natural language vs pivot programs, we included In- correct retrieved exp Pass@ swapping an explanation with a closely related that of a similar problem guarantees the explanation to be wrong, we can compare this with Incorrect Pass @1. 1. We found that on average, mistakes in nat- ural language explanations do not decrease trans- lation performance much programming lan- guage mistakes do.
@ 1 from Table Since
Formal language as intermediate step can achieve equivalent or better results than natural language. In Table 16, comparing exp (bottom row) against Correct @1 we can see that sometimes using formal language intermediate step can indeed reach surpass using explanation intermediate steps. Using higher-resource pivot language than the target language always seem to help more than using lower-resoure resource language, ex- cept for rkt-java, which could just be evaluated on
K GPT-3.5 score heuristics
In addition to two heuristics mentioned in Sec 3.4, we also try prompting GPT-3.5 to select the best explanation. We follow works in automatic genera- tion evaluation with LLMs (Koemi and Federmann, 2023; Wang et al., 2023b; Chen et al., 2023c; Fu et al., 2023; Wang et al., 2023b) and experiment with multiple-choice, direct assessment (generating a score between 0 and 100), and summarizing from

---

pivot
java pass
overall pass pass(direc((T)
pass(explT)
pass(directIF)
pass(directIF)
C++
76
42
Bash
74
81
82
89
70
74
R
70
85
77
71
68
Table 15: Translation using formal language as pivot intermediate "explanation". Results are obtained with pass @ [(==1). direct indicate translation accuracy with direct translation (no explanation), and exp means explain then translate. T indicate the pivot program is correct, and similarly for F. Hence, pass(direct!!) means the pass rate for direct translation within the subset programs that the pivot program is correct.
pivot ablation (0 shot, n=20)
php-java
rkt-java
java-php
rkt-php
java-rkt
php-rkt
correct problems
154
155
138
155
147
Correct 6e
80.3
84.3
79.7
69.6
40.8
37.9
Correct Semantic Error@
7.2
8.1
12.6
15.6
21.3
22.5
Correct Syntax Errrr@ 1
12.5
7.6
7.7
14.8
37.8
39.6
Correct @
0
0
0
# incorrect problems
154
149
140
152
140
157
Incorrect Pass@1 1 Incorrect Semantic Error@
35
75.8
43
63.9
23.7
8.3
12.1
13.7
38.3
24.4
29.2
12.3
Incorrect Syntax Error I
11.2
10.5
6.5
11.8
36.3
17.4
Incorrect Unhelpful@*
41.7
0
12.1
0
10.7
62
direct 6ass
31.3
cxp Pass
82.9
Incorrect retrieved exp Pass
75.8
73.7
31.1
Table 16: We use correct/ incorrect pivot program translation to observe the sensitivity models have in formal intermediate reasoning steps. entries indicate 0-shot Pass@ (n=20)). The column label indicate translation direction, pivot language, and the available problems generated. For example, rkt-java refers translating Python to Java, with Racket as pivot language. Semantic Error is equivalent to assertion error, Unhelpful generations include incomplete code with comments like /// TODO", "// Write your code here". All other errors are grouped under Syntax Errors

---

multiple explanations. None of these methods out- performed random selection, we do not include this method in Table GPT-3.5-scores (direct as- sessment) of the explanations almost always fall between 90-100.
Coder-reviewer details
Coder-Reviwer is a re-ranking method introduced by (Zhang al., 2022) to re-rank code generations without verifying through symbolic tools (i.e. com- pilers) in NL-code tasks. The method found that averaging the logprob score from "coder" (which estimates a Songth-normalie "reviewer" (which estimates a length-normaliin p(Ne)cood) can used as a good metric to re- rank code generations. Formally the score de- fined as:
Estimated
py py-ava exp-i
u1
py- exp 2shot
0.5
p(code) NNL)) and
044
Alhh
Figure 10: Coder-Reviewer best few-shot setting vary- ing hyper-parameter. Black dotted lines are average baseline performance where explanations are selected randomly.
(ogp(xly) + (1 a)
logp(ulx)
(1)
where: r represents the natural language descrip- tion of the code, represents generated code, and a is the hyperparameter that weighs the importance between the two terms.
indicating whether the list is sorted in non-descending order and has than occurrences of any integer. The function first creates dictionary where the keys List and the values are the number of times that integer appears in the list. then checks if any integer appears more than twice in the list, and returns False if so. Finally, checks if the list sorted non- descending order, and returns True so, and False otherwise.
In our problem, we have the inverse task of try- ing to find the best explanation given y. Since the score is symmetric, we use the same formula during re-ranking.
the integers in the
To caleulate the logprobs, used CodeGen2- I2B(Nijkamp al., 2023). We use prompt in L.1 and L.2. To obtain the best performance in esti- mated pass rate (Table 3), we try 0,1, or 2-shots (if GPU memory allows), and vary a between 0-1 with 0.1 interval (except for Python-Racket exp. which we tried 0.02 between 0.8- 1.0 in addition to the rest). In Figure 10 plot the best performing setting for each experiment trials across
#** Python verston
from typing import List
def is (lstt: List[int]) bool: count_digit dict([(i, 8) for lst]) for lst:
In section L.3 and section L.4, we show an ex- ample of correct and incorrect selection of expla- nations by coder-reviewer (To provide an idea of what explanations look like, we include only out 20 total explanations).
count_digit[i]-- any(count_digit([] for in
lst):
return False all(Lst[i-1] Ist[i] for in range(1, len(lst))):
return True
L.1
Coder prompt
else:
return False
Can you write a Python program given this explanation?
#*# Explanation
This function takes list of integers *array" input and returns sorted list of integers. The function first checks if the input list is empty. If it is, it returns an empty list.
### Explanation
This function list of integers and boolean

---

Otherwise, it sorts the list using the "sorted' function. The 'reverse'
if the list is sorted in non descending order, and returns True if so, and False otherwise.
parameter is set to 'True! if the sum of the first and last elements of the array is even (and the array will be sorted in descending order), otherwise, it is set to 'False' (and the array will be sorted in ascending order).
Python version
from typing import List
def sort_ (arraaaaayaarayy List[int]] List[int]:
#*# Python version
return [] if len(array) == 0 else sorted (array, reverse= (array [0] +array [-1]) % = 0)
from typing import List
def sort_array (arraayya List[int]) <> List[int]:
Explanation
return [] if Len(array) == else sorted (array, reverse= [array[ [-1]) % 2 == 0)
This function takes a list of integers 'array' as input and returns a sorted list of integers. The function first checks if the input list is empty. it is, it returns an empty list. Otherwise, it sorts the list using the 'sorted' function. The 'reverse* parameter is set to "True' if the sum of the first and last elements of the array is even (and the array will be sorted in descending order), otherwise, it is set to 'False' (and the array will be sorted in ascending order).
#*# Explanation
cinsert explanation here>
#*# Python version
<insert Python program here, calculate normalized log on this sequence>
L.2 Reviewer prompt
Python
<insert Python program here>
Can you explain what this Python program does in couple of sentences?
Explanation
### Python version
<insert explanation here, calculate normalized log p on this sequence>
from typing import List
def is (lst:: List[int]] -> bool: count_digit dict([(i, 8) for in lst])
L.3 Coder-Reviewer correct explanation selection
for 1 in lst:
count_dight count_diggit(j]aa if any(count_ddggt([]] for i in lst):
helpful assistant who understands all programming languages translate between them at Can you explain what this Python program does with couple of sentences goal with the explanation, is so that reader easily rewrite the program Java.
return False
if all(lst[i-1] [st[i] for range len(ts)))):
return True
else:
False
### Explanation
### Python verston
This function takes in list of integers and returns boolean indicating whether the list is sorted in non-descending arder and has than two The function first creates dictionary where the keys the integers in the List and the values times that integer appears the list. It then checks any integer appears than twice the list, and returns False if Finally, it checks
from typing import List
def match_ patch_pprresslstt: List[str]) >> str
of any integer.
def check(s):
val for in if '(:: val val else:
the number of
val
if val

---

return False
and keeping track of the number of open and closed parentheses. If the number of closed parentheses ever exceeds the number of open parentheses, the string is invalid. If the number of open and closed parentheses is equal at the end of the string, the string is valid. The function returns "Yes" if either the original concatenated string or the reversed concatenated string are valid, and 'No' otherwise.
return True if val == 0 else
False
$1 [st[a] + [st[[]
$2 lst[1] + [st[e]
return 'Yes' if check(S1) or check(
S2) else 'No"
#*# Explanation (pass 0/1
This Python program checks if it is possible to create a valid string of parentheses by concatenating two strings of parentheses. The function match_ parens takes a list of two strings of parentheses as input and returns 'Yes' if it is possible to create a valid string of parentheses by concatenating the two strings, and No otherwise.
### Explanation 4 (pass 1/1, Coder-reviewer selected, actual pass 28/20)
This Python program takes 1ist of two strings as input, and checks if the two strings can be concatenated in any order to form string with balanced parentheses. If the parentheses are ba balanced, the function returns 'Yes', otherwise it returns 'No".
The function 'check' checks if given string of parentheses is valid. It does this by iterating over the string and keeping count of the number of opening and closing parentheses. If the count of closing parentheses ever exceeds the count of opening parentheses, the string is invalid. If the count of opening and closing parentheses is equal the end of the iteration, the string valid.
The function 'check' takes string as input and checks if the parentheses in the string are balanced. It does this by iterating through the string and keeping track of the number open and closed parentheses encountered If the number of closed parentheses encountered greater than the number of open parentheses encountered, the function
far.
False. If the number of open and closed parentheses. is equal the end of string, the function returns True, otherwise it False.
The function 'match_parens'
concatenates the input strings in different orders and checks if either
valid using resulting strings 'check" function. one of the resulting strings valid, the function returns 'Yes', otherwise returns 'No'.
function then concatenates two input strings in both possible orders and checks if either of the resulting strings have balanced parentheses. at least one the strings has balanced parentheses, the function returns "Yes', otherwise it returns 'No
#** Explanation (pass 0/1)
This program checks if it is possible form string of balanced parentheses by concatenating strings from given List of strings. does this by checking if the the number of opening parentheses equal the sum of the number of closing parentheses. If the sum of the opening parentheses sum
To rewrite this program in Java, you would need to replace the 'List' type with an array of strings, and use for loop to iterate through the characters in the input strings. You would also need to use the 'StringBuilderr class to concatenate the strings. Here is an example Java implementation:
greater than the closing parentheses, then not possible to form a string balanced parentheses. The program returns 'Yes' it possible to form a string of balanced parentheses No otherwise.
'javv
public static String matchParens (String [] lst) {
StringBuilder s1 new (1st(1));
StringBuiider
#*# Explanation (pass 0/1)
StringBuilder s2 new StringBuilder (1st(2]).aapend) (Lst(o))
This program checks concatenated together form valid string balanced parentheses. does this by iterating through each character of the concatenated string
strings
if (check (shecc(().. II check( s2. toStrrigg)))) return "Yes";
else {

---

return "No";
def words striigsttingss: str) >> List[str]:
if not s: return []
public static boolean check(String s) int val = 0; for (int = i lenghh())
s list =
++)
for letter in 5: if letter == ', s_list.append("
if (s. csarrAtt()) == (() {
val++;
} else val--;
else:
}
if (val < <) return false;
list "" return s_list.pllit()
### Explanation (pass 1/1)
return val 8;
This program takes string input and returns list of words. The input string is first converted into list of characters. If comma is encountered in the string, it is replaced with space. The list of characters is then joined back into string and split into list of words using the default whitespace separator. If the input string is empty, empty list is returned.
<<<ciinend
### Explanation 5 (pass 1/1)
This program checks if it is possible
to create a valid string of parentheses by concatenating two strings from list of strings. The function match_parens" takes list of strings input and returns 'Yes" if it is possible to create valid string of parentheses by concatenating two strings from the List, and 'No' otherwise. The function 'check' checks valid by
### Explanation 2 (pass 1/1)
This program takes a string as input, and returns list of words. It does this by first checking if the string is empty, and if it is, it returns an empty list. If the string is not empty, it creates an empty list called s_list' and then iterates over each letter in the input string. If the letter is comma, it appends space to 's_list', otherwise it appends the letter itself. After iterating over all the letters in the input string, it joins all the elements in 's_list" into single string, and then splits that string into list of words using the default whitespace separator.
string of parentheses counting the number of opening and closing parentheses. If the number of closing parentheses number of opening parentheses paint, the string invalid, If the number opening and closing
greater than the any
parentheses is equal at the end of the string, the string is valid. The function " concatenates the first two strings in the list in two different orders and checks if either the resulting strings is valid. If either string is valid, the function returns "Yes', otherwise it returns ?No".
н** Explanation 3 (pass 0/1, Coder-revien selected, actual pass 8/20)
L.4 Coder-Reviewer incorrect explanation selection
This Python program takes string as input and returns list of words in string' the string. The function 'words_stre 'words_strinn'
first checks if the input string is empty, and if it is, returns an empty list. If the input string is not empty, the function creates empty list s_list', and iterates character in the input string. If the character is comma, the function appends space to 's_list', otherwise it appends the character. The function then joins 's_list' into single string, and splits the string into list of words using the 'split()' method.
You are an helpful AI assistant understands all programning languages and can translate between them at ease.
Can you explain what this Python program does with couple sentences The goal with the explanation, is so that a reader can easily rewrite the program in Java.
each
#*# Python version
from typing import List

---

### Explanation 4 (pass 0/1)
while string[beginning _u_fffi [uffii:][[:=1]]]] beginning_of_sw ofuinng +=
I= string [beginning_oo [
This program takes in a string as input and returns a list of words. The input string is first split into list of characters. If character is comma, it is replaced with space. The list of characters is then joined back into string and split into list of words. If the input string is empty, empty list is returned.
return string stringl: beginning.of.ssffix]]]:::
*** Original Explanation
This function takes a string as input and returns palindrome by appending the reverse of the string's suffix to the string, The suffix is the smallest substring that needs to be appended to the string to make it a palindrome. If the input string is empty, the function returns an empty string.
#*# Explanation 5 (pass 0/1)
This function takes a string as input and returns a list of words. It does this by first converting all commas in the string to spaces, and then splitting the string into list of words. If the input string is empty, it returns an empty list.
Obfuscated Python version
def FUNC_ (VAR_ @: str) -> str: (not VAR_0): return
VAR_1
M Alternative explanation-sslection setting
while (VAR_o[VAR_1:] I= VAR_0[VAR_
:- 1)1):
VAR_1 +=
Sec 3.4, we sample 20 explanations for each problem and generate program from each expla- nation. By sampling more than programs for each explanation, one could obtain variances of the performance estimates (by simulating pass sta- tus of each program according the pass rates of the selected explanations in the train set), but we find sampling one program to be good enough at estimating final performance. Plus, given the same budget, it is also much better to sample 20 (Table 3) than X 5 to maximize explanations diversity, and have the potential of sampling the best explanation. Table 17 shows the result for X 5 experiments.
return (VAR_8 [ 1)])
Obfuscated Explanation
This function takes a string as input returns string. If the input string is empty, returns an empty string. Otherwise, suffix the input string that is also a prefix of the reverse of the input string. It then returns the input string concatenated with the reverse of the remaining part of the input string.
finds the longest
Above is example of the program (humaneval @asl__1_mmaaa uut_maa_ppalindeeeeeeeeeeeeeeeee before and after obfuscation using tools from (Lachaux et al., 2020). After obfuscation, function and variable names are all replaced with respective surface forms, as the functionality of the program remains unchanged. As the example indicates, explanation quality does not really decrease. In fact, explanations often become more detailed just because there is generic way of describing some operation/term like "palindrome".
Comparing results in Table 17 to the main results Table 3, we see much less improvements using heuristics. It is likely that is not a large enough sample size to obtain the the correct explanation for some problems, resulting in low coverage and small improvements from heuristics.
N Program obfuscation
#*a Original Python version
To qualitatively examine explanations' effect translating semantically confusing programs, translated obfuscated Python programs using direct, exp. and exp-Ibl (Table 18). Similar Python-to-X experimen, we generate explanations with Python-Java, remove Java specific explana- tions, and ue-use explanations across the rest of the
def make_pa maake_palindeee indrrmmesstring str) -> str
if string: return
ofginniiggoo_suuffee

---

Exp Type
sre-tgt
random
line-e
line
frag
oracle
exp
py-jv
82.1 +
81.0 + 1.1
81.1 0.9
81.0 + 1.0
79.4 1.1
89.9
exp-Ibl
Py-jv
85.7 +
86.5 0.9
86.3 0.8
86.2 =
85.2 1.0
90.9
exp-Ibl-d
py-jv
85.9 + 1.1
86.4 + 0.6
85.7 0.7
85.9 = 0.6
85.8 0.7
0.7
90.3
Table 17: Explanation selection heuristics performance. We estimate heuristics performance (pass 1, n=1).
jv
php
SW
rkt
ju*
if depth < 0: return False return depth ==
direct
76.0
68.4
64.4
31.3
76.0
exp
82.9
77
70.5
41.3
82.9
exp-Ibl
82.5
77.5
75.1
39
82.5
max(A) (
16.6
31.9
*** Query Python code]
obf direct
72.6
60.3
50.9
27.8
72.6
def fib(n: int) ->
int:
obf exp
78.8
72.7
72.2
34.6
79.6
if n == 8:
obf exp-Ibl
79.9
73.2
72.7
35.9
79.4
return 0
max(A]%
10.1
21.4
42.8
29.1
if == 1:
Table 18: Translation performance after obfuscating source Python programs. Each entry measures pass (n=20). max(A)% measures maximum improvement explanations can bring on top of direct tramslation. jy* column lists explanation performance before removing Java specific information.
return 1
return fib(n - 1) + fib(n 2)
Retrieved Python code]
def fibfib(n: int) int: 8:
return 0
directions. We find that explanation robust re- gardless of surface semantics / readability of source code. In direct, we see consistent drops in perfor- mance across all translation directions. However, explanations still provide consistent improvements across all four languages. The relative improve- ments from explanation is even larger in 2 out 4 directions for obfuscated programs vs. non- obfuscated programs.
n 1:
return 0
n == 2:
return 1
return fibfib(n 1) fibfib(n
2) fibfib(n 3)
Above are examples of retrieved Python pro- grams given query programs. For retrieval, we tokenize the python program the same way as TransCoder using "tokenize" library 12, We then use BM250kap1 algorithm in rank_ bm25 library to retrieve for the most similar program within the HumanEval Dataset. Although one can improve trieval similarity by using more sophisticated meth- ods (dense embeddings such as UniXCoder or syn- tax aware similarity metrics such as CodeBLEU), we find BM25 to be cheap and effective at retriev- ing similar programs for our ablation studies.
Program Retrieval
#NR Query Python code
def prrrett_baacketnnnttrakktsss str)
bool:
depth
for b in brackets: if == ""("";
depth
else:
depth
P Intermediate step lengths
if depth
return False
In Table 19, we report the length of the intermedi- ate steps (explanation, pivot programs) and their respective ratio to the source program length. One could argue, that the increased length in interme- diate step could lead more computation in tention, decoding, which leads improvement in downstream translation. Here, we note several ob- servations and leave a detailed investigation as a direction for future work.
return depth
#*# Retrieved Python code]
def sorrecttbrakketing((aaktts: str)
bool:
depth
for in brackets:
if == #"qun;
depth
else:
depth

---

exp
exp/sre ratio
Python-to-X exp
96 + 70
1.55 + 1.04
Python-to-X exp (tgt-specific)
101 69
1.70 + 1.09
Python-to-X exp (four-shot)
122
2.06 = 0.90
Python-to-X exp (four-shot, tgt-specific)
102
1.71 = 0.79
Python-to-X exp (four-shot, coder-reviewer)
228 153
4.09 3.84
Python-to-X exp-Ibl
305 + 126
+ 2.02
Python-to-X exp-lbl (four-shot)
195
3.08 0.91
Python-to-X exp-ibl-d
316 + 121
5.26 2.27
Python-to-XX exp-Ibl-d (four-shot)
275 + 118
4.45 = 1.68
Python-to-X exp-Ibl-d (four-shot, frag)
305 + 123
5.16 + 1.96
Pivot (java)
124 + 52
2.01 + 0.75
Pivot (php)
94 + 52
1.31 + 0.23
Pivot (rkt)
80 + 51
1.13 = 0.32
Table 19: Intermediate step (explanations, pivot program) length and their ratio to source Python program
More detailed explanation is longer However, have noted in Table 1, more detailed planations do not always lead to more improve- ments. In high-low-rcsouree directions, more generic (shorter) explanations often works better. This is one of the examples where length does not correlate well with performance.
interesting to understand how does the verbosity of language correlate their usefulness as intermediate reasoning step.
Target-specific information improves perfor- mance In zero-shot four-shot settings (Ta- ble G.I, G.2), we see slight improvement with tar- get specific explanations. However, length-wise, we do not see a pattern between target-specific ex- planation vs. target-independent explanation and four-shot setting.
Heuristically selected explanations are longer Compare Python-to-X exp (four-shot, coder- reviewer) vs. Python-to-X exp (four-shot) and Python-to-X exp-Ibl-d (four-shot, frag) vs. Python-to-X exp-Ibl-d (four-shot), both heuristically selected explanations are longer than their random baselines. However, as seen in Table len heuristics do not do nearly as well as winning heuristics. This indicates that length is important, but is not all the signal in determining the success in translation.
Formal intermediate steps can be more efficient In Table 16, we see similar scale improvements from using correct pivot programs as intermediate steps. We conclude from the table that using higher resource language as pivot works better, and this case we do see higher-resource language tend to be longer than lower- lowereresource languages. would be


Reviews:
[{"review_id": "sJCXwTa4uK", "reviewer": "Reviewer_pfnB", "summary": "", "questions": "A The 3.3. Metrics section is very brief; could you elaborate on that comment \"n, k\" often dependent of sampling temperature k ? What does this dependency look like?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for a thorough review! The goal of our paper is to study how various types of explanations improve code-to-code translation with a strong model. Since ChatGPT is (near)  the SOTA model  we focused on that. However, we completely understand (and agree with) your  point about  ChatGPT-only results. Hence, we conducted additional experiments with an open-source model (CodeGen 1B). The main conclusion is that we can indeed observe similar improvements by providing explanations in smaller open source models. We will make sure to include these new results in the updated version of the paper. \n\nWe address some of the  other concerns in below:\n\n> The 3.3. Metrics section is very brief; could you elaborate on that comment \"n, k\" often dependent of sampling temperature k ? What does this dependency look like?:\n\nThe metric of pass@k was first popularized by Chen et. al. 2021 (Codex paper), which was invented so users of the code generation system have a good idea of fractions of successful code generation out of k samples. In the same paper, the authors also perform extensive experiments showing the relationship between sampling temperature and pass@k (Figure 5, 6). To summarize, lower temperature (0.2) is better when n/k is small, but higher temperature can improve when n/k is large due to the variance in the generation (higher recall). In our experiments, we focus on improving mainly pass@1, so we followed Chen et. al. 2021 and Cassano, et al 2023’s advice for using temperature 0.2. We will note this dependency in the future iteration of the paper.  \n\n> It is really hard to see a linear fit in Figure 2, seeing the data.I wonder if that correlation could be faceted by controlling for e.g. language dataset size.\n\nAs mentioned in text, the correlation is particularly strong ($r^2$\n = 0.719) if we disregard some outliers (Bash, R, Julia). Bash in particular, has poor test coverage/quality (this is from actual authors of the MultiPL-E paper) and should be analyzed lightly. Nonetheless, our intention with the figure is to bring awareness to the reader that explain-then-translate, similar to few-shot prompting, is just another method to improve performance. Such observation subsequently leads to our discussion on whether natural language or programming language is better as a self-generated context to boost performance. As to the point of whether it is related to dataset size of the programming language, we believe it is not that clear-cut of a story: lower-resource languages such as Racket and D sit on polar ends of the linear fit, and so do higher-resource languages (JavaScript vs. Java).\n\n> If the paper used an open-source model instead, or at the very least made a comparison to an OSS LM, I would be much more inclined to recommend it for acceptance.\n\nIn the time available during the rebuttal period, we are able to reproduce most of Table1, Table2 for 0-shot setting with more open source models (CodeGen2-16B, CodeGen2-1B). We plan to finish the experiments with StarCoder as well to compare with the current open source SOTA code model. Additionally, we tested whether smaller models can benefit from better explanations generated by big models (prompting CodeGen2-1B with ChatGPT generated explanations). We hope the tables of results and additional comparison of results across open-source models can be taken into consideration.\n\n### Python-to-X Translation (Similar to paper table 1) \n\nTranslation pass@1 from Python to X. Parenthesis in trial indicates # of shots. Best within same explanation source model is *italicized* and overall best is **bolded**. All parameters are kept the same as table 1 in paper.\n\n##### Table 1.1 (w/ CodeGen2-1B, self generated explanation and ChatGPT-generated explanation)\n\n| exp  | js | cpp | jv | ts | php | rb | cs | go | pl | r | rs | sc | sw | sh | lua | rkt | jl | d |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| baseline (0shot) | 4.6 | 9.4 | 5.3 | 7 | 13.4 | 0 | 6.4 | 3.5 | 3.6 | 0.6 | 4.7 | 2.6 | 3.5 | ***5.7*** | 2.7 | 0 | 2.4 | 6.2 |\n| exb (0shot) | *15.4* | *11.7* | ***7.8*** | *13.5* | 14.2 | ***5.9*** | 7.9 | 8.3 | 4.1 | 0.8 | *8.7* | *5.6* | *6.9* | 4.8 | 5.3 | 0 | *3.1* | ***7.9*** |\n| exb-lbl (0shot) | 13.9 | 11.1 | 6.5 | 11.3 | *17.3* |  |  |  |  |  |  |  |  | 3.1 |  |  |  | 7.3 |\n| exb-lbl-d (0shot) | 12.7 | *11.7* | 7.6 | 9.9 | 14.6 | 3.9 | *8.5* | *8.8* | *4.5* | ***0.9*** | 8.2 | 2.4 | *6.9* | 4.7 | ***5.4*** | 0 | 2.8 | 5.9 |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| exb (0shot) w/ ChatGPT | ***19.3*** | ***15.5*** | 5.8 | 10.8 | 16.5 | 3.9 | 11.2 | 9.8 | *4.2* | 0.4 | ***10.3*** | ***7*** | ***10.4*** | 5.2 | 4.8 | **0.1** | **3.7** | 7.2 |\n| exb-lbl (0shot) w/ ChatGPT | 16.2 | 14.3 |  | ***14.3*** | ***17.6*** | *4.7* | ***11.5*** | ***10.7*** | 3.6 | *0.7* | 9.5 | 5.4 | 10.3 |  |  | **0.1** |  |  |\n| exb-lbl-d (0shot) w/ ChatGPT |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |\n\n##### Table 1.2 (w/ CodeGen2-16B)\n\n| exp  | js | cpp | jv | ts | php | rb | cs | go | pl | r | rs | sc | sw | sh | lua | rkt | jl | d |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| baseline (0shot) | **48.4** | **41.2** |  | **44.8** | **38.4** | **8.6** | **46** |  |  |  |  |  |  |  |  |  |  | 11.7 |\n| exp (0shot) | 45.1 | 38.2 |  | 43.7 | 37.5 | 2.8 | 41.8 |  |  |  |  |  |  |  |  |  |  | 14.4 |\n| explain-lbl (0shot) | 45.1 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | **15.8** |\n| explain-lbl-d (0shot) | 43.2 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 15 |\n\n\n\n\n### X-to-X Translation (Similar to paper table 2)\n\nFor the two tables below, we report translation pass@1 between 16 different pairs of languages. **Resource** indicates the language resource levels of the source and target. **Type** indicates the source and target language typing characteristics\n(D/S=dynamically/statically typed). The best runs within the same-shot setting are **bolded**. All parameters are kept the same as table 2 in paper.\n\n##### Table 2.1 (w/ CodeGen2-1B)\n\n|  |  |  | 0 shot |  |  |  |\n| --- | --- | --- | --- | --- | --- | --- |\n| Resource | Typing | src-tgt | direct | exp | exp-lbl | exp-lbl-d |\n| High-to-High | D-D | Python - JavaScript | 4.6 |  **14.9**  | 13.9 | 13.1 |\n|  | D-T | JavaScript - Java | 7.8 |  **11.2** | 9.6 | 10.1 |\n|  | T-D | C++ - Python | 8.3 | **13.9** | 13.5 | 10.7 |\n|  | T-T | Java - C++ | 14.2 | 15.6 | 14.8 | **17.5** |\n| High-to-ExtLow | D-D | JavaScript - Racket | 0 | 0 | 0 | 0 |\n|  | D-T | Python - D | 6.2 | **7.9** | 7.3 | 5.9 |\n|  | T-D | C++ - Lua | 8.9 | **11.8** | 10 | 9.8 |\n|  | T-T | Java - Julia | 3.2 | **4.9** | 3.9 | 4.4 |\n| ExtLow-to-High | D-D | Lua - Python | 1.9 | **6** | 5.9 | 5.6 |\n|  | D-T | Racket - Java | 7.9 | 7.6 | **8.1** | 7.9 |\n|  | T-D | Julia - JavaScript | 3.7 | **6.9** | 3.9 | 5.1 |\n|  | T-T | D - C++ | 11.7 | 12.7 | 12.6 | **14.1** |\n| ExtLow-toExtLow | D-D | Lua - Racket | 0 | 0 | 0 | 0 |\n|  | D-T | Racket - Julia | 2 | **2.1** | 1.5 | 0.8 |\n|  | T-D | D - Lua | 7.7 | **8.2** | 6.3 | 7.2 |\n|  | T-T | Julia - D | 0 | **0.007** | 0 | 0 |\n\n##### Table 2.2 (w/ CodeGen2-16B)\n\n|  |  |  | 0 shot |  |  |  |\n| --- | --- | --- | --- | --- | --- | --- |\n| High-to-High | D-D | Python - JavaScript | **48.4** | 45.1 | 45.1 | 43.1 |\n|  | D-T | JavaScript - Java | **47.3** | 44.3 | 42 | 40 |\n|  | T-D | C++ - Python | 38 | **57.2** | 55.1 |  |\n|  | T-T | Java - C++ | 52.7 |  |  |  |\n| High-to-ExtLow | D-D | JavaScript - Racket | 2 |  |  |  |\n|  | D-T | Python - D | 11.7 | 14.4 | **15.8** | 15 |\n|  | T-D | C++ - Lua | 33.6 |  |  |  |\n|  | T-T | Java - Julia | 26.4 | **27.9** | 25.8 | 25.9 |\n| ExtLow-to-High | D-D | Lua - Python | 37.6 | 45.6 | **47.8** | 46.7 |\n|  | D-T | Racket - Java | **28.4** | 27.8 |  |  |\n|  | T-D | Julia - JavaScript | 40.1 | 39.7 | 40 | **42.9** |\n|  | T-T | D - C++ | **53.1** | 47.6 | 47.1 | 50.4 |\n| ExtLow-toExtLow | D-D | Lua - Racket | 0.8 |  |  |  |\n|  | D-T | Racket - Julia | 13.1 | **14.6** | **14.6** | 14.4 |\n|  | T-D | D - Lua | **29.3** | 28.1 | 28.8 | 28.9 |\n|  | T-T | Julia - D | **10.1** | 7.8 | 5.4 | 5.3 |\n\n\n\nFrom the tables above, we can draw the following additional conclusions, which we will include in the next version of the paper:\n- In weaker models, **we still see improvements with self-generated explanations across most directions**. CodeGen2-1B improves more consistently using self-generated explanations (baseline is the worst in all X-to-X directions, and in 17/18 Python-to-X directions), as much as 300%+ improvement (Table 2.3, Lua→Python, Python→JavaScript). In Python→Ruby, model with explanations obtains a pass rate of 5.9, while baseline does not generate *any* single correct translation (pass rate of 0). CodeGen2-16B shows weaker results, with baseline outperforming in 10/18 directions in Table 1 and 2. Perhaps it has a weaker alignment between natural language and programming language, resulting in worse explanations generated for each problem. The majority of errors from translation with explanation are syntactic. \n- **Better explanation leads to better translation, even in smaller model cases**. We compare CodeGen2-1B performance given self-generated explanation vs. ChatGPT generated explanation and see that the better explanation outperforms self-generated explanation in 12/18 Python-to-X directions, with a maximum improvement of 400%+ in Python → JavaScript.\n- In smaller/weaker models, **detailed explanations (exp-lbl or exp-lbl-d) do not improve as much as exp does**. Often this is due to lower quality explanations generated when the model is asked to do something it is not capable of. The line-by-line explanations often lead to repetitive content when source programs contain several repetitive lines (library import in C++ → Python direction, with CodeGen2-1B)\n- we will add these results and results from StarCoder in the revised version of our paper", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a battery of translation pairs and prompt variations. Most importantly, it summarizes the vast experimental evidence into nuggets that hopefully will be valuable to theorists. As such, it's a sound empirical work and well suited for EMNLP.", "reasons_to_reject": "To me, using a black-box, proprietary model like ChatGPT which periodically changes under the hood is a major scientific liability. This paper as a whole won't likely be reproducible in a few weeks' to months' time.  It is really unfortunate that the authors chose this and not a \"weaker\" but reproducible model, especially considering the wealth of OSS code LMs we have nowadays (CodeT5, StarCoder, etc.). I get it, ChatGPT is apparently \"strong\" in a number of areas, very convenient to use etc., but this is not what good science is about.    If the paper used an open-source model instead, or at the very least made a comparison to an OSS LM, I would be much more inclined to recommend it for acceptance.\n\n--- \n\n## Rebuttal acknowlegement\n\nI appreciate the authors' response and I gladly increase my score.", "questions_to_authors": "A The 3.3. Metrics section is very brief; could you elaborate on that comment \"n, k\" often dependent of sampling temperature k ? What does this dependency look like?", "paper_topic_and_main_contributions": "This paper is concerned with chain-of-thought prompting as applied to program source code translation. More in particular, it applies this method (originally from Chen et al., Teaching large language models to self-debug, 2023 ) extending the number of languages to 19, across low- and high-resource ones.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Only in the Conclusions section we learn that the model under test is ChatGPT. This fact should be made clear in the beginning of the work, or at least in the Experiments section.\n\nIt is really hard to see a linear fit in Figure 2, seeing the data.I wonder if that correlation could be faceted by controlling for e.g. language dataset size."}}, {"review_id": "Ap6PHpYGbi", "reviewer": "Reviewer_CiwV", "summary": "", "questions": "- could you add some comparative analysis comprising multiple LLMs focusing on the key findings?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review and comments! Taking both reviewer#1 and this feedback, we have provided additional tables that we will add to the next version of our paper paper. With the additional data points using opensource models, we can compare model performance over different training dataset and model size, as well as improving reproducibility of our claims with “white-box” models.\n\nIn the time available during the rebuttal period, we are able to reproduce most of Table1, Table2 for 0-shot setting with more open source models (CodeGen2-16B, CodeGen2-1B). We plan to finish the experiments with StarCoder as well to compare with the current open source SOTA code model. Additionally, we tested whether smaller models can benefit from better explanations generated by big models (prompting CodeGen2-1B with ChatGPT generated explanations). We hope the tables of results and additional comparison of results across open-source models can be taken into consideration.\n\n### Python-to-X Translation (Similar to paper table 1) \n\nTranslation pass@1 from Python to X. Parenthesis in trial indicates # of shots. Best within same explanation source model is *italicized* and overall best is **bolded**. All parameters are kept the same as table 1 in paper.\n\n##### Table 1.1 (w/ CodeGen2-1B, self generated explanation and ChatGPT-generated explanation)\n\n| exp  | js | cpp | jv | ts | php | rb | cs | go | pl | r | rs | sc | sw | sh | lua | rkt | jl | d |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| baseline (0shot) | 4.6 | 9.4 | 5.3 | 7 | 13.4 | 0 | 6.4 | 3.5 | 3.6 | 0.6 | 4.7 | 2.6 | 3.5 | ***5.7*** | 2.7 | 0 | 2.4 | 6.2 |\n| exb (0shot) | *15.4* | *11.7* | ***7.8*** | *13.5* | 14.2 | ***5.9*** | 7.9 | 8.3 | 4.1 | 0.8 | *8.7* | *5.6* | *6.9* | 4.8 | 5.3 | 0 | *3.1* | ***7.9*** |\n| exb-lbl (0shot) | 13.9 | 11.1 | 6.5 | 11.3 | *17.3* |  |  |  |  |  |  |  |  | 3.1 |  |  |  | 7.3 |\n| exb-lbl-d (0shot) | 12.7 | *11.7* | 7.6 | 9.9 | 14.6 | 3.9 | *8.5* | *8.8* | *4.5* | ***0.9*** | 8.2 | 2.4 | *6.9* | 4.7 | ***5.4*** | 0 | 2.8 | 5.9 |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| exb (0shot) w/ ChatGPT | ***19.3*** | ***15.5*** | 5.8 | 10.8 | 16.5 | 3.9 | 11.2 | 9.8 | *4.2* | 0.4 | ***10.3*** | ***7*** | ***10.4*** | 5.2 | 4.8 | **0.1** | **3.7** | 7.2 |\n| exb-lbl (0shot) w/ ChatGPT | 16.2 | 14.3 |  | ***14.3*** | ***17.6*** | *4.7* | ***11.5*** | ***10.7*** | 3.6 | *0.7* | 9.5 | 5.4 | 10.3 |  |  | **0.1** |  |  |\n| exb-lbl-d (0shot) w/ ChatGPT |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |\n\n\n##### Table 1.2 (w/ CodeGen2-16B)\n\n| exp  | js | cpp | jv | ts | php | rb | cs | go | pl | r | rs | sc | sw | sh | lua | rkt | jl | d |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| baseline (0shot) | **48.4** | **41.2** |  | **44.8** | **38.4** | **8.6** | **46** |  |  |  |  |  |  |  |  |  |  | 11.7 |\n| exp (0shot) | 45.1 | 38.2 |  | 43.7 | 37.5 | 2.8 | 41.8 |  |  |  |  |  |  |  |  |  |  | 14.4 |\n| explain-lbl (0shot) | 45.1 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | **15.8** |\n| explain-lbl-d (0shot) | 43.2 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 15 |\n\n\n\n\n### X-to-X Translation (Similar to paper table 2)\n\nFor the two tables below, we report translation pass@1 between 16 different pairs of languages. **Resource** indicates the language resource levels of the source and target. **Type** indicates the source and target language typing characteristics\n(D/S=dynamically/statically typed). The best runs within the same-shot setting are **bolded**. All parameters are kept the same as table 2 in paper.\n\n##### Table 2.1 (w/ CodeGen2-1B)\n\n|  |  |  | 0 shot |  |  |  |\n| --- | --- | --- | --- | --- | --- | --- |\n| Resource | Typing | src-tgt | direct | exp | exp-lbl | exp-lbl-d |\n| High-to-High | D-D | Python - JavaScript | 4.6 |  **14.9**  | 13.9 | 13.1 |\n|  | D-T | JavaScript - Java | 7.8 |  **11.2** | 9.6 | 10.1 |\n|  | T-D | C++ - Python | 8.3 | **13.9** | 13.5 | 10.7 |\n|  | T-T | Java - C++ | 14.2 | 15.6 | 14.8 | **17.5** |\n| High-to-ExtLow | D-D | JavaScript - Racket | 0 | 0 | 0 | 0 |\n|  | D-T | Python - D | 6.2 | **7.9** | 7.3 | 5.9 |\n|  | T-D | C++ - Lua | 8.9 | **11.8** | 10 | 9.8 |\n|  | T-T | Java - Julia | 3.2 | **4.9** | 3.9 | 4.4 |\n| ExtLow-to-High | D-D | Lua - Python | 1.9 | **6** | 5.9 | 5.6 |\n|  | D-T | Racket - Java | 7.9 | 7.6 | **8.1** | 7.9 |\n|  | T-D | Julia - JavaScript | 3.7 | **6.9** | 3.9 | 5.1 |\n|  | T-T | D - C++ | 11.7 | 12.7 | 12.6 | **14.1** |\n| ExtLow-toExtLow | D-D | Lua - Racket | 0 | 0 | 0 | 0 |\n|  | D-T | Racket - Julia | 2 | **2.1** | 1.5 | 0.8 |\n|  | T-D | D - Lua | 7.7 | **8.2** | 6.3 | 7.2 |\n|  | T-T | Julia - D | 0 | **0.007** | 0 | 0 |\n\n##### Table 2.2 (w/ CodeGen2-16B)\n\n|  |  |  | 0 shot |  |  |  |\n| --- | --- | --- | --- | --- | --- | --- |\n| High-to-High | D-D | Python - JavaScript | **48.4** | 45.1 | 45.1 | 43.1 |\n|  | D-T | JavaScript - Java | **47.3** | 44.3 | 42 | 40 |\n|  | T-D | C++ - Python | 38 | **57.2** | 55.1 |  |\n|  | T-T | Java - C++ | 52.7 |  |  |  |\n| High-to-ExtLow | D-D | JavaScript - Racket | 2 |  |  |  |\n|  | D-T | Python - D | 11.7 | 14.4 | **15.8** | 15 |\n|  | T-D | C++ - Lua | 33.6 |  |  |  |\n|  | T-T | Java - Julia | 26.4 | **27.9** | 25.8 | 25.9 |\n| ExtLow-to-High | D-D | Lua - Python | 37.6 | 45.6 | **47.8** | 46.7 |\n|  | D-T | Racket - Java | **28.4** | 27.8 |  |  |\n|  | T-D | Julia - JavaScript | 40.1 | 39.7 | 40 | **42.9** |\n|  | T-T | D - C++ | **53.1** | 47.6 | 47.1 | 50.4 |\n| ExtLow-toExtLow | D-D | Lua - Racket | 0.8 |  |  |  |\n|  | D-T | Racket - Julia | 13.1 | **14.6** | **14.6** | 14.4 |\n|  | T-D | D - Lua | **29.3** | 28.1 | 28.8 | 28.9 |\n|  | T-T | Julia - D | **10.1** | 7.8 | 5.4 | 5.3 |\n\n\n\nFrom the tables above, we can draw the following additional conclusions, which we will include in the next version of the paper:\n- In weaker models, **we still see improvements with self-generated explanations across most directions**. CodeGen2-1B improves more consistently using self-generated explanations (baseline is the worst in all X-to-X directions, and in 17/18 Python-to-X directions), as much as 300%+ improvement (Table 2.3, Lua→Python, Python→JavaScript). In Python→Ruby, model with explanations obtains a pass rate of 5.9, while baseline does not generate *any* single correct translation (pass rate of 0). CodeGen2-16B shows weaker results, with baseline outperforming in 10/18 directions in Table 1 and 2. Perhaps it has a weaker alignment between natural language and programming language, resulting in worse explanations generated for each problem. The majority of errors from translation with explanation are syntactic. \n- **Better explanation leads to better translation, even in smaller model cases**. We compare CodeGen2-1B performance given self-generated explanation vs. ChatGPT generated explanation and see that the better explanation outperforms self-generated explanation in 12/18 Python-to-X directions, with a maximum improvement of 400%+ in Python → JavaScript.\n- In smaller/weaker models, **detailed explanations (exp-lbl or exp-lbl-d) do not improve as much as exp does**. Often this is due to lower quality explanations generated when the model is asked to do something it is not capable of. The line-by-line explanations often lead to repetitive content when source programs contain several repetitive lines (library import in C++ → Python direction, with CodeGen2-1B)\n- we will add these results and results from StarCoder in the revised version of our paper", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Detailed study and adequate explanations regarding the reasonings\n- Detailed ablation study", "reasons_to_reject": "- this study is based on only chatgpt without any room of comparative evaluation. There needs to be some analysis comprising other llm so that the behaviors and finds could be confidently described as general.", "questions_to_authors": "- could you add some comparative analysis comprising multiple LLMs focusing on the key findings?", "paper_topic_and_main_contributions": "This paper experimented with chain-of-thought prompting in the case of programming language translation. With the aid of zero-shot and few-shot prompting, the authors came across important insights such as, adding simpler explanations turning out more useful in case of low-resourced ones and using simpler heuristics to perform better explanation selection.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SdD9HBjyTc", "reviewer": "Reviewer_XyFJ", "summary": "", "questions": "Question A: In Section 4.3, why were 0-shot explanations used for exp, while 4-shot explanations were used for exp-lbl and exp-lbl-d? I don’t see a good reason for being inconsistent here.\nQuestion B: Is there any mention in the paper of the compute resources used to run the inference performed in the experiments? Since all of the inference was performed on ChatGPT, perhaps information on the number of API keys used, as well as the time it took to run all the inference would be helpful as well.\nQuestion C: Why were only a subset of all the possible translation directions used when experimenting with alternative translation directions in Section 4.2?\nQuestion D: Why was ChatGPT mentioned as the LLM model used for inference only until the conclusion? I believe that is a relevant detail that could be included much earlier.\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for such a thorough review! We appreciate the attention to all the details. We will fix the typos and low-level mistakes pointed out and address some of the bigger questions here:\n\n> The authors claim to release their code and dataset that they used for this paper, however, I cannot find any links to these resources in the paper.\n\nPlease find the source code and artifacts generated in the zip file submitted with the original draft in OpenReview. As we finish cleaning up our repository, we will open-source our code on GitHub and make our artifacts downloadable through a public drive link. (The Github link is removed from the paper for anonymity)\n\n> Section 3, where the authors explain some of their methodology on how they are prompting the LLM, is quite vague, as it does not mention what LLM they are using nor does it clearly outline how they are using their explain-then-translate method to prompt the LLM, both of which may make reproducibility more difficult.\n\nFor the language model used in our experiments, we used ChatGPT through the Microsoft Azure completion endpoint. For the next version of the paper, we will incorporate additional experiment results with open-source models (CodeGen2-16B, CodeGen2-1B, CodeGen2-1B with ChatGPT explanations, and StarCoder), most of which can be found in the rebuttal response for reviewer#1. For detailed prompts, please see Appendix C. Prompts with explanations are generated in two separate steps. Once an explanation is generated for a source program, we append the explanation and the target code signature to the prompt (instruction, source program) and feed it to the model. Once programs are generated, we run a post-processing script (Appendix D) to remove extra natural language comments, and program statements outside of the function.\n\n> Question A:  In Section 4.3, why were 0-shot explanations used for exp, while 4-shot explanations were used for exp-lbl and exp-lbl-d? I don’t see a good reason for being inconsistent here. \n\nIn Section 4.1.2 (second paragraph) and Apx F.2, we find few-shot explanations to have lower qualities than zero-shot explanations for exp (but not for exp-lbl or exp-lbl-d), hence we use the best shot-setting to produce the pool of explanations from which we do our heuristic selection. For exp-lbl and exp-lbl-d, since the task is a bit more complex and requires few-shot guidance, we use few-shot generated explanations.\n\n> Question B: Is there any mention in the paper of the compute resources used to run the inference performed in the experiments? Since all of the inference was performed on ChatGPT, perhaps information on the number of API keys used, as well as the time it took to run all the inference would be helpful as well. \n\nThanks for pointing this out. We will include the following section in the next version of our paper:\n\nThe completion queries are made to ChatGPT (gpt-3.5-turbo-3010) between March - June 2023. We use the Azure completion endpoint (as opposed to the OpenAI chat completion endpoint). Compute credits are from XXX lab. The average query time for a single experiment (e.g. exp, Python -> Julia) takes around 10-40 minutes with one API key (of which we only used one). For Tables 1 and 2, the total query time is between 21 - 84 hours. The experiments for CodeGen2-1B are conducted on XXX and XXX clusters with NVIDIA RTX V100, A100, and A6000. Each single experiment takes 12-24 hours so the whole compute time for table 1&2 completions is around 72-144 GPU days. Code execution/evaluation is done locally on the same MacBook Pro 2015. For future studies and reproducibility, we will release docker/podman evaluations in the same way as the MultiPL-E dataset.\n\n> Question C: Why were only a subset of all the possible translation directions used when experimenting with alternative translation directions in Section 4.2? \n\nThe subset is chosen to mainly save computational cost while maintaining a representation of the dataset. Since we can divide the languages roughly by resource level and whether the languages are typed or not, we sample extreme ends of the resource level (high vs. extremely low) and make sure all combinations of type characteristics (typed -> dynamic, etc.) are included in the subset. The reason to select high-resource languages is to observe the best possible performance, while extremely low-resource languages are also crucial for understanding performance bottlenecks and are important for real-world applications (e.g. code modernization).\n\n> Question D: Why was ChatGPT mentioned as the LLM model used for inference only until the conclusion? I believe that is a relevant detail that could be included much earlier.\n\nThanks for pointing this out. Since we are adding additional experiments on open-source models, we will remove that limitation and mention the models used at the beginning of our paper.\n\nIn the time available during the rebuttal period, we are able to reproduce most of Table1, Table2 for 0-shot setting with more open source models (CodeGen2-16B, CodeGen2-1B). We plan to finish the experiments with StarCoder as well to compare with the current open source SOTA code model. Additionally, we tested whether smaller models can benefit from better explanations generated by big models (prompting CodeGen2-1B with ChatGPT generated explanations). We hope the tables of results and additional comparison of results across open-source models can be taken into consideration.\n\n### Python-to-X Translation (Similar to paper table 1) \n\nTranslation pass@1 from Python to X. Parenthesis in trial indicates # of shots. Best within same explanation source model is *italicized* and overall best is **bolded**. All parameters are kept the same as table 1 in paper.\n\n##### Table 1.1 (w/ CodeGen2-1B, self generated explanation and ChatGPT-generated explanation)\n\n| exp  | js | cpp | jv | ts | php | rb | cs | go | pl | r | rs | sc | sw | sh | lua | rkt | jl | d |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| baseline (0shot) | 4.6 | 9.4 | 5.3 | 7 | 13.4 | 0 | 6.4 | 3.5 | 3.6 | 0.6 | 4.7 | 2.6 | 3.5 | ***5.7*** | 2.7 | 0 | 2.4 | 6.2 |\n| exb (0shot) | *15.4* | *11.7* | ***7.8*** | *13.5* | 14.2 | ***5.9*** | 7.9 | 8.3 | 4.1 | 0.8 | *8.7* | *5.6* | *6.9* | 4.8 | 5.3 | 0 | *3.1* | ***7.9*** |\n| exb-lbl (0shot) | 13.9 | 11.1 | 6.5 | 11.3 | *17.3* |  |  |  |  |  |  |  |  | 3.1 |  |  |  | 7.3 |\n| exb-lbl-d (0shot) | 12.7 | *11.7* | 7.6 | 9.9 | 14.6 | 3.9 | *8.5* | *8.8* | *4.5* | ***0.9*** | 8.2 | 2.4 | *6.9* | 4.7 | ***5.4*** | 0 | 2.8 | 5.9 |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| exb (0shot) w/ ChatGPT | ***19.3*** | ***15.5*** | 5.8 | 10.8 | 16.5 | 3.9 | 11.2 | 9.8 | *4.2* | 0.4 | ***10.3*** | ***7*** | ***10.4*** | 5.2 | 4.8 | **0.1** | **3.7** | 7.2 |\n| exb-lbl (0shot) w/ ChatGPT | 16.2 | 14.3 |  | ***14.3*** | ***17.6*** | *4.7* | ***11.5*** | ***10.7*** | 3.6 | *0.7* | 9.5 | 5.4 | 10.3 |  |  | **0.1** |  |  |\n| exb-lbl-d (0shot) w/ ChatGPT |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |\n\n\n##### Table 1.2 (w/ CodeGen2-16B)\n\n| exp  | js | cpp | jv | ts | php | rb | cs | go | pl | r | rs | sc | sw | sh | lua | rkt | jl | d |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| baseline (0shot) | **48.4** | **41.2** |  | **44.8** | **38.4** | **8.6** | **46** |  |  |  |  |  |  |  |  |  |  | 11.7 |\n| exp (0shot) | 45.1 | 38.2 |  | 43.7 | 37.5 | 2.8 | 41.8 |  |  |  |  |  |  |  |  |  |  | 14.4 |\n| explain-lbl (0shot) | 45.1 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | **15.8** |\n| explain-lbl-d (0shot) | 43.2 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 15 |\n\n\n\n\n### X-to-X Translation (Similar to paper table 2)\n\nFor the two tables below, we report translation pass@1 between 16 different pairs of languages. **Resource** indicates the language resource levels of the source and target. **Type** indicates the source and target language typing characteristics\n(D/S=dynamically/statically typed). The best runs within the same-shot setting are **bolded**. All parameters are kept the same as table 2 in paper.\n\n##### Table 2.1 (w/ CodeGen2-1B)\n\n|  |  |  | 0 shot |  |  |  |\n| --- | --- | --- | --- | --- | --- | --- |\n| Resource | Typing | src-tgt | direct | exp | exp-lbl | exp-lbl-d |\n| High-to-High | D-D | Python - JavaScript | 4.6 |  **14.9**  | 13.9 | 13.1 |\n|  | D-T | JavaScript - Java | 7.8 |  **11.2** | 9.6 | 10.1 |\n|  | T-D | C++ - Python | 8.3 | **13.9** | 13.5 | 10.7 |\n|  | T-T | Java - C++ | 14.2 | 15.6 | 14.8 | **17.5** |\n| High-to-ExtLow | D-D | JavaScript - Racket | 0 | 0 | 0 | 0 |\n|  | D-T | Python - D | 6.2 | **7.9** | 7.3 | 5.9 |\n|  | T-D | C++ - Lua | 8.9 | **11.8** | 10 | 9.8 |\n|  | T-T | Java - Julia | 3.2 | **4.9** | 3.9 | 4.4 |\n| ExtLow-to-High | D-D | Lua - Python | 1.9 | **6** | 5.9 | 5.6 |\n|  | D-T | Racket - Java | 7.9 | 7.6 | **8.1** | 7.9 |\n|  | T-D | Julia - JavaScript | 3.7 | **6.9** | 3.9 | 5.1 |\n|  | T-T | D - C++ | 11.7 | 12.7 | 12.6 | **14.1** |\n| ExtLow-toExtLow | D-D | Lua - Racket | 0 | 0 | 0 | 0 |\n|  | D-T | Racket - Julia | 2 | **2.1** | 1.5 | 0.8 |\n|  | T-D | D - Lua | 7.7 | **8.2** | 6.3 | 7.2 |\n|  | T-T | Julia - D | 0 | **0.007** | 0 | 0 |\n\n##### Table 2.2 (w/ CodeGen2-16B)\n\n|  |  |  | 0 shot |  |  |  |\n| --- | --- | --- | --- | --- | --- | --- |\n| High-to-High | D-D | Python - JavaScript | **48.4** | 45.1 | 45.1 | 43.1 |\n|  | D-T | JavaScript - Java | **47.3** | 44.3 | 42 | 40 |\n|  | T-D | C++ - Python | 38 | **57.2** | 55.1 |  |\n|  | T-T | Java - C++ | 52.7 |  |  |  |\n| High-to-ExtLow | D-D | JavaScript - Racket | 2 |  |  |  |\n|  | D-T | Python - D | 11.7 | 14.4 | **15.8** | 15 |\n|  | T-D | C++ - Lua | 33.6 |  |  |  |\n|  | T-T | Java - Julia | 26.4 | **27.9** | 25.8 | 25.9 |\n| ExtLow-to-High | D-D | Lua - Python | 37.6 | 45.6 | **47.8** | 46.7 |\n|  | D-T | Racket - Java | **28.4** | 27.8 |  |  |\n|  | T-D | Julia - JavaScript | 40.1 | 39.7 | 40 | **42.9** |\n|  | T-T | D - C++ | **53.1** | 47.6 | 47.1 | 50.4 |\n| ExtLow-toExtLow | D-D | Lua - Racket | 0.8 |  |  |  |\n|  | D-T | Racket - Julia | 13.1 | **14.6** | **14.6** | 14.4 |\n|  | T-D | D - Lua | **29.3** | 28.1 | 28.8 | 28.9 |\n|  | T-T | Julia - D | **10.1** | 7.8 | 5.4 | 5.3 |\n\n\n\nFrom the tables above, we can draw the following additional conclusions, which we will include in the next version of the paper:\n- In weaker models, **we still see improvements with self-generated explanations across most directions**. CodeGen2-1B improves more consistently using self-generated explanations (baseline is the worst in all X-to-X directions, and in 17/18 Python-to-X directions), as much as 300%+ improvement (Table 2.3, Lua→Python, Python→JavaScript). In Python→Ruby, model with explanations obtains a pass rate of 5.9, while baseline does not generate *any* single correct translation (pass rate of 0). CodeGen2-16B shows weaker results, with baseline outperforming in 10/18 directions in Table 1 and 2. Perhaps it has a weaker alignment between natural language and programming language, resulting in worse explanations generated for each problem. The majority of errors from translation with explanation are syntactic. \n- **Better explanation leads to better translation, even in smaller model cases**. We compare CodeGen2-1B performance given self-generated explanation vs. ChatGPT generated explanation and see that the better explanation outperforms self-generated explanation in 12/18 Python-to-X directions, with a maximum improvement of 400%+ in Python → JavaScript.\n- In smaller/weaker models, **detailed explanations (exp-lbl or exp-lbl-d) do not improve as much as exp does**. Often this is due to lower quality explanations generated when the model is asked to do something it is not capable of. The line-by-line explanations often lead to repetitive content when source programs contain several repetitive lines (library import in C++ → Python direction, with CodeGen2-1B)\n- we will add these results and results from StarCoder in the revised version of our paper", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is to my knowledge the first highly detailed study into how self-generated natural language explanations can improve code-to-code translation performance.\n2. This paper provides detailed examples of the various prompts used in the study, which will be helpful for the future reproduction of the results in the study as well as future research in this area.  \n3. This paper provides novel heuristic selection strategies that improve the performance of its explain-then-translate method, as well as showcase the potential for growth and, thus, can be used as a blueprint for future research in the area of code-to-code translation.\n4. This paper provides results of novel ablation experiments that demonstrate how natural language explanations can be more robust and offer better performance when compared to automatically generated programming examples that are used in one-shot learning. \n", "reasons_to_reject": "1. The authors claim to release their code and dataset that they used for this paper, however I cannot find any links to these resources in the paper.\n2. Section 3, where the authors explain some of their methodology on how they are prompting the LLM, is quite vague, as it does not mention what LLM they are using nor does it clearly outline how they are using their explain-then-translate method to prompt the LLM, both of which may make reproducibility more difficult. \n", "questions_to_authors": "Question A: In Section 4.3, why were 0-shot explanations used for exp, while 4-shot explanations were used for exp-lbl and exp-lbl-d? I don’t see a good reason for being inconsistent here.\nQuestion B: Is there any mention in the paper of the compute resources used to run the inference performed in the experiments? Since all of the inference was performed on ChatGPT, perhaps information on the number of API keys used, as well as the time it took to run all the inference would be helpful as well.\nQuestion C: Why were only a subset of all the possible translation directions used when experimenting with alternative translation directions in Section 4.2?\nQuestion D: Why was ChatGPT mentioned as the LLM model used for inference only until the conclusion? I believe that is a relevant detail that could be included much earlier.\n", "paper_topic_and_main_contributions": "This paper investigates whether or not self-generated natural language explanations can improve code-to-code translation performance. The paper proposes an explain-then-translate method and analyzes its effectiveness across 19 different programming languages and 3 types of explanations. It also proposes 5 different heuristics to select better explanations to improve the performance of its explain-then-translate method. Finally, it conducts multiple ablation studies comparing the robustness and effectiveness of natural language explanations to programming language examples, commonly seen in few shot learning. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A\n", "Justification_for_Ethical_Concerns": "N/A\n"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Lines 050-051, …0-shot setting… → …0-shot performance…\nLine 094, …code generation prompt… → …code generation prompts…\nLine 111-115, …over CodeXGLUE… → …over CodeXGLUE and TransCoder…\nLine 491, …Python-Java++-PHP… → …Python-Java-PHP…\n"}}]

Author rebuttal:

Meta-review: This study applies CoT to programming translation tasks. Initially, it generates explanations for the source code and then proceeds to generate the programming code. This approach has demonstrated performance improvements in multiple settings. Reviewers have expressed concerns about the reproducibility of experiments conducted solely with ChatGPT, and the authors have presented additional experiments during the rebuttal to address these concerns. On the other hand, the authors plan to compare with the open source SOTA model in the future.
Decision: Accept-Findings