CRUXEVAL-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution
Authors: Ruiyang Xu, Jialun CAO, Yaojie Lu, Hongyu Lin, Xianpei Han, Ben He, Shing-Chi Cheung, Le Sun
Accepted at: acl25

CRUXEVAL-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution

Ruiyang Xu1,2,*, Jialun Cao3,*, Yaojie Lu1,†, Ming Wen4, Hongyu Lin1, Xianpei Han1,2, Ben He1,2,†, Shing-Chi Cheung3, Le Sun1,2
1Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences
2University of Chinese Academy of Sciences 3The Hong Kong University of Science and Technology
4Huazhong University of Science and Technology
{xuruiyang2022,luyaojie,hongyu,xianpei,sunle}@iscas.ac.cn
{jcaoap, scc}@cse.ust.hk mwenna@hust.edu.cn benhe@ucas.edu.cn

Abstract
Code benchmarks such as HumanEval are widely adopted to evaluate Large Language Models’ (LLMs) coding capabilities. However, there is an unignorable programming language bias in existing code benchmarks – over 95% code generation benchmarks are dominated by Python, leaving the LLMs’ capabilities in other programming languages such as Java and C/C++ unknown. Moreover, coding task bias is also crucial. Most benchmarks focus on code generation capability, while benchmarks for code reasoning (given input, reasoning output; and given output, reasoning input), an essential coding capability, are insufficient. Yet, constructing multi-lingual benchmarks can be expensive and labor-intensive, and codes in contest websites such as Leetcode suffer from data contamination during training. To fill this gap, we propose CRUXEVAL-X, a multi-lingual code reasoning benchmark that contains 19 programming languages. It comprises at least 600 subjects for each language, along with 19K content-consistent tests in total. In particular, the construction pipeline of CRUXEVAL-X works in a fully automated and test-guided manner, which iteratively generates and repairs based on execution feedback. Also, to cross language barriers (e.g., dynamic/static type systems in Python/C++), we formulated various transition rules between language pairs to facilitate translation. Our extensive evaluation of 24 representative LLMs reveals the correlation between language pairs. For example, TypeScript and JavaScript show a significant positive correlation, while Racket has less correlation with other languages. More interestingly, even a model trained solely on Python can achieve at most 34.4% Pass@1 in other languages, revealing the cross-language generalization of LLMs.

1 Introduction
Large language models (LLMs) have shown advanced proficiency in various domains, including code generation (Liu et al., 2024; Du et al., 2024), defect detection (Yang et al., 2024a) and program repair (Xia and Zhang, 2023; Zhong et al., 2024; Hu et al., 2024). Benchmarks such as HumanEval (Chen et al., 2021) and SWE-bench (Jimenez et al., 2023) were introduced to measure LLMs’ capabilities, providing insights into LLMs’ strengths and weaknesses.

Recent studies (Cao et al., 2024a; Chai et al., 2024; Chen et al., 2024) have spotted two significant biases in current benchmarks. First, Programming language bias. As pointed out by prior studies (Cao et al., 2024a; Chai et al., 2024; Chen et al., 2021; Austin et al., 2021), Python dominates code generation benchmarks with over 95% involvement. Other programming languages (PLs) such as Java and C/C++, despite their popularity and availability, gain less exploration. Second, Coding task bias. Most coding benchmarks focus on code generation tasks (i.e., giving descriptions in natural language and generating the program, as shown in Figure 1 (A)), while code reasoning (i.e., given

![Figure 1: Code generation vs. Code reasoning](page_820_370_601_370.png)

Figure 1: Code generation vs. Code reasoning
the program, reasoning the input or output of the program, as shown in Figure 1 (B - D)) is seldom evaluated (Chen et al., 2024). A recent work introduced a code reasoning benchmark (Gu et al., 2024; Chen et al., 2024), while it is only in Python. Figure 1 (C – D) shows that simply changing PLs from Python to C++ can turn a correct reasoning into an incorrect one. More discussion about this task can be found in Appendix J.

However, constructing multi-lingual benchmarks is not a trivial task. First, human annotation can be expensive. As reported by recent work (Chai et al., 2024), they spent a total of $12,000 US dollars for human annotators, providing the working environment, free meals, souvenirs, and free GPT-4 interface usage to construct their multi-lingual benchmark. Second, automated translation does not perform well. The latest studies (Yin et al., 2024) show that even the best LLM (i.e., ChatGPT) can only achieve an average of 64% success translation rate, which is far from practice. Rule-based translation (Cassano et al., 2023; Ling et al., 2022) usually suffers from generalizability issues, making them limited in handling prescribed code structures. Additionally, multi-lingual solutions from contest websites such as LeetCode and Codeforces were included in most LLMs training sources, thus suffering from data contamination issues (Cao et al., 2024b).

To fill the above research gaps, we introduce CRUXEVAL-X, a multi-lingual code reasoning benchmark that contains 19 popular PLs, including C++, Rust, Java, etc., expanded from CruxEval (Gu et al., 2024), a code reasoning benchmark written in Python. For each PL in CRUXEVAL-X, there are at least 600+ functions. Additionally, there are 500 samples that are fully aligned across all PLs. In total, there are 12,660 subjects along with 19K test cases for input/output reasoning.

Noteworthy that the pipeline of constructing CRUXEVAL-X works in a fully automated manner. It first translates the test cases by transition rules adapted from prior work (Cassano et al., 2023), then iterates the generation-and-repair process intensively. In particular, the transition rules are formulated to cross the language barriers. For example, Python employs a dynamically typed system where types are determined at runtime, whereas C++ uses a statically typed system requiring explicit type declarations at compile time. The rules facilitate the translation of the test cases. Additionally, inspired by prior work (Yin et al., 2024; Rozière et al., 2022), we employ a test-guided manner (Rozière et al., 2022) to generate the translation and iteratively repair the generated code using execution feedback (e.g., compilation error, runtime error) (Yin et al., 2024).

Through extensive experiments on 24 mainstream LLMs, we observe several interesting findings. First, in multiple PLs, the input reasoning and output reasoning capabilities of LLMs are comparable. Also, there is a noticeable correlation between certain PLs (e.g., JavaScript and TypeScript show a positive correlation, while Racket consistently yields the worst results). More interestingly, we observe that even if a model is only trained on Python (e.g., phi-1 and phi-1.5), it still can reach a 16% ~ 26% output reasoning success rate in other PLs, compared with 25.6% in Python. The finding indicates the cross-language generalization of LLMs.

The contributions can be summarized as follows: (1) We introduce CRUXEVAL-X, a multi-lingual code reasoning benchmark that contains 19 popular PLs. (2) We introduce an automated code translation pipeline that adopts a test-guided and iterative generate-and-repair practice. (3) We evaluate 20+ LLMs and yield inspiring findings.

2 Benchmark Construction

In this section, we describe the construction process of CRUXEVAL-X, illustrated in Figure 2, which is adapted from CRUXEVAL. It can be divided into three main steps. First, we translate function signatures via mapping variable type annotations (Step 1 in Figure 2). Then we employ a rule-based approach to translate Python test cases into other PLs (Step 2 in Figure 2). Finally, we integrate multiple LLMs to translate code by iterating the generation-and-repair process (Step 3 in Figure 2).

2.1 Step I. Function Signature Translation

To enhance the accuracy and standardization of function translation results, we first translate the function signatures and dependencies. Note that Python does not require an explicit type annotation, which may confuse the translation for the function signature. For example, as shown in Figure 2 Step I, the types of two input parameters (i.e., s1 and s2) are unclear. So we extract the input variables from the function signature using the syntax tree and match them with the tests. Based on the variable types in the tests, we annotate the input and output
variables in the function signature.

Then, we adopt the rules as prior work (Cassano et al., 2023) to map the types from Python to other PLs. In particular, we identify the data types in the annotated Python signature (e.g., parameter types, return types), mapping the types from Python to other PLs according to the rules, then structuring the signature in the corresponding PLs. Take the example in Figure 2 Step I, the Python signature def f(s1:str, s2:str) -> str is translated into that in C++ (std::string f(std::string s1, std::string s2)). After translating the tests, all 800 subjects in Python can be translated, as shown in Table 1.

2.2 Step II. Test Suites Translation

We employ a test-guided approach to ensure the correctness of the translation results, which necessitates test cases in various PLs. Prior works (Athiwaratkun et al., 2022; Cassano et al., 2023) provided various rules for mapping Python test cases to other PLs. We adopt the mapping rules from MutiPL-E (Cassano et al., 2023) to assist the transition of our test suites.

However, their rules have limited support for type handling (e.g., they cannot handle a list with hybrid types). Thus, to maximize the success rate, we made two improvements to enhance the rules. First, we enhance handling structured types such as List and Dict. For example, when handling C#, we add an equality function to check whether two Dict types are equal. Second, when dealing with variables that have complex types that are not as well-supported in some other PLs, we transform these variables into more generic types without significantly altering the original function’s functionality. For example, we change type List[Union(str, int)] into List(str) if the function keeps the same functionality. A small portion of the data that cannot be converted is discarded. The result of Step II is shown in Table 1. Further details can be found in the Appendix B.

2.3 Step III. Iterative Generation & Repair

After translating tests, dependencies, and signatures, we employ multiple LLMs to iteratively translate Python code into target PLs. Each LLM performs two steps: generation and repair. Problems that pass all test cases are excluded from subsequent iterations. Below, we will elaborate on these two substeps in detail.

2.3.1 Generation

Relying on a single LLM’s limited generations often results in a low translation success rate (Yin et al., 2024). To address this, we propose a multi-round generation method that interacts with the testing environment to determine whether to continue iterating.

Let \( A_0 \) denote the number of correctly translated codes, and \( U \) the total number of problems. For unresolved problems, LLM \( M \) generates results over multiple rounds. Let \( A_i \) represent the number of correct results in the \( i \)-th round. The maximum rounds is \( N \). Early stopping occurs if the increase in correct results compared to \( k \) rounds prior, \( A_{i-k} \),
falls below a threshold \( \delta \). The formula is as follows:

\[
A_i = Correct(P(O_i | U - A_{i-1}; M)) + A_{i-1}
\]
for \( i \in \{1, 2, \ldots, N\} \)

Stop if (\( i > k \)) and (\( A_i - A_{i-k} < \delta \))

Here, \( O_i \) is the \( i \)-th round code generated from \( P(O_i | U - A_{i-1}; M) \), and \( Correct(.) \) calculate the correct results. To leverage diverse LLMs’ strengths, we use GPT-3.5-Turbo for initial generation (see Table 1, “w/o Iter”) and DeepseekCoder-33B-Instruct for further generation. This pipeline improves upon single-LLM generation, as demonstrated in our final results. (see Table 1, “w/ Iter”)

2.3.2 Repair
Simply generating code will still result in many errors that LLMs cannot solve. Therefore, after the generation step of each LLM, we provide them with error messages for error correction. Multiple iterative error correction is costly with limited benefits. (Chen et al., 2023), so we only perform error correction once after each LLMs.

After GPT3.5-Turbo generation, we directly provide the LLM with the erroneous code along with the error messages for correction. After the Iterating Generation of DeepseekCoder-33B-Instruct, since the untranslated code can produce numerous incorrect code snippets after multiple rounds of iteration, which may contain the same errors. We first use simhash to deduplicate the erroneous code and then proceed with error correction on the deduplicated code. The error correction in each phase uses the LLM used in that phase.

2.3.3 Multiturn Repair Based on Overlap
After completing the steps above, we first calculate the intersection of correctly answered questions in different PLs. To our surprise, there were only 333 questions that all PLs answered correctly. However, 563 problems have been successfully translated correctly by at least 16 PLs. Upon analyzing the questions our LLM failed to solve, we find that each PL has its difficulties in translating from Python. For example, in Julia, the index for arrays and other collection types starts from 1, which differs from Python. The details of the difficulties can be found in Appendix C.

Based on these observations, we conduct final generation and iterative error correction on the questions that are correctly translated by more than 15 PLs. Due to the small amount of data, we utilized GPT-4o for generation and error correction. During the generation process, we provide the LLM with three corresponding typical examples based on the difficulties we find. The overlap is increased to 462 after repair of GPT-4o.

We manually refined 38 questions that GPT-4 nearly solved, expanding our dataset to 500 entries. We determined that 500 entries are sufficient to distinguish the effectiveness of the LLMs. Therefore, we use these 500 entries as our CRUXEVAL-X benchmark. The final result of our pipeline is shown in Table 1 under the column “w/ Iter”. The experimental setups and prompts can be found in Appendix D, and detail statics of our benchmark is shown in Appendix I.

2.4 Quality Analysis
After constructing the dataset, we evaluate the benchmark’s quality from four perspectives. First, Accuracy: We use a test-guided approach to ensure that each data point passes test cases, guaranteeing correctness. Second, Reliability: We build on the high-quality CRUXEVAL dataset, ensuring problems avoid randomness or multi-threading, enhancing reliability. Third, Data Leakage: We compare with Stack v2 (67.5TB of GitHub data), finding only 0.8% overlap, indicating minimal leakage risk. Fourth, Data Bias: We assess potential bias by generating evaluation data with other LLMs using the same method as described above. Results show no significant advantage for any LLM on its own generated data. The possible reason is the distinct tasks in generation and evaluation phases. Detailed results and analysis are in Appendix A.

3 Experiments
3.1 Experiment Setup
LLMs for evaluation We selected 24 LLMs across 4 types for evaluation, including general LLMs (GPT-3.5-Turbo, GPT-4o-mini, GPT-4o (Brown et al., 2020; Achiam et al., 2023), Llama3 (AI, 2024), Qwen2 (Yang et al., 2024b), phi-3-instruct (Abdin et al., 2024)), multilingual code1 LLMs (Deepseekcoder-V2 (Zhu et al., 2024), Deepseekcoder-V1 (Guo et al., 2024), CodeLlama (Roziere et al., 2023), Starcoder (Li et al., 2023a), Starcoder2 (Lozhkov et al., 2024), CodeQwen1.5-Chat (Bai et al., 2023)), instruction-tuned multilingual LLMs

1In this paper, “multilingual” refers to multiple programming languages.
<table>
  <tr>
    <th>Languages</th>
    <th>Step I</th>
    <th>Step II</th>
    <th colspan="2">Step III</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th>w/o Iter</th>
    <th>w/ Iter</th>
  </tr>
  <tr>
    <td>C# (cs)</td>
    <td>800</td>
    <td>774</td>
    <td>380</td>
    <td>670</td>
  </tr>
  <tr>
    <td>C++ (cpp)</td>
    <td>800</td>
    <td>800</td>
    <td>549</td>
    <td>733</td>
  </tr>
  <tr>
    <td>D (d)</td>
    <td>800</td>
    <td>754</td>
    <td>95</td>
    <td>629</td>
  </tr>
  <tr>
    <td>GO (go)</td>
    <td>800</td>
    <td>752</td>
    <td>293</td>
    <td>699</td>
  </tr>
  <tr>
    <td>Java (java)</td>
    <td>800</td>
    <td>774</td>
    <td>541</td>
    <td>698</td>
  </tr>
  <tr>
    <td>JavaScript (js)</td>
    <td>800</td>
    <td>800</td>
    <td>634</td>
    <td>743</td>
  </tr>
  <tr>
    <td>Julia (jl)</td>
    <td>800</td>
    <td>774</td>
    <td>410</td>
    <td>680</td>
  </tr>
  <tr>
    <td>Lua (lua)</td>
    <td>800</td>
    <td>800</td>
    <td>582</td>
    <td>741</td>
  </tr>
  <tr>
    <td>Perl (pl)</td>
    <td>800</td>
    <td>799</td>
    <td>591</td>
    <td>728</td>
  </tr>
  <tr>
    <td>PHP (php)</td>
    <td>800</td>
    <td>800</td>
    <td>622</td>
    <td>755</td>
  </tr>
  <tr>
    <td>R (r)</td>
    <td>800</td>
    <td>800</td>
    <td>542</td>
    <td>699</td>
  </tr>
  <tr>
    <td>Racket (rkt)</td>
    <td>800</td>
    <td>800</td>
    <td>264</td>
    <td>681</td>
  </tr>
  <tr>
    <td>Ruby (rb)</td>
    <td>800</td>
    <td>800</td>
    <td>658</td>
    <td>748</td>
  </tr>
  <tr>
    <td>Rust (rs)</td>
    <td>800</td>
    <td>754</td>
    <td>449</td>
    <td>690</td>
  </tr>
  <tr>
    <td>Scala (scala)</td>
    <td>800</td>
    <td>799</td>
    <td>462</td>
    <td>712</td>
  </tr>
  <tr>
    <td>Shell (sh)</td>
    <td>800</td>
    <td>763</td>
    <td>528</td>
    <td>674</td>
  </tr>
  <tr>
    <td>Swift (swift)</td>
    <td>800</td>
    <td>796</td>
    <td>415</td>
    <td>654</td>
  </tr>
  <tr>
    <td>TypeScript (ts)</td>
    <td>800</td>
    <td>774</td>
    <td>592</td>
    <td>726</td>
  </tr>
</table>

Table 1: The result of each step. The portion within parentheses in the “Language” column represents the abbreviations for various languages. Due to the constraints of page size, these abbreviations are used to better display certain charts or tables.

(Deepseekcoder-instruct-V1 (Guo et al., 2024), WizardCoder (Luo et al., 2023), CodeLlama-Python, CodeLlama-Instruct (Roziere et al., 2023)), and single or few-language code LLMs (CodeGen (Nijkamp et al., 2022), phi-1 (Gunasekar et al., 2023), phi-1.5 (Li et al., 2023b)).

Evaluation task We adopt the task settings from prior work (Gu et al., 2024), dividing the tasks into output reasoning and input reasoning. For any PLs dataset, we provide the code along with the corresponding input or output in the test cases. Input reasoning is predicting the input based on the output. Output reasoning is predicting the output based on the input. Each problem contains a test case, which follows the CRUXEVAL. The search space for test cases is exponential in nature, ensuring that LLMs do not know the answer to the problem in advance.

Evaluation method. We use pass@1 (Kulal et al., 2019; Chen et al., 2021) to evaluate both tasks. We set the temperature to 0 and employ greedy decoding for generation as prior work (Cao et al., 2024a). For closed-source LLMs, we use gpt-3.5-turbo, gpt-4o-mini, and gpt-4o from OpenAI’s API.

3.2 Overall Result

Table 2 shows Pass@1 evaluation results of various LLMs arranged in descending order of the LLMs’ parameter size. We can see that: 1) CRUXEVAL-

X is challenging for all tested LLMs. Even GPT-4o can only achieve Pass@1 around 70%. Notably, the result of the open-source LLMs Deepseekocoder-V2 is better than GPT-4o-mini, with an average Pass@1 of 62.8% on input reasoning and 65.0% on output reasoning. Furthermore, the average results show that across different types of PLs, LLMs exhibit similar capabilities in input and output reasoning, consistent with prior observations in Python (Gu et al., 2024). 2) Single-programming-language model (e.g., phi-1) and few-language models (e.g. CodeGen-muti) demonstrate unexpected generalization capabilities on unseen language tasks. During evaluation, we introduced few-language LLMs (CodeGen-multi, trained on six PLs) and single-language LLMs (phi-1.5 and phi-1, trained only on Python). Despite this, they achieved similar results across 19 PLs. Notably, phi-1, trained solely on Python, scored 11.8% Pass@1 on Python input prediction and 23.6% on Perl. We will provide a more detailed analysis of this phenomenon in the Analysis section.

4 Analysis

4.1 Key Factors for LLM Code Reasoning

To get more insight into what factors in code affect LLMs’ code reasoning ability, we explore six factors (e.g., average number of input variables, average input length) and statistics their correlation with correct/incorrect reasoning. The results are shown in Figure 3. Each column of the box plot displays the distribution of 19 PLs. In particular, the columns “Num of Input Variable”, “Num of Variable Type” are counted by averaging the number of types of input parameters in method signatures. “Input/Output Length” is the average string length of the input/output. For instance, in the test case where f(‘a’, 123) == {'a': 1}, the input length is 6, and the output length is 8. “Whether Have Complex Type” checks whether there are List, Dict, Tuple, Set types in input and output. “Execution Steps” calculates the average execution steps in Python bytecode operations, following prior work (Gu et al., 2024).

From Figure 3 (A - C ), we can see that the number/types of input variables have little impact on the code reasoning, especially Sub-figure B, which shows that the reasoning capability is slightly better when more types of variables are involved. A more counter-intuitive observation is made from Figure 3 (D - E ). They indicate that the reasoning
Table 2: The result of each LLM in CRUXEval-X. Each result is shaded with a background color from blue to white based on the Pass@1. The bluer, the larger. Deepseekcoder-V2 is a MOE LLM; the parameter activated during inference is 21B. All results are tested on 500 aligned data.

capability is negatively correlated with the length of input/output strings.

Furthermore, regarding input reasoning capability, the more input variables, the more challenging it is for LLMs to reason about the correct inputs, thus the worse the input reasoning performance (Sub-figures A and D).

4.2 Cross-language Generalization Ability

To investigate the cross-language generalization ability of LLMs, we investigate the reasoning ability of phi-1 and phi-1.5, which are trained on English and Python only. To get a better understanding, we analyze the capability in terms of syntax and semantics in 9 PLs because they provide clear error messages to distinct syntactic/semantic errors.

4.2.1 Syntactic Correctness

For a LLM trained on single PL, ensuring syntactic correctness is crucial for its generalization to other programming languages. For example, in a Racket language test case, the expected output might be
"(list 5 8 1 3 0)", but the LLM incorrectly predicts "[5,8,1,3,0]", leading to a compilation error. Figure 4 shows the number of syntactic-correct cases made by these two LLMs in both tasks. It is clear that Python has the highest syntactic correctness in both tasks, followed by Go and TypeScript. On the contrary, C++, C#, and Java witness the most syntactic errors for these LLMs. Interestingly, even though phi-1 and phi-1.5 have not trained on PLs other than Python, they can still achieve an average of 49.1% and 72.0% syntactically correctness rate in other PLs, respectively, compared with 97.0% and 98.7% achieved on Python. It indicates the cross-language generalizability of LLMs.

4.2.2 Semantic Correctness

Beyond syntactic correctness, semantic correctness poses higher requirements, i.e., passing the tests. The results are shown in the last two rows in Table 2. In particular, phi-1.5 reaches 25.8% input reasoning performance on Python, while on other PLs, an average of 19.0% can also be reached. The observation further consolidates the cross-language generalizability of LLMs.

4.2.3 Cross-NL and Cross-PL Generalization

From Figure 4 and Table 2, there is a noticeable increase from phi-1 to phi-1.5 (an average of 10.7% vs. 21.7% on input reasoning, and 15.1% vs. 21.7% on output reasoning). According to the description (Abdin et al., 2024), phi-1.5 is further fine-tuned with more synthetic texts in natural language (NL). Considering the dramatic improvement in code reasoning, it is highly likely that the improvement in NL reasoning positively impacts code reasoning.

Figure 3: Key factors for LLM code reasoning capability.

Figure 4: The syntax accuracy of each LLM.

4.3 Programming Language Correlation

To further investigate the correlations between these 19 PLs in CRUXEVAL-X, we calculate each PL pair's correlation (i.e., cosine similarity, ranging from -1 to 1), visualized in Figure 5. In particular, for each PL, we flatten the results of LLMs as a feature vector and calculate the cosine similarities for each pair of PLs.

Overall, Figure 5 shows that the correlation between PL pairs is generally similar, with an average of 0.7+ cosine similarities. Among all PL pairs, JavaScript and TypeScript correlate the most strongly (0.87 and 0.91 on both tasks). It indicates that the code reasoning capabilities on different PLs are highly correlated. Also, the correlation in output reasoning is slightly higher than in input reasoning, with an average of 0.79 vs. 0.75.

It is also noteworthy that Racket has the most minor correlation with all the other PLs. It may be because of its distinct syntax. A case study can be found in Listing 3.

Figure 5: The correlation between PL pairs.

4.4 Case Study

After identifying the phi-series-LLMs (i.e., phi-1, phi-1.5) exhibit cross-language generalization and the correlation across PLs, we further analyze the predictions of phi-1.5 to get a deeper understanding. We noticed that out of the 128 correct instances in Python by phi-1.5, 61.7% (79/128) are also correct in PHP, while only 39.8% (51/128) are correct in
Racket. Therefore, we use one example in these three PLs to understand its rationale.

Listing 1: Subject-164 (Python)
def f(lst: List[int]) -> List[int]:
    lst.sort()
    return lst[0:3]
# input: [5, 8, 1, 3, 0]
# phi-1.5 answer: [0, 1, 3]

Listing 2: Subject-164 (PHP)
<?php
function f($lst) {
    sort($lst);
    return array_slice($lst, 0, 3);
}
// input: array(5, 8, 1, 3, 0)
// phi-1.5 answer: array(0, 1, 3)

Listing 3: Subject-164 (Racket)
(define (f lst)
    (define sorted-list (sort lst <))
    (take sorted-list 3))
;; input: (list 5 8 1 3 0)
;; phi-1.5 answer: (list 5 8 1)

4.4.1 Analysis on Subject 164
Listing 1-3 demonstrates an instance where phi-1.5 generalizes Python (Listing 1)’s reasoning capabilities to other languages. From a grammar structure perspective, their respective function definitions, indentation formats, and the functions they invoke exhibit significant differences. However, overall, PHP and Python share a more similar structure, both utilizing sort for sorting and return for returning output values. Therefore, phi-1.5 is able to generalize its code reasoning abilities to PHP, but fails to comprehend the sorting command in Racket, leading to incorrect predictions.

Upon analyzing these 128 questions, we observe that excluding those where the output could be directly derived from the input, such as assert f("zej","owc") == "zej", which accounted for approximately 40% of the cases, there are still numerous examples demonstrating that phi-1.5 has developed a certain level of cross-language capabilities. From these examples, we can observe that the multilingual generalization capability of the model is positively correlated with the grammar structural similarity between languages. Even Racket, a language significantly different from others, maintains certain logical similarities in aspects such as function definitions, loops, and conditional branches. This is a key reason why Phi-1.5 can achieve considerable effectiveness across multiple languages. More case studies can be found in appendix K.

5 Related Work

Multi-Task Code Benchmark. Recently, there has been an increasing number of tasks related to code that are used to evaluate the various capabilities of LLMs in the field of coding, including code generation (Chen et al., 2021; Austin et al., 2021), code repair (Jimenez et al., 2023; Tian et al., 2024), and code description (Chai et al., 2024). However, datasets that assess the reasoning abilities of code are relatively limited, and the currently proposed reasoning datasets are confined to the Python language (Gu et al., 2024; Chen et al., 2024). In this work, we expand the Python language reasoning dataset CRUXEVAL (Gu et al., 2024) to encompass 19 PLs, thereby addressing the deficiency in reasoning datasets at the multilingual level.

Multi-Language Code Benchmark. Multilingual evaluation datasets are important for assessing the comprehensive coding capabilities of code LLMs. In the early stages, multilingual code datasets were mainly used for code translation tasks (Elnaggar et al., 2021; Ahmad et al., 2021; Roziere et al., 2020, 2021; Zhu et al., 2022b; Yan et al., 2023; Zhu et al., 2022a). These datasets often consist of problem solutions in different languages extracted from algorithm competition-related websites, thus suffering from data contamination issue. Benchmark like McEval (Chai et al., 2024), which relies on human annotation, requires a high cost. In this work, we provide a process using LLMs for multilingual code translation, which can achieve a high accuracy and low cost in creating a multilingual dataset.

6 Conclusion

In this work, we provide a fully automated process for constructing a multilingual dataset based on a Python code language dataset. Through this process, we successfully transform the CRUXEval dataset into a multilingual dataset containing 19 PLs and test its effectiveness on 24 LLMs, demonstrating the validity of the dataset. Furthermore, we find that models trained on only a few languages exhibit the ability to transfer their prediction capabilities to other languages in input/output reasoning tasks, and this ability is influenced by the model’s own reasoning capabilities.
7 Limitations

There are Three limitations in this work. First, our benchmark relies on model-generated data. While this approach effectively reduces costs and avoids issues of data leakage, it may introduce biases in the dataset towards the LLMs that generated the data. However, we select different LLMs to construct evaluation data and conduct experiments on this data. The experimental results indicate that our dataset construction method does not affect the fairness of the dataset. Additionally, the model-based translation method cannot guarantee that all data will be perfectly translated into other PLs. However, through our method, based on 800 pieces of data, we generated 500 aligned pieces of data for 19 PLs. From our evaluation results, it can be seen that this amount of data is sufficient to effectively distinguish the reasoning capabilities of each LLM. Finally, translated codes from Python cannot reflect several specific language features of other languages. Our intention is to achieve highly aligned multilingual data across 19 languages. However, preserving language-specific features make it challenging to create such aligned data, as the distinctive features of one language are inherently difficult to replicate in others. This represents a trade-off. In our future work, we plan to explore how to construct multilingual data while retaining more language-specific features.

8 Acknowledgements

We sincerely thank the reviewers for their insightful comments and valuable suggestions. This work was supported by Beijing Natural Science Foundation (L243006), Beijing Municipal Science and Technology Project (Nos. Z231100010323002), the Natural Science Foundation of China (No. 62272439, 62306303, 62476265) and the Basic Research Program of ISCAS (Grant No. ISCAS-ZD-202402).

References

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Wasi Uddin Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, and Kai-Wei Chang. 2021. Avatar: A parallel corpus for java-python program translation. arXiv preprint arXiv:2108.11590.

Meta AI. 2024. Introducing meta llama 3: The most capable openly available llm to date. Blog. Online; accessed 15-January-2024.

Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, et al. 2022. Multi-lingual evaluation of code generation models. arXiv preprint arXiv:2210.14868.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Jialun Cao, Zhiyong Chen, Jiarong Wu, Shing chi Cheung, and Chang Xu. 2024a. Can ai beat undergraduates in entry-level java assignments? benchmarking large language models on javabench. Preprint, arXiv:2406.12902.

Jialun Cao, Wuqi Zhang, Shing-Chi Cheung, and So on. 2024b. Concerned with data contamination? assessing countermeasures in code language model. Preprint, arXiv:2403.16898.

Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. 2023. Multipl-e: a scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering, 49(7):3675–3691.

Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, Ke Jin, Jiacheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, et al. 2024. Mceval: Massively multilingual code evaluation. arXiv preprint arXiv:2406.07436.

Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. 2024. Reasoning runtime behavior of a program with llm: How far are we? Preprint, arXiv:2403.16437.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

Xinyun Chen, Maxwell Lin, Nathanael Schärlí, and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128.

Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2024. Evaluating large language models in class-level code generation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13.

Ahmed Elnaggar, Wei Ding, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Silvia Severini, Florian Matthes, and Burkhard Rost. 2021. Code-trans: Towards cracking the language of silicon’s code through self-supervised deep learning and high performance computing. arXiv preprint arXiv:2104.02443.

Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. 2024. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065.

Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojgan Javaheripi, Piero Kaufmann, Gustavo de Rosa, Olli Saarikivi, et al. 2023. Textbooks are all you need. arXiv preprint arXiv:2306.11644.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming—the rise of code intelligence. arXiv preprint arXiv:2401.14196.

Xueyu Hu, Kun Kuang, Jiankai Sun, Hongxia Yang, and Fei Wu. 2024. Leveraging print debugging to improve code generation in large language models. arXiv preprint arXiv:2401.05319.

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770.

Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. 2019. Spoc: Search-based pseudocode to code. Advances in Neural Information Processing Systems, 32.

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023a. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161.

Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023b. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463.

Michael Ling, Yijun Yu, Haitao Wu, Yuan Wang, James R. Cordy, and Ahmed E. Hassan. 2022. In rust we trust – a transpiler from unsafe c to safer rust. In 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pages 354–355.

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36.

Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhatar, Jiawei Liu, Yuxiang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173.

Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardocoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568.

Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474.

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950.

Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. Advances in neural information processing systems, 33:20601–20611.

Baptiste Rozière, Jie Zhang, François Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. 2022. Leveraging automated unit tests for unsupervised code translation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.

Baptiste Rozière, Jie M Zhang, Francois Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. 2021. Leveraging automated unit tests for unsupervised code translation. arXiv preprint arXiv:2110.06773.

Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. Debugbench: Evaluating debugging capability of large language models. arXiv preprint arXiv:2401.04621.
Chunqiu Steven Xia and Lingming Zhang. 2023. Keep the conversation going: Fixing 162 out of 337 bugs for $0.42 each using chatgpt. arXiv preprint arXiv:2304.00385.

Weixiang Yan, Yuchen Tian, Yunzhe Li, Qian Chen, and Wen Wang. 2023. Codertransocean: A comprehensive multilingual benchmark for code translation. arXiv preprint arXiv:2310.04951.

Aidan ZH Yang, Claire Le Goues, Ruben Martins, and Vincent Hellendoorn. 2024a. Large language models for test-free fault localization. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pages 1–12.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024b. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

Xin Yin, Chao Ni, Tien N. Nguyen, Shaohua Wang, and Xiaohu Yang. 2024. Rectifier: Code translation with corrector via llms. Preprint, arXiv:2407.07472.

Li Zhong, Zilong Wang, and Jingbo Shang. 2024. Ldb: A large language model debugger via verifying runtime execution step-by-step. arXiv preprint arXiv:2402.16906.

Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, and Chandan K Reddy. 2022a. Xlcost: A benchmark dataset for cross-lingual code intelligence. arXiv preprint arXiv:2206.08474.

Ming Zhu, Karthik Suresh, and Chandan K Reddy. 2022b. Multilingual code snippets training for program translation. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 11783–11790.

Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. 2024. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931.

A Dataset Bias for Data-Generating LLM

The experimental results of existing papers have already demonstrated that the LLMs used for generate CRUXEVAL benchmark do not introduce unfairness (Gu et al., 2024). Therefore, our goal is to verify whether our pipeline can lead to unfairness on datasets of other programming languages (PLs). To achieve this, we reduce the number of iterations and use a single model for data generation, aiming to determine if different models can gain an advantage on the data they generated themselves.

We select Qwen2.5-Coder-32B-Instruct (Qwen 32B), deepseek-coder-33b-instruct (Dpk 33B), and deepseek-coder-6.7b-instruct (Dpk 7B) to generate data respectively, and reduce the maximum number of iterations to 5. We generated data for both commonly used PLs C++ and less commonly used PLs Racket. After taking the intersection, we obtain 566 aligned C++ data points and 206 aligned Racket data points. We then calculate the pass@1 for these three models, with the specific results shown in Table 3.

<table>
  <tr>
    <th colspan="4">Input Reasoning (C++)</th>
  </tr>
  <tr>
    <th>Evaluate</th>
    <th>Qwen 32B</th>
    <th>Dpk 33B</th>
    <th>Dpk 7B</th>
  </tr>
  <tr>
    <td>Qwen 32B</td>
    <td style="background-color:#c6efce;">76.3</td>
    <td style="background-color:#c6efce;">49.6</td>
    <td style="background-color:#c6efce;">38.3</td>
  </tr>
  <tr>
    <td>Dpk 33B</td>
    <td style="background-color:#c6efce;">75.6</td>
    <td style="background-color:#c6efce;">47.5</td>
    <td style="background-color:#c6efce;">39.2</td>
  </tr>
  <tr>
    <td>Dpk 7B</td>
    <td style="background-color:#c6efce;">76.3</td>
    <td style="background-color:#c6efce;">48</td>
    <td style="background-color:#c6efce;">37.6</td>
  </tr>
  <tr>
    <th colspan="4">Output Reasoning (C++)</th>
  </tr>
  <tr>
    <th>Evaluate</th>
    <th>Qwen 32B</th>
    <th>Dpk 33B</th>
    <th>Dpk 7B</th>
  </tr>
  <tr>
    <td>Qwen 32B</td>
    <td style="background-color:#c6efce;">65.5</td>
    <td style="background-color:#c6efce;">53.0</td>
    <td style="background-color:#c6efce;">42.6</td>
  </tr>
  <tr>
    <td>Dpk 33B</td>
    <td style="background-color:#c6efce;">67.7</td>
    <td style="background-color:#c6efce;">51.2</td>
    <td style="background-color:#c6efce;">44.3</td>
  </tr>
  <tr>
    <td>Dpk 7B</td>
    <td style="background-color:#c6efce;">67.1</td>
    <td style="background-color:#c6efce;">52.6</td>
    <td style="background-color:#c6efce;">43.2</td>
  </tr>
  <tr>
    <th colspan="4">Input Reasoning (Racket)</th>
  </tr>
  <tr>
    <th>Evaluate</th>
    <th>Qwen 32B</th>
    <th>Dpk 33B</th>
    <th>Dpk 7B</th>
  </tr>
  <tr>
    <td>Qwen 32B</td>
    <td style="background-color:#c6efce;">82.0</td>
    <td style="background-color:#c6efce;">51.9</td>
    <td style="background-color:#c6efce;">42.7</td>
  </tr>
  <tr>
    <td>Dpk 33B</td>
    <td style="background-color:#c6efce;">77.2</td>
    <td style="background-color:#c6efce;">49.0</td>
    <td style="background-color:#c6efce;">37.9</td>
  </tr>
  <tr>
    <td>Dpk 7B</td>
    <td style="background-color:#c6efce;">78.6</td>
    <td style="background-color:#c6efce;">42.2</td>
    <td style="background-color:#c6efce;">41.3</td>
  </tr>
  <tr>
    <th colspan="4">Output Reasoning (Racket)</th>
  </tr>
  <tr>
    <th>Evaluate</th>
    <th>Qwen 32B</th>
    <th>Dpk 33B</th>
    <th>Dpk 7B</th>
  </tr>
  <tr>
    <td>Qwen 32B</td>
    <td style="background-color:#c6efce;">78.6</td>
    <td style="background-color:#c6efce;">58.7</td>
    <td style="background-color:#c6efce;">47.6</td>
  </tr>
  <tr>
    <td>Dpk 33B</td>
    <td style="background-color:#c6efce;">76.2</td>
    <td style="background-color:#c6efce;">57.8</td>
    <td style="background-color:#c6efce;">46.1</td>
  </tr>
  <tr>
    <td>Dpk 7B</td>
    <td style="background-color:#c6efce;">79.1</td>
    <td style="background-color:#c6efce;">56.8</td>
    <td style="background-color:#c6efce;">44.7</td>
  </tr>
</table>

Table 3: Impact of data generating LLMs
It can be observed that the relative ordering of the three models does not change. Moreover, the model used for data generation does not significantly impact the final evaluation results. For instance, in the Input Reasoning task for C++, Qwen 32B’s pass@1 consistently remain around 76.

We analyze the possible reason: During both the data generation phase and the evaluation phase, the tasks performed by the LLMs are different. We use the model to generate code in different languages (i.e., code translation), while we evaluate the model’s ability to infer outputs/inputs based on given inputs/outputs in code (i.e., code reasoning). It is important to note that the test cases used during the code reasoning phase are not visible to the model during the code translation phase, ensuring a fair evaluation process. Therefore, the model is performing two distinct tasks (i.e., code translation and code reasoning), and there is no unfairness involved.

B The Improvements of Pipline

B.1 Enhance of the Pipline

(1) For C# we enhance the check function to include the ability to judge if the List and Dict types are equal. (2) For Julia we add the data type to empty dict, for example change input Dict() to Dict{String, String}(). Otherwise, the function can not accept the input (3) For JavaScript we change String which Contains special characters, such as"example\n" to `example\n `.

B.2 Transform Complex Types

(1) For inputs and outputs that include functions, such as "bfrerat".split("-"), we will replace the input or output with the result after the function execution. (2) When the input is of Callable type, such as lambda x: x.reverse(), we will remove the parameter and incorporate the Callable type into the main function internally. (3) For complex variables that contain multiple types, if we can convert them into a simpler type without altering the function’s functionality, we will preserve such functions. For instance, as illustrated in Listing 4, consider a dictionary d: Dict[str, Union[int, str]]. If converting all its values to the str type does not alter the function’s behavior, we will retain it; otherwise, we will discard them.

Listing 4: An example with complex type
from typing import Dict, Union, Tuple
def f(d: Dict[str, Union[int, str]]) -> Tuple[bool, bool]:
    r = {
        "c": d.copy(),
        "d": d.copy()
    }
    return (r["c"] is r["d"], r["c"] == r["d"])
def check(candidate):
    assert candidate({"i": 1, "love": "parakeets"}) == (False, True)
def test_check():
    check(f)

C The Difficulties for Translation

(1) For indexing functions, the starting position is not 0 but 1, with Julia being a typical language that exhibits this behavior. An example is shown in listing 5. (2) For the conversion between Python’s str type and its own char and string types, D language is a typical case where this issue arises. An example is shown in listing 6. (3) For the transformation and comparison of dictionary types. For example, C# need to first sort two Dictionary before compare them.

Listing 5: Julia Error Case
function f(text::String, s::Int64, e :: Int64)::Int64
    sublist = text[s+1:e] # Julia uses 1-based indexing, so adjust the start index
    if isempty(sublist)
        return -1
    end
    min_char = minimum(sublist)
    return findfirst(==(min_char), sublist) - 1 # Adjust for 0-based index in the result
end

D Details of Benchmark Construction

D.1 Experimental Setup of Each Step

In Generation step. Given the higher usage cost of closed-source LLMs, we set \( N \) to 5, \( k \) to 5, \( \delta A \) to 0, and the temperature to 0.2 for GPT3.5-Turbo. For DeepseekCoder-33B-Instruct We set \( N \) to 50, \( k \) to 5, \( \delta A \) to 0, and the temperature to 0.8. In Repair step. We set temperature to 0. For GPT-4o, we generate once, and correct errors three times.
D.2 Prompt of Each Step

In Figures 7 and 6, we include the prompts we use for our benchmark construction. We use a few-shot prompt for all models. For Generation step of each model, the prompt is show in Figure 7. The example in this Figure is used for GPT3.5-turbo. After the Generation and Repair of GPT3.5-turbo, we choose three examples from the correct generated problems which include str, List, Dict type respectively. We use these examples for Deepseekcoder-instruct-33b Generation. For GPT-4o, based on the summarized difficulties in translation, we provide three examples, as shown in Listing 6, 7, and 8. we present these examples in three distinct languages. For each specific language translation, we employ the corresponding language version of the three examples.

As illustrated in Figure 6, the prompt for Repair is shown, which depicts a single round of error correction. The compiler’s returned error messages are provided to the model for correction. For multiple rounds of error correction, subsequent error messages are appended to the dialogue after the model encounters errors again.

Listing 6: example1 for GPT-4o (D)

import std.math;
import std.typecons;
import std.conv;
import std.algorithm;
import std.array;
import std.string;

string f(string x, string y) {
    char[] yMutable = y.dup;
    yMutable.reverse();
    string tmp = yMutable.map!(c => c == "9" ? "0" : "9").array.
        map!(c => c.to!string).array.
        join("");
    if (x.isNumeric && tmp.isNumeric)
    {
        return x ~ tmp;
    } else {
        return x;
    }
}
unittest
{
    alias candidate = f;
    assert(candidate("", "sadasdnakjsda80") == "");
}
void main(){}

==================== System =====================
You are an expert programming assistant.

==================== User =====================
Please translate the language of the function from python to cpp

```python
[the python code]
```

==================== Assistant =====================
```cpp
[the error cpp code]
```

==================== User =====================
The code you translated has the following {error type} error:
{error message}

Please analyze the cause of the error and then return the repaired code in cpp.

Figure 6: Prompt of repair

==================== System =====================
You are a helpful programming assistant designed to translate code and complete code snippets.

==================== User =====================
Please translate the python function to cpp function:

```python
def add(a, b):
    return a + b
```

The function starts as follows, and your task is to complete it so that its semantics are the same as the python code above. Note that the number of packages called at the beginning of the given function cannot be reduced, but can only be increased.

```cpp
#include<assert.h>
#include<bits/stdc++.h>
long add(long x, long y) {
...
```

==================== Assistant =====================
```cpp
#include<assert.h>
#include<bits/stdc++.h>
long add(long x, long y) {
    return x + y;
}
```

==================== User =====================
Please translate the python function to cpp function:

```python
[the python code]
```

The function starts as follows, and your task is to complete it so that its semantics are the same as the python code above. Note that the number of packages called at the beginning of the given function cannot be reduced, but can only be increased.

```cpp
[the cpp function head]
```
Listing 7: example2 for GPT-4o (Swift)

import Foundation

func f(strand: String, zmnc: String) -> Int {
    var strand = strand
    var poz = strand.range(of: zmnc)!
    while poz != nil {
        strand.removeSubrange(poz!)
        poz = strand.range(of: zmnc)!
    }
    let lastIndex = strand.range(of: zmnc, options: [], range: nil, locale: nil)?.lowerBound.utf16Offset(in: strand)
    return lastIndex ?? -1
}

func ==(left: [(Int, Int)], right: [(Int, Int)]) -> Bool {
    if left.count != right.count {
        return false
    }
    for (l, r) in zip(left, right) {
        if l != r {
            return false
        }
    }
    return true
}

assert(f(strand: "", zmnc: "abc") == -1)

Listing 8: example3 for GPT-4o (Python)

from typing import Dict, Tuple

def f(d: Dict[str, int]) -> Tuple[int, int]:
    if "x" in d:
        x = d["x"]
    if "y" in d:
        y = d["y"]
    return x, y

def check(candidate):
    assert candidate({"x": 5, "y": 12}) == (5, 12)

def test_check():
    check(f)

E Programming Language Correlation in Translation

We observed that after a four-step translation process, the intersection of the 18 programming languages contained only 333 entries. This indicates that each programming language has its unique subset of correctly translated parts. Therefore, we constructed a Venn diagram to study the correlation between the sets of correct translations among different languages.

Specifically, the results are shown in Figure 8.

Due to the limitation of the number of sets that a Venn diagram can clearly represent, we explored by two methods, each selecting five representative languages. First, we chose the five languages with the largest union of results from the 18 languages to construct the first Venn diagram, which is the left half of Figure 8. Subsequently, we selected the top five most widely used languages, excluding Python, based on data from GitHut 2.0, to construct the second Venn diagram, which is the right half of Figure 8.

From Figure 8, we can observe that mainstream programming languages often have more similar syntax structures, and the model’s generation capability is stronger for these languages. Therefore, the intersection of the generation results for these five languages is relatively large, with 632 entries, while the union is relatively small, totaling 776 entries. Lua, PHP, R, Ruby, and JavaScript are among the languages with the broadest correct translation entries across all programming languages, with their union totaling 798 entries.

![Venn diagrams showing the intersection and union of code translation results for different programming languages](page_1012_1042_388_246.png)

Figure 8: The communitie of each language in code translation.

![Line chart showing the result of each step in translation for various programming languages](page_1012_1342_388_246.png)

Figure 9: The result of each step in translation
F  The Result of Each Step in Translation

Figure 9 shows the improvement brought by each step for every language during the construction of the benchmark. It can be observed that each step leads to an overall enhancement in translation performance. For mainstream languages such as C++ and Java, the number of correctly translated items can exceed 600 after one or two steps. For lower-frequency languages like D and Racket, the effect is gradually improved, eventually resulting in all languages having more than 600 correct translations.

G  GPU Usage and Total Cost of Translation

The total cost of GPT3.5-Turbo and GPT-4o is about $60 US dollars. For Deepseeker-Instruct-33b, we use 1 NVIDIA A100-80GB GPU and the generation and repair takes about 72 hours.

H  Prompt of Input/Output Reasoning

In Figures 12, 10, 11, 13, we include the prompts we use for our benchmark construction. We use a few-shot prompt for all models. The examples of few-shot is shown in Listings 9, 10, 11. All the prompts and examples are demonstrated in the C++ language.

Based on the given code, which may contain errors, complete the "????" in assert statement with the output when executing the cpp code on the given test case. Do NOT output any extra information, even if the function is incorrect or incomplete. Do NOT output a description for the assert.

```cpp
[function]
...
[ANSWER]
{function answer}
[/ANSWER]
```

Figure 10: Prompt of output reasoning (non-GPT)

You will be given a cpp function f and a check function, where you only know the output of the test case. Output the completion of the check function so that the code will run without errors by finding any input such that executing f on the input leads to the given output. There may be multiple answers, and you can output any one. Do NOT output any additional information.
...
```cpp
[function]
...
[ANSWER]
{function answer}
[/ANSWER]
```

Figure 11: Prompt of input reasoning (GPT)

You will be given a cpp function f and a check function, where you only know the output of the test case. Find any input such that executing f on the input leads to the given output. There may be multiple answers, but you should only output one. Think step by step before arriving at an answer. Finally, surround the answer, with no additional words, with [ANSWER] and [/ANSWER] tags. Express your answer as a function call that when executed will give the output.

```cpp
[example1]
...
[ANSWER]
{answer1}
[/ANSWER]
```
```cpp
[example2]
...
[ANSWER]
{answer2}
[/ANSWER]
```
```cpp
[example3]
...
[ANSWER]
{answer3}
[/ANSWER]
```
```cpp
[function]
...
[ANSWER]
{function answer}
[/ANSWER]
```

Figure 12: Prompt of input reasoning (non-GPT)

Based on the given code, which may contain errors, complete the "????" in assert statement with the output when executing the cpp code on the given test case. Do not output any extra information, even if the function is incorrect or incomplete.
...
```cpp
[function]
...
[ANSWER]
{function answer}
[/ANSWER]
```

Figure 13: Prompt of output reasoning (GPT)

Listing 9: Input/Output Reasoning example1 (C++)
```c++
#include <assert.h>
#include <bits/stdc++.h>
long f(std::vector<std::string> my_list) {
    long count = 0;
    for (std::string i : my_list) {
        if (i.size() % 2 == 0) {
            count += 1;
        }
    }
    return count;
}
int main() {
    auto candidate = f;
    assert(candidate({std::vector<std::string>({(std::string)"mq", (std::string)"px", (std::string)"zy")}}) == (3));
}
```
Listing 10: Input/Output Reasoning example2 (C++)
#include<assert.h>
#include<bits/stdc++.h>
std::string f(std::string s1, std::string s2) {
    return s1 + s2;
}
int main() {
    auto candidate = f;
    assert(candidate("ba"), ("nana")
        ) == ("banana"));
}

Listing 11: Input/Output Reasoning example3 (C++)
#include<assert.h>
#include<bits/stdc++.h>
std::tuple<long, long> f(std::map<std::string, long> d) {
    long x = 0, y = 0;
    if(d.find("x") != d.end()){
        x = d["x"];
    }
    if(d.find("y") != d.end()){
        y = d["y"];
    }
    return std::make_tuple(x, y);
}
int main() {
    auto candidate = f;
    assert(candidate(std::map<std::string, long>{{"x", 5}, {"y", 12}})) == (std::make_tuple(5, 12)));
}

<table>
  <tr>
    <th>language</th>
    <th>Rust</th>
    <th>Racket</th>
    <th>Lua</th>
    <th>Java</th>
  </tr>
  <tr>
    <td>avg line</td>
    <td>14.1</td>
    <td>17.6</td>
    <td>17.3</td>
    <td>22</td>
  </tr>
  <tr>
    <td>avg length</td>
    <td>380.4</td>
    <td>401.1</td>
    <td>413.3</td>
    <td>641.6</td>
  </tr>
  <tr>
    <th>Scala</th>
    <th>PHP</th>
    <th>Perl</th>
    <th>GO</th>
    <th>D</th>
  </tr>
  <tr>
    <td>16.1</td>
    <td>17.3</td>
    <td>21.3</td>
    <td>35.8</td>
    <td>21.5</td>
  </tr>
  <tr>
    <td>445.9</td>
    <td>407.8</td>
    <td>409.3</td>
    <td>755.2</td>
    <td>396.6</td>
  </tr>
  <tr>
    <th>C++</th>
    <th>C#</th>
    <th>TS</th>
    <th>Python</th>
    <th>R</th>
  </tr>
  <tr>
    <td>16.4</td>
    <td>22.9</td>
    <td>17.1</td>
    <td>14.3</td>
    <td>11.3</td>
  </tr>
  <tr>
    <td>481.1</td>
    <td>636.3</td>
    <td>405.1</td>
    <td>303.4</td>
    <td>347.2</td>
  </tr>
  <tr>
    <th>Julia</th>
    <th>Ruby</th>
    <th>JS</th>
    <th>Swift</th>
    <th>Shell</th>
  </tr>
  <tr>
    <td>15.2</td>
    <td>14.5</td>
    <td>16</td>
    <td>24.5</td>
    <td>23.2</td>
  </tr>
  <tr>
    <td>299.3</td>
    <td>321.9</td>
    <td>344.6</td>
    <td>551.4</td>
    <td>370.4</td>
  </tr>
</table>

Table 4: The average number of lines and average length of our benchmark

requiring rigorous logical reasoning. This task is of great reference value for evaluating the reasoning capabilities of language models.

J.2 The Applicability of Code Reasoning Tasks

In software development, understanding and predicting the input and output of code is crucial for detecting potential security vulnerabilities and ensuring the reliability of the code. For example, input validation and output filtering are key steps in preventing security vulnerabilities such as SQL injection and cross-site scripting attacks. A LLM capable of precise input and output reasoning can provide strong support in this regard.

K Additional Case Study

K.1 Error Cases of GPT-4o

As can be seen in Table 2, even GPT-4o fails to successfully solve approximately 30% of the problems. We analyze this portion of cases, specifically selecting those where less than two PLs out of 19 passed the test cases. We find that a significant number of these cases involve the processing of long strings. In input reasoning tasks, this accounted for 85.2% (46/54) of the cases. In Output reasoning tasks, it accounted for 68.4% (39/57).

Listings 12 and 13 respectively show the error cases of GPT-4o in input / output reasoning tasks. In Listing 12, the LLM fail to comprehend the code’s intention of converting the case of every alternate character. In Listing 13, the variable space_symbol consists of two parentheses, but the
LLMs mistakenly interpreted it as one parenthesis, leading to an error. This aligns with the conclusion in Section 4.1 that the longer the input or output, the more prone the LLMs are to making mistakes.

K.2 Comparative Evaluation of LLMs’ Reasoning Ability

To better understand the reasoning capabilities of different LLMs, we select three LLMs at varying levels: GPT-4o, DeepseekCoder-6.7b-base, and phi-1.5. We compare them pairwise to assess the reasoning ability of each LLM. We define that if a model correctly solves a problem in more than 17 PLs, it is considered to have mastered that problem. Conversely, if it solves the problem in fewer than 3 languages, it is deemed not to have mastered it.

By comparing the problems that GPT-4o master but DeepseekCoder-6.7b-base do not, we find that these problems mostly require multi-step operations, but the overall difficulty does not exceed the problem in Listing 12. As shown in Listings 14 and 15, one such problem involves successfully reversing ’ OOP ’ and removing extra spaces. Although DeepseekCoder-6.7b-base also execute these two steps, it fail to completely remove the spaces on both sides.

Next, by comparing the problems that DeepseekCoder-6.7b-base master but phi-1.5 do not, we find that these issues often arise because phi-1.5 fails to correctly write variables corresponding to programming languages other than Python, accounting for 90% (36/40) of the cases. As illustrated in Listings 16 and 17, phi-1.5 does not understand the meaning of the long-type variable and still outputs it as a string, leading to syntax errors.

From the above analysis, it can be seen that GPT-4o is capable of handling relatively complex tasks, DeepseekCoder-6.7b-base performs well in simple tasks but may make minor errors in complex tasks. phi-1.5, although demonstrating a certain level of Cross-PL Generalization, still performs poorly in non-Python languages. Particularly in understanding variable types and syntax.

Listing 12: Error Case of GPT-4o (Input Reasoning, C++)

#include<assert.h>
#include<bits/stdc++.h>
std::string f(std::string line) {
    int count = 0;
    std::string a;
    for (int i = 0; i < line.length() ; i++) {
        count += 1;
        if (count%2==0) {
            a.push_back(tolower(line[i]) == line[i] ? toupper(line[i]) : tolower(line[i]));
        }
        else {
            a.push_back(line[i]);
        }
    }
    return a;
}
int main() {
    auto candidate = f;
    assert(candidate("987yHnShAsHd 93275YrgSgBgsShFbSfB") == ("987YhnShAhd 93275 yRgsgBgssHfBsFB"));
}

Listing 13: Error Case of GPT-4o (Output Reasoning, Python)

def f(text: str, space_symbol: str, size: int) -> str:
    spaces = ''.join(space_symbol for i in range(size-len(text)))
    return text + spaces
def check(candidate):
    assert candidate('w', '))', 7) == 'w))))))'
def test_check():
    check(f)
test_check()

Listing 14: Correct Case of GPT-4o (Output Reasoning, Python)

def f(s: str) -> str:
    arr = list(s.strip())
    arr.reverse()
    return ''.join(arr)
def check(candidate):
    assert candidate('    OOP    ') == 'POO'
def test_check():
    check(f)
test_check()
Listing 15: Error Case of DeepseekCoder-6.7b-base (Output Reasoning, Python)

def f(s: str) -> str:
    arr = list(s.strip())
    arr.reverse()
    return ''.join(arr)

def check(candidate):
    assert candidate('   OOP   ') == '   P00'

def test_check():
    check(f)

test_check()

Listing 16: Correct Case of DeepseekCoder-6.7b-base (Input Reasoning, C++)

#include<assert.h>
#include<bits/stdc++.h>
std::string f(std::string text, long size) {
    long counter = text.length();
    for (long i = 0; i < size - (size % 2); ++i) {
        text = " " + text + " ";
        counter += 2;
        if (counter >= size) {
            return text;
        }
    }
    return text;
}
int main() {
    auto candidate = f;
    assert(candidate("7"), ("7")) == ("   7   "));
}

Listing 17: Error Case of Phi-1.5 (Input Reasoning, C++)

#include<assert.h>
#include<bits/stdc++.h>
std::string f(std::string text, long size) {
    long counter = text.length();
    for (long i = 0; i < size - (size % 2); ++i) {
        text = " " + text + " ";
        counter += 2;
        if (counter >= size) {
            return text;
        }
    }
    return text;
}
int main() {
    auto candidate = f;
    assert(candidate("7"), (11)) == ("   7   "));
}

Reviewer paper summaries: This paper propose a multilingual code reasoning corpus CruxEval-X. The authors provided an automated method that can translate existing  Python dataset into different programming languages. Evaluation of different LLMs on this dataset verify that this dataset can distinguish the reasoning ability of different LLMs.

This paper introduces CRUXEVAL-X, a multilingual code reasoning benchmark containing tests across 19 programming languages. The authors present an automated pipeline for translating Python code reasoning tasks to other languages and evaluate 24 different large language models on code reasoning capabilities. While the paper addresses an interesting problem in evaluating multilingual code reasoning, it has significant methodological flaws that undermine its contributions.
Summary of strengths: 1. The proposed corpus is useful in studying code abilities of different LLMs.

2. The automated code translation pipeline is useful in scaling code dataset and other related tasks.

3. There are some insightful analysis on the evaluation results revealing the cross-language generalizability of LLMs.

* S1: The paper identifies an important gap in existing benchmarks by pointing out the Python-centric nature of code evaluation datasets and the lack of comprehensive multilingual evaluation.

* S2: The authors make an attempt to create a large-scale multilingual dataset (600+ samples per language across 19 languages) for evaluating code reasoning capabilities.

* S3: The extensive evaluation across 24 different language models provides some interesting insights about cross-language generalization capabilities.
Summary of weaknesses: 1. The dataset is constructed from translating existing Python data to other programming languages. However, the authors did not mention where these Python data come from. Moreover, there lacks statistics of the dataset, such as the average length of each question and the number of test cases.

2. I am confused about what the task on this dataset look like. The paper mentioned syntactic correctness in section 4.2.1, but I did not understand why there can be syntax errors in input reasoning and output reasoning task because the model does not generate any code.

3. The evaluation metric pass@1 of the reasoning task is not very clear. What is considered as "pass"? Input and output are not one-to-one mapping. For example, in the input reasoning task example in Figure 1, there are infinite number of possible inputs that can give the same output. Randomness, multi-threads, etc. in code can result in different outputs given the same input. The authors should specify the scope of the questions and the evaluation metrics.

* W1: The authors use GPT-4 to generate and evaluate the dataset, creating a circular dependency that likely biases the benchmark toward patterns that current LLMs can already handle. This undermines the validity of all subsequent evaluations. The authors should have used human experts to create a smaller but high-quality seed dataset, employed multiple independent validation approaches beyond just GPT-4, conducted rigorous analysis of potential biases introduced by the generation process, and validated the dataset quality through human evaluation studies.

* W2:  The paper defines "code reasoning" tasks in terms of predicting inputs/outputs but does not justify why this represents reasoning ability. The authors should have provided theoretical grounding for their definition of code reasoning, included control experiments to validate their evaluation approach, demonstrated that their metrics actually measure reasoning rather than pattern matching, and compared against other potential evaluation frameworks.

* W3: The paper claims improvements in model performance but does not adequately control for confounding factors. The authors should have included proper baseline comparisons, conducted ablation studies to isolate the impact of different components, used statistical tests to validate significance of results, and analyzed failure cases in detail.

* W4: While the authors observe some transfer between languages, they do not rigorously analyze what enables this transfer or validate their hypotheses. They should have systematically analyzed linguistic features that enable transfer, provided detailed error analysis across language pairs, tested transfer under controlled conditions, and validated findings with targeted experiments.

* W5: The reproducibility of the work is questionable - the pipeline description lacks crucial details, and the evaluation process is not fully specified. The authors should have provided complete implementation details for their pipeline, specified all hyperparameters and configuration settings, included detailed error handling procedures, and released their code and data with proper documentation.
Comments, suggestions and typos: 1. The input/output length mentioned in Section 4.1 sounds a little bit weird to me. What is the length of complex types such as Dict? The input length of the f("a", 123) example is 6 makes more sense to me because we may want to include the quotes to distinguish string from int.

2. Line 283 python->Python

* C1: The paper needs to significantly strengthen its methodology and evaluation sections with proper experimental controls and statistical analysis.

* C2: The authors should conduct human evaluation studies to validate the quality of their generated dataset.

* C3: The paper would benefit from a thorough error analysis showing specific cases where the approach fails.
Overall assessment scores: 3

2.5
Metareview score: 3
Metareview: 1. This paper has proposed a multilingual benchmark that expands on existing Python-dominated datasets using a fully automated pipeline for dataset construction, integrating generation, repair, and cross-language translation. 
2. Sufficient results are conducted on CRUXEVAL-X.
3. This benchmark lacks a discussion on real-world applicability. And the translated codes from Python cannot reflect the several specific language features of other languages. 
4. The authors need to fix the issues and revise their paper based on the comments from other reviewers.
Summary of reasons to publish: See 1&2 in Meta review.