CodeRosetta: Pushing the Boundaries of Unsupervised Code Translation for Parallel Programming
Authors: Ali TehraniJamsaz
Arijit Bhattacharjee
Le Chen
Nesreen K. Ahmed
Amir Yazdanbakhsh
Ali Jannesari
Venue: NeurIPS 2024 poster (2024)
Keywords: unsupervised learning
code generation
HPC code generation
program translation
HPC code translation
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: Unsupervised Learning to Translate High-Performance Code

Abstract: Automatic translation of programming languages has garnered renewed interest, driven by recent advancements in large language models (LLMs). Encoder-decoder transformer models, in particular, have shown promise in translating between different programming languages. However, translating between a language and its high-performance computing (HPC) extension remains underexplored due to inherent challenges like complex parallel semantics understanding. In this paper, we introduce CodeRosetta, an encoder-decoder transformer model explicitly designed for translating between programming languages and also their HPC extensions. CodeRosetta is evaluated on C++ to CUDA and Fortran to C++ translation.
It employs a customized learning-based framework with tailored pretraining and training objectives that enable it to effectively capture code semantics and parallel structural nuances, allowing for bidirectional code translation. Our results show that CodeRosetta outperforms state-of-the-art baselines in C++ to CUDA translation by 2.9 BLEU and 1.72 CodeBLUE points while improving compilation accuracy by 6.05%. Compared to general closed-source LLMs, our proposed bidirectional learning-based method improves C++ to CUDA translation by 22.08 BLEU and 14.39 CodeBLUE with 2.75% higher compilation accuracy.
Finally, CodeRosetta exhibits proficiency in Fortran to parallel C++ translation, marking it, to our knowledge, as the first encoder-decoder model for such a complex translation task, improving CodeBLEU at least by 4.63 points compared to closed-source LLMs and Open Code LLM.

CODEROSETTA:
Pushing the Boundaries of Unsupervised Code Translation for Parallel Programming
Ali TehraniJamsaz, Arijit Bhattacharjee, Le Chen, Nesreen K. Ahmed Amir Yazdanbakhsh Ali Jannesari lowa State University, Ames, lowa, USA (tehrani, arbhatt9, lechen, jannesari |@iastate.edu
o Cisco Outshift, San Jose, CA, USA nesahmed@cisce www.cisccccoo.
Google DeepMind, Mountain View, CA, USA ayazdan@google.com
Abstract
Recent advancements Large Language Models (LLMs) have renewed interest automatic programming language translation. Encoder-decoder transformer models, in particular, have shown promise in translating between different programming languages. However, translating between a language and its high-performance com- puting (HPC) extensions remains underexplored due to challenges such as complex parallel semantics. In this paper, introduce CODEROSETTA, transformer model designed specifically for translating between programming languages and their HPC extensions. CODEROSEITA is evaluated C++ <+ CUDA and Fortran < C++ translation tasks. It uses a customized learning framework with tailored pretraining and training objectives to effectively capture both code semantics and parallel structural nuances, enabling bidirectional translation. Our results show that CODEROSETTA outperforms wfte-offthh-att baselines in C++ CUDA translation by 2.9 BLEU and 1.72 CodeBLEU points while improving com- pilation accuracy by 6.05%. Compared to general closed- source LLMss, our method improves C++ to CUDA translation by 22.08 BLEU and 14.39 CodeBLEU, with 2.75% higher compilation accuracy. Finally, CODEROSETTA exhibits proficiency in Fortran to parallel C++ translation, marking it, to our knowledge, the first encoder-decoder model for this complex task, improving CodeBLEU by at least 4.63 points compared to closed-source and open-code LLMs
encoder-decoder
Introduction
Automatic code translation between programming languages offers numerous benefits, such as modernizing legacy systems, enabling cross cross-platform development, and refactoring sequential code into parallel nigh-performance versions. However, this task poses significant challenges, primarily due to the scarcity of parallel corpora-paired datasets of the same applications written in different languages (e.g., C++ CUDA Fortran C++). This lack of data limits the effectiveness of supervised learning approaches. While recent advances in code LLMs have shown promise in general code translation, translating code that involves parallel programming paradigms (c.g., C++ CUDA) remains largely unexplored. That is primarily due to the inherent complexities in capturing and correctly replicating parallel code semantics 28
TransCoder [36] and its follow-up works (37 39 have demonstrated the potential of unsupervised
learning for code translation. However, these methods often struggle with the complexities of
Code: http::/ccoderosetta.com
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

translating between language and its specialized extensions, such as C++ to CUDA. To address this, BabelTower [46] proposes -pDA-specific metric and ranking model. Yet, its reliance on language- or library-specific metrics limits its scope, restricting it to unidirectional code translation (C++ CUDA). Moreover, extending BabelTower to other programming paradigms requires redefining syntax-specific metrics, process that is both time-consuming and dependent on domain expertise.
To address these limitations, we introduce CODEROSETTA, an encoder-decoder transformer model specificaally designed for unsupervised translation between programming languages and their high- performance computing (HPC) parallel extensions. Unlike prior methods that on language-specnin metrics, CODEROSETTA employs new pre-training and training objectives injictive--includinn Abstract Syntax Tree (AST) Entity Recognition and customized noise injection strategies for Denoising Auto- Encoding- to learn the inherent features and semantics of code in an unsupervised manner, without relying on language-specifff metrics. summary, this paper makes the following contributions:
Unsupervised code translation for parallel programming. We present CODEROSETTA, an encoder-decoder transformer model tailored for translating between programming languages and their parallel programming extension, specifically targeting C++ to CUDA and Fortran to C++.
Customized pre-training and training objectives for code translation to parallel programs. We introduce two new learning objectives for learning parallel programming syntax and nuances: (1) Abstract Syntax Tree (AST) entity recognition, enabling the model to reason about code structure by identifying and categorizing different syntactic elements, and (2) tailored denoising auto-encoding, incorporating weighted token dropping and insertion, along with an adaptive corruption rate, to help the model discern subtle differences between language constructs and their extensions.
Bidirectional translation without language-specific metrics. Unlike prior works that rely on program-specifte metrics for parallel code translation, which narrow the scope of code translation, CODEROSEITA learns bidirectionally (e.g.. C++ ++ CUDA and CUDA ++ C++) in an unsupervised manner, broadening its scope to different translation tasks.
Our results show that for C++ to CUDA translation, CODEROSETTA achieves 2.9 BLEU and 1.72 CodeBLUE improvement over existing methods while also increasing compilation accuracy by 6. 05%. Compared to closed-source LLMs, CODEROSETTA'S 's bidirectional approach exhibits even higher gains, with a 19. 84 BLEU and 14. 39 CodeBLEU improvement, and 2.75% higher compilation accuracy. To the best of our knowledge, CODEROSETTA is the first model to demonstrate proficiency in the task of Fortran to C++ translation, surpassing the performance of existing closed-source LLMs and open-code LLMs on standard metrics, with up to 4. 63-point improvement in Code
2 Related Works
Automatic parallelization. Translating from to CUDA poses a major challenge. Early efforts in this area primarily involved semi-automatti tools that required significant developer intervention. Noaje et al. [30) implemented an OpenMP [T] to CUDA translation using the OMPi compiler. Other tools, such as CUDAIy.NET and GPUcc 48 provided annotations to assist the translation process. DawnCC (27] automatically annotates C and C++ code for parallelism, utilizing static analysis to identify opportunities for optimizing execution on multicore and GPU architectures with OpenMP/Open directives. However, much of the responsibility for identifying parallelizable sections and optimizing memory usage remained with the developer. Efforts translate between C/C++ and Fortran have been more limited. FABLE 15) is one of the few frameworks designed for this, facilitating automatic translation of Fortran C++ while preserving the original code's semantics through advanced analysis and transformation techniques.
Neural machine translation. Tournavitis et al. 42 proposed framework that combines static analysis with machine learning to identify parallelizable code regions and determine the optimal parallelization scheme. This adaptive approach aims to reduce the overhead of manual parallelization while accommodating different architectures. TransCoder [36] pioneered the use of unsupervised learning techniques to translate code across various high-level languages, including Java, C++, and Python, without the need for parallel corpora. Building on TransCoder's s architecture, BabelTower [46] extends its capabilities perform parallel semantic conversion between C and CUDA.
Denoising Auto-Fncoding (DAF) has become a popular technique for training encoder-decoder models, seen in methods like CodeT5 (45) and PLBART (2]. These models typically use common

---

noising strategies such as masking and token dropping. One of the key differences in the noising strategies used by CODEROSETTA lies in its language-specift characteristics. Rather than random token dropping, CODEROSETTA employs weighted random dropping, prioritizing language-specifie reserved keywords to enhance the model's understanding of the target language's semantics. Another unique strategy is token insertion, which encourages the model to differentiate between valid and invalid tokens. These objectives enable CODEROSETTA to better distinguish between different extensions of the same programming language. In summary, CODEROSETTA is sequence-l--sequenee transformer model that learns in an unsupervised manner to translate between programming languages and parallel programming APIs. Additional related work is presented in Appendix J
3 CODEROSETTA: Unsupervised Code Translation for Parallel Programming
This section presents the design and training methodology of CODEROSETTA, our proposed encoder- decoder transformer model for unsupervised code translation. We begin by outlining the overall architecture, followed by detailed discussion of the pre-training and training objectives that enable CODEROSETTA effectively capture the nuances of both general-purpose programming languages and their parallel extensions. We focus on the C+++CUDA and C++ translation tasks.
3.1
Cross Language Masked Language Modeling
Input CUDA code
global
(ffiat
half(float *dat)
globel vold (alo int index threadidx.co [index dat(indax]]
int. index threadZdx..r
dRARR> threadidx...
(indax 3)
dat(Lndue]
Masking
(index 3)
<2AAEE (indse))
(Encoder)
ard [indec] 0.5; 1)
axd [indax] A 0.3; )}
[inden) 0.5;
)]
Figure I: Masked Language Modeling (MLM) pretraining steps in CODEROSETTA.
Pre-training plays crucial role in enabling transformer models to develop a foundational understanding of programming languages. We use Masked Language Modeling (MLM) [47], a widely adopted pre-training objective, to achieve this, outlined in Figure I1] In MLM, the model receives input code with portion of tokens randomly masked. The objective is to predict the masked tokens based on the surrounding context, thereby encouraging the model to learn both local syntactic patterns and broader semantic relationships within code.
To further challenge the model and better reflect code structure, we mask entire words rather than individual tokens. For instance, in the input code snippet " int index", the entire word " index" would be masked, requiring the model to predict the missing identifier based on its type (" int")) and its usage in the surrounding code. This approach mirrors how code comprehension often relies on understanding the roles of variables and functions within their scope.
Additionally, while MLM is typically applied to monolingual datasets, we extend it to a cross-lingual setting by training a combined dataset of both C++ and the target language (CUDA or Fortran). This cross-lingual exposure enables CODEROSETTA to learn shared programming concepts and syntactic structures across languages, such as control flow statements (if, else, while) and variable declarations, By recognizing these commonalities, the model can transfer knowledge across languages, improving its ability translate even unseen code patterns.
3.2 Abstract Syntax Tree Entity Recognition
Input CUDA code
Extracted AST entities V3: 150
Predicted entilies
Syyte
D. 3, D. T, 13, 33,
[index threadIds.ar det[index]] [index]] 0.5;
0. D,
CopeROBETTA
Figure 2: Abstract Syntax Tree Entity Recognition pretraining steps in CODEROSETTA.
3

---

Following cross-lingual MLM pre- training. we introduce a new pre-training objective called Abstract Syntax Tree (AST) Entity Recognition (AER) to further improve CODEROSETTA'S 's understanding of code structure. This approach draws inspiration from Named Entity Recognition (NER) in natural language processing [20, where models learn to classify words or phrases into predefined categories person, location, or organization). In AER, CODEROSETTA learns to recognize and categorize various syntactic components in code.
The process, illustrated in Figure 2 starts by using Tree-sitter multi-language parsing library, to generate the Abstract Syntax Tree (AST) of source code snippet. The AST representation provides a hierarchical, tree-structured view of the code, with each node corresponding to constructs such as function definitions, variable declarations, or arithmetic expressions. From this AST, we extract set of entities and their corresponding categories. Examples of categories used in our implementation include function, variable, constant, pointer, and literal.
During AER pre-training, CODEROSETTA tokenizes the input code and predicts the syntactic category of each token based on its role in the AST. Tokens that do not correspond to any specific category are labeled as "O" (Outside). This training enables CODEROSETTA to develop an understanding of the syntactic relationships between code elements, an essential step in accurately translating and generating code across different languages and extensions.
A key strength of AER is its flexibility- the set of entity categories can be easily adapted for different languages or programming paradigms. For instance, when focusing on CUDA code, we can introduce specialized categories for parallel constructs such as threadldx, blockldx, and gridDim, enabling CODEROSETTA to learn the language-specific semantics of parallel programming.
Furthermore, AER is highly adaptable. Even in cases where AST parsing is only partially available, CODEROSETTA can still leverage this pre-training, showcasing its applicability to diverse code translation tasks. The complete list of tags used in our implementation is provided in Appendix D.2
3.3
Denoising Auto Encoding with Adaptive Noise Injection
Input CUDA code
Code with noises glubel delete *are. < if linden)
Reconstructed CUDA code global void LEloat
_labbl void Mal (cloct floas *dat)
*dat) index if (Lnden 3)
(index dat[indea]
catch [tndac] aro[cooars)] 0.5, 1)
(Eroody Cucocon
dat[sndax]
acc [indax] 0.5:
)3
Macing
ero [rn]ledew] 0.5,
11
Figure 3: Denoising Auto Encoding.
While tross-lingual MLM and AST Entity Recognition effectively pre-train CODEROSETTA'S 's encoder to generate meaningful representations of source code, the decoder remains untrained at this stage. Consequently, attempting direct code translation would result in suboptimal performance due to the decoder's lack of exposure to the target language's syntax and semantics. To bridge this gap, we employ a Denoising Auto-Encoding (DAH) training strategy specifically tailored for code translation with adaptive noise injection mechanisms. In essence, DAE training involves corrupting the input source code with various types of noise and then training the model to reconstruct the original, noise-free code. This process compels the decoder learn both the underlying syntactic rules of the target language and the ability to recover meaningful code from perturbed inputs, simulating the challenges of translating real- real-world code with potential variations and inconsistencies.
To initiate the DAE training phase, we first initialize the decoder using the pre- pre-trained encoder's s weights, providing it with starting point for language understanding. Next, we apply a combination of common noise injection techniques, such as random token masking and shuffling, alongside our new noise strategies designed emphasize the distinctions between programming languages and their extensions. Figure 3 illustrates the overall process of DAE training in CODEROSETTA. We now delve into the specifics of our customized noise injection methods, which distinguish CODEROSEITA from conventional DAE-based code translation models. These strategies are crucial for enabling the model to discern the subtle but significant differences between languages like C++ and their nigh-performance counterparts like CUDA.
Patpss//tne--iitergguuubb

---

Weighted token dropping, To encourage the model to learn the distinctive features of each language and its extensions, we introduce a weighted token dropping strategy during the noise injection phase. Unlike uniform random token removal, this approach assigns higher removal probabilities to language-specific keywords, encouraging the model to focus on critical syntactic elements.
For each programming language or extension, CODEROSETTA maintains a list of reserved keywords. During token dropping, these keywords are prioritized, making them more likely to be removed than other tokens. For example, when training on CUDA code, keywords like blockIdx, threadldx, blockDim, _global_ and atomicSub are more frequently targeted for removal.
This weighted sampling creates more challenging reconstruction task for the model, compelling the decoder to develop a deeper understanding of the language-specifta semantics and parallel programming constructs. While the reserved keywords are given higher priority, the weighted random sampling still ensures that other tokens are occasionally dropped, preserving the overall balance of the noise injection process.
Language-speeific token insertion. In addition weighted token dropping, we implement a inguuag--specifii token insertion strategy to improve CODEROSETTA'S ability to discern between languages and their extensions during code generation. This method strengthens the model's robustness against out-of-vocabllary tokens, preventing it from inadvertently blending elements from different languages.
During DAE training, CODEROSETTA must distinguish between valid and invalid tokens within the target language. To facilitate this, we construct vocabulary of unique tokens for each programming language in our training dataset, tracking their frequency of occurrence. Tokens from the vocabulary of other languages are then randomly inserted into the input code based on their probability from the frequency distribution. For example, in the C++ to CUDA translation task, we insert CUDA-specte tokens into C++ code inputs during DAE training. CODEROSETTA then trained to recognize and disregard these foreign tokens while reconstructing the original C++ code. This process enables the model to develop an understanding of language boundaries, ensuring it generates syntactically and semantically valid code during translation.
Adaptive noise ratios Additionally, we introduce an adaptive noise strategy. Instead of applying fixed noise ratio, such as 10% for token dropping, we begin with an initial noise ratio and progressively increase it throughout the training process. This approach allows the model to gradually adapt to more challenging conditions as it learns to reconstruct the corrupted input sequences. As the training progresses, the input sequences become increasingly corrupted, making the reconstruction task more difficult and forcing the model to learn more robust representations.
There is maximum corruption rate that, once reached, halts further increases in noise. This prevents over-corrupting the inputs, ensuring that the model can still derive meaningful patterns. The impact of adaptive noise ratios, along with the new noise strategies, examined in our ablation study (Section 5.3).
To further support accurate code generation in the target language, we prepend special <LANG> token to each input sequence. During DAE, this token indicates the language of the corrupted input, prompting the decoder to reconstruct the code in the same language. This mechanism ensures that the model remains focused on generating code within the correct language context.
3.4 Back Translation for Unsupervised Refinement
Imput C++ code vold Mal hale (floate
Translated CUDA code vold Mil hal. (flott
Reconstructed C+4 code
-ec.-e----- vold hald (Elloee floate dati
flout* das)
"acc, cloat *dat]
for lint index 8: index
CocdRosaTTA [Eroods: [0++ to CUDA:
int index theoatids. 1e [index < 3) dos (inden]
CoodROasTTA Brodd-Doodo JOJDA C+4)
for (int indax O: indux
3) indezee)
3: indez-4) C
das (index]
dok(indon)
axe sxn]inden] 0.5; 11
sro[Andex]] 0.5;
erc[index]] 0.3;
13
33
Figure 4: Back Translation.
To further improve CONEROSETIA'S translation quality and its ability to capture complex code
semantics, we employ back translation during the training process [36). As illustrated in Figure 4
5

---

this technique leverages the model's 's bidirectional capability, enabling both source-to target and target-to-source translations, forming a weakly supervised learning loop.
In back translation, the model is trained on a source-to-target task (e.g., C++ to CUDA) while simultaneously performing the reverse translation (farget-to-source, CUDA to C++). For each batch of source code, CODEROSETTA first translates it into the target language. The generated target code then used as input for a reverse translation, where the model attempts to reconstruct the original source code.
This forward and backward translation cycle provides continuous feedback, allowing CODEROSETTA to compare the reconstructed source code with the original input, thereby learning to detect and correct errors in both translation directions. Through this iterative refinement, the model gradually improves its comprehension of nuanced language differences and complex code structures, resulting in more accurate and semantically consistent translations.
Crucially, we alternate between batches of different language pairs during back translation. This ensures that the model receives balanced exposure to both directions, preventing bias towards specific language and encouraging the development of robust, generalized translation capabilities.
3.5 Finetuning with Synthetic Data from Language Models (Optional Step)
While CODEROSETTA demonstrates promising results through unsupervised training, we explore the potential of further enhancements by leveraging the capabilities of large language models (LLMs) such as GPT-4 [[] and Gemini Ultra [41]. These LLMs, trained on extensive text and code datasets, have exhibited impressive code generation abilities. However, directly employing such large models for code translation can be computationally expensive and impractical for many real-works applications.
To address this, adopt a knowledge distillation approach [88, where these LLMs serve teacher models to generate synthetic data for fine-tuning CODEROSETTA, a smaller student model. This method allows us capture the expertise of the larger models while maintaining computational efficiency.
Specifically, we prompt GPT-4 and Gemini to translate C++ code into CUDA where feasible. After filtering out empty or invalid translations, natural text, and non-relevant data (i.e., instances lacking CUDA-specific keywords), we are left with approximately 5,000 high-quality translations from an initial set of 100,000. This significant reduction highlights the inherent challenges in C++ to CUDA translation.
The resulting synthetic dataset of C+++AUDA pairs is then used to fine-tune CODEROSETTA. This process allows CODEROSETTA to incorporate the valuable knowledge embedded in the larger LLMs without incurring their high computational costs. It is important to note that this fine-tuning step is optional and can be omitted if access to powerful LLMs for synthetic data generation is not feasible.
4 Experimental Setup
Training hyperparameters. We implement CODEROSETTA using the HuggingFace Transformers library v4.40.1 47 The model 12-layer encoder-decoder transformer, with each layer having 12 attention heads and a hidden dimension of 1,536. We initialized the tokenizer with pre-trained Byte Pair Encoding (BPE) tokenizer from UniXcoder [17, which was further trained on our specific training datasets. The training was conducted using the Adam W optimizer (4]] and batch size of 16, using gradient accumulation over two steps. The experiments were run on a single node with four Nvidia A100 SXM4 GPUs, each with 80GB of memory. To speed up the training process, mixed preed-preiisoo training was enabled. The final model consists of ~0088 billion parameters.
4.1 Datasets
We evaluate CODEROSETTA on two code translation tasks: C++ to CUDA and Fortran to C++. Table K provides overview of the datasets used. For the C++ to CUDA translation task, use the dataset from BabelTower 46 which consists of:
Unpaired training set: A collection of 243.008 C++ and CUDA source code files, meeaning there is no direct correspondance berween the files each language. To avoid any language bias, we ensure an equal number of C++ and CUDA files during training.

---

Paired validation and test sets: The validation set consists of 184 pairs, and the test set has 180 pairs of C++ and CUDA source code files. Each pair represents the same program implemented in both languages, providing a benchmark for evaluating translation accuracy.
For Fortran to C++ no dedicated parallel corpus exists for this specific translation. Thus, we construct training dataset follows:
Unpaired training set: We extract the C++ and Fortran subsets from the Stack V2 dataset [25], which includes over billion source code files across more than 600 programming languages. We ensure an equal number of files from each language prevent bias during training.
Paired fine-tuning set: For fine-tur fine-tuning, Bin et al. [9]. This set is also used for validation.
the small paired C++-Fortran dataset introduced by
Test set: To evaluate the final model performance, we use test set of 33 paired C++ and Fortran programs.
4.2 Data Preprocessing
To ensure the quality and consistency of training data, applied task-specific preprocessing steps for each translation task. C++ to CUDA. Although the BabellTower dataset [46] was reportedly cleaned, found noisy data within the CUDA files. To address this, we curated list of CUDA-specific reserved keywords and filtered the dataset, retaining only those CUDA files that contained at least one such keyword, This step significantly reduced noise and resulted a final training of 243,008 C++ files, matched by an equal number of CUDA files, The validation and test sets remained unchanged, comprising 184 and 180 paired examples, respectively.
C++ to Fortran. Preprocessing the Stack V2 dataset for C++ to Fortran translation involved managing the large imbalance between C++ and Fortran files, as well as filtering out low-quality uninformative code snippets. We implemented the following steps:
Inspired by the phi-1 model data filtering approach [6], We randomly sampled 100,000 C++ files from Stack V2 and employed GPT-3.5 to assess their "educational value" for learning C++ coding concepts. We prompted GPT-3.5 (see Figure 5 to classify each snippet as either "Yes" "No" based on its educational value. These labels were then used to fine-tune binary classifier built on the CodeSage model [49], which we applied to the remaining C++ files in Stack V2. Only files deemed educationally valuable were retained.
Educational value filtering:
Balancing language representation: From the filtered C++ files, we randomly selected subset equal in size to the number of Fortran files to create a balanced training set.
Length-based filtering: To ensure training stability and avoid biases toward very short or long code snippets, we filtered out files containing fewer than ten tokens or more than 1,000 tokens in both languages.
After these steps, the final training set for C++ to Fortran translation consisted of 474,856 files. For fine-tuning and validation, used the small paired C++-Fortran dataset from Bin et al. [19), which contains 282 samples. The model was then evaluated on a test set of 33 paired samples.
Determine the educational value of the following code for student whose goal is learn C++ coding + concepts. If it has educational value, return only "Yes", else, return "No". (code)
Educational value:
Figure 5: Prompt for determining the quality of C++ source code
4.3 Evaluation
To evaluate CODEROSETTA'S translations, we use two widely used code translation metrics: BLEU and CodeBLEU (34). We benchmark CODEROSETTA against the following baselines. For C++ to
7

---

Table I: Summary of C++ CUDA translation results across various code metrics and compilation accuracy. Second-best results are underlined.
Static Metries
Compilation Accuracy (%)
Model
BLEU
CodeBLEU
ChrF
ROGUE-L
GPT4
46.98
64.45
70.15
63.37
96.10
Gemini-Ulta
57.06
61.18
73.20
69.27
80.00
Gemini-Pro
54.82
64.20
72.58
69.82
75.50
DeepSeekCoder
26.63
21,46
28.41
15.10
57.80
StarCoder
37.58
62.58
60.16
41.84
79.40
TransCoder
72.21
71.03
N/A
N/A
83,80
BabelTower
74.00
77.12
92.80
CODEROSETTA (Ours)
76.90
78.84
81.05
82.12
98.85
CUDA, we compare (a) "BabelTower [46]
sfate-of-thee-rt unsupervised code translation model specifically designed for C++ CUDA translation, and (b) "Transcoder [36]". general unsupervised
code translation model that has demonstrated strong performance on various language pairs. Since a single evaluation metric may capture only aspect translation quality (14), supplement BLEU
and CodeBLEU with ROUGE-L [22] and ChrF [33], as recommended by [4] However, because generated translations from TransCoder and BabelTower were unavailable, ROUGE-L and ChrF scores are only provided for GPT-4, Gemini-Uni and Gemini-Pre We further compare CODEROSETTA and DeepSeckCoder
with two popular open-source code LLMs: StarCoder (starcoder2-7b) 112
For the Fortran to C++ task, evaluate CODEROSETTA against StarCoder [21] LLM model (15.5B parameters) featuring decoder-only transformer architecture, fine-tuned on a comprehensive corpus of Fortran code and DeepSeekCoder (DeepSeek- DDeepSeek-Codee------e Coder-V2-Lite--Bann
[12]. Additionally,
we evaluate CODEROSETTA alongside several prominent closed-source LLMs, including GPT-4 I and Gemini [1). by prompting them perform code translation using carefully crafted prompts (Appendix
! By evaluating against a broad spectrum of both specialized code translation models and
general-purpee purpose LLMs, we effectively gauge CODEROSETTA'S stranghts and limitations across diverse translation tasks and programming paradigms.
5 Experimental Results
5.1 C++ to CUDA
Table presents the results of CODEROSETTA for C++-ACUDA translation. For BabelTower and TransCoder the results are directly quoted from BabelTower [6). as their models and implementations are not publicly available. Comparing the performance of CODEROSETTA to other models, it demon- strates superior translation capabilities for C++ CUDA. Specifically, CODEROSETTA outperforms BabelTower by 2. BLEU points. Additionally, it achieves CodeBLEU score of 78 84, which is 1.72 points higher than BabelTower. Although GPT4 and Gemini were not specifically trained this dataset, they still reached CodeBLEU scores of 64. 45 and 64. 20, respectively. Evtikhiev et.al [14] indicate that ChrFi and ROGUE-L metries are better suited for code generation tasks than BLEU and Code Notably, CODEROSETTA also surpasses these models in both ChrF and ROUGE-L metrics.
CODEROSETTA effectively learns the necessary semantics to generate CUDA code without relying on specific metrics for training, departure from previous approaches. The compilation accuracy of CODEROSETTA is 98. 85% after post-processing. For examples of the CUDA code generated by our model compared to other baselines, please refer to Appendix B Furthermore, CODEROSETTA bidirectional, allowing it to translate both C++ to CUDA and vice versa. Please refer to Appendix A for CUDA to C++ results.
We contacted the authors of BabelTower for access to their trained model, source code, and translations but able gain access. Therefore, cite results directly from their paper.

---

Table 3: Ablation Study for C++ CUDA.
Metries
Experiment
BLEU
CodcBLEU
Removing MLM
52.12 (-24.78)
51.96 (-26.88)
Removing AER
74.98 (-1.92)
75.55 (-3.29)
Removing DAE (special noises)
72.41 (-4.49)
73.22 (-5.62)
Removing
75.08 (-1.82)
73.18 (-5.66)
Removing Fine-Tuning
73.55 (-3.35)
71.21 (-7.63)
Baseline
76.90
78.84
5.1.1 Post-processing: Compilation Error Analysis
Our test set, consisting of 180 samples, provided diverse input scenarios evaluate model's performance. We observed that 23 samples generated compilation errors when processed through the NVCC compiler with the required flags Upon manual investigation, we found that most errors were trivial and could be easily fixed with minor edits.
Specifically, 48% of the errors attributed the use of an undefined generic type Another 9% resulted from missing closing braces, while 26% were due to single missing variable initialization. Additionally, 9% of the errors were caused by incorrect function calls. Only 8% of the files contained no trivial errors. By applying quick fixes for the undefined generic type T, missing variable initializations, and missing closing braces, the overall compilation accuracy significantly improved, with 98 85% of all generated code becoming compilable. This indicates that most errors were simple and could be easily resolved by incorporating compiler feedback, which will be focus of our future work. Subsection F.T [ and Figure 13 in the Appendix presents examples of our findings.
Table 2: Types of compilation errors (28 codes with compilation error out of a total 180 codes).
Error Type
Percent
Undefined generic type T Missing variable initialization Missing closing braces Wrong function call Non-trivial errors
48
26
5.2 Runtime Evaluation
Although CODEROSETTA demonstrates more accurate translations based on the aforementioned metrics compared the reference code, these metrics are derived from static evaluations, leaving runtime performance uncertain. To address this, we randomly selected 30 translated CUDA kernels from the test set and created unique template programs to execute them. We ran the translated CUDA kernels using NVCC and found that the functional correctness of the generated code was preserved in the majority samples (approximately 93%). For further details, see Appendix Section
5.3 Ablation Study
We conduct ablation study evaluate the impact of each training objective the code translation results of CODEROSETTA. Specifically, we remove individual training objectives (e.g.. AER) while keeping the other components intact and retraining the model. Table presents the results of the ablation study for C++ to CUDA translation. As observed, removing any of the pertaining or training objectives negatively impacts translation results, with Masked Language Modeling having the most significant effect when omitted. This is expected, as Masked Language Modeling is the primary pretraining objective that enables the model to understand source code.
AER training task. CODEROSETTA employs two pre-training tasks for training its encoder: Mask Language Modeling (MLM) and Abstract Syntax Tree Entity Recognition (AER). In this phase, maintain consistent training setups except for the removal the AER component.

---

Table 4: Fortran to C++ translation results.
Model
CodeBI.EU
GPT4
19.21
Gemini- Ultra
13.62
Gemini-Pro
18.91
DeepSeckCoder
12.09
StarCoder
18.21
StarCoder (fine-tuned)
61.30
CODEROSETTA (0.8B)
65.93
Denoising Auto Encoding. We also investigate the effectiveness of various noise types and the adaptive corruption rate during Denoising Auto Encoding. For this ablation study, we train the model without weighted token dropping, insertion, and adaptive corruption rate.
Fine-tuning Data extraction from larger models is common practice. In this phase of the ablation study, we evaluate CODERoSETTA'S 's performance without fine-tuning it on the synthetic dataset. From Table we observe that the removal of each proposed learning objective negatively impacts the model's s ability to deliver improved code translation.
5.4 Fortran to C++
We train and apply CODEROSETTA for translation between C++ and Fortran. Fortran has had long-standing presence in the scientific computing community; however, its integration with modern HPC systems (38) pose significant challenges for developers. Due to the complexities involved in translating Fortran to C++, there has been limited effort to address this issue. Bin et al. [19] were the first make significant strides in this area, curating a small paired dataset specifically for this translation task and fine-tuning several open-code LLMs.
They found StarCoder (15B), when fine-tuned, benefited the most from their paired dataset. We compare CODEROSETTA with the tine-tuned StarCoder (15B), as well as with other general LLMs. The results are shown in Table 4 Fine-tuning CODEROSETTA further enhances its performance, achieving a CodeBLEU score of 65. 93. Notably, CODEROSETTA outperforms StarCoder, even though StarCoder is nearly 20 times larger, highlighting the efficiency of our model. also surpasses state-of-the-arr models like GPT-4 and Gemini by a substantial margin, achieving improvement of at least 4.63 points in CodeBL.EU.
the dataset from Bin al. [I9]
6 Conclusion
In this paper, we introduced CODEROSETTA, an encoder-decoder transformer model designed for translating between programming languages and their high-performance computing (HPC) extensions. We proposed two novel learning objectives: Abstract Syntax Tree (AST) Entity Recognition (AER) and customized Denoising Encodingg,, which incorporates weighted token dropping and insertion. These contributions enable CODEROSETTA to capture both the general syntactic structure of code and the specific nuances parallel programming constructs, without relying on language-specits metrics. Our experiments show that CODEROSETTA significantly outperforms state-of-the-arr baselines on C++ to CUDA translation, achieving improvements up to 2.9 BLEU, 1.72 in CodeBLEU, and 6.05% in compilation accuracy. Furthermore, CODEROSETTA is, the best of our knowledge, the first model to demonstrate proficiency in translating Fortran to its parallel counterpart in C++, highlighting its potential in handling diverse programming paradigms.
10

---

Acknowledgment
We would like to thank NSF for their generous support funding this project (#2211982). addition, we extend our gratitude to Intel Labs for supporting this project. We also would like to extend our gratitude towards Pengcheng Yin and Chandu Thekkath for their feedback on the early draft this work. We also appreciate the support from the extended team at Google Deep.indd We thank the Research IT tean of Iowa State University for providing access to HPC clusters for conducting the experiments of this research project. We also thank National Center for Supercomputing Applications for providing Delta GPUs through allocation CIS230375 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program [7]. Lastly, we would like to express our sincere appreciation to the anonymous reviewers, area chairs, and program chairs of NeurIPS 2024 for their valuable feedback and insights, which significantly contributed to the improvement of this work.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1ige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 Technical Report arXiv preprint arXiv:2303. 2023.
[2] Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Unified Pre-training for Program Understanding and Generation In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer. Dilek Hakkani-Tur, Beltagy, Steven Bethard, Ryan Cotterell, Tammoy Chakraborty. and Yichao Zhou, editors, NAACL, 2021.
[3] Loubna Ben Allal, Raymond Li. Denis Kocetkov, Chenghao Mou, Christopher Akiki, Car- los Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. SantaCoder: Don't 't Reach for the Stars! arXiv preprint arXiv:2301.03998. 2023.
[4] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The Falcon Series of Open Language Models, arXiv preprint arXiv:2311 1.2211111666,. 2023,
[5] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen Technical Report arXiv preprini arXiv:2309. 16609, 2023.
[6] Molided-Walid Benabderrahmane, Louis-Noël Pouchet, Albert Cohen, and Cédric Bastoul. The Polyhedral Model Is More Widely Applicable Than You Think In Proceedings of the 19th Joint European Conference on Theory and Practice of Software, International Conference on Compiler Construction. Sgringer-Verlage 2010.
[7] Timothy J. Boerner, Stephen Deems, Thomas R. Furlani, Shelley L.. Knuth, and John Towns. ACCESS: Advancing Innovation: NSF's Advanced Cyberinfrastructure Coordination Ecosystem; Services & Support In PEARC, 2023.
[8] Uday Bondhugula, Albert Hartono, J. Ramanujam, and P. Sadayappan. A Practical Automatic Polyhedral Parallelizer and Locality Optimizer In PLDDI, 2008
[9] Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin. Donald Pinckney, Ming-Ho Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. MultPL E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation IEEE Transactions on Software Engineering, 2023.
[10] Mark Chen, Jerry Tworek, Hcewoo Jun Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 2021.
[11] L. Dagum and R. Menon. OpenMP: An Industry Standard API for Shared-mennn Programming IEEE Computational Science and Engineering. 1998.
'nttps//treeaarchi..kk las iastate.edu/

---

[12] DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma. Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong. Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang. Junxiao Song. Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du, Wenjun Gao, Xuan Lu, Qinyu Chen, Yaohui Wang, Chengqi Deng. Jiashi Li, Chenggang Zhao, Chong Ruan, Fuli Luo, and Wenfeng Liang. CeepSeek-Coder-ie Breaking the Barrier of Closed-Source Models in Code Intelligence arXiv preprint arXiv: 2406. 11931, 2024.
[13] Xianzhong Ding, Le Chen, Murali Emani, Chunhua Liao. Pei- Pei-Hung Lin, Tristan Vanderbruggen. Zhen Xie, Alberto Cerpa, and Wan Du. HPC--GP:: Integrating Large Language Model for Hergh-Peroormance Computing In Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, 2023.
[14] Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin. Out of the BLEU How Should We Assess Quality of the Code Generation Models? Journal of Systems and Software, 2023.
[15] Ralf W. Grosse-Kunstleve, Thomas C. Terwilliger, Nicholas K. Sauter, and Paul D. Adams. Automatic Fortran to C++ Conversion with FABLE Source Code for Biology and Medicine, 2012.
[16] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tau- man Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks Are All You Need, arXiv preprint arXiv:2306. 1644, 2023.
[17] Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. UniXcoder: Unifid Pre-training for Code Representation In ACL, 2022.
[18] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network arXiv preprint Xrv: / 1503.0253/.31 2015.
[19] Bin Lei, Caiwen Ding, Le Chen, Pei-Hung Lin, and Chunhua Liao, Creating a Dataset for Computing Code Translation using LLMs: A Bridge Between OpenMP Fortran and C++ In HPEC, 2023.
[20]
Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. A Survey on Deep Learning for Named Entity Recognition ICDE, 2020.
[21]
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj. Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang. Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh. Sasha Luccioni, Paulo Villegas. Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Naday Timor, Jennifer Ding. Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Volan-Gavitt Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. StarCoder: May the Source be With You! Transactions on Machine Learning Research, 2023.
[22] Chin-Yew Lin. ROUGE: A Package for Automatic Evaluation of Summaries In Text summa- rization branches out, 2004.
[23] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[24] Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization In ICLR, 2017.
12

---

[25]
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Noua- mane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAAley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary. Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang. Sean Hughes, Thomas Wolf, Arjum Guha. Leandro von Werra, and Harm de Vries. [StarCoder 2 and The Stack v2: The Next Generation arXiv preprint arXiv:2402. 19173, 2024.
[26] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma. Qingwei Lin, and Daxin Jiang. Wizardeoder: Empowering Code Large Language Models with Evol- arXiv preprint arXiv:2306.08568, 2023.
[27]
Gleison Mendonça, Breno Guimarães, Péricles Alves, Márcio Pereira, Guido Araújo, and Fernando Magno Quintão Pereira. DawnCC: Automatic Annotation for Data Parallelism and Offloading ACM TACO, 2017.
[28]
Daniel Nichols, Joshua H Davis, Zhaojun Xie, Arjun Rajaram, and Abhinav Bhatele. Can Large Language Models Write Parallel Code? In HPDC, 2024.
[29]
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. CodeGGn: An Open Large Language Model for Code with Multi-Turn Program Synthesis arXiv preprint arXiv:2203. 13474, 2022.
[30] Gabriel Noaje. Christophe Jaillet, and Michaël Krajecki. [Source-to- to--uure Code Translator OpenMP C to CUDA In HPCC, 2011.
[31] Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1-13, 2024.
[32] Kishore Papineni. Salim Roukos, Todd Ward, and Wei-Jing Zhu. Evaluation of Machine Translation, In ACI., 2002.
BLEU: A Method for Automatic
[33] Maja Popović. chrlf: Character n-gram F-score for Automatic MT Evaluation In Proceedings of the tenth workshop on statistical machine translation, 2015.
[34] Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Code a Method for Automatic Evaluation of Code Synthesis
arXiv preprini arXiv:2009, 10297, 2020.
[35] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, al. Code Llama: Open Foundation 2023.
Models for Code arXiv preprint arXiv:2308.
[36]
Baptiste Roziere, Maric-Anne Lachaux. Lowik Chanussot, and Guillaume Lample. Unsupervised Translation of Programming Languages) NeurlPS, 2020.
[37] Baptiste Roziere, Jie M. Zhang, Francois Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. Leveraging Automated Unit Tests for Unsupervised Code Translation In ICLR, 2022.
[38] Thomas Sterling, Maciej Brodowicz, and Matthew Anderson. High performance Computing Modern Systems and Practices, Morgan Kaufmann, 2017.
[39] Mare Szafraniec, Baptiste Roziere, Hugh Leather, Francois Charton, Patrick Labatut, and Gabriel Synnaeve. Code Translation with Compiler Representations In ICLR, 2023.
13

---

[40] CodeGemma Team. CodeGemma: Open Code Models Based on Gemma arXiv:2406.1140% 1409, 2024.
arXiv preprint
[41]
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: A Family of Highly Capable Multimodal Models, 2024.
[42]
Georgios Tournavitis, Zheng Wang, Björn Franke, and Michael F.P. O O'Boyle. Towards a
Holistic Approach Auto-Parallelization: Integrating Profile-Driven Parallelism Detection and Machine-Lcarning Based Mapping In PLDI, 2009.
[43] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. LLaMA: Open and
Efficient Foundation Language Models, arXiv preprint 2rXiv:2332..3997 2023.
[44] Sven Verdoolaege, Juan Carlos Juega, Albert Cohen, José Ignacio Gómez, Christian Tenllado, and Francky Catthoor. Polyhedral Parallel Code Generation for CUDA ACM TACO, 2013.
[45] Yue Wang. Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. CodeT5: Identifier-award aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation 2021.
In EMNLP,
[46] Yuanbo Wen, Qi Guo, Qiang Fu, Xiaging Li, Jianxing Xu, Yanlin Tang, Yongwei Zhao, Xing Hu, Zidong Du, Ling Li, al. Babel Learning Auto-parallelized Program Translation In ICML, 2022.
[47] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: ttate-of-the-Arl Natural I Language Processing In EMNIP, 2020.
[48] Jingyue Wu, Artem Belevich, Eli Bendersky, Mark Heffernan, Chris Leary, Jacques Pienaar, Bjarke Roune, Rob Springer, Xuetian Weng, and Robert Hundt. gpucc- An Open--penuttt GPGPU Compiler In CGO, 2016.
[49] Dejiao Zhang, Wasi Ahmad, Ming Tan, Hantian Ding, Ramesh Nallapati, Dan Roth, Xiaofei Ma, and Bing Xiang. Code Representation Learning At Scale In ICLR, 2024.
[50] Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Openccdeeinterpreee:: Integrating Code Generation with Execution and Refinement arXiv preprint arXiv:2402.14655 2024.
[51] Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. CodeBERTTScore: Evaluating Code Generation with Pretrained Models Code, In EMNLP, 2023.
14

---

Appendix
Table of Contents
CUDA C++ Translation Results
15
Functional Correctness Analysis
Decontamination Analysis
16
D Insupervised Training Parameters
18
D.I Training Parameters
18
D.2 AST Entity Recognition Tags
19
D.3 Dataset Statisties
19
E Impact Beam Size
F Analysis of Generated Code from CODEROSETTA and Closed-Source LLMs
19
F.1 Common Issues and Posi-processing in ConsKossETra-GGeerralll Code
20
G Discussion on Unsupervised Training
20
G. Fine-tuning for Code Translation
20
2 Back Translation
21
Translation Pitfalls: Invalid Tokens in Target Language
21
Prompt Template and LLMS
21
Additional Related Work
22
K
Limitations
22
A CUDA to C++ Translation Results
CODEROSETTA is capable of bidirectional translation between languages. Once trained for C++ to CUDA translation, it can also translate CUDA back C++, unlike previous approaches such as Babel'ower [46]. In this section, we compare CODEROSETTA with GPT4 and Gemini on the task of translating CUDA back to C++. Table 5 summarizes the results. As shown, CODEROSETTA demonstrates higher accuracy in translating CUDA to C++. Moreover, we observed that Gemini struggles to clearly distinguish between CUDA and C++, frequently generating C++ translations that nearly identical to the original CUDA input.
Table 5: CUDA to C++ translation results across different models. We use a similar prompt as the one in Figure 15 with small adjustments.
Model
BLEU
CodeBLEU
GPT4
70.18
68.67
Gemini-Pro
35.96
61.09
CODEROSETTA (Ours)
77.03
71.28
15

---

B Functional Correctness Analysis
The metrics and results shown in Table may have limitations in capturing functional equivalence. as discussed by Evtikhiev et al. [1] To address this, we evaluated the functional correctness of the translated code by compiling and executing the generated programs. For the C++ - CUDA translation task, we randomly selected 30 generated CUDA kernels and developed a template program for their execution. We then compared the runtime results of the translated CUDA code against the reference implementations. Our findings indicate that 93% of the translated CUDA code produced results consistent with the reference.
We analyzed three representative cases of CUDA translation in detail. In the first case, shown in Figure @ the kernel is designed to he launched with a grid of thread blocks. Each thread calculates its global index i, and if i is within the array's bounds (i N), it assigns the value ALPHA to the element at index i INCX in the array X. CODEROSETTA successfully identified the optimal 2D grid struc- ture with (blockIdx. + blockIdx.y y * gridDim.x) * blockDim. + threadIdx... x, less efficient ID structure using whereas other models defaulted to This choice of grid structure significantly blockIdx. x * blockDim. + threadIdx.x.
impacts CUDA performance, and CODEROSETTA'S 's selection mirrors that of the baseline implementa- tion. Furthermore, CODEROSETTA employed the correct grid structure in four additional instances where other models did not.
The second case, illustrated in Figure 7 involves a kernel designed to initialize an array of offsets for sorting purposes. Each offset corresponds to the starting position of column in a flattened 2D grid. This often useful for parallel sorting algorithms or other operations requiring column-wiss processing. The expression int tid threadIdx.x x + blockIdx.x x blockDim. x; assigns each thread a unique index across the entire grid of blocks, enabling access to distinet elements in a global array. In contrast, the expression int tid = threadIdx. x; provides an index that is only unique within single block. Without proper offset calculations, threads across different blocks could access the same data, potentially leading to race conditions and negating the kernel's 's intended behavior. This issue was observed in several examples where Gemini-Ulte produced incorrect results due to this oversight.
The third case, depicted in Figure processes 3D arrays in parallel. Each thread calculates its 3D position, checks bounds, and updates specific elements of the array vec based on values from vecl. The kernel averages and scales values from vecl, storing the results in vec while ensuring safe memory access within the array's limits. CODEROSETTA correctly handled large block and grid dimensions by using unsigned long, whereas both GPT-4 and Gemini- Ultra failed due to the use of int, leading to index overflow.
We also analyzed Fortran to C++ translations, shown in Figure 9 The translated code snippets maintained functional equivalence, specifically in the synchronization of shared variables between threads. OpenMP, used in the Fortran code, relies on directives such as #pragma omp cri tical, #pragma omp flush, and #pragma omp atomic C to ensure synchronization and memory visibility. In the C++ translation, equivalent functionality was achieved using std: :mutex, std::atomi: c, Both approaches ensure that x is updated and visible to the
and std: atomi c_ second thread before it prints its value, effectively synchronizing the thread actions. CODEROSETTA, Gemini-Pro, and Gemini-Ultra correctly recognized the use of OpenMP in the original code, while GPT-4 did not and opted for a different approach. This highlights the limitations metrics such as BLEU, which focus on syntax rather than functionality. Despite functional equivalence, GPT-4's translation would score lower due to its syntactic divergence. This underscores the necessity of human evaluation to ensure code correctness, as single automated metric can fully capture functional behavior.
C Decontamination Analysis
The C++ to CUDA dataset was obtained from Babel'Tower [46), which has gone through deduplication and cleaning. Notably, there is paired trained data available within the dataset, meaning the model does not encounter C++ code alongside the CUDA equivalent during training. As such, the model must rely solely on self-supervised training objectives to learn to embed source code from different
16

---

global_ void fill keekeeeeeell (int float ALPHA, flat "X, int INCX) { int (blockIdx.I blockIdz.y
global void fill_ fill_gpu(int N, float ALPHA, float "x, int INCK) int (blockIdx.x blockIdx.y
if gridDin.x) blockDim.x threadIdx.x; X[1 INCU ALPHIA;
gridDim.x) blockDim.x threadIdk.x:
I[i INCX] ALPHA;
(a) Reference CUDA kernel.
(b) CODEROSEITA generated CUDA.
global_ void fill ggutinnt N, float ALPHA, float "x, INCK) {
global void fill Fill_guu(int N, float ALPHA, float "x, int INCX)
int i blockIdx.x blockDin.x
int blockIdx.x blockDim.x
threadidz.x;
threadidx.x;
if
(1
X[1 INCK ALPHA:
X[1 INCX] ALPHA;
(c) GPT4 generated CUDA.
(d) Gemini Ultra generated CUDA.
Figure 6: CUDA kernel function to fill an array X with a constant value ALPHA, parallelized across threads.
_global_ set_sorting_offset
globbal__ set_sorting_offar (const
nrows, const int ncols, int "offsets) int tid threadIdx.x blockIdx.x blockbin.x;
nrows, const int ncols, int "offsets) {
int tid blockIdx.x blockDin.x threadidx.x;
if (tid ncols)
mrows: if (tid <= acols) { offsets[tid]
offsets[tid] tid arows;
return;
return;
(a) Reference CUDA kernel.
(b) CODEROSETTA generated CUDA.
global__ void set ser_ssoring sorting_offset. offsettttttoooooo www.sotffuuaeeeeeeeoeeeoeeo cuda(const nrows, const ncols, "offsets) { int tid threadIdx.x blockIdx.x blockDin.x:
global__ void set set_ sorting offfee(conns int nrows, const ncols, offsets) { int tid threadIáx.x;
(tid ncols) {
if (tid ncols)
offsets[tid] tid
offsets[tid] tid nrows;
(d) Gemini Ultra generated CUDA.
(c) GPT4 generated CUDA.
Figure 7: CUDA kernel to set sorting offsets based on row and column indices, where each thread computes offset value for given column index.
languages into shared embedding space. Paired data is available only the test set, which we used for evaluating the model's performance.
To assess the potential overlap between the test and the training data from BabelTower, used CoReBERRS [51] to measure similarity.
Table 6: C++ Decontamination Analysis.
CodeBERTScore range
Data
0.4-0.5
0.5-0.6
0.6-0.7
0.7-0.8
0.8-0.9
0.9-1.0
C++ CUDA Train Data
0%
1.7%
44.80%
48.61%
4.78%
0.03%
C++ CUDA Synthetic Data
0%
0.8%
33%
58%
7%
0.05%
Table presents the distribution CodeBERT scores and the corresponding amount of data each range. For example, 48. 61% of training data achieved CERTSeRe between 0.7 and 0.8 when
17

---

global_ void epl.2(ffloat "vec, float *vec1, long depth, long rows, long cols)
global__ void opl23__pp(ffoot
fleat
"vecl, long depth, long rows, long cols) { unsigned long blockIdz.x blockDin.x
unsigned long threadIdr.x blockIdz.x
blockDin.x:
threadIdz.
unsigned lang y threadIdx.y blockTdx.y
unsigned long y blockIdx.y blackDin.y threadIdx.y:
blockDin.y:
unsigned long z threadIdx.2 blockIdx.2
unsigned long blockIdx.z blackDin.2
blockbin.z;
threadidx.25
unsigned long long = 2 rows cols
unsigned long long i rows cols +
cols
unsigned long long
rows cols
unsigned long long
rews cols
cols;
cols;
unsigned long size2d cols;
unsigned long sizeld cols;
unsigned lang size3d depth rows cols cols cols;
unsigned long size3d depth rows cols
cols cols;
if (x >= cols y >= rows 2 >= depth)
if (1 + cols >= size3d) return;
return;
vecfi + cols]
(vecl[í cols]
if (i cols + sizeld) return; (vecl cols]
vecl[i]);
vecfi cols]
( sizeld) return; vec[j] 0.5 (vecl[j]);
vecl
if (j sizeld) return;
vec[i] 8.5 (v)c
(b) CODEROSEITA generated CUDA.
(a) Referece CUDA kernel.
global void opL.23. gpu(float "vec, float "vec1, long depth, long long cols) int x blockIdz.x blockbim.x
global void 3pL23__guuffai "vec, float "vec1, depth, long long cols) int x blockIdx.x blockDin.x
threadIdx.x;
threadIdx.x;
int y blockIdx.y blockDim.y
int blockIdx.y blockDin.y
threadidx.y;
threadldx.y;; int blockIdx.z blockDin.2
int blockIdx.2 blockDin.z
threadIdk.2;
threadIdx.
if cols & unsigned long long = rows cols x;
depth)
if (x cols & unsigned long long 1 2 rows cols
depth)
y
unsigned long long
rows cals
unsigned long long z rews cols
y cols;
cols;
unsigned long size2d cols: unsigned long size3d depth rows
unsigned long size?d cols; unsigned long size3d depth rows cols cols cols;
cols
cols cols;
if (1 + cols + size3d) { vecfi cols]
if (1 cols 1 sizešd) { vecfi cals] vecl(i]);
(veci[i
(vecl cols]
cols] (icl
}
if ( size2d)
if ( sizezd)
vecti] = 0.5 (vecl(j]);
vec[j] 0.5 (vecl(j));
b
(c) GPT4 generated CUDA.
(d) Gemini Ultra generated CUDA.
Figure 8: CUDA kernel opI.23, averaging 3D grid elements from veel into vec, with boundary checks.
compared against test data. Ranges with no data are omitted. score below 0.8 indicates low or moderate similarity. As shown, the majority of the training samples exhibit a CodeBERT below 0.8, reflecting minimal similarity to the test set. A similar trend was observed when we applied this analysis to the synthetic dataset.
D Unsupervised Training Parameters
D.I Training Parameters
For Masked Language Modeling (MLM) training, we use a learning rate x 10-5 and train for 100 epochs with 15% masking. After each epoch, we measure the perplexity on the validation set and save the model if the perplexity is the lowest. For Abstract Syntax Tree (AST) entity recognition, use a learning rate of 5 x 10-6 and train for ten epochs. We then create the encoder-decoder
18

---

model by transferring the encoder's weights to initialize the decoder, so the decoder begins with some foundational knowledge.
For Denoising Auto-Encoding and Back Translation, we use a learning rate of 5 x 10-5 and train for 20 epochs. For Denoising Auto-Encoding, we the masking to 15%, token dropping to 25%, and token insertion to 15%, with a denoising ratio increasing by 2.5% per epoch. Finally, for fine-tuning, we use learning rate of 5 x 10-S for epochs. At each training iteration, save the model with the lowest validation loss. All the parameter values are determined empirically through detailed hyperparameter tuning.
D.2 AST Entity Recognition Tags
Table 7: AER Tags.
Tag ID
Tag Type
identifier/variable
function
5
type identifier
7
primitive type (int, float, etc.)
9
number literal
11
& pointer expression/rcference
13
*pointer declarator
constant
The AER tags used in pretraining are shown in Table 7
D.3 Dataset Statistics
A detailed overview of the dataset is shown in Table g
E Impact of Beam Size
We conducted beam search decoding with varying beam sizes, returning the top candidate in each case. The results, shown in Table 9 indicate that CODEROSETTA consistently produces the same output, regardless of the beam size.
F Analysis of Generated Code from CODEROSETTA and Closed-Source LLMs
C++ + CUDA: this part, we compare the code generated by CODEROSETTA, GPT4, and Gemini- Ultra. As the BabelTower model and its code are not publicly available, we were unable to access them, However, the BabelTower paper highlights kernel where the model failed to generate CUDA code due to syntax error when defining keyCharPtr, as shown in Figure 10 In contrast, CODEROSEITA successfully generates the correct CUDA code. recognized the if condition and improved the readability of the code by inverting the if statement, similar to the approach taken by Gemini-Ulte and GPT4. Additionally, CODEROSETTA adheres to the preferred practice of declaring variable or pointer before assigning value, which is why first keyCharPtr' is defined out of the if statement.
interesting to note that CODEROSETTA also
We demonstrate another example in Figure II where CODEROSETTA accurately reproduces the reference CUDA kernel without adding unnecessary lines of code, such host main function, which is often seen in other models.
Fortran >> C++: Figures 9 12 show examples of C++ code generated by CODEROSETTA in comparison with other LLMs. Despite CODEROSETTA" 's smaller size, it effectively translates Fortran code into correct C++ code.
Moreover, we also evaluated our model terms C++ >> Fortran translation The results indicate the capability of CODEROSETTA in translating and from Fortran code.
19

---

Table Dataset statistics for C++, CUDA, and Fortran programming languages.
Programming Pair
Train
Valid
Test
Size
C++ CUDA
243,008 (unpaired)
184
180
626.1 1 MB (Train)
139.1 KB (Valid)
141.9 KB (Test)
C++ 6+ Fortran
474,856 (unpaired)
N/A
33
1.2 GB (Train)
282 (paired)
99.0 KB (Test)
Table Effect of different beam sizes on C++ to CUDA translation.
Metrics
Beam Size
BLEU
CodeBLEU
76.47
78.43
76.90
78.84
10
76.85
78.87
25
76.70
78.67
50
76.61
78.65
Table 10: C++ Fortran translation results in terms of CodeBLEU.
CodeBLEU Gemini Ultra
Model
GPT4
35.32
33.64
Gemini-Pro
32.36
COPEROSETTA (Ours)
70.46
F.1 Common Issues and Pust- Post-processing in CopERosera-Generatte Code
Code translated by large language models like GPT-4 often includes additional caller functions that extend beyond the scope of the original function. In contrast, code translated by CODEROSETTA may occasionally fail to compile despite being syntactically correct. We identified two common issues in the code generated by CODEROSETTA and applied a simple post-processing method ensure fair comparison across models.
The first issue involves the use of generic types, which enhance code efficiency but require explicit type definitions at compile time. Figure 13a]shows the use a generic type, although the necessary definition is missing. Adding the type definition, shown in Figure 13b] resolves the compilation issue. The second issue relates to misses variable initialization in the function definition, as shown in Figure T3c By initializing the required variable, demonstrated in Figure [3d] the compilation problem is resolved. Lastly, for longer code snippets, CODEROSETTA occasionally omits the closing curly bracket.
G Discussion on Unsupervised Training
G.1 Fine-tuning for Code Translation
In the context of code translation, paired data is scarce. However, our model benefits from a strong foundational understanding of code translation acquired through unsupervised and self-supervised pre-training on 243K training examples for C++ ++ CUDA. We demonstrate that fine-tuning, even with small amount of synthetic witthou verifying the one-to-one mapping between the generated samples and the input code supervised manner-can can further improve the model's performance. Specifically, fine-tuning with merely 5K paired samples (less than 2% of total data) generated by larger
20

---

models still led to significant performance gains. While synthetic data may introduce some errors (as large models can make translation mistakes), the combination of this foundational pre-training and fine-tuning with small synthetic dataset yields further improvements.
G.2 Back Translation
Back Translation (BT) has been extensively used in unsupervised translation tasks for both natural language and code. We integrate this technique with the denoising auto-encoding (DAE) objective, ensuring that the model is not trained exclusively alternates between DAE and BT for each batch of data. This prevents the model from relying solely on BT and "cheating' by outputting the input source code an intermediate translation. To better understand this behavior, we analyzed the intermediate outputs during back translation.
single objective. During training, the model
For instance, Figure 14 shows a C++ input and its corresponding intermediate CUDA translation. As shown, while the model attempts to translate the code to CUDA, the output contains errors, such as the undefined variable j. In the back translation process, this noisy CUDA code output is fed back into the model, which then attempts to reconstruct the original C++ input. Since the model alternates between languages during back translation, it occasionally generates noisy CUDA or C++ code. This approach improves the model's robustness when handling noisy inputs in translation tasks.
H Translation Pitfalls: Invalid Tokens in Target Language
During translation between programming languages (e.g., from C++ to CUDA), certain entities, libraries, and syntaxes present the source language may not be valid or supported in the target language. For example, C++ Standard Template Libraries (STL) such as std: unique_ptrr not compatible with CUDA's device code and must be excluded from translations. The pre-training process in CODEROSETTA equips the model with semantic knowledge of both source and target languages, reducing the frequency of invalid tokens during translation. Nonetheless, there are still instances where the model may fail correctly map common source language entities to valid target language counterparts.
While test set contained no occurrences of std: :unique_ptee ptr, deliberately included this construct in a separate C++ code example to evaluate CODEROSETTA'S 's handling of STL-specific constructs. Figure 16 demonstrates this case, where the model successfully generates CUDA code by omitting the unsupported std: :unique_pp ppr in the device kernel. Instead, the use of std: : uniique que_ptr is correctly retained in the host kernel, specifically in the main function, which runs on the CPU. Since CODEROSETTA is trained to focus on device function generation, the translation accurate in this instance.
On the other hand, Figure I7 illustrates a case of incorrect translation, where CODEROSETTA, along with other large closed-soure models like GPT-4, Gemini-Ultra and Gemini-Pro, failed to generate valid CUDA code. The translated code includes the line *rho 0; which initializes the rho variable to zero. In multi multi-threaded GPU environment, executing this kernel across multiple threads and blocks simultaneously can lead to a race condition, as multiple threads would attempt to write to the same memory location concurrently. Without synchronization mechanisms like atomic operations or reduction techniques, this results in unpredictable and incorrect behavior. The correct approach would be initialize rho in the host code and use atomi cAdd accumulate values in the device code safely.
I Prompt Template and LLMs
In this section, we describe the prompt template used translate between different programming languages and libraries. The template, shown in Figure 15] served as the basis for all translation tasks, with language-specific adjustments made by updating the source and target languages as required. For this study, we use OpenAl API's GPT-4 API, using a fixed temperature of zero to ensure deterministic outputs across all models, including CODEROSEITA. All queries were executed on May 18th, 2024, ensuring consistency in results throughout the experiments.
21

---

J Additional Related Work
Automatic parallelization. Early efforts in auto-parallelization were primarily focused on identifying independent loops that could be executed in parallel. Renowned compilers like the Portland Group (PGI) and Intel's C++ Compiler (ICC) have embedded cutto-prallelization capabilities, offering bagma-based hints to guide the parallelization process. These compilers analyze loop dependencies, data flow, and potential side effects to generate parallel code, often targeting OpenMP or MPI for multi-threading and distributed computing, respectively. The advent of Polyhedral model-basee tools marked significant advancement auto-parallelizztion techniques. The Polyhedral model [6] offers powerful algebraic representation for optimizing loop nests with affine bounds and access patterns. Pluto [8] is an auto antooparallllization tool that utilizes the Polyhedral model to perform loop transformations, tiling, and fusion for effective parallel execution while considering data locality optimization. PPCG (Polyhedral Parallel Code Generation) +4 is another tool that exploits the polyhedral model to automatically optimize and generate parallel code from high-level abstractions, targeting multicore CPUs and GPUs.
Neural machine translation. STT 37 extends the original work (36 by adding automated unit testing. TransCoder-IR [39] extends even further by exploiting LLVM IR for program translation. HPC-GPT [13] uses GPT4 to create an instruction-answer dataset for two tasks (AI models and datasets for HPC and data race detection), then Llama model [43 is supervised tuned on this dataset. Pan et al. [31 provided one of the first studies on the types of errors that are often produced in code translation.
There is a growing number of large language models (LLMs) for code generation 5 35 40
50
26
4
29 3.
Most of these works focus mainly on natural language to code generation. Although these Code LLMs can generate code in various programming languages, Python, in particular, has received more attention compared others. This could be due to the number of available benchmarks that
though other programming languages have been gaining more attention recently as well 99. Despite the growing number of Code L.LMs, these models are typically not specifically trained for code translation, even though they can perform code translation to some extent, as shown by Pan et al. (31]
assess Python coding capabilities [10 23
K Limitations
While CODEROSETTA demonstrates promising results in code translation, several avenues for future work exist. Currently, CODEROSETTA targets C++ - CUDA and Fortran -> C++ translation. Extending its capabilities encompass wider range of HPC languages and parallel programming models would further broaden its scope. In addition, we plan to improve the set of entity categories used in AER to capture better representation of code semantics. This will involve incorporating additional tags for constructs like data types, control flow mechanisms, and parallel coogaamming-speciffe primitives.
22

---

#include estdio.h>
#include <stdio.h> #include comp.h>>
#include <omp.
int mainO
int main() {
int y:
int y;
#pragna oup parallel nun (2)
#pragma onp parallel mun threads
int thrd wwt.get.thrred.mmng
int thrd omp-get.. thae
(thrd = 0) {
#pragna oup critical
if (thrd 0) #pragna opp critical
== 18;
praama cup flush(x)
#pragua atonic write
#pragna onp flush
y =
#pragna cup atonic write
else
y
int tmp 0;
else {
while (trp {
txp
#pragua oup atonic read acquire tap y;
shile (tip == 0)
#pragna omp atonic acquire
pragma oup critical
#pragna omp critical
if (x I= 10) printf("x xdvn", x):
18) printf("x sala", x):
return 0;
return 0;
(a) Reference C++
(b) CODEROSETTA generated C++.
#include ciastrean>
#include <iostrean>
#include <atonics
#include comp.h>
#include <thread>
#include <mutex>
int nain() {
std::atooie-iint x(0);
int x, y. thrd; int tmp;
std::atonic y(0);
std:: idduttuex ntx;
vold (hrt thrd) if (thrd = 0) {
0;
#pragna oup parallel t2) private(thrd, tmp)
std::lock guard gtd::dook.u mutex: lock(mtx);
x1store std: meenory_arder_relnee
thrd tmp_get_thread_mmm)
std::atomic. fence std: std::menory_com order release
fence(
if (thrd 0) #pragna oup critical
);
y.store(l, menory order relaxed); } else tmp 0; while (trp 0)
= 18;
#pragna flush(x)
tap
#pragna oup atonic write
loldd menory menory wwwp:deemooooorreereeeeeeeee acdirui
}
else {
tmp
muter mutex: lock(ntx);
while (tep == 0) {
std::cout "x
#pragma orp atonic acquire
<6 loaddddt:d: menory relaaee
<< sed::endl;
}
#pragna omp critical
std::cout "x std::
int main(
std:: (threeadFmm,, 8):
std:: thread t2(uhreadFunc, 1):
tl.join();
joinn)
return 0;
return 0:
(c) GPT4 generated C++.
Gemini Ultra generated C++.
Figure 9: A C++ OpenMP example with thread sync using atomic operations and critical sections.
23

---

void kerne (umsigme int key, char"
global_.. void kernel key, char "input_ str cuda, unsigned char tttttteeetttttuut.tt cuda, int lnput_)engtt)
input. cuda, unsigned char" possible plaintext str_ssstrse cuda, int input_ int id: keyCharPtr;
int id threadIdx.x blockDin.x blockIdx.xx;
if (id on input_length) return;
for 8: id < Input__length. id++) { int keyIndex id
int keyIndex id % char "keyCharPtr ((char *)8key); keyChar keyCbarPurkeyIIdee]]]
keyCharPtr (char*))key; char keyChar keyCharPur(keyIndez]]
possible_plade possible.plaintet..u baalle_paintextssr_..oe [ida(idd] [i]] keyChar input_str_cuda(id]:
keyChar
input_str_cudaid]]
(a) C++ code.
(b) CUDA reference.
global void KernelXor(
global_
(unsigned key,
unsigned int key,
char" input_str_cudt unsigned char"
char iinput_str__cude
possible_plan wtt.bleepllintettsst..oom int input Length) { int id blackIdx.x blockDim.x threadIdx.x;;
unsigned char "possible plaintext cuda, lenut_length)
int id blockIdx.x blockDin.x threadIdx.x;
char keyCharPtr;
if (id >m input_length)
if (id input_length) {
return;
keyIndex keyCharPtr ((char))(key); char keyChar keyCharPtr bessible_pliineer_sst_...u. keyChar input_str [ipu__str.cudaid]]
int keyIndex
keyCharPtr ((char ')akey);
char keyChar [eychar)ttr
bussible_plaintet..ee wtw.eeettsssuutaaaaeeeeeeaaaaeee [id] keyChar
input_str_coudaid]]
(c) BabelTowwer
(d) CODEROSETTA.
gglobal void kerneloor(umsigned int key,
global void kernel int key, char" input
char "input_str. cuda,
unsigned char
unsigned char
possible_plaintee www.pplltnttstttteeeeeeeseeeee strpcctsssr__uuuu int input_length) {
bussible_plaintet.t..
int input. length)
int id threadIdx.x blockIdx.x blockDin.x;
blockIdx.x blockDin.x
threadldx.x;;
(id inpuut_lnnghh)
if (id input_length)
int keyindex = id x 4;
Int keyIndex char "keyCharPtr (char *)akey;
char keyCharPur ((char") char keyChar = keyChar
char keyChar leychar
possible plaintext str. cudalid] keyChar input_str. stt]ccddd_dd]];;
possible_plainte hacll__lliittxt_ttee www.ssttaaaaaaeeeeeeeeeeeeeaaass [id] keyChar input_str__udd(id]]
(f) GPT4,
(c) Gemini Ultra.
Figure 10: Comparison the generated kernelXor CUDA kernel.
24

---

void pou cpunt N. float ALPHA, float" x, int INCI, float" Y. int INCV) { int
-lloball.. void pow_ www.kernll(ff (int ы, float ALPHA, float "I, int INCX, float "Y INCT) { int i (blackIdx.x blockIdx.y gridDin.x) blockDim.x + threadIdk.x:
for
Y[i INCY] pow(X INCX], ALPHA);
if (1 < M) Y[i INCY] powf INCX], ALPHA);
(a) C++ code.
(b) CUDA reference.
global void (int N, ALPHA, float "x, int IMCK, float "Y, int INCY) int 1 (blockIdx.2 + blockIdx.y
global__ pix_
ALPHA, float 'X, int INCX, float "Y, int INCY) ( int blockIdx.x blockDin.x threadIdx.x;
gridDin.x) * blockDim.x threadIdx.x; if N) Y[i INCY] pow(X[i + INCX], ALPHA);
(i < X) INCY] powf(X(i INCK], ALPHA);
void pow cpu(int N, float ALPHA, float "x, int
(c) CODEROSEITA.
INCX, float "T, int INCY)
int threadsPertlack int blocksPerGrid (K threadsPerSlock
256;
1) threadsPerittck,
www.gpe:ccbloccksPerrrddd
threadssPerElock==>0.. ALPHA, X. INCX, INCI: cudaberviceSyyccheemieee))
(d) GPT4.
// CUDA program to calculate the power each element of an array
global__ void (int N, flaat ALPHA, float "x, int INCX, float "Y, int INCY) { int blockidx.x blockDin.x threadIdx.
1f (1 8)
Y[i IECY] powf([[i IKCX, ALPHA);
int minn {
// Rumber of elements
array
float ALPHE:; // Exponent to raise each element to float // Input array
int INCK; // Stride of the input array
float 'F; // Output array
int INCY; Stride of the output array
// Initialize the input and output arrays
1/ Allocate nemory for input output arrays device
// Copy the input array to the device
// Call the kernel to compute the power of each element of the array
// Copy the output array back the host
// Free the device nemory
0;
(c) Gemini Ultra.
Figure 11: Power of elements CUDA kernel.
25

---

subroutine add sp_data implicit none integer if (timeron) call timer starrssttrtttttttt
void add {
int 1, k, int thread_id orp-get_.hreaa wwp_get.thheedd_mm (timeron thread_id timer timer_staree start(T ADD);
do 1, nz2
do ny2
#pragna for for (k 1; l < nz2; k-t) { for for 1; < nx2; i++) for
do nx2
ny2; jos)
unn, 1, 3. 10 ubn, 1, J, rhs 3.
m+)
end do
end do end
}
if (timeron) call tiner_stop(t_add)
return
end subroutine
if (tineron & thread_id = 0) timer_stop(T_ADD)
(a) Fortran code.
(h) C++ reference.
add(
sinclude "sp_data.h" void add() { int 1, 3. k, mi
int 1, k, m; thread id oup-get..three munO: if (timeron & thread_id 8) timer stne__s)artT
if (tineron) { start(t add):
#pragna omp for
nz2; kit)
1; nz2; kit)
for (j 1; j on ny2: j+4) {
for (j ny2; j++)
for (1 = 1: 1 <= ax2; 1++) {
for = 1; <= nx2; 1++) {
for [1]
m++)
for [[m [iml
m--)
u(k)
[bs
if (timeron & thread id 0) { tiner Sttr_stopTT_ADD)_
if (tineron) { timer_stop (imer_stop(t_add)
}
(c) CODEROSEITA.
GPT4.
void addO
using namespace sp_data; J. m; if (tineron) [
timer start(t add):
for Ok 1; du nz2; k+t) { for 1: ny2; j+t] {
for (i = 1; i <s nx2; i+t)
1: 5; m=+) {
uln, 3. 10 uGn, 1. rhs(n, 1. J. 10:
if (timeron) { timer_stop(t_add)
return;
(e) Gemini Ultra.
Figure 12: Flement- wiinent-wiieeee addition of two multi-dimennional arrays.
26

---

global__ void vet_valid_mask.guucce T "score, score_thr, int nask, int dins)
template <typename T>
-lloball... void set_valid mask gpu(const (oot "score, I score thr, int "valid mask, int dims) {
int tid blockIdx.x blockDin.x threadIdx.x;
int blockIdx.x blockDin.x
if (tid < dims)
threadIdx.x;
(score[tid) score_thr)
dims) {
valid_nask(tid]]
if (score[tid) score_thr) (
} else maskiiis
malid_mask(tid) = else { valid.mask(tidd = 0;
}
(a) Generated CUDA code.
(b) Corrected CUDA code.
global void get_ev(double "old_arr, double new) {
global__ void (double *old_arr, double "new_are acr, int size)
blockIdx.x blackDin.x threadidx.
int blockTdx.x blackDin.x
threadidr.x;
if (tid size)
(tid size) {
awr[tr]] old arr[tid];
wwwaar[[tid] old arr[tid);
Generated CUDA code
(d) Corrected CUDA code
Figure 13: Post Compilation fixes on CUDA kernel.
static makexgraph(graph "g, xsord tb,
.lloblll.... void makexgraph(graph "g, xword "h,
int n)
setword gi; int
setword gi;
int blockIdx.x blockDim.x
aword hi;
threadIdx.x;
Hi)
hi;
hi 8:
for (: i < blockDin.x griddin.x)
9[1];
while (gi) {
hi 8;
j = PIRSTBIIIZZGi);
gi = g[i];
bit[i];
while (gi)
hi
[|= xBIT());
j FIRSTBITHZ(!1); gi bit
h[i] bi:
bi | XBIT());
h[i] hi;
}
(a) Input C++ code.
(b) Intermediate CUDA generated code.
Figure 14: Back translation intermediate results.
You are an expert translating C++ programs to CUDA programs. Given the C++ program below, translate it to CUDA. Ensure that the CUDA
program is compatible with the C++ program and preserves the semantics of the original code.
Just print the CUDA program remove any unnecessary comments. Surround the generated CUDA program in #start and #end.
### C++ Program: ccpprammmcppcccceecoomtee
## CUDA Version:
Figure 15: Prompt for translating C++ to CUDA.
27

---

void InitiallzeArray(st::: pte-in(()+ arrey, int size)
globball....................bb.. void Enttiallieehrryyiit "array, size) {
(int 8; 1 size: ++1) {
int i blockIdz.x blockDin.x
array[ij 1 10;
threadIds.x:
( < size)
array 10;
(a) Example of C++ code with std:::tiiee :unique_ptr
(h) Translated CUDA code.
Figure 16: Example of translation of a C++ code with std: :unique_ptr
void int munOfNucl, psi, const double" occilo, double" rho, char debug)
double"
_llbaal.... void getRho(connst numOffuucl, const double "psi, const double "accNo, double "rho, char debug)
"rho 0; for (int = 8: 1 nunDfNucl; ++1) { *rho occNo[s] psi[i]] psi [3];
"rho
for (int blockIdx. blockDin.x threadidx.x; numöffuur;; blockDin.x griddin.x) { occNo[i] psi[i] psi[il;
if (debug
printf("DEBUG print of RB0:AA *fnhhss last line (RHO). \n\n", *rho);
(debug 1) printf("DEBUC print of RHO:\nRHO %fnnhhs the last line (RHO). AmAn", "rho);
(a) C++ Code
(b) Wrong translated CUDA code.
Figure 17: Example of a failed C++ to CUDA translation.
28

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer:[Yes)
Justification: We provide bidirectional encoder-decodee transformer model that is capable of learning to translate programming languages as well as their HPC extensions. Results indicated
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of our study are addressed in Appendix(Section K First, our research does not extend CODEROSETTA to a broader range of applications. Second, we have not incorporated several potentially beneficial AER tags. Despite these limitations, our methodology has demonstrated significant effectiveness in two critical tasks in the HPC community: translating C++ to Fortran and C++ to CUDA. We plan to address these limitations in future research efforts.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or a speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
29

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We submitted the code as supplemental material and also created an anonymous repository (https:// /atp:////onmmmuss 4open. scciencerr/eeuu http://eerrissccccrootttttttttt CAB2/) coderosetta-
include everything needed to reproduce the experiments. Please refer the README file in the repository for detailed steps.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
30

---

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but should possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We submitted the code as supplemental material and also created an anonymous repository (https:/ hanp:s///nnmmmouus.mmomm 4open. science r/neuripsss..eeeee..rrrrrrr wwtpsssoeooeooeeoeeeeee CAB2/) to include both data and code. Please refer to the README file in the repository for details.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips.co
wwt.iiisssooooocyi.o for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines http:iss.../plllli/aauuaa/////io
for more details. The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4 discussed experimental settings/details such as data preprocessing, data splits, syper-parammters, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: resources.
Our work does not include statistical experiments due to limited computational
Guidelines:
31

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?| Yes]]
Question:
Justification: We discussed the information the computer resources at the beginning of Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
32

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators of the data and models used in this work have been properly credited. The license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
33

---

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, [paperswi thcode. wwt.cooo/caaa. da has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: Our work does not release new assets. We use existing, publicly available resources for analysis.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
34

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval. you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "P8yxKISd1J", "reviewer": "Reviewer_9e24", "summary": "The paper looks into the problem of unsupervised code translation with a particular focus on parallel programming languages like C++ to CUDA. It trains (relatively) small encoder-decoder models compared to current LLMs while achieving very strong performance on translation tasks. It incorporates program-specific terms in loss such as token frequencies, and AST structure to achieve this.", "questions": "1. The AST structure loss seems interesting. Do the authors believe it would also improve embeddings for monolingual encoder models?\n\n2. Backtranslation-based unsupervised learning. The authors use back-translation to improve the translation performance of the model. However, for given languages pairs A and B (going from A -> B -> A), if the intermediate sample B is not constrained, the model can cheat and not do the translation job currently (as an extreme example, just copy the tokens from A). Can the authors clarify how the setup avoids such failure models?\n\n3. What tokenizers do the authors use for these languages which are usually less well represented in the web data?", "limitations": "Authors have limitations in the the appendix.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Significance - This is an impactful problem and the resulting approach competes with very large LLMs with ~small models. The authors perform appropriate ablations and the proposed AST structure and token frequency-based denoising in loss are also effective. \n\n2. Clarity - The paper is well-written and provides appropriate context and background around methods.   \n\n3. Originality - While people have tried including program-specific constructs in loss previously for encoder models applying it to the translation domain in novel. The denoising scheme based on language-specific token frequencies is intuitive and is proven effective from thorough ablations \n\n4. Qualitative analysis of compilation errors - It was good to see authors performing an analysis of mistakes in model-translated samples. The distribution or errors and thus motivated future direction of using execution feedback does look promising.", "weaknesses": "1. Evaluation and metrics. Perhaps a challenging aspect of the results is the evaluation benchmarks and datasets used. \n  a. Benchmarks. The authors achieve considerably strong results on the C++ to CUDA evaluation sets with BLEU and CodeBLEU numbers higher than 75. Do authors perform some analysis on contamination or deduplication on the benchmark and collected training sets?\n  b. Evaluation metric. Functional correctness has emerged to be the winning metric in the code generation space. However, it seems more challenging to apply that to the translation problems. High BLEU might not necessitate better translations. Similarly, comparability as a metric does not ensure that the translation is correct. While it might be hard to design functional correctness-based evaluation, the authors do not perform any other evaluation to assess the quality of translation beyond the, somewhat unreliable automated metrics.\n\n2. Performance on proprietary LLMs. GPT-4 and Gemini-Pro depict considerably worse BLEU for some language pairs. It would be useful to highlight the failure modes since it might also highlight some issues in BLEU-like metrics.\n\n3. Missing details on the datasets. The authors can provide more details on the datasets and benchmarks used in terms of document length and document counts. Similar statistics for the benchmarks and model generations would be useful to understand the complexity of the task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Deduplication analysis**\n\nThe C++ to CUDA dataset was obtained from BabelTower [26]. This dataset has already gone through a rigorous deduplication and cleaning process. Moreover, there is no paired trained data available in the training set. This means the model can not see a C++ code and its CUDA equivalent during training as such data does not exist. The model has to rely on the self-supervised training objectives to learn to embed source code in different languages into the same embedding space. Only for the test set, we have paired data, which we have used to evaluate our model.\nWe further evaluated the similarity between the test and the trainset from BabelTower [26] using CodeBERTScore. The following table shows the ranges for CodeBERT score and the amount of data that falls in each range. For example, 48.61% of training data have CodeBERTScore between 0.7 and 0.8 when compared against test data. Ranges with zero data are omitted. A score below 0.8 shows low or moderate similarity. As can be seen, the majority of samples in the training set have CodeBERTScore lower than 0.8\n| 0.4-0.5 | 0.5-0.6 | 0.6-0.7 | 0.7-0.8 | 0.8-0.9 | 0.9-1.0 |\n|---------|---------|---------|---------|---------|---------|\n| 0%      | 1.7%    | 44.80%  | 48.61%  | 4.78%   | 0.03    |\n\nWe randomly examined the training samples that had high similarity scores.\nFor example, this is a sample from the test set:\n\n```__global__ void scale_dev(float *array, float scale, int N) {\n    int idx = blockIdx.x * blockDim.x + threadIdx.x;\n    if (idx < N) {\n        array[idx] *= scale;\n    }\n    return;\n}\n```\n\nAnd the most similar sample from the trainset is this one with CodeBERTScore of 0.94:\n\n```\n__global__ void set_val(float *data, float val, int size) {\n    int idx = blockIdx.x * blockDim.x + threadIdx.x;\n    if (idx >= size) return;\n    data[idx] = val;\n}\n```\n\nWe can see that despite some similarities between the two code snippets, their functionalities are different. One is scaling the values of an array; the other is populating an array.\n\n**Functional Correctness**\n\nPlease refer to the global response.\n\n**More details on datasets**\n\nDetails of the datasets have been added to the appendix. Please let us know if any other information needs to be added. We will make sure to move the datasets' table to the main paper if space permits.\n\n\n**AST for monolingual models**\n\nAST entity recognition (AER) pre-training task enables the model to identify various entities in the source code. Examples of these entities are identifiers, functions, and primitive types. By allowing the model to identify such entities, the model can recognize where in the structure of the source code each entity should be. For instance, it recognizes that a type identifier precedes a function entity in C++. This would essentially improve monolingual encoders. However, we do not believe it can fully replace self-supervised training objectives such as MLM. Instead, it can be considered a fine-tuning step on top of MLM to improve the model's embedding capabilities further.\n\n\n**Back translation cheat avoidance**\n\nBack Translation (BT) technique has been used in unsupervised translation-related works both for natural language and code translation. We couple this training objective with the denoising auto-encoding (DAE) training objective in a way that the model is not solely trained on one of these objectives. Instead, during training, the model iterates between DAE and BT for each batch of data. This, in hindsight, prevents the model from solely depending on BT and cheating. To further investigate this, we looked into intermediate outputs in back translation (Language B, as you mentioned).\nFor example, this is one of the C++ input codes to the model (language A):\n```\nstatic void makexgraph(graph *g, xword *h, int n) {\n    setword gi;\n    int i, j;\n    xword hi;\n\n    for (i = 0; i < n; ++i) {\n        hi = 0;\n        gi = g[i];\n        while (gi) {\n            j = FIRSTBITNZ(gi);\n            gi ^= bit[j];\n            hi |= XBIT(j);\n        }\n        h[i] = hi;\n    }\n}\n```\n\nAnd this is the intermediate result (language B):\n\n```\n__global__ void makexgraph(graph *g, xword *h, int n) {\n    setword gi;\n    int i = blockIdx.x * blockDim.x + threadIdx.x;\n    xword hi;\n\n    for (; i < n; i += blockDim.x * gridDim.x) {\n        hi = 0;\n        gi = g[i];\n\n        while (gi) {\n            j = FIRSTBITNZ(gi);\n            gi ^= bit[j];\n            hi |= XBIT(j);\n        }\n\n        h[i] = hi;\n    }\n}\n```\n\nWe can see that the model has tried to translate the code to CUDA code. However, the translated code contains issues. For instance, variable `j` is not defined. In the context of back translation, this noisy translated CUDA code will be used as an input to the model, and the model is responsible for reconstructing the original input C++ code. Since we iterate over languages with back translation, sometimes the model has to generate noisy CUDA code and sometimes C++ code. This step also enables the model to be able to translate from noisy input data.\n\n\n**Details on tokenizer**\n\nBPE is one of the most potent and popular types of tokenizers. For CodeRosetta, we loaded a pre-trained BPE tokenizer from uniXcoder[1] and then trained it further on our trainsets. Training tokenizers from scratch is notoriously time-consuming; this is why we load a pre-trained tokenizer that has already seen some C++ data. \n\n[1]Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., & Yin, J. (2022). Unixcoder: Unified cross-modal pre-training for code representation. arXiv preprint arXiv:2203.03850.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G8idhcDCtL", "reviewer": "Reviewer_djHa", "summary": "The paper presents CodeRosetta, a transformer model designed for unsupervised code translation and their high-performance computing extensions, such as C++ to CUDA and Fortran. By introducing novel pre-training objectives like AST Entity Recognition and customized Denoising Auto-Encoding, CodeRosetta effectively learns parallel programming syntax. The model demonstrates superior performance, surpassing state-of-the-art baselines metrics like BLEU / CodeBLEU and compilation accuracy.", "questions": "1. Is there an error in the left part of Figure 4?\n2. The phrase \"compare CodeRosetta against their best fine-tuned model, StarCoder (15B)\" is unclear. While StarCoder's pre-training data does include Fortran code, the notion of \"best fine-tuned\" is not well explained. Could the authors clarify this?\n3. The performance of the three closed-source models in Table 4 is surprisingly low. Could the authors provide more details on the testing procedures?", "limitations": "I do not identify potential negative societal impacts in this paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Strengths**\n\n1. CodeRosetta introduces novel pre-training objectives to solve translation tasks about CUDA and Fortran, such as AST Entity Recognition and customized Denoising Auto-Encoding.\n2. CodeRosetta is one of the first to handle code translation of code HPC extensions, marking an advancement in code intelligence.\n3. The model outperforms state-of-the-art baselines in key metrics, achieving higher BLEU and CodeBLEU scores", "weaknesses": "**Weaknesses**\n\n1. Although novel pre-training objectives have been introduced, the authors have not methodologically distinguished these from predecessors using AST or Denoising objects. Training with code structures is prevalent in code learning, as seen in models like CodeT5[1] and PLBART[2]. The authors should discuss these previous methods and variants of structural modeling, as referenced in papers like [3,4].\n2. The experimental section lacks some rigor. For instance, in the Educational value filtering, \"randomly sampled 100,000 C++ files from Stack V2 and employed GPT-3.5 to assess their ‘educational value’ for learning C++ coding concepts\" poses challenges to reproducibility. The use of GPT-3.5 to evaluate C++ coding concepts and the prompt engineering methodology warrant further clarification. Additionally, some experimental details in Table 4 are not clearly presented.\n3. Upon reviewing the data composition of models like StarCoder (StarCoderDataV1), which includes 58,355 CUDA and 165,446 Fortran files, it appears that existing models like StarCoder or DeepSeekCoder could be directly employed for research. Significant engineering effort in the paper was focused on vocabulary issues, such as inserting CUDA-specific tokens, yet there is no discussion on how existing code LLMs handle out-of-vocabulary problems with languages like CUDA.\n\n[1] CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation\n\n[2] Unified Pre-training for Program Understanding and Generation\n\n[3] A Survey of Neural Code Intelligence: Paradigms, Advances and Beyond\n\n[4] A Survey on Pretrained Language Models for Neural Code Intelligence", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Difference with relevant works, CodeT5[1] and PLBART[2]**\n\nThank you for providing us with these references. Indeed, Denoising Auto Encoding (DAE) is a popular technique when it comes to training Encoder-Decoder models, as both CodeT5 and PLBART use it, too. Some of the noising strategies, such as masking and token dropping, are common. One of the key differences in the noising strategies employed by CodeRosetta is their language-specific characteristics. For example, instead of random token dropping, we employ weighted random dropping that gives a higher probability to language-specific reserved keywords, forcing the model to develop a deeper understanding of the target language’s semantics and its structure. Another noising strategy is token insertion, which encourages the model to distinguish between valid and invalid tokens for the target language. We make sure to update the paper and distinguish CodeRosetta’s noising strategies vs. common strategies in the literature, specifically in the related works section.\n\n**Details on filtering CPP files from StackV2**\n\nWhen training CodeRosetta, We aim to present the same amount of data for the languages involved in the translation. For the translation of Fortran to C++, we used data from the StackV2 data corpus, which contains far more C++  than Fortran. It is possible to randomly subsample C++ source code files to have an equal number of samples for C++ and Fortran. However, instead of random subsampling, we tried to extract C++ samples that have ‘good quality’. We employ a technique similar to “Textbooks are all you need”[1]. Using a larger model to assess the quality of some portion of C++ data in StackV2 (due to the budget) and then training a classifier model to classify all C++ data in StackV2, then randomly selecting from the C++ files that have been predicted to have good educational value. We make sure to rephrase this part of the paper and provide more clear explanations with references. Thank you.\n\n[1] S. Gunasekar, Y. Zhang, J. Aneja, C. C. T. Mendes, A. Del Giorno, S. Gopi, M. Javaheripi, P. Kauffmann, G. de Rosa, O. Saarikivi et al.,“Textbooks are all you need,” arXiv preprint arXiv:2306.11644, 2023`\n\n\n\n**Clarification on Table 4**\n\nThe table presents Fortran to C++ translation results (apologies for the mistake in the caption of the title; we make sure to rectify its title) on the dataset provided by Bin et al. [9]. In this dataset, the test set contains 33 paired samples, meaning for each Fortran code there exists an equivalent C++ code. We aim to analyze the performance of CodeRosetta for the task of translating from Fortran code to C++ code. The experimental results in this table indicate that for the Fortran to C++ translation, CodeRosetta is effective.\n\n**Comparison with StarCoder or DeepSeekCoder**\n\nThank you for suggesting a comparison with StarCoder and DeepSeekCoder.\nWe have provided the results in the following table using the prompt format provided in Figure 6 of the paper, with adjustments depending on which languages we are targeting. Results indicate that StarCoder is relatively better than DeepSeekCoder, and it has a CodeBLEU score close to Gemini-Ultra. We make sure to update the paper and reflect the result of these open Code LLMs.\n\n|                              |       | C++ to CUDA |      | Fortran to C++ |\n|------------------------------|-------|-------------|------|----------------|\n|                              | BLEU  | CodeBLEU    | BLEU | CodeBLEU       |\n| DeepSeek-Coder-V2-Lite-Base (16B)  | 26.63 | 21.46       | 0.77 | 12.09          |\n| Starcoder2-15b-instruct-v0.1 | 37.58 | 62.58       | 5.71 | 18.21          |\n\n\n**Fine-tuned StarCoder(15B)**\n\nStarCoder has been fine-tuned for the task of C++ to Fortran Translation by Bin *et al.* [9]. They have fine-tuned other models as well, among them, fine-tuned StarCoder was able to translate C++ to Fortran with higher accuracy. We compared CodeRosetta against the fine-tuned StarCoder since this model was the one that had the best results among others.\n\n\n**Low scores of ChatGPT and Gemini**\n\nBaseline papers have used BLEU and CodeBLEU to evaluate the translation results. We have used the same metrics. However, LLMs generally do not perform one to one translation. They typically provide descriptions and instructions about code and how to execute it. This could impact the metrics. Moreover, to reveal why other LLMs' results are lower than CodeRosetta, we manually examined the generated code. Please refer to the global response for further explanation on this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C1VAF516Hh", "reviewer": "Reviewer_W1wi", "summary": "This paper introduces CodeRosetta, an encoder-decoder transformer model for unsupervised translation between programming languages and their high-performance computing (HPC) extensions. The main contributions include unsupervised code translation: CodeRosetta translates between programming languages (e.g., C++) and their parallel programming extensions (e.g., CUDA and Fortran). Customized pre-training and training objective: The model employs Abstract Syntax Tree (AST) Entity Recognition and customized Denoising Auto-Encoding (DAE) to learn parallel programming syntax and nuances.\n\nExperimental results demonstrate that CodeRosetta outperforms state-of-the-art methods in C++ to CUDA translation, with improvements of 2.9 BLEU and 1.72 CodeBLEU points, and a 6.05% increase in compilation accuracy.", "questions": "- In Section 5.2 Ablation Study, Table 3: Ablation Study for C++ to CUDA, it was found that removing the fine-tuning with data from large models resulted in the most significant performance drop. Although the authors stated that this step is optional, the model's performance without this step did not surpass the baseline.\n\n- Section 3.4 Back translation for unsupervised refinement and Section 3.5 Finetuning with synthetic data from language models both mention generating new data using large models. There is a potential risk of data leakage during this process. It is crucial to ensure that the data generated by large models does not overlap with the test data, which could affect the validity of the test results.\n\n- Minor issues.  Line 27,  \"C++ ↔ CUDA or C++ ↔ CUDA\"  should be \"C++ ↔ CUDA or C++ ↔ Fortran\"?", "limitations": "The authors have addressed the limitations of their approach.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces two new learning objectives: Abstract Syntax Tree (AST) Entity Recognition (AER) and customized Denoising Auto-Encoding (DAE) with weighted token dropping and insertion.\n\n2. CodeRosetta can learn both the general syntactic structure of code and the specific nuances of parallel programming constructs without relying on language-specific metrics.\n\n3. CodeRosetta outperforms the current state-of-the-art baseline models.", "weaknesses": "- The evaluation only included BLEU and CodeBLEU, which are syntax-based metrics, and compilation correctness. It did not test the runtime correctness of the code, indicating a lack of completeness in the evaluation metrics. \n\n- In Section 5.2 Ablation Study, the ablation study did not include parts about Masked Language Modeling (MLM) and back translation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Evaluation metrics and runtime correctness**\n\nEvaluation metrics were selected for comparison as they have been utilized in the baselines. We understand that these metrics can have limitations. We tried to address this point by manually analyzing the translated code and executing it. Please refer to the global response for further details on this. Thank you.\n\n**MLM and BT in ablation**\n\nWe have performed an ablation study on the parts that are contributions of our work. The intention was to show the contribution of each proposed training objective. Technically, there is no issue with performing an ablation study on MLM and BT. We make sure to add them to the paper per your request. However, since this requires retraining the model two more times, it would not be possible to finish it within the rebuttal time constraint.\n\n**Fine-tuning Performance**\n\nWe have added fine-tuning as an additional optional step to our pipeline.\nFor translation works, one of the biggest obstacles is the lack of paired data corpora. For example, it is very challenging to find a C++ program and its equivalent CUDA code. However, even if such data exists, it would not be sufficient enough to train a model. Therefore, developing an unsupervised translation model is inevitable. One of the benefits of having such a model is that we can train it using self-supervised and unsupervised training techniques as proposed in the paper and then fine-tune it on a small set of paired data. This is a key ingredient as the model already has knowledge of translation, and with a small set of paired data, it can gain a significant boost in its performance.\n\n**Contamination/Deduplication for synthetic data:**\n\nDuring Back Translation (BT), no new model or training data is involved. In this training task, only CodeRosetta model and the train set are involved. CodeRosetta is asked to translate from A to B. However, since the model has not translated any code in the past, B would be of poor quality with noises. Then B is used as input to the model to reconstruct A. So, the model leverages weak supervision from its own knowledge, and the test set is not involved at all.\nFor synthetic, we take random samples only from the train set and ask a larger model to translate them.\nTo investigate this further, we applied CodeBERTScore to measure the functional similarity of each sample in the test set against the synthetic dataset.\nAnd randomly analyzed some samples that have high CodeBERTScore.\n\nFor example, this is one of the samples from test set\n```\n__global__ void l1_kernel(int n, float *pred, float *truth, float *delta) {\n    int i = blockIdx.x * blockDim.x + threadIdx.x;\n    if (i < n) {\n        float diff = pred[i] - truth[i];\n        delta[i] = (diff > 0) ? 1 : -1;\n    }\n}\n```\nThe most similar CUDA code in the trainset with CodeBERTScore of 0.92 is this:\n\n```\n__global__ void softmax_x_ent_cpu(int n, float *pred, float *truth, float *delta, float *error) {\n    int i = blockIdx.x * blockDim.x + threadIdx.x;\n    if (i < n) {\n        float t = truth[i];\n        float p = pred[i];\n        error[i] = (t) ? -log(p) : 0;\n        delta[i] = t - p;\n    }\n}\n```\n\nAs we can see, the only common thing about these kernels is the fact that both of them are loss functions. However, the first kernel is L1 loss, and the second kernel is cross-entropy loss.\n\nHere, we are showing the amount of data that falls in each range of similarity score. The ranges with zero data are omitted.\n\n| 0.4-0.5 | 0.5-0.6 | 0.6-0.7 | 0.7-0.8 | 0.8-0.9 | 0.9-1.0 |\n|---------|---------|---------|---------|---------|---------|\n| 0%      | 0.8%    | 33%     | 58%     | 7%      | 0.05%   |\n\nA score below 0.8 shows moderate to small similarity. As can be seen, the majority of the files don’t have high similarity with the test data, and even with those ones that had high similarity, we checked manually and realized the reason for the high similarity is that they belong to the same domain with slight similarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3bQ1bSEmej", "reviewer": "Reviewer_BqwA", "summary": "This work intorduces an encoder-decoder transformer model, CodeRosetta, translating between programming languages and also their high-performance computing (HPC) extensions parallelly.\n\nThe authors claim that CodeRosetta outperforms baselines and general LLMs on C++ to CUDA and Fortran translation tasks.", "questions": "I am unfamiliar with the programming language compilers, but AST conversion should vary for C, CUDA, and Fortran. CodeRosetta may be unable to distinguish different languages and much less to generate executable code? Additionally, finetuning with synthetic data from language models can be considered as supervised learning, which may conflict with the paper topic?", "limitations": "The authors addressed limitations in their work.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. In the translation task from C++ to CUDA and Fortran, the paper claims that the CodeRosetta 0.8B model outperforms much larger language models, including GPT-4, Gemini-Ultra, and StarCoder 15B.\n2. This research employed various techniques, such as Abstract Syntax Tree (AST) Entity Recognition, Noise Injection, Token Dropping, and Language-specific Token Insertion, combined with supervised fine-tuning to enhance the model's performance on the code translation task.", "weaknesses": "1. Both BLEU and CodeBLEU might not serve as accurate translation performance indicators [1, 2]. These semantic metrics struggle to capture the nuanced behaviors of code. While Compilation Accuracy appears to be a robust metric, it lacks a formal definition in the paper.\n2. The functional consistency of the original code and the translated code is more important than whether the translated code is compilable. Adding functional correctness evaluation may make the experiment sound more reliable.\n3. For the C++ to CUDA translation task, the paper notes that the model is trained on an unpaired collection of C++ and CUDA source files. This training method can explain how the model learns the knowledge of each individual language. However, a more detailed explanation is needed to clarify the source of the model's language translation capability.\n\n[1] Evtikhiev, M., Bogomolov, E., Sokolov, Y., & Bryksin, T. (2023). Out of the bleu: how should we assess quality of the code generation models?. *Journal of Systems and Software*, *203*, 111741.\n\n[2] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., ... & Zaremba, W. (2021). Evaluating large language models trained on code. *arXiv preprint arXiv:2107.03374*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Metric issue and Formal definition of compilation**\n\nWe understand the metrics used may not be comprehensive enough, as noted in [1]. To address this, we manually analyzed and executed the translated code. Please reference to global response.\nMoreover, as it was mentioned in Out of the BLEU [1] that ChrF is a better fit for evaluating the generated code, we calculated the results based on this metric. The results on ChrF show that CodeRosetta achieves a good score in comparison to other models. Note that we omitted BabelTower from the baseline as neither its model nor its results are available.\n\n| Model      | CodeRosetta | GPT4  | Gemini Ultra | Gemini Pro |\n|------------|-------------|-------|--------------|------------|\n| ChrF(CUDA) | 81.05       | 70.15 | 73.20        | 72.58      |\n\nCompilation Accuracy is the ratio of compilable generated code to the total number of reference codes in the test set.\n\n**Functional correctness**\n\nPlease refer to the global response.\n\n**Details on the model's language translation capability.**\n\nThe bilingualism of our model stems from several pre-training and training objectives that we have utilized and developed. For example, the Mask Language Modeling helps the model to learn cross-lingual representation. For instance, when a `if` keyword, which is a shared keyword between C++ and Fortran, is masked, forces the model to learn the context of an if statement; that is how such a statement looks like in each language. On the other hand, AST entity recognition teaches the model about language-specific entities and cross-language entities to further enable the model to map similar programs in different languages to the same embedding space (encoder model). Then, Denoising Auto Encoding and Back Translation enable the model to leverage the embeddings learned in previous steps to train an encoder-decoder model in order to generate code in the target language.\n\n\n**Fine Tuning with synthetic data**\n\nFinetuning is an optional step. One of the biggest challenges in training a code translation model is the lack of paired data. Therefore, a common practice is to train a model in an unsupervised way. However, even if paired data is available, the amount of data could be insufficient to train a model from scratch. Here, we aimed to evaluate how a model will perform if a small set of paired data exists for fine-tuning. Since the model already has knowledge of code translation, this fine-tuning step can further improve the result of the model and make its performance comparable to a much larger model. CodeRosetta is one of the few unsupervised trained models that support fine-tuning if the data exists. \n\n**Differences in ASTs of different languages**\n\nThat is indeed true. The ASTs differ slightly depending on which language we are dealing with. However, some entities are common between ASTs, such as identifiers. These common entities enable the model to learn how they are utilized in the source code. For example, how identifiers are being used in C++ and CUDA programs. Some other entities are not common among the languages, such as pointers. The absence of such an entity in a programming language forces the model to not predict the type of an entity as a pointer. This, on the other hand, helps the model to realize that pointers are available in a programming language but are not available in another language.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers, 

Thank you very much for your invaluable feedback and comments. We acknowledge that evaluation metrics may not capture all nuances and that systematically evaluating the generated code against references is challenging. However, these metrics have been used widely in the baseline papers. Nonetheless, to address this issue, we manually analyzed the models’ results to reveal how CodeRosetta (0.8B) differs from other proprietary models.

We analyzed 30 randomly selected CUDA kernels by creating unique template programs using reference code and verifying the outputs of each generated CUDA code. The code was executed using NVCC. Our experiments show that CodeRosetta can generate functionally correct code, and we provide insights from our manual execution of translated code. Due to limitations, we only show the code snippet rather than the full kernel.

---
 
**Case 1**

Reference CUDA code: 

`__global__ void fill_kernel(int N, float ALPHA, float *X, int INCX) {int i = (blockIdx.x + blockIdx.y * gridDim.x) * blockDim.x + threadIdx.x;`


CodeRosetta CUDA translation: 
`int i = (blockIdx.x + blockIdx.y * gridDim.x) * blockDim.x + threadIdx.x;`


GPT-4 Translation:
`int i = blockIdx.x * blockDim.x + threadIdx.x;`

Gemini Pro Translation:
`int i = blockIdx.x * blockDim.x + threadIdx.x;`

Gemini Ultra Translation:
`int i = blockIdx.x * blockDim.x + threadIdx.x;`

This kernel is designed to be launched with a grid of thread blocks. Each thread calculates its global index `i`, and if `i` is within the array's bounds (`i < N`), it assigns the value `ALPHA` to the element at index `i * INCX` in the array `X`. The translated code from CodeRosetta correctly identifies the 2D grid structure with `(blockIdx.x + blockIdx.y * gridDim.x) * blockDim.x + threadIdx.x`, while other models use a 1D structure with `blockIdx.x * blockDim.x + threadIdx.x`. The choice of grid structure significantly impacts CUDA performance, and our model successfully finds the optimized result similar to the baseline. Unlike the other models, we observed four other instances where CodeRosetta used the correct grid structure.

---

**Case 2**

Reference CUDA code : 
`__global__ void set_sorting_offset(const int nrows, const int ncols, int *offsets) {int tid = threadIdx.x + blockIdx.x * blockDim.x;`

CodeRosetta Translation: 
`int tid = blockIdx.x * blockDim.x + threadIdx.x;`

Gemini Ultra Translation: 
`int tid = threadIdx.x;`

The purpose of this kernel is to initialize an array of offsets for sorting, where each offset corresponds to the starting position of a column in a flattened 2D grid. This is useful for parallel sorting or column-wise operations. Using `threadIdx.x + blockIdx.x * blockDim.x;` gives each thread a unique index across the entire grid, suitable for accessing unique elements in a global array. In contrast, `threadIdx.x;` only provides a block-unique index, risking data races when accessing global data. The code translated from Gemini Ultra suffers from this issue, causing data races and failing to fulfill the kernel's intention. It behaves in a similar fashion over some other kernels too. Other models have successfully translated the code correctly.

---

**Case 3**

Reference CUDA code: 

`__global__ void opL23 ( float * vec , float * vec1 , long depth , long rows , long cols ) { unsigned long x = threadIdx.x + blockIdx.x * blockDim.x ; unsigned long y = threadIdx.y + blockIdx.y * blockDim.y; unsigned long z`

CodeRosetta translation: 
`unsigned long x = blockIdx.x * blockDim.x + threadIdx.x ;`

GPT-4 Translation:
`int x = blockIdx.x * blockDim.x + threadIdx.x;` 

Gemini Pro Translation:
`int i = threadIdx.x + blockIdx.x * blockDim.x;`

Gemini Ultra Translation:
`int x = blockIdx.x * blockDim.x + threadIdx.x;`

This kernel function processes 3D arrays in parallel. Each thread calculates its 3D position, performs bounds checks, and updates specific elements of the `vec` array based on `vec1`. It averages and scales values from `vec1` and stores the results in `vec`, ensuring safe memory access within array limits. CodeRosetta ensures optimal translation without index overflow by using `unsigned long`, unlike GPT-4 and Gemini Ultra, which fail with large block and grid dimensions due to `int` usage. Gemini Pro absolutely fails in this translation producing erroneous results. Gemini Pro also misses the `const` type qualifier over different kernels, overwriting read-only data.

---

**Case 4**

We also analyze some examples from Fortran to C++ translation in the test set. 
Reference C++ Code:

`#include <stdio.h>
#include <omp.h>
int main() {
    int x = 0, y;
    #pragma omp parallel`

CodeRosetta Translation:
`#include <omp.h>\n\nint main(){#pragma omp parallel`

Gemini Pro: 
`#include <omp.h>\nint main(){#pragma omp parallel`

Gemini Ultra: 
`#include <omp.h>\n\nint main() {#pragma omp parallel`

GPT4 Generated Translation: 

`#include <atomic>
#include <thread>
#include <mutex>
std::mutex mtx;`

The code snippets are similar in functionality, showing synchronization of shared variables between threads. The main differences are the synchronization primitives used. OpenMP uses directives (`#pragma omp critical`, `#pragma omp flush`, `#pragma omp atomic`) for synchronization and memory visibility. C++ uses `std::mutex, std::atomic`, and `std::atomic_thread_fence` to achieve the same objectives from their headers.

Both approaches ensure `x` is correctly updated and visible to the second thread before it prints its value, synchronizing the threads' actions. The Fortran code included OMP, which CodeRosetta, Gemini Pro, and Gemini Ultra identified, but GPT-4 did not, instead using a different method. This highlights the limitations of BLEU like metrics, which focus on syntax rather than functionality. Despite the functional equivalence, GPT-4's code would score lower. This underscores the need for human evaluation to ensure code correctness, as no automated metric or benchmark can fully capture this.

Meta-review: The paper investigates the problem of unsupervised code translation, focusing on parallel programming languages like C++ and CUDA. By introducing novel pre-training objectives like AST Entity Recognition and customized Denoising Auto-Encoding, CodeRosetta effectively learns parallel programming syntax. Experimental results demonstrate that CodeRosetta outperforms state-of-the-art methods in C++ to CUDA translation with relatively small models (0.8B). The reviewers generally appreciate the method's performance and the newly introduced training objectives. Reviewers' concerns were mostly addressed during the rebuttal.
Decision: Accept (poster)