CodeRosetta: Pushing the Boundaries of Unsupervised Code Translation for Parallel Programming

Authors: ali  tehranijamsaz
arijit  bhattacharjee
le  chen
nesreen k ahmed
♢  amir
ali  jannesari
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:59:21

Abstract: Automatic translation of programming languages has garnered renewed interest, driven by recent advancements in large language models (LLMs). Encoder-decoder transformer models, in particular, have shown promise in translating between different programming languages. However, translating between a language and its high-performance computing (HPC) extension remains underexplored due to inherent challenges like complex parallel semantics understanding. In this paper, we introduce CodeRosetta, an encoder-decoder transformer model explicitly designed for translating between programming languages and also their HPC extensions. CodeRosetta is evaluated on C++ to CUDA and Fortran to C++ translation.
It employs a customized learning-based framework with tailored pretraining and training objectives that enable it to effectively capture code semantics and parallel structural nuances, allowing for bidirectional code translation. Our results show that CodeRosetta outperforms state-of-the-art baselines in C++ to CUDA translation by 2.9 BLEU and 1.72 CodeBLUE points while improving compilation accuracy by 6.05%. Compared to general closed-source LLMs, our proposed bidirectional learning-based method improves C++ to CUDA translation by 22.08 BLEU and 14.39 CodeBLUE with 2.75% higher compilation accuracy.
Finally, CodeRosetta exhibits proficiency in Fortran to parallel C++ translation, marking it, to our knowledge, as the first encoder-decoder model for such a complex translation task, improving CodeBLEU at least by 4.63 points compared to closed-source LLMs and Open Code LLM.

Main paper: 
 introduction: 
 Automatic code translation between programming languages offers numerous benefits, such as modernizing legacy systems, enabling cross-platform development, and refactoring sequential code into parallel high-performance versions. However, this task poses significant challenges, primarily due to the scarcity of parallel corpora-paired datasets of the same applications written in different languages (e.g., C++ ↔ CUDA or Fortran ↔ C++). This lack of data limits the effectiveness of supervised learning approaches. While recent advances in code LLMs have shown promise in general code translation, translating code that involves parallel programming paradigms (e.g., C++ to CUDA) remains largely unexplored. That is primarily due to the inherent complexities in capturing and correctly replicating parallel code semantics [28] .TransCoder [36] and its follow-up works [37, 39] have demonstrated the potential of unsupervised learning for code translation. However, these methods often struggle with the complexities of Pre-training plays a crucial role in enabling transformer models to develop a foundational understanding of programming languages. We use Masked Language Modeling (MLM) [47] , a widely adopted pre-training objective, to achieve this, as outlined in Figure 1 . In MLM, the model receives input code with a portion of tokens randomly masked. The objective is to predict the masked tokens based on the surrounding context, thereby encouraging the model to learn both local syntactic patterns and broader semantic relationships within code.To further challenge the model and better reflect code structure, we mask entire words rather than individual tokens. For instance, in the input code snippet " int index", the entire word " index" would be masked, requiring the model to predict the missing identifier based on its type (" int") and its usage in the surrounding code. This approach mirrors how code comprehension often relies on understanding the roles of variables and functions within their scope.Additionally, while MLM is typically applied to monolingual datasets, we extend it to a cross-lingual setting by training on a combined dataset of both C++ and the target language (CUDA or Fortran). This cross-lingual exposure enables CodeRosetta to learn shared programming concepts and syntactic structures across languages, such as control flow statements (if, else, while) and variable declarations. By recognizing these commonalities, the model can transfer knowledge across languages, improving its ability to translate even unseen code patterns. Extracted AST entities 0, 7, 3, 0, 7, 13, 13, 0, 7, 13, 13, 0, 0, 7, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 9, 0, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 9, 0, 0, 0 0:O 1:Var 2:Func ...0, 7, 3, 0, 7, 13, 13, 0, 7, 13, 13, 0, 0, 7, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 9, 0, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 9, 0, 0, 0 Following cross-lingual MLM pre-training, we introduce a new pre-training objective called Abstract Syntax Tree (AST) Entity Recognition (AER) to further improve CodeRosetta's understanding of code structure. This approach draws inspiration from Named Entity Recognition (NER) in natural language processing [20] , where models learn to classify words or phrases into predefined categories (e.g., person, location, or organization). In AER, CodeRosetta learns to recognize and categorize various syntactic components in code.Label InputThe process, illustrated in Figure 2 , starts by using Tree-sitter 2 , a multi-language parsing library, to generate the Abstract Syntax Tree (AST) of a source code snippet. The AST representation provides a hierarchical, tree-structured view of the code, with each node corresponding to constructs such as function definitions, variable declarations, or arithmetic expressions. From this AST, we extract a set of entities and their corresponding categories. Examples of categories used in our implementation include function, variable, constant, pointer, and literal.During AER pre-training, CodeRosetta tokenizes the input code and predicts the syntactic category of each token based on its role in the AST. Tokens that do not correspond to any specific category are labeled as "O" (Outside). This training enables CodeRosetta to develop an understanding of the syntactic relationships between code elements, an essential step in accurately translating and generating code across different languages and extensions.A key strength of AER is its flexibility-the set of entity categories can be easily adapted for different languages or programming paradigms. For instance, when focusing on CUDA code, we can introduce specialized categories for parallel constructs such as threadIdx, blockIdx, and gridDim, enabling CodeRosetta to learn the language-specific semantics of parallel programming.Furthermore, AER is highly adaptable. Even in cases where AST parsing is only partially available, CodeRosetta can still leverage this pre-training, showcasing its applicability to diverse code translation tasks. The complete list of tags used in our implementation is provided in Appendix D.2. While cross-lingual MLM and AST Entity Recognition effectively pre-train CodeRosetta's encoder to generate meaningful representations of source code, the decoder remains untrained at this stage. Consequently, attempting direct code translation would result in suboptimal performance due to the decoder's lack of exposure to the target language's syntax and semantics. To bridge this gap, we employ a Denoising Auto-Encoding (DAE) training strategy specifically tailored for code translation with adaptive noise injection mechanisms. In essence, DAE training involves corrupting the input source code with various types of noise and then training the model to reconstruct the original, noise-free code. This process compels the decoder to learn both the underlying syntactic rules of the target language and the ability to recover meaningful code from perturbed inputs, simulating the challenges of translating real-world code with potential variations and inconsistencies. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The limitations of our study are addressed in Appendix(Section K) . First, our research does not extend CodeRosetta to a broader range of applications. Second, we have not incorporated several potentially beneficial AER tags. Despite these limitations, our methodology has demonstrated significant effectiveness in two critical tasks in the HPC community: translating C++ to Fortran and C++ to CUDA. We plan to address these limitations in future research efforts. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
denoising auto encoding with adaptive noise injection: 
 To initiate the DAE training phase, we first initialize the decoder using the pre-trained encoder's weights, providing it with a starting point for language understanding. Next, we apply a combination of common noise injection techniques, such as random token masking and shuffling, alongside our new noise strategies designed to emphasize the distinctions between programming languages and their extensions. Figure 3 illustrates the overall process of DAE training in CodeRosetta. We now delve into the specifics of our customized noise injection methods, which distinguish CodeRosetta from conventional DAE-based code translation models. These strategies are crucial for enabling the model to discern the subtle but significant differences between languages like C++ and their high-performance counterparts like CUDA.Weighted token dropping. To encourage the model to learn the distinctive features of each language and its extensions, we introduce a weighted token dropping strategy during the noise injection phase. Unlike uniform random token removal, this approach assigns higher removal probabilities to language-specific keywords, encouraging the model to focus on critical syntactic elements.For each programming language or extension, CodeRosetta maintains a list of reserved keywords. During token dropping, these keywords are prioritized, making them more likely to be removed than other tokens. For example, when training on CUDA code, keywords like blockIdx, threadIdx, blockDim, __global__, and atomicSub are more frequently targeted for removal.This weighted sampling creates a more challenging reconstruction task for the model, compelling the decoder to develop a deeper understanding of the language-specific semantics and parallel programming constructs. While the reserved keywords are given higher priority, the weighted random sampling still ensures that other tokens are occasionally dropped, preserving the overall balance of the noise injection process. Language-specific token insertion. In addition to weighted token dropping, we implement a language-specific token insertion strategy to improve CodeRosetta's ability to discern between languages and their extensions during code generation. This method strengthens the model's robustness against out-of-vocabulary tokens, preventing it from inadvertently blending elements from different languages.During DAE training, CodeRosetta must distinguish between valid and invalid tokens within the target language. To facilitate this, we construct a vocabulary of unique tokens for each programming language in our training dataset, tracking their frequency of occurrence. Tokens from the vocabulary of other languages are then randomly inserted into the input code based on their probability from the frequency distribution. For example, in the C++ to CUDA translation task, we insert CUDA-specific tokens into C++ code inputs during DAE training. CodeRosetta is then trained to recognize and disregard these foreign tokens while reconstructing the original C++ code. This process enables the model to develop an understanding of language boundaries, ensuring it generates syntactically and semantically valid code during translation.Adaptive noise ratios Additionally, we introduce an adaptive noise strategy. Instead of applying a fixed noise ratio, such as 10% for token dropping, we begin with an initial noise ratio and progressively increase it throughout the training process. This approach allows the model to gradually adapt to more challenging conditions as it learns to reconstruct the corrupted input sequences. As the training progresses, the input sequences become increasingly corrupted, making the reconstruction task more difficult and forcing the model to learn more robust representations.There is a maximum corruption rate that, once reached, halts further increases in noise. This prevents over-corrupting the inputs, ensuring that the model can still derive meaningful patterns. The impact of adaptive noise ratios, along with the new noise strategies, is examined in our ablation study (Section 5.3).To further support accurate code generation in the target language, we prepend a special <LANG> token to each input sequence. During DAE, this token indicates the language of the corrupted input, prompting the decoder to reconstruct the code in the same language. This mechanism ensures that the model remains focused on generating code within the correct language context. To further improve CodeRosetta's translation quality and its ability to capture complex code semantics, we employ back translation during the training process [36] . As illustrated in Figure 4 , this technique leverages the model's bidirectional capability, enabling both source-to-target and target-to-source translations, forming a weakly supervised learning loop.In back translation, the model is trained on a source-to-target task (e.g., C++ to CUDA) while simultaneously performing the reverse translation (target-to-source, CUDA to C++). For each batch of source code, CodeRosetta first translates it into the target language. The generated target code is then used as input for a reverse translation, where the model attempts to reconstruct the original source code.This forward and backward translation cycle provides continuous feedback, allowing CodeRosetta to compare the reconstructed source code with the original input, thereby learning to detect and correct errors in both translation directions. Through this iterative refinement, the model gradually improves its comprehension of nuanced language differences and complex code structures, resulting in more accurate and semantically consistent translations.Crucially, we alternate between batches of different language pairs during back translation. This ensures that the model receives balanced exposure to both directions, preventing bias towards a specific language and encouraging the development of robust, generalized translation capabilities.While CodeRosetta demonstrates promising results through unsupervised training, we explore the potential of further enhancements by leveraging the capabilities of large language models (LLMs) such as GPT-4 [1] and Gemini Ultra [41] . These LLMs, trained on extensive text and code datasets, have exhibited impressive code generation abilities. However, directly employing such large models for code translation can be computationally expensive and impractical for many real-world applications.To address this, we adopt a knowledge distillation approach [18] , where these LLMs serve as teacher models to generate synthetic data for fine-tuning CodeRosetta, a smaller student model. This method allows us to capture the expertise of the larger models while maintaining computational efficiency.Specifically, we prompt GPT-4 and Gemini to translate C++ code into CUDA where feasible. After filtering out empty or invalid translations, natural text, and non-relevant data (i.e., instances lacking CUDA-specific keywords), we are left with approximately 5,000 high-quality translations from an initial set of 100,000. This significant reduction highlights the inherent challenges in C++ to CUDA translation.The resulting synthetic dataset of C++↔CUDA pairs is then used to fine-tune CodeRosetta. This process allows CodeRosetta to incorporate the valuable knowledge embedded in the larger LLMs without incurring their high computational costs. It is important to note that this fine-tuning step is optional and can be omitted if access to powerful LLMs for synthetic data generation is not feasible. 
experimental setup: 
 Training hyperparameters. We implement CodeRosetta using the HuggingFace Transformers library v4.40.1 [47] . The model is a 12-layer encoder-decoder transformer, with each layer having 12 attention heads and a hidden dimension of 1,536. We initialized the tokenizer with a pre-trained Byte Pair Encoding (BPE) tokenizer from UniXcoder [17] , which was further trained on our specific training datasets. The training was conducted using the AdamW optimizer [24] and a batch size of 16, using gradient accumulation over two steps. The experiments were run on a single node with four Nvidia A100 SXM4 GPUs, each with 80GB of memory. To speed up the training process, mixed-precision training was enabled. The final model consists of ∼0.8 billion parameters.We evaluate CodeRosetta on two code translation tasks: C++ to CUDA and Fortran to C++. Table 8 provides an overview of the datasets used. For the C++ to CUDA translation task, we use the dataset from BabelTower [46] , which consists of:• Unpaired training set: A collection of 243,008 C++ and CUDA source code files, meeaning there is no direct correspondance between the files in each language. To avoid any language bias, we ensure an equal number of C++ and CUDA files during training.• Paired validation and test sets: The validation set consists of 184 pairs, and the test set has 180 pairs of C++ and CUDA source code files. Each pair represents the same program implemented in both languages, providing a benchmark for evaluating translation accuracy.For Fortran to C++, no dedicated parallel corpus exists for this specific translation. Thus, we construct our training dataset as follows:• Unpaired training set: We extract the C++ and Fortran subsets from the Stack V2 dataset [25] , which includes over 3 billion source code files across more than 600 programming languages. We ensure an equal number of files from each language to prevent bias during training. • Paired fine-tuning set: For fine-tuning, we use the small paired C++-Fortran dataset introduced by Bin et al. [19] . This set is also used for validation. • Test set: To evaluate the final model performance, we use a test set of 33 paired C++ and Fortran programs.To ensure the quality and consistency of training data, we applied task-specific preprocessing steps for each translation task. C++ to CUDA. Although the BabelTower dataset [46] was reportedly cleaned, we found noisy data within the CUDA files. To address this, we curated a list of CUDA-specific reserved keywords and filtered the dataset, retaining only those CUDA files that contained at least one such keyword. This step significantly reduced noise and resulted in a final training set of 243,008 C++ files, matched by an equal number of CUDA files. The validation and test sets remained unchanged, comprising 184 and 180 paired examples, respectively. C++ to Fortran. Preprocessing the Stack V2 dataset for C++ to Fortran translation involved managing the large imbalance between C++ and Fortran files, as well as filtering out low-quality or uninformative code snippets. We implemented the following steps:• Educational value filtering: Inspired by the phi-1 model data filtering approach [16] , We randomly sampled 100,000 C++ files from Stack V2 and employed GPT-3.5 to assess their "educational value" for learning C++ coding concepts. We prompted GPT-3.5 (see Figure 5 to classify each snippet as either "Yes" or "No" based on its educational value. These labels were then used to fine-tune a binary classifier built on the CodeSage model [49] , which we applied to the remaining C++ files in Stack V2. Only files deemed educationally valuable were retained. • Balancing language representation: From the filtered C++ files, we randomly selected a subset equal in size to the number of Fortran files to create a balanced training set. • Length-based filtering: To ensure training stability and avoid biases toward very short or long code snippets, we filtered out files containing fewer than ten tokens or more than 1,000 tokens in both languages.After these steps, the final training set for C++ to Fortran translation consisted of 474,856 files. For fine-tuning and validation, we used the small paired C++-Fortran dataset from Bin et al. [19] , which contains 282 samples. The model was then evaluated on a test set of 33 paired samples.Determine the educational value of the following code for a student whose goal is to learn C++ coding concepts. If it has educational value, return only "Yes", else, return "No". To evaluate CodeRosetta's translations, we use two widely used code translation metrics: BLEU [32] and CodeBLEU [34] . We benchmark CodeRosetta against the following baselines. For C++ to CUDA, we compare (a) "BabelTower [46] ",3 a state-of-the-art unsupervised code translation model specifically designed for C++ to CUDA translation, and (b) "Transcoder [36] ", a general unsupervised code translation model that has demonstrated strong performance on various language pairs. Since a single evaluation metric may capture only one aspect of translation quality [14] , we supplement BLEU and CodeBLEU with ROUGE-L [22] and ChrF [33] , as recommended by [14] . However, because generated translations from TransCoder and BabelTower were unavailable, ROUGE-L and ChrF scores are only provided for GPT-4, Gemini-Ultra, and Gemini-Pro. We further compare CodeRosetta with two popular open-source code LLMs: StarCoder (starcoder2-7b) [21] and DeepSeekCoder (DeepSeek-Coder-V2-Lite-Base) [12] .For the Fortran to C++ task, we evaluate CodeRosetta against StarCoder [21] , an LLM model (15.5B parameters) featuring a decoder-only transformer architecture, fine-tuned on a comprehensive corpus of Fortran code and DeepSeekCoder (DeepSeek-Coder-V2-Lite-Base) [12] . Additionally, we evaluate CodeRosetta alongside several prominent closed-source LLMs, including GPT-4 [1] and Gemini [41] , by prompting them to perform code translation using carefully crafted prompts (Appendix I). By evaluating against a broad spectrum of both specialized code translation models and general-purpose LLMs, we effectively gauge CodeRosetta's stranghts and limitations across diverse translation tasks and programming paradigms.  
post-processing: compilation error analysis: 
 Our test set, consisting of 180 samples, provided diverse input scenarios to evaluate our model's performance. We observed that 23 samples generated compilation errors when processed through the NVCC compiler with the required flags. 4 Upon manual investigation, we found that most errors were trivial and could be easily fixed with minor edits. Specifically, 48% of the errors were attributed to the use of an undefined generic type T. Another 9% resulted from missing closing braces, while 26% were due to a single missing variable initialization. Additionally, 9% of the errors were caused by incorrect function calls. Only 8% of the files contained no trivial errors. By applying quick fixes for the undefined generic type T, missing variable initializations, and missing closing braces, the overall compilation accuracy significantly improved, with 98.85% of all generated code becoming compilable. This indicates that most errors were simple and could be easily resolved by incorporating compiler feedback, which will be a focus of our future work. Subsection F.1 and Figure 13 in the Appendix presents examples of our findings.Although CodeRosetta demonstrates more accurate translations based on the aforementioned metrics compared to the reference code, these metrics are derived from static evaluations, leaving runtime performance uncertain. To address this, we randomly selected 30 translated CUDA kernels from the test set and created unique template programs to execute them. We ran the translated CUDA kernels using NVCC and found that the functional correctness of the generated code was preserved in the majority of samples (approximately 93%). For further details, see Appendix Section B.We conduct an ablation study to evaluate the impact of each training objective on the code translation results of CodeRosetta. Specifically, we remove individual training objectives (e.g., AER) while keeping the other components intact and retraining the model. Fine-tuning Data extraction from larger models is a common practice. In this phase of the ablation study, we evaluate CodeRosetta's performance without fine-tuning it on the synthetic dataset. From Table 3 , we observe that the removal of each proposed learning objective negatively impacts the model's ability to deliver improved code translation.We train and apply CodeRosetta for translation between C++ and Fortran. Fortran has had a long-standing presence in the scientific computing community; however, its integration with modern HPC systems [38] can pose significant challenges for developers. Due to the complexities involved in translating Fortran to C++, there has been limited effort to address this issue. Bin et al. [19] were the first to make significant strides in this area, curating a small paired dataset specifically for this translation task and fine-tuning several open-code LLMs.They found StarCoder (15B), when fine-tuned, benefited the most from their paired dataset. We compare CodeRosetta with the fine-tuned StarCoder (15B), as well as with other general LLMs.The results are shown in Table 4 . Fine-tuning CodeRosetta on the dataset from Bin et al. [19] further enhances its performance, achieving a CodeBLEU score of 65.93. Notably, CodeRosetta outperforms StarCoder, even though StarCoder is nearly 20 times larger, highlighting the efficiency of our model. It also surpasses state-of-the-art models like GPT-4 and Gemini by a substantial margin, achieving an improvement of at least 4.63 points in CodeBLEU.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We submitted the code as supplemental material and also created an anonymous repository ( https://anonymous.4open.science/r/neurips_coderosetta-CAB2/) to include both data and code. Please refer to the README file in the repository for details. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion: 
 In this paper, we introduced CodeRosetta, an encoder-decoder transformer model designed for translating between programming languages and their high-performance computing (HPC) extensions. We proposed two novel learning objectives: Abstract Syntax Tree (AST) Entity Recognition (AER) and customized Denoising Auto-Encoding, which incorporates weighted token dropping and insertion. These contributions enable CodeRosetta to capture both the general syntactic structure of code and the specific nuances of parallel programming constructs, without relying on language-specific metrics.Our experiments show that CodeRosetta significantly outperforms state-of-the-art baselines on C++ to CUDA translation, achieving improvements up to 2.9 BLEU, 1.72 in CodeBLEU, and 6.05% in compilation accuracy. Furthermore, CodeRosetta is, to the best of our knowledge, the first model to demonstrate proficiency in translating Fortran to its parallel counterpart in C++, highlighting its potential in handling diverse programming paradigms. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The creators of the data and models used in this work have been properly credited. The license and terms of use are explicitly mentioned and properly respected. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: Our work does not release new assets. We use existing, publicly available resources for our analysis.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects. 
 
 Appendix: The metrics and results shown in Table 1 may have limitations in capturing functional equivalence, as discussed by Evtikhiev et al. [14] . To address this, we evaluated the functional correctness of the translated code by compiling and executing the generated programs. For the C++ → CUDA translation task, we randomly selected 30 generated CUDA kernels and developed a template program for their execution. We then compared the runtime results of the translated CUDA code against the reference implementations. Our findings indicate that 93% of the translated CUDA code produced results consistent with the reference.We analyzed three representative cases of CUDA translation in detail. In the first case, shown in Figure 6 , the kernel is designed to be launched with a grid of thread blocks. Each thread calculates its global index i, and if i is within the array's bounds (i < N), it assigns the value ALPHA to the element at index i * INCX in the array X. CodeRosetta successfully identified the optimal 2D grid structure with (blockIdx.x + blockIdx.y * gridDim.x) * blockDim.x + threadIdx.x, whereas other models defaulted to a less efficient 1D structure using blockIdx.x * blockDim.x + threadIdx.x. This choice of grid structure significantly impacts CUDA performance, and CodeRosetta's selection mirrors that of the baseline implementation. Furthermore, CodeRosetta employed the correct grid structure in four additional instances where other models did not.The second case, illustrated in Figure 7 , involves a kernel designed to initialize an array of offsets for sorting purposes. Each offset corresponds to the starting position of a column in a flattened 2D grid. This is often useful for parallel sorting algorithms or other operations requiring column-wise processing. The expression int tid = threadIdx.x + blockIdx.x * blockDim.x; assigns each thread a unique index across the entire grid of blocks, enabling access to distinct elements in a global array. In contrast, the expression int tid = threadIdx.x; provides an index that is only unique within a single block. Without proper offset calculations, threads across different blocks could access the same data, potentially leading to race conditions and negating the kernel's intended behavior. This issue was observed in several examples where Gemini-Ultra produced incorrect results due to this oversight.The third case, depicted in Figure 8 , processes 3D arrays in parallel. Each thread calculates its 3D position, checks bounds, and updates specific elements of the array vec based on values from vec1. The kernel averages and scales values from vec1, storing the results in vec while ensuring safe memory access within the array's limits. CodeRosetta correctly handled large block and grid dimensions by using unsigned long, whereas both GPT-4 and Gemini-Ultra failed due to the use of int, leading to index overflow.We also analyzed Fortran to C++ translations, shown in Figure 9 . The translated code snippets maintained functional equivalence, specifically in the synchronization of shared variables between threads. OpenMP, used in the Fortran code, relies on directives such as #pragma omp critical, #pragma omp flush, and #pragma omp atomic to ensure synchronization and memory visibility.In the C++ translation, equivalent functionality was achieved using std::mutex, std::atomic, and std::atomic_thread_fence. Both approaches ensure that x is updated and visible to the second thread before it prints its value, effectively synchronizing the thread actions. CodeRosetta, Gemini-Pro, and Gemini-Ultra correctly recognized the use of OpenMP in the original code, while GPT-4 did not and opted for a different approach. This highlights the limitations of metrics such as BLEU, which focus on syntax rather than functionality. Despite functional equivalence, GPT-4's translation would score lower due to its syntactic divergence. This underscores the necessity of human evaluation to ensure code correctness, as no single automated metric can fully capture functional behavior.The C++ to CUDA dataset was obtained from BabelTower [46] , which has gone through deduplication and cleaning. Notably, there is no paired trained data available within the dataset, meaning the model does not encounter C++ code alongside the CUDA equivalent during training. As such, the model must rely solely on self-supervised training objectives to learn to embed source code from different model by transferring the encoder's weights to initialize the decoder, so the decoder begins with some foundational knowledge.For Denoising Auto-Encoding and Back Translation, we use a learning rate of 5 × 10 -5 and train for 20 epochs. For Denoising Auto-Encoding, we set the masking to 15%, token dropping to 25%, and token insertion to 15%, with a denoising ratio increasing by 2.5% per epoch. Finally, for fine-tuning, we use a learning rate of 5 × 10 -5 for ten epochs. At each training iteration, we save the model with the lowest validation loss. All the parameter values are determined empirically through detailed hyperparameter tuning. The AER tags used in pretraining are shown in Table 7 .A detailed overview of the dataset is shown in Table 8 .We conducted beam search decoding with varying beam sizes, returning the top candidate in each case. The results, shown in Table 9 , indicate that CodeRosetta consistently produces the same output, regardless of the beam size.C++ → CUDA: In this part, we compare the code generated by CodeRosetta, GPT4, and Gemini-Ultra. As the BabelTower model and its code are not publicly available, we were unable to access them. However, the BabelTower paper highlights a kernel where the model failed to generate CUDA code due to a syntax error when defining keyCharPtr, as shown in Figure 10 . In contrast, CodeRosetta successfully generates the correct CUDA code. It is interesting to note that CodeRosetta also recognized the if condition and improved the readability of the code by inverting the if statement, similar to the approach taken by Gemini-Ultra and GPT4. Additionally, CodeRosetta adheres to the preferred practice of declaring a variable or pointer before assigning a value, which is why first keyCharPtr is defined out of the if statement.We demonstrate another example in Figure 11 , where CodeRosetta accurately reproduces the reference CUDA kernel without adding unnecessary lines of code, such as a host or main function, which is often seen in other models. FortranCode translated by large language models like GPT-4 often includes additional caller functions that extend beyond the scope of the original function. In contrast, code translated by CodeRosetta may occasionally fail to compile despite being syntactically correct. We identified two common issues in the code generated by CodeRosetta and applied a simple post-processing method to ensure a fair comparison across models.The first issue involves the use of generic types, which can enhance code efficiency but require explicit type definitions at compile time. Figure 13a shows the use of a generic type, although the necessary definition is missing. Adding the type definition, as shown in Figure 13b , resolves the compilation issue. The second issue relates to misses variable initialization in the function definition, as shown in Figure 13c . By initializing the required variable, as demonstrated in Figure 13d , the compilation problem is resolved. Lastly, for longer code snippets, CodeRosetta occasionally omits the closing curly bracket.In the context of code translation, paired data is scarce. However, our model benefits from a strong foundational understanding of code translation acquired through unsupervised and self-supervised pre-training on 243K training examples for C++ ↔ CUDA. We demonstrate that fine-tuning, even with a small amount of synthetic data-without verifying the one-to-one mapping between the generated samples and the input code in a supervised manner-can further improve the model's performance. Specifically, fine-tuning with merely 5K paired samples (less than 2% of total data) generated by larger models still led to significant performance gains. While synthetic data may introduce some errors (as large models can make translation mistakes), the combination of this foundational pre-training and fine-tuning with a small synthetic dataset yields further improvements.Back Translation (BT) has been extensively used in unsupervised translation tasks for both natural language and code. We integrate this technique with the denoising auto-encoding (DAE) objective, ensuring that the model is not trained exclusively on a single objective. During training, the model alternates between DAE and BT for each batch of data. This prevents the model from relying solely on BT and 'cheating' by outputting the input source code as an intermediate translation. To better understand this behavior, we analyzed the intermediate outputs during back translation.For instance, Figure 14 shows a C++ input and its corresponding intermediate CUDA translation. As shown, while the model attempts to translate the code to CUDA, the output contains errors, such as the undefined variable j. In the back translation process, this noisy CUDA code output is fed back into the model, which then attempts to reconstruct the original C++ input. Since the model alternates between languages during back translation, it occasionally generates noisy CUDA or C++ code. This approach improves the model's robustness when handling noisy inputs in translation tasks.During translation between programming languages (e.g., from C++ to CUDA), certain entities, libraries, and syntaxes present in the source language may not be valid or supported in the target language. For example, C++ Standard Template Libraries (STL) such as std::unique_ptr are not compatible with CUDA's device code and must be excluded from translations. The pre-training process in CodeRosetta equips the model with semantic knowledge of both source and target languages, reducing the frequency of invalid tokens during translation. Nonetheless, there are still instances where the model may fail to correctly map common source language entities to valid target language counterparts.While our test set contained no occurrences of std::unique_ptr, we deliberately included this construct in a separate C++ code example to evaluate CodeRosetta's handling of STL-specific constructs. Figure 16 demonstrates this case, where the model successfully generates CUDA code by omitting the unsupported std::unique_ptr in the device kernel. Instead, the use of std::unique_ptr is correctly retained in the host kernel, specifically in the main function, which runs on the CPU. Since CodeRosetta is trained to focus on device function generation, the translation is accurate in this instance.On the other hand, Figure 17 illustrates a case of incorrect translation, where CodeRosetta, along with other large closed-source models like GPT-4, Gemini-Ultra, and Gemini-Pro, failed to generate valid CUDA code. The translated code includes the line *rho = 0;, which initializes the rho variable to zero. In a multi-threaded GPU environment, executing this kernel across multiple threads and blocks simultaneously can lead to a race condition, as multiple threads would attempt to write to the same memory location concurrently. Without synchronization mechanisms like atomic operations or reduction techniques, this results in unpredictable and incorrect behavior. The correct approach would be to initialize rho in the host code and use atomicAdd to accumulate values in the device code safely.In this section, we describe the prompt template used to translate between different programming languages and libraries. The template, shown in Figure 15 , served as the basis for all translation tasks, with language-specific adjustments made by updating the source and target languages as required. For this study, we use OpenAI API's GPT-4 API, using a fixed temperature of zero to ensure deterministic outputs across all models, including CodeRosetta. All queries were executed on May 18th, 2024, ensuring consistency in results throughout the experiments.Automatic parallelization. Early efforts in auto-parallelization were primarily focused on identifying independent loops that could be executed in parallel. Renowned compilers like the Portland Group (PGI) and Intel's C++ Compiler (ICC) have embedded auto-parallelization capabilities, offering pragma-based hints to guide the parallelization process. These compilers analyze loop dependencies, data flow, and potential side effects to generate parallel code, often targeting OpenMP or MPI for multi-threading and distributed computing, respectively. The advent of Polyhedral model-based tools marked a significant advancement in auto-parallelization techniques. The Polyhedral model [6] offers a powerful algebraic representation for optimizing loop nests with affine bounds and access patterns. Pluto [8] is an auto-parallelization tool that utilizes the Polyhedral model to perform loop transformations, tiling, and fusion for effective parallel execution while considering data locality optimization. PPCG (Polyhedral Parallel Code Generation) [44] is another tool that exploits the polyhedral model to automatically optimize and generate parallel code from high-level abstractions, targeting multicore CPUs and GPUs.Neural machine translation. TransCoder-ST [37] extends the original work [36] by adding automated unit testing. TransCoder-IR [39] extends it even further by exploiting LLVM IR for program translation. HPC-GPT [13] uses GPT4 to create an instruction-answer dataset for two tasks (AI models and datasets for HPC and data race detection), then Llama model [43] is supervised tuned on this dataset. Pan et al. [31] provided one of the first studies on the types of errors that are often produced in code translation.There is a growing number of large language models (LLMs) for code generation [5, 35, 40, 50, 26, 4, 29, 3] . Most of these works focus mainly on natural language to code generation. Although these Code LLMs can generate code in various programming languages, Python, in particular, has received more attention compared to others. This could be due to the number of available benchmarks that assess Python coding capabilities [10, 23] , though other programming languages have been gaining more attention recently as well [9] . Despite the growing number of Code LLMs, these models are typically not specifically trained for code translation, even though they can perform code translation to some extent, as shown by Pan et al. [31] .While CodeRosetta demonstrates promising results in code translation, several avenues for future work exist. Currently, CodeRosetta targets C++ → CUDA and Fortran → C++ translation. Extending its capabilities to encompass a wider range of HPC languages and parallel programming models would further broaden its scope. In addition, we plan to improve the set of entity categories used in AER to capture a better representation of code semantics. This will involve incorporating additional tags for constructs like data types, control flow mechanisms, and parallel programming-specific primitives.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer:[Yes] Justification: We provide a bidirectional encoder-decoder transformer model that is capable of learning to translate programming languages as well as their HPC extensions. Results indicated Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.Code: https://coderosetta.com 38th Conference on Neural Information Processing Systems (NeurIPS2024).https://tree-sitter.github.ioWe contacted the authors of BabelTower for access to their trained model, source code, and translations but were not able to gain access. Therefore, we cite results directly from their paper.https://developer.nvidia.com/cuda-11-8-0-download-archivehttps://researchit.las.iastate.edu/ 


Decision: The paper investigates the problem of unsupervised code translation, focusing on parallel programming languages like C++ and CUDA. By introducing novel pre-training objectives like AST Entity Recognition and customized Denoising Auto-Encoding, CodeRosetta effectively learns parallel programming syntax. Experimental results demonstrate that CodeRosetta outperforms state-of-the-art methods in C++ to CUDA translation with relatively small models (0.8B). The reviewers generally appreciate the method's performance and the newly introduced training objectives. Reviewers' concerns were mostly addressed during the rebuttal.

Reviews: review: paper_summary: paper_summary: The paper looks into the problem of unsupervised code translation with a particular focus on parallel programming languages like C++ to CUDA. It trains (relatively) small encoder-decoder models compared to current LLMs while achieving very strong performance on translation tasks. It incorporates program-specific terms in loss such as token frequencies, and AST structure to achieve this.
strength_weakness: strength_weakness: 1. Evaluation and metrics. Perhaps a challenging aspect of the results is the evaluation benchmarks and datasets used. 
  a. Benchmarks. The authors achieve considerably strong results on the C++ to CUDA evaluation sets with BLEU and CodeBLEU numbers higher than 75. Do authors perform some analysis on contamination or deduplication on the benchmark and collected training sets?
  b. Evaluation metric. Functional correctness has emerged to be the winning metric in the code generation space. However, it seems more challenging to apply that to the translation problems. High BLEU might not necessitate better translations. Similarly, comparability as a metric does not ensure that the translation is correct. While it might be hard to design functional correctness-based evaluation, the authors do not perform any other evaluation to assess the quality of translation beyond the, somewhat unreliable automated metrics.

2. Performance on proprietary LLMs. GPT-4 and Gemini-Pro depict considerably worse BLEU for some language pairs. It would be useful to highlight the failure modes since it might also highlight some issues in BLEU-like metrics.

3. Missing details on the datasets. The authors can provide more details on the datasets and benchmarks used in terms of document length and document counts. Similar statistics for the benchmarks and model generations would be useful to understand the complexity of the task.
questions: questions: 1. The AST structure loss seems interesting. Do the authors believe it would also improve embeddings for monolingual encoder models?

2. Backtranslation-based unsupervised learning. The authors use back-translation to improve the translation performance of the model. However, for given languages pairs A and B (going from A -> B -> A), if the intermediate sample B is not constrained, the model can cheat and not do the translation job currently (as an extreme example, just copy the tokens from A). Can the authors clarify how the setup avoids such failure models?

3. What tokenizers do the authors use for these languages which are usually less well represented in the web data?
limitations: limitations: Authors have limitations in the the appendix.
paper_summary: paper_summary: The paper presents CodeRosetta, a transformer model designed for unsupervised code translation and their high-performance computing extensions, such as C++ to CUDA and Fortran. By introducing novel pre-training objectives like AST Entity Recognition and customized Denoising Auto-Encoding, CodeRosetta effectively learns parallel programming syntax. The model demonstrates superior performance, surpassing state-of-the-art baselines metrics like BLEU / CodeBLEU and compilation accuracy.
strength_weakness: strength_weakness: **Weaknesses**

1. Although novel pre-training objectives have been introduced, the authors have not methodologically distinguished these from predecessors using AST or Denoising objects. Training with code structures is prevalent in code learning, as seen in models like CodeT5[1] and PLBART[2]. The authors should discuss these previous methods and variants of structural modeling, as referenced in papers like [3,4].
2. The experimental section lacks some rigor. For instance, in the Educational value filtering, "randomly sampled 100,000 C++ files from Stack V2 and employed GPT-3.5 to assess their ‘educational value’ for learning C++ coding concepts" poses challenges to reproducibility. The use of GPT-3.5 to evaluate C++ coding concepts and the prompt engineering methodology warrant further clarification. Additionally, some experimental details in Table 4 are not clearly presented.
3. Upon reviewing the data composition of models like StarCoder (StarCoderDataV1), which includes 58,355 CUDA and 165,446 Fortran files, it appears that existing models like StarCoder or DeepSeekCoder could be directly employed for research. Significant engineering effort in the paper was focused on vocabulary issues, such as inserting CUDA-specific tokens, yet there is no discussion on how existing code LLMs handle out-of-vocabulary problems with languages like CUDA.

[1] CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation

[2] Unified Pre-training for Program Understanding and Generation

[3] A Survey of Neural Code Intelligence: Paradigms, Advances and Beyond

[4] A Survey on Pretrained Language Models for Neural Code Intelligence
questions: questions: 1. Is there an error in the left part of Figure 4?
2. The phrase "compare CodeRosetta against their best fine-tuned model, StarCoder (15B)" is unclear. While StarCoder's pre-training data does include Fortran code, the notion of "best fine-tuned" is not well explained. Could the authors clarify this?
3. The performance of the three closed-source models in Table 4 is surprisingly low. Could the authors provide more details on the testing procedures?
limitations: limitations: I do not identify potential negative societal impacts in this paper.
paper_summary: paper_summary: This paper introduces CodeRosetta, an encoder-decoder transformer model for unsupervised translation between programming languages and their high-performance computing (HPC) extensions. The main contributions include unsupervised code translation: CodeRosetta translates between programming languages (e.g., C++) and their parallel programming extensions (e.g., CUDA and Fortran). Customized pre-training and training objective: The model employs Abstract Syntax Tree (AST) Entity Recognition and customized Denoising Auto-Encoding (DAE) to learn parallel programming syntax and nuances.

Experimental results demonstrate that CodeRosetta outperforms state-of-the-art methods in C++ to CUDA translation, with improvements of 2.9 BLEU and 1.72 CodeBLEU points, and a 6.05% increase in compilation accuracy.
strength_weakness: strength_weakness: - The evaluation only included BLEU and CodeBLEU, which are syntax-based metrics, and compilation correctness. It did not test the runtime correctness of the code, indicating a lack of completeness in the evaluation metrics. 

- In Section 5.2 Ablation Study, the ablation study did not include parts about Masked Language Modeling (MLM) and back translation.
questions: questions: - In Section 5.2 Ablation Study, Table 3: Ablation Study for C++ to CUDA, it was found that removing the fine-tuning with data from large models resulted in the most significant performance drop. Although the authors stated that this step is optional, the model's performance without this step did not surpass the baseline.

- Section 3.4 Back translation for unsupervised refinement and Section 3.5 Finetuning with synthetic data from language models both mention generating new data using large models. There is a potential risk of data leakage during this process. It is crucial to ensure that the data generated by large models does not overlap with the test data, which could affect the validity of the test results.

- Minor issues.  Line 27,  "C++ ↔ CUDA or C++ ↔ CUDA"  should be "C++ ↔ CUDA or C++ ↔ Fortran"?
limitations: limitations: The authors have addressed the limitations of their approach.
paper_summary: paper_summary: This work intorduces an encoder-decoder transformer model, CodeRosetta, translating between programming languages and also their high-performance computing (HPC) extensions parallelly.

The authors claim that CodeRosetta outperforms baselines and general LLMs on C++ to CUDA and Fortran translation tasks.
strength_weakness: strength_weakness: 1. Both BLEU and CodeBLEU might not serve as accurate translation performance indicators [1, 2]. These semantic metrics struggle to capture the nuanced behaviors of code. While Compilation Accuracy appears to be a robust metric, it lacks a formal definition in the paper.
2. The functional consistency of the original code and the translated code is more important than whether the translated code is compilable. Adding functional correctness evaluation may make the experiment sound more reliable.
3. For the C++ to CUDA translation task, the paper notes that the model is trained on an unpaired collection of C++ and CUDA source files. This training method can explain how the model learns the knowledge of each individual language. However, a more detailed explanation is needed to clarify the source of the model's language translation capability.

[1] Evtikhiev, M., Bogomolov, E., Sokolov, Y., & Bryksin, T. (2023). Out of the bleu: how should we assess quality of the code generation models?. *Journal of Systems and Software*, *203*, 111741.

[2] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., ... & Zaremba, W. (2021). Evaluating large language models trained on code. *arXiv preprint arXiv:2107.03374*.
questions: questions: I am unfamiliar with the programming language compilers, but AST conversion should vary for C, CUDA, and Fortran. CodeRosetta may be unable to distinguish different languages and much less to generate executable code? Additionally, finetuning with synthetic data from language models can be considered as supervised learning, which may conflict with the paper topic?
limitations: limitations: The authors addressed limitations in their work.

Comments: title: Thank you for your time and recognition of our efforts!
Response from reviewer
Thank you for the support!
Thanks for the response
Thank you for your time, thoughtful engagement, and recognition of our efforts!
Thank you for your effort
Thank you! --- Part 2
Thank you! --- Part 1
Response to Author Rebuttal
comment: Dear Reviewer djHa,

Thank you for your thoughtful engagement with our work and for taking time to carefully read our rebuttal. We are glad to hear that our explanations and additional experiments have largely addressed your concerns, and we sincerely appreciate your decision to raise your evaluation.

We apologize for the misunderstanding your point regarding Figure 4. Upon review, we identified a typo in the figure, and we have since corrected it to align with the text.

We are committed to incorporating these additional results and discussions in the final version of the paper, ensuring that they accurately reflect our discussions.

Thank you once again for your support and valuable feedback.


Best regards,

The Authors
Thank you for your rebuttal. My question was about Figure 4, not Table 4. My concerns have been largely addressed, and I have raised my evaluation.
Dear Reviewer 9e24,

Thank you for your thoughtful review and for recognizing the significance of our contribution. We appreciate your support and the time you took to engage with our detailed analysis. Your feedback has been instrumental in improving our work.

We will incorporate the discussion on BLEU and the potential noise in evaluations into the final version of the manuscript, as you suggested.

Thank you once again for your encouraging feedback and supporting our work.

Best Regards,

The Authors
Thank you for performing the detailed analysis and not shying away from the limitations of BLEU. I would encourage the authors to add this discussion to the manuscript. Such incidents can add a fair amount of noise to the evaluations. However, the improvements achieved by the work are considerable and I remain cautiously optimistic for the paper.
Dear Reviewer BqwA,

Thank you for your thoughtful engagement and for taking time to carefully review our response. It means a lot to us. We appreciate your recognition of our efforts and are delighted that our explanations and additional experiments have addressed your concerns.

We're especially grateful for your decision to increase the score and advocate for the acceptance of our paper. Your feedback has improved the quality of our work, and we believe this will benefit the broader research community.

We are committed to revising the manuscript for the final version to carefully reflect our discussions.

Thank you once again for your support.

Best Regards,

The Authors
Dear authors,

Thank you for your diligent effort and thorough response!

The experiments and explanations you provided have satisfactorily addressed all of my concerns. As a result, I am pleased to elevate my score from 3 to 6 and advocate for the paper's acceptance. 

Good luck!
## Model’s Translation Capability

Thanks for this great question. As you mentioned, there are entities, libraries, keywords, and syntaxes from the source language (e.g., C++) that may not be valid in the target language (e.g., CUDA). `std::unique_ptr` and other C++ Standard Template Libraries (STL) belong to this category and must be avoided in the translation. The pre-training process in CodeRosetta (esp. Weighted Token Dropping) equips the model to gain a semantic understanding of source and target languages. It reduces the occurrences of invalid tokens in the translation. Nonetheless, there may be cases in which the model fails to correctly translate from common entities in the source code to valid entities in the target code.

Looking into our dataset, there was no occurrence of `std::unique_ptr` in the test set. However, as the training set has `std::unique_ptr`, we intended to test the capability of CodeRosetta by developing a C++ code that uses `std::unique_ptr`, we made sure this sample does not exist in the training set:

### Reference C++ Code

```
void initializeArray(std::unique_ptr<int[]>& array, int size) {
    for (int i = 0; i < size; ++i) {
        array[i] = i * 10;
    }
}

```

### CodeRosetta CUDA Generation

```
__global__ void initializeArray(int *array, int size) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < size) {
        array[i] = i * 10;
    }
}
```

This sample is about array initialization. As can be seen, the model is able to provide the CUDA code. It removes the `std::unique_ptr` from the CUDA kernel as it is not supported in the device code. In CUDA, `std::unique_ptr` is called in the host kernel, which is the main function. Our model focuses on the generation of device function and hence is successful in this case.  

Moreover, from the test set, we checked where our model could potentially fail. Here is an example of a translation that CodeRosetta struggles with:

### Reference C++ Code: 
```
void getRho(const int numOfNucl, const double* psi, const double* occNo, double* rho, const char debug) {
    *rho = 0;
    for (int i = 0; i < numOfNucl; ++i) {
        *rho += occNo[i] * psi[i] * psi[i];
    }
    if (debug == 1) {
        printf("DEBUG ▁ print ▁ of ▁ RHO:\n▁ RHO ▁ = ▁ %f\nThis ▁ is ▁ the ▁ last ▁ line (RHO).\n\n", *rho);
    }
}
```
### CodeRosetta CUDA Generation: 

```
__global__ void getRho(const int numOfNucl, const double *psi, const double *occNo, double *rho, const char debug) {
    *rho = 0;
    for (int i = blockIdx.x * blockDim.x + threadIdx.x; i < numOfNucl; i += blockDim.x * gridDim.x) {
        *rho += occNo[i] * psi[i] * psi[i];
    }
    if (debug == 1) {
        printf("DEBUG print of RHO:\nRHO = %f\nThis is the last line (RHO).\n\n", *rho);
    }
}
```

In this example, CodeRosetta, along with all larger models like GPT-4, Gemini Ultra, and Gemini Pro, failed to translate the code correctly. We verified the generated code by executing it. The line `*rho = 0;` initializes the variable rho to zero. However, if this kernel is executed by multiple threads across different blocks simultaneously (which is very likely), all threads will write to the same memory location. This results in a race condition where multiple threads are simultaneously updating the value of *rho without any synchronization mechanism, such as atomic operations or reduction techniques. As a result, each thread's write could overwrite others, leading to incorrect and unpredictable outcomes. To rectify this issue, `rho` should be set to 0 in the host code, and `atomicAdd` should be included in the computation.

---

We hope these clarifications address your feedback and questions effectively and you find our rebuttal and additional discussion satisfactory.

We would greatly appreciate it if these additional results and clarifications lead you to reevaluate our work.

At the end, if there are any other lingering questions or requests, please do not hesitate to ask. Our goal is to work with experts like you to improve the quality of our paper and make our research more valuable for the community.

Thank you,

The Authors
Dear Reviewer BqwA,

Thank you for your quick response, your thoughtful feedback, and the opportunity to address your questions.

---

## Metrics and Compilation Definition

**ChrF Metric:** Yes, ChrF refers to the Character n-gram F-score. We apologize for not providing the full name earlier. We used the HuggingFace Evaluate Metric library with the default $\beta$ value set to 2. We used the ChrF metric following the recommendations from the *“Out of the BLEU: How Should We Assess Quality of the Code Generation Models?”* paper, which states that: *``ChrF and ROUGE-L are the best-performing metrics for the assessment of code generation models among the metrics we consider.``* We have also included the results  using the ROUGE-L metric:

| Model         | CodeRosetta | GPT4  | Gemini Ultra | Gemini Pro |
|---------------|-------------|-------|--------------|------------|
| Rouge-L(CUDA) | 82.12       | 63.37 | 69.27        | 69.82      |


**Compilation Accuracy:** We acknowledge the limitation of using compilation accuracy as a metric in code translation, as it is possible for a model to generate code that compiles successfully but diverges from the intended functionality. To partially address this, we conducted a manual evaluation of 30 generated code samples to compare their behavior against reference implementations. While we fully understand the limitations of such evaluations, we found that the functional correctness of generated code was preserved in the majority of samples (~93%). We also did run those generated codes against the reference and found them to be compilable and functionally correct producing the same outputs with the same inputs as the reference. In addition, the use of the compilation accuracy metric enabled us a direct comparison with BabelTower (ICML 2022), especially given that their codebase is not open sourced.

---

Following this discussion, we will dedicate a section in our paper that covers the limitations of existing metrics in code translation and our efforts to partially address these limitations through manual inspection. We will also include the samples in the Appendix. 

Taking all of these additional results into account, we believe we have made our best effort to address this concern, though we acknowledge it may not be perfect. That said, we would be more than happy to include any additional metrics you suggest that may better characterize the performance of our work.

---

## Fine-Tuning 

`Paper-Tables-1 & 2` report the results from the fine-tuned model. `Paper-Table-3-Page-9` reports the results from the pre-trained model. 

**Discussion on Fine-tuning for Code Translation w/o Verification in Supervised Manner.** In code translation, paired data is scarce. However, our model benefits from a foundational understanding of code translation acquired through unsupervised and self-supervised pre-training (243K training examples). We show that finetuning, even on limited synthetic data—*without verifying the generated samples and their one-to-one mapping to input code in a supervised manner*—generated by larger models (merely 5K paired samples---less than 2% of total data points), can further boost the model’s performance. While synthetic data may introduce some errors (as large models can make mistakes in their translation), the combination of foundational understanding during pre-training and fine-tuning with such a small number of synthetic data points can lead to additional improvements.
Dear Authors,

Thank you for your prompt response.

**Metric Issue and Formal Definition of Compilation**

* **ChrF Metric:** I assume 'ChrF' refers to the 'Character n-gram F-score'. Could you share the $\beta$ value used in your settings? Much like token-level metrics, the ChrF metric at the character level may also fail to accurately reflect the true code translation capabilities.

* **Compilation Accuracy:** In natural language translation tasks, language models often generate specious responses. I am concerned that your model might generate code that compiles correctly but is functionally different from the intended solution.

**Functional Correctness**

* I understand that it is not feasible to conduct extensive experiments to evaluate the functional correctness of the generated code within the rebuttal period. I appreciate your efforts in manually analyzing the model's outcomes.

**Fine-Tuning with Synthetic Data**

* Could you clarify whether the scores reported in Tables 1 and 2 originate from the pre-trained model or the fine-tuned model?

**Model’s Language Translation Capability**

* Given that you are pre-training the model with unsupervised objectives, I recognize that the model can acquire monolingual capabilities, and perhaps some bilingual capabilities from frequently occurring common usages—the area where rule-based deterministic algorithms excel. However, I am curious about the model's ability to handle novel cases. For example, you mentioned that "some entities are not common among languages, such as pointers." How does the model handle the conversion of 'cuda::unique_ptr' to C++ code?

**Differences in ASTs of Different Languages**

* Thank you for providing an answer to my question.