Function-to-Style Guidance of LLMs for Code Translation
Authors: Longhui Zhang
Bin Wang
Jiahao Wang
Xiaofeng Zhao
Min Zhang
Hao Yang
Meishan Zhang
YU LI
Jing Li
Jun Yu
Min Zhang
Venue: ICML 2025 poster (2025)
Keywords: Code Translation
Large Language Models
Primary area: deep_learning->large_language_models

Abstract: Large language models (LLMs) have made significant strides in code translation tasks. However, ensuring both the correctness and readability of translated code remains a challenge, limiting their effective adoption in real-world software development. In this work, we propose F2STrans,  a function-to-style guiding paradigm designed to progressively improve the performance of LLMs in code translation. Our approach comprises two key stages: (1) Functional learning, which optimizes translation correctness using high-quality source-target code pairs mined from online programming platforms, and (2) Style learning, which improves translation readability by incorporating both positive and negative style examples. Additionally, we introduce a novel code translation benchmark that includes up-to-date source code, extensive test cases, and manually annotated ground-truth translations, enabling comprehensive functional and stylistic evaluations. Experiments on both our new benchmark and existing datasets demonstrate that our approach significantly improves code translation performance. Notably, our approach enables Qwen-1.5B to outperform prompt-enhanced Qwen-32B and GPT-4 on average across 20 diverse code translation scenarios.

Funttion-to-Style Guidance of LLMs for Code Translation
Longhui Zhang' Bin Wang'
Jiahao Wang' Jing Li
Xiaofeng Zhao? Min Zhang? 2
Hao Yang*
Meishan Zhang'
Yu Li
Jun Yu
Min Zhang'
Abstract
sinclude ciostroze
Input() len(s) For in rangeto 1): If s(a] s[1 1]: printr "Bad")
using nasespace std;
Large language models (L.LMs) have made sig- nificant strides in code translation tasks. How- ever, ensuring both the correctness and readabil- ity of translated code remains challenge, lim- iting their effective adoption in real-world soft- ware development. In this work, we propose F2STRANS, a function-to-sttyee guiding paradigm designed to progressively improve the perfor- mance of LLMs in code translation. Our ap- proach comprises two key stages: (1) Functional learning, which optimizes translation correct- ness using high-quality source-target code pairs mined from online programming platforms, and (2) Style learning, which improves translation readability by incorporating both positive and negative style examples. Additionally, we in- troduce novel code translation benchmark that includes up-to-date source code, extensive test cases, and manually annotated ground-truth trans- lations, enabling comprehensive functional and stylistic evaluations. Experiments on both our new benchmark and existing datasets demon- strate that our approach significantly improves code translation performance. Notably, our ap- proach enables Qwen,.s 1.5 to outperform prompt- enhanced Qwens2B and GPT-4 on average across 20 diverse code translation scenarios.
inz rain { Jos::sync,nee with: (tt__iiit_tddi(fallpp cin. tin((); string sea;
pránt("Caod"))
cin see;
for (int 1-0; icseq, size()) 1; 1+1)
[+e[[i] seuli 1]} {
Correctacss Evalustire:
cout "Bad" <c undl;;
loop leminctioon
Inetrrn
Readability Evalustion:
Incossitest variblle
Baterdent
cout << "Gooe" ondi;
miemaid (mnii))
Ignoral opiinization
(4) Ctt code its functional and style informationn
(b) Pythsn code transbated by Quen-7B
with carrectuos readabiliey evalisations.
Figure 1: Limitations in the correctness and readability of code translation by LLMSs.
plex cases manually (Zhong et al., 2010). The advent of deep learning has led to the development of various learning- based strategies. A prominent approach involves pretraining monolingual code datasets, followed by fine-tuning with bilingual corpora to improve translation accuracy (Lachaux et al., 2020; Wang et al., 2021). Although learning-base -based techniques have significantly outperformed traditional rule- based methods, their performance remains insufficient for real-world deployment (Yang et al., 2024).
Large language models (LLMs), such as GPT-4 (Achiam et al., 2023), have revolutionized various coding tasks, in- cluding code generation (Liu et al., 2024a) and program repair (Fan et al., 2023), with their remarkable performance. This success has spurred growing interest among researchers in leveraging LLMs for code translation (Yan al., 2023b). Simple prompt-based learning allows LLMs to translate code effectively, and optimized prompts can further enhance their performance. For example, Bhattarai et al. (2024a;b) improved translation accuracy using a retrieval-augmmentd generation (RAG) strategy. Yang et al. (2024); Pan et al. (2024) utilized compiler feedback to iteratively refine trans- lations. Powered by L.MMs, current code translation models have reached unprecedented levels of performance, surpass- ing traditional approaches (Tao et al., 2024).
1. Introduction
Code translation involves converting code from one pro- gramming language another, task usually required for application porting or software migration (Nguyen et al., 2013). Traditionally, this task primarily relied on rule-based methods that required skilled programmers to handle com-
'Harbin Institute of Technology, Shenzhen, China. Huaaei Translation Services Center, Beijing, China. *Zhejiang Uni- versity, Hangzhou, China. Correspondence Jing Li <jingli.phd hotmail.com
While the effectiveness of LLMs in code translation tasks is widely acknowledged, most models face two critical lim- itations, as illustrated in Figure 1: (i) Correctness: For instance, StarCoder achieves an average success rate of only 7% on the traditional code translation benchmark Co-
Proceedings of the 4rnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Function-to-Style Guidance of LLMs for Code Translation
deNet (Pan et al., 2024). (ii) Readability: Even when translations are functionally correct, they often fail to pre- serve the source code's s style, including code structure and variable naming conventions (Weisz et al., 2022). This lack of readability imposes a substantial burden on developers, as reading poorly structured code often takes longer than writing it from scratch (Martin, 2009). Undeniably, more powerful LLMs can produce higher-quality code transla- tions, but they come with notable drawbacks, such as mas- sive model sizes (e.g., Qwen32n (Qwen et al., 2025)) or limited accessibility (e.g., GPT-4). Relying solely on the inherent capabilities of LLMs to overcome these issues is only a short-term solution.
2. Methodology
As illustrated in Figure 2, F2STRANS employs a two-stage progressive learning paradigm. First, to ensure functionally consistent code translation, we perform functional learning using cmnction-consisten code pairs. Second, to improve the style alignment between source and target code, we propose a novel style-learning mechanism based on positive and negative style translation examples.
2.1. Function-oriented Guidance
A fundamental requirement of code translation is func- tional consistency- produce identical outputs for the same inputs. A simple strat- egy (Khan et al., 2024) is to fine-tune LLMs using source and target language code that solves the same programming problem on online programming platforms. Platforms such as Codeforces host numerous programming problems, along with test cases and code solutions submitted by developers worldwide, making this strategy feasible.
that source and target code
To address these challenges, we propose F2STRANS, a two-stage guidance framework improve both correct- ness and readability: functional learning followed by style learning. Functional learning involves training the model to generate target code that preserves the functionality of the source code. To achieve this, F2STRANS mines cross- language code pairs from online programming platforms, selects pairs with consistent solutions and functionality, and utilizes these pairs for functional learning. Style learning ensures the model accurately preserves the stylistic features of the source code in the translated target code, thereby improving readability. In this stage, F2STRANS generates target code samples exhibiting both good and poor stylis- tic quality, enabling the model to recognize and prioritize maintaining stylistic consistency through the style learning.
However, this strategy can create ambiguity in the model's 's understanding of the optimal translation, since program- ming problems typically have multiple solutions. More critically, solution inconsistencies may lead to divergent output behaviors between technically correct code solutions. For example, while both Dijkstra's and Floyd-Warshall's Warshall's algorithms solve the shortest path problem, they may pro- duce different but equally valid results for the same graph. Obviously, training on code pairs with inconsistent outputs is sub-optimal. These limitations motivate us enhance the quality of the data generated by this strategy.
We conduct extensive experiments to validate our approach. To overcome the limitations of existing benchmarks, such as outdated source code, insufficient test cases, and missing ground-truth translations, we construct a new benchmark. Utilizing this new benchmark, along with the traditional benchmarks, we evaluate F2STRAN ANS across 20 code trans- lation scenarios encompassing five programming languages: C, C++. Go. Java, and Python. The results demonstrate that our approach is effective across LLMs of varying types and scales, including StarCodergz and Qwenu.s-7B. No- tably, with our approach, Qweni.5n surpasses GPT-4 in code translation tasks.
Data Construction. To develop our dataset, we first select highly relevant source- source-target target code pairs from a large number of solutions to the same program- ming problem, and then retain those exhibiting identical ouput- behavior.
Relevance-driven Code Pair Selection. To balance ef- fectiveness and efficiency, we employ a two-step method to identify highly relevant code pairs. First, we use a lightweight code embedding model Jina (Günther et al., 2024) to retrieve the top ten similar code pairs based on cosine similarity of embeddings. Next, we apply a rating basle method where an LLM judge assesses the so- lution consistency of code pairs using fine- mrne-graine labels. The scoring process is defined as:
We summarize the key contributions of our work as follows:
We propose F2STRANS, a funetion-to-style guid- ance framework designed to enhance both the correct- ness and readability of code translations generated by LLMs.
We create comprehensive benchmark rigorously evaluate the functional accuracy and stylistic consis- tency of code translations.
score (src, tgt) = Σ p (ks (s fgt)) k
(I)
exp (sk)
P [(k] Pr(src, Σe exp (sw)
Our approach significantly improves the quality of translated code across various types and sizes of LLMs in 20 investigated translation scenarios.
where we use QwenyB as the LLM judge. Here, sre and tgt represent source and target code, respectively, while
2

---

Function-to-Style Guidance of LLMs for Code Translation
Stage Function-oriented Guidance
Top-1
let
src
Translate code into Y:
sEC
Top-k
Differential Test
Input
sre
Ltun
fgt
Mfun
Code Snippets
Code Pairs Recall
L.M Judgel
M
tgt
Relevance-driven Code Pair Selection
(a) Function-consistent Data Construction
Functional Learning
Stage 2. Style-oriented Guidance
StyleSim
Differential
Translate code into Y:
tgt
tgt
Test
Input
src
Style-aware LLM
Style Consensus Selection
Laty
Paro
> tgt
Matyle
Source Code
M
Snippels
Agr
fgt
tgt
Mfann
Style-poor Translation Selection
(c) Style-oriented Data Construction
(d) Style Learning
Figure 2: Overview of our F2STRANS. Firstly, the base LLM Mbsse is transformed into an intermediate model M after function-oriented guidance. Subsequently, Minn is refined into the final model May through style-oriented guidance.
(src, tgt) is the prompt for the LLM judge, shown in Appendix F.1. The variable & is an integer ranges from to K, with higher values signifying greater relevance, and 8y the log- likelihood score of label i generated by the LLM judge when prompted with P. (sre, lgl). We use fine- grained labels {1. . }} instead of binary labels, because code pairs for the same programming problem often exhibit varying degrees of solution similarity. Besides, to ensure each code pair receives a distinct score, we aggregate the tog-likelihood scores s, of all labels to obtain continuous relevance scores.
translate the source code src. The trained model is Miun-
2.2. Style-oriented Guidance
Although our functional guidance ensures data quality at the solution and function levels, style inconsistencies inevitably persist in code pairs derived from online programming plat- forms. These inconsistencies usually include variations in variable naming, function signatures, code structure, and comments between source and target code. Training on such style- byyee-inconsistent data can limit the model's learning of the code style, thereby diminishing the readability of the trans- lated code. Therefore, we introduce style-oriented guidance to alleviate this issue.
Differential Testing. To ensure functional consistency, perform differential testing (McKeeman, 1998) the most relevant code pairs. This process involves executing identi- cal input on both the source and target code and comparing their outputs for discrepancies. Only code pairs whose input and output behavior are exactly the same are retained.
Style-oriented Data Construction. We construct positive and negative code translation data to help Minn discern the desired code style through comparison. Positive translations maintain stylistic consistency with the source code, while negative translations do not.
Functional Learning. Instruction Fine-tuning (IFT) in- volves training LLMs instruction-outpur pairs with next- token prediction objective to enhance the models' ability to follow instructions (Zhang et al., 2024). We preliminary im- prove the code translation performance of base LLM Musse by applying IFT on the constructed imncion-consistenn code translation data as follow:
Positive Translation Construction. We use strong LLM Qweng28 to generate multiple style-aware translations and then select the optimal translation from these candidates. Specifically, we first design style-aware prompt, as de- tailed in Appendix F.I, which explicitly instructs the L.M to adhere to the stylistic conventions of the source code. Using this prompt. Qwenzan generates m translations of the source code. Next, we perform differential testing to filter these translations, retaining only the subset T that successfully passes all test cases, thereby ensuring func- tional correctness. Finally, we select the target code tgt
Ltan (scr; Lg()
Σ log (lgl, P (src) Aglte))
where tgt, denotes the i-th token of target code tgt, tgtze represents the token sequence preceding the i-th token in tgt, and P(rcc) code translation prompt designed to
3

---

Function-to-Style Guidance of LLMs for Code Translation
that best preserves the source's style from T+
#I #Lang
#Code
Date
#Cases
GT
A major challenge in this process is how to choose the best translation tgt* from T While our style-aware prompt contributes to good code style, the occasional style error is still inevitable. Existing style evaluation methods, such as CSSim (Li et al., 2024), are limited to code pairs within the same language. To avoid the use of abnormal code trans- lation, we propose a style consensus selection mechanism, which identifies the optimal translation by selecting the one with the highest stylistic similarity to the other candidate translations. The selection mechanism can be mathemati- cally represented as follow:
CodeNet
5 200 x Pre-2021
x 20
X
F2STRANS (Ours)
5
1000 x 20 Mid-2024
50 x 20
Table I:
Comparison of CodeNet and our evaluation bench- mark. Both datasets cover 20 translation scenarios across five languages (I (Lag)). Our benchmark surpasses CodeNet in terms of a larger and more up-to-date codebase (Code and Date), extensive test cases (Cases), and manually annotated ground-truth translations (GT).
after meticulous data cleaning, leading to inaccurate evalua- tions (Xu et al., 2024). For example, the latest data for the CodeNet benchmark comes from 2020 (Puri et al., 2021). To accurately assess the code translation capabilities of LLMs, benchmark based on more recent data is essential
tgt arg max
Σ
StyleSim (Igt,, tgt;),
(3)
tot, ETT
where StyleSim(.) is a function that measures the stylistic similarity between two code snippets in the target language. We use the CSSim metric (Li et al., 2024) as StyleSim, which quantifies code stylistic similarity based on variable naming, API invocation, and code structure. A detailed explanation of CSSim can be found in Appendix A.
Motivated by this insight, we introduce a new code transla- tion benchmark with three key advantages over traditional benchmarks: (i) Up-to-date source code: We first collect the most recently released programming problems from Codeforces, and then select up to two code solutions from each problem source code. (ii) Extensive test cases: Each source code undergoes extensive cases that encompass edge conditions and difficult scenarios of programming problems. (iii) Consistent functional and stylistic translations: We manually translate each source code into the target language to support functional and stylis- tic evaluation of the translated code. The detailed statistics presented in Table 1.
Negative Translation Collection Construction. To ensure that negative translations are representative, we use Mfan as the negative translator. Specifically, MM fun first generate multiple candidate negative translations. Then, we evaluate their style consistency with the positive translations tyt* using the CSSim metric and retain negative translations T- whose CSSim values are less than a.
annootate test
Style Learning. Inspired by contrastive learning (Chen et al., 2020), we propose list-wise loss function to encour- age Maun generate soyeesconsittet target code tgt while suppressing inconsistent translations T
3.2. Experimental Settings
Implementation Details. In function-oriented guidance, we set the maximum algorithmic consistency label K in Eq. 1 to 5. In the style-oriented guidance, we set both the numbers of positive translations T* and negative transla- tions T- namely m and n. to 10, with the value of a in negative translation collection construction set to 0.8 and the trade-off hyperparameter 33 in Eq. 5 fixed 0.6. We combine training data from various translation scenarios for mixed training, enabling single LLM translate among all the investigated languages. The LLM-judge prompt P. in function-oriented style-oriented guidance, and the prompt P of F2STRANS in Eq. 2 and 4 are shown in Appendix F.1. Appendix B shows more detailed implementation details.
exp(S(src, ))
Cia(sre, lggtt ,T") = log
Enger exp(SS Lg())
(4)
S(src, fgt)=
II p (tgt; | (P(sre),
where S(sre, tgt) denotes the probability that M trans- lates the source code src into the target code tgt, and T = T U }. Additionally, we apply IFT on pos- itive translations to emphasize their importance:
oriented guidance, the style-aware prompt in
Cay = 3. Cis(serr igit, )
) + (1 -1-18 B) (a(ssr, (gt ) (5)
where B is a trade-off hyperparameter ranging between O and 1, and Lin is the IFT loss, calculated the same way as Crn in Eq. 2.
Baselines and Metric. We evaluate our approach against two SOTA LLMs: Qweng2B and GPT-4. For each model, we implement four established prompt learning strategies: Direct Prompt Learning (Yang et al., 2024), Chain of Thought (CoT) (Yan et al., 2023b), RAG (Bhattarai et al., 2024a;b), Self-Debug Prompt Learning (Yang et al., 2024; Pan et al., 2024). These strategies have been proposed and
3. Experiments
3.1. Benchmark Construction
Since pre-training LLMs requires handling vast datasets, traditional benchmarks face the risk of data leakage even
4

---

Function-to-Style Guidance of LLMs for Code Translation
Translation - (}
Translation C++ - O
Translation Go - O}
Translation Java - ()
Translation Py - {
Method
LLM
Avg.
C++
Go
Java
Py
C
Go
Java
Py
C
C++
Java
C
C++
Go
Py
C
C++
Go
Java
(I) CodeNet Benchmark
Direct
88.0
68.9
76.4
61.3
83.5
69.0
81.0
59.5
69.4
74.9
75.4
58.3
71.2
79.3
64.2
61.1
76.9
81.4
69.9
88.5
72.9
CoT
82.9
72.4
75.4
62.8
80.5
61.5
81.0
65.5
75.9
72.4
77.9
63.3
78.3
80.3
58.6
61.6
73.4
82.4
76.4
88.4
73.5
Qwensgas
RAG
87,5
76.5
79.0
70.1
83.1
73.1
81.7
65.6
69.0
75.4
76.1
66.0
77.4
78.4
71.9
66.3
84.6
81.5
77.0
87.9
76.4
Self-debog
89.1
74.4
77.9
70.1
86.6
71.7
83.9
67.3
74.3
78.4
79.9
68.0
76.6
82.5
68.8
69.4
81.0
85.0
76.0
90.5
77.6
Direct
91.0
69.5
80.9
62.6
83.3
68.2
84.0
64.0
77.4
80.4
77.9
69.4
77.8
83.9
66.7
67.3
76.9
84.2
58.1
82.3
75.3
CoT
88.8
73.2
81.1
65.2
82.7
67.0
83.7
68.8
81.3
80.8
80.6
73.5
86.3
86.2
63.5
67.1
73.7
84.5
62.6
83.5
76.7
GPT-4
RAG
76.2
72.0
84.6
70.6
84.7
76.2
78.3
77.0
72.6
77.7
83.2
72.9
72.1
84.4
86.3
64.3
78.9
delfu
93.5
72.5
87.5
78.5
86.5
72.0
87.5
74.0
84.5
88.0
88.5
81.0
87.0
89.5
70.0
78.5
85.5
87.0
62.0
85.0
81.9
Qweno. au
91.0
78.9
81.9
73.9
84.5
82.0
84.5
75.5
70.9
73.9
77.4
76.4
74.2
85.4
77.3
75.8
74.4
80.9
73.4
85.4
78.9
Qwenj.a
83.9
90.5
90.0
87.5
80.0
79.4
78.4
86.4
82.3
84.4
87.4
85.1
F2STRANS (Ours)
Qwensa
93.6
86.4
90.5
82.9
90.0
89.0
93.5
84.0
79.4
81.4
86.4
83.9
85.9
88.9
85.9
84.8
89.9
88.4
85.4
91.0
87.1
Qwenza
91.5
92.5
93.5
86.5
92.0
92.5
93.0
88.0
82.9
86.9
89.4
87.9
88.4
89.9
88.4
87.4
93.0
89.4
87.4
94.5
89.8
StarCoderae
92.0
87.9
89,9
84.4
92.0
89.5
91.5
83.5
80.4
81.9
84.9
84.9
86.4
85.4
83.8
85.4
86.9
85.9
84.9
92.5
86.7
(II)
F2STRANS
Benchmark (Ours)
Direct
48.5
57.4
33.6
56.4
39.1
63.2
69.9
71.6
63.1
32.7
37.1
32.1
53.9
CoT
80.3
43.7
68.2
51.9
54.4
34.3
56.0
42.0
66.4
72.7
64.9
53.8
68.0
78.5
44.4
63.4
29.9
40.9
37.0
45.5
54.8
Qweasya
RAG
81.3
57.9
67.6
57.2
56.6
46.6
55.3
41.6
61.4
69.3
60.9
54.6
71.1
77.3
59.9
68.2
34.3
37.8
34,1
44.0
56.9
Self-debug
83.1
69.8
56.0
60.4
35.7
59.0
47.2
68.5
72.2
57.9
77.8
51.3
71.8
35.3
39.4
37.3
58.2
Direct
89.9 58.1
81.3
51.4
68.9
52.6
70.7
52.6
70,9
81.0
74.1
68.9
77.2
87.7
60.9
62.4
34.4
44.4
38.3
46.7
63.6
CoT
60.2
80.8
54.0
69.2
50.2
71.2
56.5
75.1
82.2
75.7
81.5
89.2
57.4
63.1
34.2
42.8
40.5
44.5
64.5
GPT-4
RAG
89.7 61.4
87.0
59.7
69.2
56.0
72.2
61.8
71.4
80.8
73.4
75.1
81.0
87.0
64.6
67.5
39.5
47.6
42.5
49.3
66.8
debfu
91.3 61.5
85.6
63.2
71.4
55.7
74.1
61.4
72.4
83.6
78.7
76.7
84.5
88.7
67.4
72.9
40.6
46.8
42.8
51.7
68.5
Qweno.ag
71.5
76.1
64.8
66.0
60.6
53.2
64.8
74.8
70.3
84.0
73.9
72.8
25.1
35.1
27.6
34.1
62.1
Qwenj.aa
93.5
75.7
83.6
71.3
75.1
68.2
71.2
62.3
70.7
79.9
73.2
76.6
79.5
89.2
83.7
82.1
35.1
44.8
35.7
44.6
69.8
F2STRANS (Ours)
Qwensa
95.2
81.3
86.7
76.1
78.2
71.8
75.4
68.9
75.2
82.5
77.4
80.7
81.1
91.1
86.8
84.6
38.9
48.4
43.6
50.4
73.7
Qwenza
76.1
79.5
75.8
77.6
69.4
76.4
85.1
H2.7
91.0
88.2
84.7
51.3
75.9
StarCoderas
94.3
78.8
86.5
76.3
77.8
71.0
75.1
66.7
76.6
82.0
77.2
80.2
82.0
90.2
87.0
83.4
37.9
45.2
42.9
46.5
72.9
Table 2: Code translation results of various models on CodeNet and our benchmark. Following standard practice, we adopt CA as the evaluation metric. The bold values represent the best results, while the underlined values indicate the second-best.
validated in related works to enhance model performance in code translation tasks. Detailed descriptions of each strategy are provided in Appendix C.
we recognize the immense potential of LLMs in code trans- lation tasks. For instance, F2STRANS enables Qweng.s. to surpass RAG-based Qwena2n in average performance across the various translation scenarios examined. While F2STRANS lags behind GPT-4 with self-debug prompt learning in some specific cases, such as Go-to-C transla- tion, is important to consider the increased computational cost associated with the additional reasoning steps required by self-debug prompt learning for bug fixing. Therefore, the advantages of F2STRANS remain significant.
Following standard practice (Lachaux et al., 2020), we adopt Computational Accuracy (CA) as our evaluation metric, which measures the proportion of translated code that pro- duce identical execution results to the source code across all inputs. We allowed each LLM only one translation attempt per source code. However, for self-debug prompt learning, which inherently requires multiple I.I.M invocations, we allow an additional iteration for bug fixing.
Evaluation on F2STRANS Benchmark. As shown in Table 2(II), all evaluated models score at least 10 points lower on average in our benchmark compared to CodeNet, highlighting the greater complexity of our proposed bench- mark. An interesting observation is the significantly weaker performance of all models Python translation within our benchmark. This may be attributed to Python's inter- preted nature, contrasting with the compiled nature of the other languages. This disparity in language types intro- duces potential challenges in code translation. Nonetheless, the overall trend observed on our benchmark consistent with that of the CodeNet benchmark. F2STRANS enables
3.3. Main Results
To verify the broad applicability of F2STRANS, our experiments involve various types and sizes of LLMs, including Qweno.5-78 (Qwen et al., 2025) and StarCoderg, (Lozhkov et al., 2024). The performance of our model and baselines on CodeNet and our benchmark is presented in Table 2, and additional benchmark results in Appendix D.
Evaluation on CodeNet Benchmark. From Table 2(1,
5

---

Function-to-Style Guidance of LLMs for Code Translation
Base
Bas+Fuu
Base+Style.
Base+Pun (Suy)..QOus)
Qwena
StarCodersa
90
83,5
7
72.9
Method
CodeNet
Ours
CodeNet
Ours
80
73.9
78.9
79.5
62.1
63.3
67.2
F2STRANS (Ours)
Tunction-oriented Guidance
78.9
62.1
86.7
72.9
70
71.3
544
wfo RdSel
75.5
57.5
83.7
68.8
50
49.6
w/o LLM Judge
77.5
59.5
84.3
70.2
10
10.....
4.5
7:4
10
2.1
5.3
Style-oriented Guidance
w/o Dif. Test
78.2
60.2
85.8
71.0
StarCoda''s
0
Qwitlem
StarCodeF
w/o StyPro
78.6
61.5
86.3
72.4
(a) CodeNet Benchmark
(b) F2STrans Benchmark (Ours)
w/o SCS
77.7
58.6
84.2
69.4
w/o SpTS
78.4
60.3
86.3
71.4
Model performance under different training strate- gies: base LLMs, function guidance (Fun.), style guidance (Style.), and our mmnction-to-stlee guidance.
Figure 3:
wfo List
74.3
56.8
84.1
68.8
w/o Cin
75.9
59.0
85,4
70,8
Table 3: Ablation results of our F2STRANS, evaluating the contribution of the following components: relevance-driven code pair selection (RdSel), LLM judge and differential test (Dif. Test) in function guidance, along with style-aware prompt learning (StyPro.), style consensus selection in Eq. (SCS), ayyl-poor translation selection (SpT'S) and the loss function in style guidance.
Qwen1.sB to surpass both Qweng28 and GPT-4 in average performance, despite the latter utilizing self-debug prompt learning. Furthermore, increasing the model size can further enhance performance.
3.4. Ablation Analysis
Our ablation analysis quantifies the contribution of each part of F2STRANS to the overall performance improvements.
strong task generalization capabilities of LLMs.
Differential Testing. Although we have strived to maxi- mize the solution relevance of the training data, overlooking differential testing has undeniably adversely affected the training process. This phenomenon indicates that assess- ing the input-output behaviors of code pairs is the most effective approach to verifying their functional consistency.
Stytyee Guidance. Figure 3 shows the indi- vidual contributions of function and style guidance based on Qweno.5 5 and StarCoderap. As shown, both training stages exhibit significant performance gains, validating their importance to optimal final results. In particular, the in- fluence of style guidance is highly remarkable. This is unsurprising, as style guidance involves both positive and negative translations, and the positive translations are de- rived from Qweng2B rather than online data. Without these two stages, both Qweng.5 and StarCoderyn score below 10, underscoring the inherent limitations of naive LLMs in code translation tasks.
Style-oriented Guidance. As shown in Table 3 and Fig- ure 4, we evaluate four key modules of style supervision:
Style-aware Prompt Learning. We employ style-aware prompts guide LLMs in generating positive translations that align with the source code style. To evaluate the ne- cessity of this configuration, we remove all style-related information from the prompt, thereby allowing the LLMs to generate code without stylistic constraints. As shown in Ta- ble under "w'o StyPro", both Qweno.5B and StarCodery exhibit decreased performance, indicating that well-defined styles in the training data facilitate LLMs' adaptation to code translation tasks
Funetion-oriented Guidance. The first part of Table 3 examines the impact of three modules in this stage:
Relevance-driven Code Pair Selection. To validate the effectiveness of this module, compared F2STRANS variant that randomly selects code pairs, disregarding solu- tion differences. The notable performance decline observed with this variant confirms our hypothesis that the solution gaps between code versions are crucial and cannot be ig- nored, Bridging these gaps is essential for achieving optimal model performance.
Style-oriented Data Selection of Positive and Negative Candidate Translations. As illustrated in Figure 2, we em- ploy a style consensus strategy to select the optimal positive translation from those generated by Qwenga and utilize negative translations that significantly differ in style from the positive data. In the ablation studies presented under the "w/o SCS" and "w/o SpTS" entries in Table 3, we omit these two data selection methods and instead randomly select one positive candidate translation and n negative candidate translations for the style learning in Eq. 5.
LLM Judge. To assess the impact of LLM judgments, implemented variant that directly uses the embedding model Jina to select the top-ranked relevant code pair for training. The observed performance decline in this model variant indicates that, although Jina is specifically designed for relevance assessment, LLM judgments can effectively refine its retrieval results. This further corroborates the
The results demonstrate that random data is of inferior qual-
6

---

Fusytion-to-Style Guidance of LLMs for Code Translation
dpo
Liss
&in (Ous)
62
72
90
80
60
70
85.5
U4
86.7
75
71.4
72.9
58
2
Qwen
8
10
68
2
4
StarCodor
8
38
10
80-
78.2
-7899
82.2
65
70
61.1
62.1
66.7
08.8
60
(a) The Scale Positive style Translations (m)
75.7999
74.3
55.7
56.8
56
50
62
72
Owen
Sharooder
Qwenass
(a) CodeNat Benchmark
(b) F2SThan Benchmerk (Ouss)
60
70
58
Qwen 1158
68
StarCoder
38
Figure 5: Results of style learning based on various loss functions.
22
4
6
8
10
2
4
8
10
(b) The Scale of Syyle-porrrranslatioos Translations (12)
ing strategies, such as RLHF (Lambert et al., 2022) and PPO (Schulman et al., 2017). Here, we compare our list- wise loss function Lis in Eq. 4 with Direct Preference Opti- mization (DPO) (Rafailov et al., 2024), a leading preference learning method known for its simple and effective training process. Since DPO uses only one positive and one negative data, in our experiment, we align the positive sample of DPO with F2STRANS, while the negative data is randomly selected from the negative data of F2STRANS. Additionally, following our Eq 5, we also compare the model's perfor- mance with the instruction tunnnn loss function Ln as an auxiliary. The results shown in Figure demonstrate that, under both experimental setups, our list-wise loss function Lust significantly outperforms DPO. Furthermore, including Lit as an auxiliary also improves DPO performance.
Figure 4: Impact of the number of positive translations gen- erated by Qwen and the number of negative translations used in style learning, based on our benchmark.
ity compared to our carefully curated dataset. This yields two key insights: (I) Even functionally correct translations generated by powerful LLMs may contain subtle stylistic inconsistencies. (II) Ensuring a clear stylistic distinction between positive and negative translations facilitates more effective style learning.
The Number of Positive and Negative Candidate Transla- tions. Figure 4 illustrates the influence of these two numbers, m and n, based our benchmark. We observe that increas- ing both m and n significantly enhances model performance, with results stabilizing around m = 8 and n = 10. This is expected, as a larger number of positive candidates increases the likelihood of identifying the optimal translation, while more negative translations expose potential deficiencies of M Furthermore, the steeper curves demonstrate that scaling these data yields greater benefits for StarCoder indicating that larger model sizes achieve higher perfor- mance ceilings.
LLM Judge of Function-oriente oriented Guidance. In Figure 6, we provide an in-depth analysis of LLM judge from two perspectives: the number of fine- fine-grained labels K and the comparison with explicit scoring.
The Number of Fine- Labels K. First, compared to binary labels, the advantage of using more fine-grained labels is evident. Performance initially improves with in- creasing label details, peaking around K = 5, and then declines. This suggests that while finer labels enable LLMs to make more accurate judgments about data quality, exces- sive refinement may cause confusion in the model.
Loss Function. Our loss function comprises Lis and Lin- Ablation studies in Table 3 indicate that Cist significantly enhances model performance, and the addition of L.n further increases these benefits. Moreover, List outperforms Lin because it not only encourages the generation of positive translations like Lit does, but also suppresses the generation of negative translations.
The Comparison with Explicit Scoring. Unlike the log- likelihood score that integrates all labels as shown in Eq. 1, here we directly use the labels generated by LLMs, with the number of labels K set to 5. It is observed that explicit scoring performs worse than our method, indicating that the log-likelihood score contains more valuable information than a simple LLM-generated response. Moreover, even with explicit scoring, the results outperform those achieved with binary labels. This further highlights the importance of fine- fine-grained labeling.
3.5. Discussion
In this subsection, we conduct detailed experimental analy- ses to comprehensively evaluate F2STRANS.
Comparison Between Our List-wise Loss Function List and Preference Learning Loss Functions. Our style learning can be realized using traditional preference learn-
Evaluation from a Style Perspective. Using our manu- ally annotated target code within our benchmark as refer- ences, we assess the stylistic quality of translations gener-
7

---

Function-to-Style Guidance of LLMs for Code Translation
79.5
87.0
73.0
62.0
79.0
86.5
72.5
61.5
78.5
86.0
72.0
61.0
78.0
85.5
71.5
60.5
77.5
85.0
71.0
2
3
4
2
3
4
5
6
3
4
5
(a) CodeNet Banchmak
(b) CodeNet Benchmrak
(c) F2STrans Bonchmark (Qweenaa))
(d) F2STraan Benchmark (StrCookagg)
(Qwertue)
(StarCooergg)
Figure 6: The impact of the fine- fine-grained label count K (blue curve) and the use of explicit scoring (red dashed line) on the results the LLM judge. In the explicit scoring setting, we set K to 5 and directly use the scores generated by QwenrB as to measure the relevance between source and target code. In cases of ties among the highest scores, one selected randomly.
Dis
Dis
Disatru
CSSim f
Compilation
Runtime
Rundionll Froms
lemmnneinng
Sucassful Percation
Qwemsan
Piros
Frmrs
Porcation
Direct
17.7
24.6
28.6
76.4
52% (f21%)
CoT
18.9
24.3
28.1
76.2
RAG
16.5
23.6
27.2
77.6
64%
(16%))
Self-debug
16.3
23.4
27.0
77.8
GPT4
71%
(135)
Direct
16.8
23.2
26.4
77.9
CoT
17.3
22.8
25.8
78.0
67% (185)
RAG
Self-debug
15.4
15.2
21.8
21.8
24.5
24.8
79.5
79.3
Moson
Function
Guidorce
M Misew
Punction-to Style
Gurdance
Mayle
F23TRANS (Ourx)
Oweng.5B
13.8
20.1
23.8
80.7
Figure 7:
Comparison of error correction in base StarCoder: translations using function guidance and our function-to-style guidance. For the former, 31 % of the com- pilation errors are successfully corrected, while the latter increases the percentage 52%.
Qwen1.5B
12.7
19.1
22.1
82.0
Qwens 11.9
18.5
20.8
82.9
Qwen7H 11.1
17.9
19.6
83,8
StarCoderg 12.0
18.3
21.1
82.8
Table 4: Style evaluation of generated translations against the ground truth in our benchmark. The assessment mea- sures code differences in variable naming Disvar, API in- vocation Dis and code structure Disstru. with CSSim integrating all three aspects. Details of these metries are shown in Appendix A.
function guidance successfully rectifies the majority of the base LLMs' translation errors, while only correcting 31 % of compilation errors. However, our function-to-style guid- ance achieves comprehensive improvements across all error types, increasing the correction rate for compilation errors by 21%. This may be because when LLMs diligently adhere to the source code's style, they can avoid superficial code errors such as undeclared identifiers, which often lead to compilation errors. This further underscores the importance of style guidance.
ated by various models, employing the CCSim metric (Li et al., 2024). The results presented in Table demon- strate that F2STRANS significantly improves LLMs' stylis- tie awareness and generalizes across different scales and types of tested LLMs. A compelling piece of evidence is that T2STRANS-improver Qweno.5p exhibits superior code style compared to self-debug-basee based GPT-4, despite GPT-4's better functional performance shown Table 2. This indi- cates that functionally correct code can still exhibit stylistic deficiencies. Nonetheless, our F2STRANS effectively miti- gates this problem.
Multilingual Modeling Strategies. We evaluate four mul- tilingual modeling strategies (Yan et al., 2023b) for code translation across the five programming languages used in our work: one2one (training separate models for each lan- guage pair), all2one (translating code from multiple source languages into one target language using single model), one2all (translating code from one source language into multiple target languages using single model), and all2all (a unified model for all language pairs).
Analysis of F2STRANS in Correcting Errors of Base LLMss. Based on our benchmark and the previously es- tablished code eиror classification (Pan et al., 2024), we evaluate the effectiveness function guidance and function- tt-style guidance in correcting erroncous translations pro- duced by the naive StarCoderag. illustrated in Figure 7,
As illustrated in Figure 8, the onelall, all2one, and all2all approaches significantly outperform one2one, with all2all achieving the highest improvements. These results indicate
8

---

Function-to-Style Guidance of LLMs for Code Translation
on
aone
one
eall (Ours)
explored the impact of output formatting LLM perfor- mance. Bhattarai et al. (2024a;b) improved translations of low-resource programming languages using RAG strategy. Yang et al. (2024); Pan et al. (2024); Yin et al. (2024) intro- duced the self-dehugging strategy, where I.LM-generated target code is compiled, and any detected bugs are incorpo- rated into subsequent prompts to guide precise fixes. Huang et al. (2023): Szafraniec et al. (2022); Sun et al. (2024) used unified intermediate representation a pivot for translat- ing between programming languages, enabling models to capture language-agnostic code semantics effectively. Addi- tionally, multi-agent systems (Yuan et al., 2024) and human- machine interactive systems (Liu et al., 2024b) leveraging LLMS provide developers with more transparent reasoning processes and facilitates the generation of higher-quality translated code.
RL4
FiR
87.9
280
28.5
70.3
8
m2
75.6
3
758
CH
BLA
83
C+
Lava Go
LB
81.6
Laa Go
81
782
774
7A3
BL
B&G
BLG
G)) Quemb
(b) Ctareo
Figure 8: Comparison of various multilingual modeling strategies based on the CodeNet benchmark.
that mixed training across multiple programming languages enhances performance in F2STRANS. Notably, translations from Go exhibit the lowest success rates, which can be at- tributed to Go's relatively recent emergence in 2009 (Dono- van & Kernighan, 2015), resulting in limited knowledge within LLMs. However, the all2all strategy provides the most substantial gains when translating from Go. suggesting that it effectively bridges the knowledge gap by enabling LLMs to better understand the relationships between Go and other languages.
5. Conclusion
In this study, we introduced a novel feature-to-style training paradigm to effectively adapt LLMs for code translation tasks. Initially, we conducted functional learning using high-quality source-target code pairs from online program- ming platforms, generating functionally correct translations. Subsequently, we applied style learning based on positive and negative translation data, yielding more readable and stylistically consistent translations. This two-stage train- ing paradigm significantly improved the performance of LLMs across various scales and types on our newly con- structed benchmark as well as traditional code translation benchmarks, even surpassing GPT-4. The substantial gains achieved by our approach highlight its effectiveness in ad- vancing code translation capabilities of LLMs.
4. Related Work
Code translation is essential in software development and maintenance, and has been a subject of extensive research for decades (Mossienko, 2003). Early rule-based and pro- gram analysis- based methods, such as CxGo (c2g, 2023), were costly and required developers to possess a deep un- derstanding of both source and target languages. In the era of deep learning, Chen al. (2018) minimizes manual in- tervention by leveraging a Icarnable attention mechanism to transform the source language's 's syntax tree into its coun- terpart in the target language. To alleviate the reliance on high-quality parallel code pairs, a series of unsupervised training strategies have been proposed (Xue et al., 2024; Rozière et al., 2022). For example, Lachaux et al. (2020) em- ploys masked pre-training on large-scale monolingual code data and back-translation strategies learn the mapping between source and target languages. Another approach is to use code solutions to identical programming problems in different languages from competitive platforms as weakly supervised parallel data (Ahmad et al., 2021; Zhu et al., 2022; Yan et al., 2023a; Xie et al., 2023).
Impact Statement
This work introduces a function-to-style guiding paradigm to improve the correctness and readability of code generated by translation models. We anticipate that our approach will contribute to more efficient software development and maintenance. However, it is crucial to acknowledge the potential for misuse, such as the translation of malicious code. Ultimately, adhering to ethical guidelines is essential to ensure the responsible application of this technology.
A Acknowledgement
This work was supported in part by National Science Foundation of China (62336008, 62476070, 62125201, U24B20174), Shenzhen Science and Technology Program (ICX1200411202123500000 GXWD20231 128111332002000000 ZDSYS2023062609 1030008 KQTD20240729102155 and Department of Science and Technology of Guangdong
Recent advancements in LLM-based code translation have been substantial. Yan et al. (2023b) evaluated ChatGPT's performance on code translation tasks using standard infer- ence techniques such as direct prompting, few-shot learning, and COT, demonstrating the effectiveness of L.Mss. Re- searchers have since sought to enhance LLMs' code transla- tion capabilities from multiple angles. Macedo et al. (2024)
(2024A1515011540)

---

Function-to-Style Guidance of LLMs for Code Translation
References
Huang. Y., Qi, M., Yao, Y., Wang, M., Gu, B., Clement, C., and Sundaresan, N. Program translation via code distillation. arXiv preprint arXiv:2310.1 1476, 2023.
C to go translator.
https://github: com/ gotranspile/cxpee 2023.
Khan, M. A. M. Bari, M. S., Long, D., Wang, W., Parvez, M. R., and Joty, S. Xcodeeval: An execution-based large scale multilingual multitask benchmark for code under- standing, generation, translation and retrieval. In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 6766-6805, 2024.
Achiam, J., Adler, S., Agarwal, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Ahmad, W. U., Tushar, M. G. R., Chakraborty, S., and Chang, K.-W. Avatar: A parallel corpus for java-python program translation. arXiv preprint arXiv:2108. 1590, 2021.
Lachaux, M.-A., Roziere, B., Chanussot, L.., and Lample, G. Unsupervised translation of programming languages. arXiv preprint arXiv:2006.033: 2020.
Bhattarai, M., Santos, J. E.. Jones, S., Biswas, A., Alexan- drov, B., and O' O'Malley, D. Enhancing code translation in language models with few-shot learning via retrieval- augmented generation. arXiv preprint arXiv:2407. 19619, 2024a.
Lambert, N., Castricato, L., von Werra, L., and Havrilla, A. Illustrating reinforcement learning from human feedback (rlhf), Hugging Face Blog, 2022.
Bhattarai, M., Vu, M., Santos, E., Boureima, I., and Malley, D. O. Enhancing cross-language code transla- tion via task-specific embedding alignment in retrieval- augmented generation. arXiv preprint 2:X05:2412.055159 2024b.
Lee, J., Wang, Y., Li, J., and Zhang, M. Multimodal reason- ing with multimodal knowledge graph. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL.), 2024.
Li, H., Zhou, X., and Shen, Z. Rewriting the code: A simple method for large language model augmented code search. arXiv preprint urXiv:2401.045// 2024.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual repre- sentations. In Proceedings of the 37th International Con- ference on Machine Learning (ICML). pp. 1597-1607, 2020.
Liu, F., Liu, Y., Shi, L., Huang, H., Wang, R., Yang, Z., Zhang, L., Li, Z., and Ma, Y. Exploring and evaluating hallucinations in llm-powered code generation. arXiv preprint arXiv:2404.0090 2024a.
Chen, X., Liu, C., and Song, D. Tree-to-tree neural networks for program translation. arXiv preprini arXiv: 2018.
Liu, J., Zhang, F., Zhang, X., Yu, Z., Wang, L., Zhang, Y., and Guo, B. hmcodetrans: Human-machine interac- tive code translation. IEEE Transactions on Software Engineering, 2024b.
Donovan, A. A. and Kernighan, B. W. The Go programming language. Addison- Addison-Wesley Professional, 2015.
Du, G., Li, J., Liu, H., Jiang, R., Yu, S., Guo, Y., Goh, S. K., and Tang, H.-K. Knowledge fusion by evolving weights of language models. In Find- ings of the Association for Computational Linguistics: ACL 2024, 2024. doi: 10. 10.18653/v1/22244fmmll 698. URL https:/ /t//aalaante hww.aaanttoooo..ogg//000.00 Andings-acc- 698/.
Lozhkov, A., Li. R., Allal, L. B.. Cassano, F,, Lamy-Poiriess J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., Liu, T., Tian, M., Kocetkov, D., Zucker, A., Belkada, Y., Wang,
Z., Liu, Q., Abulkhanov, D., Paul, I., Li, Z., Li, W.-D., Risdal, M., Li, J., Zhu, J., Zhuo, T. Y., Zheltomozhskii, E., Dade, N. O., Yu, W., Krauß, L., Jain, N., Su, Y., He, X., Dey, M., Abati, E., Chai, Y., Muennighoff, N., Tang, X., Oblokulov, M., Akiki, C., Marone, M., Mou, C., Mishra, M., Gu, A., Hui, B., Dao, T., Zebaze, A., Dehaene, O., Patry, N., Xu, C., McAAley. J., Hu, H., Scholak, T., Pa- quet, S., Robinson, J., Anderson, C. J., Chapados, N., Patwary, M., Tajbakhsh, N., Jernite, Y., Ferrandis, C. M., Zhang, L.. Hughes, S.. Wolf, T., Guha, A., von Werra. L., and de Vries, H. Starcoder and the stack v2: The next generation. arXiv preprint 20X.x:2402.191777 2024.
Fan, Z., Gao, X., Mirchev, M., Roychoudhury. A., and Tan, S. H. Automated repair of programs from large language models. In Proceedings of the 2023 IEEE/ACM 45th International Conference Software Engineering (ICSE), pp. 1469-1488 2023.
Günther, M., Ong, J., Mohr, I., Abdessalem, A., Abel, T., Akram, M. K., Guzman, S., Mastrapas, G., Sturua, S., Wang, B., Werk, M., Wang, N., and Xiao, H. Jina embed- dings 2: 8192-token general- purpose text embeddings for long documents. arXiv preprint arXiv:2310 /0. 19923, 2024.
Macedo, M., Tian, Y., Cogo, F., and Adams, B. Exploring the impact of the output format on the evaluation of large language models for code translation. In Proceedings of
10

---

Function-to-Style Guidance of LLMs for Code Translation
the 2024 IEEE/ACM First International Conference on Al Foundation Models and Software Engineering (FORGE), pp. 57-68, 2024.
unsupervised code translation. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
Martin, R. C. Clean code: a handbook of agile software craftsmanship. Pearson Education, 2009.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 1rXiv:1707.063444 2017.
McKeeman, W. M. Differential testing for software. Digital Technical Journal, pp. 100-107, 1998.
Shi, Z. and Zhou, Y. Topic-selective graph network for topic-focused summarization. In Advances in Knowledge Discovery and Data Mining 27th Pacific-Asia Confer- ence on Knowledge Discovery and Data Mining, PAKDD 2023, Osaka, Japan, May 25-28, 2023, Proceedings, Part IV. pp. 247-259, 2023.
Mossienko, M. Automated cobol to java recycling. In Proceedings of Seventh European Conference Soft- ware Maintenance and Reengineering (CSMR), pp. 40- 50, 2003.
Nguyen, A. T., Nguyen, T. T., and Nguyen, T. N. Lexical statistical machine translation for language migration. In Proceedings of the 2013 9th Joint Meeting on Founda- tions of Software Engineering (FSE), pp. 651-654, 2013.
Sparck Jones, K. A statistical interpretation of term speci- ficity and its application in retrieval. Journal of documen- lation, 1972.
Paaßen, B. Revisiting the tree edit distance and its back- tracing: A tutorial. arXiv preprint arXiv:1805.06630 2018.
Sun, T., Chai, L., Yang, J., Yin, Y., Guo, H., Liu, J., Wang, B., Yang, L.,, and Li, Z. Unicoder: Scaling code large language model via universal code. arXiv preprint arXiv:2406. 16441, 2024.
Pan, R., Ibrahimzada, A. R., Krishna, R., Sankar, D., Wassi, L. P., Merler, M., Sobolev, B., Pavuluri, R., Sinha, S., and Jabbarvand, R. Lost in translation: A study of bugs introduced by large language models while translating code. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE), pp- 1-13, 2024.
Szafraniec, M., Roziere, B., Leather, H., Charton, F., La- batut, P., and Synnaeve, G. Code translation with com- piler representations, arXiv preprint wrXiv:2207.03578 2022.
Tao, Q., Yu, T., Gu, X., and Shen, B. Unraveling the poten- tial of large language models in code translation: How far are we? arXiv preprint arXiv:2410.09812, 2024.
Puri, R., Kung, D. S., Janssen, G., Zhang, W., Domeni- coni, G., Zolotov, V., Dolby, J., Chen, J., Choudhury, M., Decker, I.,, et al. Codenet: A large-scale for code dataset for learning a diversity of coding tasks. arXiv preprint arXiv:2105.
Wang, Y., Wang, W., Joty, S., and Hoi, S. C. Codet5: Identifier-awarr unified pre- pre-trained encoder-decoder mod- els for code understanding and generation. arXiv preprint 1rXhv:2109.00853 2021.
2021.
Qwen, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J. Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M.. Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report. arXiv preprint 241.:2212211111 2025.
Weisz, J. D., Muller, M., Ross, I., Martinez, F., Houde, S., Agarwal, M., Talamadupula, K., and Richards, J. Better together? evaluation of ai-supported code translation. In Proceedings of the 27th International Conference on Intelligent User Interfaces, pp. 369-391, 2022.
Xie, Y., Naik, A., Fried, D., and Rose, C. Data augmentation for code translation with comparable corpora and multiple references. arXiv preprint
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct preference optimization: Your language model is secretly reward model. arXiv preprint arXiv:2305. / 18290, 2024.
2023.
Xu, R., Wang, Z., Fan, R.-Z,, and Liu, P. Benchmark- ing benchmark leakage in large language models. arXiv preprint arXiv:2404. 18824, 2024.
Ristad, E. S. and Yianilos, N. Learning string-edit dis- tance. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 1998.
Xue, M., Andrzejak, A., and Leuther, M. An interpretable error correction method for enhancing code-to-coe to-code trans- lation. Proceedings of the International Conference on Learning Representations (ICLR), 2024.
Rozière, B., Zhang, J., Charton, F., Harman, M., Synnaeve, G., and Lample, G. Leveraging automated unit tests for

---

Function-to-Style Guidance of LLMs for Code Translation
Yan, W., Liu, H., Wang, Y., Li, Y., Chen, Q. Wang, W., Lin, T., Thao, W., Zhu, L.., Sundaram, H., et al. Codescope: An execution-based multilingual multitask multidimensional benchmark for evaluating Iims on code understanding and generation. arXiv preprint arXiv:231 2023a.
Yan, W., Tian, Y., Li, Y., Chen, Q., and Wang, W. Code- transoceam: A comprehensive multilingual benchmark for code translation. arXiv preprint arXiv:2310.0495. 2023b.
Yang, Z.,, Liu, F., Yu, Z., Keung. W., Li, J., Liu, S., Hong, Y., Ma, X., Jin, Z., and Li, G. Exploring and unleash- ing the power of large language models in automated code translation. In Proceedings of the ACM on Software Engineering, pp. 1585-1608, 2024.
Yin, X.. Ni, C., Nguyen, T. N., Wang, S., and Yang, X. Rectifier: Code translation with corrector via llms. arXiv preprint arXXv:2407.07477 2024.
Yuan, Z., Chen, W., Wang, H., Yu, K., Peng, X., and Lou, Y. Transagent: An llm-based multi-agent system for code translation. arXiv preprint arXiv:2409. 19894, 2024.
Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wu, and Wang, G. Instruc- tion tuning for large language models: A survey. arXiv preprint arXiv:2308. 107999 2024.
Zhong, H., Thummalapenta,
Xie, T., Zhang, L., and Wang, Q. Mining upi mapping for language migration. In Proceedings of the 32nd ACM/IEEE International Con- ference Software Engineering (ICSE), pp. 195-204, 2010.
Zhu, M., Jain, A., Suresh, K., Ravindran, R., Tipirneni, S., and Reddy, C. K. Xleost: benchmark dataset for cross-lingual code intelligence. arXiv preprint NxXiv:2006.084744 2022.
12

---

Function-to-Style Guidance of LLMs for Code Translation
A. Background
CCSim. CCSim, proposed by Li et al. (2024), measures stylistic similarity of code by considering the edit distances (Ristad Yianilos, 1998) of variable naming, API invocation, and code structure.
Calculation of Variable Name Edit Distance Disvar- First, extract all variable names, V1 and V/2 from the two code respectively. Then, compute the edit distance between these two sets of variables follows:
Disv, =
E eEVi
A min ED my)
wyeV2
Disv
A min ED(,, 15) Wev,
(6)
||112 eVVV2
Disv, + Disv2
Disvar
2
where ED is the Edit Distance (Ristad & Yianilos, 1998), and X is the normalized inverse document frequency (IDF) (Sparck Jones, 1972) of variable naming ve, which is used decrease the impact of common words.
Calculation of API Invocation Edit Distance - -Dssapi It is calculated similarly to Dis except that variable names are replaced with API names.
Calculation of Code Structure Edit Distance Disserus The measurement code structure is based on the Tree Edit Distance (TED) (Paaßen, 2018) of abstract syntax tree. Specifically, measures the structural difference of two code snippets by determining the fewest insertions, deletions, and replacements needed to transform one tree into the other.
Based on the edit distances of variable names, API invocation, and code structure described above, CCSim measures the stylistic similarity between two code snippets as follows:
Disvar + Disapi + Disstru
CSSim
3
where CSSim, Disyar, Dis Disstru [0, 1], and higher CCSim values indicate greater similarity.
Computational Accuracy. We utilize Computational Accuracy (CA) to assess the functional correctness of code translations produced by various models. Given all source code, their corresponding target code generated by the models, and the input data [(src), (NPPT.))) (sTeN, tgtN, INPUTN)), CA is calculated as follows:
CA E ca (STCk. tgtx)
N
(8)
1, if exec (srck, imputt exec (tglx; imput), Vinput INPUT
ca (srck, tgtx)=
0, otherwise
where exec(.) denotes the result of executing the code with a given input.
B. More Implementation Details.
In the function-oriented training, we construct approximately 5,000 code pairs for each translation scenario, such as translating from C++ to Python, with a corresponding scale of 10,000 in the style-oriented training. Throughout both training stages, we maintain consistent hyperparameters, employing 2 epochs and learning rate of x 10-5. During inference, we set the temperature of the LLMs 0.7. All our experiments are carried out on a machine equipped with eight NVIDIA 8000-XXM4-80GB GPUs.
C. Baseline Details
We adopt Qwenz2n and GPT-4 our baselines, implementing four previously established prompt learning strategies. The prompts are shown in Appendix F.2.
Direct prompt learning (Direct), This straightforward strategy (Yang et al., 2024) provides the LLM with prompt that includes the source language, target language, source code, and concise task description. Owing to the powerful
13

---

Function-to-Style Guidance of LLMs for Code Translation
Translation C + {
Translation C++
Translation Go
(
Translation Java + ()
Translation Py {
Method
LIM
Avg.
C++
Go
Java
Go
Java
Py
C
C++
Java
C++
Go
Py
C++
Go
Java
Direct
90.3 57.1 81.6
62.5
73.6
54.6
71.0
5688
79.2
83.1
80.4
61.7
78.1
75.3
49.9
68.6
70.0
71.0
61.5
78.5
70.5
CoT
85.9 61.9
76.6
63.1
74.5
51.9
76.3
62.2
74,7
83,4
81.2
66.3
75.7
77.0
45.0
70.7
68.4
75.6
65.3
82.6
70,9
Qwentag
RAG
87.3 69.5
58.2
83.5
82.2
60.8
74.5
74.5
76.1
74.2
Self-debog
91.2 64.9
83.2
67.4
82.5
58.6
78.1
64.4
82.2
85.1
82.6
68.2
79.4
82.5
55.8
77.3
73.6
79.7
69.8
84.3
75.5
Direct
95.5 64.5
90.2
65.6
87.9
63.5
85.5
66.6
87.5
91.4
88.1
77.4
84.1
82.2
57.4
71.9
71.5
76.0
58.9
76.6
77.1
CoT
92.6 70.7
83.7
62.8
85.7
71.0
85.3
92.6
84.2
85.4
54.5
72.9
70.6
77.7
64.8
78.0
GPT-4
RAG
93.5 72.1
91.3
70.1
84.4
67.9
83.9
74.1
85.6
91.5
85.9
84.3
85.1
84,9
64.2
76.2
77.1
82.9
66.8
84.7
80.3
Self-dehug
96.9
70.2
75.0
89.2
87.1
75.1
88.9
93.4
89.1
84.7
85.6
90.2
64.6
82.3
78.3
81.3
65.1
85.0
82.0
Qweng SM
96.2
76.0
87.5
77.3
87.0
73.6
80.3
72.1
80.5
84.7
84.2
81.4
78.9
81.2
68.9
80.7
64.9
70.0
60.6
71.9
77.9
Qwen
97.3
83.8
91.6
82.7
90.2
82.8
87.1
78.9
87.2
90.3
90.1
87.6
85,6
86.0
79.0
90.3
76.3
80.0
73.7
85.4
85.3
F2STRANS (Ours)
Qwentg
91.9
85.2
89.6
81.2
92.1
89.5
81.6
90.8
79.2
27.4
87.4
Qwenzy
97.7
88.4
94.6
84.9
93.0
86.8
89.9
85.4
89.5
93.6
93.2
91.0
89.8
90.4
83.7
93.5
81.2
85.8
80.1
89.1
StarCoderag
97.5
86.
92.4
84.8
91.2
84.9
88.9
82.1
88.8
91.6
90.9
88. 1
86.0
89.1
80.6
90.8
78.0
83.0
76.8
85.9
86.9
Table 5: Code translation results of various models on XcodeEval benchmarks.
79
87
62
78
61
72
77
71
76
85
59
75
58
70
74
84
0.0 0.2 044 0.6 088 1.0
0.0
0.2
04
0.6
0.8
10
0.0
0.2 0.4
0.6
0.8
1.0
0.0 0.2
0.4
0.6
088 10
(a) CodeNet Benchmark (Qwen
(b)) CodeNet Benchmark
(c) F2SThans Benchmerk (Qwenn
(d) F2STrans Benchmark (StarCodkr..)
(StarCooer..))
Figure 9: The impact of trade-off hyper- pper-parameters 33 in the loss function Eq. 5 on the results.
instruction- following abilities of LLMs, direct prompt learning has proven highly effective across a broad range of tasks (Du et al., 2024; Shi Zhou, 2023).
Chain of thought (CoT). This strategy, proposed by Yan al. (2023b), first encourages the model to thoughtfully
consider the translation process and identify potential challenges before undertaking code translation. This strategy usually achieves better results than direct prompt learning (Lee et al., 2024).
Retrieval-augee augmented generation (RAG). Consistent with previous RAG-based code translation approaches (Bhattarai et al., 2024a;b), we retrieve the most similar source-target pair from our style-oriented positive translation dataset using the code embedding model Jina. This retrieved pair provides auxiliary context for the model.
Self-debug prompt learning (Self-debug). This method (Yang et al., 2024; Pan et al., 2024) employs LLMs to generate initial translations and test cases, verifies the translated code's correctness using these tests, and subsequently corrects any errors based on the test results and compiler error messages.
D. Additional Results
We further evaluate F2STRANS on xCodeEval (Khan et al., 2024), as shown in Table 5. We can find that F2STRANS continues to demonstrate a significant advantage. Notably, F2STRANS enables Qweno.5B to outperform both QwenggB and GPT-4 on average across 20 code translation tasks. Qwen1.5B even surpasses the self-debugging GPT-4.
E. Additional Discussion
The 8 in the Loss Function Eq. 5 of Style Learning. Figure illustrates the performance of the model with different trade-off hyperparameters B. It is evident that, during the style learning, the weight of the instruction fine-tuning loss Lia increases, the model's performance improves. The best performance is achieved when 3 z 0.6, after which the performance gradually declines. Furthermore, when 3 = 1.0 (i.e., using only Liaa), the model consistently outperforms the case when = 0.0 (i.c., using only Lit). This is because Liss takes both positive and negative data into account, whereas Lnn considers only the positive data.
14

---

Function-to-Style Guidance of LLMs for Code Translation
F. Prompt Settings
F.1. Prompts Used by F2STRANS
Ludee Prompt.
You are given source code in [SOURCE.LANG] and a translated code in TTARGET.LANG] Please evaluate the translation by scoring it on scale from 1 to 5. where:
1: The translated code has significant differences in logic, structure, implementation compared to the source code, and would likely work as intended.
The translated code works, but there are noticeable differences in logic, style, or structure that deviate from the original solution.
3: The translated code is mostly similar to the source code but has minor differences optimizations that do not impact overall functionality.
The translated code is very close to the original code, minor, non-critical deviations in style or structure.
5: The translated code is highly consistent with the source code, both terms of logic and structure, and works as intended.
### {SOURCE, LANG) Code:
(SOURCE,COM) CCDEE
### (TARGET LANG) Code:
(TARGET CODE)
#illi Score:
Style-aware Prompt.
Translate the following (SOURCE LANG) code to (TARGET LANG] while preserving the source code style, including variable names, function names, and code structure. Adhere the following guidelines:
Variable and Function Names:
Maintain the same variable and function names as the source code.
If necessary due to language constraints, adjust names minimally while keeping them similar to the original.
2. Code Structure:
Preserve the overall structure and logic flow of the code.
Maintain same control structures (c.g., loops, conditionals) and their nesting levels.
Libraries and APIs:
Replace source language libraries and functions with equivalent [TARGET LANG) libraries and functions.
Keep the variable and parameter
the same in the source code where possible.
Comments:
Retain any comments present in the source code.
Translate comments to (TARGET LANG) | if applicable, maintaining their position and style.
Code Formatting:
Maintain similar code formatting style the code, including indentation, spacing, and line breaks, much possible within conventions (TARGET LANG).
Print only the translated (TARGET LANG) code and end with the comment "End of Code".
###f Source code:
[SOURCE.CODE]
Prompts P in Eq. and During the Training and Inference of F2STRANS.
Translate the (SOURCE.LANG) code (TARGET code.
### ESOURCE_LANG, Code;
(SOURCE_CODE)
### [TARGET LANG) Code:
15

---

Function-to-Style Guidance of LLMs for Code Translation
F.2. Prompts Used by Baselines
Direct Prompt Learning.
Translate following (SOURCE.LANG) code (TARGET Print only the (TARGET.LANG) code with the comment "End of Code". ### Source code: (SOURCF_CODE)
Chain-of of-thought Prompt Learning.
First, understand the functionality of following (SOURCE LANG] code predict execution output. Then, translate the (SOURCE.LAN) code into (TARGET while maintaining the same functionality, ensuring that the translated code can be successfully executed. ###f Souree code: (SOURCE.CODE)
Retrieval Augmented Generation Prompt Learning.
Translate the following [SAURCELLANG] code to STARGETLLANG, (TARGET Print only the (TARGET LANG) code and end with the comment "End of Code".
###I Source code:
(SOURCE CODE EXAMPLE)
#iHl Target code:
### Souree code:
(SOURCE CODE}
#illi Target code:
Self-debug Prompt Learning When Effect is COMPILE ERROR or RUNTIME ERROR.
You were asked to translate the following [SOURCE.LANC) code to LANG): (SOURCE, [CDE)
Your response the following [LARGET_LANG] code:
(TRANSLATED CODE)
Executing your generated code gives the following error because it syntactically incorrect:
(STDERR)
Can you re-generate your response and translate the above (SOURCE.LAN) LANG) code (TARGET Print only the (TARGET LANG) code and not add any other natural language description your output. Make sure your generated code syntactically correct.
Self-debug Prompt Learning When Effect is INCORRECT OUTPUT.
You were asked translate the following (SOURCE.LANG, code to TTARGETLANN) {SOURCE, .ODEC
Your response was following (TARGET_LANN code:
(TRANSLATED .CDDE}
Executing your generated code gives the following output:
[GENERATED OUTPUT)
instead of the following expected output:
(EXPECTED OUTPUT)
Can you re-generate your response and translate the above (SOURCE.LANG) code to (TARGET Print only the (TARGET LANG) code and do not add other natural language description in your output. Make sure your generated code is syntactically correct. Your generated (TTARET code should take the following input and generate the expected output:
Input:
(TEST INPUT)
Expected Output:
TEXPECTED.OUTPUT
16


Reviews:
[{"review_id": "3NUMTWL8Xz", "reviewer": "Reviewer_p79m", "summary": "Authors present F2STRANS, a novel function-to-style guiding paradigm that enhances the code translation performance of smaller LLMs by addressing functional correctness and stylistic readability. \n\nAuthors propose a 2-stage learning approach: \n1) The functional learning stage is based on instruction fine tuning(FT) using good quality source-target code pairs collected by authors from online programming platforms (codeforces). This step optimizes for code translation correctness by FT a base LLM. The quality of code pairs is maintained by (a) Qwen-7B as LLM judge (b) testing. \n2) The style learning stage improves code readability by training model fine-tuned in stage 1 using constrastive learning (positive and negative style examples). The positive example is generated by Qwen-32B while negative style translations are generated from model fine-tuned in stage 1. \n\nAuthors present extensive evaluations which shows that F2STRANS significantly improves code translation performance across benchmarks for 4 different prompting approaches based on FT smaller Qwen and starcoder models and comparing these against larger Qwen-32B and GPT-4. \n\nAuthors also introduces a new larger code translation benchmark F2STRANS.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful and valuable comments. \nWe respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns. \nIf our response meets your expectations, we would greatly appreciate it if you could consider raising your score (currently Rating: 3: Weak Accept). \nIf further concerns remain, please let us know, and we are committed to addressing them and refining our submission accordingly.\n\n\n>**Q1:**\nHeavy dependence on the availability of larger LLMs to judge translations and generate correctly styled translations.\n\n\n**A1:**\nUsing larger LLMs to build training data can indeed enhance model performance. \nHowever, we would like to clarify two key points:  \n\n(1) **While it is important to acknowledge the limitations of this approach, we should not overlook its significant benefits.**\n\nFor instance, our training framework enables Qwen-1.5B to outperform GPT-4 across a wide range of code translation benchmarks.\nThe specific results are as follows:\n\n||CodeNet|F2STrans|xCodeEval|CodeScope|\n|-|:-:|:-:|:-:|:-:|\n|Qwen-32B| 72.9 | 53.9 |  70.5 | 58.4 |\n|GPT-4| 75.3 | 63.6 | 77.1 | 64.6 |\n|Ours (Qwen-1.5B)| **85.1** |  **69.8** |  **85.3** | **73.9** |\n\n\n(2) **Even when we use LLMs of the same scale as the trained model to construct training data, our two-stage training method remains effective.**\n\n\nHere, we trained Qwen-3B exclusively on data constructed entirely by itself. \nThe results are presented in the table below.\nWe find that, without relying on larger LLMs, our framework still enables Qwen-3B to surpass GPT-4 across these code translation benchmarks.\n\n\n||CodeNet|F2STrans|xCodeEval|CodeScope|\n|-|:-:|:-:|:-:|:-:|\n|Qwen-32B| 72.9 | 53.9 |  70.5 | 58.4 |\n|GPT-4| 75.3 | 63.6 | 77.1 | 64.6 |\n|Ours (Judge-**Qwen-3B**, Positive Translation Generation-**Qwen-3B**, Model Training-**Qwen-3B**)| **85.5** | **70.3** | **85.7** | **75.1** |\n\n---\n\n>**Q2:**\nCould you please explain how the negative-style translations are generated using $M_{fun}$ in stage 2 along with prompting techniques used. \nHow did you validate that there is sufficient variability/diversity in the generated negative style samples and that they are different from one another.\n\n\n**A2:**\nWe set $M_{fun}$'s temperature to 1.5 and perform multiple sampling generations with the prompt described in line 825 of our paper. \nA translation is retained only if its CCsim score with any prior translation is **below 0.9**; otherwise, it is discarded.\nCCSim, as detailed in lines 606 to 633 of the paper, measures stylistic similarity between code snippets, and a threshold of 0.9 ensures that the retained samples are sufficiently distinct from one another.\nIf fewer than the desired number of translations are retained after 20 sampling attempts (a rare scenario), we incrementally increase the temperature to encourage greater diversity in the $M_{fun}$'s outputs.\nThe table below illustrates the average similarity between our constructed negative-style translations, measured using two metrics: CCSim and CodeBLEU.\n\n|         | C    | C++  | Go   | Java | Python | Average |\n|---------|:------:|:------:|:------:|:------:|:--------:|:---------:|\n| CCSim   | 82.4 | 84.3 | 77.5 | 81.7 | 73.6   | 79.9    |\n| CodeBLUE| 78.1 | 79.7 | 72.1 | 76.4 | 68.7   | 75.0    |\n\n\n\n\nWe further evaluated the performance of the trained Qwen-1.5B model on various benchmarks using different CCsim filtering thresholds.\nThe results shown in the following table indicate that omitting negative translation filtering—disregarding the diversity of negative translations—led to a noticeable decline in performance. \nMoreover, setting the threshold too low (e.g., 0.8) also impaired performance.\nThis may be because an excessively low threshold introduces some highly diverse but low-quality negative translations.\n\n\n| Benchmark       | 1.0 (w/o Negative Translation Filtering) | 0.95   | 0.9    | 0.85   | 0.8    |\n|-----------------|:------------------------------------:|:------:|:------:|:------:|:------:|\n| CodeNet     |                84.0                   |  84.5  | **85.1** | **85.1** |  84.9  |\n| F2STrans    |               68.9                  |  69.3  | **69.8** |  69.7  |  69.5  |\n| xCodeEval   |               84.6                  |  84.8  | 85.3 | **85.4**  |  85.1  |\n| CodeScope   |               73.1                  |  73.5  | **73.9** |  73.7  |  73.4  |", "claims_and_evidence": "Key technical Innovations claimed include: \n1. Relevance-driven code pair selection: \n(a) Use of Jina code embedding model & LLM judge to assess solution consistency \n(b) Differential testing ensures code pairs exhibit identical input-output behavior\n2. Style consensus selection: \na) Leverages CSSim metric to ensure stylistic similarity among candidates translations. \n3. New code translation benchmark dataset with 5 programming languages. \n\nMain results claimed include: \n1. F2STRANS improves translation performance on both the new (fs2trans) and existing benchmarks (codenet). F2STRANS enables smaller Qwen and starcoder models to outperform Qwen32B and GPT-4 on several language pairs. \n2. Ablation studies demonstrate that functional and style learning stages contribute substantially to performance gains. F2STRANS shows improved effectiveness in correcting various types of errors in base LLM translations compared to function guidance alone.\n3. Evaluation using CCSim metric shows F2STRANS enhances stylistic quality of translated code. \n4. Experiments with multilingual training strategies indicate that cross-lingual training enhances the performance of F2STRANS.", "methods_and_evaluation_criteria": "Yes . Authors use (i) a new larger benchmark (ii) CCSim to evaluate the stylistic similarity (iii) Comparison with stronger LLMs QWEN-32B and GPT-4. (iv) Ablation Studies to attribute the contribution of different components of FS2TRANS. (v) Analysis of types of errors corrected by F2STRANS.", "theoretical_claims": "None", "experimental_designs_or_analyses": "NIL", "relation_to_broader_scientific_literature": "Broadly, this work makes a good contribution in the code translation space, by building upon ideas from literature and contributing a new benchmark.", "essential_references_not_discussed": "NIL", "other_strengths_and_weaknesses": "- Weakness: Heavy dependence on the availability of larger LLMs to judge translations and generate correctly styled translations.", "other_comments_or_suggestions": "None", "questions_for_authors": "- Could you please explain how the negative-style translations are generated using Mfun in stage 2 along with prompting techniques used. \nHow did you validate that there is sufficient variability/diversity in the generated negative style samples and that they are different from one another.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kEl81u3C0W", "reviewer": "Reviewer_Lczm", "summary": "Previous work in code translation has focused on improving the performance of LLM-based code translation by focusing on multilingual training and various test-time inference strategies. In this paper, the authors hypothesize that optimizing for program correctness and program readability can improve the performance of LLMs on code translation tasks. Specifically, the authors mine codeforces to collect a dataset of cross-lingual code pairs that are functionally consistent and present a contrastive learning framework to optimize for coding style. The authors find that (1) evaluating on their curated benchmark reduces mean average score by about 10% highlighting greater complexity (2) ablating the style-consistent training and the functional consistency checks significantly reduce the overall effectiveness of the model, (3) for many tasks, a 0.5B parameter Quen model outperforms GPT4 model.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the contributions of our work and providing valuable feedback.\nWe respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns. \nIf so, we would deeply appreciate it if you could raise your score (currently Rating: 2: Weak reject). \nIf not, please let us know your concerns, and we will continue actively responding to your comments and improving our submission.\n\nBefore addressing your specific points, we would like to clarify that the \"CodeEval\" mentioned in the review likely refers to the \"CodeNet\" benchmark.\n>**Q1:**\nIn addition to the CodeNet and F2STrans benchmarks, the paper should evaluate performance on a broader range of benchmarks, such as CodeTransOcean.\n\n**A1:** \nBesides the CodeNet and F2STrans benchmarks, we also report results on xCodeEval and CodeScope benchmarks in **Table 5 of Appendix D**.\nHere, we show more results on CodeTransOcean and Multiple-E benchmarks.\nFor CodeTransOcean, we adopt the CodeBLEU metric, consistent with the standard practice in the CodeTransOcean paper.\n\n||xCodeEval|CodeScope|Multiple-E|CodeTransOcean\n-|-|-|-|-\nQwen32B|70.5|58.4|59.9|27.5\nGPT4|77.1|64.6|68.0|29.0\nOurs (Qwen1.5B)|**85.3**|**73.9**|**74.2**|**29.7**\n---\n>**Q2:**\nThe paper lacks evaluation of code quality.\n\n**A2:**\nIn **Table 4 of Section 3.5**, we assess the stylistic quality of translations based on the CCSim metric.\nThe results are as follows:\n\n||CSSim\n|-|:-:|\nQwen32B|76.4\nGPT4|77.9\nOurs (Qwen1.5B)|**82.0**\n---\n>**Q3:**\nThe paper uses Qwen7B and Qwen32B for LLM Judge and Style Aware LLM Translation, respectively. It does not analyze the impact of inherent biases in these LLMs on the model training.\n\n**A3:**\nWe build training datasets based on different LLMs (Qwen-7B/32B and DeepSeek-6.7B/33B) and show the training results of Qwen1.5B based on F2STrans benchmark:\n\nLLMs Used by Judge and Style Aware Translation|C|C++|Go|Java|Py|Avg.\n-|:-:|:-:|:-:|:-:|:-:|:-:|\nQwen-(7B/32B)|81.0|69.2|75.1|83.6|40.0|69.8\nDeepSeek-(6.7B/33B)|80.4|67.1|77.3|82.2|43.4|70.1\nHybrid Mining|**81.3**|**70.5**|**77.8**|**84.1**|**43.6**|**71.5**\n\nWe find that the inherent biases of LLMs cause uneven performance across languages.\nBy adopting a **Hybrid Mining** strategy—using Qwen LLMs for C, C++, and Java, and DeepSeek LLMs for Go and Python—we achieved consistent performance improvements. \nThis demonstrates that assigning tasks according to each model's strengths can alleviate the impact of LLMs’ inherent biases and improve the quality of training data.\n\n---\n>**Q4:**\nSome non-trivial insights into the overall performance decline on the F2STrans benchmark are required.\n\n**A4:**\nWe have identified two main reasons for the overall performance decline:\n\n- **Number of Test Cases.**\nF2STrans includes 50 human-annotated test cases for each code snippet (see Table 1), whereas CodeNet provides only one.\nThe scarcity of test cases in CodeNet may lead to inflated evaluation results, introducing evaluation errors.\nTo quantify this impact, we conducted 10 code translation evaluations, each using a random test case from F2STrans samples.\nThe results, shown below, reveal that relying on a single test case introduces at least a 2.4% evaluation error.\n\n||1 Case|50 Cases|Evaluation Error\n|-|:-:|:-:|:-:|\nQwen32B|56.7|53.9|3.8\nGPT4|66.0|63.6|2.4\nOurs (Qwen3B)|76.1|73.7|2.4\n\n- **Code Length.**\nAs shown below, source code in F2STrans usually contains more tokens than in CodeNet, which makes evaluations more challenging.\n\n||C|C++|Go|Java|Py\n|-|:-:|:-:|:-:|:-:|:-:|\nCodeNet|298.5|264.8|300.6|256.0|91.3\nF2STrans|397.3|368.7|350.6|315.2|216.4\nDelta|98.8|103.9|50.0|59.2|125.1\n---\n>**Q5:**\nThe Python translation performance of various models shows a significant decline in the F2STrans benchmark, yet the paper lacks an analysis.\n\n**A5:**\nWe find that Python code in our F2STrans benchmark is more complex, characterized by longer average length and more frequent use of built-in APIs, such as \"map\" and \"lambda\".\nAs a result, models are more prone to generating code with compilation errors when translating these complex examples.  \nFor instance, over half of the errors produced by Qwen32B are due to compilation failures, as follows:\n\n||CodeNet|F2STrans\n-|:-:|:-:|\n#Token|91.3|216.4\n#API|4.3|7.5\nCompilation Error Ratio|21.4|51.8\n\n---\n>**Q6:**\nPresenting concrete numbers for the mean average success rate delta, both with and without Python, can help verify the benchmark advantages of F2STrans.\n\n**A6:**\nThe table below shows the mean average success rate delta for each language between the CodeNet and F2STrans benchmarks.\nEach model's performance drops by at least 11.7 on F2STrans compared to CodeNet, and by at least 6.0 even without Python.\nThis confirms that our F2STrans benchmark is highly challenging.\n||C|C++|Go|Java|Py|Avg.|Avg. w/o Py\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\nQwen32B|12.9|26.6|8.7|4.0|42.8|19.0|13.0\nGPT4|5.8|13.6|2.6|1.9|34.4|11.7|6.0\nOurs (Qwen1.5B)|6.4|17.8|5.9|1.7|44.5|15.3|8.0", "claims_and_evidence": "Generally yes. I think the authors have presented reasonable evidence that supports their claims. Specifically, I looked at these claims:\n - **Claim 1:** F2STrans is an effective benchmark for code translation.\n - Evidence 1: The mean average success rate of models drops 10% from CodeNet to F2STrans. \nComments: The performance of Python translation drops signficiantly from CodeEval to FS2Trans. The authors mention that this could be because python is an interpreted language but a deeper answer would be useful here. what is the mean average success rate ignoring Python translation tasks?\n - **Claim 2**: Functional consistency training is beneficial to code translation.\n - Evidence 2: An ablation experiment shows that, without functional consistency training, translation success rate drops slightly.\n - **Claim 3:** Style-oriented training is beneficial to code translation.\n - Evidence 3: An ablation showcases that model accuracy suffers without style oriented training.", "methods_and_evaluation_criteria": "This paper is generally missing a comparison with the CodeTransOcean benchmark. Evaluating how the trained methods performed on this dataset would be very informative since the CodeEval seems to be sourcing data from the same underlying data distribution as F2STrans (codeforces).\n\nFurthermore, I find it concerning that the data mining algorithm for both the functional consistency and the style-guidance seem to rely heavily on having an LLM as a critic (Quen-7B as the \"LLM Judge\" and Quen-32B as the \"Style Aware LLM\") but there was no evaluation done as to how the inherent biases of these models affects the final performance of a model training on the F2STrans benchmark.", "theoretical_claims": "none.", "experimental_designs_or_analyses": "For Table 2:\n> L258R: As shown in Table 2(II), all evaluated models achieve scores that are at least 10 points lower on average in our benchmark compared to CodeNet, highlighting the greater complexity of our proposed benchmark\n\nThis seems to be the only quantitative evidence in the main paper that argues for whether FS2Trans is a better benchmark than CodeNet. This is a major claim which deserves a finer-grained analysis than the current analysis presented. Some recommendations:\n - Please present concrete numbers of the mean average success rate delta, ideally by each language category.\n - It seems that Python translation tasks have an outsized influence on the aggregate number presented. If we remove Python, what does the delta difference look like? \n\n> L258R: ... weaker performance on Python...\n\nIts pretty interesting that Python translation success rate decreases so much between the datasets. The reasoning presented in the paper is that this is because of the \"interpreted nature [of Python]\". I'm not sure this argument holds up very well. If this argument were true, CodeEval results should also have been in the 25%-55% range. Yet, on CodeEval, the average performance is between 70%-90%.  I think a deeper qualitative analysis would be beneficial here, such as a qualitative analysis of the code and a study of what kinds of failures the models fell into on CodeEval and on F2STrans.\n\n__Overall__: For Table 2, I agree that the overall reduction in success rate is interesting. However, there are many other non-trivial insights we can take from this experiment that are not presented. A deeper analysis is required.\n\nAdditionally, both CodeEval and FS2Trans source their data from codeforces. I do not think the code-writing style on codeforces is a good indicator of how well code translation would work \"in the wild.\" For instance, we generally modularize and structure code in Java around certain object oriented classes while in C we modularize and structure code around shared procedures. I think evaluating and comparing on a code translation dataset that sources it's data from other sources would be very useful here. (eg: CodeTransOcean but the authors might know other ones as well).\n\n\n\n> While the effectiveness of LLMs ... most models face two critical limitations .. (i) Correctness [and] (ii) Readability\n\nTable 2 and related experiments seem to only measure the correctness of the generated code against a dataset. How is code quality improved by the presented approach? How can this be measured quantitatively? Even some qualitative examples of how the code correctness and readability changes before and after training would be beneficial.", "relation_to_broader_scientific_literature": "Just in terms of the benchmark, I don't think the results with the style guidance and functional consistency are particularly surprising. I definitely see this benchmark complementing other preexisting benchmarks but I'm not sure what additional non-trivial insight a user would gain by benchmarking their model on CodeEval as compared to F2STrans, other than the fact that their model will be slightly worse on FS2Trans. I believe this can be solved by improving the presentation of the paper to motivate the hypothesis. a bit more. For instance, by showcasing a non-trivial tasks where function consistency and style-oriented training truly brings a large difference in performance.\nIn terms of the algorithmic insight, the training methodology and loss functions do not seem to be novel contributions.", "essential_references_not_discussed": ".", "other_strengths_and_weaknesses": "- The notation for denoting LLM prompts $\\mathcal{P}(\\cdot)$ looks very similar to $P(\\cdot)$. It might be better to express these as *serializations* of various variables. e.g: instead of using $\\mathcal{P}(src),tgt_{<i}$ to denote the \"prompt\" for translating source code with the text generated so far, we can instead denote it as $\\langle \\text{src} \\rangle;\\langle\\text{tgt}_{<I}\\rangle$ to denote the serialization of the source code concatenated with the target code generated so far.\n\n - > L63: Relying solely on the inherent capabilities of LLMs to overcome these issues is only a short-term solution.\n    - I can agree with this statement but more justification for this statement (in what use cases? Under what constraints?) would be useful.\n\n - In the related work section, it would be beneficial to comment on how the proposed pipeline relates to each of the sub-fields mentioned.", "other_comments_or_suggestions": "Overall, I'm leaning towards Rejection (Weak Rejection). While F2STrans presents an interesting hypothesis (that style consistency and functional consistency are important for code translation), the supporting analysis for the experimental observations is inadequate to support these claims. I'm willing to raise my score after discussion with the authors.\n\n----\nAfter discussions with the Authors, I'm increasing my score to Weak Accept.", "questions_for_authors": "(merged with other sections)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NvjXBb7f9h", "reviewer": "Reviewer_a86m", "summary": "This paper proposes F2STrans, an approach for improving code-to-code translation by optimizing large language models (LLMs). The main idea of this paper is to provide fine-tuning processes that instruct knowledge on good code and bad code in terms of two criteria: Functional Learning and Style Learning. For functional learning, which requires LLM to generate translated code with a high level of functional correctness, this approach designs a paradigm called Function-oriented Guidance by two steps: Function-consistent Data Construction using code embedding and ranking candidate code using LLMs and Functional Learning, which uses Instruction Fine-tuning to return the optimized LLM for code translation. For Style learning, this approach uses strong LLMs to generate positive translation examples with stylistic similarity scores to find the best example and the same process to find negative translation examples. Next, it uses a lit-wise loss function to instruct LLM to generate code with better style. In the experiment, this work uses CodeNet and their proposed benchmark called F2STrans for evaluation. Their results show that F2STrans can provide better Computational Accuracy and stylistic accuracy compared to state-of-the-art LLMs like GPT-4 and Qwen-32B.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful and valuable comments. \n\n>**Q1:**\nFurther study is needed to analyze the case when LLM judges provide low-quality output.\n\n**A1:**\nWe observe that LLM Judge is prone to errors when evaluating source code that is either lengthy or contains numerous built-in APIs, e.g., \"map\", \"lambda\", etc., in Python. \n\nTo evaluate its performance in assessing translation quality, we introduce an **LLM Judge benchmark**.\nThis benchmark comprises source code and human-annotated translation from our F2STrans benchmark, along with translations generated by Qwen-32B and GPT-3.5.\nWe measure the performance of a Qwen-7B-based LLM Judge by its success rate in identifying the human-annotated translation as the best translation for a given source code.\nThe table below summarizes results for source code with varying API counts and token lengths.\n\nWhile performance deteriorates with increased complexity, our analysis indicates that LLM Judge still reliably identifies high-quality translations in most scenarios.\n\n#API|[0,3]|[4,7]|[8,11]|Overall\n-|-|-|-|-\nResults|83.4|75.6|61.3|77.5\n\n#Token|[0,250]|[251,500]|[501,750]|[750, $+\\infty$]|Overall\n-|-|-|-|-|-\nResults|84.3|79.6|64.6|46.3|77.5\n\n---\n\n>**Q2:**\nWhy were style and functional correctness chosen as two aspects for improvement?\n\n**A2:**\nWe focused on style and functional correctness for two main reasons:\n1. **Fundamental requirements**: Functional correctness ensures the translated code maintains the same behavior as the source code—an essential requirement for any translation system. \nStylistic correctness enhances readability and maintainability, making the translated code easier to understand and modify.\n2. **Translation-specific factors**: While efficiency, security, and other aspects are certainly important in code development, these factors are largely determined by the source code itself rather than the translation process.\n\nWe are actively exploring ways to optimize code translation quality from a broader perspective in our ongoing research.\n\n---\n\n>**Q3:**\nThe research on code preferences (CodeUltraFeedback) should be discussed.\n\n**A3:**\nWe agree that the CodeUltraFeedback is highly relevant to our work. \nIt inspires us to explore incorporating user-defined code styles and preferences into our code translation system. \nAdditionally, it illuminates important future challenges beyond our current scope, particularly how to extend our approach to address other crucial non-functional factors like efficiency, security, and maintainability in the translated code. \nOur paper will incorporate a more detailed discussion of CodeUltraFeedback and its implications for our research.\n\n---\n\n>**Q4:**\nClarification on I/O optimization in motivation example.\n\n**A4:** \nIn the C++ code, the lines\n```cpp\nios::sync_with_stdio(false);\ncin.tie(0);\n```\ndisable synchronization between C++ I/O streams and untie cin from cout, reducing overhead and accelerating input/output operations.\nIn Python, a comparable optimization would involve using faster input methods, such as ```sys.stdin.readline()``` or reading all input at once via ```sys.stdin.read()```.\n\n---\n\n>**Q5:**\nThe paper lacks a detailed description of the process for manually annotating the ground-truth dataset.\n\n**A5:**\nOur annotation team consisted of 30 professional software developers with industry experience, 15 graduate students, and 5 doctoral students in computer science.\nDuring code translation, each source code was translated by at least 5 different annotators, ensuring multiple candidate translations. \nDoctoral students then selected the best translation or refined problematic ones.\nFor test case annotation, we maintained a 95% branch coverage rate, prioritizing edge cases and challenging scenarios.\nThe entire annotation process took approximately 5 months.\n\n---\n\n>**Q6:**\nLack of comparison with non-LLM approaches, like CodeRosetta.\n\n**A6:**\nWe appreciate the suggestion to compare our work with non-LLM approaches for a more comprehensive evaluation.\nWhile CodeRosetta focuses on translations between C++ and CUDA/Fortran (which differs from our scope), we will include a more detailed discussion of it in our paper.\nBelow, we compare our approach with two prominent non-LLM code translation systems, TransCode-ST and TransCode-IR.\n\nCodenet Benchmark|C++↔Go|C++↔Java|C++↔Py|Go↔Java|Java↔Py\n-|-|-|-|-|-\nTransCoder-ST|-|66.3|72.7|-|71.2\nTransCoder-IR|54.3|69.4|-|50.6|-\nOurs (Qwen-0.5B)|77.9|84.9|78.2|77.3|80.6\n\n\nF2STrans Benchmark|C++↔Go|C++↔Java|C++↔Py|Go↔Java|Java↔Py\n-|-|-|-|-|-\nTransCoder-ST|-|50.2|26.3|-|32.6\nTransCoder-IR|41.7|53.2|-|54.3|-\nOurs (Qwen-0.5B)|67.0|72.3|44.1|70.6|53.4\n\n---\n\n>**Q7:**\nDid authors try other well-known code embedding methods, like UniXcoder?\n\n**A7:**\nThe training results of Qwen-1.5B using various embedding models are as follows:\n| |Size|CodeNet|F2STrans\n-|-|-|-\nBGE|110M|84.3|68.8\nUniXcoder|123M|85.2|70.1\nJina-embeddings|161M|85.1|69.8\nCodeXEmbed|400M|85.4|70.5", "claims_and_evidence": "The claims made in the submission are clear and convincing.", "methods_and_evaluation_criteria": "The evaluation criteria are clear and convincing, with the benchmark constructed and the evaluation metrics. For the proposed method, It is good to know the correctness of Judge LLM used in Function Learning and how it can impact the accuracy of code translation. In Table 3, the authors reported the accuracy without LLM Judge and it underperforming the default configuration. However, the authors need to analyze the case when the judge returns bad results. Other parts of the Methods are clear and convincing.", "theoretical_claims": "This paper proofs that the existing dataset for evaluating LLM-based code translation like CodeNet is somehow outdated. They claim that their proposed dataset can capture better the complexity and nature of LLM-based code translation. Through the experiment, they successfully proved that claim since the accuracy of multiple configurations of their tool for code translation tends to be lower with their constructed benchmark.", "experimental_designs_or_analyses": "Experiment designs and analysis are sound and valid.", "relation_to_broader_scientific_literature": "This paper proposed a new direction of improving LLM-based code translation with a focus on important aspects of translated code as functional correctness and stylistic information. Their results show that with proper improvement, simple open LLMs can still outperform large scale open LLM and closed LLMs. They also contributed to the science community a dataset that will be helpful in evaluating code translation.", "essential_references_not_discussed": "I suggest the authors explain why they chose stylistic and functional correctness as two aspects for improvement. They should add papers about studies on code preferences (CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences - Wessay et al.) to describe the potential/ challenges of applying new aspects to their approach.", "other_strengths_and_weaknesses": "Strength:\n- This paper is well-written and has a good structure to follow.\n- The strategy of providing better samples for the fine-tuning process is a good direction. \n- The selections of loss functions for functional learning and style-oriented learning were based on well-known prior works.\n- The result has an acceptable level of outperformance compared to existing works.\n- Integrating manually annotated ground-truth translation. \n\nWeakness: \n- Unclear motivation example: Can you define which part of the translated code should include the code for I/O optimization?\n- For Function-consistent Data Construction, there is a risk that the LLM Judge can provide incorrect output. Further study is needed to analyze the case when LLM judges provide low-quality output.\n- While human annotation for ground-truth dataset creation is important, there is are lack of description of this process in the current version of this paper.", "other_comments_or_suggestions": "Lack of comparison with non-LLM approaches. Authors can consider comparing their work with transformer-based approaches such as CodeRosetta (CodeRosetta: Pushing the Boundaries of Unsupervised Code Translation for Parallel Programming).", "questions_for_authors": "1. The scale of the manual ground-truth translation creation  (mentioned in section 3.1). How many people are required to do this process? How do authors ensure that human responses were correct?\n\n2.Code Pair embedding. Did authors try other well-known code embedding methods, such as UniXcoder (Guo et al. - UniXcoder: Unified Cross-Modal Pre-training for Code Representation)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9HRCtOMPrj", "reviewer": "Reviewer_stzs", "summary": "This paper studies code translation. Different from previous settings where researchers only care about the functional correctness of translated codes, the authors in this paper also care the functional consistency. They propose a method to handle this by splitting it into a two-stage training framework, function learning and style learning. Besides existing benchmarks, the authors also propose a more up-to-date benchmark for code translation. Experimental results demonstrate the effectiveness of their proposed ;earning framework and extensive ablation studies explore the impact of different design choices.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's insightful and constructive feedback, and we have carefully addressed each point in our response to resolve your concerns.\nIf our response has satisfactorily addressed your questions, we kindly request your consideration of raising the score (currently Rating: 3: Weak Accept).\nShould any further issues remain, please feel free to share your additional comments, and we will continue actively responding to your comments and improving our submission.\n\n>**Q1:**\nComparison between DPO and list-wise loss function.\n\n**A1:**\nThe list-wise loss function is designed to improve the generation of positive translations by learning from one positive example and multiple negative examples.\nThis approach differs significantly from DPO, which optimizes models based on pairwise comparisons between a single positive and a single negative sample.\n\nTo evaluate the effectiveness of these two methods, we conduct experiments on the CodeNet and F2STrans benchmarks, with the results presented in **Figure 8 of Appendix E** in our paper. \nThe average performance across three training runs is summarized in the following table, from which we can observe that our list-wise loss function is more effective.\nFor more details on the experimental setup and additional analyses, please refer to **Figure 8 of Appendix E** in our paper.\n\n|Dataset|Our Models|DPO|List-wise Loss|Delta|\n|-|-|-|-|-|\n|CodeNet|Qwen-0.5B|78.2|78.9|+0.7|\n||StarCoder-3B|85.5|86.7|+1.2|\n|F2STrans|Qwen-0.5B|61.1|62.1|+1.0|\n||StarCoder-3B|71.4|72.9|+1.5|\n\n---\n\n>**Q2:**\nIn addition to the CodeNet and F2STrans benchmarks, the paper should evaluate performance on a broader range of benchmarks, such as Multiple-E.\n\n**A2:**\nWe fully recognize the importance of comprehensive testing benchmarks.\nTherefore, **Table 5 of Appendix D** in our paper demonstrates the model's performance on the **xCodeEval** and **CodeScope** benchmarks.\nThe average translation results for various source languages ​​are as follows:\n\n|xCodeEval|C|C++|Go|Java|Py|Avg.|\n|-|-|-|-|-|-|-|\n|Direct (Qwen-32B)|72.9|65.5|76.1|68.0|70.3|70.5|\n|CoT (Qwen-32B)|71.9|66.2|76.4|67.1|73.0|70.9|\n|RAG (Qwen-32B)|75.8|68.8|78.1|72.6|75.6|74.2|\n|Self-debug (Qwen-32B)|76.7|70.9|79.5|73.8|76.9|75.5|\n|Direct (GPT4)|79.0|75.9|86.1|73.9|70.8|77.1|\n|CoT (GPT4)|78.8|75.8|86.8|74.2|74.3|78.0|\n|RAG (GPT4)|81.8|77.6|86.8|77.6|77.9|80.3|\n|Self-debug (GPT4)|83.1|79.9|89.0|80.7|77.4|82.0|\n|Ours (Qwen-0.5B)|84.2|78.2|82.7|77.4|66.8|77.9|\n|Ours (Qwen-1.5B)|88.8|84.8|88.8|85.2|78.8|85.3|\n|Ours (Qwen-3B)|90.4|87.0|90.3|87.1|82.2|87.4|\n|Ours (Qwen-7B)|**91.4**|**88.8**|**91.8**|**89.3**|**84.1**|**89.1**|\n|Ours (StarCoder-3B)|90.2|86.8|89.8|86.6|80.9|86.9|\n\n\n|CodeScope|C|C++|Go|Java|Py|Avg.\n|-|-|-|-|-|-|-\n|Direct (Qwen-32B)|59.4|56.0|67.4|60.7|48.6|58.4\n|CoT (Qwen-32B)|67.6|53.5|65.3|60.1|52.8|59.9\n|RAG (Qwen-32B)|75.4|58.8|65.7|67.8|59.7|65.5\n|Self-debug (Qwen-32B)|75.3|61.4|71.8|67.9|60.7|67.4\n|Direct (GPT4)|69.3|64.4|77.8|62.1|49.3|64.6\n|CoT (GPT4)|74.6|65.6|75.9|68.2|55.0|67.9\n|RAG (GPT4)|80.1|68.2|77.4|73.5|59.8|71.8\n|Self-debug (GPT4)|80.8|70.5|81.8|74.8|61.2|73.8\n|Ours (Qwen-0.5B)|75.4|69.4|75.8|67.9|45.7|66.8\n|Ours (Qwen-1.5B)|84.5|71.9|78.2|76.8|58.2|73.9\n|Ours (Qwen-3B)|88.6|75.3|80.7|82.5|65.5|78.5\n|Ours (Qwen-7B)|**92.8**|**86.6**|82.6|**85.3**|**68.5**|**83.2**|\n|Ours (StarCoder-3B)|87.3|78.1|**83.2**|78.9|62.5|78.0\n\n\nIn response to your suggestion, we have conducted additional tests on the **Multiple-E** and **CodeTransOcean** benchmarks.\nFor Multiple-E, since it does not include C code, we focus on evaluating its performance across other languages.\nFor CodeTransOcean, we adopt the CodeBLEU metric, consistent with the standard practice described in the CodeTransOcean paper.\nFor other benchmarks, we continue to use the Computational Accuracy metric as outlined in our paper.\n\n|Multiple-E|C++|Go|Java|Py|Avg.\n|-|-|-|-|-|-\n|Direct (Qwen-32B)|58.6|64.8|72.0|44.3|59.9\n|CoT (Qwen-32B)|61.8|65.8|77.0|47.3|63.0\n|RAG (Qwen-32B)|62.9|69.4|77.5|49.1|64.7\n|Direct (GPT4)|71.2|75.9|78.4|46.5|68.0\n|CoT (GPT4)|71.6|77.6|79.1|48.8|69.3\n|Self-debug (GPT4)|73.5|77.2|81.6|52.3|71.1\n|Ours (Qwen-0.5B)|71.7|71.9|81.7|40.0|66.3\n|Ours (Qwen-1.5B)|79.3|77.9|89.5|50.0|74.2\n|Ours (Qwen-3B)|82.6|80.6|92.4|54.7|77.6\n|Ours (Qwen-7B)|**84.6**|**83.6**|**93.3**|**58.2**|**79.9**|\n|Ours (StarCoder-3B)|82.3|80.9|92.1|52.9|77.1\n\n|CodeTransOcean|C|C++|Go|Java|Py|Avg.\n|-|-|-|-|-|-|-\n|Direct (Qwen-32B)|27.6|27.0|27.1|31.6|24.1|27.5\n|CoT (Qwen-32B)|27.3|26.5|27.5|32.1|25.2|27.7\n|RAG (Qwen-32B)|28.2|27.4|28.1|32.4|24.7|28.2\n|Direct (GPT4)|29.1|29.0|28.4|32.9|25.7|29.0\n|CoT (GPT4)|29.2|29.3|28.6|32.2|26.2|29.1\n|Ours (Qwen-0.5B)|30.3|29.0|28.4|30.8|25.7|28.9\n|Ours (Qwen-1.5B)|30.8|29.7|29.8|32.3|25.8|29.7\n|Ours (Qwen-3B)|**31.2**|29.9|**30.9**|33.2|26.7|30.4\n|Ours (Qwen-7B)|**31.2**|**30.6**|30.4|**34.3**|**27.1**|**30.7**\n|Ours (StarCoder-3B)|30.8|29.8|30.0|32.8|26.6|30.0", "claims_and_evidence": "Yes. Experimental results demonstrate that style learning benefits code translation.", "methods_and_evaluation_criteria": "In style learning, the authors use a list-wise loss function to train LLMs to favor style-consistent target codes. The motivation of using such a loss function is not clear. This preference could also be learned through reinforcement learning (e.g DPO). The authors should study the impact of replacing the list-wise loss function with RL algorithms in their ablation studies.\n\nAt least for me, the performance of list-wise loss doe not have to be better than DPO. This is because I consider these loss functions as designing variations of the framework. In the future, there may be more advanced loss functions to achieve even better performance.", "theoretical_claims": "There are no theoretical claims in this paper.", "experimental_designs_or_analyses": "There are other benchmarks that could be used for code translation, other than CodeNet, though not specifically designed for code translation. For example, the Multiple-E [1]. The author should also apply their framework on such benchmarks.\n\n[1] Multi-Programming Language Evaluation of Large Language Models of Code", "relation_to_broader_scientific_literature": "This work is specifically designed for code translation.", "essential_references_not_discussed": "N.A", "other_strengths_and_weaknesses": "N.A", "other_comments_or_suggestions": "N.A", "questions_for_authors": "See my comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper gives a new approach to LLM-based code translation. The method operates in two stages. First, instruction finetuning is used to translate code into a functionally correct form in the target language. Second, a contrastive learning method converts this functionally correct translated code into stylistically idiomatic code. The paper evaluates the method rigorously and also introduces a new benchmark.

The reviewers were generally in favor of accepting the paper. There were a few questions (most notably about alternative baselines and analysis), but the reviewers have answered the questions diligently and provided additional experimental evidence in their rebuttal. Given this, I am recommending acceptance. Please incorporate the reviewers' feedback and the new results carefully in the final version.
Decision: Accept (poster)