On-Policy Optimization with Group Equivalent Preference for Multi-Programming Language Understanding
Authors: Haoyuan Wu
Rui Ming
Jilong Gao
Hangyu Zhao
Xueyi Chen
Yikai Yang
Haisheng Zheng
Zhuolun He
Bei Yu
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Reinforcement Learning
Multi-Programming Language Understanding
Primary area: deep_learning

Abstract: Large language models (LLMs) achieve remarkable performance in code generation tasks.
However, a significant performance disparity persists between popular programming languages (e.g., Python, C++) and others. 
To address this capability gap, we leverage the code translation task to train LLMs, thereby facilitating the transfer of coding proficiency across diverse programming languages.
Moreover, we introduce OORL for training, a novel reinforcement learning (RL) framework that integrates on-policy and off-policy strategies.
Within OORL, on-policy RL is applied during code translation, guided by a rule-based reward signal derived from unit tests.
Complementing this coarse-grained rule-based reward, we propose Group Equivalent Preference Optimization (GEPO), a novel preference optimization method.
Specifically, GEPO trains the LLM using intermediate representations (IRs) groups.
LLMs can be guided to discern IRs equivalent to the source code from inequivalent ones, while also utilizing signals about the mutual equivalence between IRs within the group.
This process allows LLMs to capture nuanced aspects of code functionality.
By employing OORL for training with code translation tasks, LLMs improve their recognition of code functionality and their understanding of the relationships between code implemented in different languages. 
Extensive experiments demonstrate that our OORL for LLMs training with code translation tasks achieves significant performance improvements on code benchmarks across multiple programming languages.

On-Policy Optimization with Group Equivalent Preference for Multi-Programming Language Understanding
Haoyuan Wa
Rui Ming' Jilong Gao Hangyu Zhao'
Xueyi Chen'
Yikai Yang'
Haisheng Zheng" Zhuolun He Bei Yu²
The Chinese University of Hong Kong
"ChtEDDA Tech
(hywu24, byu) @cse. cwwk.edu.... hk
Abstract
Large language models (LLMs) achieve remarkable performance code genera- tion tasks. However, a significant performance disparity persists between popular programming languages (e.g.. Python, C++) and others. To address this capability gap, we leverage the code translation task to train LLMs, thereby facilitating the transfer of coding proficiency across diverse programming languages. Moreover, we introduce OORL for training, novel reinforcement learning (RL) framework that integrates on-policy and off-policy strategies. Within OORL, on on-policy RL is applied during code translation, guided by rule-based reward signal derived from unit tests. Complementing this coarse- coarse-grained aule-based reward, we propose Group Equivalent Preference Optimization (GEPO), novel preference optimiza- tion method. Specifically, GEPO trains the LLM using intermediate representations (IRs) groups. LLMs can be guided to discern IRs equivalent to the source code from inequivalent ones, while also utilizing signals about the mutual equivalence be- tween IRs within the group. This process allows LLMs capture nuanced aspects of code functionality. By employing OORL for training with code translation tasks, LLMs improve their recognition of code functionality and their understanding of the relationships between code implemented in different languages. Extensive experiments demonstrate that our OORL for LLMs training with code translation tasks achieves significant performance improvements on code benchmarks across multiple programming languages.
Introduction
With the rapid advancement of LLMs [2], code-specific language models have garnered significant attention within the research community. Building upon pre-trained LLMs, code LLMs such as the StarCoder series [12, 16], CodeLlama series [23], DeepSeekCoder series [27]. QwenCoder [20] series, and CodeStral [18] have demonstrated superior performance in code generation tasks. However, although current state-of-the-art code generation LLMs excel at generating code using popular programming languages (e.g., Python, C++), their performance diminishes when applied to solving the same problems using other programming languages [19]. This capability gap limits the universal applicability of these powerful LLMs and hinders developers working in diverse programming language ecosystems.
Consequently, it is crucial to enhance LLMs' proficiency across a wider range of programming languages to match their performance in widely used ones [26, 19]. We address this challenge by training LLMS on code translation tasks, requiring them to translate code from one programming
'Equal Contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

language to another accurately. Intuitively, if LLMs can accurately translate Python code into other programming languages, they can achieve comparable performance in Python code generation tasks in those programming languages. Moreover, focusing on code translation facilitates the L.M's understanding of inter-language similarities and differences and allows it to learn from exemplary code structures.
Recently, on-policy RL algorithms [24, 1, 25, 14, 9] utilizing rule-based rewards have demonstrated significant performance in code generation tasks [7, 6]. However, the inherent coarse-grained nature of these rule-based rewards, lacking process-level supervision, limits their effectiveness in guiding LLMs to recognize the functional equivalence of intermediate code components during training. Implementing fine-graine tne-grained process-leve rewards is challenging, particularly given that multiple valid implementations may exist for the same code functionality. Given that leveraging preference data can guide L.M behavior towards desired outcomes, preference optimization [21, 3] offers an opportunity to incorporate process-level constraints during training.
Functional equivalence crucial during the training process for code translation tasks. High- level programming languages can be too abstract for effective functional understanding by LLMs, especially for less common ones [26]. Consequently, interlingual intermediate representations (IRs) can be leveraged to facilitate cross-language transfer from high- to low(er)-resource programming languages [19, 10, 26]. Compiler IRs are agnostic to the source programming language and target execution platform, providing a method to align constructs from different programming languages semantically [5]. Therefore, we construct groups of IRs for the preference optimization process. By guiding the L.M to distinguish IRs equivalent to the source code from inequivalent ones and utilizing signals about the mutual equivalence between IRs within the group, the LLM can capture fine-grained information regarding code functionality.
This study introduces OORL, a novel RL framework integrating on-policy and off-policy strategies for training. First, we employ on-policy RL with a binary, rule-based reward signal. This component ensures the fundamental correctness of the code translation, verifying that the generated code adheres to formatting requirements, compiles successfully, and passes predefined unit tests. This provides a strong. unambiguous signal for task completion. Second, we introduce group equivalent preference optimization (GEPO) to incorporate finer-grained quality distinctions and address the unique nature of code generation, where multiple equivalences can exist. GEPO is a novel preference optimization method that extends beyond traditional pairwise comparisons. It leverages preference data comparing groups of "winner" (equivalent IRs) and "loser" (inequivalent IRs) responses. Crucially, GEPO explicitly models the concept of functional equivalence within the winner group, guiding the model to recognize that multiple distinct IRs can be equally valid. This allows the model to learn nuanced aspects of code functionality across multiple programming languages.
Our contributions can be summarized as follows:
Introduce OORL, a novel RL framework for training with code translation tasks to enhance LLMs' understanding across multiple programming languages.
Develop GEPO, novel preference optimization method that extends beyond pairwise comparisons by explicitly modeling equivalence within IRs groups
Conduct extensive evaluations demonstrating the superior performance of our methods across various code benchmarks using multiple programming languages.
2 Preliminaries
2.1 RL with Explicit Reward Function
RL [24, 7] has been widely used in preference optimization for LLMs. Let मit denote an LLM obtained after supervised fine-tuning (SFT). RL optimizes the policy model ne typically initialized from wee with parameters 9, by maximizing an expected reward signal. This requires defining a reward function R(z, y) that assigns a scalar score to a complete response y given the prompt æ. This score reflects the accuracy, quality, or alignment of the response with human preferences.
Once the reward function R(x, y) is established, an RL algorithm can be employed to optimize the policy model mg. The objective typically to maximize the expected reward, often incorporating KL divergence penalty term to prevent Ty from deviating too far from the initial reference policy

---

01
Variance Constraint
y
y
Yw
Palicy Model
02
Unit
72
<oon + LoEPo
y
y
V1
Group Preference
On
In
Figure I: Overview of the OORL integrating on-policy RL with the rule-based reward and on-policy preference optimization (GEPO).
model Fref, thereby maintaining generative capabilities and diversity:
max Ez- D prang. CeiianeEyyrnn Eyrona [R(x,, y) 30Dxx. (<((=)][Tre((()))))
where D prompt is a dataset of prompts and is hyperparameter controlling the KL penalty.
2.2 Direct Preference Optimization
Direct Preference Optimization (DPO) [21] an alternative approach that leverages the preference dataset Pree directly optimize the policy model No without the need for explicitly training separate reward model. DPO establishes a direct link between preference probabilities and policy probabilities, deriving an objective equivalent to optimizing against learned reward model within the RLHF framework. Specifically, DPO assumes an implicit reward function (2 y) exists such that the human preference data can be modeled via the Bradley-Terry model:
Ply > 31|2) = (ro(2, yw) ro yt)),
where > denotes "is preferred over". Theoretical derivation shows that the RI. objective of aligning with this implicit reward (including the KI. penalty) can be transformed into a maximum likelihood objective computed directly on the preference data. The DPO loss function is defined as:
To (yw (r) Ted (yw
Te( )]
LDPO Feet) = - RE she (n) [log o (B log
,3 log Teer( (xr)
(3)
where is the policy model being optimized, net is the reference policy model, and [ is scaling factor for the implicit reward.
Methods
Integrating on-policy (Section 3.1) and off-policy (Section 3.2) strategies, as illustrated in Fig- ure introduce OORL (Section 3.3), novel RL framework for multi- programmin language understanding.
3.1 On-policy RL for Code Translation
Although LLMs achieve remarkable performance in code generation tasks, a significant performance disparity persists between high-resource programming languages (e.g., Python, C++) and others. To address this capability gap, we leverage the code translation task to train LLMs, thereby facilitating the transfer of coding proficiency across diverse programming languages. For example, if LLMs can accurately translate Python code into other programming languages, they can achieve comparable performance in Python code generation tasks while using those programming languages.
RL with Binary Code Translation Reward. Recently, on-policy RL algorithms utilizing rule-based rewards have demonstrated significant performance in code generation tasks [7. 6]. Consequently. we apply on-policy RL to training with code translation tasks. Let's first define the RL setup for the code translation task: (1) State (s): The state typically consists of input y with source code and potentially the sequence of tokens generated so far Oct with translated code; (2) Action (a): The
action corresponds to the generation of the next token o; by the policy model; (3) Policy (mg): The
policy model (als) ()) is the LLM being optimized, parameterized by 0. It's typically initialized from

---

eus seguentist integersjint connti cuccent pas
define doo local 132 Sour, sequett intogers 1232 nounder Scounc) local_unnane pun.redd entzy:
scrpa Tcmp ege cer..codd.ceemm.
Label
secucn cursent suM
foz. boo.y...eehceeerr
60 add new 132 tcomnt,
11 sas- uneg 132 10 133
detine des Local 532 bean, sequential, Integeze (132 noundet unneaed cdde
32 add new troent,
sex:
tail osl1 132 311mm 132 tcount, 132
15 Tehr M4,
10 zest 132 timax 133
16 tzunz
133
132
sdd new 137 tonasy
87 add 132 kcount, 56
12 cest 132 11 .33
12 edd 132 V7,
mal 40,
be label sfar.cond,claenn for.cond.eles cleanup
13,
testry 38,
ffo. preheadez
132 seureant Iemms
declare 132 311vs, moax 132 32, 1321
Source code Xg and corresponding function equivalenct IRs group Yw
Joun sequeatial Insegers (132 noundef 10001_ unnimed_4sdd
Bour. sequencial intagero nounder sesaii_anasad__ee
tail call 132 1llvm. anex, (132 scouns, 132 60 wds new 132 tonax,
tirax tail call 122 ww3.. tezunt, 132
13 rest neeg 137 kanax to 133
11 cest 132 10 .33 mal 40,
11 add naw 132 tazax, -1
nect
declare 132 313's,, mnax. 1 3232 1321
deotare 132 attvm, wwa.. 132 1329 F1
Inequivalenct IRs group Y, augmented from equivalanct IRs group
Figure 2: Overview of the equivalent and equivalent IRs groups for the training process with GEPO. The equivalent IRs group is constructed from different LLVM optimization levels (e.g.. -Oz, -03) of the source code. The inequivalent IRs group is augmented from the equivalent IRs group.
the SFT reference model Tet- During the on-policy RL process, given the query and the trajectory o, we employ a binary reward function, Raute o), assigned upon completion of the generation o:
if represents successful code translation from g. 0, otherwise.
P (q. 0)
Specifically, successful translation demonstrates that o must adhere to the standard format specified in q and the translated code extracted from can be successfully compiled and pass unit tests. The policy model Ty is optimized using an on-policy RL algorithm (e.g.. PPO, GRPO) to optimize Equation (1). The on-policy RL algorithm operates by iteratively sampling trajectories (g. o) from the current policy model Ty. For each action in each sampled trajectory, it estimates the advantage A,, which quantifies how much better the received reward R2 is compared to expected baseline. The policy parameters are then updated using clipped objective function designed to maximize the expected advantage while limiting the change in the म. each step. The on-policy RL loss can be defined as:
Loa - -) [min(r: ( Ar, clip(r:(mo)) 1 e, + A)),))
where (TT) [e(a))) the probability ratio between new and old policies. hyperparameter
Te. [a]s)
defining the clipping range. This rule-based RL component, focused on maximizing the binary success signal defined in Equation (4). provides strong, unambiguous signal for task completion, which is complemented by the fier-ggrain preference information incorporated through the GEPO loss, as described in the subsequent section.
3.2 Group Equivalent Preference Optimization with IRs
The inherent coarse-grained nature of rule-based rewards, which lack process-level supervision, limits their effecctivnness in guiding LLMs to recognize the functional equivalence of intermediate code components. Implementing fine- mme-grained process-level rewards is challenging, particularly as multiple valid implementations data can effectively guide LLM behavior towards desired outcomes, preference optimization [21, 3] offers promising avenue incorporate process-level constraints during training.
exist for the same code functionality. Given that leveraging preference
Functional equivalence is crucial code generation scenarios. However, the functions implemented with high-level programming languages can be too abstract for LLMs to understand, especially

---

for less common ones. In contrast, compiler IRs expose detailed and low-level operators that are significantly easier for LLMs to understand [26]. Consequently, we propose performing preference optimization using IR translation tasks, which require L.MMs to translate code written in high-level programming languages into IRs. This process facilitates the LLM's understanding of the code functionality expressed in the high-level languages. Furthermore, equívalent IRs resulting from different compiler optimization stages can guide LLMs in discerning the equivalent functionality of different code implementations.
However, traditional preference optimization methods [21, 3] typically rely on pairwise comparisons, which are insufficient in this context. Specifically, methods like DPO [21] do not account for the equivalence among IRs. Therefore, we introduce Group Equivalent Preference Optimization (GEPO) and construct groups sper with equivalent and inequivalent IRs for the preference optimization process. GEPO compares groups of IR responses and specifically incorporates the concept of equivalence within the group of preferred ("winner")) IR responses. By guiding the LLM to distinguish IRs equivalent to the source code from inequivalent ones and utilizing signals about the mutual equivalence between IRs within a group, GEPO enables the LLM to capture detailed information regarding code functionality.
For each prompt Fy with source code in gepner for GEPO, there exists group of "winner" responses group of "loser" responses Y; = {ut- ,... )(""} and Y/ E (Y.,. U Vi). Specifically, y represents functionally equivalent IRs while Y7 represents m inequivalent IRs. Following [21]. we assume an implicit reward function ( M/) underlies the preferences. Instead of comparing the individual winner IR response and the loser IR response, GEPO compares the average reward of Y, to the average reward of Y,. Furthermore, it explicitly enforces equivalence within Y. by constraining the variance of their rewards. The initial optimization problem for GEPO, formulated in terms of the implicit reward Te. can be defined as:
Yu = [yw. V .. ).
min LOEOO = - E (agi Mi) Desd Dog Dog ac 1 Σ
1 Σ
( Vin) -
ro D1
1=1
R=
s.t. Bixg Yw). (aa((r (Zgz Yw))] <E
Var(ro Yn)) = Σ
(ra("ig Vln)
Σ rolog p2.))
(6)
1=1
The primary objective term maximizes the probability that the average winner reward exceeds the average loser reward, modeled using the sigmoid function. The constraint bounds the variance of rewards within the winner group Yu by a small threshold e, encouraging To (Zgg viw) To (Z. vie) for all v Na E Y..
According to Appendix A, we can establish a direct relationship between the implicit reward function To and the optimal policy no being optimized, relative to the reference policy model Tee- Accordingly, the optimal policy model T² that maximizes Equation (1) can be given by T (Uglzz) x ( (x)) exp( (1 Hg)). This implies that the reward function can be expressed in terms of the policy probabilities, up to partition function Z(xg). as follows:
ro(Tig Ug) = llg (o(yg(zg) + 3 log ((xg), Tree (Uglzng)
where Te is the policy model to be optimized. Substituting this expression for TT Ua) into Equation (6), the partition function terms B log Z(x,)) cancel out within the reward differences and the variance calculation. This yields objective function that depends on the policy model ne, the reference policy model Tet, and the preference data (Ig, Yu, Yi). which can be formulated as follows:
33 Σ Fo vi 1)
min C'EEO = -E( Vi) Dame ever Dlog o n
Σ
Ao 6o vio)
m k=1
f n
n
s.t. Be Yw) n
Σ
(Fo(E ven
fo(agg w2u)) <
7
1=1
To (Vggl (Mr x
Fo(FggVb) Ug) = log
(8)
5

---

Here, fo(2g, Mg) represents the log- proo-abbbilltyy ratio proportional to the implicit reward score. The constraint now applies the variance of these pro-abbbbbyl ratios within the winner group. To solve this constrained optimization problem, we introduce Lagrange multiplier À 2 0 for the variance constraint, transforming it into an unconstrained objective. The GEPO loss can be formulated as:
m Fo(Øo vt 1fI)
GEPO Yo.
Vi)- Dog (B(
Fo Vin)
m
B
+ AB Y.. D gaa
Σ (Fo( Vin)
Σ
ro (5 )9
(9)
Finally, CIEPO consists of two terms. The first term is analogous to the DPO loss but operates on the average log- pog-pobbability ratios of the winner and loser groups, pushing the model to prefer winners over losers on average. The second term, weighted by the hyperparameter A, explicitly penalizes the variance of the bog-rbbabiylity ratios within the winner group Yw. Minimizing this term encourages ro("g Vg) to be similar for all winners yhe E Yu., thereby enforcing the desired equivalence. The hyperparameter À 1 controls the strength of this equivalence regularization relative to the main preference objective. By minimizing LGEPO, we directly optimize the policy model wy to enhance the likelihood of high-quality, functionally equivalent code translation (Yw) compared undesired ones (Yi), while simultancously promoting diversity and consistency among the preferred solutions.
3.3 OORL
Building upon the on-policy RL strategy and GEPO, we develop OORL, a novel RL. framework for training with code translation tasks, as shown in Figure 1. Specifically, following the on-policy nature used for the RL part, we periodically sample trajectories (4, o) using the current policy model Ty and compute the binary reward Ra,e(g, o). This allows the estimation of advantages A, and the computation of the on-policy RL objective (OmP(0), which drives the policy model towards successful task completion according to the defined rules. In parallel, we utilize the static preference dataset D geeds- = {(xgg Yu. } containing groups of winner and loser responses to calculate LGERO (To; aet,), which guides the policy model align with fine-grained function equivalences among desirable IRs.
The parameters 9 of тe are updated using combined signal derived from both objectives. A conceptual representation of the combined objective to be minimized during policy updates can be expressed as:
L w/ LoMP + Wgepo LGEPO:
(10)
where the non-negative wa and Wgepo serve as weights that balance the contribution of the rule-based task completion signal from RL against the nuanced quality and equivalence signal from GEPO. Specifically, we set wy 1 and Wgepo 0.01 for the training process with OORL.
Experiments
4.1 Experiment Settings
The training dataset for OORL is constructed separately. For on-policy RL, construct 2400 code translation problems with unit tests from the SYNTHETIC-I [17] and APPS [8] datasets. Specifically, this batch of training data includes translation from Python and C++ as source codes to Python, C++, Java, PHP, Bash, and JavaScript as target codes. Meanwhile, GEPO is performed with 9600 curated C-to-IR groups from the SLTrans dataset [19].
Training Data.
Implementation Details. In this paper, implement our OORL framework based on Qwen3- 8B [20]. Specifically, we employ REINFORECE++ [1, 9] as the on-policy RL algorithm with OORL. The training process entailed using cosine learning rate schedule with a warm-up ratio of 0.03, and the AdamW W [15] optimizer with a learning rate of 5 x 10-7, no weight decay, a batch size of 8, and sequence length of 8192 tokens. The models underwent instruction tuning for epoch using DeepSpeed-Zen stage2 with offload [22] on 4 A100 GPUs, each with 80G memory.
Baselines. Our Qwen3-8B-OORL. trained with the OORL framework, is compared against existing LLMs with comparable parameters. These include DS- V2-Lite- [27], Qwen2.5-Coder-7BB

---

Translation from Python
Translation from C++
Translation from Java
Translation from Go
P
10
Scaa
C++
Scab
Pythen Scala
Pythen Dast
Pythan
Naskel
Nasball
DS-Ceder-V--Lite-lnst
Qwen3-8B (Non-Thinking)
-wen3-88-OORL (Non-Thinking)
Figure 3: Code Translation Performance of LLMs on CrossPLEval.
Table 1: Statistics in the CrossPLEval Benchmark.
Programming Language
# Solution Files
Avg. Lines Code (LoC)
Proportion of Files (%)
F(Pyton
7.81
Python
119x10 10
19.94
30.36
C++
92x 10
35.49
23,47
Go
92x 10
47.60
23.47
Java
89 10
39.72
22.70
Total Weighted Avg.
x92 10
34.57
100.00
Avg. LoC the toal is weighted avenge. Proportions are based on the total 2920 flle.
Inst [11], and Qwen3-8B [20]. Specifically, the experiments with Qwen3-8B are conducted using its non-thinking mode.
Evaluation Benchmarks. To comprehensively evaluate proficiency in multi-programming languages understanding, utilize the MultiPL-E [4] benchmark and select eight mainstream languages for evaluation, including Python, C++, Java, PHP, TypeScript, C#, Bash, and JavaScript. In addition, multilingual code translation benchmark, for further evaluation. CrossPL.vve comprises 119 programming problems carefully selected from the TACO dataset [13], with details illustrated in Section 4.2. For evaluation, MultiPL-EE and CrossPLEval benchmarks.
develop CrossPLEval,
the pass @ rate the metric for
4.2 CrossPLEval
CrossPLEval is designed to assess the ability of models to translate a canonical solution of problem from a source language to a target language. Specifically, the model receives the function declaration and the canonical solution in the source language (c.g.. Python) as input and is tasked with generating the corresponding solution in the target language. To facilitate evaluation, the target language function declaration is also provided, which helps constrain function names and variable types.
The 119 core problems in CrossPLEval originate from the TACO dataset [13], initially defined in Python. Recognizing the non-trivial nature of developing high-quality, semantically aligned test data across languages, we undertook meticulous manual processing for these problems ensure data diversity and quality. This process involved two main efforts. First, concerning canonical solutions, while reference Python solutions were available for all 119 problems, we manually rewrote these into equivalent implementations in other mainstream programming languages. This yielded 89 solutions for Java, 92 for Go, and 92 for C++, resulting 273 high-quality translation task instances from Python to these three target languages. Second, to ensure rigorous evaluation of functional correctness, we manually rewrote the original test cases for each problem into ten different programming languages, including C++, Java, PHP, JavaScript, Bash, Scala, Go, TypeScript, C#. and Haskell. These multilingual test cases enable verification translation accuracy across broad linguistic spectrum. Through this construction process, the CrossPI.Eve benchmark comprises 3920 independent coding problems, each of which includes at least unit tests, thereby providing solid foundation for evaluation. The details are described in Table 1.
7

---

Table Performance of different LLMs on MultiPL-E [4].
Python
C++
Java
PHP
Bash
JS
TS
C#
Avg.
DS- DS-Coder-V2-Lite--ie 81.10
32.91
68.35
72.67
19.62
81.36
83.33
41.13
60.06
(ren2.5-Coder-7BBmmg
88.40
57.14
70.88
73.91
43.67
78.88
84.27
52.53
68.71
Qwen3-8B
82.60
71.42
70.25
50.31
36.07
83.22
84.27
42.41
65.07
Qwen3-8B-OORL
90.06
83.23
83,54
78.26
46.20
89.44
85.53
54.22
76.31
Table 3: Results of code translation compared with other LLMs on CrossPLEval.
Target Language
Model
Python
C++
Java
PHP
JS
Bash
Scala
Go
TS
C#
Haskell
Avg.
DS-Coder-VV V2-Lite-Inst
41.17
46.21 46.21
57.14
27.73 32.77 34.45
53.78
21.00
11.76
37.22
Owen2.5-Codde--7B.
36.13
33.61
47.89
52,94
18.54
20.16
29.41
54.62
32.77
15.12
34.12
Py
Qwen3-8B
44.53
41.17
50.42
75.54
24.36
22.68
37.25
59.66
35.29
15.13
40.60
Qwen3-8B-OORL.
54.62
52.94
63.94
75.63
35.29
35.29
59.32
71.42
52.94
28.48
52.99
DS-Coder- Inss
73.91
71.73
75.00
72.82
33.69
42.39
45.65
69.56
20.65
20.65
52.61
Qwen2.5--Code-----iss
69.56
56.52
66.30
71.73
34.78
33.69
43.47
65.21
45.65
17.39
50.43
C++
Qwen3-8B
85.86
63.04
69.56
78.26
31.52
40.65
54.71
80.43
47.82
20.65
57.25
Qwen3-SB-OORL.
91.30
72.82
78.15
85.86
41.30
46.73
74.02
80.43
65.65
38.91
67.52
75.28
66.29
64.04
65.16
25.84
50.56
43.82
65.82
25.84
17.97
50.06
Qwen2.5-Codez-7B. BB-nntt
75.28
46.06
66.29
73.03
22.47
32.58
42.69
70.78
40.44
19.10
48.87
Java
Qwen3-8B
84,64
50.56
68.53
74.15
32.58
48.31
47.19
71.91
52.80
22.47
55.31
Qwen3-8B-OORL
92.13
71.91
75.28
78.65
41.34
58.42
68.53
76.40
76.40
28.53
66.76
DS-Coder- -2-Lite-Inst
78.26
40.21
57.60
53.26
71.73
32.60
32.60
61.95
20.65
17.39
46.63
Swen2..-Codee--BB
71.73
41,30
40.21
56.52
77.17
25.00
26.08
68.47
43.47
14.13
46.41
Go
Qwen3-8B
81.52
43.47
50.00
67.39
76.08
31.52
26.08
75.00
39.13
16.30
50.65
Qwen3-8B-OORL.
90.21
72.06
71.73
74,45
82.39
35.86 48.26
81.08
68.47
28.80
65.33
To illustrate the difficulty of CrossPLEval across different languages, visualize the code translation performance in Figure 3. Current SOTA code LLMs with around 8B parameters achieve low score on average performance across multiple programming languages.
4.3 Excellent Multi-Programming Language Understanding
To evaluate proficiency in understanding and generating code across multiple programming languages, we conduct evaluations using the established MultiPL-EE [4] benchmark and our newly introduced CrossPLEval benchmark. Besides Python, C++, Java, PHP, Bash, and JavaScript, we also include Scala, Go, TypeScript, C#, and Haskell in our evaluation, noting that the latter five languages are not included in the training dataset. Moreover, Bash, Scala, and Haskell are particularly noteworthy as low-resource programming languages.
Table 2 presents the code generation performance on MultPLLEE, while Table 3 and Figure 3 show code translation results on CrossPLEval. Across both benchmarks and all tested models, our Qwen3- 8B-OORL model consistently demonstrates superior performance. On MultiPL-E, it achieves the highest average score 76.31, significantly outperforming the second highest Qwen2.5-Coder-77 Inst with a significant margin. This strong performance extends across all eight languages evaluated in MultiPL-E, which are generally well-represented in large code corpora. This highlights the excellent ability to generate correct code from natural language instructions in these programming languages.
Furthermore, the CrossPLEval results demonstrate the code translation capabilities and, critically, its generalization ability. Qwen3-88 -aann------OLL.. achieves the highest average translation scores across all source languages evaluated. The CrossPLEval benchmark is particularly valuable for evaluating performance on languages potentially less represented in training data, such as Scala, Go, TypeScript, C#, and the low-resource Haskell. Although absolute translation scores into target languages like Bash, Scala, and Haskell are lower compared to more common targets, our Qwen3-8B-OORd exhibits significant relative improvements over the baselines on these more challenging targets. For instance, when translating from Python to Haskell, (ren3-8B-OORL scores 28.48, representing substantial

---

Table 4: Ablation studies on various benchmarks using OORL.
On- On-Policy RL Strategy
Off-Policy RL Strategy
CrossPLEval
MultiPL.-E
Python
C++
Java
Go
Avg.
65.06
40.60
57.25
55.32
50.65
50.96
REINFORCE +++ REINFORCE:++
73,60
47.56
63.26
62.13
60.87
58.46
Qwen3-8B
DPO GEPO
73.48
50.93
62.93
62.80
58.15
58.70
REINFORCE++
76.31
52.99
67.52
66.76
65.33
63.15
gain over the Qwen3-8B. These results underscore the effective generalization beyond its direct training data and its ability to provide significant performance boosts for tasks involving low-resource programming languages.
The evaluation results demonstrate that Swen3-8B-OORE not only excels in core code generation tasks on well-represented languages but also effectively translates and generalizes to improve perfor- mance untrained and low-resource programming languages, marking a notable advancement in multilingual code understanding and generation.
4.4 Ablation Studies
To evaluate the individual contribution of each key component within our OORL framework, we conducted thorough ablation studies. The results of these studies are detailed Table The investi- gation systematically deconstructed OORL by starting with the Qwen3-8B model and incrementally adding the on-policy RL strategy (REINFORCE++)) followed by a comparison different off policplly preference optimization strategies, specifically DPO [21] and our proposed GEPO.
On-Policy RL Strategy. As illustrated in Table 4, training with code translation tasks using the REINFORCE+ [9, 1] algorithm increases the MultiPL-E score 73.60 and the average CrossPL- Eval score to 58.46, demonstrating a significant performance improvement. This substantial gain demonstrates the effectiveness of on-policy RL, which directly refines nulti-programming language understanding of LLMs through the rule-based reward derived from unit tests.
Off-Policy RL Strategy. As shown Table 4, the combination of REINFORCE++ [9, 1] and DPO [21] only offers marginal improvement on CrossPLEval. The effectiveness of DPO in code understanding is found be limited. This limitation stems from the difficulty standard preference optimization methods face in fully understanding the functional equivalence between different code implementations. In contrast, our GEPO strategy is specifically designed to utilize functional equivalence information from IRs. It's worth noting that integrating GEPO with REINFORCE++ achieved significantly higher performance, reaching 76.31 on MultiPL-E and a notable average of 63.15 CrossPI.Eval This demonstrates the ability of our GEPO leverage functional equivalence, providing substantial additional performance gains over standard off-policy methods in the context of multi-programming language understanding.
5 Related Works
5.1 RL for LLMs
RL [24, 25, 21, 3] has emerged prominent paradigm for aligning LLMs with desired behaviors and preferences. Investigations in this area have explored both on-policy and off-policy RI. algorithms. On-policy methods [24, 1. 25, 14, have received considerable attention due to their inherent stability and effectiveness in directly optimizing the policy using data collected under the current policy, For instance, PPO [24] has been widely adopted for fine-tuning LLMs based on human preference data. Similarly, REINFORCE-
methods, such as ReMax [14], RLOO [1], and GRPO [25], have been extensively employed in extending mathematical and general reasoning abilities, often utilizing rule-based rewards exclusively. Concurrently, off-policy algorithms offer potential advantages in terms of sample efficiency by allowing the agent to learn from prior experiences or data generated by different policies. DPO [21] and IPO [3] have been proposed as more stable and efficient RL. techniques that implicitly learn reward function from preference data and directly optimize the policy. These methods can be conceptualized as implicitly performing off off-policy evaluation and

---

optimization. Building upon the successes of both on on-policy and off off-policy RL. we introduce OORL, novel RL framework that integrates on-policy and off-policy strategies for training. We also incorporate GEPO, novel off-policy method that leverages the concept of functional equivalence, to provide a more nuanced and effective approach for training L.MSs code translation tasks.
5.2 Code Understanding with IRs
Intermediate Representations (IRs) have been increasingly leveraged in recent work to enhance LLMs for code generation tasks. The umguage-agnostec nature of IRs allows LLMs abstract away from specific syntax and focus more effectively program logic. For instance, to improve multilingual code generation and cross-lingual transfer, IRCoder [19] employs continued pre-training of LLMs on SLTrans, parallel dataset of source code and its corresponding LLVM IR, thereby aligning code and IR semantics. Similarly, Transcoder-IR(26, also utilizes LLVM IR, augmenting L.M training by exploring it as pivot language. Acknowledging the potential complexities of standard compiler IRs for translation tasks, CoDist [10] proposes a custom, distilled code as simplified intermediate representation and translation pivot. Furthermore, for compiler-specific applications, LLM Compiler [5] is specialized through extensive further pre-training Code LLMs directly on large corpora of LLVM IR and assembly code. This specialization aims to enhance performance on tasks such as code optimization and disassembly. In this paper, we adopt a unique approach by leveraging LLVM IRs within our GEPO preference optimization process. We construct and compare groups of IRs derived from translated code. This method guides the LLM to discern fine-gratd -reggraieed functional equivalence, which is crucial for effective cross-lingual code understanding.
6 Conclusion
In this paper, we introduce OORL, a novel RL framework that integrates both on-policy and off-policy strategies for training LLMs. Within the OORL framework, on-policy RL applied during the code translation process, guided by rule-based reward signal derived from unit tests. Complementing this coarse-grt gearse-grained rule-based reward, we propose Group Equivalent Preference Optimization (GEPO), novel preference optimization method. GEPO specifically extends beyond traditional pairwise comparisons by explicitly modeling equivalence within groups of IRs. Extensive experiments demonstrate that our OORL framework achieves significant performance improvements code benchmarks across multiple programming languages, highlighting its effectiveness for enhancing multilingual code understanding and generation. In general, this work presents a novel integrated RI. approach that leverages functional equivalence and offers a new perspective for advancing LLMs' understanding capabilities across diverse programming languages.
Limitations
A primary limitation of OORL stems from the group-based optimization mechanism in GEPO, which processes grouped preference responses for each source prompt. Compared to previous preference optimization methods [21, 3] that handle a pair of responses per step, GEPO manages larger groups of preference data, necessitating more GPU memory resources. This increased memory can be particularly pronounced with larger group sizes or longer response sequences. One potential strategy mitigate this issue is serialize the processing of individual responses within each preference group, which can reduce the number of padding tokens and thereby lower memory overhead. Although the serialization is beneficial for memory efficiency, introduces an increase in training latency due to the sequential handling of responses previously processed in parallel within the group.
Acknowledgement
This work is partially supported by The Research Grants Council of Hong Kong SAR (No. RF$2425- 4S02 and No. (201642,))
10

---

References
[1] A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee. J. Kreutzer, O. Pietquin. A. Üstün, and S. Hooker. Back basics: Revisiting reinforce style optimization for learning from human fcedback in llms. arXiv preprint ArXiv:2402. 14740, 2024,
[2] Anthropic. Claude Sonnet. https: hopps://wwrr /www. anthropic. ctw/ clt.e//aaaeeeee/eeee/eeeaaaeeee/ sonde//sonet..../ 2025.
Azar, Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. general theoretical paradigm to understand learning from human preferences. International Conference on Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024,
[4] F. Cassano, J. Gouwar, D. Nguyen. S. Nguyen, L. Chipps-Costin, D. Pinckney, M.-H. Yee. Y. Zi, C. J. Anderson, M. Q. Feldman, et al. MultiPL-E: A scalable and extensible approach to benchmarking neural code generation. arXiv preprint ar0iv:2208 08227, 2022.
[5] C. Cummins, V. Seeker, D. Grubisic, B. Roziere, J. Gehring, G. Synnaeve, and H. Leather. LLM Compiler: Foundation Language Models for Compiler Optimization. In Proceedings of the 34th ACM SIGPLAN International Conference on Compiler Construction, pages 141-153, 2025.
[6] A. Du, B. Gao. B. Xing, C. Jiang. C. Chen, C. Li, C. Xiao, C. Du, C. Liao, et al. Kimi LI. 5: Scaling reinforcement learning with LLMs. arXiv preprint arXiv:2501. 12599, 2025.
[7] D. Guo, D. Yang. H. Zhang, Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, Wang. X. Bi, et al. DeepSeek-RI:
Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint 1rX1v:2501.22458,
2025.
[8] D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puramik, H. He, D. Song,
et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2 09938, 2021.
[9] J. Hu. REINFÖRCE++: ++ A Simple and Efficient Approach for Aligning Large Language Models. arXiv
preprint 2rX1v:2501.03662, 2025.
[10] Y. Huang, M. Qi, Y. Yao, M. Wang, B. Gu. C. Clement, and N. Sundaresan. Program translation via code distillation. arXiv preprint arXiv:2310.1 1476, 2023.
[11] B. Hui, J. Yang. Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, et al. Qwen2. S-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
[12] R. Li, L. B. Allal, Y. Zá, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, al. 2023.
Starcoder: may the source be with you! arXiv preprint arXiv:2305.06/6
[13] R. Li, J. Fu, B. Zhang. T. Huang, Z. Sun, C. Lyu, G. Liu, Jin, and G. Li. TACO: Topics algorithmic code generation dataset. arXiv preprint arXiv:2312. 14852, 2023.
[14] 7. Li, T. Xu, Y. Zhang, 7. Lin, Y. Yu, R. Sun, and Z.-Q. Luo. ReMax: A simple, effective, and efficient reinforcement learning method for aligning large language models. arXiv preprint arXiv:2310. 10505,
2023.
[15] I. Loshchilov and F. Hutter. Decoupled Weight Decay Regularization. arXiv preprint
2017.
[16] A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang. D. Pykhtar, J. Liu, Y. Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint 2xxv:2402..1177 19173, 2024.
[17] J. Mattern, S. Jaghouar, M. Basra, J. Straube, M. D. Ferrante, F. Gabriel, J. M. Ong. V. Weisser, and
J. Hagemann. SYNTHETIC-I: Two Million Collaboratively Generated Reasoning Traces from Deepseck-
R1, 2025.
[18]
MistralAI. Codestral. pettps:/misstenn
hes///ws//odesssralllll/a ww//ew/////////aa///////aaaassta codestral, 2024. 2024.
[19]
I. Paul, G. Glavaš, and Gurevych. IRCoder: Intermediate representations make language models robust
multilingual code generators. arXiv preprint arXiv:2403.0328
2024.
[20]
Qwen Team. Qwen3. ttsp:: / // /qwen Im. gi thub. ic/blog/////oooo qwwn3/. ic/bLog//wwer//, 2025.
[21]
R. Rafailov, A. Sharma, Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct Preference Optimization:
Your Language Model is Secretly Reward Model. Anmual Conference on Neural Information Processing
Systems (NIPS), 2023.
[22] S. Rajbhandari, J. Rasley, O. Ruwase and Y. He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking. Storage and Analysis, pages 1-16. IEEE, 2020.
[23] B. Roziere, J. Gehring. F. Gloeckle. S. Sootla. I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remez, et al. Code Llama: Open foundation models for code. arXiv preprini arXiv:2308. 12950, 2023.
[24] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O, Klimov, Proximal policy optimization algorithms. 1707:/70070655
arXiv preprint
2017.
[25] Z. Shao, P. Wang. Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. MeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.0334 2024.
[26] M. Szafranice, B. Roziere, H. Leather, F. Charton, P. Labatut, and G. Synnacve. Code translation with
compiler repressttioontttsnnn arXiv preprint arXiv:2207.03572 2022.
[27] Q Zhu, D. Guo, Z. Shao, D. Yang, P. Wang, R. Xu, Y. Wu, Y. Li, H. Gao, S. Ma, et al. Deepseck-coder-v2.
Breaking the barrier of dosed-source models in code intelligence. arXiv preprint arXiv:2406. 022440661911, 11911, 2024.
11

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work.
12

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: For each theoretical result, the paper provides the full set of assumptions and complete (and correct) proof.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper discloses all the information needed to reproduce the main experi- mental results.
Guidelines:
The answer NA means that the paper does not include experiments.
13

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We will open-source our code and data after publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines futtps://nips.. cc/
public/ guides/ ProdeSubmit asionPooicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (ht (https: / / /tcppscccpppp htpp/uubiccc/ /taiii//uiiie ht//guuiieeee des / Code Submi as i cooooooliii icy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
14

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The paper specifies all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The paper does not need to provide the experiment statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provides sufficient information on the computation resources.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
15

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tle:://neurtae /neuriisss. wwtpp//lcc/tcccccceeaaocccceceee
Answer: [Yes]
Justification: The research is conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
16

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The original owners of assets, used the paper, are properly credited.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of the package should be provided. For popular datasets, papers anperswitin swithcode com/ dat aastaseteo has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- ce-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
17

---

Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy (https: / iepurrips.. ce/ /c//Co Con//eenfeeessssccceeeeeeeeoceee 2025/ LLM) for what should should not described.
18

---

A Derivation of the Implicit Reward
This section details the derivation of the relationship between an implicit reward function To (2 Mg)) and the optimal policy To (g)) under the standard reinforcement learning objective with a KL- divergence penalty against a reference policy Ter(Vyl This formulation foundational to methods like Direct Preference Optimization (DPO) and our proposed Group Equivalent Preference Optimization (GEPO).
Following [21], the objective is to find policy Ta that maximizes the expected reward ro(#g Ug) while remaining close to reference policy Thts controlled by coefficient, (3:
max
Ex, D.g mol (1 va)! 31 [Te (xng (Tee (y)
To (8 xgg)
min
Ex, LD,,DBB (us 1x llog
log ((x,)]
Z(we)) The (Vgl (g)) exp( 3 ( Ex x ( (no (Ug (ry) (yg) - log (srg))
Ro
min 70
(11)
Here, D represents the distribution of prompts Tg. and n-" T" ( is defined as the optimal policy distribution:
n' ("Vg)
Z(x,) TTrrly mg)) exp( ( Bg)).
(12)
The term (5gg) is the partition function, ensuring that x" (yn (zg) normalizes valid probability distribution over all possible responses Vy:
Z(xr)= = Σ
expl ( Un)).
(13)
My
The KL divergence term DEL. (To ( 11== xg))) in Equation 11 is minimized (becomes zero) when the policy no (Vg [gg) is identical to n² (Vg gz Therefore, the optimal policy Ty that solves the optimization problem
To (o (.g) = x (vglxg)) = Z(xrg) Teet (Vgl exp( (aap(7"loen (o (g))
(14)
To derive the expression for the implicit reward ra (Z Mg). we can rearrange Equation 14:
To(yo |r, 2(())) = exp( 1
8 gra(#g: Wg))
Teer (yg (Fgg)
log( Te( (r,g) ) (((rg)
3 ( Ug)
Teerlyy (r:g)
B(og (e [rg)
+ log Z (xx))) = r No)-
Weef Un (2g)
This gives us the final form for the implicit reward function:
ro (T Mg) = 3 log To(yg]=a) + 3 log ((rg). (vel(ygaag)
(15)
This relationship (Equation 15) demonstrates that the implicit reward (T Fo) is proportional to the log- probability ratio of the policy Ta (Vglzg) with respect the reference policy ( xgg). plus a term related to the partition function. This expression is leveraged in Section 3.2 (specifically, Equation 7) to transform the GEPO objective, which is initially defined in terms of r (Tig Ug). into an objective that directly depends on the policy probabilities no (yg (grg) and ( xg))
19


Reviews:
[{"review_id": "wSZoguy4Dd", "reviewer": "Reviewer_xoTJ", "summary": "In the present paper, the authors present OORL a combination of on-policy RL in the form of Reinforce++, and off-policy RL with group equivalent preference optimization. Applied to code, the authors leverage the underlying compiler infrastructure in the form of language-independent compiler representations to extend the code model's performance to lower-resource languages such as C#. The main novelty of the paper lies in the development of the GEPO off-policy approach, and how it leverages compiler IR. The performance of the approach is proven across a number code translation benchmarks on which the proposed approach outperforms other contemporary models.", "questions": "- Compiler intermediate representation themselves have a heavy distribution bias towards certain programming languages such as C++, both with regards to the length and amount of IR available per language. How do you seek to remedy, or potentially even exploit, this in practice?\n\n- The lines of code in the average code sample contained in the evaluation are fairly short, and may not be as representative as real world code. Do the authors imagine it to be possible to construct a \"hero\"-showcase where longer forms of code (& IR) are used to better illustrate the approaches' limitations in practice. Particularly this quote from the limitations \"GEPO manages larger groups of preference data, necessitating more GPU memory resources. This increased memory can be particularly pronounced with larger group sizes or longer response sequences.\" gives me pause of concern here.\n\n- Have the authors' considered how their approach could potentially also scale to more HPC-focussed languages? For example, there exist the Joseph Burkardt database of Fortran routines <https://people.sc.fsu.edu/~jburkardt/f77_src/f77_src.html> for which there are also direct function implementations in other languages such as C, C++, Java, and more available. Evaluating on a subset of the database would provide ample evaluation of the approaches' performance on HPC-centric programming languages.\n\n- Do the authors know whether their approach is sensitive to different generations of LLVM intermediate representation? I.e. LLVM 16 vs 17 vs 18 vs 19 vs 20. Have the authors for example tested their approach with different version of the LLVM compiler to make sure that the evaluation performance stays constant? Testing even with one different LLVM version would aid in assessing the approaches' sensitivity to this compiler specification.\n\n- As is, the potential of larger models, when prompted correctly, is not considered here at all. Have the authors considered to for example prompt larger models such as Claude Opus, or DeepSeek R1 to perform the same task? Adding this point of comparison would aid greatly in assessing the paper's contributions more comprehensively.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper has its main strengths in its intellectual clarity, its clear derivation of its optimization objectives enabling reproducibility with a reasonable amount of effort, and its style of evaluation. Especially its evaluations cover a wide range of programming languages, and as such provide a fair assessment of the approaches' strength across programming languages.\n\nThe main weaknesses of the paper break down into the following points:\n- The average lines of code per function, i.e. table 1, is too short for a representative benchmark. Especially the length does seem to not be representative of production code. See e.g. ComPile (https://huggingface.co/datasets/llvm-ml/ComPile), figure 3 for a more typical distribution of the length of production code when looking at the total instruction count.\n- The authors point out in the limitations, that GEPO occupies a lot of memory due to its group equivariant nature. Continuing on from the previous point, the paper fails to show when this limitation would apply in practice. Even considering a synthetic example to illustrate the limitations in terms of the length of code would improve the paper greatly here.\n- The evaluation only considers a highly limited number of models. Large language models which have seen LLVM IR during training such as StarCoder 2, or Meta's LLM-Compiler are not included in the evaluation, and modern large models such as Claude Opus are not considered in their entirety. As stands, the evaluation fails to capture these points of comparison and makes it hard to place its performance in relation to these models.", "was_revised": true, "final_justification": "I firmly recommend the acceptance of the present work as I believe for it to be a significant advancement towards extending machine learning for code to lower resource languages, a pressing issue of modern models.\n\nSpecifically, the following points made me arrive at my final score:\n- Algorithmic clarity, in the review especially, the authors managed to provide further clarity around training- vs inference-time usage of componentry. This addressed questions around the usage of IR, as well as around the memory footprint of the proposed approach.\n- Further discussion on the distinction between the presented approach, and the scale its evaluations is connected, and the inherent motivation behind it.\n\nAs such I wholeheartedly recommend the acceptance of this work.", "rebuttal": "We sincerely appreciate the **Reviewer xoTJ**'s insightful comments and questions. We are glad that **Reviewer xoTJ** recognized the innovation of our method and comprehensive experiments. Below are responses to address the concerns raised.\n\n___\n>Weaknesses 1 & Questions 2:\n\nWe appreciate your comments regarding the representativeness and length of our benchmark. We'd like to offer the following clarifications:\n1.  **Benchmark Representativeness**: As shown in Table 1, our newly proposed benchmark **CrossPLEval**, features an average of 34.57 Lines of Code (LoC) per function. This is approximately 5 times longer than traditional MultiPL-E benchmarks, marking a significant step towards more realistic code samples while remaining within a scope that allows for meaningful performance evaluation with current LLMs. Datasets that are excessively complex can sometimes obscure subtle but important improvements, especially given the current capabilities of models.\n2.  **Focus on Function-Level Evaluation**: Our evaluation primarily uses code samples at the function level. This was a deliberate choice to align with our core objective: to verify whether our proposed GEPO method can effectively enhance a model's deep understanding of code's \"functional semantics\" by learning the equivalence and non-equivalence of IRs. Functions are the smallest, self-contained units that encapsulate specific functionalities. Evaluating at this level provides the most direct and clear means to measure our method's ability.\n3.  **ComPile Dataset**: We acknowledge ComPile as a valuable IR translation dataset. It could potentially be used for training GEPO by constructing positive and negative groups, but it is not ideal for our evaluation, as our tasks involve high-level code generation. **Because our model operates the same pipelines as existing LLMs on programming benchmarks without requiring any IRs during inference**.\n4.  **GEPO Memory Usage Clarification**: It's crucial to clarify that GEPO's additional GPU memory resources are primarily required during training, not during inference. The increased memory consumption in training stems from managing larger groups of preference data. Although this is an overhead, it leads to increased data utilization efficiency. Importantly, the trained model, once deployed, operates with standard memory footprints, making it viable for practical application on longer code samples without additional memory concerns.\n\n___\n>Weaknesses 2:\n\n1. As previously mentioned, GEPO's additional memory occupation occurs exclusively during the training stage. \n2.  The increased computational overhead during training is a direct consequence of the enhanced data utilization provided by GEPO. By processing groups of equivalent and non-equivalent IRs, GEPO extracts more robust and nuanced semantic understanding. From a data efficiency perspective, this trade-off is well-justified by the significant improvements in model performance and its ability to generalize across programming languages.\n\n___\n>Weaknesses 3: \n\nWe appreciate your desire for a broader comparison against more diverse models. Our evaluation strategy is guided by specific objectives:\n1.  Our proposed OORL framework is validated on models within the 7B parameter scale. Direct comparisons with significantly larger models, such as Claude Opus, which operate at a much larger scale, are not directly comparable given the vast difference in model sizes and computational resources required. Our focus is on demonstrating the effectiveness of GEPO as an optimization method for LLMs, rather than on achieving new SOTA results with the largest models.\n2.  Our primary goal is to enhance multi-programming language understanding capabilities. **We do not utilize LLVM as a reference input for the model during the evaluation process**. Consequently, models like StarCoder2 and LLM-Compiler do not consistently demonstrate competitive performance when evaluated on our specific high-level code generation tasks. We strategically chose the strongest available open-source models of comparable size to rigorously demonstrate the effectiveness of our proposed method in improving code understanding and generation across diverse languages.\n    \n___\n>Questions 1: \n\nWe are grateful for this insightful question regarding the inherent distribution bias of compiler IRs. **Reviewer xoTJ** accurately highlights that the current compiler IR ecosystem is heavily structured around languages like C/C++. We wish to clarify that this focus is **not an oversight, but a conscious methodological choice, which we indeed seek to exploit.**\n\nOur core idea is to leverage the high-quality IR generated from resource-rich languages such as C/C++ as an ideal **knowledge bridge**. As we discuss in the paper, compiler IR possesses language-agnostic properties, making it an excellent medium for aligning the semantics of diverse programming languages. By training our model to learn the mapping from C code to its various functionally equivalent IR forms, we enable the model to grasp a deeper, function-level logic that is independent of any specific language syntax.\n\nOnce this abstract understanding of \"functional equivalence\" is established, it can effectively **generalize to other programming languages**, particularly those low-resource languages that often lack mature IR toolchains. As shown in Table 2 and 3, our method significantly improves code understanding and generation capabilities across a variety of languages, including low-resource languages Bash and Haskell, which are not explicitly present in the GEPO training phase.\n\n___\n>Question 3: \n\nThe core mechanism of our framework relies on learning functional semantics captured from compiler Intermediate Representations (IRs), rather than being tied to the syntax of specific source languages. While our current evaluation focuses on a range of mainstream and lower-resource languages, there are **no fundamental barriers that would prevent its application to HPC languages**, such as Fortran.\n\nOur method has already demonstrated strong generalization capabilities to programming languages that were not explicitly included in the training set (e.g., Bash and Haskell). This suggests that the framework's learned understanding of functional equivalence can effectively extend to other languages, including those relevant to HPC, without requiring direct training on them. This scalability to new domains is a key strength of our IR-based approach.\n\nIndeed, the Joseph Burkardt database is an excellent resource, and evaluating on a subset of it would be a valuable avenue for future work.\n\n___\n>Questions 4: \n \nWe appreciate this insightful question regarding the sensitivity of our approach to different LLVM IR versions.\n1.  It is crucial to clarify that our **test process does not involve the use of LLVM** for inference. The model, once trained, directly generates high-level code based on a high-level code prompt. Consequently, the evaluation performance is **not affected by the specific LLVM version** during testing, **as LLVM is not part of the inference pipeline**.\n2.  The enhancement in our model's capabilities primarily stems from its learning of functional semantics directly from the compiler's IRs **during the training phase**, coupled with the multi-programming language translation tasks. While different LLVM versions might produce slightly varied IRs for the same source code, our method's strength lies in learning the underlying *equivalence relations* between these IR forms. This indicates that the core knowledge acquisition happens at the training stage, making the inference robust to variations in LLVM versions during evaluation. For future work, we could explore training on data generated from multiple LLVM versions to further enhance robustness, but for the current study, the inference process remains unaffected.\n\n___\n>Questions 5:\n\nWe acknowledge the value of comparing with larger, state-of-the-art models for a comprehensive assessment.\n\nOur paper primarily introduces the OORL framework as a post-training method for LLM training, aimed at enhancing multi-programming language understanding capabilities. The framework can be applied to any code-domain-specific model to improve its code understanding. Naturally, larger models (Claude4 Opus, DeepSeek R1, etc.) with significantly more parameters and extensive pre-training are expected to outperform our 7B-scale model in absolute terms.\n\nRegarding prompting, as clarified previously, our testing phase does not involve the use of LLVM code. Therefore, there are no specific \"special prompt tricks\" related to LLVM. We utilize a standard evaluation pipeline [1, 2], ensuring a fair comparison.\n\n[1] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388, 2025.\n\n[2] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek-V3 Technical Report. arXiv preprint arXiv:2412.19437, 2024.\n\n___\n>Paper Formatting Concerns 1: \n\nThanks for meticulously pointing out the typographical errors. We sincerely apologize for these oversights. We will conduct a thorough review of the entire paper to correct all spelling and grammatical errors, ensuring a polished and professional final version. Your attention to detail is greatly appreciated.\n\n___"}, {"review_id": "8uCfgGeHcX", "reviewer": "Reviewer_9pZg", "summary": "The paper proposes OORL, a RL framework designed to improve LLMs' understanding of code across diverse programming languages, especially in low-resource scenarios. OORL combines on-policy RL with rule-based binary rewards from unit tests and a new off-policy method called Group Equivalent Preference Optimization (GEPO). GEPO guides LLMs to distinguish functionally equivalent IRs from non-equivalent ones by comparing groups of IRs, leveraging fine-grained equivalence signals. This enables models to better generalize code translation and generation across languages.", "questions": "What does the learning curve of the proposed RL training look like? Does the good performance emerge gradually or appear suddenly?", "limitations": "I don't see obvious negative societal impact of this work.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n* The research problem of code translation is practically useful. For example, it can accelerate system upgrades of old industrial codebases written in legacy or low-resource languages, such as Bash, Haskell, by translating them into modern languages like Python or C#.\n* The IR-based reward design is well-motivated and reasonable. It provides a good way to incorporate functional alignment into RL training beyond surface forms, and the design is intuitive and grounded in the nature of programming languages.\n* The experimental results support the claims, demonstrating significant improvements in code translation between four popular languages and a set of target languages, including low-resource ones.\n\n\n**Weaknesses**\n\nI don’t see significant weaknesses.", "was_revised": true, "final_justification": "I don’t see significant weaknesses in this paper, and the authors have addressed my question regarding the experiments.", "rebuttal": "We sincerely appreciate the **Reviewer 9pZg**'s insightful comments and questions. We are glad that **Reviewer 9pZg** recognized our useful and well-motivated method and comprehensive experiments. Below are responses to address the question raised.\n\n___\n>Questions 1: What does the learning curve of the proposed RL training look like? Does the good performance emerge gradually or appear suddenly?\n\nThe learning curve of our RL training exhibits the following behavior:\n1. Performance shows a sharp increase at the beginning. This initial surge is primarily driven by the model quickly learning the desired response format and basic task structure.\n2. Following the initial rapid increase, the performance continues to improve steadily but more slowly.\n3. Finally, the learning curve gradually stabilizes and converges within a certain range, indicating that the model has reached a plateau in its learning for the given task.\n\n___"}, {"review_id": "dBS0vhJCmj", "reviewer": "Reviewer_ms4a", "summary": "This work aims to bridge the performance gap of large language models (LLMs) across different programming languages through code-to-code translation. The proposed approach, OORL, is a reinforcement learning (RL) framework that integrates on-policy fine-tuning using an enhanced REINFORCE algorithm (REINFORCE++) with off-policy preference learning. A key component of the method, GEPO, learns to rank groups of functionally equivalent compiler intermediate representations (IRs), enabling deep semantic alignment through preference optimization. The approach yields performance improvements over vanilla RL, achieving higher pass@1 scores on both MultiPL-E (from 73.6 to 76.3) and the newly introduced CrossPLEval benchmark (from 58.5 to 63.2). The main contributions include a unified on-/off-policy RL framework tailored for multilingual code, a novel group-based preference optimization method leveraging IRs, and the introduction of CrossPLEval, a new benchmark for evaluating cross-lingual code generation.", "questions": "- You claim a “significant performance disparity” between popular and low-resource languages motivates your work. How large is this gap after simply continuing SFT on the low-resource slices of public code corpora? Why is a much heavier RL pipeline needed instead of that cheaper remedy?\n\n- Translating between languages does not necessarily expose the model to new algorithmic patterns. What evidence shows translation is causal for closing the cross-language gap, rather than just correlational?\n\n- You frame GEPO as a general technique for structured preference learning. But all experiments are narrowly scoped to IR-based code translation.Have you tested GEPO on any non-code task or natural language reasoning setup?", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strength\n\nUnder-explored area, efforts to curate the training data and establishing signals would be important to the community. \n\n## Weaknesses\n1. The method presupposes a robust suite of cross-language unit tests and IR mappings. In real-world scenarios, such robust testing and IR equivalences are seldom available, especially for less popular languages.\n\n2.  Many programming languages differ dramatically not only in syntax but in execution paradigms, side-effect behaviors, and semantic subtleties (e.g., imperative vs. functional vs. logic-based languages). Representational equivalence at the IR-level might be overly simplistic or even fundamentally incorrect.\n\n3. Missing “realistic” baselines like continuing pre-training on target-language code or retrieval-augmented fine-tuning, which could be cheaper and simpler.\n\n4. If the goal is genuinely closing the gap in resource-poor languages, the more straightforward solution would be to directly generate synthetic data or leverage powerful multilingual pre-trained models (e.g., GPT-4, Code LLaMA), rather than employing complex RL frameworks that require sophisticated translation tasks and fine-grained reward engineering.", "was_revised": true, "final_justification": "The authors did not address my concerns sufficiently e.g. \n> Question 1: You claim a \"significant performance disparity\" between popular and low-resource languages motivates your work. How large is this gap after simply continuing SFT on the low-resource slices of public code corpora? Why is a much heavier RL pipeline needed instead of that cheaper remedy?\nand \n> Weakness 4: If the goal is genuinely closing the gap in resource-poor languages, the more straightforward solution would be to directly generate synthetic data or leverage powerful multilingual pre-trained models (e.g., GPT-4, Code LLaMA), rather than employing complex RL frameworks that require sophisticated translation tasks and fine-grained reward engineering.\n\nStill we don't see a necessity for RL, unless seeing a clear-outperforming results against simple baselines.", "rebuttal": "We sincerely appreciate the **Reviewer ms4a**'s comments and questions. We are glad that **Reviewer ms4a** recognized the importance of our method. Below are responses to address the concerns raised.\n\n___\n>Weaknesses 1: The method presupposes a robust suite of cross-language unit tests and IR mappings. In real-world scenarios, such robust testing and IR equivalences are seldom available, especially for less popular languages.\n\nWe appreciate this concern, and our response clarifies the role of IR data and the generalizability of our method:\n1.  It's crucial to understand that our method utilizes IR data exclusively during the training stage. During inference, our model operates the same pipelines as existing LLMs on programming benchmarks, **without requiring any IRs**.\n2.  The core contribution of our method lies in its ability to enhance programming proficiency across diverse languages through code translation tasks and the recognition of IR logical equivalences. This capability is designed to be applicable in real-world scenarios, improving the model's understanding and generation of code in various languages, even those with limited resources.\n\n___\n>Weakness 2: Many programming languages differ dramatically not only in syntax but in execution paradigms, side-effect behaviors, and semantic subtleties (e.g., imperative vs. functional vs. logic-based languages). Representational equivalence at the IR-level might be overly simplistic or even fundamentally incorrect.\n\nWe acknowledge the semantic complexities across different programming paradigms. Our approach to equivalence within GEPO is based on **functional equivalence**. For any two code snippets, if they achieve the same functionality, they are considered equivalent in GEPO. Our algorithm explicitly accounts for subtle differences by minimizing the variance of their corresponding implicit rewards rather than forcing exact equality. Although this doesn't introduce perfect equivalence at a detailed level, our experimental results demonstrate that incorporating this functional-level approximate equivalence significantly improves overall performance.\n\n___\n>Weakness 3: Missing \"realistic\" baselines like continuing pre-training on target-language code or retrieval-augmented fine-tuning, which could be cheaper and simpler.\n\nWe appreciate the suggestion for additional baselines and would like to clarify our position:\n1.  Our method **does not utilize LLVM code** during the evaluation stage, making a direct application of RAG less straightforward for comparison. Our primary contribution is the OORL framework, which serves as a general method to enhance the multi-programming language understanding capabilities of Language Models (LLMs). It can be integrated into the training pipeline of any code-domain-specific model.\n2.  CPT isn't cheaper. Our training process with OORL requires only 4 A100 GPUs for full-parameter fine-tuning over one day, using 2400 samples. This demonstrates a highly efficient training process.\n3.  SFT on low-resource languages often necessitates extensive data collection, including correct reasoning processes and corresponding code, which incurs substantial human labor. Such datasets typically require large scales (100k+ examples). In contrast, our method leverages unit tests, where test case conversion across languages can be easily **automated via rules**, significantly reducing the data collection burden, especially for low-resource languages.\n\n___\n>Weakness 4: If the goal is genuinely closing the gap in resource-poor languages, the more straightforward solution would be to directly generate synthetic data or leverage powerful multilingual pre-trained models (e.g., GPT-4, Code LLaMA), rather than employing complex RL frameworks that require sophisticated translation tasks and fine-grained reward engineering.\n\nWe argue that our approach offers distinct advantages over direct synthetic data generation:\n1.  Directly synthesizing or generating high-quality low-resource language data for fine-tuning is resource-intensive, requiring significant human effort and time, and it struggles to cover a wide range of functionalities. \n2. Our OORL training framework is applied to LLMs after post-training. Consequently, our method does not conflict with the use of the SFT method.\n2. Moreover, as the multi-programming language understanding improves during RL training, it can itself generate higher-quality low-resource language data during rollouts, which further enhances its training in a self-improving loop. This inherent mechanism makes our method a powerful data augmentation technique.\n3.  Crucially, our training method demonstrates strong generalization capabilities, improving the model's proficiency even in programming languages not explicitly present in the training set. This is typically challenging for SFT-based approaches, whereas RL algorithms generally exhibit superior generalization. \n\n___\n>Question 1: You claim a \"significant performance disparity\" between popular and low-resource languages motivates your work. How large is this gap after simply continuing SFT on the low-resource slices of public code corpora? Why is a much heavier RL pipeline needed instead of that cheaper remedy?\n\n1.  Even with current code LLMs that have been extensively trained with various programming languages, a substantial performance disparity persists between different programming languages for the same coding requirements (As shown in Table 2 and Qwen2.5 Coder [1]). \n2.  Our RL pipeline is not \"heavier\". It is quite efficient. As mentioned previously, we achieved our results with only 4 A100 GPUs and 2400 samples for just one day.\n3.  Although SFT on low-resource data might reduce the performance gap, it often leads to a degradation in overall performance. Our goal is comprehensive multi-programming language understanding, not merely focusing on low-resource languages. As shown in Tables 2 and 3, our method demonstrates improvements in both widely-used languages and less common ones.\n\n[1] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024.\n\n___\n>Question 2: Translating between languages does not necessarily expose the model to new algorithmic patterns. What evidence shows translation is causal for closing the cross-language gap, rather than just correlational?\n\nAs detailed in Table 4, we provide empirical evidence that the code translation task is indeed causal in closing the cross-language performance gap. We perform code translation tasks with an RL algorithm and achieve significant improvement. This is one of the key contributions of our work, demonstrating that this specific task design within our framework directly leads to improved multi-programming language understanding.\n\n___\n>Question 3: You frame GEPO as a general technique for structured preference learning. But all experiments are narrowly scoped to IR-based code translation.Have you tested GEPO on any non-code task or natural language reasoning setup?\n\n1.  The GEPO algorithm is developed to specifically address the problem of the significant performance disparity between popular and low-resource languages in code. While it is fundamentally a general preference learning algorithm, its application requires defining a specific notion of \"equivalence\" for the task at hand. In this work, we leverage **code functional equivalence**. Applying GEPO to other natural language tasks would necessitate defining task-specific equivalence concepts, making it a highly specific application rather than something easily testable on general preference learning datasets.\n2.  Within the context of this paper, we conducted an ablation study comparing GEPO with the DPO algorithm on code-related tasks (Table 4). Our results demonstrate that GEPO outperforms the general DPO algorithm in this domain. From a formal perspective, DPO can be considered a special case of GEPO where the winner group and loser group each contain a single sample.\n\n___"}, {"review_id": "zcPeTBhEYZ", "reviewer": "Reviewer_w3YY", "summary": "This paper presents VeriReason, a novel framework that integrates supervised fine-tuning with Guided Reward Proximal Optimization (GRPO) and structured reasoning for Verilog RTL code generation. The model leverages both testbench feedback and AST-based structural heuristics as part of its reward signal to address hallucinations and misalignments between natural language specifications and generated Verilog code. The method demonstrates strong performance on the VerilogEval benchmark, achieving 83.1% functional correctness, outperforming both fine-tuned models and larger commercial LLMs such as GPT-4 Turbo. This work represents the first attempt to embed reinforcement learning and symbolic reasoning in the RTL domain, offering a scalable path to automated hardware design synthesis.", "questions": "1. Reasoning Module Efficiency: Could the authors discuss whether reasoning outputs can be cached or distilled into a faster model to reduce inference overhead?\n2. Generalization: How does the system perform when faced with ambiguous or underspecified prompts? Are there any fallback strategies?\n3. Reward Ablation: Could the authors provide a detailed ablation of individual reward components (testbench pass rate, structural match, spec match), possibly as a table?\n4. Limitations in Industrial Deployment: Can the method scale to real-world industrial-grade circuits, or is it currently constrained to educational/benchmark-level Verilog?\n5. Transfer to Other HDLs: Have the authors considered whether the same approach applies to VHDL/SystemVerilog, or are there domain-specific barriers?", "limitations": "Yes. The authors have discussed the computational cost of reasoning and RL training. I would recommend adding a short paragraph discussing potential bias from limited testbench coverage and its implications on model robustness.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: High impact and relevance: This paper tackles an important problem in EDA and LLM alignment, with immediate real-world applications. Well-designed methodology: The integration of GRPO and symbolic reasoning steps is innovative and well-motivated. Thorough evaluation: The results are compelling, supported by extensive comparisons, ablations, and generalization tests. Reward formulation: The mix of functional (testbench-based) and structural (AST-based) metrics for reward shaping is novel and effective. Scalability to multiple model sizes and families: The system is tested on Qwen, CodeLlama, and DeepSeek, showing robust improvements across the board.\n\nWeaknesses: Inference latency: The reasoning module increases inference time by 2.5× to 3×, which may limit real-world deployment. Lack of theoretical analysis: GRPO is introduced with empirical effectiveness, but lacks a deeper theoretical justification or convergence discussion tailored to the structured code domain. Limited generalization examples: The paper could benefit from showcasing more detailed examples of how the model generalizes to out-of-distribution circuits or novel control/dataflow patterns. Code/model not yet released: Although promised, the unavailability of code and model weights at submission time hinders reproducibility.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the **Reviewer w3YY**'s time and effort in reviewing our manuscript. \n\nWe believe there might have been a mix-up with the review comments. Specifically, the comment appears to be unrelated to our submission. It seems to refer to a different paper.\nWe understand that handling numerous submissions can be challenging, and such mix-ups can occasionally occur.\nWe respectfully request your assistance and thanks you for your attention to this issue and your continuous support in this review process."}]

Author rebuttal:

Meta-review: This paper uses the code translation task (between different programming languages) to improve code generation. The reviewers seem to think that the paper has a significant value, and that the claims are strong. The authors have answer questions meticulously.
Decision: Accept (poster)