DuPO: Enabling Reliable Self-Verification via Dual Preference Optimization
Authors: Shuaijie She
Yu Bao
Yu Lu
Lu Xu
Tao Li
Wenhao Zhu
Jianbing Zhang
Shujian Huang
Shanbo Cheng
Lu Lu
Yuxuan Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Self-Verification
Dual Learning
Preference Optimization
Large Language Model
Primary area: foundation or frontier models, including LLMs

Abstract: We present DuPO, a dual learning-based preference optimization framework that generates annotation-free feedback via generalized duality. DuPO addresses two key limitations: Reinforcement Learning with Verifiable Rewards (RLVR)’s reliance on costly labels and applicability restricted to verifiable tasks, and traditional dual learning’s restriction to strictly dual task pairs (e.g., translation and back-translation). Specifically, DuPO decomposes a primal task’s input into known and unknown components, then constructs its dual task to reconstruct the unknown part using the primal output and known information (e.g., reversing math solutions to recover hidden variables), broadening applicability to non-invertible tasks. The quality of this reconstruction serves as a self-supervised reward to optimize the primal task, synergizing with LLMs’ ability to instantiate both tasks via a single model. Empirically, DuPO achieves substantial gains across diverse tasks: it enhances the average translation quality by 2.1 COMET over 756 directions, boosts the mathematical reasoning accuracy by an average of 6.4 points on four challenge benchmarks, and enhances performance by 9.3 points as an inference-time reranker~(trading computation for accuracy). These results position DuPO as a scalable, general, and annotation-free paradigm for LLM optimization.

DUPO: ENABLING RELIABLE SELF-VERIFICATION
VIA DUAL PREFERENCE OPTIMIZATION
Shuaijie She
Yu Bao
Yu Lu
Lu Xu
Tao Li
Wenhao Zhu
Jianbing Zhang
Shujian Huang
Shanbo Cheng Pf
Lu Lu Yuxuan Wang
Nanjing University
ByteDance Seed
ABSTRACT
We present DuPO, dual learning-based preference optimization framework that generates annotation-free feedback via generalized duality. DuPO addresses two key limitations: Reinforcement Learning with Verifiable Rewards (RLVR)'s re- liance on costly labels and applicability restricted to verifiable tasks, and tradi- tional dual learning's restriction strictly dual task pairs (e.g., translation and back-translation). Specifically, DuPO decomposes a primal task's input into known and unknown components, then constructs its dual task reconstruct the unknown part using the primal output and known information (e.g., reversing math solutions to recover hidden variables), broadening applicability to non-invertible tasks. The quality of this reconstruction serves self-supervisee reward to opti- mize the primal task, synergizing with LLMs' ability to instantiate both tasks via a single model. Empirically, DuPO achieves substantial gains across diverse tasks: it enhances the average translation quality by 2.1 COMET over 756 directions, boosts the mathematical reasoning accuracy by average of 6.4 points on four challenge benchmarks, and enhances performance by 9.3 points as an inference- time reranker (trading computation for accuracy). These results position DuPO as a scalable, general, and annotation- free paradigm for LLM optimization.
INTRODUCTION
Large Language Models (LLMs) Yang et al. 2025
Grattafiori et al. 2024 Liu al. 2024 Ope-
nA
2024 Al 2024 DeepMind
2025) have shown remarkable progress in tasks like mathemat-
ical
reasoning Yang al.) 2024b Shao al. 2024 Chen al. 2025) and multilingual transla-
tion (Cheng et al. 2025 Zhu et al.
2024 Li et al.) c024b To further enhance these capabilities, researchers have increasingly adopted reinforcement learning (RL) paradigms like Reinforcement
Learning from Human Feedback (RLHF) (Grattafiori et al. 2024 Liu et al.) 2024| Yang et al. (025) and Reinforcement Learning with Verifiable Rewards (RLVR) DeeepSeekAAlle (1025 Team 2025 Yu et al. 2025 He et al. 2025: Hu et al. 2025b) have gained traction. Specifically, RLHF aligns mod- els with human preferences but relies on costly, inconsistent human annotations (Lee et al.) 2023] Zhang et al. 2024). RLVR addresses this for objective tasks (e.g., math, code) via binary rewards from verifiable answers, reducing annotation burdens. However, RLVR still depends on external supervision: acquiring verifiable answers remains bottleneck, limiting scalability. Moreover, it struggles with generative tasks (e.g., multilingual translation), where single references cannot cap- ture diverse high-quality outputs (Jia et al. 2025 Callison-Burees et al. 2006). Recent attempts (e.g.. AI- IL-Fee (Lee et al. (023). Constitutional AI (Bai et al. (2022) merely swap dependen- cies (human labels - teacher models/rules), failing to resolve the core bottleneck.
Dual learning (He et al.) 2016) offers self-supervised alternative by leveraging task duality to generate intrinsic feedback: through paired "primal" and "dual" tasks (e.g.. translation and back- translation (Sennrich et al. (0155), models validate outputs via cycle consistency, eliminating re- liance on external labels. Given that LLMs possess diverse capabilities from extensive pretraining, they could be trained across various tasks. However, applying this framework to LLMs non-trivial, which faces two critical challenges:
"Correspnndigg author.
"This work done during Shuaijie's internship ByteDance Seed.

---

1. Limited Duality in Mutually Non-Invertible Tasks: Most real-world LLM tasks (e.g., math reasoning) lack strict invertibility. LLM's output (e.g., a math solution) rarely con- tains enough information to reconstruct its input (e.g., the original problem), breaking the duality cycle.
2. Bidirectional Competence Asymmetry: LLMs often exhibit uneven performance across primal/dual tasks (e.g., strong at solving math problems but weak at generating problems from solutions). Noisy self-signals from asymmetric tasks hinder optimization, reducing the framework's utility.
These mismatches render traditional dual learning ill-suited for general LLM optimization, leaving it open challenge.
In this paper, we propose DuPO (Duality-based Preference Optimization), framework that aligns LLM generalization with a (relaxed) duality applicable to general tasks. it core lies a generalized duality framework ($3.2) built on complementary relationships: it decomposes each input into disjoint known (Ixz) and unknown (In) components, then designs the dual objective to reconstruct only Tu. from the primal output y and the known input Tk, rather than inverting the full input. This framework resolves two asymmetries: it restores sufficient information flow hetween the primal and dual tasks (task asymmetry) and reduces the complexity burden the dual task side (capability asymmetry). The formulation naturally synergizes with LLMs: their broad foundational capabilities allow a single model instantiate both primal and dual tasks without specific architectures, while the dual task converts the model's outputs into self-supervised reward signals, enabling continual improvement without external annotations. This bidirectional benefit addresses critical challenge in LLM development: obtaining high-quality feedback for capability enhancement.
We empirically validate DuPO on two representative tasks: mathematical reasoning and multilingual translation, demonstrating significant and consistent improvements. By applying DuPO to one the strongest translation LLM, Seed-X-7B- Instrruct (Cheng al. 2025 we demonstrate significant performance gain of 2.1 COMET points on the multilingual translation benchmark, bringing the 7B model to performance comparable ultra-large SOTA systems. In mathematical reasoning. method yields robust gains across models of varying scales, from 1.5B to 7B parameters; notably, DuPO improves the Qwen3-4B (Yang al.. 2025) model's score four challenging mathematical benchmarks by 6.5 percentage points. Our comprehensive ablation studies confirm that our design, the generalized duality, crucial for achieving these results. Beyond training, DuPO acts as reranking mechanism at inference, boosting performance by 9.3 points without finetuning enabling smaller models to outperform stronger ultra-large LLMs like DeepSeek-RI (DeepSeek-AI) (025) even without training. In summary, DuPO reimagines task duality for non-invertible LLM tasks. It eliminates external annotation reliance, scales across tasks/domains, and enhances both training and inference, offering a scalable path to align LLMs with diverse goals using self-supervised feedback, marking a promising first step toward unlocking verficeriiination for broader, open-ended domains.
PRELIMINARIES
We can cast various tasks as conditional generation: with input space X' and output space V, the model generates y E y via the LLM TT x) given an input & E X.
2.1 PREFERENCE OPTIMIZATION
Preference optimization steers LLMs' behavior by assigning scalar rewards to responses based on their quality, higher-rewarded outputs are preferred and reinforced by the optimization. Formally, given a reward function X' y I that quantifies output quality, the objective is:
max 8
In practice, r(z, y) is commonly derived from human preferences (RLHF), LLM judgments (RLAIF), or verifiable correctness (RLVR). While numerous effective algorithms (e.g.. PPO (Schul- man et al. 2017). REINFORCE+ (Hu et al., 2025a),, GRPO (Shao et al.) 2024)) have been devel- oped to optimize this objective, their performance ultimately hinges on the quality of the reward signal. Consequently, the key challenge lies in obtaining accurate and scalable rewards, given that existing sources are plagued by high costs, inherent biases, and limited coverage.
2

---

A B =
X
(a) Non-unique reconstruc- tion of x fromy
(b) Failure to reconstruct X given
(c) Predicting a subcompo- nent B of the input to pre- serve uniqueness and adapt dual task difficulty
Figure I: Challenges in Dual Learning and Solutions via Relaxed Duality Constraints. Dilem- mas applying dual learning - (a) Non-unique reconstruction of from y breaks the closed-loop: (b) Failure to reconstruct from due to competence asymmetry. Resolutions by relaxing duality restrictions - (c) Predicting unknown component B to preserve uniqueness and adapt dual task difficulty
2.2 DUAL LEARNING
Dual learning offers self-supervision signal by utilizing the task duality. We begin by formalizing the task duality between a primal task and its dual counterpart.
Definition 1. A primal lask is a mapping Tp X Y, and dual task is mapping Ta: : - X. The pair (Tp: Ta) is said to form dual pair if they satisfy the consistency principle:
E d(x, Ta(Tp(x))) Ex,
where d(.): :x)x R a domain-specific distance metric, and ex 2 0 is a tolerance threshold that quantifies acceptable reconstruction errors in each space.
Leveraging this duality, we can construct a self-supervised reward to quantify the quality of primal- task output. Given input x and its corresponding output (x).,. we could define reward
y(x,y)
exp(->-d( dx, Ta(y))),
where > controls the sensitivity of the reward to reconstruction error. High-quality outputs maximize the expected reward B[r(y)] by preserving information that is recoverable through the duality cycle. This principle has been successfully applied in various domains, including machine
translation (He al. 2016 Zou al. 2025).
DUAL LEARNING-BASER PREFERENCE OPTIMIZATION FRAMEWORK
In this section, we present DuPO (Duality-basee Preference Optimization). Its core objective is to leverage the intrinsic relationships between tasks and their dual counterparts to generate self- supervised rewards, enabling LLMs improve performance without relying on expensive human annotations complex handcrafted rules.
3.1 CHALLENGES IN DUAL LEARNING-BASHD OPTIMIZATION
While task duality offers promising self-supervised paradigm, its application to LLM optimization non-trivial, it confronts two critical challenges that disrupt the closed-loop information flow.
Challenge I: Limited Duality in Non-Mutually Implicative Tasks. The utility of task duality hinges on Tn and Ta being mutually implicative - specifically, the output y of T contains suffi- cient information to reconstruct via Ta. and vice versa. This property holds for canonical tasks like machine translation, where y (a translation) preserves the semantic content of x (the source sentence), enabling Ta (back-translation) to recover x.
However, most real-world tasks lack this mutual implicativity (Fig. Ia reasoning, where Tp solves problem x (e.g., "A box contains 3 red and blue balls; what is the total?") to produce y = 8. Here, (the total count) is insufficient to uniquely reconstruct x via Ta 8 could answer infinitely many disparate questions, such "What is 10 2?" "What is the
Consider mathematical
3

---

atomic number of Oxygen?". This underdetermined relationship breaks the duality loop: Ta cannot reliably recover x from y, making the self self-supervised reward (based on x 2e x) untrustworthy.
Challenge II: Bidirectional Competence Asymmetry. Even for mutual invertibility tasks, dual- ity optimization is sensitive to the bidirectional competence of the LLM the performance gap between Tp and Ta- If Tp is strong but Ta weak, Ta may produce noisy & that distorts the su- pervision signal (Fig. [b]. This asymmetry is particularly pronounced in L.MMs, where extensive pretraining creates diverse but uneven capabilities across tasks, even within the same domain.
For instance, in machine translation optimization, let = "The quick brown fox jumps the lazy dog" (English) and y "Der schnelle braune Fuchs springt über den faulen Hund" (correct German translation). A high-quality y should enable Ta to back-translate to & struggles with nuanced vocabulary (e.g.. "schnelle" + "fast" instead of "quick"), ) x might be "The fast brown fox jumps over the lazy dog" divergence from x that erroneously penalizes y despite its correctness.
However, if Ta
Using separate models for Tp and Ta, as was common in traditional dual learning, merely sidesteps the challenge of intrinsic competence asymmetry (Wang et al., 2018). This imbalance still arises from the distinct natures and complexities the primal-dual tasks, destabilizing the self-supervised feedback loop.
3.2 GENERALIZED DUALITY REWARD VIA COMPLEMENTARY TASK
To address the two-fold challenges, we propose generalized duality that redefines task duality through complementary dependencies. transcends traditional duality's strict input-output rever- sal requirement by leveraging partial and stable dependencies between task components, enabling robust sulf-super rewarding even for tasks lacking inherent mutual implicativity.
Definition 2. Let the input space X of primal task Tp be decomposed into two disjoint subspaces: N (known components) and X, (unknown components), such that X X₂ x X,. The primal task Tr is mapping Tp X - y that maps X an output space y E y. Its complementary dual task Tea is a mapping that leverages and the known component X2 to reconstruct the unknown component x,, E Xx:
Tod : (y,8k) + Su.
The pair (Tp: Tea) is said to form a generalized dual pair if they satisfy the complementary consis- tency principle:
Vx E y= d(xu, Tealy, xx)) (eely,x))))))))) Se:
where d(-): X2 x XX + R is a domain-specife distance metric, and 2 0 is tolerance threshold.
Leveraging this generalized duality, we can construct a self-supervised reward quantif'y the pref- erence of a primal-tast output analogously to Def. [ Given an input x E X' with decomposition (æg, and its corresponding output y = (x)., the reward is defined as
y(x,y) (-p d (X.: Tealy, xx))),
where À 0 controls reward sensitivity.
simple two-sum task illustrated in Example [ When generalizing to practical applications, DuPO also splits the input and selects component of the input (e.g., a numerical variable in math problem) to serve as the unknown part I, and construct the corresponding dual task. To further improve the task duality, the selection could follow principles like Answerability of the Dual Ques- rion and Uniqueness of the Correct Completeness (see Appendix for details). Moreover, DuPO's flexibility also allows for task-specific distance metrics d(-). We could employ BLEU scores for multilingual translation, while for mathematical reasoning, we evaluate variable equality, yielding binary rewards, with case studies provided in Appendix D
4

---

Example 1: Generalized Duality Reward for a Two-Sum Task: A+
The primal task Tp output as:
Xx + X3 is to compute the sum of two numbers, with its input and
The imput x is decomposed as + (A, B). where X known number) and X B (an unknown number, without loss of generality).
-The output y is the result of sum: C
The complementary dual task Ted X 4-y - X is designed to reconstruct the unknown compo- nent X,, using the primal output y (i.e. C) and the known X (i.e. A):
&u + B
Then, we can directly quantify whether B (original unknown) and B' (reconstructed unknown) are consistent as reward signal:
yx x exp (
Here, T(-) is an indicator function: it equals 0 if B = B' (consistent) and 1 otherwise (inconsis-
tent). This ensures the reward is maximized when B and B' match, and reduced otherwise.
3.3 POLICY OPTIMIZATION WITH GENERALIZED DUALITY REWARD
The core of Dual Learning-baat based Policy Optimization (DuPO) framework is optimize LLMs using duality-derived self-supervised complementary rewards r r(x, y), without external annotations. The objective is to maximize the expected reward based on the (complementary) dual task:
T(0) (ypx) y)]
where (y())) denotes the LLM (parameterized by 8) for generating output y given input (Xu; xx). Notably, DuPO is compatible with various RL algorithms (e.g., PPO, REINFORCE++), adopt GRPO (Shao et al. 2024) in our experiments for its simplicity and efficiency.
Remark 1. Compared to traditional dual learning, which suffers from strict mutual implicativ- ity (i.e., y must fully encode x) and bidirectional competence asymmetry, our generalized duality framework offers three fundamental advantages:
1. Overcomes the Invertibility Constraint. By redesigning the dual objective from recon- structing the full input x to only a selected unknown component X our framework fun- damentally bypasses the stringent requirement of task symmetry. This relaxation the key to unlocking dual learning for tasks that are inherently non-invertible, where the primal output does not contain sufficient information to recover the entire input.
2. Mitigates the Competence Asymmetry. The difficulty of the dual task is significantly re- duced in two ways. First, the known component Xx acts as a strong contextual anchor, constraining the solution space for reconstruction. Second, select an Xa, that is not only feasibly reconstructible but also act as a faithful reward signal for the primal task's ss solution quality (Appendix(A). This directly addresses the "weak dual" pitfall and ensures the reff-superviee reward is reliable and informative.
can simply yet effectively
3. Enables Broad Applicability. It unlocks dual learning for broad class of tasks previously considered unsuitable, including complex domains such as mathematical reasoning, code generation, and dialogue systems where input-outpul relationships are partial.
EXPERIMENT
We validate the efficacy of DuPO representative tasks: multilingual translation and math- ematical reasoning. Below, we detail the experimental setup, datasets, and evaluation metrics for each task, followed by key results.
4.1 EXPERIMENT SETUP
Base Model. We evaluate DuPO on diverse set of strong and popular base models to demonstrate its effectiveness and robustness. For translation tasks, we employ Sosd-X--B-Instrreee (Cheng et al.) 2025), one of the strongest open-source translation models. For mathematical reasoning, select models of varying scales and capabilities, including small-scale yet powerful DeepSeek-Rl-Distie Qwen-1.5B (DeepSeek-Al) 2025) and its larger counterpart DeepSeek- R -Distill-Qwen-71. both
5

---

Model
BLEU
COMET BLEURT Avg.
Google
Qwen3-8B
21.7
84.8
65.8
57.4
36
Doubao-1.55 Thinkkng
26.2
87.9
71.7
61.9
Qwen3-235B-22B
28.4
88.8
73.9
63.7
DeepSeek-RI-Os
30.2
89.2
75.0
64.8
13
Seed-X-7B-Instree
28.8
87.0
72.6
62.8
DuPO (ours)
30.3
74.6
64.7
Figure 2: Human Evaluation Scores (0- 4) on the Seed-X-Challenge for 14 Lan- guage Directions. DuPO achieves per- formance comparable to or even surpass- ing ultra-large models such as GPT-4o and DeepSeek-RI-ON 1-00100, while significantly out- performing Google Translate.
Table 1: Multilingual Translation Perfor- mance Across 756 Translation Directions 28 Languages. DuPO significantly improves all metries and performs comparably to its strong counterparts (DeepSeek models).
distilled from the SOTA DeepSeek-RI. We also include Qwen3-4B (Yagg et al. 2025), the latest strong small L.M,, and the most capable open-source reasoning model, Open Nemooonnn--- 7B (Moshkov et al., 2025). These models represent strong and representative baselines within their respective model scales, ensuring comprehensive evaluation. Additionally, we also include some SOTA and impressive ultra-large models like coubao-1.55/..6TTii
(ByteDance Seed Team 2025). Mhude--oonet4-Thinniin and DeepSeek-RI (DeepSeek-A] 2025) for comparison.
Dataset. For translation tasks, we focus on 28 languages that are aligned with the language cover- age of Seed-X, selecting 1,000 prompts for each language from a multilingual pre-training dataset to create our training prompt set. Additionally, we collect 7,000 parallel data entries across these specified languages to support our experiments from the dev set of Flores-200 (NLLB Team (024)) For mathematical reasoning tasks, we utilize questions from mixture of publicly available math- ematics datasets"] These diverse datasets are widely used to synthesize SFT data or provide oracle labels for RL, training LLMs complex reasoning tasks.
Benchmarks. For comprehensive evaluation, we include various tasks and benchmarks:
We construct an automatic evaluation set by randomly sam- pling 50 instances for each of the 756 translation directions (across 28 languages) from the Flores test ser"] This dataset, comprising 37,800 samples, will be released to facilitate reproducibility. We employ BLEU (Papineni et al. 2002), COMET (Rei et al.) 2020). and BLEURT (Sellam et al. 2020 as evaluation metrics. For human evaluation, we use the Seed-X-Challenge (Cheng al. 2025 a challenging benchmark test the boundaries of LLMs' translation capabilities with diverse linguistic elements across multiple domains. Human experts assess accuracy, fluency, and idiomaticity. scoring translations from Chi- nese or English to seven languages on 0-4 scale (higher score denotes better translation).
Multilingual Translation:
We evaluate our approach on multiple henchmarks, including AMC23 (MAA) 2023), AIME24, AIME25 and HMMT (Balunović et al., 2025). assess performance in standardized contest environments. For each problem, we sample 32 re- sponses using a temperature of 0.8 and maximum reasoning budget of 32,000 tokens, then report the average accuracy (Avg@) 32).
Mathematical Reasoning:
Ultra-large models like DeepSeck-R. R1 and Doubao- 1.6- 6-Thinning are accessed via their official APIs. More details about training are provided in Appendix C
4.2
MAIN RESULTS
4.2.1
DuPO BOOSTS LLM'S PERFORMANCE ON VARIOUS TASKS
DuPO achieves strong performance on diverse tasks, including multilingual translation and mathe- matical reasoning. On multilingual translation, DuPO elevates the backbone a SOTA performance level, rivaling and even surpassing significantly ultra-large LLMs. As detailed in Tab. I applying DuPO to the Beed-x--B-Instruct model boosts its performance by 1.5, 2.1, and 2.0 across three automatic evaluation metrics. This performance even surpasses that of current SOTA closed-source
Mooe details math data preparation be found Appendix https: aca.facce.coo http://hhwggggeeoaaaa///e//e//ae https://gi thub. com/ kyte lacce Swwd/ <ttp://yeeaaneeeeeee///e/eee-ed- /aaar///eeeeeeeeeeaaa//eerrrraaa http://aannnncalllg_se_ss

---

Model
AMC23
AIME24
AIME25
HMMT
Avg.
DeepSeck-R1-0120
97.7
79.8
70.0
44.2
72.9
Sonnde--oonet-Thinkiii
97.5
82.5
70.0
48.3
74.6
Doubao-1. Thinkknng
99.4
86.3
73.3
57.7
79.2
Doubao- 6 T-inniin
98.8
88.4
83.4
60.1
82.7
DeepSeek-RI 1-0528
99.4
91.4
87.5
71.4
87,4
DeepSeck-R11DD RI-Distill Saalll-wen--.n. w/ DuPO (ours)
67.5
20.0
20.0
13.3
30.2
72.5
30.0
26.7
16.7
36.5 (+6.3)
DeepSeek-RI RI-Distill D---isiill-Qwen-22 Qwen-7B
85.0
56.7
36.7
20.0
49.6
w/ DuPO (ours)
90.0
63.3
40.0
26.7
55.0 (+5.4)
Qwen3-4B w/ DuPO (ours)
95.0
70.0
66.7
40.0
67.9
97.5
83.3
70.0
46.7
74.4 (+6.5)
BaaReassoning-Nemotroo
95.0
83.3
73.3
56.7
77.1
w/ DuPO (ours)
97.5
83.3
90.0
66.7
84.4 (47.3)
Table 2: Mathematical Reasoning Performances (%) on Representative Benchmarks. DuPO significantly improves the performances across models with varying base capabilities, enabling Qwen3-4B to outperform DeepSeck-R1-0122 and OppuReesoniing--enooro----1 to achieve impres- sive performance (+7.3).
ultra-large language models, such Doubao-I.S-Thinking (+2.8) and Qwen3-235B-22B (+1.0), and is on par with the performance of the latest DeepSeek-R1. As shown in Fig. 2 DuPO demon- strates remarkable performance, achieving results comparable tfte-oofthhearr ultra-large models such as GPT-4o and DeepSeek-R1. Moreover, DuPO substantially outperforms widely-used com- mercial closed-source systems like Google Translate, showcasing a clear advantage translation quality perceived by human evaluators.
On mathematical reasoning, the results in Tab. clearly demonstrate that DuPO yields consistent and significant performance improvements across all models at different scales and baseline reason- ing ability. On the most powerful DpenReasoninggNNeootroo the average score from 77.1% to 84.4%, achieving impressive performance. This trend of signif- icant gains continues 6.5 points from 67.9% to 74.4%, even surpassing the ultra-large model DeepSeek -eaccckRR-D00 R1-0120. The approach remains remarkably effective on DeepSeck's distilled models well. Even on DeepSeek- RI-Distill-Qwen--1.. 1.55, the least reasoning capability among the strong baselines, we still achieved 6.3-point increase in average accuracy. Our framework's performance is further validated by crete examples in multilingual translation and mathematical reasoning (case studies in Appendix D
model, applying DuPO increased
the mid-sized Qwen3-4B model, which saw its average score boosted by
4.2.2 DUPO SCALES TO VARIOUS BACKBONES EFFECTIVELY
To validate the robustness and generalization of our proposed DuPO framework. we extend our evaluation to the LlaMA architectural family. Our experiments are conducted on two LlaMA archi- tectural models: 1.aMA-3.1-88 1-8B (Grattafirri et al., 2024) and OctoThinker- Bhhinker-8B-Hybriiee -B-HHyrid-BBB Barie (Wang et al. 2025), the latter of which has undergone middle training on mathematical reasoning knowl- edge. Considering the significant difference model ability, we select two benchmarks moderate difficulty, AMC23 (MAA 2023) and MATH500 (Hendrycks et al. 2021). For a fair comparison, all models are finetuned using identical training data and settings. Results are listed in Tab. 3
As seen, DuPO's effectiveness is not tied to a specific model architecture; it serves as robust and generalizable enhancement, delivering significant improvements to diverse backbones regardless of their initial reasoning proficiency. DuPO lifts the average score LlaMA-3. 1-BB to 32.1 %, +24.0 percentage-point gain over the vanilla model, and surpasses SimpleRL-Zon (Zeng et al.) 2025) (which relies on oracle-labeled answers during training) by 13.1%. When applied to the Octo Tetnkeiinee--B... HebrHyiii-BBee performance improvements of +50.0 on AMC23 and +27.4 on MATH500, achieving an average performance of 62.5.
BB-Hybrid-Base Base (Wang al. 2025), our DuPO approach yields more impressive

---

Model
AMC23 MATH500 Avg.
LlaMA-3.1-3
2.5
13.6
8.1
SimpleRL-Zoo
15.0
23.0
19.0
Accuracy (%)
w/ DuPO (ours)
20.0
44.2
32.1
40-
MooTThinker-BB-Hybrrii -Bas
5.0
42.6
23.8
w/ DuPO (ours)
55.0
70.0
62.5
20-
50
100
Table 3: Performances (%) of DuPO on Dif- ferent Backbone Models. DuPO even surpasses SimpleRL-Zoo, which utilizes labeled answers as reward. DuPO's potential is further exemplified by Thinkei, which underwent additional mid- dle training.
Steps
Figure 3: Training Progress of DuPO on Qwen3-4B-Base The performance consis- tently improves on the primal/dual task (For- ward/Backward Acc) and the benchmarks.
DuPO (curs)
Oracle-RLVI
Orecle-AAPD
Model
A1ME24 AIME25
Avg.
DeepSeck-R1-0120
79.8
70.0
74.9
35
Clande-Somner--Thinnk.
82.5
70.0
76.3
30
DeepSeck-R1-Disiill-..tt
20.0
20.0
20.0
25
Awerage
w/ DuPO rewarding
53.3
24.1
38.7 (+18.7)
20
Qwen3-4B
70.0
66.7
68.4
15
w/ DuPO rewarding
86.6
68.9
77.7 (49.3)
1d0
200
300
400
500
6do
Table Inference-Time Scaling on Mathematical Rea- soning Using DuPO Rewarding (Backward Acc) for Reranking. Our method improves the performance of policy models with varying base ability, without requir- ing additional training.
Step
Figure 4: Training Process of DuPO vs. Oracle Baselines. DuPO tracks the oracle baselines that uses ground-truth for verifiable reward.
4.2.3 DUPO INCENTIVIZES REASONING CAPABILITY ON BASE MODEL
Further validating DuPO's versatility, we demonstrate its effectiveness even when applied directly to base model without a preliminary supervised fine-tuning (SFT) phase for activation of reasoning behavior. We track the learning dynamics by simultaneously collecting the primal task accuracy ("Forward Acc") and complementary dual task accuracy ("Backward Acc") on the training set and its generalization performance on three unseen test set.
As shown in Fig. DuPO provides stable and effective pathway to awaken and generalize the latent reasoning abilities of base model, validating its utility as a powerful training methodol- ogy. Specifically,, the training dynamics reveal clear and substantial improvement the primal task, with the "Forward Acc" soaring from a nascent 15.2% to 56.5%. Our automated unknown- component selection strategy (based Qwen3-4B-Insttee ing relatively high initial accuracy (52.6%) and effective rewards. As the model improves during training, it naturally solves more dual questions, thereby unlocking richer reward signals. This up- ward trajectory provides direct evidence that the reward signal derived from our dual-task serves as an effective guide for enhancing the model's reasoning. More importantly, this acquired skill demonstrates robust generalization. Performance on the test set AMC23 leaped from 20% to 70%, with similarly significant gains observed on the AIME24 and AIME25 datasets.
adjusts dual-question difficulty, yield-
4.2.4 DUPO ACHIEVES SIMILAR PERFORMANCE WITH ORACLE
Further analysis reveals that DuPO achieves near-optimal performance, effectively approaching the results obtainable with ground-truth supervision. To quantify this gap, construct two oracle base- lines, Oracle-RLVe and Oracle-DAPO, where both utilize ground-trutte truth answers to verify rollouts reward signals. Specifically, Oracle-RLD utilizes the exact same dataset as DuPO, serving direct baseline. In contrast, Oracle-DAPO employs the DAPO Yu et al.) (2025) dataset, which is a representative high-quality annotated dataset from the open-source community. These oracles indi-
8

---

100
1.38 Bassline
48 Basel ma
70
D.PD
95
W0
DJPO
DJPD wel 1i
DuPD w/o flt
60
90
30
Acc
75-
233
747
70
28.0
20
AMC23
AIME24
AIME25
Avg
AMC23
AIME24
AIME25
Avg
Figure 5: Performance Ablation of ical Reasoning. Our unknown component selection strategy reduces training noise and improves these models' performance across three benchmarks.
Mathemat-
cate the performance upper bound of the RLVR paradigm given accurate supervision. For efficiency, we conducted experiments using Qwen3-4B-o Base with 8192 maximum output length.
As shown in Figure 4 DuPO closely tracks Oracle-RLVR throughout the training process. By step 600, both methods reach nearly identical performance levels at approximately 35% aceuracy. This close alignment indicates that our self-verification reward signals are accurate and effective. Furthermore, we observe that the performance of Oracle-DAPO is comparable to that of both DuPO and DLVR. This similarity suggests that H. selection strategy has minimal impact on the data distribution, thereby demonstrating the robustness of DuPO.
4.2.5 DUPO SCALES REASONING DURING INFERENCE WITHOUT TRAINING
Beyond serving as a reward signal for RL training, the DuPO mechanism can be naturally applied training-free,, inference-time reranking strategy to improve the reasoning capabilities of any LLM. The process unfolds in three stages: 1) Similar to the rollout stage during RL process, we prompt any given policy model to generate diverse reasoning trajectories. 2) For each candidate trajectory, we use its final answer to ask the policy model to solve the corresponding dual question automatically constructed without accessing labeled answer. We could apply more computation by performing K (K = 8 in our experiments) sampling runs on each dual question for more reliable reward estimate, a practice distinct from RL training. 3) Finally, for each test set question, select the trajectory with the highest backward accuracy on its dual questions as the final output.
As presented in Tab. 4 the experimental results demonstrate that DuPO provides accurate reward signals, effectively guiding models towards correct reasoning, serving as an efficient approach for scaling reasoning capabilities even without training. applying DuPO as a reranking method improves the average performance of Qwen3-4B by 9.3 points, elevating its accuracy from 68.4% to 77.7% without any additional training. Notably, the DuPO-enhanced Qwen3-4B surpasses DeepScek-RI and Claudo-Sonnet4-hhiniie (77.7% vs. 74.9% on average). The impact on DeepSeek-RI- Ra----Iiiiill-weiieeeee Sessill-Qwen-..5B is even more pro- nounced, with an 18.7 point increase (20.0% to 38.7%).
the two challenging AIME benchmarks,
4.2.6 EFFECTS OF TASK DUALITY To verify the efficacy of component selection strategy, conduct ablation study by remov- ing it and training on the entire unfiltered dataset. The results, illustrated Figure reveal a clear trend: removing the selection strategy results in a significant performance degradation across most benchmarks. This is further evidenced by an average accuracy drop of 3.6 and 5.4 percentage points for the 1.5B and 4B models, respectively. The result validates our strategy's effectiveness, confirm- ing that by improving task duality, it provides a cleaner reward signal that is crucial for achieving superior performance.
RELATED WORK
5.1 PREFERENCE OPTIMIZATION FOR LLMS
Preference optimization is pivotal for aligning LLMs with desired behaviors, with current research dominated by three paradigms reliant on external supervision. RLHF (Ouyang et al.) (022) aligns models with human preferences, but is fundamentally hampered by the cost and inconsistency of

---

human annotation (Lee al. 2023 Zhang et al. 2024). As cost-effective alternative, LLM-as-a- Judge (Zheng et al., 2023 Lee et al. 2023) utilizes a powerful LLM for evaluation. The reliability is perpetually undermined by the judge model's own performance limitations and intrinsic biases, such as sensitivity to response ordering or stylistic artifacts (Wang et al.) 2024] Gudibande et al.)
2023
Li al.] (024a). In parallel, RLVR has shown success in domains like mathematics by using ground-truth outcomes as reward signals (DeepSeck-AI, 2025 Team 2025] Yang et al., 2025
However, this paradigm's 's continued reliance on labeled answers external supervision makes it ill- suited for free-form tasks, such as multilingual translation, that inherently lack a single, definitive ground truth. Recent self-supervised paradigms explore alternative reward sources. Some leverage consistency checks between problem paraphrases (Zhang et al. 2025), while others utilize self-play to generate a curriculum (Huang et al. 2025), which requires managing a multi-agent adversarial process. RIT (Cetin et al.) 2025) focuses on training a teacher model to generate more effective distillation data, by rewarding the teacher's explanations based on the student's performance. DuPO differs by sourcing its reward from the intrinsic, dual structure of a task itself and provides reliable and self-contained verification signal, sidestepping dependencies on auxiliary data generation or the complexities of adversarial dynamics.
5.2 DUAL LEARNING
Dual learning enhances model performance by leveraging intrinsic task symmetry, where a primal task and its complementary dual task mutually provide supervision. He et al. (2016) first introduced dual learning for machine translation, which uses bidirectional tasks (e.g., En -Zh and Zh->En) to generate pseudo-labels via back-translation (Sennrich et al. al.]|200..) (211)). reducing reliance on parallel cor- pora-a breakthrough for low-resource language pairs. Building on this foundation, the paradigm has proven highly versatile persatile-spanning multi-modal (Yi al.] 2017: Zhu et al.) 2017 Ren et al.] 2020) and knowledge reasoning (Dognin et al. 2020), and extending to reinforcement learning (Luo) ct al. 2019; Bahng et al., (2225). In modern LLMs, it further refines output quality and enforces semantic consistency (Zou et al., 2025; Chen et al., 2024). However, the reliance strict task duality, requiring mutually invertible tasks -pecludes its application to open-ended or creative do- mains. In this work, we generalize the dual learning paradigm, moving beyond this rigid invertibility constraint.
CONCLUSION
We introduce DuPO, dual learning-based preference optimization framework that eliminates the need for costly human annotations and handcrafted rewards in LLM training. Its core innova- tion, generalized duality, generates self-supervised feedback by decomposing and reconstructing input spaces, addressing critical limitations of traditional dual learning and preference optimization paradigms. DuPO's effectiveness validated across diverse tasks: in mathematical reasoning, boosts average accuracy by 6.4 percent points across models from 1.5B to 7B, while in multilingual translation, elevates a 7B model to rival larger SOTA LLMs with COMET score gains of up to 2.1 across 756 translation directions. Furthermore, models to outperform significantly larger LLMs by up to 9.3 points. This mogel-aggostic and task- versatile design positions DuPO as a scalable, unnotation-efficien solution for more autonomous, adaptable, and cost-effective LLM optimization.
training-free reranker, DuPO enables smaller
REPRODUCIBILITY STATEMENT
To ensure the full reproducibility of our work, we provide a comprehensive description of our methodology and experimental setup. We detail the step-by-step application of our method in Ap- pendix A and the pipeline for training data construction in Appendix B For training and evaluation, Section 4. specifies the large language models and datasets utilized, while Appendix [ lists the corresponding hyperparameters and configurations used. Furthermore, illustrative case studies are presented in Section offer qualitative insights into our method's application. To further promote reproducibility and facilitate future research, we will also open-source our code.
REFERENCES
Mistral AI. Mistral large. http://ts///.tttaaaaaaa/aaa/aaaa
2024. 2024.
Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, and Nick Haber. Big-Math: A large-scale,
10

---

high-quality math dataset for reinforcement learning in language models, 2025. URL https:
//
Hyojin Bahng. Caroline Chan, Fredo Durand, and Phillip Isola. Cycle consistency as reward: Learn- ing image-text alignment without human preferences. arXiv preprint arXiv:2506.0200 02005, 2025.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Ols- son, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li. Eli Tran- Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer EI Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen- Tom Con- erly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback. 2022. URL /ttt/tt:///aatt /ttp:////axvvv.... org/ abs abs/2212.0800
Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanovié, and Martin Vechev. MathArena: Evaluating lims on uncontaminated math competitions, 2025.
ByteDance Seed Team. Seed1 T5Thhnning: Advancing superb reasoning models with reinforcement learning, 2025.
Chris Callison-Burch, Miles Osborne, and Philipp Koehn. Re-evaluating the role of Bleu in machine translation research. In Diana McCarthy and Shuly Wintner (eds.), 11th Conference of the Eu- ropean Chapter of the Association for Computational Linguistics. pp. 249 256, Trento, Italy, April 2006. Association for Computational Linguistics. URL https: /tle://aaall /aapaaaaaaaant........
org/E06-10 0332/
Edoardo Cetin, Tianyu Zhao, and Yujin Tang. Reinforcement learning teachers of test time scaling, 2025. URL Utttsst:s///axxiiti wwwpe/iww.orggbb//55000 08338
Andong Chen, Lianzhang Lou, Kehai Chen, Xuefeng Bai, Yang Xiang, Muyun Yang, Tiejun Zhao, and Min Zhang. DUAL-REFLECT: Enhancing large language models for reflective translation through dual learning feedback mechanisms, 2024. URL/tttps://axxi// org/ absp//bb//424
07232
Yang Chen, Zhuolin Yang. Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. AceReason-Nemotroon Advancing math and code reasoning through rein- forcement learning. arXiv preprint arXiv:2505. 16400, 2025.
Shanbo Cheng, Yu Bao, Qian Cao, Luyang Huang, Liyan Kang, Zhicheng Liu, Yu Lu, Wenhao Zhu, Jingwen Chen, Zhichao Huang, Tao Li, Yifu Li, Huiying Lin, Sitong Liu, Ningxin Peng. Shuaijie She, Lu Xu, Nuo Xu, Sen Yang. Runsheng Yu, Yiming Yu, Lichao Zou, Hang Li, Lu Lu, Yuxuan Wang, and Yonghui Wu. Seed X: Building strong multilingual translation 1lm with 7b parameters, 2025. URL httpsss/aarxi..orr///////2.111
DeepMind, Gemini 2.5. htttos//deeepllll.ooooe////////s Accessed: 2025-04-18.
2025.
DeepSeek-AL DeepSeck-RI: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025.
Pierre L. Dognin, Igor Melnyk, Inkit Padhi, Cícero Nogueira dos Santos, and Payel Das. Dualtkb: A dual learning bridge between text and knowledge base. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 8605-8616. Associ- 694. URL
ation for Computational Linguistics, 2020. doi: www.pp.........pppp.mm...p
694
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. ArXiv preprint, abs/2407.21 277783,, 2024. URL https: arxiv. org/ abs/2407.
21783
11

---

Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprictary llms. CoRR, hbsps/2335.15111 2023. doi: 1044550/ARXI.2305555777 URL 2305. 15717
Di He, Yingee Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Yan Liu, and Wei-Ying Ma. Dual learning for machine translation. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Sys- tems 29: Annual Conference Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 820 828, 2016. pttppp/peeepppooctiiiioooooooooe paper 201 /tt.///....555000233300000000110
Abst .ttml
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng. Tianwen Wei, Cheng Cheng, Yang Liu, and Yahui Zhou. Skywork open reasoner series, 2025. Notion Blog.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. ArXiv preprint, www//113.03872. 2021. URL htthttt:::
0
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. REINFORCE+:: An efficient rlhff algorithm with robustness both prompt and reward models, 2025a. URL htt.hhtss///rxiii.orrggg abs/
2501.003 03262
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. reen-reasoner-zero:: An open source approach to scaling up reinforcement learning the base model, 2025b. httllttts://axi......
aas///b///// 2503.50322 24290
Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-zero: Self-evolving reasoning Ilm from zero data, 2025, URL
htttp::/arxiv.. httpp////raaii.og/////////////// 0500000000000000
Yunjie Ji, Sitong Zhao, Xiaoyu Tian, Haotian Wang, Shuaiting Chen, Yiping Peng. Han Zhao, and Xiangang Li. How difficulty- staged reinforcement learning enhances Ilms' reasoning capa- bilities: A preliminary experimental study, 2025. URL
abs/g//bbs/5000
00829
Ruipeng Jia, Yunyi Yang. Yongbo Gai, Kai Luo, Shihao Huang, Jianhe Lin, Xiaoxi Jiang, and Guanjun Jiang. Writing-zero: Bridge the gap between non-verifiable tasks and verifiable rewards, 2025. URL
org httpg////bb//.....///////// 22500000033
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Vic- tor Carbune, and Abhinav Rastogi. RLAIF vs. RLHF: Scaling reinforcement learning from hu- man feedback with ai feedback. In International Conference on Machine Learning, 2023. URL wtw.app.seeaaannscc..osos.ee
https://ags hwppp/pp.... www.stmatttttee icscholar www.aaarrrrrrCoooeeeeee
Dawei Li, Bohan Jiang, Liangjie Huang. Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang. Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. From generation to judgment: Opportunities and challenges of llm-as-a-judge. CoRR, 11./2411.11594, 2024a. doi: 10.48550/ARXI..211...
URL D 4855:////......500..........
2411 1594
Jiahuan Li, Shanbo Cheng, Shujian Huang, and Jiajun Chen. Mt-patcher: Selective and extendable knowledge distillation from large language models for machine translation, 2024b. URL https:
7/arxiv. wrg. abs 2403. 09522
Jiahuan Li, Hao Zhou, Shujian Huang, Shanbo Cheng, and Jiajun Chen, Eliciting the translation abil- ity of large language models via multilingual finetuning with translation instructions. Transactions of the Association for Computational Linguistics, 12:576-599 592, 2024c. doi: 10. 1021ttaac URL htt.tt:://acaattholae
org/ 2024 tacl 11..-1.33 327
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang. Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseck-vy v3 technical report. ArXiv preprint, abs/2412, 19437, 2024. |Rthttts. /ttp::///aiiiii abbs//////// /211. 19437
12

---

Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Xu Sun, and Zhifang Sui. A dual reinforcement learning framework for unsupervised text style transfer. In Sarit Kraus (ed.), Proceedings of the Twenty-Eighth International Joint Conference Artificial Intelli- gence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 5116-5122, ijcai.org, 2019. doi: 10.24963/IICAI.201999
UKI.https:///o..rrg//130
MAA. Ame 2023 problems, 2023. URL httppss//arrrfppooooomooooooooao
index. php/2023 _0000020_______000220200002022 12A Problems
Accessed: 2025-05-11.
Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. AIMO-2 Winning Solution: Building state-of-the-art math- ematical reasoning models with openmathreasoning dataset, 2025. URL https://arxiv:
org/abs/ 2504 22504166880 16891
NLLB Team. Scaling neural machine translation to 200 languages. Nature, 630(80188:441-846 2024. ISSN 1476-4687, doi: 10.003544156--004--73355- URL https:///bi.ozg//10 1038 0038/44558-0244-177
OpenAl. Hello 2025-04-18.
2024. Accessed:
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL ftlt:p://arxiv.ort wttp://r//aa/........./ 02155
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp- 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguis- URL http:pppe/aaallaatttooo......//a
tics. doi: 10.3115/1073303.1073311
10407
Ricardo Rei, Craig Stewart, Ana Farinha, and Alon Lavie COMET: neural framework for mt evaluation, 2020. htt.l//pee///arrii.orr///a/a///p 09025
Yi Ren, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu, Almost unsupervised text to speech and automatic speech recognition, 2020. URL 06791
wtwpvvv..rg/bb//190000 abs/1905.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707. 0074076477 2017. URL http:://axxia wrwleoorg/eee abs/
1707. 06347
Thibault Sellam, Dipanjan Das. and Ankur Parikh. BLEURT:
Learning robust metrics for text generation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881-7892, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2022 main. 704. URL. wtt.t:ttt//......... //a:: antho wrwpy//0.g000000. -ma i
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation mod- els with monolingual data. CoRR, abs/1511.06700 2015. URL http:://axii...tt//////// owtpb//.orrg////////////
1511.06709
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li. Y Wu, et al. Deepseek Math: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Kimi Team. Kimi k1.5: Scaling reinforcement learning with 1lms. ArXiv, abs/2501.129999, 2025.
URL htl.htt::: /apii www.m//ieechh......oooo/oppsappa
13

---

Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong. Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Anmual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pp. 9440-9450, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/ 2024.acl-long.511 utthttps: htttt:/://cllattooooooooo/o///
Yijun Wang, Yingce Xia, Li Zhao, Jiang Bian, Tao Qin, Guiquan Liu, and Tie-Yan Liu. Dual transfer learning for neural machine translation with marginal distribution regularization. In Sheila A.
Mcllraith and Kilian Q. Weinberger (eds.). Proceedings of the Thirty-Secont AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI- 18), and the Sth AAAI Symposium on Educational A Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 5553 5560. AAAI Press, 2018. doi: 10.1609/AAALV3211 11999. URL http:: http::://ooo.org////// 1609/ /wtp/aaal.../ v32il 11999
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling, 2025. WRLhttps:://
org/ abaa htlbs//556.200
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. ArXiv preprint, 2oss24422.11155 15, 2024a. htRp:tts:: arxiv. org/ abs /
/ 2212/44222..0 1144121150 15115
An Yang. Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-Man Technical Report: Toward mathematical ex- pert model via self-improvement, 2024b. URL htttttttt///axiiii wwtp:/o///////aa...o..oo..
12122
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https: / org/ abs/ 0 09388
Zili Yi, Hao (Richard) Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-image translation. In IEEE International Conference on Computer Vision, ICCV 2017, Venice Italy, October 22-29, 2017, pp. 2868 2876. IEEE Computer Society, 2017. doi: 10. 10.1109/1CCV.201..33
doi http://oo..ogg///////// 100 1109/ICCVV 2017. 310
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin. Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjic Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song. Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: An open-source IIm reinforcement learning system at scale, 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025.
URL http:tttp//aaaae /ttp://aaxvvvvv.. org/ abs/ wwtp.////w/.b//........ 18892
Michael JQ Zhang, Zhilin Wang, Jena D. Hwang, Yi Dong, Olivier Delalleau, Yejin Choi, Eunsol Choi, Xiang Ren, and Valentina Pyatkin. Diverging preferences: When do annotators disagree and do models know?, 2024. URL WtRL.hhtps://aal www.rxxivvorr/aaa. hbtp://b///21... 14632
14

---

Zizhuo Zhang, Jianing Zhu, Xinmu Ge, Zihua Zhao, Zhanke Zhou, Xuan Li, Xiao Feng, Jiangchao Yao. and Bo Han. Co-reward: Self-supervised reinforcement learning for large language model reasoning via contrastive agreement, 2025. URLAttps://axii.orrg wtt.orggg/aaaa//.//
00410
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL htttp://papers.at nips cc cctpaper.pppppeeeeeepeeeeeee...e www.ss/ppeeeeeaaaeaa Benchmarks. html
_nd
Yun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. Multilingual machine translation with large language models: Empirical results and analysis, 2024. URL
Wei Zou, Sen Yang, Yu Bao, Shujian Huang, Jiajun Chen, and Shanbo Cheng. TRANS-ZERO: Self-play incentivizes large language models for multilingual translation without parallel data. In Wanxiang Che. Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings the Association for Computational Linguisties; ACL 2025, pp. 12337-12347, Vi- enna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256- 5. doi: 10. 18613166611225555 htw.://///2.....nn---o-aaa Anddings-acl.637 URL https: http://aaaaaatclcoorrroooo org/222 Findings acl. 637/
15

---

A CONSTRUCTION OF DUAL QUESTION FOR MATH REASONING
We propose simple approach for construction of dual question for mathematical reasoning. The algorithm operates on mathematical expressions and performs the following key steps:
1. Pattern Recognition and Exclusion: The algorithm first identifies numerical candidates within the expression while excluding numbers in specific contexts: subscripts (æ1, (x), in- equality constraints (x 2 5), common exponential bases (2", 10"), and function arguments (f(3)).
2. Variable Generation and Replacement: For each valid numerical candidate, the system generates a unique variable identifier of the form Variable; where str is a randomly generated lowercase string. The original number is then substituted with this variable.
3. Question Generation Dual Task:
Using the transformed expression and the original answer, the algorithm constructs inverse problems following templates such "Given that the correct answer is {answer), }, determine the value of (variable)."
This methodology enables systematic generation of problem variants while preserving mathematical validity and semantic coherence. From a single primal question, multiple dual questions can be derived. To ensure that these dual questions robustly satisfy the properties of duality, we filter the candidates using the following two principles:
1. Answerability of the Dual Question: For the set of sampled answers collected for a given primal question, at least one answer must be capable of correctly solving the corresponding dual question.
2. Uniqueness of the Correct Completion: Among the same set of sampled answers, at most one should correctly answer the dual question.
Taken together, these two principles ensure that for any selected dual question, there is one and only one correct answer within the pool of candidate solutions for the primal task. This establishes the one-to-one correspondence necessary for generating reliable self-supervised reward signal.
B MATH RL DATASET PREPARATION
Our dataset preparation process began with the collection of 1,815,942 prompts from various pub- licly available datasets (Chen et al., 2025 Albalak et al.) 2025 He et al. 2025 Ji et al.] 2025). After deduplication,
obtained 318,649 649 primal questions and generated 1. 1,59,67 dual questions
through our designed steps as discussed above. After that, we employed Qwen2.5-78 2 (Yang al. 2024a) sample 32 candidate answers for each primal question and then prompted it to an- swer the corresponding dual question based on these candidates. Subsequently. we rigorously fil- tered out all dual questions that failed to meet our predefined principles above. We repeated this sampling and filtering process with Qwen3-4B (Yang et al., (2025,, this time with 8 candidate an- swers per question. The resulting collection of high-quality, diverse mathematical questions along with corresponding dual questions formed our final RI. training set, providing a robust foundation for our reinforcement learning process in the mathematical domain.
C EXPERIMENT DETAILS
We present more details about our training as follows. For the training process, we use a train hatch size of 512, mini batch size of 32, sampling temperature of 1.0, and 16 rollouts per prompt. We employ the AdamW W optimizer with learning rate of le-6 and weight decay of 0. (following the default configuration in verl). with gradient clipping set to 1.0. For translation tasks, we the maximum input length to 2,048 tokens and output length to 4,096 tokens. For mathematical tasks, we use the same input length but extend the maximum output length 30,000 tokens.
D CASE STUDY
To illustrate the efficacy of our DuPO approach, we present two representative scenarios in Tab. that demonstrate how DuPO provides reliable reward signal across diverse domains.
Scenario 1: Mathematical Reasoning Validation. mathematical reasoning, DuPO derives dual task questions from the primal task question where key numerical parameters replaced with
16

---

Scenario DuPO on Mathematical Reasoning
Primal Task
Let AABC have circumcenter O and incenter / with IA - 01, circumradius 13, and inradius 6. Find AB AC. (Correct Answer: 468)
Dual Task #1
Let AABC have circumcenter O and incenter with -A-OI,
circumradius Vak: and inradius 6, Find AB AC. Check your work: the solution for above question is answer what must VsR have been?
Dual Task #2
Let's examine: Let AABC have circumcenter O and incenter with IA 1 O1, circumradius 13, and inradius V,j. Find AB - ACC. When the solution for above question is answer what's the corresponding V,
Candidates
Answer: 468
Backward Accuracy: 69.1%
Answer: 108
Backward Accuracy: 0%
Answer: 312
Backward Accuracy: 0%
Scenario 2: DuPO on Machine Translation (MT)
Primal Task
Translate to Chinese: As knowledge of Greek declined, the West found
itself cut off from its Greek philosophical and scientific roots.
Reference
2 JT
Ri
Primal MT #1
- T (BLEU: 45.85)
Dual MT #1
As knowledge Greek declined, the West found itself cut off from its philosophical and scientific roots in (BLEU: 82.07)
Primal MT #2
(BLEU: 28.65)
Dual MT #2
As understanding of the Greek language gradually fades, the West finds itself cut off from the roots of Greek philosophy and Science.(BLEE) 16.11)
Table 5: Case Studies of DuPO on Mathematical Reasoning and Machine Translation. DuPO validates each candidate's 's quality through corresponding dual task, reliably identifies the superior solution over inferior ones.
variables, and the model tries to work backwards conditioned on candidate answers. When given geometry problem about triangle properties, three candidate answers are sampled: 468, 108, and 312. DuPO automatically derives two dual questions by replacing the circumradius (13) and inradius (6) with variables, asking the model to deduce these values from the proposed answer. The candidate answer 468 achieves 69. % accuracy on dual task, while the incorrect answers (108 and 312) totally fail to answer the dual task.
Scenario Machine Translation Quality Assessment. For translation tasks, DuPO leverages re- verse direction translation as the dual task to evaluate translation quality. Given English sentence about Greek philosophical decline, two Chinese translation candidates are generated and subse- quently back-translated English. The first translation achieves BLEU score of 45.85 in the forward direction and 82.07 in the mack-translation, demonstrating semantic preservation and trans- lation fidelity. In contrast, the second candidate shows degraded performance with BLEU scores of 28.65 and 16.11, respectively, indicating semantic drift and poor translation quality.
These case studies validate DuPO's core hypothesis: high-quality solutions maintain consistent in- formation across dual task formulations, while inferior solutions exhibit significant degradation. This dual validation mechanism provides robust framework for automatic quality assessment with- out requiring ground truth labels.
E LIMITATION DISCUSSION
DuPO has demonstrated its effectiveness on mathematical reasoning (representing complex reason- ing tasks) and machine translation (representing constrained generation tasks), providing novel
17

---

pathway for self-supervised training in LLMs. However, fully open-ended tasks remain inherently challenging. Tasks such as creative writing, where outputs are highly unconstrained and evaluation criteria are subjective, require further exploration effectively extract the unknown component and construct meaningful dual problems. Intuitively, taking creative writing as an example, specific con- straints in user instruction (e.g., theme, style, tone) might be extracted as the unknown component Ta for reconstruction. However, the practical implementation and validation of such mechanisms in these open-ended domains constitute important directions for future work.
THE USE OF LARGE LANGUAGE MODELS
We utilized Large Language Model (LLM) as an assistive tool to proofread the manuscript for grammatical errors and to provide suggestions for improving clarity and flow. All suggestions were manually reviewed by the authors, who made the final decisions on all textual modifications to enhance the paper's s readability.
18


Reviews:
[{"review_id": "OXpuW5CjTC", "reviewer": "Reviewer_S26D", "summary": "The authors introduce DuPO, a consistency-based preference optimization framework which exploits task duality to increase post-training robustness. The authors achieve success over existing baselines on math datasets, and extend their framework to work at inference-time as well.", "questions": "- Please address weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written, easy to follow, and the figures are detailed with the appendices including relevant information and concrete examples\n- The idea of using dual problems in RLHF is interesting and (to my knowledge) novel\n- The experimental results appear rather promising", "weaknesses": "- My primary concern with DuPO is regarding the difficulty of constructing the dual problems, verifying the correctness of those duals problems, and how generalizable it is to different domains. \n- In addition, it seems rather expensive to have to run DuPO at inference-time since the dual problems would also need to be generated on the fly?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "77vZk4cWNp", "reviewer": "Reviewer_EbeA", "summary": "This paper proposes DuPO (Dual Learning-based Preference Optimization), a novel framework for optimizing Large Language Models (LLMs) without external supervision. DuPO addresses the limitations of traditional dual learning—which requires strictly invertible task pairs—and Reinforcement Learning with Verifiable Rewards (RLVR)—which relies on costly ground-truth labels. The core innovation is a generalized duality framework: it decomposes the input of a primal task into known and unknown components, and then defines a dual task that reconstructs only the unknown component using the primal output and the known input. The quality of this reconstruction serves as a self-supervised reward signal to optimize the primal task. The authors demonstrate DuPO's effectiveness on both mathematical reasoning and multilingual translation, showing significant performance gains during training and as a training-free inference-time reranker.", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Fully Self-Supervised: The method requires no human labels, reference answers, or external reward models, which is a major advantage for scalability and cost-efficiency.\n2. Practical Impact: The ability to use DuPO as a simple, training-free reranker that boosts performance by +9.3 points is a highly practical contribution.", "weaknesses": "1. Insufficient experiments: In most experiments, DuPO is compared to untrained models, which is unfair. For translation tasks, RLVR may be difficult, but it is still worth using as a baseline. For mathematical tasks, standard RLVR should be added as a baseline. In addition, SFT can be added to each benchmark as a baseline.\n2. Task Construction: While the paper provides principles for selecting the unknown component (Appendix A), the process still seems somewhat heuristic and task-specific. I still have concerns about the generality of this method on other tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6xYeKVcukD", "reviewer": "Reviewer_tLFR", "summary": "The paper proposes DuPO, a self-supervised preference-optimization framework that replaces external reward signals (human or verifiable answers) with a generalized duality signal. The idea is to decompose each input into known and unknown parts $(x_k, x_u)$ and define a dual task that reconstructs only the unknown parts from the model’s output and the known context. The reward is the distance between the true $x_u$ and its reconstruction. DuPO then optimizes the promal policy $\\pi_{\\theta}(y|x)$ with standard RL updates (GRPO in experiments). The authors conduct the experiments on translation, math reasoning, and inference reranking tasks.", "questions": "* How should the inputs be decomposed into known and unknown parts in tasks such as creative writing? \n* How does the choice of known and unknown parts affect the reward? For example, if the unknown parts are too simple, can the model predict them from the known parts alone?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "* The idea of integrating dual learning into self-supervised LLM fine-tuning is relatively new and underexplored.\n* The authors conduct experiments across several tasks, including both LLM fine-tuning and reranking.\n* The paper includes a nice ablation study on component selection.", "weaknesses": "* Although the idea of dual learning is relatively new, the motivation for why it improves performance remains unclear. The notion of generalized duality is defined axiomatically, but there are no guarantees showing that optimizing the DuPO reward leads to better true task utility, nor theoretical bounds relating to the choice of decomposition. For tasks like machine translation, the intuition makes sense as the forward and backward tasks are naturally aligned, but for other tasks, it is unclear why DuPO would outperform existing methods such as RLVR or other self-play approaches.\n* Several claims in the paper are problematic. For instance, the authors state that RLVR and RLHF rely on costly labels. However, constructing an appropriate dual task for DuPO may be even more challenging. Moreover, this may not even be possible for tasks such as creative writing. While the authors briefly mention the creative writing task, they do not discuss how it could be resolved.\n* Reward design: It is unclear how the reward is defined when there are multiple valid outputs for the reconstructed unknown. For example, in a math problem with a single correct answer but many possible solution paths, how is the reward computed consistently?\n* Baselines: The paper lacks RLVR baselines on the math reasoning task, as well as self-play LLM baselines such as [1-4].\n\n[1] Chen, Zixiang, et al. \"Self-play fine-tuning converts weak language models to strong language models.\" ICML 2024.\n\n[2] Pang, Richard Yuanzhe, et al. \"Iterative reasoning preference optimization.\" NeurIPS 2024.\n\n[3] Dong, Qingxiu, et al. \"Self-boosting large language models with synthetic preference data.\" ICLR 2025.\n\n[4] Wu, Yue, et al. \"Self-play preference optimization for language model alignment.\" ICLR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HwmYfdE5uI", "reviewer": "Reviewer_P36R", "summary": "This paper introduces a self-verification reward idea for reinforcement learning in LLMs. The general setting is given a prompt $x$, for the LLM to provide an answer $y$, as $\\pi(y|x)$. The core idea of dual learning is to use the prediction of the prompt from the answer as a dual self-supervised training signal. The authors observe that it is not sufficient or possible for many tasks, identifying non-unique reconstruction or failure to reconstruct as core problems. To tackle this, they propose to decompose the prompt $x$ into  a known and an unknown parts, $x = (x_k, x_u)$, to use a model to predict the unknown part from both the answer and the known part, $\\hat{x}_u = \\mathcal{T}(y,x_k)$, and to use this to define a (self-verification) reward signal $r(x,y) \\propto \\exp(-\\lambda d(x_u, \\hat{x}_u))$, with $d$ being a task-dependent distance function. This reward is then optimized using RL. The proposed approach is then quite thoroughly experimented.", "questions": "I'm supportive of accepting this paper, which is really great, provided that the questions below are answered, especially those regarding clarifying the proposed approach (which is probably a rather minor modification)\n\n### More details on the proposed approach \n\n* The paper currently doesn’t say what model is used for generating $x_u$ from $y$ and $x_k$. From the context, it is likely the LLM being trained, but is it the initial frozen LLM, the LLM being trained (as training progresses), or even another LLM?\n* In all the above cases, the reward being optimized depends heavily on the capacity of the LLM to provide correct answer to the dual problem (predict $x_u$). If it’s not capable of making that prediction correctly enough, there is not learning signal. Can you comment on this? Notably, for the Llama experiment (table 3), the initial score is pretty low, suggesting that probably predicting $x_u$ may not be good enough, which makes the huge improvement quite surprising. Or is it that predicting $x_u$ is much easier than predicting $y$?\n* Is there a mechanism for improving the prediction of $x_u$ (if the current LLM is used for building the reward function)? Or is it a byproduct of improving the prediction of $y$? This would be an interesting experiment to add, measuring the initial reward quality (given the ground truth $y$), and how it evolves along learning, once the precise setup has been clarified.\n\n### Reproducibility details\n\nSome parameters are not provided, such as the optimizer, learning rate and other related parameters, but more importantly the proposed reward function relies on hyper parameters that are not provided, and they should be provided but also ideally the sensitivity of the approach to their value should be studied. Notably, the choice of $\\lambda$ and of the distance $d$.\n\n### Other comments and questions \n\n* What is called preference optimization (eg title of sec 3.3) is not preference optimization, it’s RL. It is semantics, but also misleading. \n* Everywhere but in Eq (1), the reward does not depend on $x$, which is hopefully a typo\n* A paper that would be interesting to discuss (no experiment needed) is “Reinforcement Learning Teachers of Test Time Scaling” by Sakana. They address a different problem, but there is some interesting similarity in the idea of decomposing the prompt and predicting part of it.\n* Sec. 4.5.2 (scaling reasoning during inference without training) is very interesting. Please discuss the additional cost. Also, an experiment that would be quite interesting would be to run this on huge close-source strong models like the latest R1, through the API (as there is no learning), to see the amount of possible improvement there.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is very well written and structured, a pleasure to read\n* The idea of decomposing the prompt for extending applicability of dual learning, and using it as self-verified reward signal is simple yet elegant and very interesting \n* The experimental section is thorough and the results seem to be quite strong", "weaknesses": "* The approach is presented as being very general (with mentions to code generation, dialogue systems, open-ended and creative domains), while it is strongly dependent on the prompt decomposition which has to be defined manually for every task. Math reasoning and translation are the considered tasks, and only math reasoning calls for such a decomposition. This is fine, but this is a limitation that should be acknowledged and discussed. \n* If the paper does a very good job at providing the high-level intuition, it somehow fails to provide a precise enough description of the proposed approach, harming understanding and reproducibility. See questions for more details.\n* There are missing details harming reproducibility. The authors promise for open-sourcing the code, which is very nice, but some additional details could be provided in the appendix. See questions for more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)